Saturday, August 28, 2010

The Immaturity of Programming for the Internet

Although we are a couple of decades into the age of the internet, some lessons still have not been learned by some programmers and their management. Some sites, even multi-million and multi-billion dollar sites, occasionally confront disaster when releasing a new version of something that has been working well.

Mainframe processors went through that a lot in the 1970's, a time when applications were getting larger and dealing with higher volumes, and when remote users began to have access to host systems.

The reasons then and now, I suspect, were twofold: an unwarranted confidence in an organization's ability to make system changes and a reluctance to spend money - quality control is expensive.

One way to divide the universe of a data processor's responsibilities to users is:
  1. Systems currently in production and *large* systems being developed for production.

  2. Smaller one-time jobs.
It is critical that those in the first category be vetted by someone independent of the development, and in fact independent of the data processing area itself. One organization I worked for had a quality control person who vetted every single change to software used by clients. All proposed releases went through this QC person, and if problems were found the data processing area was notified and the release rejected.

No one in the company had the authority to make him change his mind.

The problems had to be fixed and the release resubmitted to him.

Another company I worked with - not as an employee, but in tandem with - had one person who was paid six figures and whose sole responsibility was to tell the company when to change hardware and when to change operating systems. Under no other circumstances could anyone else in the company - throughout the world - replace a mainframe or an operating system.

Such precautions are expensive but like the mills of the gods they grind exceeding small. In all the years I knew them, neither of these companies ever had a major problem that their controls were designed to prevent.

Contrast that with one large auction site, for example, which several years ago put into production a new billing system. They "tested" it in production by picking half (I think) of their sellers, leaving the others alone. For *months* the site could not bill the half under the new system. What was the cost in lost revenue, lost interest, and perhaps even lost sellers?

Never should have happened.

Never should have happened.

Never should have happened.

Another large site, one that pays people to write articles, is currently approaching the death rattle stage. Whatever possessed management I don't know, but it was decided that the current software be replaced using a new software package. Worse yet, the new software package had not been released commercially. Worse than that, even, was that the package hadn't even been beta tested.

Now when writers try to post an article it comes out garbled. Paragraphs appear in random order, functions that are supposed to work fail miserably, and "hit counts," the basis on which writers are paid, are hopelessly muddled. And readers are staying away in droves. How many will never return, even after things are stabilized?

In addition to a complete lack of quality control, a reason for failure was something that mainframers learned the hard way too, about using new products: Never be first. Never be last.

I commend that to you regarding your PC, your Mac, your laptop, your internet service: Never be first. Never be last.

Now I must confess that when I was a mainframe assembler programmer there were times when I was a cowboy. I did my own testing on programs that I wrote and ultimately pronounced them fit. I must confess also that occasionally, I (and my employer) paid for it. But these were all category two items, one time jobs, quickly fixed, sometimes not even seen in their problem state by the client.

We learned, and they will too.