28-Nov-2007: The Boondoggle
I'm going to skip ahead here a bit in my saga of the computing revolution and my very, very small part in it,
and go to the late 1980's after I had left the government and become a consultant. At first, I worked as an employee
for an outsourcing company now called ISM, Inc. My job was to go to computer sites where ISM had contracts to supply
computer support. At first, they had hired me to go back to Energy, Mines, and Resources to take over my old job there,
as EMR simply did not have any VMS expertise on staff after I had left. So, for about six months I was back in my old
job, making roughly the same amount of money, except that now my employer was external to the civil service. There was
a union grievance filed when I returned, but it went nowhere. Skill-sets are skill-sets and union seniority was irrelevant. (There had been
a union protest when I was first assigned to manage the VAXes because, even though they knew nothing about systems management,
people with more seniority in the union felt that they were more entitled to the position than I was.)
After a few other short contracts, my ISM boss suggested I go to the Department of Supplies and Services (DSS), now called PWGSC, and
see if I could help out. DEC had the computer hardware and support contract for a project with DSS, but had subcontracted operator-support
to ISM. Apparently the program had been under development for a year and there were still serious performance issues. Maybe I could take a
look?
I fixed the performance issue within a few minutes after logging on to the system. I had been asked to create my own working
account and I was appalled by the default parameter settings the system had given me. "There's no way I can work with this!" I exclaimed. So, getting
the go-ahead I adjusted the account parameters for everyone, setting them to more reasonable limits. The phone started ringing immediately with users
asking what we had done; the system was now working as it was supposed to. I was called into the Director's office and asked if what I had done
to cause the system to work was common knowledge among system managers. I answered that it was basic and that anyone calling himself a system
manager should be very familiar with the parameters whose settings I had corrected. Shortly afterwards, the VMS system manager, a Digital
employee, came to tell our group that she had been asked to leave the site.
Apparently Digital had been advising DSS for the past year to purchase more hardware and buy their in-house systems tuning software and DSS,
rightly, had refused, saying that Digital had contracted to supply a working system for a given price, so don't give us any of this crap about buying
more hardware and software. When I had fixed the systems by performing a very simple and basic setup change, the project management contract with Digital was torn up
and handed over to ISM. I was there for more than three years with nothing much to do except tweak the systems now and then.
It was during this time that ISM landed a very small contract to analyse the problems that the Department of External Affairs was having with their new
VAX installation. This one was in the newspapers who were complaining that External Affairs had spent more than $265,000,000.00 on a computer system that
did not work and wanted to spend roughly another $300,000,000 on it. Its purpose was to provide a secure world-wide link between Canada's embassies
overseas and head office in Ottawa. I was asked to go in and do two things: 1) determine whether the software acceptance
testing would need to start over from the beginning after the contractor had made major changes to the software; and 2) look at the system and write up a list of
recommendations on how to improve performance. Everyone I spoke to at External Affairs was defensive and upset about the newspaper stories, taking care to explain
to me that it was a two phase project and the $265,000,000.00 had been for the first stage only.
That story was of no interest to me. I wanted a look at the systems. However, security was so tight that I was not allowed to touch a keyboard. I had to
sit at least an arm's length away from the security officer and dictate to him what to type. (Though I have worked in a military research station and in the Prime Minister's
office, I have never seen such detailed security.) As with DSS, I spotted the problem right away. It didn't take brain surgery to realize that a batch job running
at an elevated priority was blocking interactive access to the system. When I pointed this out, I was told that the job was necessary because what it was doing
was checking email messages to see if they had the string "***Urgent***" in them and so could flash a message on the recipient’s screen. In answer to my inevitable and
annoying habit of asking, "Why?" I was told that it was a key deliverable. "Why so high a priority?" The answer: because the contract said that email had to be scanned every
five minutes and the program took so long to executed that it could not complete within five minutes unless they raised its priority.
Show me its source code. Sure enough, it was the poorest designed program I had ever seen and, to add insult to injury, it was written in DCL, a command line
interpreted language. What the code did was search out the mailbox of every account on the system and search through all its stored messages looking for the
magic word. It didn't matter that the account was a batch or system one and so never had any email, or if the messages it was searching every five minutes were six months
old. Even if the program had been written in FORTRAN, one of the main systems working languages of the day, it would still take forever to execute. Opening files and searching
for text strings is a very slow event in a CPU's life, seeing as most of the work was being done at the disk level millions of times slower than a CPU.
While there, I also looked at the acceptance tests that had been done and remained to be done, and examined the changes the contractor had make. There was no doubt in my
mind that the change was significant enough that all the previous successfully-run tests had to be re-run.
Then I went off to meet with the contractor for the system. Fenco Engineering. As far as I knew they built bridges, not computer systems. I met some very nervous
people there and then was introduced to their programmer. After talking with him for a few minutes I realized that he had no concept of what he was doing. He had very little
programming experience and had no idea what I was talking about when I asked him why he wasn't using the system service calls (functions built into an operating
system to give programmers access to its basic operations). He worked for the competition so I wasn't going to give him any more hints than that. His problem though,
was far more basic than knowing how to use the tools available in the system: he was thinking backwards. To do what the contract obliged his program to
do did not require a batch job continuously running. Nor did it involve reading through tons of old messages. What he wanted to do was intercept new incoming messages
and scan for the "***Urgent****" string in them. Old files were irrelevant. If he had written an interrupt-driven piece of code in a compiled language and done it properly,
the program would run only when a new message entered the system, and the message would be scanned in its binary form in micro-seconds. No more performance issue because of this requirement in
the contract.
So, I wrote up my suggestions for improving performance, including a detailed description of this one piece of code that was killing their hundreds of millions of dollars
system and met with External officials. I told them in plain English why the acceptance testing had to be re-run. "I like how this guy thinks," observed one of them.
Then I gave them a run-down of my recommendations for improving performance. They thanked me, and I went back to drinking coffee at DSS.
Not long after that, the program was axed. Again, banner headlines in the newspapers about how much money had been squandered. I don't know how External Affairs
keeps in touch with our embassies now, and I don't want to know, but chances are very slim that there are any VAXes in External Affairs buildings. One thing that bothered
me was how an engineering company with no large systems experience had won such a huge contract. I wondered about that for a few years until I read Stevie Cameron's
book, On the Take: Greed and Corruption during the Mulroney Years. She devoted an entire chapter to the External Affairs boondoggle. (To those who aren't familiar with the
term in its Canadian usage: a boondoggle is a scheme where a politician uses his position and influence to benefit his friends, especially financially, when that
project is unnecessary or highly over-priced. An example would be approving the building of a bridge that leads nowhere.)
External Affairs and the Department of Supply and Services weren't the only places where I saw huge contracts in serious jeopardy because of the inexperience
of someone in a key position. However, the prevailing attitude seemed to be that the big bucks were reserved for the "managers" who prided themselves on taking
the "10,000 foot view" (which means, in reality, that they had no idea how the individual parts of a system fit together, nor what was required to
make them work), and one spent as little as possible on the people performing the essential tasks, like writing computer code and managing the systems on
a day-to-day basis. Whenever there were cuts to be made, instead of getting rid of one MBA who had no experience with reality, they'd cut five programmer and support
positions (for about the same salary savings). And, when systems starting failing as a result, the solution was to hire more MBAs, though this time in their role as
consultants.
I think that attitude harkened back to the earliest days of computing in the 1940's and 1950's. Before Rear Admiral Grace Hopper developed the first compilers in the 1950's,
"programming" was piecing together the binary instructions for a CPU to process. It was male scientists and mathematicians (there simply
weren't many female scientists) who designed the wiring and circuitry, but they left it to the, largely, female "operators" to actually implement the instructions.
These operators were the first "computer programmers," and they often came up with ingenious ways to code the instructions for the CPU. Even the first commercially viable computer
language COBOL was created by a female, Rear Admiral Hopper. What I am suggesting is that, right from its earliest beginnings, the computing world was
divided between those who designed and those who implemented. And, because the division was mainly along sexual lines,
the designers (the males) were paid many times more than the implementers (the females). This basic attitude still exists in today's computing world, even though
the original sexist justification no longer exists. Those who can do are treated like expendable servants; those who can't are the managers.