“What if a demon were to creep after you one night, in your loneliest loneliness, and say, ‘This life which you live must be lived by you once again and innumerable times more; and every pain and joy and thought and sigh must come again to you, all in the same sequence. The eternal hourglass will again and again be turned and you with it, dust of the dust!’ Would you throw yourself down and gnash your teeth and curse that demon? Or would you answer, ‘Never have I heard anything more divine’?”
Today’s story is an amusing anecdote from a project that goes way back in time. It probably doesn’t have any element of learning to it, but it’s an OSS story that still brings a wry smile to my face and a shake of the head when I recall it.
We had an enigmatic release manager named Russell who had been tasked with introducing a major database and application change that was the culmination of months of work. The release had been packaged up by head office in 6 separate parts and it had been thoroughly tested before being sent to us (and Russ) at site.
But on the day of release, things started off well but then went horribly awry, ending with a system shut-down. The change was quickly rolled back and the diagnosis was sent back to head office for evaluation. Head office made a couple of minor tweaks, but basically couldn’t find any significant problems.
Russell was given a new release and asked to try again. He tried again and the same thing happened. Another quick roll-back and more scratching of heads. Again, the logs were completely different from what head office was expecting.
Then a seasoned veteran named John from our SME team (who had little knowledge of the release process, databases, applications, etc.) decided to sit with Russell and just watch what he was doing (and act as a calming influence) as he ran through the release process for a third time.
Always the nervous type, Russell was literally twitching when stepping through the run-list. Each step and each command was fully documented so John just watched Russell cut and paste from the release notes in the following manner:
- Step 1 – ran successfully and the logs were analysed, showing no issue
- Step 2 – ran successfully and the logs were analysed, showing no issues
- Step 3 – ran successfully and the logs were analysed, showing no issues
- Step 5 – was just about to be launched by Russell when John jumped up, exclaiming “Whoa! Hold on Russell! What about step 4?”
Russell had somehow managed to completely skip step 4 not once, not twice, but was on the verge of skipping it for a third time in a row.
Needless to say, once step 4, 5 and 6 were then run in sequence after 1, 2 and 3, the release worked perfectly and disaster was averted.
Why was step 4 missed? Who knows! As I said earlier, Russell is an enigma (and a really nice guy)!Read the Passionate About OSS Blog for more or Subscribe to the Passionate About OSS Blog by Email