Who, Me? Rise and shine, dear readers, it’s Monday and therefore time to sink your teeth into a new week of work, a fact The Register celebrates with a new edition of Who, Me? This is the column in which you share stories of when you bit off more than you can chew, but fumbled through.
This week, meet a reader we’ll Regomize as “Blair” who once worked as a Unix sysadmin for a company that ran a colossal PABX that underpinned a mid-sized nation’s entire telephone system.
When Blair started the job, the company and the network were in a terrible mess.
“The datacenter cabling was a rat’s nest, the UPS was not networked, there was no monitoring, no documentation, and the list of passwords was on a scrap of stained grid paper,” he told us.
One thing did work: backups of Oracle databases.
Blair tidied things up, made them more resilient, and eventually reached the point at which he could turn his attention to a documentation system.
“I created a mediawiki and used LDAP for authentication.” This worked well. So well that the company’s software engineer asked him to build another to replace the shadow IT wiki he was using.
“He didn’t just like the new wiki but also appreciated being able to use the same login as his PC and email,” Blair wrote.
The request was then extended to applying LDAP to the customer ticketing system.
Blair decided that was possible after a simple change that he scheduled for the same day he planned to spend an afternoon with his dentist.
It turned out the change wasn’t so simple as switching over to LDAP authentication because the ticketing system deleted all existing users.
Remember how we said this was a mid-sized country? Blair had deleted millions of users.
He felt something might be awry as while he lay in the dentist’s chair, his phone scarcely stopped buzzing.
Blair bolted back to the office and gave himself a crash course in Oracle database recovery.
“I fixed it in the wee hours of the next day, and I confirmed that all users were present and accounted for,” he wrote.
A little later, he learned the full horror of his error by finding documentation that warned his plan was dangerous.
“It wasn’t in bold or caps, there were no warning symbols or information boxes,” Blair argued in his defense. “It was just another sentence in the paragraph: ‘Enabling LDAP authentication will delete all existing users’.”
Have you ever been unable to escape and fix a mess of your own creation? Click here to send a confession email to Who, Me? so we can issue a fresh life lesson on a future Monday. ®
