“Nothing good happens at 3 AM.”

Probably not entirely true, but it seems to be the time – thereabouts – when shit happens in IT.

Case in point: yesterday. I was winding down for the day, somewhere around midnight-ish, dealing with some guy who was dithering on a backup restore after one of his developers apparently hosed something. He ultimately decided against it, as whatever it was got worked out. Shortly thereafter, after about 1:15 in the morning, one of the servers just went offline. Poof!

We’ve seen this from time to time over the years, where the primary NIC (that’s network interface card, for those of you who do not speak Geek)  shuts itself off. There are a variety of fixes, or possible fixes for it, as with anything. Usually, rebooting the thing bring it back online. That depends on the PDU (power distribution unit, AKA, fancy power strip that allows remote reboots) working. We have probably six that need to be replaced, as they are errored out and do not work for anything other than just plain old power – hence the fancy power strip moniker I’ve given them. This, alas, was one of them.

So I had to haul myself to the NOC at 1:30 in the morning, only to get there and find that the errored PDU just shut off the outlet. And only that one outlet. Easy enough to fix by swapping it to another, working PDU. Then the drive home. In bed by 4. Up again by 8. Work work work. It really puts a crimp in your day.

Some days are like that in IT. Just so anyone who thinks it’s all glamorous and files transfer in nanoseconds and no one ever makes a typo and who may think IT is just super cool and easy money doesn’t get any wrong ideas about it all.

That’s it for now. Until next time, peeps: be well.