One of the downs I guess I neglected to mention last entry is when your connection goes down, or the power goes down … I had the worst Internet day of my life yesterday … I got home from an excellent Saturday night at Bootleggers, probably around 5AM or so … The PA was loud and the band sounded awesome … Incredibly full sound packed with a punch that I haven’t heard much at Bootleggers … I think that’s cause I really haven’t ever turned it up that loud there cause I never have been allowed to … New owners, new attitude … There was a wet T-shirt contest in between the 2nd and 3rd sets, and by the time they wrapped that up, Octane didn’t start playing until 1:45 … They played until 2:10 … Breakin the law, breakin the law … Did I say it was loud? Awesome …
But I digress, so I do my normal channel flipping cause I fall asleep to the TV all the time … The TV drowns out my tinitus … So I get engrossed in this movie with Jennifer Connelly and Ben Kingsley about some chick that lost her house and ol’ Ben moved in … That ends, and the screen shows that The Man Without a Face is coming on next … So, I have to watch that … It’s a great movie … Should buy the DVD … So the credits are rolling, and poof, the power goes out, for like a second … Damn, first thing that shoots through my mind is the computers and how I haven’t had a power loss for over a year and a half or more …
This fries the power supply in the main Windows Server, the domain controller … I am very tired now … I didn’t expect to last through two movies and then have a power outage requiring my attention … So I am going to send the wife up to CompUSA to purchase a power supply … I remember looking at the clock to see if they were open, and as I recall they were, so maybe this was around 9:30? Heck, I have logs … hang on a sec … Okay, here’s the story … The power dropped at 9:20AM … I could tell she wasn’t too excited about this, so I figured I would molest another machine and go buy replacements later … (I have another machine allocated to Windows that needs a power supply as well, so I should pick up 3 or 4 of them) … So I replace the power supply, and bring up the Windows server at 10:02 … I am poking around on there, and boom! 10:11 the power goes out again … It’s out this time for about 30 minutes …
Once I get everything synched back up again, I can’t bring up my own web site, I can’t bring up any other site except for the ICUHost.Net site … So I figure this is some sort of DNS issue since the only zone in my local Active Directory enabled DNS is the ICUHost.Net domain … I see activity through the firewall that millions of HTTP hits have gone through, so I know things are working okay from the outside, I just can’t get around … I put my FreeBSD name server as a secondary in the Windows configuration and sies started coming up for me … So, okay, I have some sort of internal DNS issue, but I don’t necessarily know why, so I figure I will leave it until later … Everything seems to be working for the people outside, and that’s all that matters … We were having people over for the 3 year old’s birthday, and I hadn’t gotten any sleep yet … The wife called and delayed the invites another hour to allow for this … When she came to me and said people were there, I mosied to another sleeping location and got another hour basically …
I came down and checked out the machines … I logged into all the others that I hadn’t logged into yet … I brought up the game servers and their streaming log file applications that produced the web displays and log capturing for stats … Just brought up more things that are normally running that I didn’t fire up earlier cause I wanted sleep … So back upstairs to the house fulla people celebrating the little guy’s birthday …
Then the phone rings and the wife answers it and it’s a reseller who cain’t FTP … I do preliminary help, and nothing is resolved so I stick him on hold and head down the steps to the dungeon I call home … I browse around and I can’t figure out why he can’t FTP … The service is running, I see his connection with netstat … It’s just not making sense to me … While he’s on the phone, he has company show up, so we both decide that I will look at it later and get back to him when I think I know what’s going on with this issue … No one else had called, so I wasn’t entirely sure if it was all on my end … I supposed it was though … So I go back upstairs … The wife also informs me that someone overflowed the toilet … as I noticed this when the top of the kitchen stairs were wet … Sheesh …
As I am sitting there with this going through my mind, I start to think that maybe I never saved the router config when I applied ACLs to it, and maybe that when the power went out it cycled an old configuration that had these problems in it … and that was probably also the reason that I was having internal/external surfing issues … Those were the symptoms I had had when I initially started trying to lock out the baddies working on the router ACLs … So at the optimum time I head back downstairs to apply this theory … I find the files I was using to go through these changes, and I find the most recent ACL text file that I have and I apply that to the router … I go over to the syslog and I see in it that the Serial Interface changed state to down, and I saw that the Internet activity LED that is normally pretty much solid is off … and I’m thinking that the Serial Interface never was taken down when I went in to configure the router … Weird … So I hop back over to HyperTerminal and exit from the router … Still no Internet light … Hmmm … so I go around to the other side and check out the CSU/DSU lights and sure enough there is that little amber guy staring at me … “AL” … Alarm … My connection is down … No! Not what I think it is …
Yes, the toilet juice ran down through the floor and was all over the T1 connection box … drips of water hanging from the bottom … I probably turned red … I was livid … There was everybody upstairs, and with the problems I had already had today, my T1 is screwed up cause someone used too much toilet tissue … 2 floors away! I pick up the intercom and get the wife’s attention … She doesn’t pick up, and I’m not about to broadcast my concerns to all those present … So I hung up and called the home phone, she answered and said she heard me, and I said slowly through my teeth, “Bring me the hair dryer” … She says, “you’re kidding” like she knows why … After a wait much longer than I expected, I became more .. uhm .. upset I guess … So she finally shows up, and I simply don’t say a word, plug in the hair dryer and point it at the device that I have no knowledge of … For one thing, it’s locked … I can’t get into it … So I blow hot air into the vents thinking that maybe something will dry out and miraculously I will have an Internet connection again … After 5 or 10 minutes of this, and no result, I give up and call for service …
The guy takes my number down, etc … asks me when I am available at this number, etc and I’m like 24 hours a day … and asking what hours are tomorrow etc, and I’m like … ‘This is the location, we’re here now and through the night, we will be here for the guy whenever the guy shows up’ … So I think I conveyed my desire to have ASAP service … Especially after I expressed my concern that someone wouldn’t be out tonight to fix this … That was probably the clue, and he said he would call and see what could be done and get back to me … So, nothing else I could do, I went upstairs waiting for the phone to ring … I had to leave for the Tango to go to work in a short while … I told the wife that when the phone rang her answers were “yes” and “anytime, right now is good” … I was just sitting there at the dinig room table clutching my coffee mug ever so tightly … My father came over and asked me questions and stuff … But he didn’t know what was up with the thing, etc (he’s not a computer genius like myself) so I snapped “I don’t really wanna talk about this right now” and he walked away … I felt bad, but it was the truth … So I figured I would just go to work … After all, there was alcohol there …
The wife called at 10:30 and said that the technician will come out when the sun comes up, whenever that is … They prefer to work during the light hours … I thought that was completely odd … When I got home she said that Savvis had called around 3AM and said the guy would be here at 8AM … He didn’t show til about 8:40 (the bastard) … It was fixed in about 20 minutes … Down for about 12 hours …
Of course at this point I already know that anyone who hosts through me is already searching for a new host … I mean, I would, so I can’t blame anyone for feeling that way … For me, I don’t sweat the downtime … I don’t think anything is that important … I am dissatisfied at the response time … I will certainly be looking into that further … I pay a huge chunk of change per month … I think the service should reflect that … I don’t charge people a lot that host through me though, and I give no limitations on anyone except for disk space … I sent a way less stuffed shirt email to everyone who hosts through me and so far the response has been good … On the other hand, besides demanding that my host remain available 24/7 with backups out the B-hind, I can appreciate that before this outage (that was explained in graphic detail) that there was an uptmie of over a year … What did someone actually miss? Email delayed, another offensive guest book entry? It’s like what my wife and I did one power outage … We played cards … We don’t ever play cards … but then again, she doesn’t read email … But cards, not even for clothes or anything … Strip Gin is a thing of the past … We’re grandparents now, you know … I just hope people understand what went down … How and why, and that 12 hours is nothing compared to the 19, 680 some odd hours before that since June 28, 2002 …