OK, WTF@#K Happened THIS Time???

Announcements and items of immediate importance.
Post Reply
User avatar
Paul_Schatzkin
Site Admin
Posts: 992
Joined: Thu Jun 14, 2001 12:49 pm
Real name: aka The Perfesser
Contact:

OK, WTF@#K Happened THIS Time???

Post by Paul_Schatzkin »

I am composing this missive the morning of Wednesday , February 23, 2022.

As I write, fusor.net - as well as all my other web-based operations, have been offline for FIVE DAYS.

And while I know there have errors and mis-cues on our part, I am laying the blame for this disruption entirely at the feet of our Internet Service Provider. They sold us a bill of goods and have simply failed to deliver.

GoDaddy is misnamed. It's really StopDaddy.

To say I am disappointed is an Understatement of Epic Proportions.

I thought I could make his short but... here are all the sordid details:

1. On Thursday Feb 17, I attempted to perform a backup of one of my Wordpress installations. The backup required ~500MB of disk storage.

2. At the time I initiated that backup, the disk usage in my account showed that we had ~85GB of data, which displayed as ~94% of the 95GB disk allowance.

3. The fusor.net database that has been built up over the past 20+ years weighs in at ~20+GB. My other sites are all in the 1GB-2GB range. The reason so much of our storage was in use is because there were redundant copies of the fusor site on the server that had been stashed in an (apparently vain) attempt to prevent data loss. Regardless, there should have been sufficient storage available to perform a .5GB backup.

4. It turned out that, according to Math By GoDaddy, 85GB of 95GB = 100%. That maxing out of our storage meant that normal operations started breaking down and fusor.net/board users started getting error messages.

5. Friday morning I called GoDaddy and made the command decision to elevate our server accommodations with 'more resources' - primarily by doubling our storage capacity from 100 (actually ~85GB) to 200GB. That upgrade was sold to me for less than $100, and I figured another 100GB would be sufficient to manage a 500MB backup...

6. What was not clearly conveyed, or, arguably, I failed to fully grasp, was that this addition of 'more resources' involved more than simply expanding a partition or increasing storage. I cringed when Frank sent me a screen shot that said "your website is being migrated..." about noon EST on Saturday.

7. Come Sunday morning, all hell broke loose. The unanticipated 'migration' had been implemented but there was one small problem: of the ~85GB of data stored in the prior installation, only ~35GB actually made it across the digital wilderness to the new homestead. 50GB of data was left ashore in the 'migration'.

8. Needless to say I spent the better part of my Sunday (the one day a week I try to 'go analog' by reading actual books and listening to vinyl records..) barking at telephone support personnel at GoDaddy. It didn't take long to get to the four magic words of any support scenario, "Please escalate the call." But all Tier 2 could do was create a "cPanel Business Support Request" - which just meant "put it in the 'fix it' queue."

9. I was told at the time that the 'ticket' would be given top priority, but that was Sunday and this is ('was' at the time of actual posting) Wednesday. The 72 hours the ticket says to expect a resolution has lapsed and there is still no site.

10. On Monday I called Tier 2 again. This time I was assured that the original installation is still intact with the entire 85GB (remember, that's 100% of 95GB) of data still sitting on the original server. I told the Tier 2 advisor to just "put everything back the way it was on Thursday" and we'll deal with the storage issue ourselves.

11. Tier 2 informed me that this could not be done, that the migration would have to be completed. Our resident savior, The Coalman, who has a keyboard full of 'magic wands', says otherwise but... at that point we were entirely at the mercy of unseen forces working in unknowable spaces with unfathomable tools. We were, in simpler terms... fucked.

So we waited. All day Monday. All day Tuesday. I checked email compulsively all day long hoping-against-hope that a message from cPanel Support would show up saying that 'your migration has been completed' and that all would be restored and there would be great rejoicing throughout the land.

And now look... I am finally posting this on Friday.

Which means it has been a solid week since this site has been fully functional.

One thing I have learned during that week is that GoDaddy is a blackbox. The ONLY way to talk to a human at that company is by calling their support line - which these days can entail hold times from 10 to 30 minutes. And then sometimes you get a Tier 1 advisor, sometimes you go straight to Tier 3.

Absent any other phone number, over the past several days I have been sending frantic messages to every email address I could find on the GoDaddy website with a pdf attached detailing the cascading litany of failures outlined above. I even went so far last night as to open an account with a new host/ISP and register a new domain: GoDaddyDestroyedMyWebsite.com" (I tried to register the domain w GoDaddy but surprise surprise, they said that domain was "unavailable." Turns out that was just an exercise in corporate censorship!).

I have no way of knowing if my email barrage had any impact, but this morning things finally started coming unstuck. I got emails saying that new tickets had been opened, and then more emails saying the case was 'resolved' - though nothing had changed. This afternoon I managed to get a Tier 3 advisor on the line, he afforded some insights on what was going on behind the scenes, another 'host ticket' showed up and... lo and behold, around 6:30 this evening the site was restored.

I have been on the Internet since 1993. In those nearly 30 years, this is the single worst experience I have ever endured.

I understand that technology can be unreliable, that it's great when it works and horrendous when it doesn't.

But GoDaddy's conduct in this case has been inconceivable, unconscionable, and unforgivable.

I guess it's great that we have the site back, but now we have to take a cold hard look at how we make absolutely certain that nothing like this ever happens again.

By the same token we need to look at some of the internal operations that have caused the loss of many file attachments and make sure we never get blindsided by the sort of limitation (what, ftp stops downloading at 10,000 files??) that we ran into here.

I think the single most discouraging moment I have had in the past week was hearing that there was scuttlebutt among the faithful that the site was lost forever, that it was 'over.' I mourned the thought that those 20 years of irreplaceable knowledge might have been liquidated.

I cannot tell you how relieved I am (for the moment, at least) that is not the case.

I am grateful beyond words to Frank Sanns for staying with me through this ordeal; to 'The Coalman' for coming to our rescue and offering continued support, to Richard Hull for being patient in the face of certain despair, and to each and every one of you has contributed to the incalculably valuable body of knowledge that has been compiled under the aegis of 'Fusor.net'

Now I/we have to make 100% certain we do not get caught in a bind like this again.

Which brings me to the next post...
Paul Schatzkin, aka "The Perfesser" – Founder and Host of Fusor.net
Author of The Boy Who Invented Television: 2023 Edition – https://amz.run/6ag1
"Fusion is not 20 years in the future; it is 60 years in the past and we missed it."
thecoalman
Posts: 189
Joined: Sun Feb 06, 2022 1:26 pm
Real name:

Re: OK, WTF@#K Happened THIS Time???

Post by thecoalman »

I'm going to have to say it's very unfortunate this happened right after the previous issues. Also note that everything has been restored to exactly how it was very early Sunday. No data or files have been lost with this incident. Keep in mind the results you were seeing were from the migrated version. The source server was completely intact and even usable as if nothing happened with some custom DNS.
11. Tier 2 informed me that this could not be done, that the migration would have to be completed. Our resident savior, The Coalman, who has a keyboard full of 'magic wands', says otherwise but... at that point we were entirely at the mercy of unseen forces working in unknowable spaces with unfathomable tools. We were, in simpler terms... fucked.
The tech I talked to last night was able to give me more insight. The information relayed to her was that it was a "permission" issue that really doesn't say much but the other thing she noted was that it was being worked on by the developer team. If you have developers working on something it's pretty serious.

She did say it would be possible to cancel the migration and revert the DNS changes but she was also informed if that was done the problem may make it's way back to the origin server. This was probably why it wasn't offered to begin with.

Assuming this is all true and me speculating it would appear a Godaddy system bridged to WHM/Cpanel had some kind of failure.

There is no excuse for this though, the account should of been manually migrated days ago. The other issue is lack of communication, GoDaddy has no ticketing system in their panel, It's phone or chat only. How you can operate a hosting company without a ticket panel for the customer is beyond me. This is guaranteed to lead to communication issues.
John Futter
Posts: 1848
Joined: Wed Apr 21, 2004 10:29 pm
Real name: John Futter
Contact:

Re: OK, WTF@#K Happened THIS Time???

Post by John Futter »

Thanks Guys
especially you Paul
have you got any hair left?- i doubt it
RobertMendelsohn
Posts: 85
Joined: Mon Jul 20, 2020 4:58 pm
Real name: Robert Mendelsohn

Re: OK, WTF@#K Happened THIS Time???

Post by RobertMendelsohn »

Thank you so much to everyone who has helped maintain this amazing site.

If it would be helpful, could a folder of Fusor.net's data be made available for public download, as an additional precaution / store of the files? I'd be happy to make several SSD backups.

I have had similar problems with GoDaddy, really a scummy company.
User avatar
Bob Reite
Posts: 576
Joined: Sun Aug 25, 2013 9:03 pm
Real name: Bob Reite
Location: Wilkes Barre/Scranton area

Re: OK, WTF@#K Happened THIS Time???

Post by Bob Reite »

I've never had trouble with GoDaddy, but I only use them for a registrar. I host my own web and email servers right in my home office. I like the idea of having physical control of the hardware.
The more reactive the materials, the more spectacular the failures.
The testing isn't over until the prototype is destroyed.
User avatar
Paul_Schatzkin
Site Admin
Posts: 992
Joined: Thu Jun 14, 2001 12:49 pm
Real name: aka The Perfesser
Contact:

Re: OK, WTF@#K Happened THIS Time???

Post by Paul_Schatzkin »

Bob Reite wrote: Sat Feb 26, 2022 3:36 pm I've never had trouble with GoDaddy, but I only use them for a registrar. I host my own web and email servers right in my home office. I like the idea of having physical control of the hardware.
I think that is going to be my approach as well. At least for the registry side of things. Hosting, I'm just not up to doing that on my own, but, then, clearly, neither is GoDaddy.

Also, Bob, I know I've read it before but I just laughed outloud at your .sig line 'The testing is not over until the prototype is destroyed.'
Paul Schatzkin, aka "The Perfesser" – Founder and Host of Fusor.net
Author of The Boy Who Invented Television: 2023 Edition – https://amz.run/6ag1
"Fusion is not 20 years in the future; it is 60 years in the past and we missed it."
User avatar
Richard Hull
Moderator
Posts: 14975
Joined: Fri Jun 15, 2001 9:44 am
Real name: Richard Hull

Re: OK, WTF@#K Happened THIS Time???

Post by Richard Hull »

Well, is seems we are back up. I am checking for my last bits of updating the images. I will report if I find anything missing.

Richard Hull
Progress may have been a good thing once, but it just went on too long. - Yogi Berra
Fusion is the energy of the future....and it always will be
The more complex the idea put forward by the poor amateur, the more likely it will never see embodiment
Frank Sanns
Site Admin
Posts: 2117
Joined: Fri Jun 14, 2002 2:26 pm
Real name: Frank Sanns

Re: OK, WTF@#K Happened THIS Time???

Post by Frank Sanns »

Everything should be there Richard. Backups were made up to about an hour of losing access to the site. I know this because I was in the process of posting when the site vanished. Only my very last post appears to be missing. You should be good but let us know.
Achiever's madness; when enough is still not enough. ---FS
We have to stop looking at the world through our physical eyes. The universe is NOT what we see. It is the quantum world that is real. The rest is just an electron illusion. ---FS
User avatar
Richard Hull
Moderator
Posts: 14975
Joined: Fri Jun 15, 2001 9:44 am
Real name: Richard Hull

Re: OK, WTF@#K Happened THIS Time???

Post by Richard Hull »

Yes, it all appears intact as of my last updating of the FAQ images. This is great.

Richard Hull
Progress may have been a good thing once, but it just went on too long. - Yogi Berra
Fusion is the energy of the future....and it always will be
The more complex the idea put forward by the poor amateur, the more likely it will never see embodiment
thecoalman
Posts: 189
Joined: Sun Feb 06, 2022 1:26 pm
Real name:

Re: OK, WTF@#K Happened THIS Time???

Post by thecoalman »

I still have access to the server as I finish up this last script so Sunday morning when I saw this occurring I dumped the database and downloaded the most recent uploaded files from the original server. Even if GoDaddy's datacenter were to wiped off the face of the earth it could of been restored. The only thing I'm aware of lost was a single post that was made shortly after the site came back up and even that could of been fixed if I wanted to take the time. My apologies to the poster but it wasn't anything important.

If I had known this was going to carry on for this many days I would of suggested simply moving it to another host, it could of been up late Sunday for most people. Never would I have expected a host to take 5 days to resolve an issue.
Bob Reite wrote: Sat Feb 26, 2022 3:36 pm I've never had trouble with GoDaddy, but I only use them for a registrar. I host my own web and email servers right in my home office. I like the idea of having physical control of the hardware.
What happened here is another thing I have added to my list of reasons not to use your host for domain registration/nameserver. It makes things relatively easy for the customer but issues like this where they are able to unilaterally make DNS changes is not only problematic for data continuity but also an issue if they point it at a botched migration.

Since the origin server was still intact what could of done Sunday was revert the DNS changes to original server and get an assurance from GoDaddy the original server would remain available for a few days after successful migration. Once there was successful migration disable the forum on the original , export the database, import into new server and change the DNS to new server. This is actually what should of been done to begin with to maintain data continuity. Even if this migration was successful there would of been a few missing posts starting around 1:30 Sat. Not a huge problem if you catch it but if people started posting on new server before importing the the most up to date data then you have a problem.

Being able to host in your home/office is nice but to make that work well you usually need business plan from your ISP and in my neck of the woods those are not cheap. The primary difference between them and a residential plan is you get a dedicated IP and increased upload speeds. While there is ways around not having a dedicated IP it's PITA not having one for web server. I could never justify the increased cost relative to what I could get with paid hosting. If you can mange server in your house then certainly you could do it remotely, this is what you can get with unmanaged VPS for $20/month.
2 Core CPU Processor
4 GB Guaranteed RAM
75 GB Raid-10 SSD Storage
2 TB Premium Bandwidth
2 IPv4 Addresses
Rex Allers
Posts: 570
Joined: Sun Dec 30, 2012 3:39 am
Real name:
Location: San Jose CA

Re: OK, WTF@#K Happened THIS Time???

Post by Rex Allers »

Back in 2004 (I think) I set up a hosting package on Godaddy. It had more hosting storage and bandwidth than I would ever need and a large number of available emails (50 or 100?)

Last fall Godaddy informed me they were going to stop providing my domain email product. They were slamming me into a Microsoft email package that would cost me a monthly fee after a free intro period of a couple months. I found it didn't even provide one of the key email features I used and needed.

So I shopped around. I found a mail package at "NameCheap" that met my needs and was way cheaper than this Microsoft thing. Bye bye email through godaddy/MS.

I have kept my web hosting on godaddy because I'm grandfathered in from my long-time hosting to a good price for unlimited storage and bandwidth. (Not that I'm testing any limits, but why give up a good thing.)

My domain registration was close to expiring so I moved that to NameCheap too. Much cheaper than godaddy's renewal. Took me a couple days to figure how to set all the reg parameters but a one-time thing and done. Having done that, I don't know why anyone would keep their registration through godaddy.

EDIT: Post moved here as it is more relevant than resurrecting an 8 month old thread. FS
Rex Allers
thecoalman
Posts: 189
Joined: Sun Feb 06, 2022 1:26 pm
Real name:

Re: OK, WTF@#K Happened THIS Time???

Post by thecoalman »

Those shared hosting plans with "unlimited" anything are largely marketing gimmicks. In the TOS you are going to find limitations on CPU usage, how you can use it like not being able to use it as backup service and other things that can give them an excuse to cut you off.

Theses large hosts offering shared hosting make money by packing hundreds or thousands of sites onto a single sever with the expectation most of those customers are going to utilize very few resources. If you start using a lot of them inevitably you'll get notice about some limitation being exceeded.
User avatar
Paul_Schatzkin
Site Admin
Posts: 992
Joined: Thu Jun 14, 2001 12:49 pm
Real name: aka The Perfesser
Contact:

Re: OK, WTF@#K Happened THIS Time???

Post by Paul_Schatzkin »

thecoalman wrote: Sun Feb 27, 2022 3:53 pm Theses large hosts offering shared hosting make money by packing hundreds or thousands of sites onto a single sever with the expectation most of those customers are going to utilize very few resources. If you start using a lot of them inevitably you'll get notice about some limitation being exceeded.
That would describe perfectly what happened with Media Temple.

Would that we had known at the time that going from MT to GD was out of the frying pan and into the fire.

--P
Paul Schatzkin, aka "The Perfesser" – Founder and Host of Fusor.net
Author of The Boy Who Invented Television: 2023 Edition – https://amz.run/6ag1
"Fusion is not 20 years in the future; it is 60 years in the past and we missed it."
User avatar
Paul_Schatzkin
Site Admin
Posts: 992
Joined: Thu Jun 14, 2001 12:49 pm
Real name: aka The Perfesser
Contact:

Re: OK, WTF@#K Happened THIS Time???

Post by Paul_Schatzkin »

thecoalman wrote: Sun Feb 27, 2022 4:43 am get an assurance from GoDaddy
🤣

I'm just highlighting that because... well, the words "assurance" and "GoDaddy" just don't belong in the same sentence.

I am reviewing Scott/TheCoalman's recommendations now.

--PS
Paul Schatzkin, aka "The Perfesser" – Founder and Host of Fusor.net
Author of The Boy Who Invented Television: 2023 Edition – https://amz.run/6ag1
"Fusion is not 20 years in the future; it is 60 years in the past and we missed it."
User avatar
Bob Reite
Posts: 576
Joined: Sun Aug 25, 2013 9:03 pm
Real name: Bob Reite
Location: Wilkes Barre/Scranton area

Re: OK, WTF@#K Happened THIS Time???

Post by Bob Reite »

For a dirt cheap VPS, you can't beat the $28 a year offering at vpscheap.net. But bear in mind, it's only good for a site that does not get too much traffic. It's fine for a backup site, or a small Wordpress site. But when I tried to do audio streaming through vpscheap.net it would just shut down without warning, even though I limited it to ten listeners at a time. No data was lost, I would just have to manually restart it so I switched the audio streaming from one of my home machines.

And yes, I have a business account with 5 static IPs It's bundled with my business phone and yes, it runs about $200 a month, but all deductible as a business expense for me.
The more reactive the materials, the more spectacular the failures.
The testing isn't over until the prototype is destroyed.
thecoalman
Posts: 189
Joined: Sun Feb 06, 2022 1:26 pm
Real name:

Re: OK, WTF@#K Happened THIS Time???

Post by thecoalman »

Years ago on my site we did "Coalcast", it was a bit duct taped together. I was using Teamspeak which was limited to 8 people for those participating in the conversation, then I routed Teamspeak output to Icecast for the public stream. Like I said it was duct taped together but it worked.

In any event I recommended a smaller host I have been using for about 5 years now that built their business on VPS/Dedicated hosting. They have been extremely fast no matter what, I once ordered an IP and had it in 5 minutes later. Initial response time to tickets is 1 to 2 minutes. If you go into their forum and start a tonic you might get a response from the CTO. :) .

I can't say that is the way it will remain. The host I was with previously was exactly the same, then they started buying up a bunch of smaller shared hosting companies. Everything went to crap literally overnight. I went from ticket resolutions in usually half an hour to initial responses that took hours. Just move on.
Chris Mullins
Posts: 67
Joined: Sun Feb 19, 2017 11:32 am
Real name: Chris Mullins
Location: Shenandoah Valley, VA
Contact:

Re: OK, WTF@#K Happened THIS Time???

Post by Chris Mullins »

I've used Digital Ocean both personally for small sites and professionally. One nice feature is the ability to make a VM snapshot of the entire server, which you can restore back on the original virtual machine or a new one. I've had good experiences with Namecheap for DNS.
Post Reply

Return to “Announcements and Site Administration Topics”