Page 1 of 2

Not good news about the lost links on the site

Posted: Mon Nov 04, 2019 11:01 pm
by Frank Sanns
Since I am supposed to be the tender of the backups, I shall be the one to break the bad news to everybody. There are a great deal of images and files that have be irreparably lost.

All along, I have done regular backups around every two months of the database as well as the phpBB. Both structure and data are backed up. I keep them on my raid drive and also on the host server.

Before we even began to do the migration, I did the usual backups plus I did, out of abundance of caution, a full download via ftp of every single file from Media Temple. There really was not reason to do so except for my paranoia with the change of something, somewhere getting changed during the migration. All should have been good in the world with three layers of backups before the first step of migration. This was August 2nd. I also put a note up on the forum that the migration was soon to come.

About August 12 or so, we had the new host at GoDaddy configured and ready for the move. I did another database and phpBB backup so the most recent posts would be migrated to the new host. Unfortunately, by this time, and with no explanation, the Media Temple site went down a day or two before we started the move. It was nothing that any of us did on our end. It was all MT.

Andrew and had to update the database and phpBB versions to match and we did the migration. This is the short fast version of the story. To make a very long story not so long, the backups from mid August from MT were not good. We struggle through to near the end of August before we realized that. Finally we agreed to use the August 2nd backups since that was before any problems with MT.

There were many more problems and the site had many holes. We worked to patch those up. There were many holes and many lacking attachments. Since I had the full backup via FTP, we selectively loaded all of the images from that back up back to the site. Only some of the attachments showed up.

Fast forward to today when Paul brought on a specialist with php. He confirmed that we are missing a little over 3,000 files including attachments.

What I thought was a complete backup in three versions, was a pile of Media Temple corrupted and lost versions of our site. Paul tried to solicit their help but they only keep backups for one week. That means we would have had to have recognized that there was a problem with the site at the end of July or certainly by August 2nd for them to be able to restore THEIR loss of our site. That means that before we even started to preparations for the migration, the fusor site was already lost.

It is astonishing that Media Temple cannot get our site back considering that THEY and THEY alone lost the essence of our site. Sure I had all of the backups but they were corrupt and I did not have a full FTP backup because it really was not necessary but rather out of abundance of caution. Keep in mind that Paul kept the MT site funded and operational while we did this move. Technically, we should not have had to have any special backups as we did not touch the original site and had no plans to until the new one was up and running and it was backed up.

It is a tragic story of false sense of security with three versions of my backups AND supposedly a still fully operational Media Temple site. That is quad redundancy! And still a failure.

I shall take the heat for not doing the over 1 hour of ftp backup and 8GB of data to my computer more regularly but it is a ton of data and time beyond the normal two backups and the host backup. I learned my lesson and feel extraordinary bad about the loss of data that I am supposed to be trusted in securing. And here I thought I was being overly paranoid. I was not. If you do not see it, have it in your hand, store in in many forms in many places, then in this digital age, you do not have it.

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 3:34 am
by Richard Hull
With what apparently seems to be zero lost data on hand, may I now, at long last, proceed to search over 3 computers here to obtain and replace the tiniest fraction, mainly in the FAQs, data that has ceased to be available. It will be long and tedious effort and I will become very tired doing it. I might take long rests from it as I get dispirited. If I am finally given the go ahead, I will once again request any one who stumbles on to a crippled FAQ, to PM me and I will see what I can do.

Question ....Do I have permission to repair what I can.

Richard Hull

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 3:42 am
by Rich Feldman
Please don't lose sleep over it.
I did some spot checking of posts from 15 years ago. Some images "no longer present", some are just fine.

Any chance there are 3rd party backups somewhere?

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 4:16 am
by Richard Hull
Third party backups?! Don't hold your breath. However, those of us who have posted over the years in non-FAQ posts have pictures missing. It is up to those individuals, If they so desire, to reload their images and diagrams. Many people who were significant contributors are no longer here. As such, their stuff is gone forever.

Over time, we can rebuild the core of fusor. net. First up must be the FAQs for we tend to chide the newbies for not reading them to get core information on key aspects of technical issues needed to get started in so many areas of fusor construction, safety and measurement.

I am going to put up a list of FAQ's being worked on and update as needed.

Richard Hull

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 4:47 am
by Nicolas Krause
I can confirm that some information has been backed up at the wayback machine. For example the following. In addition, there are some images, but it looks like trawling through the archives would be quite a bit of work.

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 8:14 am
by Richard Hull
Thanks for the effort, but what is broken here is also broken there. Apparently, it is referencing images that no long exist as is ours.

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 1:02 pm
by Frank Sanns
Please give us a few more days before fixing the links. Andrew and I are searching our old hard drives (pre2017) for backups that we know we made. I tended to purge after having two full backups so there may be something out there of the very old files. Do not count on it but give us that time.

We are also going to try something else that is far reaching but the site WILL be physically down for a day or two. When it comes back up, if it looks different, please do not be alarmed as that will be only temporary and we will be back to this state soon after. This down is imminent and it WILL HAPPEN starting later today.

Re: Not good news about the lost links on the site

Posted: Tue Nov 05, 2019 2:21 pm
by Paul_Schatzkin
About a week ago, I got an email from Richard that said something to the effect that "this is all so sad..."

And I rebuked Richard by saying, "it's not sad, it's just complicated..."

Now, I fear, Richard may be right.

But as you can see from the other comments in this thread, that sadness has yet to descend in to total despair. We're still scratching for some kind of solution to this perplexing dilemma.

There is a strong temptation to hold Media Temple's feet to the fire for all this. I have the beginnings of a letter in forming in my mind (do we have any lawyers among our members?) but before I "go after" MT, I need to be clearer on the exact sequence of events that demonstrates their culpability in this fiasco. I'll be discussing that with Frank and Andrew later.

I "spoke" (a chat line) with somebody at MT yesterday who offered to "restore a backup" for $79. She said their backups go all the way back to... wait for it.... all the way back to A WHOLE WEEK AGO. Boy, that's sure helpful.

At some point I will contact Media Temple and, assuming their direct culpability, demand the restoration of a backup older than a week. Surely.... well, you'd like to think.... you'd hope...

This reminds me a little bit of the first time, 25 years ago, when I suffered a hard drive crash that took several days of current work with it. At the time I likened the experience to "a death in the family." I realize that is a bit of an extreme perspective, but considering that we're talking about 20 years of stuff that has been gathered on this site, it does not seem entirely unreasonable.

There is definitely some grief to process, if in fact we cannot find a path to restore some (if not all) of what has been lost. There is some solace in the knowledge that there is still an enormous body of useful information in this database. But it will greatly diminish the seamless experience we have enjoyed here if users keep running in to "Forumula 404 - page not found" when digging into the legacy posts.

So let us be sad, but let us not yet despair. Not yet, anyway.

Finally, let me express my deep and sincere gratitude to Frank Sanns and Andrew Robinson for the time and effort that they have put into this migration. I know they are deeply frustrated at the less than satisfactory current state of affairs, but that does not diminish the fact of their dedication and diligence. Let us all be thankful for that.

Now let's see what other vague possibilities we can find...


Re: Not good news about the lost links on the site

Posted: Thu Nov 07, 2019 1:24 am
by Rex Allers
Some thoughts on the problem...

I think it is possible that the files may still present on the site but the links to them are broken.

Here is something I have observed about the problem by looking at posts with and without the missing attachments:
Only for *Certain Users* for every upload they made in posts prior to the move, the uploaded files indicate "The selected attachment does not exist anymore." in the posts. Usually these files are images, but they could be anything (pdf, etc.)

I have described this before. See:
Re: Missing photos
Post by Rex Allers ยป Tue Sep 24, 2019 6:08 pm

Here is my partial understanding of how this works, gotten by browsing support messages.

When a user uploads a file (usually as a message attachment) the file is stored in the /files directory of the phpBB installation. This file is given some kind of a hashed filename (not the original name). I think this is to guarantee each file has a unique name. So if two users upload the file xyz.jpg they won't conflict. They both get saved with a unique name. I suspect the contents of the file is unmodified, not encrypted or compressed, but I am not certain of this.

I think all uploaded files in the /files directory are stored the same (not under sub-dirs, etc). The linkage to get to these uploaded files must take place via tables in the database. I don't know any details of how this would work. Since the problem is related to all uploaded files by certain users, I would think there must be some component of these entries that relates to user-id or user-name.

Maybe it is possible to look for one or two files to see if it really is still there. My thought: Find a post with one of these broken link files. Get the date and time of the post with the broken link. With an FTP client, look in the /files directory, sort the view by date. See if there are any files posted around that time and if so download a copy of one or a few of the files. Examine them on a local PC and see if any seem to have contents that match the missing file. (Probably rename them to something.jpg -- whatever the post showed for the file.)

I wanted to do some experiments to look at how this stuff works. I had my own phpBB site I installed years back, but never made it public so it is a simple basic installation. I planned to make some test posts with and without attachments, then look at my installation and database to see if I could work out the details.

Unfortunately, the first step I made was to update my 3.0 phpBB to current. That went bad. Even trying to do a complete new install is failing. (I'm on Godaddy too. Not sure if that is part of my problem, but I suspect it is.)

I'm working on trying to resolve my installation issues and hope I can do some tests soon.

I'm willing to try to help sorting these problems out, if you want my help.

Re: Not good news about the lost links on the site

Posted: Thu Nov 07, 2019 9:19 pm
by Bob Reite
That hashed filename is of the form XX_[UUID}. I think just a random UUID is generated, not a "hash".

But in any case the XX (which can actually be one, two or even three digits if we have that many users) before the underline character is the same as the poster_id in the database. the phpbb_attachments table is where this information is stored, including the real_filename that will be displayed on the web site. The poster_id in this table is the same value as the user_id in the phpbb_users.

My thought is to use phpmyadmin to see how many attachments are supposed to be on the site (phpbb_attachments table), then see how many files are actually in /files.

Look at all the backups to see how many files are in /files as well.