About Me

that was weird...

apparently my database was restored to about 10/10/08... which is annoying since I've done a little SINCE then... but oh well.. I had a partial backup... but at least it's up and running again...

Mood: more website whoas... will it ever end?!?!

10/25/2008 12:27:30 AM

ken

well here's what happened...


Dear Ken Liao,

The web server that your site is hosted on has been offline due to some hardware failures in the RAID setup.

RAID stands for "Redundant Array of Independent Disks" and is a technology that employs the simultaneous use of two or more hard disk drives to achieve greater levels of reliability and performance.

Your website is stored across the RAID system twice over different hard drives, if one of the hard drives fails your web site will continue to run. The failed hard drive is replaced and the data that was on the drive copied again from the other drives within the RAID, this is known as rebuilding the RAID, and normally happens seamlessly without any effect to the web hosting server or your website. This is a daily task performed in our data centers and is standard for large data storage systems such as used in the web hosting environment.

In this instance, we replaced the failed drive with a new drive and the RAID started to rebuild. While this was happening the rebuild process failed, corrupting all the data within the RAID set. This should not happen and we have open tickets with the RAID manufacturer to understand what went wrong in this case and to ensure that they can prevent this for the future.

Our system administrators do not rely on the RAID system as our only source of backup. We run a rolling backup of the live system to external backup servers to ensure that in a case like this we have a restore solution.

After the RAID corruption occurred, our engineers analyzed the situation and found that the only solution left to us was to recover the data from our backup systems. At this point the RAID was reinitialized ready to receive data, this process itself takes several hours to perform.

Currently we are copying and restoring the data from our backup systems to the web hosting server that your site runs from. The restore process takes time and is expected to finish early tomorrow morning. When the data is restored to the server we will then turn on the services that deliver your website to the Internet. A small amount of data loss may occur if you uploaded new files to your web space between the time that the backup was made and the failure occurred.

Since the system problems began we have had a dedicated team of administrators working around the clock to monitor the copy of data from our backups and to ensure that all settings are restored so that your website will run again.

We apologize for any inconvenience and thank you for your patience. We will update you again as soon as there is additional information available.

Sincerely
1&1 Internet Inc.

10/27/2008 1:52:18 AM

Name


Comment