After seeing these symptoms and a machine that is totally unresponsive, I chose to blindly reboot the machine, as I was running the server headless.  Not something I would ever do in a production environment, but it’s my home server.  After a reboot, the server wouldn’t even respond to ping requests, and I saw in my router that it had not registered its DHCP lease upon boot.  My obvious next step was to connect a monitor to this machine and actually see what was going on.  As I moved the machine around to plug in the monitor cable, I noticed that the steel case was warm (borderline hot, in fact) to the touch.  Once the monitor was connected, I immediately saw the root of the problem.  I was staring at a BIOS screen telling me the boot drive had failed.

Knowing what the problem was made for a clear path for recovering.  I located a replacement IDE hard drive and swapped it into the case and reloaded a stock installation of Debian 6, Squeeze.  Recreating the basic installation was no problem, and I added Webmin back onto the system.  Fortunately, I had no user data on the orignal hard drive, and I had kept the data volume in its own volume group, to avoid problems like this, should they ever come up.  The only parts of the configuration that were located on the original drive were mounting, NFS, Samba, and DNS configurations.  On the other hand, I had not gotten around to backing up that configuration to another machine to avoid such a problem.  ;-)

After getting a new boot drive with an OS on it, I moved to getting the data volume up and running.  Based upon my previous post, the data lives inside of a LVM logical volume on top of a physical RAID1 array.  Since I was simply trying to locate and re-enable an existing array, I ran: “mdadm –assemble –scan” which found the two partitions in the RAID1 and activated them.  Since they had not been touched in quite some time, no resync was even necessary.

The next step was to locate and reactivate the logical volume so that I could actually mount and use the data stored on the volume.  I ran each of the LVM *scan commands, “vgscan”, “pvscan”, and “lvscan” to confirm the metadata had been preserved.  Everything looked good, so I just had to reactivate the volume with the “vgchange -y a” command, which tells LVM to activate all available volume groups and logical volumes.  I was finally able to mount the data drives and verify that all of the expected data was there.

Now that the data drives were healthy, I had to finish recreating the lost configurations.  I added a permanent entry in /etc/fstab to mount the volume at boot, along with my NFS exports.  Also, I added the local users to the system and gave Samba passwords.  My only time-consuming task was to re-enter the forward and reverse DNS entries for my internal machines and set the DNS forwarders.

Now that the configuration was back to where I wanted it, I took the extra step of using Webmin’s backup module to export all of the configuration information from the system to my laptop.

Once I really knew what was hapenning, the fix really didn’t take that long.  I spent about 4 hours between loading the OS, reactivating the data drives, and getting the local services set back up again.  With the added knowledge of RAID and LVM recovery in Linux, as well as configuration backups, it would probably only take about an hour; most of that time would be waiting on the OS to install to the drive.

The hardware you can use for this project is fairly straightforward. I used a hodgepodge of parts I had laying around. I used:

• Case: older Antec case
• Motherboard: Mini-ITX M10000N small form-factor motherboard, nearly everything integrated
• System Hard Disk Drive: IDE drive I had lying around, 30 GB

I had the bulk of what I needed for the project, but I still needed the drives and a way to attach them to the motherboard. I ordered:

• 2 SATA hard drives, 1TB, from NewEgg
• SATA Host-Bus Adapter to add SATA ports to the system from Deal Extreme
• Assorted SATA data and power cables from MonoPrice

Once all the parts arrived, I started my build. For anyone who has built a computer or replaced parts knows how simple the build really is. The motherboard is mounted into the case, along with the drives. The SATA card is placed into this motherboard’s PCI slot, and the cabling is connected. About an hour later (I removed a VERY old motherboard from this case.), I had the system built and ready for the software for the project.

Several distributions of free software could manage both the storage array and the services designed to present the array to the rest of the network. I chose to use a stock installation of Debian 6.0 stable, with as few packages installed as possible, to keep the bloat down. Of the specific software that I made sure to have installed was the SSH server, NFS file sharing services, and Samba file sharing services. After I installed Debian, I manually installed WebMin to provide a web interface for configuring everything.

First business is to create the array from the SATA disks. I used the cfdisk utility at the command-line to create a single partition over the entire drive, using a partition type of DA, which is listed as a Non-FS data partition. I then repeated the procedure for the second disk. Next, I used the webmin panel to create a RAID1 array (/dev/md0) with both of the disks. After I had the array, I created a new LVM volume group to contain the data array. Creating the volume group in Webmin requires a physical volume to be added to the newly created volume group; I chose the RAID array. Finally, I created an ext4 filesystem and waited for the disks in the array to sync. The synchronization process took roughly eight hours.

The next day, I used webmin to create some SMB shares and NFS filesystems. These will allow my Linux machines to use the data as if it were local and give access to the couple of Windows-centric devices available on my network.

With two fairly short days, and some waiting for disk synchronizing, my file storage device is complete. I am in the process of copying my data from my unreliable desktop to the device as we speak. Among my future improvements and enhancements to the configuration will be a tweaking the mounts and setting up a dedicated space for each user, instead of just a single wide-open storage space.  From the hardware side, I eventually will upgrade to a hot-swap carriage from IcyDock.  I can also add up to two more hard drives to the existing card.  As long as the drives are at least 1TB, I can expand the existing RAID1 into a RAID5.  If the drives are larger, I will create one partition as large as the partitions on the original drives.  I can then create a new partition for the rest of the new drives and create a new array.  This is, to the best of my knowledge, is the way the Drobo allows for expanding storage.

The major problem was that the two clients I was running could ping each other, but not the server. I suspected a cable or simple configuration problem, but I tried to ping from the server to the clients, and that worked. I knew it wasn’t the cable.

Next, I tried examining the IP configuration of the clients. All looked normal.

Finally, it dawned on me that I had set up a firewall at one point through the Mandrake Control Cent
er. I took down the ethernet interface and flushed the chains from iptables. That wasn’t successful, so I took eth0 down again and removed the iptables kernel modules entirely. A quick ping later, and I was in business.

Now, the connectivity is restored; Samba actually logged me onto multiple machines and allowed me to connect to the server with the shares that I had set up.

Today, I configured Samba to allow for storage of the critical files on their network to occur on the server instead of their Windows 9x clients. I was able to use smbclient on the server and get logged in and browsing around with that.

So far, no actual network gear has been donated or purchased, but I hope that will come soon. A few more phone calls will have to be made, and then a decision as to where to make our purchases (Staples ).