After seeing these symptoms and a machine that is totally unresponsive, I chose to blindly reboot the machine, as I was running the server headless.  Not something I would ever do in a production environment, but it’s my home server.  After a reboot, the server wouldn’t even respond to ping requests, and I saw in my router that it had not registered its DHCP lease upon boot.  My obvious next step was to connect a monitor to this machine and actually see what was going on.  As I moved the machine around to plug in the monitor cable, I noticed that the steel case was warm (borderline hot, in fact) to the touch.  Once the monitor was connected, I immediately saw the root of the problem.  I was staring at a BIOS screen telling me the boot drive had failed.

Knowing what the problem was made for a clear path for recovering.  I located a replacement IDE hard drive and swapped it into the case and reloaded a stock installation of Debian 6, Squeeze.  Recreating the basic installation was no problem, and I added Webmin back onto the system.  Fortunately, I had no user data on the orignal hard drive, and I had kept the data volume in its own volume group, to avoid problems like this, should they ever come up.  The only parts of the configuration that were located on the original drive were mounting, NFS, Samba, and DNS configurations.  On the other hand, I had not gotten around to backing up that configuration to another machine to avoid such a problem.  ;-)

After getting a new boot drive with an OS on it, I moved to getting the data volume up and running.  Based upon my previous post, the data lives inside of a LVM logical volume on top of a physical RAID1 array.  Since I was simply trying to locate and re-enable an existing array, I ran: “mdadm –assemble –scan” which found the two partitions in the RAID1 and activated them.  Since they had not been touched in quite some time, no resync was even necessary.

The next step was to locate and reactivate the logical volume so that I could actually mount and use the data stored on the volume.  I ran each of the LVM *scan commands, “vgscan”, “pvscan”, and “lvscan” to confirm the metadata had been preserved.  Everything looked good, so I just had to reactivate the volume with the “vgchange -y a” command, which tells LVM to activate all available volume groups and logical volumes.  I was finally able to mount the data drives and verify that all of the expected data was there.

Now that the data drives were healthy, I had to finish recreating the lost configurations.  I added a permanent entry in /etc/fstab to mount the volume at boot, along with my NFS exports.  Also, I added the local users to the system and gave Samba passwords.  My only time-consuming task was to re-enter the forward and reverse DNS entries for my internal machines and set the DNS forwarders.

Now that the configuration was back to where I wanted it, I took the extra step of using Webmin’s backup module to export all of the configuration information from the system to my laptop.

Once I really knew what was hapenning, the fix really didn’t take that long.  I spent about 4 hours between loading the OS, reactivating the data drives, and getting the local services set back up again.  With the added knowledge of RAID and LVM recovery in Linux, as well as configuration backups, it would probably only take about an hour; most of that time would be waiting on the OS to install to the drive.

The hardware you can use for this project is fairly straightforward. I used a hodgepodge of parts I had laying around. I used:

• Case: older Antec case
• Motherboard: Mini-ITX M10000N small form-factor motherboard, nearly everything integrated
• System Hard Disk Drive: IDE drive I had lying around, 30 GB

I had the bulk of what I needed for the project, but I still needed the drives and a way to attach them to the motherboard. I ordered:

• 2 SATA hard drives, 1TB, from NewEgg
• SATA Host-Bus Adapter to add SATA ports to the system from Deal Extreme
• Assorted SATA data and power cables from MonoPrice

Once all the parts arrived, I started my build. For anyone who has built a computer or replaced parts knows how simple the build really is. The motherboard is mounted into the case, along with the drives. The SATA card is placed into this motherboard’s PCI slot, and the cabling is connected. About an hour later (I removed a VERY old motherboard from this case.), I had the system built and ready for the software for the project.

Several distributions of free software could manage both the storage array and the services designed to present the array to the rest of the network. I chose to use a stock installation of Debian 6.0 stable, with as few packages installed as possible, to keep the bloat down. Of the specific software that I made sure to have installed was the SSH server, NFS file sharing services, and Samba file sharing services. After I installed Debian, I manually installed WebMin to provide a web interface for configuring everything.

First business is to create the array from the SATA disks. I used the cfdisk utility at the command-line to create a single partition over the entire drive, using a partition type of DA, which is listed as a Non-FS data partition. I then repeated the procedure for the second disk. Next, I used the webmin panel to create a RAID1 array (/dev/md0) with both of the disks. After I had the array, I created a new LVM volume group to contain the data array. Creating the volume group in Webmin requires a physical volume to be added to the newly created volume group; I chose the RAID array. Finally, I created an ext4 filesystem and waited for the disks in the array to sync. The synchronization process took roughly eight hours.

The next day, I used webmin to create some SMB shares and NFS filesystems. These will allow my Linux machines to use the data as if it were local and give access to the couple of Windows-centric devices available on my network.

With two fairly short days, and some waiting for disk synchronizing, my file storage device is complete. I am in the process of copying my data from my unreliable desktop to the device as we speak. Among my future improvements and enhancements to the configuration will be a tweaking the mounts and setting up a dedicated space for each user, instead of just a single wide-open storage space.  From the hardware side, I eventually will upgrade to a hot-swap carriage from IcyDock.  I can also add up to two more hard drives to the existing card.  As long as the drives are at least 1TB, I can expand the existing RAID1 into a RAID5.  If the drives are larger, I will create one partition as large as the partitions on the original drives.  I can then create a new partition for the rest of the new drives and create a new array.  This is, to the best of my knowledge, is the way the Drobo allows for expanding storage.

I began investigating by looking into the metadata for the voicemails which were left; all were from unusual phone numbers outside of our area code.  My next trip was to the CDR records and my provider’s call history.  These sources showed that over 1000 phone calls were attempted to be connected through our phone system to various communities around the country.  Two patterns emerged while looking at the details of the calls:

• The calls were being generated in a non-sequential pattern; clearly an unscrupulous telemarketer was war-dialing to South Carolina, Michigan, and Virginia.
• All of the outbound calls were originating with my girlfriend’s CallerID.

So, I knew what was going on, but I had yet to determine to how this incident occured.  Further digging revealed that the Linksys ATA being used internally was lagging significantly and intermittently.  It was a moment of epiphany; I understood where the problem was, and I could take action to address it.

Several years ago, I installed an Asterisk server to provide home phone service with a company that provided an IAX trunk.  At the time, all we were using softphones to make and receive calls.  Over time, I upgraded to a Cisco 7940 for myself and an analog cordless phone on a Linksys ATA for my girlfriend.  At the time I installed the ATA, I had not used a significant amount of security of the SIP peer registration, as the network was firewalled from the Internet.

Fast forward to 5 months ago…  I obtained a business phone service through a major provider using SIP.  Due to their configuration, I was required to open the SIP and RTP ports to the general Internet and forward them to my Asterisk server.

Now, we return to the present time.  Several days ago, this telemarketer scanned my router, found open SIP ports, and began to issue a brute-force attack against my Asterisk server.  The telemarketer scanned every possible 3 and 4 digit peer name, was able to find the Linksys ATA’s peer name, and was quickly able to brute-force the password.  With this information, the telemarketer was able to register as that peer and make calls outbound, according to the dialplan configured for my girlfriend.  My Cisco’s peer was not impersonated, but it would not be likely to be responsible for outbound calling due to the unique dialplan I am using.

With the understandng of what has happened, I could work to correct and block the means by which this telemarketer was using my phone system and service:

1. I unloaded chan_iax.so to remove the possibility of additional phone calls being made.
2. I then enabled a moderate amount of security, particularly a random password for each of my SIP devices.
3. Next, I modified my firewall configuration to drop all packets through my router which contained the IP address from which the attack came.
4. I then re-enabled the IAX phone service by reloading the chan_iax.so module.

After the initial security containment, I turned my focus to handling the inbound phone calls being returned based upon the messages left with the various victims around the country.  I recorded an announcement indicating what had happened and that the compromise was taken care of.  Finally, I crafted the dialplan to play this announcement to every caller not from a local phone number.

Overall, this breach was not as extensive as it could have been, but it does illustrate that security must be vigilantly monitored and maintained.  My complacency in the security of my network ultimately led to this incident.  Fortunately, it does not seem to have led to any major financial loss to any party, including myself.

• Obtaining and reflashing a Linksys WRT54G router, either the GL or GS.
• Playing with and/or hacking a LaFonera wireless access point
• Building my power lockout device for my ham radio
• Building a PC for my car with wireless capability for a media center
• Building a decent amateur radio station, including packet station
• Building a podcast station
• Upgrade my radio license to General Class

At some point in the future, I want to do some kind of podcast. But before I would commit to that, I need to find a topic that I can regularly update and contribute. Part of the issue of the lack of updates is the company for which I work. MCPR has me doing some very interesting things, but they are marketable ideas that I have to keep quiet. The only project that I can talk about is our Asterisk dabblings. We are offering the ability to connect analog or IP phones to the system and connect to about any outbound media out there, including SIP trunks.

But much beyond that, I don’t have the ability to talk about the details of what I do at work with the world at large, between protecting our clients’ privacy and not wanting to give any competitors any ideas about our strategy.

My next challenge was to get the network card working. Keep in mind that this laptop does not have working PCMCIA sockets because it fell off the table…. Since this had been a laptop I had been previously using, I had already found a suitable USB network adapter, the D-Link. I first tried to install the drivers for the network card, but I could not get the provided 2.4.18 kernel source from the Debian CD’s to match the pre-compiled kernel. Since I had to compile a kernel from scratch, anyway, I went ahead and used my USB Zip drive to copy over the 2.4.24 kernel source code. Then, I extracted, configured, and compiled my kernel.

Next, I compiled and installed the drivers for the network interface, available from The Linux-WLAN Project. After installation of the drivers, the log file said that the device had no driver claiming it. So, I actually found the source code that links the product/vendor code to a driver and edited the prism2sta.c file to add the cod
es in. I recompiled the modules and reinstalled. At this point, the interface was recognized.

The next step was to get the kernel to set up the network interface automatically. On my home network, I have a DHCP server set up, but there are two commands that are needed to turn on the radio and associate with the Access Point. To get these to be executed automatically when you plug in the card, place the following two commands in /etc/network/interfaces:
pre-up wlanctl-ng wlan0 lnxreq_ifstate ifstate=enable
pre-up wlanctl-ng wlan0 lnxreq_autojoin ssid=”” authtype=opensystem

Once the network is configured, the next step is to install the streaming software. At slimdevices.com, I downloaded the tarball, rather than the RPM. After extracting the code, I
copied it into place and ran the software. It defaults to using port 9000, but that can be changed in the server interface. I changed it to port 80, for ease of access. At the same time, I changed the directory of the music library to point to my collection, once I got it copied to the server.

After getting the software laid out in the right locations, I decided I wanted to add it into the init system, so I needed to have a daemon startup and shutdown script. It turns out that the one for the sympa mailing list manager worked well for me, after some slight modifications. After creating the script for the init.d directory, I created the appropriate symbolic links in the rcX.d directories.

There are some last minute details that I am still working out, such as using NFS for native Linux mounting from other machines and Samba for Windows file sharing. The only other large project yet to conquer is a firewall. Of course, the Shorewall package will dramatically simplify that project.

In a matter of a couple of days of experimentation, I have set up my old laptop to stream and share my entire music collection to any machine that is on my network, giving me the freedom to listen to my music anywhere in the house or out in the yard, when it finally warms up in Ohio.

Fortunately for me, the internal Winmodem is a Lucent Technologies chipset that just happens to have a Linux driver, available at http://www.physcip.uni-stuttgart.de/heby/ltmodem/ The package was simple to install, and it works quite well, although I should try to figure out why the driver told me that it is a v.92 modem, or if this is normal with a controllerless modem.

My major problem was dealing with the issue of how to treat my wireless and wired ethernet cards (Wifi used at school and wired at home) differently for the purposes of Shorewall. It was not until reading more documentation on the Shorewall website was I able to figure out the hosts file and get my laptop to understand that the home zone was a subset of the net zone.

I decided to treat North Central as a hostile environment, though it would be less hostile than hanging this machine off a cable modem directly. Me, being the paranoid security nut that I am, chose the more secure environment. Nearly everything is closed off, particularly anything inbound not directly related to my browsing or other activities.

For home, I have opened up SSH and FTP (inbound and outbound), the two services I regularly use on my home network. If I need more, I can always add rules or take down the firewall temporarily. Of course, the same outbound connections are enabled so that I can connect to the internet using my desktop machine as a gateway.

Now that I have a better understanding of Shorewall and its internals, I have decided that it is very cool. It does a great job of blocking unusual traffic and common spoofed traffic while making it easy to configure what traffic should go through.

The default and preferred method of connecting to their chat system is through their Java Jabber applet, but it is possible to connect to their server with a standard client. I will not reveal those details here, in case TechTV does not want those released, but it makes it nice to use TKabber under Linux, where Java support can get a little hairy if not set up properly.

Just a few minutes ago, I figured out the problem. All programs attempting to connect to the ‘lo’ interface (all my local servers) were unable to work because both my eth0 and lo interfaces were non-functional… Something knocked out my /etc/network/interfaces files, which is the Debian method for configuring all the network interfaces (except ppp). I recreated the file, and issued an ‘ifup eth0′ and ‘ifup lo’, and things seem to be working again.