Recently in Off Topic Tech Category

Remember that giddy feeling when in high school you first ate lunch on the grass with that special someone, the object of your springtime infatuation? Ahhh. So sweet. I'm reliving that feeling with my newly installed NAS. I tenderly call her "323" for short, but her parents call her "D-Link DNS-323 2-Bay Network Storage Enclosure." I can see beyond her toaster looks...

I don't blog about every early Christmas present, but this NAS is so geek-winningly hackable, and I wound up doing such a number on my home network for it that I can't help but share the story. This may be helpful to other web wanderers, just as I relied on many blog posts, discussion forums, and so forth to get set up.

Benefits

First the benefits of this relationship:

1. Network Attached Storage -- You know at the office how nice it is to always have access to those never-ending shared drives that corporate IT provides. I now have it at home. Instead of keeping only select music stored on my computer and the rest locked away on that external USB drive in my wife's office, it's all available. The old cables and plugs were a barrier to access.

2. Peace of Mind -- With RAID-1 and two SATA drives, my data won't get lost when a hard drive fails.  And every hard drive fails sooner or later.

3. Openness -- The 323 runs an embedded Linux, and D-Link built a hook to let folks access the core. Extend it with Subversion, SSH, MySQL, or if you're crazy enough you can even install a new Debian.

4. FTP -- The built-in FTP server and granular security model lets me access, share, or backup content from outside the home.

5. iTunes Server -- The device can discover its music then broadcast it to iTunes clients on the network.

6. Scheduled Downloads -- It can schedule downloads of files and folders from an FTP server, web server, or local network share. I don't want to fully rely on my web hosting providers to backup my data, and this lets me keep a copy too.

The feature list is rich, but not all of it applies to me -- yet. We'll see how my thinking shifts as she and I get to know each other better. Other people though are interested in its BitTorrent feature, UPnP AV server, or others

Now for some details.

The Hard Drives

In keeping with the do-it-yourself offering, the 323 doesn't come with hard drives. It's just an enclosure. So what did I buy? I admit that I was driven by price rather than features, but I still wound up with a pair great drives. Amazon was selling Western Digital's energy efficient WD10EADS drives cheaper than any of the other 1TB options, at least with 7200 rpm. It's cool to be green (no matter what Kermit says), but I'm more excited about the cool temperature than the green energy savings. As drives heat up, the probability of failure increases dramatically. More on failures later. The 323 has a feature to monitor the temperature and at high levels, send an email alert and then shut down. I want this feature, but I also don't want it to ever be triggered. The drives were $69 each when I bought, but perhaps for the holidays they have since risen to $84.

Installing the drives to the 323 was easy. I just tore open the drive packages from Amazon, slid the front plate off the 323, and pushed in the drives. No tools required.

Improving the Home Network

In order for my wife and me to share the NAS, our laptops need IP addresses from the same network. Previously, we didn't have this. The Internet drop and primary wireless router (an old WRT54g) are in my office. Since her office is on the other side of the house, and since the house has built-in ethernet wiring from the location of the Internet drop, we put a secondary wireless router (an older BEFW11S4) near her office that pulls data from the ethernet port.That router though was configured the easy way, with DHCP enabled and placing her on a different network. I was on 10.1.10.x, and she was on 10.1.11.x. So here's what I did:

1. Made sure the primary router ran normally, with our ISP's Internet provided through the router's WAN port
2. Made sure the secondary router ran normally, with the primary router's Internet provided through the secondary router's WAN port
3. Changed the secondary router to use a static IP, which in retrospect may not have been necessary
4. Configured the secondary router's Setup->Advanced Routing page to both send and receive RIP 1, which may not have been necessary but one blog suggested
5. Moved the secondary router's ethernet cable from the WAN port to port 1 which does uplink.

That was it. Now when my wife connects to the secondary router by her office, it acts as a switch to get to the primary router, and the primary router gives her an IP address in 10.1.10.x so we can both communicate with the NAS.

Improving the 323

Out of the box the 323 is nice, but it really starts to get cool once you start treating it as a customizable Linux box instead of just a hard drive. The device has a thriving community supporting it, and it's a great example of how a company's decision to open their product up can improve its usefulness and cultivate buzz (e.g. this blog post). The best site for the product may be http://wiki.dns323.info/. I proceeded cautiously installing my first "fun_plug" file to execute my commands at startup based on instructions at that site. Once my feet were wet, I installed a package of Unix tools called "ffp" (Fonz fun_plug) following the instructions at nas-tweaks.net.

In no time at all, I had logged in through telnet, disabled that insecure service, set up SSH, and begun looking around. I then followed the instructions at another blog to install the usb-storage.ko module allowing me to mount Fat32 USB devices through the 323's USB port. I got my wife's old iPod Mini loaded up with little effort.

Breaking Up, not Backing Up, with Standard USB Hard Drives

After setting up a few directories on the hard drives with proper security, I powered up the old 250 gb USB drive that started all this. The prior time my wife started it was ten days ago, and that time like this time it behaved the same way: a few minutes of near-silent, slow clicking, then an awakening and normal operation. We suspect it's on the verge of death. Before going to sleep, I dragged the old drive's folders to the 323, and let it run through its 10 hour transfer from sinking skiff to reliable coast guard cutter.

What's the problem with standard USB drives, and why should you not rely on them? Every drive fails, and you don't want to backup to a device with a fuse burning toward self destruction. Before moving to the RAID solution and while looking for a replacement for the old USB drive, I realized that every drive Amazon sells, given enough reviews, will have some frightening proportion of customers saying this like "It died after two weeks and I lost all my data! I'm never buying from this company again! Avoid this drive!" Every drive does this. Really?

I did a little research, and I found a great paper put together by some Google engineers. The guys who support the Google infrastructure have to buy a lot of drives and must know something about failure rates, right? Failure Trends in a Large Disk Drive Population may be more scientific than you're interested in, but at least consider this picture:

Enough said.

Anyway, that's my personal tech journal for the week.

Enjoy!

Here's a post that will be of little interest to my normal readers but that may be helpful to Googlers. If this helps you, please drop a comment letting me know. I need encouragement to go so far off topic from my normal posts.

Several years ago I bought the history book on which the movie Amistad had been based. The Amistad was a ship carrying slaves to the Americans, and its captives revolted. The movie, which I didn't see, was apparently exciting enough, but the book was tedious. I wanted to never revisit it or anything like it again. But alas, I've encountered what could be called a slave revolt.

MySQL has a strange behavior on slaves with the CHANGE MASTER command that cost me a few hours of sleep. Sometimes when values are set with the command, those values merge into the master.info file. However in other cases after using the command, the values in master.info are lost. A sequence of commands that seemed reasonable to me left me without the proper master bin-log and offset log position, and this caused my slave to get errors like 'Duplicate entry for key 1.'

Here's how I discovered this behavior:

First, I created a dump using the syntax that places within the dump an update statement to set the master's position:

mysqldump --all-databases --master-data=1 --add-locks -u myuser -p > full.db.`date +"%F"`.dmp

Afterward, I can check my dump and find that indeed, it provides the master's bin-log and position:

CHANGE MASTER TO MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;

I then bring the dump to my slave server. If I first import the dump and rely on its values to set the master's position, I'll get errors when replication begins. The errors are caused because the replication picks up at the oldest bin-log instead of the right one. The errors, found after running "show slave status\G;" are like this:

Last_Error: Error 'Duplicate entry '3363837' for key 1' on query. Default database: 'myapp'. Query: 'INSERT INTO mytable (
                                    blah,
                                    blah2,
                                    blah3
                                ) VALUES(
                                    '1',
                                    '2009-11-01T00:06:16-05:00',
                                    'stuff'
                                )'
                                
What I really should have done to avoid the errors would have been to run a CHANGE MASTER command that stated everything rather than skipping the details that the dump included.

After looking into this further, I find that as expected, the dump creates a master.info file with the master's proper bin-log and offset, and that master.info doesn't yet have the server connection details. Then after providing just the connection details through CHANGE MASTER, contrary to my expectation it then wipes out the bin-log and offset values rather than properly merging. I can fix this by then providing just the bin-log and offset values, which are properly merged into master.info.

Commands illustrating this are below:

[root@myhost ~]# # import the master's data
[root@myhost ~]# mysql -u root -p{secret} < /tmp/full.db.2009-11-14.dmp
[root@myhost ~]# # see what the dump put into master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this first iteration of the file has no connection info
14
bin-log.000494
169

test

3306
60
0





[root@myhost ~]# # set the partial details as documented
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_HOST='10.1.1.14', MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='supersecret';"
cat /var/lib/mysql/master.info
[root@myhost ~]# # check if that put anything in master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this second iteration dropped the bin-log and log position
14

4
10.1.1.14
repl
supersecret
3306
60
0





[root@myhost ~]# # set the remaining details as though nothing had been in dump
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;"
[root@myhost ~]# # check if that put anything in master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this third iteration merged in the bin-log and log position
14
bin-log.000494
169
10.1.1.14
repl
supersecret
3306
60
0





[root@myhost ~]# # set everything and see the results:
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_HOST='10.1.1.14', MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='supersecret', MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;"
cat /var/lib/mysql/master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this fourth iteration that sets everything looks like the third iteration
14
bin-log.000494
169
10.1.1.14
repl
supersecret
3306
60
0



So in short, don't rely on the dump to set master.info values for you. Just put them all into your mysql prompt similar to this:

mysql> CHANGE MASTER TO MASTER_HOST='10.1.1.14',
    -> MASTER_PORT=3306,
    -> MASTER_USER='repl',
    -> MASTER_PASSWORD='supersecret',
    -> MASTER_LOG_FILE='bin-log.000494',
    -> MASTER_LOG_POS=169;

Enjoy!

To my normal readers, don't worry! This blog will soon return to its regular programming!

While installing Pandora on my Blackberry Pearl, I hit an obscure error message. Google only knew of a single page in the whole wide world wide web that discussed this error, but that page had no solution. So now for the perfect stranger who came here from Google, who doesn't care about Oracle software, who doesn't know me, and who just wants to listen to his or her reggae/polka/rap/country/classical music through the Blackberry, I bring you this special announcement:

Pandora provides different application downloads based on which device it thinks you're using, and it determines your device by your browser's emulation mode. So if you're not using the Blackberry emulation mode, then your device may try to download the wrong one. In my case, I saw the error "Unsupported media type: application/x-cabinet," and the device wanted to download pandora.cab. I was using MS Pocket IE mode when that happened. You may be using a different but still wrong mode, so try setting it properly. To do so:

1. Open your browser (and it must be the default browser--don't try Opera Mini or some such thing)
   
2. Click the Blackberry menu button
   
3. Select "Options"
4. Select "Browser Configuration"
5. Set your emulation mode to Blackberry
6. Navigate to http://www.pandora.com (or refresh if you had already been there)
7. Download, and you'll be in business
8. For good sound, use a headset of some sort
   
9. Give me a comment if this really did help you

Enjoy!