Into the Wild Spaces

My first attempt was to write a little script that would work from the C shell, to be run by root. Rsync was clearly the right tool for this job, probably something like:

rsync -av /directory/to/sync /mnt/destination/directory

And there would have to be a mount command there too. Plugging a USB drive into the slot and watching the contents of dmesg revealed the machine was recognizing this FAT-formatted drive as /dev/da1. So as root, I tried:

mount_msdosfs -o /dev/da1 /mnt/external

The -o flag is necessary so the file system is prepped for a large drive. That seemed to work at first, but I wanted the errors to go to a log file to be sure, so I used instead:

rsync -av /directory/to/sync /mnt/destination/directory &2>errorlog.txt

That was important because I spotted a huge number of errors. First of all, the FAT filesystem doesn't recognize owners or groups of files, so I had to add the -o and -g flags to rsync to tell it to ignore ownership. But more seriously, rsync choked on any file with foreign characters, and in my music collection alone there were hundreds of files not being synced because of this issue. I went to the rsync developer's for help and didn't find any easy solutions. How could so many people be using this software and having no trouble?

The answer was in the filesystem. The problem was some combination of the MSDOS filesystem (ie, FAT) being mounted with inappropriate options, or the FAT system simply not liking the foreign characters. I tried an experiment: I opened up an rsync account at rsync.net and tried syncing the same troublesome files. No problem whatsoever; not a single error message. So that confirmed for me the problem was the file system on those external drives, not rsync itself. After some grunting and complaining, I decided to reformat the external drives to UFS and make them essentially "Unix disks"

This is no easy decision to make: hardly any system other than FreeBSD can read UFS. Linux supports it poorly and most other OSes don't support it at all. But hell, if it backs up my stuff and gives me some protection from disaster, who cares? And I'm never too far away from a FreeBSD box anyway, happily.

This blog gave me the way forward in formatting an external drive to the UFS filesystem. Catch: you need a FreeBSD box to do it, and if you don't have one, you've got to do it from the NAS itself. Dangerous, because if you format the wrong disk you hose your system. Fun times! Again, stick in the drive and monitor dmesg to see how it is recognized. In my case, it was (again) /dev/da1 (meaning /dev/da0 was probably part of my NAS: need to remember to avoid touching that one). OK, so:

fdisk -BI /dev/da1  # Format the external drive: one huge partition, bootable
bsdlabel -wB /dev/da1s1 # Write standard (bootable) disk label to the 1st partition
newfs -O2 -U /dev/da1s1a  # formatthe partition with UFS2 and soft updates

That worked without a hitch, and was as methodical and precise as I would have expected from FreeBSD. And now, to mount it. I rewrote my little script as follows:

#!/bin/sh
mount /dev/da1s1a /mnt/external
rsync -av /mnt/sharkstuff/randymon /mnt/external 2> rmonbkup-error.log
umount /mnt/external

It's not perfect, but it works. So now on a weekly basis, I plug in an external drive, ssh into the NAS and run:

sudo mkdir /mnt/external
sudo name-of-my-script

It copies everything over to the external drive and logs the errors to the error.log (but these days there are no errors, happily). Then it unmounts the drive. I have two drives and two separate data collections to back up so I have two copies of the script, one for each. Then I file the hard drives away somewhere physically separate from the NAS in case fire or water ever take out my box. And now I sleep easily! It would be no sweat to automate the script to run at a given time every Sunday but I'd have to be concious and organized to make sure the box were turned on and the right hard drive attached at the indicated time. This non-automated version works well enough.

Adventures in Unix!

Post script: I was told on the FreeNAS forums that support for the UFS file system would disappear in a future release of FreeNAS, Not sure what kind of uber-geek dick-swinging that was, but as of November 2020 I am running an up-to-date version of FreeNAS and this script still works perfectly. (If they do drop support for UFS drives, I'll simply format the external drives with ZFS and use "zfs pool send" or equivalent - cross that bridge when the time comes).