Sunday, September 30, 2012

ReadyNAS NV+ backup to DreamHost Part 2

One critical point I forgot to mention in my previous post about using Duplicity on a ReadyNAS NV+. I'd forgotten that I had installed an updated version of OpenSSH (5.2) which installs an sftp client among other things. Duplicity uses sftp for practically everything. Even when you specify --use-scp, it still expects to find sftp to check file lists etc.

However a sad discovery I made when I started to upload large files was that the upload stalls when uploading to DreamHost. I have no problem copying large files and directories to my MacBook Air on my LAN. And I have no problem of stalling when I copy the same files and directories to DH from my MBA through the same router and ADSL modem.

I've spent days searching the Net trying to find a fix which will work for the NV+. I've tried the 'MTU' fix, the 'pipe to ssh' fix, the '-l 8192' fix, the '-C' fix, the sysctl.conf fix, the 'rsync' fix etc. etc. All make absolutely no difference.

The only difference I can find in the logs of the two machines' uploads is that the MBA has a long series of 'debug2: channel 0: rcvd adjust nnn' lines and the NV+ has only one of those lines and then it stalls.

So for the moment Duplicity is unusable on the NV+ for backing up to DreamHost.

Friday, September 28, 2012

ReadyNAS NV+ backup to DreamHost Part 1

(NB The following assumes a fairly good knowledge of Debian Linux at the commandline level.)

Following a total loss of 3TB of files on my ReadyNAS NV+ I decided I needed a backup strategy. (Thankfully I have offsite copies of my most critical data.)

My strategy now is to use a surplus Drobo 1 as an onsite backup of the NV+ (see below), and my DreamHost (DH) account for an offsite backup of critical files. I decided that Duplicity would be a good tool to handle encrypted offsite backups and rsync can handle the backup to the Drobo.

I found a great article How to install Duplicity on a Netgear ReadyNAS Duo (the Duo is the 2-disk version of the NV+, both use Sparc chips) and I was able to follow it with no glitches. The sections about installing apt-get and using it to download .deb packages to create a usable Sparc compiling environment and using the '--build=sparc-linux' option were really useful as I discovered when trying to install the Drobo.

Then I followed this article to create a private and public key and put the public key in the .ssh/authorized_keys file in my DH ssh account. Then I created .ssh/config on the NV+ with the following contents:

Host dh
  HostName myhost.com
  User myname
  IdentityFile ~/.ssh/dh_id_rsa


(I renamed id_rsa to dh_id_rsa so as not to confuse it with other private keys I have in .ssh.)

I checked that I could log into my DH account from the NV+ by typing
ssh dh

at the NV+ prompt. So far so good.

Finally I typed the following Duplicity command:

duplicity --exclude-globbing-filelist $HOME/exclude.list \
--use-scp -v4 /media/Music scp://dh/backup/nas


and my Music directory was encrypted and copied to my DH account. So now it's just a case of setting up a cron entry to run this command and duplicate for other directories.

ReadyNAS NV+ backup to a Drobo 1

The Drobo 1 only allowed me to choose HFS+ or FAT32 for formatting it. But FAT32 has a volume limit of 2TB and I want a single 8TB volume so I have to use HFS+. I've only got 5TB of disks installed at the moment but I eventually want to replace them with 4x2TB and having a single (albeit virtual) volume makes upgrades a lot easier. The Drobo is surplus because it only has a USB 1 connector. No Ethernet, no FireWire etc. So it's very limited both in transfer speed and connectivity. I have it connected to my LAN using an Apple Airport Extreme Base Station (AEBS) which makes the Drobo into a (slow) NAS.

It seemed a pretty simple thing to connect the Drobo to one of the USB ports on the NV+ and use FrontView to schedule a periodic backup of my media folder to the Drobo. No such luck. The NV+ doesn't recognise the Drobo.

OK, so obviously I merely had to put the Drobo back into the AEBS and network mount the Drobo to the NV+. Unfortunately, while the NV+ supports AFP (Apple File Protocol) to allow the NV+ look like a Mac disk to a Mac, there is no corresponding driver on the NV+ to allow a Mac disk to be network mounted to it.

A Google search turned up afpfs-ng which uses FUSE (filesystem in user space).  I moved into /root/src (created when installing Duplicity) and downloaded the source file:
wget http://downloads.sourceforge.net/project/afpfs-ng/afpfs-ng/0.8.1/afpfs-ng-0.8.1.tar.bz2

Unfortunately there is no bunzip2 utility already installed on the NV+ and the version of tar installed is too old to handle bzip2 files, so once again:
apt-get install bzip2

Then I unzipped the afpfs-ng file:
bunzip2 afpfs-ng-0.8.1.tar.bz2
tar xf afpfs-ng-0.8.1.tar.bz2
and moved into the afpfs-ng-0.8.1 directory.

From the INSTALL file I found that afpfs-ng needs libgcrypt, libgmp, readline and libfuse (≥ 2.7.0). I used apt-get to install libgmp and readline:
apt-get install libgmp3 libgmp3-dev
apt-get install libreadline4 libreadline4-dev


[Edit] Also afpfs-ng build needs the g++ compiler installed. Once again:
apt-get install 'g++'
[Edit] And it needs libgpg-error-dev:
apt-get install libgpg-error-dev

apt-get told me that libgcrypt11 was already installed but it didn't appear to be linked correctly so I added the following:
ln -s /usr/lib/libgcrypt.so.11.2.2 /usr/lib/libgcrypt.so

I later discovered that gcrypt.h was missing from /usr/include but when I tried to
apt-get install libgcrypt11-dev
I got an error about package conflicts. I figured it was only one missing header file which wouldn't be much different between versions. So I tracked down the .deb package and downloaded it into /root/src:
wget http://archive.debian.org/debian/pool/main/libg/libgcrypt11/libgcrypt11-dev_1.2.0-11.1_sparc.deb

unpacked it into /root/src/usr:
dpkg -x libgcrypt11-dev_1.2.0-11.1_sparc.deb .
and copied the include directory files to /usr/include:
cp usr/include/*.h /usr/include
(Note no slash on first 'usr' but a slash on the 2nd)

Unfortunately the latest fuse .deb package is for 2.6.3 so I decided to compile fuse-2.7.0 from source.

So the steps to install fuse-2.7.0 from source are:

1) ssh login into the NV+ and
cd /root/src 
2) Download fuse-2.7.0 source:
wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.7.0/fuse-2.7.0.tar.gz
3) Unzip the file:
tar zxf fuse-2.7.0.tar.gz
4) Move into the fuse source directory
cd fuse-2.7.0
5) Configure, make and install
./configure --build=sparc-linux
make
make install

So finally I was ready to build afpfs-ng. The INSTALL file suggested adding the /usr/local directory to the compile search list:
CFLAGS="-I/usr/local/include -L/usr/local/lib" ./configure --build=sparc-linux

Then 'make'.

[Edit] The '--build=sparc-linux' config option causes the wrong fcntl.h header file to be included which causes "error: redefinition of 'struct flock'" to crash the make. Solution is to edit fuse/fuse_int.c, lines 27-31:
#ifdef __linux__
#include <asm/fcntl.h>
#else
#include <fcntl.h>
#endif

removing lines 27-29 and line 31 so that only line 30 appears:
#include <fcntl.h>
Save the file and re-run 'make'. This should now complete successfully.

Run 'make install'.
[Edit] run 'ldconfig' (to rescan the loadable libraries) and run 'afpfsd' (to start the FUSE daemon).

Next, using Airport Utility, on the disk tab, change authentication process to Accounts and add at least one account. Save the new config and when the AEBS front light returns to green mount the Drobo on the NV+ as follows:
mkdir /mnt/drobo
mount_afp afp://myuser:mypass@192.168.1.1 /mnt/drobo
where 192.168.1.1 is the IP address of the AEBS.

See here for info on how to make the mount happen automatically.

Now it's simply a matter of using rsync to backup NV+ directories to the Drobo.