Recently I switched my last computer over to linux, and this meant I had to rethink my backup strategy. I like to copy all important files over to a central area, maybe even to a remote NAS box. In windows this was done via a simple batch file each user ran when they felt like it to copy it to a shared drive on the network. On linux, I can utilise the power of rsync, and do it all from a local terminal. So I cooked up the following bash script.
There are a few dependencies this has, it requires the following:
- s3cmd installed and configured to the S3 account
- rsync installed on all machines. (Not installed as standard on Debian systems.)
- An available drive or NAS box attached.
- If passwordless required for connecting to the other boxes, you will need to set up passwordless SSH, which I’ll not cover here, but there are countless other tutorials to cover this.
It can be run from a simple cron job, or manually when required. There are command line options that can be run
- -clean
- -clean command line will cause rsync to delete any files that are no longer on the source from the archive. It will also remove any files that have been copied, but now excluded from the exclusion file.
- -s3
- Will cause an upload event and syncronise the Amazon S3 archive with the local. This will not however trigger a local update event
- Empty command line
- Will simply backup files to the target machine from the sources, this will not clean the archives, nor will it trigger a S3 upload.
So, lets break down the source and have a look.
if mountpoint -q $BCK_DEST
then
echo "Backup Location is mounted"
if [ -z "$1" ];
then
echo rsync command is: -$RSYNC_CMD_STD
#--------------------------------------------------------------------------------------
#Copy To Local Storage
echo Backing Up
rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
#--------------------------------------------------------------------------------------
fi
This first section is the overall backup, responsible for copying the source to the target. We initially connect the rsync over to the target via SSH, and copy the entire home folder, minus the excluded files and directories. Its only run, should the command line be empty, the default run if you like. The variable $1 is the first command line option passed to the script. One of the important things we do here, is ensure that the drive is actually mounted in the system. If this is not mounted, everything else will fail.
if [ "$1" = "-clean" ];
then
echo RSync Clean Command is: -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN
echo Backing Up
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
fi
This section is the clean up. It runs the standard backup, along with the rsync options “–delete-after –delete-excluded” which clean up the archives.
if [ "$1" = "-s3" ];
then
echo S3 destination is: $S3_BUCKET
echo Amazon Upload Proceding
s3cmd sync $S3_CMD $BCK_DEST/1 $S3_BUCKET/1
exit
fi
This section is the amazon upload. Its not quite complete,.and there are more directories to add to the upload. However, uploading to amazon, is not the quickest thing in the world, and I’ll add the other important directories as and when the uploads complete.
And here it is, in its entire bash like glory.
#!/bin/bash
#VARIABLES
BCK_DEST=/mnt/sdc1
EXCLUDE_FILE=rsync_exclude
S3_BUCKET=S3 Bucket
RSYNC_CMD_STD=avzh
RSYNC_CMD_CLEAN=" --delete-after --delete-excluded"
S3_CMD="-rH --skip-existing --delete-removed --acl-private"
echo backing up systems
echo ______________
echo Exclude File Path: $EXCLUDE_FILE
echo running on: $HOSTNAME
echo destibation is: $BCK_DEST
echo Command line passed: $1
#ENSURE DRIVE IS MOUNTED
if mountpoint -q $BCK_DEST
then
echo "Backup Location is mounted"
if [ -z "$1" ];
then
echo rsync command is: -$RSYNC_CMD_STD
#--------------------------------------------------------------------------------------
#Copy To Local Storage
echo Backing Up
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
#--------------------------------------------------------------------------------------
fi
if [ "$1" = "-clean" ];
then
echo RSync Clean Command is: -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN
echo Backing Up 1
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
fi
if [ "$1" = "-s3" ];
then
echo S3 destination is: $S3_BUCKET
echo Amazon Upload Proceding
s3cmd sync $S3_CMD $BCK_DEST/1 $S3_BUCKET/1
exit
fi
else
echo "Backup Location is not mounted"
exit
fi
Hope you find this useful, and if you have any ideas how to improve it, let me know.
Mikes Blog 