Another backup post

So I had a thought of redoing my backup scripts, as my existing script file was becoming huge, complicated and difficult to follow.

It was a good way for me to learn Bash scripting however. The idea was to vastly simplify the whole process. I began by thinking the best way, and decided to use each machine to independently backup to a network drive, rather than having a single machine doing the grunt work running a script.

So each computer on the network had the drive mounted, and the script file placed into the main users crontab. I could of used the root crontab to copy the whole /home/ directory of course, but each machine only has one real user so I opted for that.

The file that runs from crontab is very simple

source /mnt/dlink_nfs/backup-script/var-dec
rsync -va --delete-after --delete-excluded --exclude-from=$FOLDER_NFS/backup-script/exclude.lst /home/$USER $FOLDER_NFS/backup-test/$DIRNAME

And that’s it. Just uses rsync to copy the contents to the network drive . The referenced source file is just shared variable declarations.

Now, that’s not quite enough for me to be happy with a backup system, so I use a raspberry pi, to run a second set of scripts from its crontab. Those files are responsible for uploading to Amazon S3, and also copying to a secondary NAS.

#!/bin/bash

source /mnt/dlink_nfs/backup-script/var-dec
echo "Script Started: $(date)" >> uploads3.log

if pidof -x "$script_name" -o $$ >/dev/null;then
   echo "An another instance of this script is already running"
echo "Script Already running, exiting" >>  uploads3.log
echo "-----------------------"
   exit 1
fi

if [[ $1 == 'clean' ]]
	then
		echo "clean command passed" >> uploads3.log
		rsync -vruO --delete-after $FOLDER_NFS/backup-test /mnt/samba/
		echo "Clean compleated $(date)"	
		exit 1
else
	if mountpoint -q /mnt/samba
		then
echo "Samba share mounted, started RSYNC" >> uploads3.log
		rsync -vruO $FOLDER_NFS/backup-test /mnt/samba/
	fi

	cd $FOLDER_NFS/backup-test/
echo "Starting S3 uploads" >> /home/pi/uploads3.log
	shopt -s dotglob
	shopt -s nullglob
	array=(*/)
	echo runing s3

	for dir in "${array[@]}"
	 do 
		echo "Currently Running S3 on $dir" >> /home/pi/uploads3.log
		dir=${dir%/}
	        timeout 30m s3cmd $s3_cmd $dir $s3_bucket
		echo "Compleated uploading $dir" >> /home/pi/uploads3.log

	 done
echo "Finished Script: $(date)" >> /home/pi/uploads3.log
echo "--------------------" >> /home/pi/uploads3.log
fi

And that file basically, ensures the script isn’t already running, copies the backup to another NAS, then iterates through each directory uploading to S3. I use timeout to limit each upload to 30mins to prevent overruns. Once the initial upload has completed, this limit can be removed.

You can view the most up to date git repository at my github site https://github.com/mikethompson/new-backup