Backups.. (Again!)

For a little while now, I’ve been working on a new backup script, as switching from Windows to a full Linux environment rendered my last script unusable.

So, here is the full script, it works well, but I’m making tweaks to it.

One issue I Have, is certain users don’t exist on certain machines, so my development branch doesn’t quite work. I’ve managed to loop through an array contains user names, and another looping through IP addresses, however I’ve not worked out away yet to filter those arrays to remove IP and user names that don’t exist on the target.

But basically, this script connects to machines using RSYNC, over SSH and copies the files to the backup location, and optionally uploads them to Amazon S3.

So, without further ado, here is a wall of code..

#!/bin/bash#Main Script file for Back ups.#See Bitbucket Repo for further information https://bitbucket.org/thompsonmichael/backup-sys#Michael Thompson 2018# mikethompson@gmx.co.uk (GPG Key-ID: 062C03D9)#Version 0.0.1#VARIABLESBCK_DEST=/mnt/Logical_DataEXCLUDE_FILE=/home/michael/Script/rsync_excludeS3_BUCKET=s3://RSYNC_CMD_STD="azh --progress"RSYNC_CMD_CLEAN=" --delete-after --delete-excluded"S3_CMD="-rHv --skip-existing --acl-private --continue-put --storage-class=STANDARD_IA --no-delete-removed --exclude-from=s3_exclude"S3_EXTRA=$2LOG_FILE="/home/michael/Script/log_file.log"REM_HOST="192.168.0.2"BLUE="\e[1;34m"RED="\e[1;31m"NORMAL_COL="\e[0m"if ! [ -z "$2" ];thenif ! [ "$2" = "-clean" ];thenecho "Running Custom S3 Command"S3_CMD=$S3_CMD" "$S3_EXTRAfifiecho Backing up systemsecho ______________echo -e ${BLUE} S3 Bucket Configured: $RED $S3_BUCKET${NORMAL_COL}echo -e ${BLUE}S3 Command is: $RED $S3_CMD${NORMAL_COL}echo -e ${BLUE}Exclude File Path: $RED $EXCLUDE_FILE${NORMAL_COL}echo -e ${BLUE}Running on: $RED $HOSTNAME${NORMAL_COL}echo -e ${BLUE}Destination is: $RED $BCK_DEST${NORMAL_COL}if [ -z "$1" ];thenecho -e ${BLUE}Command line passed: Empty ${NORMAL_COL}elseecho -e ${BLUE}Command line passed: $1 ${NORMAL_COL}fiechoecho -----------------------------------------------------------------------echo#error function. pass as func error,code,messagefunction_error () {echo -e ${RED}"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"echo -e "CRITICAL ERROR!"echo -e "Error occured in: " $1 "Error returned was: " $2if [ -z "$3" ];thenecho -e "Unknown Error, cannot advise. Check FAQ"fiecho -e "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"${NORMAL_COL}return 1exit}#ENSURE DRIVE IS MOUNTEDif mountpoint -q $BCK_DESTthenecho -e ${BLUE}"Backup Location is mounted " $BCK_DEST ${NORMAL_COL}elsefunction_error "Backup Location not mounted" $? "Mount location and restart"exitfiif [ -z "$1" ];then#--------------------------------------------------------------------------------------#Copy To Local Storageecho Command Returned: $?echo -e ${BLUE}Backing Up Dads${NORMAL_COL}wget -q --tries=10 --timeout=20 --spider http://google.comif [[ $? -eq 0 ]]; thenecho -e "Internet Connected"elsefunction_error "Internet Connection Down" "INT_DOWN $?" "Check Internet Connection"exitfiping -c1 ${REM_HOST} -q 2>&1 >/dev/nullRET=$?if [ ${RET} -eq 0 ]; thenecho -e ${BLUE}"Host Is Alive"${NORMAL_COL}echo -e ${BLUE}Ping Command Returned: $? ${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )"${NORMAL_COL}rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dadsecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}if ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"firsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USERif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}elseecho -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}echo -e ${RED}"Ping Command Returned (Ping):" $? ${NORMAL_COL}echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}fiecho -e ${BLUE}Backing up $HOSTNAME${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( $HOSTNAME )"${NORMAL_COL}rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debianif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $?${NORMAL_COL}fiif [ "$1" = "-clean" ];thenping -c1 ${REM_HOST} -q 2>&1 >/dev/nullRET=$?if [ ${RET} -eq 0 ]; thenecho -e ${BLUE}"Host Is Alive"${NORMAL_COL}echo -e ${BLUE}"Ping Command Returned (Ping):" $? ${NORMAL_COL}echo -e ${BLUE}"Host is Alive" ${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )" ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dadsif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USERif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho "Command Returned (RSync):" $?elseecho -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}echo -e ${RED}"Ping Command Returned:" $? ${NORMAL_COL}echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}fiecho -e ${BLUE}Backing up $HOSTNAME ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debianif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}#-----------------------------------------------------------------------------------------------------------------------------------# -s3clean has been added as a command line option, and must be passed a second command to -clean# it will cause a S3 clean event to be processed.# -clean on its own will pass only a standard archive clean. S3 is not routinly cleaned, unless explicity passed with -s3clean.if [ "$2" = "-s3clean" ];then#Call Clean_S3source s3_cmd.scfi#-----------------------------------------------------------------------------------------------------------------------------------fiif [ "$1" = "-s3" ];thenecho S3 destination is: $S3_BUCKETecho Amazon Upload Proceding...echo Uploading $BCK_DEST/Backup/Dads/william/Pictures/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Pictures/ $S3_BUCKET/Dads/Pictures/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Dads/william/Documents/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Documents/ $S3_BUCKET/Dads/Documents/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Dads/william/Videos/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Videos/ $S3_BUCKET/Dads/Videos/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Michael-Debian/s3cmd sync $S3_CMD $BCK_DEST/Backup/Michael-Debian/ $S3_BUCKET/Michael-Debian/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?exitfi#EOF

To Strava, or not to Strava

I’ve been a loyal user of strava for years. But I’ve been thinking recently of dumping my premium subscription. Why? 

Well, in all honesty, it just no longer fits my use requirements. I’ve never really used the segments, other than beating my own times. I’m not interested in bragging rights or KOMS. One of strava’s issues is that there will always people out there who will game the system to get the bragging rights. The leaderboard is knackered due to this.

Garmin recently brought segments to many GPS units. I used to use it, but due to a bug with the garmin firmware and the inability to set a default target it fell out of favour.

So what else is out there? Well, there are a few logging sites. None of them have the same pull as strava, but my current usage is Training Peaks and Ridewithgps.com. 

Training peaks is arguably the grand father of them all. With no limits to the way you can view data, and the sheer amount you can do, it is the greatest at seeing your performance improve. But it can be a massive data overload. 

Ridewithgps is simple, but powerful. With support for ride tagging, gear management and maintenance logs. It’s recently started to gear itself more toward clubs, but it’s still a great choice for lone riders who arnt looking for a social aspect, although it does offer that, but so far has failed to pick up the type of following strava enjoys. 

And that’s what strava has going, the social aspect. It’s simply unmatched in that regard. And that’s why I’ll continue to keep a strava account, but not a premium. It just has no benefit for me.