Backups.. (Again!)

For a little while now, I’ve been working on a new backup script, as switching from Windows to a full Linux environment rendered my last script unusable.

So, here is the full script, it works well, but I’m making tweaks to it.

One issue I Have, is certain users don’t exist on certain machines, so my development branch doesn’t quite work. I’ve managed to loop through an array contains user names, and another looping through IP addresses, however I’ve not worked out away yet to filter those arrays to remove IP and user names that don’t exist on the target.

But basically, this script connects to machines using RSYNC, over SSH and copies the files to the backup location, and optionally uploads them to Amazon S3.

So, without further ado, here is a wall of code..

#!/bin/bash

#Main Script file for Back ups.

#See Bitbucket Repo for further information https://bitbucket.org/thompsonmichael/backup-sys

#Michael Thompson 2018

# mikethompson@gmx.co.uk (GPG Key-ID: 062C03D9)

#Version 0.0.1

#VARIABLES

BCK_DEST=/mnt/Logical_Data

EXCLUDE_FILE=/home/michael/Script/rsync_exclude

S3_BUCKET=s3://

RSYNC_CMD_STD="azh --progress"

RSYNC_CMD_CLEAN=" --delete-after --delete-excluded"

S3_CMD="-rHv --skip-existing --acl-private --continue-put --storage-class=STANDARD_IA --no-delete-removed --exclude-from=s3_exclude"

S3_EXTRA=$2

LOG_FILE="/home/michael/Script/log_file.log"

REM_HOST="192.168.0.2"

BLUE="\e[1;34m"

RED="\e[1;31m"

NORMAL_COL="\e[0m"

if ! [ -z "$2" ];

then

if ! [ "$2" = "-clean" ];

then

echo "Running Custom S3 Command"

S3_CMD=$S3_CMD" "$S3_EXTRA

fi

fi

echo Backing up systems

echo ______________

echo -e ${BLUE} S3 Bucket Configured: $RED $S3_BUCKET${NORMAL_COL}

echo -e ${BLUE}S3 Command is: $RED $S3_CMD${NORMAL_COL}

echo -e ${BLUE}Exclude File Path: $RED $EXCLUDE_FILE${NORMAL_COL}

echo -e ${BLUE}Running on: $RED $HOSTNAME${NORMAL_COL}

echo -e ${BLUE}Destination is: $RED $BCK_DEST${NORMAL_COL}

if [ -z "$1" ];

then

echo -e ${BLUE}Command line passed: Empty ${NORMAL_COL}

else

echo -e ${BLUE}Command line passed: $1 ${NORMAL_COL}

fi

echo

echo -----------------------------------------------------------------------

echo

#error function. pass as func error,code,message

function_error () {

echo -e ${RED}"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

echo -e "CRITICAL ERROR!"

echo -e "Error occured in: " $1 "Error returned was: " $2

if [ -z "$3" ];

then

echo -e "Unknown Error, cannot advise. Check FAQ"

fi

echo -e "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"${NORMAL_COL}

return 1

exit

}

#ENSURE DRIVE IS MOUNTED

if mountpoint -q $BCK_DEST

then

echo -e ${BLUE}"Backup Location is mounted " $BCK_DEST ${NORMAL_COL}

else

function_error "Backup Location not mounted" $? "Mount location and restart"

exit

fi

if [ -z "$1" ];

then

#--------------------------------------------------------------------------------------

#Copy To Local Storage

echo Command Returned: $?

echo -e ${BLUE}Backing Up Dads${NORMAL_COL}

wget -q --tries=10 --timeout=20 --spider http://google.com

if [[ $? -eq 0 ]]; then

echo -e "Internet Connected"

else

function_error "Internet Connection Down" "INT_DOWN $?" "Check Internet Connection"

exit

fi

ping -c1 ${REM_HOST} -q 2>&1 >/dev/null

RET=$?

if [ ${RET} -eq 0 ]; then

echo -e ${BLUE}"Host Is Alive"${NORMAL_COL}

echo -e ${BLUE}Ping Command Returned: $? ${NORMAL_COL}

echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )"${NORMAL_COL}

rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dads

echo -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USER

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

echo -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}

else

echo -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}

echo -e ${RED}"Ping Command Returned (Ping):" $? ${NORMAL_COL}

echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}

fi

echo -e ${BLUE}Backing up $HOSTNAME${NORMAL_COL}

echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( $HOSTNAME )"${NORMAL_COL}

rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debian

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

echo -e ${BLUE}"Command Returned (RSync):" $?${NORMAL_COL}

fi

if [ "$1" = "-clean" ];

then

ping -c1 ${REM_HOST} -q 2>&1 >/dev/null

RET=$?

if [ ${RET} -eq 0 ]; then

echo -e ${BLUE}"Host Is Alive"${NORMAL_COL}

echo -e ${BLUE}"Ping Command Returned (Ping):" $? ${NORMAL_COL}

echo -e ${BLUE}"Host is Alive" ${NORMAL_COL}

echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )" ${NORMAL_COL}

rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dads

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

echo -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}

rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USER

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

echo "Command Returned (RSync):" $?

else

echo -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}

echo -e ${RED}"Ping Command Returned:" $? ${NORMAL_COL}

echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}

fi

echo -e ${BLUE}Backing up $HOSTNAME ${NORMAL_COL}

rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debian

if ! [ "$?" = "0" ];

then

function_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"

fi

echo -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}

#-----------------------------------------------------------------------------------------------------------------------------------

# -s3clean has been added as a command line option, and must be passed a second command to -clean

# it will cause a S3 clean event to be processed.

# -clean on its own will pass only a standard archive clean. S3 is not routinly cleaned, unless explicity passed with -s3clean.

if [ "$2" = "-s3clean" ];

then

#Call Clean_S3

source s3_cmd.sc

fi

#-----------------------------------------------------------------------------------------------------------------------------------

fi

if [ "$1" = "-s3" ];

then

echo S3 destination is: $S3_BUCKET

echo Amazon Upload Proceding...

echo Uploading $BCK_DEST/Backup/Dads/william/Pictures/

s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Pictures/ $S3_BUCKET/Dads/Pictures/

if ! [ "$?" = "0" ];

then

function_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"

fi

echo "Command Returned (S3CMD):" $?

echo Uploading $BCK_DEST/Backup/Dads/william/Documents/

s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Documents/ $S3_BUCKET/Dads/Documents/

if ! [ "$?" = "0" ];

then

function_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"

fi

echo "Command Returned (S3CMD):" $?

echo Uploading $BCK_DEST/Backup/Dads/william/Videos/

s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Videos/ $S3_BUCKET/Dads/Videos/

if ! [ "$?" = "0" ];

then

function_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"

fi

echo "Command Returned (S3CMD):" $?

echo Uploading $BCK_DEST/Backup/Michael-Debian/

s3cmd sync $S3_CMD $BCK_DEST/Backup/Michael-Debian/ $S3_BUCKET/Michael-Debian/

if ! [ "$?" = "0" ];

then

function_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"

fi

echo "Command Returned (S3CMD):" $?

exit

fi

#EOF

To Strava, or not to Strava

I’ve been a loyal user of strava for years. But I’ve been thinking recently of dumping my premium subscription. Why? 

Well, in all honesty, it just no longer fits my use requirements. I’ve never really used the segments, other than beating my own times. I’m not interested in bragging rights or KOMS. One of strava’s issues is that there will always people out there who will game the system to get the bragging rights. The leaderboard is knackered due to this.

Garmin recently brought segments to many GPS units. I used to use it, but due to a bug with the garmin firmware and the inability to set a default target it fell out of favour.

So what else is out there? Well, there are a few logging sites. None of them have the same pull as strava, but my current usage is Training Peaks and Ridewithgps.com. 

Training peaks is arguably the grand father of them all. With no limits to the way you can view data, and the sheer amount you can do, it is the greatest at seeing your performance improve. But it can be a massive data overload. 

Ridewithgps is simple, but powerful. With support for ride tagging, gear management and maintenance logs. It’s recently started to gear itself more toward clubs, but it’s still a great choice for lone riders who arnt looking for a social aspect, although it does offer that, but so far has failed to pick up the type of following strava enjoys. 

And that’s what strava has going, the social aspect. It’s simply unmatched in that regard. And that’s why I’ll continue to keep a strava account, but not a premium. It just has no benefit for me.