Backups.. (Again!)

For a little while now, I’ve been working on a new backup script, as switching from Windows to a full Linux environment rendered my last script unusable.

So, here is the full script, it works well, but I’m making tweaks to it.

One issue I Have, is certain users don’t exist on certain machines, so my development branch doesn’t quite work. I’ve managed to loop through an array contains user names, and another looping through IP addresses, however I’ve not worked out away yet to filter those arrays to remove IP and user names that don’t exist on the target.

But basically, this script connects to machines using RSYNC, over SSH and copies the files to the backup location, and optionally uploads them to Amazon S3.

So, without further ado, here is a wall of code..

#!/bin/bash#Main Script file for Back ups.#See Bitbucket Repo for further information https://bitbucket.org/thompsonmichael/backup-sys#Michael Thompson 2018# mikethompson@gmx.co.uk (GPG Key-ID: 062C03D9)#Version 0.0.1#VARIABLESBCK_DEST=/mnt/Logical_DataEXCLUDE_FILE=/home/michael/Script/rsync_excludeS3_BUCKET=s3://RSYNC_CMD_STD="azh --progress"RSYNC_CMD_CLEAN=" --delete-after --delete-excluded"S3_CMD="-rHv --skip-existing --acl-private --continue-put --storage-class=STANDARD_IA --no-delete-removed --exclude-from=s3_exclude"S3_EXTRA=$2LOG_FILE="/home/michael/Script/log_file.log"REM_HOST="192.168.0.2"BLUE="\e[1;34m"RED="\e[1;31m"NORMAL_COL="\e[0m"if ! [ -z "$2" ];thenif ! [ "$2" = "-clean" ];thenecho "Running Custom S3 Command"S3_CMD=$S3_CMD" "$S3_EXTRAfifiecho Backing up systemsecho ______________echo -e ${BLUE} S3 Bucket Configured: $RED $S3_BUCKET${NORMAL_COL}echo -e ${BLUE}S3 Command is: $RED $S3_CMD${NORMAL_COL}echo -e ${BLUE}Exclude File Path: $RED $EXCLUDE_FILE${NORMAL_COL}echo -e ${BLUE}Running on: $RED $HOSTNAME${NORMAL_COL}echo -e ${BLUE}Destination is: $RED $BCK_DEST${NORMAL_COL}if [ -z "$1" ];thenecho -e ${BLUE}Command line passed: Empty ${NORMAL_COL}elseecho -e ${BLUE}Command line passed: $1 ${NORMAL_COL}fiechoecho -----------------------------------------------------------------------echo#error function. pass as func error,code,messagefunction_error () {echo -e ${RED}"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"echo -e "CRITICAL ERROR!"echo -e "Error occured in: " $1 "Error returned was: " $2if [ -z "$3" ];thenecho -e "Unknown Error, cannot advise. Check FAQ"fiecho -e "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"${NORMAL_COL}return 1exit}#ENSURE DRIVE IS MOUNTEDif mountpoint -q $BCK_DESTthenecho -e ${BLUE}"Backup Location is mounted " $BCK_DEST ${NORMAL_COL}elsefunction_error "Backup Location not mounted" $? "Mount location and restart"exitfiif [ -z "$1" ];then#--------------------------------------------------------------------------------------#Copy To Local Storageecho Command Returned: $?echo -e ${BLUE}Backing Up Dads${NORMAL_COL}wget -q --tries=10 --timeout=20 --spider http://google.comif [[ $? -eq 0 ]]; thenecho -e "Internet Connected"elsefunction_error "Internet Connection Down" "INT_DOWN $?" "Check Internet Connection"exitfiping -c1 ${REM_HOST} -q 2>&1 >/dev/nullRET=$?if [ ${RET} -eq 0 ]; thenecho -e ${BLUE}"Host Is Alive"${NORMAL_COL}echo -e ${BLUE}Ping Command Returned: $? ${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )"${NORMAL_COL}rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dadsecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}if ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"firsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USERif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}elseecho -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}echo -e ${RED}"Ping Command Returned (Ping):" $? ${NORMAL_COL}echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}fiecho -e ${BLUE}Backing up $HOSTNAME${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( $HOSTNAME )"${NORMAL_COL}rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debianif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $?${NORMAL_COL}fiif [ "$1" = "-clean" ];thenping -c1 ${REM_HOST} -q 2>&1 >/dev/nullRET=$?if [ ${RET} -eq 0 ]; thenecho -e ${BLUE}"Host Is Alive"${NORMAL_COL}echo -e ${BLUE}"Ping Command Returned (Ping):" $? ${NORMAL_COL}echo -e ${BLUE}"Host is Alive" ${NORMAL_COL}echo -e ${BLUE}rsync command is: "-$RSYNC_CMD_STD ( ${REM_HOST} )" ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE william@${REM_HOST}:/home/william $BCK_DEST/Backup/Dadsif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN -e ssh --exclude-from $EXCLUDE_FILE $USER@${REM_HOST}:/home/$USER $BCK_DEST/Backup/$USERif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho "Command Returned (RSync):" $?elseecho -e ${RED}"Host ${REM_HOST} failed ping monitoring on `date`"${NORMAL_COL}echo -e ${RED}"Ping Command Returned:" $? ${NORMAL_COL}echo -e ${RED}"${REM_HOST} is Dead"${NORMAL_COL}fiecho -e ${BLUE}Backing up $HOSTNAME ${NORMAL_COL}rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE /home/michael $BCK_DEST/Backup/Michael-Debianif ! [ "$?" = "0" ];thenfunction_error "RSYNC" $? "Check RSYNC Command Line, and RSYNC Dirs"fiecho -e ${BLUE}"Command Returned (RSync):" $? ${NORMAL_COL}#-----------------------------------------------------------------------------------------------------------------------------------# -s3clean has been added as a command line option, and must be passed a second command to -clean# it will cause a S3 clean event to be processed.# -clean on its own will pass only a standard archive clean. S3 is not routinly cleaned, unless explicity passed with -s3clean.if [ "$2" = "-s3clean" ];then#Call Clean_S3source s3_cmd.scfi#-----------------------------------------------------------------------------------------------------------------------------------fiif [ "$1" = "-s3" ];thenecho S3 destination is: $S3_BUCKETecho Amazon Upload Proceding...echo Uploading $BCK_DEST/Backup/Dads/william/Pictures/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Pictures/ $S3_BUCKET/Dads/Pictures/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Dads/william/Documents/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Documents/ $S3_BUCKET/Dads/Documents/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Dads/william/Videos/s3cmd sync $S3_CMD $BCK_DEST/Backup/Dads/william/Videos/ $S3_BUCKET/Dads/Videos/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?echo Uploading $BCK_DEST/Backup/Michael-Debian/s3cmd sync $S3_CMD $BCK_DEST/Backup/Michael-Debian/ $S3_BUCKET/Michael-Debian/if ! [ "$?" = "0" ];thenfunction_error "S3CMD" $? "Check S3CMD Command Line, and Dirs"fiecho "Command Returned (S3CMD):" $?exitfi#EOF

Power of the edits..

While I always like to process images as little as possible, if at all, some times it’s a requirement of every shot.

Here, for instance this shot taken from my Spark. It lacks depth and colour. Visually, it just didn’t stand out.

However, a little time spent in the Google Photos editor tweaking exposure, clarity, saturation and blacks improve things.

Shadows have a bit more depth, the colours are less muted but still hold the autumn setting.

You don’t take a photograph, you make it

Ansel Adams

So, don’t be afraid to play with the sliders, just don’t overdo it, and keep it realistic and you’ll do just fine.

Wind.

The spark has continued to impress with its ability in windy conditions. I’ve flown it in some questionable wind speeds, bit being careful not to take it too far. So far, there’s only been one occasion where I’ve noticed it not being able to cope and that was on blubber houses crag. And by god, it was seriously windy.

Up til now, I have had no way of actually knowing the wind speed conditions, apart from looking on apps and the weather reports, so I’ve invested little more than a tenner for an anemometer from Amazon to measure the wind speed.

This will let me know exactly what the wind speed is, and honestly, for £12, it’s a good buy.

Range..

So, in Europe the power of a radio transmitter used in drones is limited to 18dbm, well below the ability of the controller’s 25dbm.

This equates to a massive cut in the range of the spark, about 500 meters is the limit before the video signal gives up and the drone returns. But fear not, there is something you can do to get the full range.

It basically involves fooling the DJI Go app to think your in a country that uses the FCC rules instead of the CE rules. To do it, follow these steps.

  • Ensure the drone and RC are both off
  • Turn aeroplane mode on your phone or tablet to On, and make sure to fully quit the DJI Go app
  • Install a fake GPS app and enable it.
  • Set your location somewhere in America or Canada.
  • Turn on the RC and drone and allow them to.connect.
  • Connect to the aircraft’s WiFi signal
  • Once the app starts up, it’ll ask you if you want to change the WiFi settings. Select ok.
  • Wait for the drones/RC’s network to come back online, and power everything off
  • Quit the app and the fake GPS provider, restart the drone/RC and app
  • When asked again to alter the WiFi settings, hit cancel

And that’s it. Each time you launch the app.from here, it’ll ask you to change the settings, always cancel the dialog.

To confirm your in FCC mode, goto the wireless page in the app, and look if there is a channel 12/13, if not your in FCC mode. Range in this mode will get you in excess of 1000 meters, as opposed to the 500 in the CE mode.

Happy flights!

A tad breezy

A bit windy for drone work, but the little spark put up a good effort in battling the wind, even if it did kill the battery in short order.

One thing I seem to be having issues with, is the controls. I have it set to the default at the moment, which is mode 2. This means the left stick controls elevation and turning movement, the right stick controls yaw and forward/rear movement. My mind always seems to try and reverse this in my hands and find I use the wrong stick for the command. So maybe I’ll stick with it and try and retrain my mind, or just change it.

The only other thing is working out which way the drone is going to move. Based on the drones orientation, moving the stick may not yield the results you expect. So knowing the orientation of the drone is important, but working it out can be a bit of a headache.

Little Drones

I recently purchased a DJI Spark drone. I’m having loads of fun working out how to fly it, and get those fantastic cinematic shots.

So far, I haven’t really got the hang of shooting with it, but damn it’s fun. However, finding somewhere to fly it, legally is troublesome. Your not supposed to fly within 50 meters of any person or property. My first flight didn’t obey that. We have an area in Harrogate called the Stray. A huge grass area, I played about scuttling the drone around, up high and just working out how it all works. My second flight was amongst the walkers, cyclists and dog walkers on Bilton fields. I found a nice quiet area and sent the drone off on its merry dance. All fine until a dog took an unhealthy interest. Watched the owners chase it about for ages until they caught it.

As I’m not using Windows anymore, finding a Linux based video editor that ticks all my boxes is a little complicated. I’ve settled for kdenlive at the moment. But time will tell if I settle with it. The cache editor built into the DJI app is quite good for quick edits.

Camera..

I picked up my trusty Nikon for the first time in ages, and headed out to the deer park at Studley Royal.

I thought the deer may of started rutting. They hadn’t, but I found a few deer, and they where play fighting for a while.

I had mounted my longest zoom, mounted it on a monopod as the weight of the lens and camera make hand holding almost impossible.

https://www.instagram.com/p/BmlpNStByhF/?utm_source=ig_share_sheet&igshid=1mnklzolrjxhv

I think this image worked better in B&W, I had difficulty getting the exposure, as they where sat under a large oak, giving plenty of shade.

Blocking China

Tired of seeing Chinese IP addresses in my server logs trying SSH access, I decided to completly block it off.

Easy way todo this? Yup.


#!/bin/sh

cd /home/michael/System\ Scripts/Sineo_IPTABLES_Block/

wget http://www.okean.com/antispam/iptables/rc.firewall.sinokorea

sed -i 's/INPUT/SINEO/g' /home/michael/System\ Scripts/Sineo_IPTABLES_Block/rc.firewall.sinokorea

sed -i '/iptables -A SINEO -m state --state ESTABLISHED,RELATED -j ACCEPT/d' /home/michael/System\ Scripts/Sineo_IPTABLES_Block/rc.firewall.sinokorea

chmod +x /home/michael/System\ Scripts/Sineo_IPTABLES_Block/rc.firewall.sinokorea

sudo /home/michael/System\ Scripts/Sineo_IPTABLES_Block/rc.firewall.sinokorea

This will download a formatted list of chinese and Korean IP addresses, change the iptables chain to point to a new chain rather than the input chain, remove the now pointless established rule, and install the chain. Obviously, change the file paths to suit your structure.

It will however, require root to install the chain.

Backup, Its Important!

Recently I switched my last computer over to linux, and this meant I had to rethink my backup strategy. I like to copy all important files over to a central area, maybe even to a remote NAS box. In windows this was done via a simple batch file each user ran when they felt like it to copy it to a shared drive on the network. On linux, I can utilise the power of rsync, and do it all from a local terminal. So I cooked up the following bash script.
There are a few dependencies this has, it requires the following:

  1. s3cmd installed and configured to the S3 account
  2. rsync installed on all machines. (Not installed as standard on Debian systems.)
  3. An available drive or NAS box attached.
  4. If passwordless required for connecting to the other boxes, you will need to set up passwordless SSH, which I’ll not cover here, but there are countless other tutorials to cover this.

It can be run from a simple cron job, or manually when required. There are command line options that can be run

  1. -clean
    1. -clean command line will cause rsync to delete any files that are no longer on the source from the archive. It will also remove any files that have been copied, but now excluded from the exclusion file.
  2. -s3
    1. Will cause an upload event and syncronise the Amazon S3 archive with the local. This will not however trigger a local update event
  3. Empty command line
    1. Will simply backup files to the target machine from the sources, this will not clean the archives, nor will it trigger a S3 upload.

So, lets break down the source and have a look.


if mountpoint -q $BCK_DEST
then
echo "Backup Location is mounted"

if [ -z "$1" ];
then
echo rsync command is: -$RSYNC_CMD_STD
#--------------------------------------------------------------------------------------
#Copy To Local Storage
echo Backing Up
rsync -$RSYNC_CMD_STD -e ssh --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2

#--------------------------------------------------------------------------------------
fi

This first section is the overall backup, responsible for copying the source to the target. We initially connect the rsync over to the target via SSH, and copy the entire home folder, minus the excluded files and directories. Its only run, should the command line be empty, the default run if you like. The variable $1 is the first command line option passed to the script. One of the important things we do here, is ensure that the drive is actually mounted in the system. If this is not mounted, everything else will fail.

if [ "$1" = "-clean" ];
then
echo RSync Clean Command is: -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN
echo Backing Up
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
fi

This section is the clean up. It runs the standard backup, along with the rsync options “–delete-after –delete-excluded” which clean up the archives.

if [ "$1" = "-s3" ];
then
echo S3 destination is: $S3_BUCKET
echo Amazon Upload Proceding
s3cmd sync $S3_CMD $BCK_DEST/1 $S3_BUCKET/1
exit
fi

This section is the amazon upload. Its not quite complete,.and there are more directories to add to the upload. However, uploading to amazon, is not the quickest thing in the world, and I’ll add the other important directories as and when the uploads complete.

And here it is, in its entire bash like glory.

#!/bin/bash

#VARIABLES
BCK_DEST=/mnt/sdc1
EXCLUDE_FILE=rsync_exclude
S3_BUCKET=S3 Bucket
RSYNC_CMD_STD=avzh
RSYNC_CMD_CLEAN=" --delete-after --delete-excluded"
S3_CMD="-rH --skip-existing --delete-removed --acl-private"

echo backing up systems
echo ______________
echo Exclude File Path: $EXCLUDE_FILE
echo running on: $HOSTNAME
echo destibation is: $BCK_DEST
echo Command line passed: $1

#ENSURE DRIVE IS MOUNTED
if mountpoint -q $BCK_DEST
then
echo "Backup Location is mounted"

if [ -z "$1" ];
then
echo rsync command is: -$RSYNC_CMD_STD
#--------------------------------------------------------------------------------------
#Copy To Local Storage
echo Backing Up
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2

#--------------------------------------------------------------------------------------
fi

if [ "$1" = "-clean" ];
then
echo RSync Clean Command is: -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN
echo Backing Up 1
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/1
echo Backing up $HOSTNAME
rsync -$RSYNC_CMD_STD$RSYNC_CMD_CLEAN --exclude-from $EXCLUDE_FILE REMOTE SYSTEM $BCK_DEST/2
fi

if [ "$1" = "-s3" ];
then
echo S3 destination is: $S3_BUCKET
echo Amazon Upload Proceding
s3cmd sync $S3_CMD $BCK_DEST/1 $S3_BUCKET/1
exit
fi

else
echo "Backup Location is not mounted"
exit
fi

Hope you find this useful, and if you have any ideas how to improve it, let me know.

Another stupid maintenance story.

The bearings have gone in my fulcrum quattro, not unusual as Fulcrum ship with bearings that are sealed on one side only. So they do tend to go quite quick.

But I couldn’t be bothered to change them out yet. So I grabbed a spare set of wheels, a Mavic Aksium pair. Cassette and tyres mounted, on the bike and spinning. They are certainly not the best wheels in the world, but they work.

Off I went for a ride. Noise. Rubbing noise. Any noise on a bike annoys me. I checked the tyre clearances, break pads. All fine. Noise was still there. I adjusted the pads on the front wheel, I could swear the noise was coming from the front wheel. Checked spokes, all appeared to be tight enough with a simple squeeze test.

No idea what this noise was. Then, all of a sudden it disappeared, nothing, no noise apart from the road noise.

Shifted up into the large chain ring, and there it was again. I thought the cable may of come slightly loose as I hadn’t adjusted it since changing the cables, so turned the barrel adjuster a turn, span the cranks and the noise was worse. What the hell.

On look, the cable end was rubbing on the wheel…