{"id":615,"date":"2017-11-22T13:00:31","date_gmt":"2017-11-22T13:00:31","guid":{"rendered":"http:\/\/chewett.co.uk\/blog\/?p=615"},"modified":"2017-11-22T21:42:28","modified_gmt":"2017-11-22T21:42:28","slug":"compressing-raspberry-pi-backups-automatically","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/615\/compressing-raspberry-pi-backups-automatically\/","title":{"rendered":"Compressing Raspberry Pi backups automatically"},"content":{"rendered":"<p>Today I will be improving the basic script used to backup my Raspberry Pi&#8217;s so that it compresses the files after they have been copied. This will allow me to store several backups instead of having a single copy stored on my computer. This assumes you have followed the first tutorial to <a href=\"https:\/\/chewett.co.uk\/blog\/569\/creating-raspberry-pi-backup-script-scp-bash-windows\/\">create a Raspberry Pi backup script<\/a>.<\/p>\n<p><!--more--><\/p>\n<h2>Compressing and archiving our backups<\/h2>\n<p>Our previous script copied over the files from the Raspberry Pi using a program called <code>scp<\/code>.\u00a0 This created local copies of the files from our Raspberry Pi.<\/p>\n<p>Each time the copy runs the previous files were overwritten which means I only have the current copy and nothing historic. Today I am going to improve the backup script by downloading the files and then creating an archive file. In addition to creating an archive file I am going to compress it so that it takes less space on my computer.<\/p>\n<h2>What is the .tar.gz file format?<\/h2>\n<p>A <code>.tar.gz<\/code> file, or <code>.tgz<\/code>\u00a0file, is a combination of two operations intended to create a compressed archive. The first step in creating a <code>.tgz<\/code> file is creating a <code>tar<\/code> archive of the files you want to compress. <code>tar<\/code>\u00a0is used to combine a file and folder structure into a single file that can be easily moved, the name comes from tape archive. This is because\u00a0originally tar was used to create a single file that could be written to a tape archive.<\/p>\n<p>Once we have created our tar file we then want to compress it so it uses less space on our filesystem. This can be done using the gzip compression algorithm.<\/p>\n<p>To create tar files I can use the unix\u00a0tool <code>tar<\/code>.\u00a0One of the useful features is that this natively supports a number of compression algorithms, including <code>gzip<\/code>.<\/p>\n<h2>Writing our script<\/h2>\n<p>The first thing we are going to do is decide what we want to call our backup file. I have decided that I am going to call it after the Raspberry Pi I am backing up (bunker-master2). To archive several backups I am also going to add the current date and time to the end of the archive name.<\/p>\n<p>To create the backup name with the current date and time in it I will use the date function. This will return a date string formatted based on any given date string. Once I have called it I will save it to a variable name.<\/p>\n<pre>BACKUPNAME=$(date +\"bunker-master2-backup_%d-%m-%Y_%H-%M.tgz\")<\/pre>\n<p>Here I save the backup name in a variable called <code>BACKUPNAME<\/code>. We will use this in the tar command to refer to the backup name. The string after the <code>date<\/code>\u00a0command sets what the date string will look like. We use the special formatting strings such as <code>%d<\/code>\u00a0which\u00a0the date program will convert to the current date. We use the following date strings in the above command:<\/p>\n<ul>\n<li><code>%d<\/code> &#8211; Day of month<\/li>\n<li><code>%m<\/code> &#8211; Month number<\/li>\n<li><code>%Y<\/code> &#8211; Year number<\/li>\n<li><code>%H<\/code> &#8211; Hour (in 24 hour clock)<\/li>\n<li><code>%M<\/code> &#8211; Minute of hour<\/li>\n<\/ul>\n<p>Now I have the filename I can call tar to create the archive.<\/p>\n<pre>tar -czvf \"$BACKUPNAME\" home var<\/pre>\n<p>I use the declared variable name to hold the file I want to save the backup to. Here I am passing <code>tar<\/code> four flags each of which are described below:<\/p>\n<ul>\n<li>c &#8211; Create a tar file<\/li>\n<li>z &#8211; Compress the tar file with the gzip algorithm<\/li>\n<li>v &#8211; Use verbose mode to describe what is currently running (can be removed to reduce command line output)<\/li>\n<li>f &#8211; Allows us to pass the filename of the created file as the next parameter.<\/li>\n<\/ul>\n<p>The third and fourth parameters are the folders that I want to backup. These are the folders that hold the files I have just copied over from the Raspberry Pi. There can be as many of these as required to backup all your folders and files. <code>tar<\/code> will create the archive recursively so if you specify a folder all files and folders inside that will be included.<\/p>\n<p>Now I have a backup script that will copy the files from my Raspberry Pi and save them in a compressed archive. This will let me easily keep several backups. The full script is copied below for ease of use:<\/p>\n<pre>rm -rf home var\r\nscp -r bunker-master2:\/home\/* home\/\r\nscp -r bunker-master2:\/var\/www\/* \/var\/www\/\r\nBACKUPNAME=$(date +\"bunker-master2-backup_%d-%m-%Y_%H-%M.tgz\")\r\ntar -czvf \"$BACKUPNAME\" home var<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Today I will be improving the basic script used to backup my Raspberry Pi&#8217;s so that it compresses the files after they have been copied. This will allow me to store several backups instead of having a single copy stored on my computer. This assumes you have followed the first tutorial to create a Raspberry [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":866,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[122],"tags":[159,158,39,37,101,148,156],"class_list":["post-615","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-raspberry-pi-cluster","tag-backup","tag-bash","tag-linux","tag-raspberry-pi","tag-raspberry-pi-cluster","tag-windows-10","tag-windows-subsystem-for-linux"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/11\/compressing_raspberry_pi.jpg?fit=800%2C800&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-9V","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":569,"url":"https:\/\/chewett.co.uk\/blog\/569\/creating-raspberry-pi-backup-script-scp-bash-windows\/","url_meta":{"origin":615,"position":0},"title":"Creating a Raspberry Pi backup script with scp and Bash on Windows","author":"Chewett","date":"August 30, 2017","format":false,"excerpt":"This post describes a simple way to keep backups of a Raspberry Pi or any other Linux computer. Why do I need backups of my Raspberry Pi? Some of the nodes in my cluster are going to be compute only, meaning that they will just run what is asked of\u2026","rel":"","context":"In &quot;Informational&quot;","block_context":{"text":"Informational","link":"https:\/\/chewett.co.uk\/blog\/category\/informational\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/08\/backing_up_a_rasperry_pi.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/08\/backing_up_a_rasperry_pi.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/08\/backing_up_a_rasperry_pi.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/08\/backing_up_a_rasperry_pi.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":613,"url":"https:\/\/chewett.co.uk\/blog\/613\/backup-raspberry-pi-rsync\/","url_meta":{"origin":615,"position":1},"title":"Backup a Raspberry Pi with Rsync","author":"Chewett","date":"December 20, 2017","format":false,"excerpt":"I am going to upgrade our previous Raspberry Pi backup script that previously used Secure Copy to one that uses rsync. The primary reason for this is that rsync will only copy new or changed files over. This post goes through the reasons why you might want to use rsync\u00a0instead\u2026","rel":"","context":"In &quot;Raspberry Pi Cluster&quot;","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rsync_backup_raspberry_pi.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rsync_backup_raspberry_pi.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rsync_backup_raspberry_pi.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/12\/rsync_backup_raspberry_pi.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2734,"url":"https:\/\/chewett.co.uk\/blog\/2734\/rsync-entire-raspberry-pi-to-windows\/","url_meta":{"origin":615,"position":2},"title":"Rsync Entire Raspberry Pi to Windows","author":"Chewett","date":"November 21, 2020","format":false,"excerpt":"This post includes a short snippet to copy your entire Raspberry Pi filesystem to Windows using Rsync. Copying and Archiving particular directories If you want to rsync specific directories rather than the entire Raspberry Pi I suggest you read my previous blog post. It details how to backup a Raspberry\u2026","rel":"","context":"In &quot;Raspberry Pi Cluster&quot;","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_rsync_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_rsync_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_rsync_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_rsync_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2020\/11\/raspberrypi_rsync_posticon_OUTPUT.png?fit=1200%2C628&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":567,"url":"https:\/\/chewett.co.uk\/blog\/567\/upgrading-raspbian-jessie-raspbian-stretch\/","url_meta":{"origin":615,"position":3},"title":"Upgrading Raspbian Jessie to Raspbian Stretch","author":"Chewett","date":"November 4, 2017","format":false,"excerpt":"This post describes how you can upgrade Raspbian Jessie, version 8, to Raspbian Stretch, version 9. Raspbian Stretch was recently released and while they recommend installing from fresh you are able to upgrade the versions. Here I follow the steps of upgrading the Raspberry Pi's in my cluster including some\u2026","rel":"","context":"In &quot;Raspberry Pi Cluster&quot;","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/upgrading_to_raspbian_stretch.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/upgrading_to_raspbian_stretch.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/upgrading_to_raspbian_stretch.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2017\/10\/upgrading_to_raspbian_stretch.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2002,"url":"https:\/\/chewett.co.uk\/blog\/2002\/raspberry-pi-cluster-node-12-automatic-start-with-rc-local\/","url_meta":{"origin":615,"position":4},"title":"Raspberry Pi Cluster Node \u2013 12 Automatic Start with rc.local","author":"Chewett","date":"March 6, 2019","format":false,"excerpt":"This post builds on\u00a0my previous posts in the Raspberry Pi Cluster series by creating a script which automatically starts the slave and master with rc.local. What is the rc.local file? The rc.local file is used on many Linux operating systems to schedule operations to run after services are started on\u2026","rel":"","context":"In &quot;Raspberry Pi Cluster&quot;","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/02\/rpi_cluster_12_automatic_start.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/02\/rpi_cluster_12_automatic_start.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/02\/rpi_cluster_12_automatic_start.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/02\/rpi_cluster_12_automatic_start.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1001,"url":"https:\/\/chewett.co.uk\/blog\/1001\/raspberry-pi-cluster-node-04-configuration-files-configparser\/","url_meta":{"origin":615,"position":5},"title":"Raspberry Pi Cluster Node \u2013 04 Configuration Files with ConfigParser","author":"Chewett","date":"February 10, 2018","format":false,"excerpt":"This post builds on the\u00a0third step to create a Raspberry Pi Cluster Node\u00a0to store our configuration settings in a config file. Here we move all the configuration settings in our script into a useful .cfg file using the python ConfigParser. Why use Configuration Files? When developing a system there will\u2026","rel":"","context":"In &quot;Raspberry Pi Cluster&quot;","block_context":{"text":"Raspberry Pi Cluster","link":"https:\/\/chewett.co.uk\/blog\/category\/raspberry-pi-cluster\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/rpi_cluster_04_config_parser.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/615","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=615"}],"version-history":[{"count":4,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/615\/revisions"}],"predecessor-version":[{"id":868,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/615\/revisions\/868"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/866"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=615"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=615"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=615"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}