Backups & Snapshots


What if you loose all your photos, yes every single one of your photos in your phone, laptop, usb drive. Even worse imagine loosing every single bit of your digital data. What if you had to start from a fresh laptop, mobile phone. That's a pretty close explanation of what I had to go through in 2010. 

Overview

 A few years ago, around 2010 to be exact I lost all my data due to a hard drive failure. I was able to recover some of it but to this day regret the incident. I was keeping all my data. on a single usb hard drive. I should have known better. Hindsight is 2020. Once bitten, twice shy, I have rotated between various complex, high maintenance solutions to the current compromise. In this article I will explain how I do this at present day. 

Backups vs Snapshots

First lets talk about snapshots. Having backups is good but what if you need to recover a photo you deleted 2 weeks ago. What if you wanted to recover a version of a file you changed 2 days ago. This is where snapshots comes into play. The key is to find the right balance between snapshot cadence vs retention.

I use rsnapshot to achieve this. It's available for most linux distributions. The one I use is rsnapshot for Arch. The idea is to have a config for different snapshots. I have one for daily, weekly and monthly snapshots. Then choosing the right execution mechanism.  

  • Daily snapshots kept for 7 days
  • Weekly snapshots kept for 5 weeks
  • Monthly shapshots kept for 12 months

Space / Disk Capacity

Having many snapshots doesn't mean you are gonna have many duplicate copies of data. For example, having 7 daily snapshots doesn't mean you will have 7 copies of the same data. rsnapshot is based on rsync. It it only stores one full copy and then the deltas needed to generate the rest. If nothing changed the same source will be symlinked into all snapshots. This means the space usage is going to depend on the variation. 

Lets assume you only changed one file in your backup target within the last 7 days. This means the 7 days snapshots will have only 1 duplicate file. Thus this is relatively cheap on storage. 

Snapshot Executions

I use a systemd based timer and a service to fire the snapshots. You can achieve the same using cron if thats your preference. I will post some examples towards the bottom. 

Backups 

My data lives on a nvme ssd attached to one of my servers. The snapshots live in a NAS drive. This does give some redundancy. To prepare for hardware failures I have another layer of protections. I keep a rsync based backup of the snapshots on 2 external usb ssd drives. One of those USB drives are stored outside of my home. Syncs on the hard drive kept away from my residence are only done a few times per annum. It still provides a reasonable redundant solution. It's all about the compromises ...

Lets get into some tech details now.

Technical Details

I'm not gonna talk about setting up rsync, cron. Nor wanna go into details of systemd services and timers. That's a subject of its own. I will provide some sample scripts though. 

For rsnapshot I needed a configuration. The installation generally comes with a default well document config just as any good linux package. 

Example Section

















The settings are self explanatory. Some notable ones are explained hereafter. 

Backup levels section was changed from the default "alpha,beta...." to read "daily, weekly, monthly" to increase readability. The number next to it denotes how many backups of each category are stored. 















This means I have a in time snapshot of my data for the last 7 days. In time backup of my data for last 5 weeks etc.

The systemd service file thats used to run the snapshots are given below. The filename of the service is 'ranapshot@.service'. This becomes relevant when writing the systemd timers.



Then each snapshot has it's own timer set to run at the correct cadence














That's a wrap forks. Hope you enjoyed. 





Comments

Post a Comment

Popular posts from this blog

Setting up KDiff3 to work with TortoiseGIT

Image
Setting up KDiff3 to work with TortoiseGIT When I work with git I do most of my work on the command line except for a few tasks. Resolving conflicts is one of them. So when it comes to resolving conflicts I prefer the KDiff3 instead of the TortoiseGIT diff. I was googling around to find the proper way and after grabbing things from difference places here is what I did. Obvious steps  :) Install git Install KDiff3 Install TortoiseGIT Then after that Open tortoise GIT settings Go to "Diff Viewer" Go to section "Configure the program used for comparing different revisio...." Choose "External" Enter the following "[ kdiff3 instalation path ]\kdiff3.exe %base %mine --L1 %bname --L2 %yname" Go to "Merge Tool" Choose "External" Enter the following "[ kdiff3 instalation path ]\kdiff3.exe %base %mine %theirs -o %merged" Click apply I will soon write something to show how to actually resolve...
Read more

Nextcloud and PHP8

Image
  PHP8 broke my Nextcloud instance !!! I run a Nextcloud instance on a self hosted Arch  linux server. Arch linux because I like the challenge 😁. Nextcloud due to privacy reasons. Recently Arch upgraded all php packages to php8 . Nextcloud 20.x, the version I'm using currently can't work on php8 thus my instance went offline. In this article I'm gonna describe how I managed to fix this issue. I run timeshift in my server via the cli. So I could have downgraded the php version. That would mean I would have to refrain from updating php and risk zero day vulnerabilities. That would nullify the whole purpose of running a self hosted Nextcloud instance for my sensitive data. Instead I went with dual php versions. I run php-fpm with Apache in my server so now I have to figure out how to run Fast CGI process manager with 2 php versions in the same Apache server. If you are thinking "well you are a SDE and you should know it", let me start by saying I neither use PHP...
Read more

Nextcloud on Arch Linux (Encrypted System) [Part 01 - Preparation]

Image
 In this series of posts I'm going to outline the process I used to create my own Nextcloud server.  Network Setup First step is to prepare my network. I'm planning to host this at home. Due to obvious security concerns I don't want to host this on my home network. I used pfSense  to achieve what I want. The pfSense setup is outside the scope of this post. If you do need help or is curious about setting up pfSense let me know. Given below is an overview diagram of my network,  The basic firewall rules in this network are Server can't talk to the switch.  Server can't talk to the pfSense router interface Server interface has Suricata IDS/IPS running.  Drive Preparation and Encryption Secure erase the drive.  Note: The drive I used was an old HDD. What I didn't notice was that the partition table was of type DOS. Had to change that to GPT and repeat this process in order for it to work. I'm going to use a 250GB HDD for my instalation (not a SSD). First...
Read more