How to backup Flickr into Amazon S3/Dropbox/Google Cloud/Drive...
The following recipe is an easy way to get your full
Flickr account backed up in Amazon S3 or similar cloud storage.
Unless you have a really good Internet connection at home or you are doing a backup of a small library the first thing to do would be to create a machine in the cloud, such as Digital Ocean or Amazon EC2 so you can have a high throughput. Once the process is finished you can destroy the machine and the whole operation will cost you a few cents. When creating the droplet/instance make sure to create a machine that has enough disk space to hold your library.
In my case I had some free credit in Digital Ocean so I did it there, but of course you can create a machine in EC2 and persist directly in S3, or you can also do this in Google Cloud.
The process consists in two simple steps:
- Download all the pictures from Flickr using Flickrtouchr
- Sync all the downloaded pictures to S3 or other cloud storage using Rclone
Install the software
apt-get update
apt-get install --yes git unzip screen
wget http://downloads.rclone.org/rclone-v1.28-linux-amd64.zip unzip rclone-v1.28-linux-amd64.zip mv ~/rclone-v1.28-linux-amd64/rclone /usr/local/bin/rclone
The substitute the following lines with your own Amazon Access Key ID and secret
echo " [S3] type = s3 env_auth = false access_key_id = ABCDEFGH1234567890IJK secret_access_key = aBcDefghHijKl123456mnopqrst region = us-east-1 endpoint = location_constraint = " > ~/.rclone.conf
Install the script that downloads pictures from Flickr:
git clone https://github.com/dan/hivelogic-flickrtouchr.git cd hivelogic-flickrtouchr mkdir FlickrBackupFolder/
Download all your Flickr pictures
It is recommended to use screen so you can recover the session anytime
screen -d -m -S "Flickr download" python flickrtouchr.py FlickrBackupFolder # screen -ls # screen -r PID # CTRL+a :detach
Upload pictures to S3
rclone copy ~/hivelogic-flickrtouchr/FlickrBackupFolder/ S3:your-bucket-name
Done!