Thoughts Community Previous Thought Next Thought

Python S3 Sync Helper

At Paragon we back up everything.


Storage, data, backups… oh my! At Paragon we back up everything. That one thing you asked us to do back in 2002? You know the one with the sweet flash splash page? Yep, we still got it.

The difficulty in keep that much data is finding where on Earth to put it so the announcement this month of the Amazon S3 price reduction sent a happy dance through my bones. We use S3 and Glacier for backups but the process has been manual and mind numbing. Sorting, zipping, uploading, waiting… while it is necessary it does mean that time passes before something hits the cloud.

In an effort to try and get things into S3 quickly for increased redundancy I set out to try to find a bit of software to keep archived work in sync with S3. I found plenty of software to mount S3 as a drive but wasn’t impressed with keeping local files backed up to S3. So, I thought, how hard could it be to write something?

s3_upload_helper

What we now have chewing through our local archives is called “Python S3 Sync Helper.” Thank goodness I’m not in charge of branding around here. You can take a look on GitHub if you’d like to try it out. It has only been tested in Windows so, Mac peeps, give it a shot and let me know what breaks.

To get it working, you’ll need Python 2.7 and Boto installed. Next, edit the config file and add your S3 API credentials (Access, Secret) along with the bucket where you’d like to store your files, and the location of the folder on your local hard drive. Optionally add an S3 folder name if you’d like the files to upload in a specific directory of your bucket… I know S3 doesn’t really use folders but you know what I mean.

If everything is installed and configured, run the backup.py file to start uploading files. The program gets a list of all files in the local directory you provide, uploads them to S3, and gives you a rundown of what happened when it is done.

If you do give it a try, let me know how/if it works for you and what you think.

Avatar

Philip Joyner

Not only can the man stare down CSS code until it writes itself in sheer terror, but he is famous around 220 E. Hall St for what we like to call his “happy dance”. Few have seen it, and those who have can’t get enough.

Comments

Share