You appear to be using an outdated browser. For the best experience on this site, please install Chrome Frame, update to Internet Explorer 8 or later, or use another browser like Chrome or Firefox.

ZFS on Linux on Amazon EC2

Our clinical trials section (CTS) needs to have off site backup.   Seems the traditional way FDA Part 11 compliant studies do this is do encrypted backups to tape and ship them off with Iron Mountain.  I cringed at that idea.  It’s cumbersome and much less useful than a sending stream of ZFS snapshots off to another site.

After burning way to many hours trying to get OpenIndiana running on Amazon EC2, I started looking for alternatives.   It turns out the ZFS on Linux project is almost in lock step with the Illumos kernel in OpenIndiana.

I started with a t1.micro instance of Amazon Linux.   This is for backup purposes so I’m not worried about performance here.  Should there arise a need to do a large restoration from this server or some actual work from it I could easily shutdown and started back up as a larger instance.

Even though Amazon claims redundancy of their EBS storage, they don’t provide any specifics but only an annual failure rate of between 0.1% and 0.5%.   I decided to take advantage of ZFS to increase that reliability by building RAIDZ1 VDEVs with 8 EBS volumes each.  I then created my ZPOOL out of 5 of these VDEVs.   Since I don’t want to pay for a bunch of unused storage my EBS volumes all started at 1GB each.   This stripes across 40 EBS volumes at 1GB each.  I’ll cover later how I grow this storage and keep my ZPOOL balanced.

In the process of setting this up I’ve created a bitbucket repository of the scripts I’ve used to set up and grow this zfs pool in the cloud:  https://bitbucket.org/ctsadmin/aws_zfs_tools

The pool creation and maintenance is all handled externally to the ec2 instance.   The scripts reside on the primary ZFS server.

To perform a backup the primary ZFS server kicks off scripts that starts the instance, supplies the encryption key to the scripts to mount the ZFS pool, pushes a zfs send, and shuts down the instance.   Since the encryption key is only in memory on the ec2 instance once shut down the data on the EBS storage is encrypted and safe.

Since this is for an FDA Part 11 project, I am taking the security of this data even further.  The primary ZFS server will have two pools of data.  The production pool and a file level encrypted copy.   It will maintain the encrypted copy with additional scripts.  It is the encrypted copy that will be backed up to the cloud.  Two encryption keys will be needed for retrieval of data from the backup in the cloud.

After some kernel panics I’ve switch to using and m1.small instance instead of the t1.micro.  I’ve also added some swap space for the few times when growing the pool I’ve seen memory use cause kernel panics on the small instance.   Since the instance will be on for less than an hour a day to receive backups, the 8 cent instance hour charge per day is reasonable.

I may later play with deploying it as a spot instance or further tuning ZFS for the small memory environment of a micro instance.   But it comes down to how much work do I want to put into pinching those last few pennies.

 

 

This entry was posted in ZFS Storage and tagged , , , , , . Bookmark the permalink.

19 Responses to ZFS on Linux on Amazon EC2

  1. Amazon has now introduced Glacier and it is prices at 1/10 the storage cost of EBS.

    If your cloud backup model is like ours in that it is there for DR purposes and really won’t be needed for any file level recovery, Glacier is a much more appealing processes.

    I haven’t test Glacier yet, but from what I can see it is driven by the Amazon API in Java or .NET. They provide simple Java and .NET code samples to push data up and retrieve data from Glacier.

    I would attempt to implement this for ZFS by choosing an upload repeat cycle based on the rate of data change I’m backing up. For example push a full ZFS send once a month, then daily increments. Since there is no ZFS receive on Glacier you would have one large archive and smaller daily archives. After a month cycle a full send would happen again, and when confirmed successful from Amazon the previous cycle gets deleted. If a restore is needed the whole batch of archives would need to be retrieved and the zfs pool reassembled from the archives. I would probably add a check before delete that the new data set is larger than the previous month to prevent something really bad from happening in an automated way.

    As they market this it is not good for regular retrievals of data, it will take 3 to 5 hours to get data ready to download.

    The architecture I designed for our ZFS folders and snapshot schedules so that all of our backups are in snapshots. Anything that needs to be kept for long periods of time are in folders that snapshots stick around for years or longer. This is kept separate from data with higher change rates with shorter snapshot deletion cycles.

    If lots of your data is changing frequently, Glacier is most likely not the backup solution for you.

  2. You should take a look at https://forums.aws.amazon.com/thread.jspa?messageID=131127#131127 and the attached PDF you may well not be achieving your hoped for redundancy/durability.

  3. Very interesting read.

    Actually, I’m not implementing raidz1 for redundancy. It’s so I can dynamically grow the pool vertically instead of horizontally and always have a balanced pool of EBS devices.

    This is a backup system, if I get a report of data loss via scrub, I will simply re-push all the data to the backup.

    If this were for the primary site, I would completely reconsider how I implement redundancy.

  4. Hi Chip,

    You mention that ZFS was not really working on EC2 as intended even on higher end machines. did this happen when you were pushing a lot of data in to the ZFS EC2 system ? I am trying to ask if it would work for low rate of data transfer.

    Thanks
    Sankar

    • It worked flawlessly when data rates were low. The problem always happened when I would push data a high rates. It would happen pretty consistently around the time I pushed about the same amount of data as RAM allocated to the EC2 instance.

      I suspect it is related to this bug: https://github/zfsonlinux/spl/issues/174 which has since been fixed. I have changed my backup strategy to utilize Glacier, so I haven’t yet tested it the fix.

      When I start working on DR plans that include EC2, I will be revisiting ZoL on EC2.

    • Yes. It always happened when pushing lots of data. It always seemed to happen about the time I pushed the equivalent as the ram allocation.

      At the time there was no Illumos/OpenSolaris based distribution on EC2. I tried to get OI going, but failed. Now there are prebuilt OmniOS instances. I would definitely use it on EC2 if doing it now. The ZoL bug that was causing my problem may be fixed now too.

  5. FYI, zfs on linux needs to run in 64bit kernel to be effective. otherwise it will look like its working until you start to push data to it.. then it just fails. also ZFS needs LOTS of RAM.

  6. Hi Chip,

    Noted. I had tried to install zfs on a EC2 M1.Medium ,64 bit,Ubuntu 12.04 based on the ubuntu/stable PPA. However, after the installation
    # modprobe zfs says FATAL:module not loaded
    and
    # dmesg | grep ZFS: doesnot indicate anything

    I had tried with ver 11.10 also.

    I presume, you had used Ubuntu 12.04, Is there a specific procedure to follow for the installation. Any help would be appreciated.

    Thanks in advance
    Sankar

  7. I had installed it on Ubuntu 12.04 and on Amazon Linux.

    On Ubuntu is was straight forward. I used the method from ZoL FAQ: http://zfsonlinux.org/faq.html#HowDoIInstallIt

    $ sudo add-apt-repository ppa:zfs-native/stable
    $ sudo apt-get update
    $ sudo apt-get install ubuntu-zfs

  8. Good choice in technology for EC2 – ZFS on Linux.

    We also use ZFS on Linux in SoftNAS(tm), a full NAS virtual storage appliance for EC2, VMware and Hyper-V that we just launched yesterday.

    Rick

  9. The link was wrong when I wrote the article. It should have been https://bitbucket.org/ctsadmin/aws_zfs_tools. Article has been corrected.

    Sorry about that.

  10. Actually the link in the article is still broken (says ec2 not aws). The one in the comment before this one is the one that works.

    A question though, now that ec2 offers provisioned IOPS, should this be able to replace all the hard work of aggregating 40 disks in raid-z groups (assuming one doesn’t need redundancy)

    • Link should be fixed now. I must have not saved the update.

      Yes, Amazon is offering provisioned IOPs, however how much do you want to pay for your IOPs.

      By spanning ZFS horizontally across any number of block devices you can get all the IOPs you ever wanted by spreading your zpool across what may be 1000′s of spindles in the AWS data center.

      I haven’t used ZFS on Linux in quite some time now. If I wanted ZFS on Amazon now I would use one of the OmniOS AMIs as starting point. My experience with ZFS has been much better on Open Solaris derivatives than on Linux. When I wrote the article getting any Solaris running on AWS was a steep hill to climb.

      The scripts have evolved more in to managing ZFS than working with EC2. The backup to glacier portion is in production and has worked without flaw for many months now. I’m working on extending it to automatically cycle the incremental ZFS sends with a fresh full ZFS send. I hope to have that complete within the next few weeks. After that’s complete, my goal is to fork this into two separate projects, one for managing ZFS the other for backups to AWS.

  11. Great write-up, I certainly think ZFS is a necessity if you run on Amazon.

    My company ran on EC2 in 2010-2011, until we migrated to our own servers in colo. We ran OpenSolaris 2009.06 en m1.large instances, with ZFS on instance storage and PostgreSQL replication across availability zones for disaster recovery. Our decision to shun EBS saved us from the many well-publicized incidents on that platform, and ZFS saved us a number of times from silent data corruption. Another major benefit is the ability to efficiently replicate data off AWS using zfs send – we migrated several terabytes from us-east 1 to California

    For more details:

    http://broken.net/uncategorized/los-hicimos-we-did-it-we-broke-the-bonds-of-amazon-aws-ec2/

  12. Very interesting, what are your plans to manage encryption if you move to OmniOS? With FreeBSD you could use Geli much like dm-crypt so I’m thinking about using that instead. Right now I’m using OpenIndiana but need to dump close to 8TB worth of data so it really boils down to part performance as well.. Have you tried pulling out any of the data from Glacier etc?

    • I haven’t built any systems with OmniOS on EC2 yet, however my first approach would be to use lofiadm and layer the zpool on top of encrypted block devices.

      This could be managed the same way I managed encryption with ZoL.

      I have tested the ability to restore a few times now. I have only done it with small zfs folders so I don’t rack up a massive bill, but it has worked as planned.

  13. You should not use saved ZFS streams for backup as they are not resilient to data corruption and meant only as a temporary transfer mechanism. Your suggested method with glacier could lead to your backups being unrecoverable and no way to tell that until you need them.

    • That is why sha256 sums are taken of every stream and stored. If the stream returned from glacier is corrupt it will be detect via the check sum.

      Glacier also does their own check summing to ensure there is no data corruption. While there is no other redundancy besides what Glacier builds under the covers, this is why periodic testing of backups is essential.

      If you still don’t trust the method, use something like PAR2 to split each send into redundant parts that upload.

Leave a Reply

Your email address will not be published. Required fields are marked *


× 3 = twenty seven

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>