I Am Unikernel (And So Can You!)

Page content

Julia Evans, prolific blogger and rad person, gave me several kind comments on the “Why I Unikernel” posts (security, self-hosting). She also asked, quite reasonably, whether I’d written a high-level summary of how I host my blog from a unikernel. “No, but I should,” I said, and unlike most times I say I should do something, I actually did it.

Here’s the very-high-level overview:

  • use brain to generate content that some human, somewhere, might want to read (hardest step)
  • write all that stuff in Markdown
  • use Octopress to generate a static site from that Markdown
  • use Mirage to build a unikernel with the blog content
  • upload the unikernel to an EC2 instance running Linux
  • build a new EC2 instance from the uploaded unikernel
  • make sure that newly generated instance looks like my website with new content
  • shut down the Linux host that made the new EC2 instance
  • make somerandomidiot.com point to the new EC2 instance
  • kill the EC2 instance which previously served somerandomidiot.com

And below, one can find the gory details.

Use Brain To Generate Content

I can’t really give you great advice on this, but A. Jesse Jiryu Davis can.

Write All That Stuff In Markdown

I say Markdown, but really, write it in whatever thing makes sense for how you’re going to deploy it. If you’re going to use Octopress, you want Markdown. There are a lot of Markdown summaries floating around the ’net, but my favorite one is potatodown.

For the purposes of this walkthrough, let’s say I’ve written my masterpiece into octopress/source/_posts/2014-08-18-i-am-unikernel.markdown.

Use Octopress To Generate A Static Site

I happen to use Octopress, but I see no reason why you couldn’t use any static site generator. Amir Chaudry, who was making unikernel blog entries before I’d even cracked a book on OCaml, uses Jekyll and much smarter automation than I use.

If you want to start an Octopress blog, the instructions on getting started are fairly straightforward. There’s a potentially tricky substep here, which is “get a working Ruby environment that can build Octopress”. If you already use Ruby, you’re probably fine; if you don’t, you may appreciate this documentation for Bundler, the module which gave me a couple of hours of trouble.

Once Octopress is initialized, configured, and ready to go, I can regenerate my blog with new content:

user@computer:~/octopress$ rake generate
## Generating Site with Jekyll
identical source/stylesheets/screen.css 
Configuration from /home/user/octopress/_config.yml
Building site: source -> public
Successfully generated site: source -> public

Now I have a bunch of files in octopress/public that comprise the entirety of my blog, including public/blog/2014/08/18/i-am-unikernel/index.html, my new post.

Use Mirage to Build a Unikernel

To serve my website, I need something that can do a series of things:

  • read HTTP requests from interested clients (e.g., “show me the index of somerandomidiot.com”)
  • provide the content for a given request
  • construct a valid HTTP response wrapping the content
  • respond to requests that can’t be fulfilled with an error page (i.e., serve a “page not found” page)

Handily, there’s an example of such a thing in mirage-skeleton. I made a _mirage subdirectory in my octopress directory and copied the config.ml and dispatch.ml over into it. I had to edit config.ml to look for my content in a different location, and edit dispatch.ml to assume that any request with a trailing / should have an index.html appended for lookup; other than that, the code is stock. Here’s the edited dispatch.ml and the edited config.ml; if you like, use them yourself.

Once this is in place, DHCP=yes mirage configure --xen and mirage build gets me a unikernel called mir-www.xen. I generally boot it locally and look it over, find something wrong, fix it, and rebuild a few times.

Upload the Unikernel to an EC2 Host Running Linux

I run my unikernel on EC2, pretty much entirely because it works and it’s free. In order to deploy a unikernel to EC2 as cheaply as possible, one currently needs to do some auxiliary operations on something running on EC2. I got a Linux host set up with the shell EC2 tools to do this. I keep the host stopped when I’m not using it to build a unikernel, to avoid having to (1) pay for compute time and (2) worry about its credentials being brute-forced. The Linux host is tagged with role=host, so I can distinguish it from my little mayfly unikernels when manipulating my EC2 instances programmatically.

I start up the build host, then once it’s responding, I scp my unikernel, mir-www.xen, over to its filesystem. The blog is currently about 17 megabytes (mostly high-resolution embroidery photos), so this is starting to take a while; soon I’ll have to do something smarter.

Build the New Unikernel EC2 Instance

Building a bootable t1.micro instance from a unikernel is surprisingly involved. I have a janky shell script that lives on the Linux EC2 host and does this automatically. If you’re interested in the details, it’s up on gist. The summary, shamelessly copied from the first time I did this a few months ago, is this:

  • create an EBS block to attach to the Linux image
  • attach the EBS block to the Linux image
  • make a bootable filesystem on the EBS block
  • copy the Mirage kernel to the EBS block over the network
  • make a workable Grub configuration on the EBS block
  • snapshot the EBS block
  • associate the EBS block snapshot with an Amazon Kernel Image that boots PV-GRUB, which gives you an Amazon Machine Instance backed by an Elastic Block Store
  • finally, fire up your instance from the previous step.

Check the New Instance

At the end of all this, a successful build generates a new instance at some nonpredictable public IP, which is discoverable from the Amazon console or with ec2-describe-instances. I usually browse to it and make sure it’s up and responding before making this version of the blog the authoritative one, but this is somewhat unnecessarily paranoid.

Stop the Build Host

I don’t need the Linux EC2 host running anymore, so I stop it with ec2-stop-instances.

Point Traffic at the New Instance

somerandomidiot.com points to an Amazon Elastic IP, which can be reassigned to a different instance very quickly – much faster than DNS propagation. Once I’m satisfied with the new instance, I move the elastic IP over to it with ec2-associate-address, so that the IP that somerandomidiot.com resolves to now points to the newest instance.

Kill the Previous Instance

The EC2 instance that was serving the blog until the previous step is still up and running. ec2-terminate-instances (a command that not many people get to use as often as I do!) destroys it completely. If I needed to, I could regenerate it from the EBS block snapshot.

Bonus Extras!

I haven’t had to worry about this yet, but there’s a charge for storing large numbers of snapshots on EC2 - a few pennies per gigabyte-month once you pass 30 gigabytes. Each deployment makes a 1GB snapshot, so soon I’ll have enough snapshots on EC2 that I’ll have to go in and delete a few.

The Future!

My fellow OPW intern, Jyotsna Prakash, has been working on EC2 API bindings for OCaml. Soon, I hope it will be possible to have a unikernel do the work of packaging up unikernels into Amazon Machine Instances, as well as the management work of spinning them up, shutting them down, and reassigning IPs. It’s a bit embarrassing that I have a Linux host just for running a shell script that repeatedly calls other shell scripts that invoke the JVM in order to do REST API calls, and it’ll be a great day when I can replace that with a few lines of OCaml.