Verb Your Own Noun
This blog has been running on a Mirage OS unikernel hosted on Amazon EC2 since April 3rd:
$ ec2-get-console-output --region the-best-region i-0123abcd
2014-04-03T16:42:58+0000
Xen Minimal OS!
In that time, I’ve done some stuff:
- submitted a successful pull request to Mirage
- made a permanent home for Secret Project Glow Cloud’s code
- successfully applied to the Outreach Program for Women to work on Mirage some more
- broke and fixed my OCaml development environment repeatedly
- sang in public in front of other Hacker Schoolers
- looked at a lot of neat objects and buildings
- started work on a fuzzing framework to scratch my own itch for testing network clients
I figured it was time to tell you about some of it, but first I did some other stuff:
- upgraded some packages on my build machine
- broke the build on my blog
- learned about how Mirage makefiles are generated by trying to get mine working again
You’d rather hear about all of that, right?
The first step of writing a post here is to make an empty shell for the content in Octopress:
$ cd octopress
$ rake new_post["Verb Your Own Noun"]
mkdir -p source/_posts
Creating new post: source/_posts/2014-04-23-verb-your-own-noun
which makes me something to write in, so I write some stuff (that’s the boring part, ho hum). Once I have something I’d like to look at, I regenerate the entire blog:
$ rake generate
## Generating Site with Jekyll
identical source/stylesheets/screen.css
Configuration from /home/dundermifflin/octopress/_config.yml
Building site: source -> public
Successfully generated site: source -> public
## Building Elm code from _elm/elm-finds-kitten
mkdir -p _elm/elm-finds-kitten/build
elm --make -r "elm-runtime.js" -b "build/" rfk.elm
Generating HTML ... Done
cp ~/.cabal/share/Elm-0.11/elm-runtime.js "build/"
mkdir -p public/rfk
Now I have a new /public
directory, which I can use for the content of my unikernel. So let’s make a unikernel:
$ rake mirage
## Beginning unikernel build ...
MIRAGE Using the scanned config file: config.ml
MIRAGE Compiling and dynlinking /home/dundermifflin/octopress/_mirage/config.ml
MIRAGE + Executing: rm -rf /home/dundermifflin/octopress/_mirage/_build/config.*
MIRAGE + Executing: cd /home/dundermifflin/octopress/_mirage && ocamlbuild -use-ocamlfind -tags annot,bin_annot -pkg mirage config.cmxs
www CONFIGURE: /home/dundermifflin/octopress/_mirage/config.ml
www 1 job [Dispatch.Main]
www Installing OPAM packages.
www + Executing: opam install --yes crunch cstruct lwt mirage-clock-xen mirage-console-xen mirage-http mirage-net-xen mirage-types mirage-xen tcpip
www + Executing: ocamlfind printconf path
www Generating main.ml
www Generating /home/dundermifflin/octopress/_mirage/static1.ml.
www + Executing: ocaml-crunch -o static1.ml /home/dundermifflin/octopress/_mirage/../public
MIRAGE Using the scanned config file: config.ml
MIRAGE Compiling and dynlinking /home/dundermifflin/octopress/_mirage/config.ml
MIRAGE + Executing: rm -rf /home/dundermifflin/octopress/_mirage/_build/config.*
MIRAGE + Executing: cd /home/dundermifflin/octopress/_mirage && ocamlbuild -use-ocamlfind -tags annot,bin_annot -pkg mirage config.cmxs
www BUILD: /home/dundermifflin/octopress/_mirage/config.ml
www + Executing: make build
www make: *** [main.native.o] Error 10
www amlfind -pkgs lwt.syntax,cstruct,io-page,lwt,mirage-clock-xen,mirage-console-xen,mirage-http,mirage-net-xen,mirage-types,mirage-types.lwt,tcpip.stack-direct -tags "syntax(camlp4o),annot,bin_annot,strict_sequence,principal" -cflag -g -lflags -g,-linkpkg,-dontlink,unix main.native.o
www ocamlfind ocamldep -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -modules main.ml > main.ml.depends
www ocamlfind ocamldep -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -modules static1.mli > static1.mli.depends
www ocamlfind ocamlc -c -g -annot -bin-annot -principal -strict-sequence -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -o static1.cmi static1.mli
www ocamlfind ocamlc -c -g -annot -bin-annot -principal -strict-sequence -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -o main.cmo main.ml
www ocamlfind ocamldep -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -modules static1.ml > static1.ml.depends
www ocamlfind ocamlopt -c -g -annot -bin-annot -principal -strict-sequence -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -o static1.cmx static1.ml
www ocamlfind ocamlopt -c -g -annot -bin-annot -principal -strict-sequence -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o -o main.cmx main.ml
www ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o dispatch.cmx static1.cmx main.cmx -o main.native.o
www + ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj -package tcpip.stack-direct -package mirage-types.lwt -package mirage-types -package mirage-net-xen -package mirage-http -package mirage-console-xen -package mirage-clock-xen -package lwt -package io-page -package cstruct -package lwt.syntax -syntax camlp4o dispatch.cmx static1.cmx main.cmx -o main.native.o
www File "_none_", line 1:
www Error: No implementations provided for the following modules:
www Re_str referenced from dispatch.cmx
www Command exited with code 2.
[ERROR] The command "make build" exited with code 2.
…that’s… not supposed to happen.
Let’s check this out:
www Error: No implementations provided for the following modules:
www Re_str referenced from dispatch.cmx
dispatch.cmx
is generated from dispatch.ml
, one of the two files containing actual, human-written code involved in the construction of my Mirage unikernel. Re_str
, I happen to know, is provided by the re
package, which I have installed:
$ opam info re
package: re
version: 1.2.1
upstream-url: https://github.com/ocaml/ocaml-re/archive/ocaml-re-1.2.1.tar.gz
upstream-kind: http
upstream-checksum: 13b3f2aa3710d03c82e3338feefec669
depends: ocamlfind
installed-version: re.1.2.1 [system-mirage-tcpip system 4.01.0]
available-versions: 1.0, 1.1.0, 1.2.0
description: RE is a regular expression library for OCaml
But it’s not being found by the compiler. Let’s have a look at the failed compilation step:
www + ocamlfind ocamlopt -g -linkpkg -dontlink unix -output-obj
-package tcpip.stack-direct -package mirage-types.lwt -package mirage-types
-package mirage-net-xen -package mirage-http -package mirage-console-xen
-package mirage-clock-xen -package lwt -package io-page -package cstruct
-package lwt.syntax -syntax camlp4o dispatch.cmx static1.cmx main.cmx -o main.native.o
There are an awful lot of packages getting pulled in here, but none of them are named re_str
or re
or (the actual name, I find out after some blind experimentation) re.str
. Manually descending into _mirage/_build
and retrying the failed step with an added -package re.str
seems to work, hooray! But that’s no kind of solution; I need my Makefile to work properly.
The list of things that get included as -package
directives in the unikernel build seem to be specified as the variable PKGS
in the Makefile. Adding re.str
to the end of this comma-separated list and rerunning make
seems to work! But there’s a fly in this ointment:
$ head -1 Makefile
# Generated by Mirage (Wed, 23 Apr 2014 19:10:50 GMT).
We need to go deeper.
Mirage automatically generates this Makefile every time I call mirage --configure unix
or mirage --configure xen
. In order to make this change persist, I have to somehow tell Mirage that re.str
is a required package. I dig into the Mirage repository and quickly find some Makefile-generating code, referenced by a general configure
function. A little more digging reveals several module
s which have defined packages
functions, like this one for Direct_kv_ro
(a direct key-value-store filesystem that is read-only):
let packages t =
match !mode with
| `Xen -> Crunch.packages t
| `Unix -> [
"mirage-types";
"lwt";
"cstruct";
"mirage-fs-unix";
]
Unfortunately, none of these reference re.str
, and while mapping the path between the config.ml
definitions of required module
s and the end result in PKGS
is a good exercise, it doesn’t give me the name of something I can reference to get re.str
in my Makefile.
I take a different tack and look around in mirage-skeleton, the cloneable repository of example code referenced in the “Hello Mirage World” to see if any code there is trying to add a package to the default list. Sure enough:
$ find . -name config.ml|xargs grep package
./stackv4/config.ml: add_to_opam_packages ["mirage-http"];
./xen/static_website+ip/config.ml: add_to_opam_packages ["mirage-http"];
./dns/config.ml: add_to_opam_packages ["dns"];
./mirage/lib_test/ip/config.ml: add_to_opam_packages ["tcpip"];
./mirage/lib_test/basic_http/config.ml: add_to_opam_packages ["mirage-http"];
It looks like add_to_opam_packages
sticks the argument into the list of packages included in opam install --yes huge-list-of-packages
. Looking at the surrounding context, it looks like I also need add_to_ocamlfind_libraries["my_totally_sweet_package"]
in order to get the package added to the list of -package
arguments to ocamlbuild
. The new main
method of my config.ml
looks like this:
let () =
add_to_opam_packages["re"];
add_to_ocamlfind_libraries["re.str"];
register "www" [
main $ default_console $ fs $ server
]
and it works!
$ rake mirage
## Beginning unikernel build ...
MIRAGE Using the scanned config file: config.ml
MIRAGE Compiling and dynlinking /home/dundermifflin/octopress/_mirage/config.ml
MIRAGE + Executing: rm -rf /home/dundermifflin/octopress/_mirage/_build/config.*
MIRAGE + Executing: cd /home/dundermifflin/octopress/_mirage && ocamlbuild -use-ocamlfind -tags annot,bin_annot -pkg mirage config.cmxs
www CONFIGURE: /home/dundermifflin/octopress/_mirage/config.ml
www 1 job [Dispatch.Main]
www Installing OPAM packages.
www + Executing: opam install --yes crunch cstruct lwt mirage-clock-xen mirage-console-xen mirage-http mirage-net-xen mirage-types mirage-xen re tcpip
www + Executing: ocamlfind printconf path
www Generating main.ml
www Generating /home/dundermifflin/octopress/_mirage/static1.ml.
www + Executing: ocaml-crunch -o static1.ml /home/dundermifflin/octopress/_mirage/../public
MIRAGE Using the scanned config file: config.ml
MIRAGE Compiling and dynlinking /home/dundermifflin/octopress/_mirage/config.ml
MIRAGE + Executing: rm -rf /home/dundermifflin/octopress/_mirage/_build/config.*
MIRAGE + Executing: cd /home/dundermifflin/octopress/_mirage && ocamlbuild -use-ocamlfind -tags annot,bin_annot -pkg mirage config.cmxs
www BUILD: /home/dundermifflin/octopress/_mirage/config.ml
www + Executing: make build
$ ls _mirage/*.xen
_mirage/mir-www.xen
Now we have a unikernel! I can run it locally with Xen, fire it off to the cloud, put it on another computer, invoke several thousand copies of it on my local network to implement an extremely silly DHCP exhaustion attack, put peanut butter on it, whatever.
This is one of the many things you can learn by eating your own dog food.