I used to do something like this but with ZFS on an OpenSolaris / Illumos storage server, exporting copy-on-write clones of snapshots of iSCSI volumes to boot Xen VMs on neighbouring blades, with tagged VLANs from each host because there were multiple guests to launch with varying roles in the application cluster. We made the VLAN number match 12 bits of the IP addresses and numbered the clones similarly. It merely remained to create new snapshots every release and any dev could launch an bugfix/test/showcase/etc environment for that version, and connect to it via a VPN. I was always worried about scale if we hired more than 4096 developers but fortunately the company was acquired and its product discontinued before that happened.
That was in 2007 so the control plane (scheduler and automation) were built from scratch and we had very few reference points for the overall design. If I was building that today I’d probably still use ZFS clones but at filesystem level instead of block devices, and serve jails over NFS if I can get away with it, the iSCSI part was always a little janky.
I built a reverse proxy server for this kind of usecase specifically. I have a chroot launcher with a simple config file and the reverse proxy is auto configured over RPC and automatically acts as a CA root so internal communication is HTTPS. You can also set the API key as an env var and just run your binary on the command line for testing and it will auto configure and be available on your URL with https.
https://github.com/fsmv/daemon/
It's a bad time for me to be mentioning it because I have a major update that's not quite ready to release that changes some client APIs and makes the whole thing much nicer with fully automatic lets encrypt. I haven't had the space to work on it for a while unfortunately.
For those unfamiliar with FreeBSD, this is using base OS tools to manually create this type of immutable jail/container. This can be done with 'less effort' by using a jail manager.
The ezjail[0] port is another option for achieving the article's stated goal:
FreeBSD's native support for ZFS snapshots and jails
provides a powerful foundation for immutable deployments.
Detailed writeup here: https://docs.skunkwerks.at/s/fUiAmi4pE
If you want to do this kind of thing but don’t like the looks of this indepth tutorial, I highly reccomend installing Bastille FreeBSD instead and let the bastille tooling do the heavy lifting of managing zpool cloning and what not. Their getting started tutorial should get you 90% of the way there and has Docker like tooling. [0]
Also found this self host on a Raspberry Pi helpful.[1]
[0] https://bastillebsd.org/getting-started/
[1] https://www.sharpwriting.net/project/bastille-jail-managemen...
I hope OCI containers support make this boilerplate obsolete.
Isn't this just docker with extra steps?
I highly recommend this approach. I run ~all of my digital footprint on FreeBSD jails. After surveying how many jail managers have come and go in the last decade, I decided to just roll my own using a single shell script called jailctl [0].
Nothing fancy, just VNET jails based on ZFS templates (vanilla FreeBSD rootfs) and epair interfaces (which I truck to various VLANs on the host's egress interface).
One pattern that I've found useful is to give each jail a persistently delegated ZFS dataset called "data." This lets me reprovision the OS image for the jail without having to backup and restore its application data (such as a Postgres DB). It also allows each jail to manage its own ZFS snapshots.
The only thing that was a bit hairy was generating unique interface names and MAC addresses for each jail's VNET interface. My first instinct was to derive the interface name from the jail name, but interface names on FreeBSD are limited to 15 characters, and occasionally I'd hit this limit.
In the end I did some dark magic using md5 sums of the jail name / host interface MAC address. Kind of ugly but I really didn't want to introduce any dependencies besides /bin/sh.
[0] https://github.com/cullumsmith/infrastructure/blob/master/fi...