Containing web services with iocell

March 5, 2017

I'm a huge fan of the FreeBSD jails feature. It is a great system for splitting services into logical units with all the performance of the bare metal system. In fact, this very site runs in its own jail! If this is starting to sound like LXC or Docker, it might surprise you to learn that OS-level virtualization has existed for quite some time. Kudos to the Linux folks for finally getting around to it. 😛

If you're interested in the history behind Jails, there is an excellent talk from Papers We Love on the subject:

It's long, but this is some seriously cool stuff.

Getting started

There are plenty of options when it comes to setting up the jail system. Ezjail and Iocage are popular, or you could do things manually. Iocage was recently rewritten in python, but was originally a set of shell scripts. That version has since been forked under the name Iocell, and I think it's pretty neat, so this tutorial will be using Iocell.

To start, you'll need the following:

A FreeBSD install (we'll be using 11.0)
The iocell package (available as a package, also in the ports tree)
A ZFS pool for hosting the jails

Once you have installed iocell and configured your ZFS pool, you'll need to run a few commands before creating your first jail. First, tell iocell which ZFS pool to use by issuing iocell activate $POOLNAME. Iocell will create a few datasets. Below is the list from my test environment, with the poolname 'tank':

NAME                    USED  AVAIL  REFER  MOUNTPOINT
tank/iocell             492K  7.02T   108K  /iocell
tank/iocell/download     96K  7.02T    96K  /iocell/download
tank/iocell/jails        96K  7.02T    96K  /iocell/jails
tank/iocell/releases     96K  7.02T    96K  /iocell/releases
tank/iocell/templates    96K  7.02T    96K  /iocell/templates</code></div>

ZFS datasets created by iocell

As you can imagine, your jails are contained within the /iocell/jails dataset. The /iocell/releases dataset is used for storing the next command we need to run, iocell fetch. Iocell will ask you which release you'd like to pull down. Since we're running 11.0 on the host, pick 11.0-RELEASE. Iocell will download the necessary txz files and unpack them in /iocell/releases.

Building the jail

Here's where things get interesting. Instruct iocell to create a jail by using the aptly-named create subcommand: iocell create tag=my-web-server. Iocell will return with the UUID of the jail it created, which is great, but not very user friendly. By specifying the tag option we have a name for this jail, as well as a symlink (in /iocell/tags) we can access it with.

Boot the jail by running iocell start my-web-server. Notice how we used the tag name. You could have instead specified the UUID, but let's assume you're not a glutton for pain. With the jail started, you may be wondering how to check the state of your jails. To do so, run iocell list:

~ » sudo iocell list
JID  UUID                                  BOOT  STATE  TAG            TYPE      IP4  RELEASE
1    b9a6ce54-0215-11e7-95e2-0cc47acd8c64  off   up     my-web-server  basejail  -    11.0-RELEASE

Listing the states of all iocell jails

The JID column is the FreeBSD identifier allocated to this jail. UUID and TAG represent the aforementioned iocell attributes. BOOT specifies whether or not this jail should be started upon system boot. You can set the boot state by issuing iocell set boot=on my-web-server. Keep in mind iocell must be enabled in /etc/rc.conf for this flag to work. Run sysrc iocell_enable="YES" to do so.

The STATE column gives us the current status of the jail. Since we issued the start subcommand, it is presently up. You can stop the jail with the stop subcommand: iocell stop my-web-server.

TYPE is the type of jail we created with iocell. There are several kinds of jails within iocell, but to keep things simple we're using a basejail. IP4 is the IPv4 address currently assigned to this jail. We have not set this yet, so there's a dash in this field. We'll touch on networking soon. Lastly, RELEASE refers to the FreeBSD release we used to build this jail. Since we picked 11.0 during the fetch command earlier, that was the release used.

Networking

Depending on what your containing, you may not need a network connection. For a web server though, that is definitely not the case! The traditional way of assigning networking for a jail is to allocate an IP address as an alias of the host's network adapter. To do so, let's check what interfaces are available. Run ifconfig to see the interfaces list:

~ » ifconfig
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:cd:8a:0e
    inet 10.0.0.18 netmask 0xffffff00 broadcast 10.0.0.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:cd:8a:0f
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

ifconfig command output

As you can see, we have three network interfaces in this example: two intel gigabit NICs and a loopback device. To setup aliasing for our jail to the NIC with an internet connection (igb0), we need to set the ip4_addr property. First, offline the jail with iocell stop my-web-server, then run iocell set ip4_addr="igb0|10.0.0.19" my-web-server. Adjust the interface and IP as necessary for your own configuration. Then bring the jail back online with iocell start my-web-server. If everthing went well, your NIC should have another address:

~ » ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:cd:8a:0e
    inet 10.0.0.18 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.19 netmask 0xffffffff broadcast 10.0.0.19
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

ifconfig output for our external NIC after giving the jail an ip

If we enter the jail (with the command iocell console my-web-server) and run the same ifconfig command, however, we see only the address assigned to us via iocell:

root@b9a6ce54-0215-11e7-95e2-0cc47acd8c64:~ # ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:cd:8a:0e
    inet 10.0.0.19 netmask 0xffffffff broadcast 10.0.0.19
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

ifconfig output for our external NIC from within the jail

Right about now you may be tempted to test our network connection with a popular tool like ping. You might be surprised then by the `ping: ssend socket: Operation not permitted` you'll be greeted with. Rest assured, our network connection is working correctly, but we have not enabled support for raw sockets. Be careful with this one - as the manpage warns:

Since raw sockets can be used to configure and interact with various network subsystems, extra caution should be used where privileged access to jails is given out to untrusted parties.

With that said, if you need to enable this, run iocell set allow_raw_sockets=1 my-web-server and you can ping all you want.

Installing software

A jail is pretty useless without any software. Fortunately the pkg utility is included to make downloading and installing software a snap. Iocell provides a wrapper for pkg in the form of the pkg subcommand. Run iocell pkg my-web-server upgrade to initialize and update the package database inside our jail. Then, to actually install software, pass the pkg subcommand install and the package name. We'd like to use our jail as a webserver, so we'll need to install nginx and php70. Run iocell pkg my-web-server install nginx php70.

Pkg will resolve the requested packages and their dependencies, and ask you to confirm:

Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
nginx: 1.10.2_3,2
php70: 7.0.16
pcre: 8.39_1
libxml2: 2.9.4

Number of packages to be installed: 4

The process will require 28 MiB more space.
4 MiB to be downloaded.

Proceed with this action? [y/N]:

Installing nginx and php

Once installed, we need to configure nginx and php-fpm. Enter the jail and edit the nginx configuration file at /usr/local/etc/nginx/nginx.conf to look like ours:

load_module /usr/local/libexec/nginx/ngx_mail_module.so;
load_module /usr/local/libexec/nginx/ngx_stream_module.so;
worker_processes  1;
events {
worker_connections  1024;
}
http {
include       mime.types;
default_type  application/octet-stream;
sendfile        on;
keepalive_timeout  65;

    server {
        listen       80;
        server_name  my-web-server;

        root /usr/local/www;
        index index.php;

        location / {
            try_files $uri $uri/ /index.php?$query_string;
        }

        location ~ \.php$ {
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            fastcgi_pass   unix:/var/run/php-fpm.sock;
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            include        fastcgi_params;
        }
    }
}

Slightly modified nginx config to listen for php-fpm

We're almost out of the woods here. We just need to modify php-fpm to work with our new nginx configuration. Edit /usr/local/etc/php-fpm.d/www.conf to match our minimal version:

; pool www
[www]
; Unix user/group of processes
user = www
group = www

; The local socket php-fpm listens on
listen = /var/run/php-fpm.sock
; permissions
listen.owner = www
listen.group = www
listen.mode = 0660

; process-manager mode config
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

Slightly modified php-fpm config to match nginx

Almost there! Create the file /usr/local/www/index.php and add these lines:

<?php
phpinfo();

Our mock index.php

With these changes in place, we have to enable php-fpm and nginx in /etc/rc.conf. Run sysrc nginx_enable="YES" and sysrc php_fpm_enable="YES". Finally, run service php-fpm start and then service nginx start. Open your web browser to the jail's IP. You should see the stock PHP information page.

It's working!

That's all there is to it. You now have a FreeBSD webserver contained within a jail. We've only scratched the surface of what is possible, however. resource limits, nullfs mounts, jail backups/replication, nesting and more can be done. Be sure to check out the iocell documentation for ideas.