class: center, middle # FreeBSD Jails from Scratch ## EuroBSDcon 2024 ## Dublin, Ireland --- ----------------------------------------------------------------------------- # `whoami` ## [dch] - Dave Cottlehuber - <dch@skunkwerks.at> | <dch@FreeBSD.org> - FreeBSD enthusiast - Couch-driven Developer --- ## admin > goal is 1 tutorial for end-to-end jail usage - mix of slides, hands-on, and Q&A - morning is internals, afternoon is OCI containers - grab your VM IP and login as `root@....` - move fast, break things, ask questions - I'm here all conf for follow-ups - follow [slides](https://skunkwerks.at/~dch/FreeBSD/diy-jails/tutorial.html) [src]( https://git.sr.ht/~dch/diy-jails-tutorial/tree/main/item/tutorial.md) - get your ssh up - import this [ssh key](https://git.sr.ht/~dch/diy-jails-tutorial/blob/main/keys/jails_ed25519) https://git.sr.ht/~dch/diy-jails-tutorial/blob/main/keys/jails_ed25519 --- ## Agenda - what even is this "jail" ? As the kernel sees it - the underlying truth about all jail tools - shuffle packets with abandon - wield & jail ZFS datasets like a ninja - what does the FreeBSD installer do? - how to share data between hosts and jails - summon customised jails like an arcane sorcerer - acquaint yourself with jail security <i> > As this is the third time I've done this tutorial, I don't know > how far we'll get. If we have stuff left over, or stuff you're > not clear about, please let me know! </i> ??? And, if we have time: --- ## ... Hidden Agenda, Time Permitting - OCI container usage on FreeBSD - containerd & runj, nerdctl, `podman(1)`, `buildah(1)` - learn about `tarfs(5)` from 14.0-RELEASE - learn about `vxlan(4)` to make multiple jail hosts appear as one --- ## Jails Tools - ezjail, qjail, iocell, iocage, py-iocage, quickjail - bastille, mkjail, pot, nomad, cbsd, appjail, jailer - jadm, jest, runj, ocijail, podman, buildah, ansible-jails - yours too ? --- ## Jails as the kernel sees it - a single syscall that locks the *current* process inside it and results in, optionally: - a process with a restricted view of the filesystem - restricted view of processes, devices `/dev/*` - including network interfaces and IP addresses ```c struct jail { uint32_t version; char *path; char *hostname; char *jailname; unsigned int ip4s; unsigned int ip6s; struct in_addr *ip4; struct in6_addr *ip6; }; ``` ```c int jail(struct jail *jail); int jail_attach(int jid); int jail_remove(int jid); int jail_get(struct iovec *iov, u_int niov, int flags); int jail_set(struct iovec *iov, u_int niov, int flags); ``` If your new process is `/sbin/init` it looks awfully like a full server. ??? As far as API calls go, this isn't very intimidating. Define a jail struct, attach a jail, set some parameters on it, easy peasy. What's important is that many jail parameters are *not* stored in the struct, they're ephemeral. ## Inside the kernel ```c /* sys/sys/jail.h */ enum prison_state { PRISON_STATE_INVALID = 0, /* New prison, not ready to be seen */ PRISON_STATE_ALIVE, /* Current prison, visible to all */ PRISON_STATE_DYING /* Removed but holding resources, */ }; /* optionally visible. */ #define JAIL_MAX 999999 ``` ??? The kernel tracks very little state for jails - its up, down or dying. The rest is up to the userland tools to manage. --- ## The Truth about all Jail Tools - there is only 1 [jail syscall] - it doesn't provide - filesystem manipulation - network and firewall rules - DNS or anything else - everything else is just smoke, mirrors, and shell scripts - you don't even *need* jail.conf to do jails ??? From the user's perspective, the jail syscall itself is less interesting. It's the dirty work of mounting filesystems, setting up network interfaces and firewall rules, before spawning processes inside the jail environment, and eventually cleaning it all up. But before we can understand that, we need to cover a little bit about these components first to get to our first jails. The plan is to go pretty fast through these, answer any questions as we get started, and then get some hands-on fun, knowing we have this as reference material. --- ## ... about FreeBSD - not really an Operating System - a toolbox, and a pile of legos, to *build your own OS* --- ## Hands On #0 My First Jail - welcome to [jail(8)] and [jexec(8)] - try out [ps(1)], [top(1)] both inside and outside jails - what happens if you use a string for `name` instead of a number? - what is the return code of the [jail(8)] invocation? why? ```shell # uname -a # freebsd-version -kru # jail -vc path=/ name=12345 command=/bin/sh jail_set(JAIL_CREATE) persist path=/ name=12345 created run command in jail: /bin/sh # sysctl security.jail.jailed security.jail.jailed: 1 # ps auxwwd USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 37 0.0 0.0 13500 3308 12 SJ 22:49 0:00.00 /bin/sh root 9922 0.0 0.0 13560 3148 12 R+J 22:49 0:00.00 - ps auxwwd # exit 1 # echo $? ``` ??? Note that the error code of the process is propagated back --- ## Hands On #0 Persisting Jails - the old jail is gone, make a new one that persists - try leaving things running, and reconnecting ```shell # jail -c path=/ name=12345 persist=true command=/usr/local/bin/tmux # ps auxwwwd -J 12345 ### run top(1), then exit tmux via "^b d" leaving top running ### outside # ps auxwwd -J 12345 ### rejoin # jexec 12345 /usr/local/bin/tmux new -DAs # sysctl security.jail |less ### "^b d" to exit again # jail -rv 12345 # ps auxwwd -J 12345 ``` ??? Secret hack: by picking a *name* that's numeric, we get to assign a jail id. I would like this as a proper parameter, as I like names too. And tags. --- ## Hands On #0 Jailed Filesystems - we already have a tmpfs on the host system - make a jail as before ``` root@ /# mount zroot/ROOT/default on / (zfs, local, noatime, nfsv4acls) # df -h /tmp/ Filesystem Size Used Avail Capacity iused ifree %iused Mounted on tmpfs 120G 166M 120G 0% 268 472M 0% [restricted] root@ /# df -h Filesystem Size Used Avail Capacity Mounted on zroot/ROOT/default 41G 1.2G 40G 3% / root@ /# zfs list no datasets available # sysctl security.jail |less ``` - use `jail -rv 12345` - repeat [jail(8)] with `enforce_statfs=1` as well, and =2 ??? Check tmpfs by `mount -t tmpfs` and if missing, run: echo "tmpfs /tmp tmpfs rw,mode=01777,size=1g 0 0" | tee -a /etc/fstab mount -a Both process views and filesystem views are restricted. statfs=2 (default) shows almost nothing statfs=1 allows the jail to see parent mountpoints statfs=0 allows the jail to see everything (all filesystems) --- ## Networking: basics For a jail to access the network, it needs *either* - a shared interface + IP with the host - a loopback IP (to access host & other jails) - routing or firewall rules to get out to the network - it's own network interface entirely (physical or VNET stack) - some DNS goodness - Unix Domain Socket as a proxy outside the jail ??? Notes --- ## Networking: Shared IP - basic configuration in `/etc/rc.conf` on the host ```shell # assuming we are on 10.0.0.1 # provide additional 5 IP aliases # we can assign these to jails ifconfig_vtnet0_aliases="inet 10.0.0.201-206/24" # you must *reboot* the VM here, otherwise you will lose access ``` ``` # ifconfig vtnet0 vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU, ...> ether 02:00:17:05:14:a0 inet6 fe80::17ff:fe05:14a0%vtnet0 prefixlen 64 scopeid 0x1 inet 10.0.0.82 netmask 0xffffff00 broadcast 10.0.0.255 inet 10.0.0.201 netmask 0xffffff00 broadcast 10.0.0.255 inet 10.0.0.202 netmask 0xffffffff broadcast 10.0.0.102 inet 10.0.0.203 netmask 0xffffffff broadcast 10.0.0.103 inet 10.0.0.204 netmask 0xffffffff broadcast 10.0.0.104 inet 10.0.0.205 netmask 0xffffffff broadcast 10.0.0.105 inet 10.0.0.206 netmask 0xffffffff broadcast 10.0.0.106 media: Ethernet autoselect (10Gbase-T <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> ``` ??? Notes --- ## Networking: Loopback IP - again, configuration in `/etc/rc.conf` on the host ```shell # jail networks using CGN (Carrier Grade NAT) range # we can start from zero, but networking people hate this # provide a separate interface just for our jails cloned_interfaces="${cloned_interfaces} lo1" ifconfig_lo1_aliases="inet 100.64.0.0-15/24" # reboot again or run `/etc/netstart` inside a tmux ``` ??? CGN is very handy, as it's almost never used for local networks. no IP range conflicts! Same for lo1, just with different IP ranges. Note the group there, we'll come back to this later. https://www.cisco.com/c/en/us/support/docs/ip/dynamic-address-allocation-resolution/13711-40.html https://tools.ietf.org/html/rfc1878 --- ## Networking: Firewalls but not Routing - [pf(4)] and [pf.conf(5)] - TLDR: use anchors for runtime modification of redirections - uses `cloned_interfaces` function from `/etc/rc.conf` earlier - run `kldload pf; touch /etc/pf.jails /etc/pf.conf` ```config # /etc/pf.conf extl_if="vtnet0" jail_if="lo1" jail_net=$jail_if:network jail_ports="{ http, https }" host_ports="{ ssh }" set block-policy return scrub in on $extl_if all fragment reassemble set skip on lo # jails are allowed outbound connections via NAT nat on $extl_if inet from $jail_net -> ($extl_if:0) # ... to be continued ``` We are not going to talk much about routing because we'd need another week. Assign your jails IP addresses, do the routing things, profit. --- ## Networking: Firewalls, Redirection ... ```config # insert after NAT rule # redirect port 80 to a specific jail, insert in /etc/pf.conf # redirect incoming traffic to our delicious jail rdr pass on $extl_if proto tcp from any \ to any port $jail_ports -> 100.64.0.1 # or use a table to do dynamic redirection table <jails> persist file "/etc/pf.jails" rdr-anchor "jails/*" pass in quick on $extl_if proto tcp from any to any port $host_ports # secure the things block in all pass out quick antispoof for $extl_if inet # oh ssh how I love you ``` - lets try it before rebooting ```shell # touch /etc/pf.jails /etc/pf.conf # service pf onestart # pfctl -vnf /etc/pf.conf # pfctl -vef /etc/pf.conf ### we get disconnected now, but can reconnect again # sysrc pf_enable=YES # reboot ``` ??? --- ## Networking: Dynamic pf(4) redirects It's possible to update pf rdr and other settings dynamically. As we have outbound NAT setup via the jail interface itself, it's only necessary to configure specific per-jail redirects. ```shell ### check the anchor exists # pfctl -sn | grep anchor rdr-anchor "jails/*" all ### check if a custom rule exists # pfctl -a jails/my_jail_rule -Psn pfctl: DIOCGETRULES: Invalid argument ### add a dynamic rule to the table echo 'rdr pass on vtnet0 proto tcp from any port https -> 100.64.0.1 port https' \ | pfctl -a jails/my_jail_rule -f- ### show rules again # pfctl -a jails/my_jail_rule -Psn rdr pass on vtnet0 inet proto tcp from any port = 443 to any -> 100.64.0.1 port 443 ### burn it down pfctl -a jails/my_jail_rule -Fn nat cleared ``` ??? Remember the golden rule of pf, packets only match if they *cross* a device. You cannot use a redirect from a localhost IP to another localhost IP for this reason. Ask me how I know. Debugging tip - use ngrep or tcpdump on the pflog0 interface to dump blocked traffic. tcpdump will show you the rule number that blocked it, as seen from `pfctl -vef /etc/pf.conf` from earlier. enable pflog via [rc.conf(5)]. --- background-image: url(...) ## Networking: VNET (VIMAGE) jails Jails support a fully virtualised IP stack, including pf. This requires either a logical software bridge to connect to, or routing to let the packets flow, or using netgraph. - use bridges and epairs to wire up your virtual switch to jails - use netgraph to create arbitrarily complex topologies ??? We'll not cover these specifically today, there's already too much to get through. The tutorial docs are open, so keep an eye out for upgrades. --- ## DNS: even jails need it When a jail starts up, you'll want to add packages to it, or configure applications. This means DNS is needed. ```shell # sysrc local_unbound_enable=YES # service local_unbound setup # echo nameserver 9.9.9.9 | tee /etc/resolv.conf ``` ```config # /etc/unbound/conf.d/jails.conf server: interface: 100.64.0.0 interface-automatic: yes access-control: 100.64.0.0/24 allow access-control: 0.0.0.0/0 deny ``` Restart it, and use `nameserver 100.64.0.0` in the jail's `/etc/resolv.conf` ??? My preferred config is to have a locked-down caching DNS server on the host, and require that all jails connect to it, and only to it, and to use a different resolver for the host itself. That's it. Nothing more required. The key thing here is permitting our jail network range to use local_unbound as a caching resolver, and also to tell unbound to ensure requests and responses use the appropriate interfaces on their little trips. The reason for using a different resolver for the host is to ensure that the host isn't at risk of cache pollution from the jails, who are not to be trusted! --- ## Unix Domain Sockets Most modern application servers don't need TCPIP to shuffle traffic around. They can use UDS instead, a filesystem-based path. The main advantages are - sockets are filesystem paths so they can have permissions - they can be deployed prior to jail start - allows passing traffic through proxies (haproxy, nginx, traefik ...) - faster than the IP stack (statistics please) - you can deploy network-less jails to confuse everybody ```shell # echo 'syslogd_flags="-ss -S /var/run/logpriv \ -p /var/run/log \ -p /jails/foo/var/run/log \ -S /jails/foo/var/run/logpriv"' | tee /etc/rc.conf.d/syslogd # service syslogd restart # sockstat -l |grep syslog root syslogd 55496 6 dgram /var/run/logpriv root syslogd 55496 7 dgram /var/run/log root syslogd 55496 8 dgram /jails/foo/var/run/log root syslogd 55496 9 dgram /jails/foo/var/run/logpriv ``` ??? We'll do a syslog example, now there's no need for jails to run their own syslogd process. Personally I use a dedicated syslog-ng process for this, instead of the main system syslogd, but thats up to you. For network traffic, consider running haproxy or nginx in a dedicated jail, and using nullfs mounts to make their unix domain sockets available in the other jails. This allows you to have jails that are networkless, for key applications, with no network stack at all, just mediated access through a proxy jail. ------------------------------------------------------------------------------- --- ## Hands On #1 Networking - connect to your VM - add a 100.64.0.0-15/15 cloned loopback interface to /etc/rc.conf - run `kldload pf; touch /etc/pf.jails /etc/pf.conf` - configure /etc/pf.conf for your jails - enable local_unbound and add the jail secret sauce - spawn up a jail bound to a specific IP - grab a coffee ```shell # sockstat -46lp 53 # mv /etc/resolv.conf /etc/resolv.disabled # jail -vc name=12345 path=/ ip4.addr='lo1|100.64.0.1' command=/bin/sh ## jailed # drill foo.com # echo nameserver 127.0.0.1 > /etc/resolv.conf # drill foo.com # echo nameserver 127.0.0.2 > /etc/resolv.conf # mv /etc/resolv.disabled /etc/resolv.conf # ping -c 1 127.0.0.1 # ping 127.0.0.2 ``` Re-try the above, with `allow.raw_sockets=true` ??? Exercise: add a pf.conf rule to block jail traffic apart from our local caching resolver. Question: why does 127.0.0.1 work here? But not 127.0.0.2? ------------------------------------------------------------------------------- --- ## Filesystems: so tasty - [devfs(5)] - get at the external world - has rulesets attached to selectively open /dev - defaults to `/dev/{null,random,zero}` and `stdin,stderr,stdout` - [nullfs(5)] - make a host filesystem appear inside the jail - [zfs(5)] - do useful awesome things - [fusefs(5)] - do terrible despicable things - [tmpfs(5)] - for java things [procfs(5)], python things [fdescfs(5)] - [tarfs(5)] - the ultimate read-only jail Not all filesystems are jail-safe - use [lsvfs(1)] to check: <pre> Filesystem Num Refs Flags cd9660 0x000000bd 0 read-only devfs 0x00000071 1 synthetic, jail fusefs 0x000000ed 0 synthetic, jail msdosfs 0x00000032 1 nfs 0x0000003a 0 network ***** patches available ** nullfs 0x00000029 0 loopback, jail procfs 0x00000002 0 synthetic, jail tarfs 0x00000031 0 read-only (14.0-RELEASE and up) tmpfs 0x00000087 1 jail ufs 0x00000035 0 ************** use nullfs ********* zfs 0x000000de 9 jail, delegated-administration </pre> --- ## devfs: Exposing devices devfs rules in the host OS allow defining rules that filter `/dev/` nodes, to make them available in jails. The config format is crufty but straightfowards. Use the `devfs_ruleset=...` parameter on jail creation to modify this. ``` # /etc/defaults/devfs.rules but edit /etc/devsfs.rules [devfsrules_jail=4] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path fuse unhide add path zfs unhide [devfsrules_jail_vnet=5] ... same as before, with add include $devfsrules_jail add path pf unhide ``` ??? The jail syscall requires a path, to be the new jail's / root. We can mount additional filesystems under the jail, but not all filesystems are jail-safe. We're focusing mainly on ZFS because it has the most interesting functionality. There are patches for nfs clients in jails here, YMMV https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251347 Use [net/unfs3](https://unfs3.github.io/) for a jailable NFSv3 server. See https://vermaden.wordpress.com/2022/05/22/nfs-server-inside-freebsd-vnet-jail/ for a guide. --- ## What does [bsdinstall(8)] do? - selects and partitions disks (not much use for jails) - unpacks tarballs - sets up DNS via /etc/resolv.conf - sets up timezones via [tzsetup(8)] - deploys a minimal `/etc/rc.conf` - disables/enables a bunch of sysctls and daemons on request - would be nice if it also installed some packages Try running `bsdinstall jail /tmp/myjail` ??? --- ## Hands On #3 Filesystem Jails and bsdinstall ```shell # mkdir -p /tmp/myjail/examples # bsdinstall jail /tmp/myjail # mount_nullfs -o ro /usr/share/examples /tmp/myjail/examples # echo gonzo | tee /etc/gonzo # touch /tmp/myjail/gonzo # mount_nullfs -o ro /etc/gonzo /tmp/myjail/gonzo # jail -vc name=12345 path=/tmp/myjail devfs_ruleset=5 command=/bin/sh ``` ```shell ## jailed but what do we see? # ls -AFGhl /dev # cat /etc/rc.conf # pfctl -s all # mount -t devfs devfs /dev # mount -t tmpfs tmpfs /tmp # mkdir /examples/foo # cat /gonzo # exit ``` ??? Most of these commands fail, for various reasons. We need to ensure tmpfs and devfs are available, and that appropriate directories are readonly or writable as required. --- ## Hands On #3 Dying Jails Repeat, making a new jail, adding just one new parameter each time: - allow.mount - allow.mount.devfs - enforce_statfs=1 Once you succeed with the mount command inside the jail, exit the jail. - run `jls` to show there's nothing left... or is there? - what does `jls --libxo json -v | jq .` show you? - what does `jls --libxo json -vd | jq .` show you? - try to make a new jail with same jid Why does this fail? How can you fix it? ??? You need all of these to be able to mount filesystems in the jail. Personally, I prefer to do the dev/tmpfs mounting *outside* the jail, so that you don't have that. ------------------------------------------------------------------------------- --- ## Filesystems: ZFS in 3 mins - copy-on-write checksum zero-cost snapshot jail-aware fs - nestable datasets with inheritable and overridable properties - pure awesome ```shell # zfs create -o mountpoint=/jails \ -o compression=zstd-19 -o checksum=skein zroot/jails # zfs create -o canmount=off zroot/jails/downloads # zfs create -o canmount=off zroot/jails/instances # zfs create -o canmount=off zroot/jails/templates # zfs create zroot/jails/downloads/13.4-RELEASE-arm64-aarch64 # zfs create zroot/jails/templates/13.4-RELEASE-arm64-aarch64 # zfs snapshot zroot/jails/downloads/13.4-RELEASE-arm64-aarch64@empty #### fill it with stuff # zfs snapshot zroot/jails/downloads/13.4-RELEASE-arm64-aarch64@tasty ### duplicate that data # zfs send -LevcPR zroot/jails/downloads/13.4-RELEASE-arm64-aarch64@tasty \ > /tmp/tasty.zfs #### annihilate that tasty data # zfs rollback zroot/jails/downloads/13.4-RELEASE-arm64-aarch64@empty #### recover that data from our duplicate # zfs recv -Fuvs zroot/jails/downloads/13.4-RELEASE-arm64-aarch64 \ < /tmp/tasty.zfs ``` --- ## Templates: zfs style ```shell # cd /jails/downloads/13* # curl -#LO https://download.freebsd.org/ftp/releases/arm64/aarch64/13.4-RELEASE/base.txz # curl -#LO .../kernel.txz # cd /jails/templates/13* # tar xvpf /jails/downloads/13*/base.txz -C . # tar xvpf /jails/downloads/13*/kernel.txz -C . # zfs snapshot zroot$(pwd)@unpacked ``` ??? Once we've done a few more steps, we will clone this template for each jail we want to use. --- ## Templates: tweak my ride - in `etc`, amend `resolv.conf`, `rc.conf`, `freebsd-update.conf` - run `tzsetup -s -C . UTC` - s/quarterly/latest/ for `/etc/pkg/FreeBSD.conf` - snapshot again `zfs snapshot zroot$(pwd)@tweaked` ```shell # echo nameserver 127.0.0.1 | tee etc/resolv.conf ``` ``` # /etc/rc.conf zfs_enable=YES cron_enable=NO ntpd_enable=NO sendmail_enable=NONE sshd_enable=NO syslogd_enable=NO ``` ``` # append to /etc/freebsd-update.conf Components world kernel StrictComponents yes IDSIgnorePaths /etc/freebsd-update.conf IDSIgnorePaths /etc/master.passwd IDSIgnorePaths /etc/passwd IDSIgnorePaths /etc/pkg/FreeBSD.conf IDSIgnorePaths /etc/pwd.db IDSIgnorePaths /etc/shells IDSIgnorePaths /etc/spwd.db ``` ??? Most of this is pretty clear, tzsetup runs in a chroot, we want the best freshest packages, and we'll use the DNS resolver we built. Remember how the kernel remaps the DNS IP addresses on our behalf inside the jail. --- ## Templates: patching and verification - we use [freebsd-update(8)] to update and verify our jail - then, snapshot it for posterity and security ```shell # /usr/sbin/freebsd-update -b $(pwd) \ --currently-running 13.4-RELEASE \ --not-running-from-cron \ fetch install \ | tee /dev/null # zfs snapshot zroot$(pwd)@patched ``` - check that nothing got tampered with in transit ```shell /usr/sbin/freebsd-update -b $(pwd) \ -f ./etc/freebsd-update.conf \ --currently-running 13.4-RELEASE IDS zfs snapshot zroot$(pwd)@verified ``` --- ## Templates: pkg installs ```shell # ABI="FreeBSD:13:aarch64" # PKGS="ftp/curl sysutils/tmux sysutils/htop www/nginx" # mount -t devfs devfs ./dev # mount -t tmpfs tmpfs ./tmp # /usr/local/sbin/pkg -o ASSUME_ALWAYS_YES=yes -o AUTOCLEAN=yes \ --chroot ./ \ bootstrap -f # /usr/local/sbin/pkg -o ASSUME_ALWAYS_YES=yes -o AUTOCLEAN=yes \ --chroot . \ install -r FreeBSD -y \ ${PKGS} # umount ./dev # umount ./tmp # zfs snapshot zroot$(pwd)@packaged ``` ??? This is a large wall of text, just trust me :) - our template is a tuple of arch, machine, release - We need devfs for /dev/null and /dev/zero in the jails - install packages, then clean up - this is where you can customise further --- ## Hands On #4 Templating with ZFS - create a zfs tree for your jails with downloads and templates - fetch `base.txz`,`kernel.txz` into `/jails/downloads/13.4-RELEASE-arm64-aarch64/` from https://download.freebsd.org/ftp/releases/arm64/aarch64/13.4-RELEASE/ - do the unpacking, tweaking, patching, packages as above - run a final *zfs snapshot zroot$(pwd)@ready* for cloning from ??? ------------------------------------------------------------------------------- --- ## Phases of Jail Invocation - tasks running as root in the host OS (create datasets, mount fs etc) - kernel prepares process to be jailed & kexecs into it - `exec.created` tasks run on host - `exec.start` runs in jail, typically your /etc/rc init script - do your jaily things - `exec.stop` runs inside the jail - `exec.poststop` tasks run as root in the host OS after jail teardown - many more of these phases, see [jail(8)] for details ??? NB jail cleanup is .. messy. Sometimes post-exec scripts don't run, filesystems get left behind, nothing works consistently. Manual wrappers. --- ## Hands On #5 Creating a Jail Prepare a new jail from our template. You can modify it as required. Let's get nginx to run on a readonly zfs dataset, softlinking things into the future jail /tmp dir, and removing an inconvenient symlink. ```shell # zfs clone zroot/jails/templates/13.4-RELEASE-arm64-aarch64@ready \ zroot/jails/instances/dublin # cd /jails/instances/dublin # echo nginx_enable=YES | tee ./etc/rc.conf.d/nginx # rm -rf var/tmp/nginx var/log/nginx var/run # ln -s /tmp var/tmp/nginx # ln -s /tmp var/log/nginx # ln -s /tmp var/run # rm usr/local/www/nginx # cp -av usr/local/www/nginx-dist usr/local/www/nginx # zfs snapshot zroot/jails/instances/dublin@ready # zfs set readonly=on zroot/jails/instances/dublin ``` --- ## Hands On #5 Creating a Jail ```shell # jail -vc name=dublin \ path=/jails/instances/dublin \ ip4.addr='lo1|100.64.0.1' \ exec.created='mount -t tmpfs tmpfs /jails/instances/dublin/tmp' \ exec.start='/bin/sh /etc/rc' \ exec.poststop='umount -f /jails/instances/dublin/tmp' \ exec.release='umount -f /jails/instances/dublin/dev' \ allow.mount \ allow.mount.zfs \ enforce_statfs=1 \ host.hostname='dublin' \ mount.devfs \ command=/bin/sh -c "env -i TERM=$TERM tmux -u new-session -DAs default" ``` --- # curl localhost try with these additional parameters - what do they do? persist osreldate=1304000 osrelease=13.4-RELEASE securelevel=3 allow.mount.devfs allow.mount.tmpfs allow.nomount allow.raw_sockets allow.reserved_ports see if you can get syslogd to run inside the jail. You'll need this on the host: ``` # echo 'syslogd_flags="-ss -S /var/run/logpriv -p /var/run/log -p /jails/instances/dublin/var/run/log -p /jails/instances/dublin/var/run/logpriv"' \ | tee /etc/rc.conf.d/syslogd # service syslogd restart ``` ------------------------------------------------------------------------------- --- ## ZFS: jailing datasets - immutable jails, mutable data - make a dataset on the host available within the jail - uses zfs nested datasets and custom mountpoint property - will overwrite the `/usr/local/www/nginx` dir we made earlier ```shell # zfs create -o canmount=off -o mountpoint=none zroot/jailed # zfs create \ -o canmount=on \ -o jailed=on \ -o mountpoint=/usr/local/www/nginx \ zroot/jailed/dublin # zfs list -o name,jailed,canmount,mounted,mountpoint,name -r zroot/jailed ``` ??? # zfs list -o name,jailed,canmount,mounted,m NAME JAILED CANMOUNT MOUNTED MOUNTPOINT NAME zroot/jailed off off no none zroot/jailed zroot/jailed/dublin on on no /usr/local/www/nginx zroot/jailed/dublin --- ## Hands On #6 Jailing ZFS datasets - run through previous slide steps - restart the jail with new parameters ``` # jail -vcmr name=dublin \ path=/jails/instances/dublin \ ip4.addr='lo1|100.64.0.1' \ exec.created='zfs jail dublin zroot/jailed/dublin' \ exec.created="mount -t tmpfs tmpfs /jails/instances/dublin/tmp" \ exec.start="/bin/sh /etc/rc" \ exec.poststop='zfs unjail dublin zroot/jailed/dublin' \ exec.poststop="umount -f /jails/instances/dublin/tmp" \ exec.release="umount -f /jails/instances/dublin/dev" \ host.hostname="dublin" \ allow.mount \ allow.mount.tmpfs \ allow.mount.zfs \ allow.reserved_ports \ devfs_ruleset=4 \ enforce_statfs=1 \ mount.devfs \ osreldate=1302001 \ osrelease=13.4-RELEASE-p3 \ persist \ command=/bin/sh -c "env -i TERM=$TERM tmux -u new-session -DAs default" ``` ------------------------------------------------------------------------------- # OCI Containers - s/containers/jails/ - out of the docker frying pan, into the freebsd fire - OCI is a standard for container runtimes across Linux, Windows, and now FreeBSD - enormous amount to cover, we will skip a lot - fill in the gaps during the conference! - major efforts by Samuel Karp, Doug Rabson, and many others to make this work on FreeBSD ??? everywhere you see "container" just mumble "jail" --- ## container basics - standards for container runtimes - "the spec" - an OCI container is a filesystem bundle - a simplified model is a tarball with a config file - a container can be made up of layers of tarballs - layers are shared using clever filesystem tricks ??? - e.g. kernel layer, minimal config, small static binary layer - next layer could be base.txz and more applications ??? Along with runj, nerdctl, containerd too/ --- ## runtime basics - runtimes run instances of containers - they are responsible for the lifecycle - from creation to destruction - but once the container is running, the runtime is hands-off - its just jails again ??? - containers can be started/stopped/destroyed using a runtime - podman has been ported to FreeBSD, along with nerdctl, and runj, & more --- ## Registry basics - containers can be stored in a registry - like a git repo but for containers - checksums, layers, versions, metdata - mainly we search, push, and pull from them ??? There are notes on running a local registry in the tutorial docs, but we will skip these for expediency. Feel free to experiment and ask. --- ## Hands On #7 OCI Containers - get set up: ``` # touch /etc/pf.conf /etc/pf.jails # service pf start # sysctl net.pf.filter_local=1 # zfs create -o mountpoint=/var/db/containers zroot/containers # zfs snapshot zroot/containers@empty # mount -t fdescfs fdesc /dev/fd # pkg install -yr FreeBSD podman-suite ``` ??? --- ## Hands On #8 OCI exploration There are a number of config files, we don't need them initially, but do explore at leisure. ``` # pkg list buildah podman conmon ocijail containers-common \ containernetworking-plugins \ |grep /etc/ /usr/local/etc/rc.d/podman /usr/local/etc/rc.d/podman_service /usr/local/etc/containers/containers.conf.sample /usr/local/etc/containers/policy.json.sample /usr/local/etc/containers/registries.conf.sample /usr/local/etc/containers/storage.conf.sample /usr/local/etc/containers/pf.conf.sample ``` --- ## Hands On #9 OCI exploration ``` # podman image search --list-tags docker.io/dougrabson/hello NAME TAG docker.io/dougrabson/hello latest # podman run --rmi docker.io/dougrabson/hello ... !... Hello Podman World ...! # podman ps -a # fd -tf . /var/db/containers ``` ??? A number of files, locks, and json files. /var/db/containers/storage/db.sql /var/db/containers/storage/defaultNetworkBackend /var/db/containers/storage/storage.lock /var/db/containers/storage/userns.lock /var/db/containers/storage/zfs-containers/containers.lock /var/db/containers/storage/zfs-containers/volatile-containers.json /var/db/containers/storage/zfs-images/images.json /var/db/containers/storage/zfs-images/images.lock /var/db/containers/storage/zfs-layers/layers.json /var/db/containers/storage/zfs-layers/layers.lock /var/db/containers/storage/zfs-layers/volatile-layers.json --- ## Hands On #10 OCI public images - https://github.com/orgs/skunkwerks/packages ``` # podman image search --list-tags ghcr.io/skunkwerks/freebsd14.1-minimal # buildah from ghcr.io/skunkwerks/freebsd14.1-minimal:latest-aarch64 # buildah run freebsd14.1-minimal-working-container freebsd-version -ru 15.0-CURRENT <----- your host kernel 14.1-RELEASE-p4 <-- your container userland ``` - run `ping -c 3 100.64.0.1` in the container - try `df -h /` in the container ??? - https://github.com/orgs/skunkwerks/packages is a public repo - we have a number of images there, 14.0 and 14.1 - they are platform & architecture-specific (linux,freebsd,amd64, arm64) - we want to fetch one and run it - note the different userland and kernel images - note the path to the container, its just zfs again NAME TAG ghcr.io/skunkwerks/freebsd14.1-minimal latest ghcr.io/skunkwerks/freebsd14.1-minimal latest-amd64 ghcr.io/skunkwerks/freebsd14.1-minimal latest-aarch64 --- ## Hands On #11 OCI inside the images ``` # podman image list # podman image tree 068b35dd5edc # buildah from ghcr.io/skunkwerks/freebsd14.1-small:latest-aarch64 # podman image list ``` ??? Note when we fetch the next image, the blobs (layers) are local - space is saved. ``` REPOSITORY TAG IMAGE ID CREATED SIZE ghcr.io/skunkwerks/freebsd14.1-minimal latest-aarch64 068b35dd5edc 2 days ago 34.7 MB Image ID: 068b35dd5edc Tags: [ghcr.io/skunkwerks/freebsd14.1-minimal:latest-aarch64] Size: 34.74MB Image Layers ├── ID: 6465bbfd83f4 Size: 1.195MB ├── ID: a51e0f440026 Size: 4.994MB ├── ID: bb5e44e48370 Size: 9.954MB └── ID: 0f90814d2d6d Size: 18.6MB Top Layer of: [ghcr.io/skunkwerks/freebsd14.1-minimal:latest-aarch64] ``` --- ## Hands On #12 OCI building - we can create our own images using buildah - like git, we fork the image layers - make changes, commit them into a new layer - tag the image, push it to a registry ``` # mkdir /var/cache/buildah # buildah images # buildah containers # export buildah from \ ghcr.io/skunkwerks/freebsd14.1-minimal:latest-aarch64 # buildah from --volume /var/cache/buildah:/var/cache/pkg \ --volume /tmp:/tmp \ --name web-builder \ ghcr.io/skunkwerks/freebsd14.1-minimal:latest-aarch64 # buildah list # buildah run web-builder pkg install -y www/nginx ``` --- ------------------------------------------------------------------------------- snippet shelf pushing images ``` $ for i in (podman images -n | cut -wf 1 | grep freebsd | sed -e s,localhost/,,); buildah push $i docker://ghcr.io/skunkwerks/$i; end Getting image source signatures Copying blob c01a330e0b2c done | Copying blob ad5272764424 done | Copying blob 4f1f27c7bec0 done | Copying blob ba36bfad5bb8 done | Copying blob 6465bbfd83f4 done | Copying config 2a9aa442af done | Writing manifest to image destination Getting image source signatures Copying blob 93bc9f13a476 skipped: already exists Copying blob 71d2080dadf8 skipped: already exists Copying blob e7086f1818ab skipped: already exists Copying blob 6eb5801da48e skipped: already exists Copying blob cc1dff9ff8f3 skipped: already exists Copying config 2a9aa442af done | Writing manifest to image destination Getting image source signatures Copying blob cc1dff9ff8f3 skipped: already exists Copying blob e7086f1818ab skipped: already exists Copying blob 71d2080dadf8 skipped: already exists ``` ## oci metadata ``` # jq -C .freeformTags.repo /var/run/oci_metadata.json "https://git.sr.ht/~dch/diy-jails-tutorial" # jq -C .freeformTags.script /var/run/oci_metadata.json "deploy.sh" ``` ## security - the more specifically bound ip:port combo gets the data - shared IPs are trouble - root in jails can change the MAC address - UNIX sockets are preferred over IP addresses - SYSVIPC security (postgres & similar shared memory esgments) - dockerish jails - https://github.com/jackvz/oci-freebsd - https://github.com/samuelkarp/runj - https://akhramov.github.io/posts/2021-09-16-FreeBSD-OCI-jails.html - https://github.com/containerd/nerdctl/issues/868 - https://github.com/michael-yuji/xca jail-aware commands - jail, jexec - ps, mount, unmount, top, sockstat, fstat, procstat, ifconfig ... why isn't netstat jail-aware? MACHINE=$(sysctl -n hw.machine) # hw.machine_arch: aarch64 | amd64 ARCH=$(sysctl -n hw.machine_arch) SETS="base kernel" SITE="https://download.freebsd.org/ftp/releases/${MACHINE}/${ARCH}/" # what do we we fetch RELEASE="13.4-RELEASE" VERSION="${MACHINE}-${ARCH}-${RELEASE}" ABI="FreeBSD:13:${ARCH}" -------------------------------------------------- ``` # /etc/pf.conf extl_if="vtnet0" jail_if="lo1" jail_net=$jail_if:network jail_ports="{ http, https }" host_ports="{ ssh }" set block-policy return scrub in on $extl_if all fragment reassemble set skip on lo # redirections first rdr pass inet proto tcp from any port $jail_ports -> 100.64.0.1 # or use a table to do dynamic redirection table <jails> persist file "/etc/pf.jails" rdr-anchor "jails/*" # jails are allowed outbound connections via NAT nat on $extl_if inet from $jail_net -> ($extl_if:0) # allow ssh and mosh in all their glorious incantations pass in quick on $extl_if proto tcp from any port $host_ports # secure the things # block in all pass out quick antispoof for $extl_if inet # oh ssh how I love you pass in quick on $extl_if proto tcp from any port $host_ports ``` -------------------------------------------------- [bsdinstall(8)]: https://man.freebsd.org/bsdinstall [dch]: mailto:dch@FreeBSD.org [devfs(5)]: https://man.freebsd.org/devfs [fdescfs(5)]: https://man.freebsd.org/fdescfs [freebsd-update(8)]: https://man.freebsd.org/freebsd-update [fusefs(5)]: https://man.freebsd.org/devfs [jail syscall]: https://man.freebsd.org/jail_set [jail(8)]: https://man.freebsd.org/jail [jexec(8)]: https://man.freebsd.org/jexec [nullfs(5)]: https://man.freebsd.org/nullfs [pf(4)]: https://man.freebsd.org/pf [pf.conf(5)]: https://man.freebsd.org/pf.conf [procfs(5)]: https://man.freebsd.org/procfs [ps(1)]: https://man.freebsd.org/ps [rc.conf(5)]: https://man.freebsd.org/rc.conf [tarfs(5)]: https://man.freebsd.org/top [tmpfs(5)]: https://man.freebsd.org/tmpfs [top(1)]: https://man.freebsd.org/top [tzsetup(8)]: https://man.freebsd.org/tzsetup [zfs(5)]: https://man.freebsd.org/zfs [lsvfs(1)]: https://man.freebsd.org/lsvfs