Problem
As soon as I got TrueNAS relatively stable and configured, I tried to migrate my Docker services over to the platform as part of my grand plan to re-home the majority of my infrastructure on to my R630. But my containers weren’t networking properly; in particular, they didn’t have external connectivity. Some quick Googling surfaced the answer:
- the default configuration sets
iptables=false
so the containers have to route through a Kubernetes ingress object (like a LoadBalancer), which I didn’t have nor want - the default configuration assumes that you’re managing everything through the Web UI (which doesn’t satisfy my use cases) and running on Kubernetes (which is overkill for my use cases)
TrueNAS actively enforces system configuration at reboot by overwriting local changes from the system database, so some modifications are required to run vanilla Docker. Some quick cleanup and scripting to update the Docker configuration and I should be all set, right? Well…
Solution
Turns out there were two key pieces of information required to ultimately fix the issue:
- I needed a custom Docker daemon configuration to be applied at reboot that disabled the system configuration and replaced it with my user configuration, enabling a “vanilla” Docker experience
- I needed a custom Docker volume (more on this below) to be mounted at reboot to avoid performance and stability issues running Docker over ZFS with the default
zfs
storage driver
Since I had been playing around with the built-in Docker/k8s capability in the Web UI, the first thing I had to do was disable that capability by unassigning the pool. I didn’t have anything valuable stored at that location, so I also deleted the ix-applications
dataset as part of the cleanup.
Now I needed a way to reliably configure the system at startup. After doing some research, I elected to not use the Docker zfs
storage driver due to the snapshot issue and instead elected to create a new zvol
— a block storage device in ZFS — formatted as ext4
and mounted at /var/lib/docker
. Since this appears as an ext4
volume to the system, I can use the default overlay2
storage driver for Docker as well as the default location. To make this persistent across restarts, I assembled the following script, which runs PREINIT
as an “Init” script in TrueNAS:
#!/usr/bin/env bash
####################################################
### pre()
####################################################
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
# Write activities to local log file
exec > "${SCRIPT_DIR}/docker-output.log" 2>&1
####################################################
### main()
####################################################
echo "Stopping daemon..."
systemctl stop docker
### mount user volume for Docker storage
echo "Checking mounts..."
NUM_MOUNTS=$(mount -l | grep "/var/lib/docker " | wc -l)
if [ "$NUM_MOUNTS" = "1" ]; then
# expected re-run behavior
# something is already mounted; skip this block
MOUNTPOINT=$(mount -l | grep "/var/lib/docker ")
echo "Found 1 mount: $MOUNTPOINT"
elif [ "$NUM_MOUNTS" = "0" ]; then
# expected first run behavior
# remove any system data and mount user volume
echo "No mounts found. Removing system data..."
rm -rdf /var/lib/docker/*
echo "Mounting user Docker volume..."
mount /dev/data00/docker /var/lib/docker
echo "User Docker volume mounted!"
# exit 2
else
# this is an illegal condition and should never trigger
echo "Found $NUM_MOUNTS -- exiting..."
exit 1
fi
### install user configuration for Docker daemon
echo "Replacing daemon.json config. file..."
cmp --silent "${SCRIPT_DIR}/docker-config.json" /etc/docker/daemon.json || echo "Updating Docker config..." && cp -f "${SCRIPT_DIR}/docker-config.json" /etc/docker/daemon.json
chown root:root /etc/docker/daemon.json
echo "User daemon config. file installed!"
echo "Starting daemon..."
systemctl start docker
####################################################
### post()
####################################################
echo "Done!"
exit 0
The script is a little verbose (and could definitely be hardened) but basically does the following:
- stop the
docker
daemon - remove any lingering data at
/var/lib/docker
- mount my custom
/dev/data00/docker
volume - install my custom
config.json
file for Docker - start the
docker
daemon
The script and the associated configuration file are stored in my user home directory. The custom configuration file is pretty close to the factory default for Docker:
{
"data-root": "/var/lib/docker",
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"storage-driver": "overlay2",
"iptables": true,
"bridge": "",
"dns": ["1.1.1.1"]
}
with the key field being { "iptables": true }
to enable vanilla Docker networking for Swarm and Compose.
Creating the zvol
itself was straight-forward with the following caveats:
- the
zvol
will appear under/dev/{ pool_name }/{ zvol_name }
- don’t bother with
fdisk
; just format the raw volume viamkfs.ext4 /dev/{ pool_name }/{ zvol_name }
- don’t bother with adding the mount to
/etc/fstab
, which is overwritten at startup from the TrueNAS database - if in doubt, make the
zvol
smaller; it’s trivial to grow it from the Web UI and almost impossible to shrink it
For reference, I settled on making my “docker” zvol
128 GiB in size and located on my NVMe-backed “data00” pool
. I would have made it larger except the storage is preallocated and my pool is only 512 GB, so I wanted to keep utilization <50%.
To mirror previous deployments, I also created a dataset
for my Docker applications, which I called “stacks” and placed on the same “data00” pool
. This is native ZFS and used for bind-mounts into containers; if I run into issues, I’ll likely convert this to a zvol
and mount it similarly to my “docker” zvol
.
I have some test services running — namely Traefik and Portainer — so the real test will come when I migrate some of my real workloads over. Stay tuned!
References:
- TrueNAS Scale – Use Vanilla Docker
- Using Docker on TrueNAS SCALE (no Kubernetes)
- Setup TrueNAS Scale to work with portainer
- Home Server Setup (Ubuntu / Docker / ZFS) Sanity Check; in particular, this comment by u/myerscarpenter
- docker ZFS driver creates hundreds of datasets and doesn’t clean them #41055; in particular, this issue comment by kraduk