Ubuntu 18.04 LXD/LXC, ZFS, Docker, and Advanced Networking

by Will | May 20, 2018 | Linux

Note on 2018.5.20: This is a first draft. This will undergo many revisions.

I used many, many other tutorials, blogs, and Q&A posts to assemble this. Relevant links are sprinkled throughout the guide.

This is a complete, step by step tutorial on configuring the following:

• Ubuntu 18.04 install on a server with two NICs
• One NIC for host traffic
• Other NIC for LXC/Docker traffic
• Plex, Sonarr, Radarr, Jackett in Docker on host
• rTorrent, ruTorrent, Flood, and OpenVPN nested in Docker in LXC container on host

Topology from a visual perspective:

Topology from a CLI perspective:

INSTALL UBUNTU

At Filesystem Setup for 120GB SSD:
1. Leave bootloader partition alone
2. I gave 20GB to / partition
3. I gave 60GB to /home partition
4. I left the rest as free space so it can be used later in this guide for our ZFS pool

CREATE USER, ADD TO SUDO GROUP, SWITCH TO USER

adduser will
usermod -aG sudo will
su will

CONFIGURE HOST NETWORK INTERFACE

Get names of network interfaces
ip a

enp1s0f0 is for my host network
enp1s0f1 is for my container network

Edit the existing YAML file
sudo vim /etc/netplan/50-cloud-init.yaml

Apply changes
sudo netplan apply

Schedule script every boot to set the physical interface used for containers to be UP with PROMISC ON. This is necessary right now on Ubuntu 18.04 due to a bug documented here.
mkdir ~/scripts
echo "ip link set enp1s0f1 up && ip link set enp1s0f1 promisc on" > ~/scripts/enp1s0f1.sh
chmod +x ~/scripts/enp1s0f1.sh

The following command will write to crontab so that your script runs as root at boot.
This did not work with only the first sudo, so I threw a bunch of extra sudos in there to make it work. I don’t know if they are all necessary.
sudo crontab -u root -l | { sudo cat; sudo echo "@reboot /home/will/scripts/enp1s0f1.sh"; } | sudo crontab -

SET UNUSED INTERFACES DOWN

I have extra physical interfaces that I’m not using right now, so I’m shutting them down.
sudo ip link set en131s0f0 down
sudo ip link set en131s0f1 down

INTERFACE TROUBLESHOOTING

You can do the following to remove an IP from an interface. For example, I accidentally assigned 10.1.20.24/24 to enp1s0f1 but I want that IP on enp1s0f0 instead:
sudo ip address del 10.1.20.24/24 dev enp1s0f1

You can do the following to restart an interface:
sudo ip link set enp1s0f1 down && sudo ip link set enp1s0f1 up

If you need to fix 127.0.0.53 being in resolv.conf (this happened to me):
sudo rm -f /etc/resolv.conf

CREATE YOUR HOST UPGRADE SCRIPT

Create update script:
cd ~
vim update.sh

Make update script executable:
chmod +x update.sh

Run script to update. You can run this whenever you want to update.
./update.sh

Install whatever common packages you use:
sudo apt install tree unrar ncdu -y

INSTALL ZFS AND IMPORT ZFS POOL FROM HBA CONTROLLER

sudo apt install zfsutils-linux -y
zpool import
zpool import tank

Import zpools at boot. Not necessary if you reference the disk by-id when doing initial import as discussed here.
sudo systemctl enable zfs-import-cache

INSTALL AND CONFIGURE SAMBA

This section is entirely optional. I like being able to access my entire media ZFS pool from Windows.

Install Samba as described here.
sudo apt install -y samba
sudo nano /etc/samba/smb.conf

Turn SMB sharing on:
sudo zfs set sharesmb=on tank

Create Samba user and define password
sudo pdbedit -a will

Find out who owns the media folder
ls -l /tank/media

I don’t care about UID 1420, but I do want to name the GID 1420 group “media”
sudo groupadd media
sudo groupmod -g 1420 media

Observe changes:

Add myself to the media group and restart Samba
sudo adduser will media
sudo systemctl restart smbd nmbd

CONFIGURE LXC PROFILE TO USE MACVLAN

Run lxc without typing sudo every time:
sudo setfacl -m u:will:rwx ~/.config/lxc

Create and edit the YAML file:
lxc profile create lxcnet
lxc profile edit lxcnet

CREATE ZFS PARTITION ON LOCAL SSD

Find disk/partition to be used
sudo fdisk -l

Check lxc version. On 18.04 it’s 3.0.0 right now.
lxc info

Start configuration of LXC
sudo lxd init

The outcome of these steps is that the network configuration from the lxcnet profile is copied to the default profile, and the default profile is populated with the ZFS pool information.

You can see this with the following:
lxc profile show default

INCREASE FILE AND INODE LIMITS

sudo vim /etc/sysctl.conf

sudo vim /etc/security/limits.conf

Now reboot:
sudo reboot

VERIFY HOST NETWORK, LXD NETWORK, AND ZFS

Observe your primary network interface matches what you set in /etc/netplan/00-netcfg.yaml
ip a

Note that a pool has been created with datasets:
sudo zfs list

CREATE AN LXC CONTAINER FOR TORRENT/VPN

Note, this is an empty Ubuntu 16.04 container. I’m naming it ‘torrent’
lxc launch ubuntu:xenial torrent

See that the container has started:
lxc list

Look for the MAC address in the container:
lxc config show --expanded torrent

In my case, I see the following:
volatile.eth0.hwaddr: 00:16:3e:e1:65:36

On my DHCP server, I create a new MAC reservation:
Name: torrent
IP: 10.1.20.12
MAC: 00:16:3e:e1:65:36

Enter the torrent container:
lxc exec torrent bash

Remove the dynamic IP so you can get the static one assigned
ip addr flush dev eth0
rm /var/lib/dhcp/dhclient.eth0.leases
dhclient -r; dhclient

You should see the DHCP-assigned static IP address:
ip a

Now exit the container
exit

MOUNT HOST DIRECTORY INTO LXC CONTAINER

Stop it if it’s running:
lxc stop torrent

Make it privileged to avoid file ownership issues as noted here:
lxc config set torrent security.privileged true

Mount /tank/downloads to /downloads:
lxc config device add torrent downloads disk source=/tank/downloads path=/downloads

Allow Docker inside LXD container:
lxc config set torrent security.nesting true

CREATE NON-ROOT USER AND ASSIGN PRIVILEGES

Start the container and enter it:
lxc start torrent
lxc exec torrent bash

Create user in the container and assign permissions:
adduser will
usermod -aG sudo will
groupadd media
adduser will media
usermod -u 1420 will
groupmod -g 1420 media

Log into user and create command so you can run sudo as documented here:
su will
vim ~/.bashrc

Install the .bashrc:
source ~/.bashrc

INSTALL DOCKER IN LXC CONTAINER

Instructions pulled from here.
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt-get install -y docker-ce

Start it and enable it to start at boot:
sudo systemctl start docker
sudo systemctl enable docker

Let user do docker things without typing sudo:
sudo gpasswd -a will docker
sudo service docker restart
sudo systemctl enable docker

Leave and come back:
exit
su will

INSTALL PIA VPN AND TORRENT DOCKER IN LXC

Enter container if you aren’t already in it:
lxc exec torrent bash

Prepare host (LXC container) and create torrent config directory:
mkdir -p ~/torrent/config/openvpn
mkdir ~/torrent/openvpn_all
cd ~/torrent/openvpn_all
wget https://www.privateinternetaccess.com/openvpn/openvpn.zip
sudo apt install unzip
unzip openvpn.zip

Copy only the key, cert, and ovpn file you want to use:
cp *.crt ~/torrent/config/openvpn
cp *.pem ~/torrent/config/openvpn
cp US\ Midwest.ovpn ~/torrent/config/openvpn

Create and run torrent docker:

If you need to add a flag on the fly to a running container, here’s an example:
docker update --restart=always torrent

Troubleshooting Docker:
docker events&
docker start sonarr
docker logs 345fceb9d7589a51c6b2d40c4b84c2e7b4e23463a363a24d7bd47fffd3dec013

Enter a Docker container for troubleshooting:
docker exec -it torrent /bin/bash

INSTALL AND SET UP DOCKER ON HOST

Install Docker in Ubuntu 18.04:
curl -fsSL test.docker.com | sh

Create macvlan for hosts
docker network create -d macvlan \
--subnet=10.1.20.0/24 \
--gateway=10.1.20.1 \
-o parent=enp1s0f1 mvdock0

Create and run Plex container (source):
docker run -id \
--name plex \
--network=mvdock0 \
--ip=10.1.20.11 \
-h plex \
-e VERSION=latest \
-e TZ="America/Chicago" \
-e ADVERTISE_IP="http://plex.paw.blue:32400/" \
-e PLEX_UID=1420 -e PLEX_GID=1420 \
-v /tank/plexdata/config:/config \
-v /tank/media/tv:/tv \
-v /tank/media/movies:/movies \
-v /tank/media/education:/education \
-v /tank/transcode:/transcode \
plexinc/pms-docker

Sonarr:
docker run -d \
--name sonarr \
--network=mvdock0 \
--ip=10.1.20.15 \
-p 8989:8989 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-e PUID=1420 -e PGID=1420 \
-e TZ=America/Chicago \
-e DEBUG=false \
-v /etc/localtime:/etc/localtime:ro \
-v /home/will/sonarr/config:/config \
-v /tank/downloads:/downloads \
-v /tank/media/tv:/tv \
--restart=always \
linuxserver/sonarr

Radarr:
docker run -d \
--name=radarr \
--network=mvdock0 \
--ip=10.1.20.16 \
-p 7878:7878 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-e PGID=1420 -e PUID=1420 \
-e TZ=America/Chicago \
-v /etc/localtime:/etc/localtime:ro \
-v /home/will/radarr/config:/config \
-v /tank/downloads:/downloads \
-v /tank/media/movies:/movies \
--restart=always \
linuxserver/radarr

Jackett:
docker run -d \
--name=jackett \
--network=mvdock0 \
--ip=10.1.20.17 \
-p 9117:9117 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-v /home/will/jackett/config:/config \
-v /tank/downloads:/downloads \
-e PGID=1420 -e PUID=1420 \
-e TZ=America/Chicago \
-v /etc/localtime:/etc/localtime:ro \
--restart=always \
linuxserver/jackett

DOCKER EXPERIMENTAL THINGS

If you want to use an ipvlan instead of a macvlan in Ubuntu 18.04, you will have to start docker in experimental mode.

Enabling experimental mode:
dockerd --experimental=true

Examples of layer 2 and layer 3 ipvlan networks:

EXPLANATIONS ARE IN ORDER

Q: Why don’t you just run the vpn/torrent docker container on the host?
A: I tried to do that with the network=mvdock0 and ip flags, but it wasn’t working. I think it has something to do with the way the VPN is influencing the network connection. By making the vpn/torrent docker container use the “host” network and having the “host” actually be an LXC container, I can still ensure this traffic passes through enp1s0f1 on the physical host.

Q: Why not do all LXC or all Docker? Why mix and match?
A: First, so I could learn both. Second, Docker is so easy to get my applications up and running.
Third, my inspiration for this project was this amazing post by Jason Bayton,  and I loved the idea of hosting LXC containers in ZFS. As it turns out, I only hosted one. But perhaps more soon!

SUMMARY

This took me many hours to assemble, as I had very little LXC or Docker experience before setting out on this journey. I’m sure people will point out many, many flaws in this tutorial. Please comment so that I can fix them!

If this guide helped you, please consider a small crypto donation!
BTC: 3LypUnvR1RktXqF4FAA7WT5tCZXxjSDH1f
ETH: 0xc23D1cb22324873Bf7dc4e3FFaD81621Ce50F8EC
LTC: MJb1YqnFSdPkM6vaX1ApnGz96donGZfAU3