Skip to content

Ubuntu 18.04 LXD/LXC, ZFS, Docker, and Advanced Networking

This is a complete, step by step tutorial on configuring the following:

  • Ubuntu 18.04 install on a server with two NICs
  • One NIC for host traffic
  • Other NIC for LXC/Docker traffic
  • Plex, Sonarr, Radarr, Jackett in Docker on host
  • rTorrent, ruTorrent, Flood, and OpenVPN nested in Docker in LXC container on host

Topology from a visual perspective:

topology

Topology from a CLI perspective:

will@ara:~$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS                  PORTS               NAMES
3bccf3d47433        linuxserver/radarr    "/init"             5 hours ago         Up 5 hours                                  radarr
cecc3e98ae8e        linuxserver/sonarr    "/init"             5 hours ago         Up 5 hours                                  sonarr
4137c943105c        linuxserver/jackett   "/init"             18 hours ago        Up 18 hours                                 jackett
e5011d300ead        plexinc/pms-docker    "/init"             29 hours ago        Up 29 hours (healthy)                       plex
will@ara:~$ lxc list
+---------+---------+----------------------+------+------------+-----------+
|  NAME   |  STATE  |         IPV4         | IPV6 |    TYPE    | SNAPSHOTS |
+---------+---------+----------------------+------+------------+-----------+
| torrent | RUNNING | 172.17.0.1 (docker0) |      | PERSISTENT | 0         |
|         |         | 10.1.20.12 (eth0)    |      |            |           |
+---------+---------+----------------------+------+------------+-----------+
will@ara:~$ lxc exec torrent bash
root@torrent:~# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                                                                                            NAMES
71af0f693c10        binhex/arch-rtorrentvpn   "/usr/bin/tini -- /b…"   19 hours ago        Up 19 hours         0.0.0.0:3000->3000/tcp, 0.0.0.0:8118->8118/tcp, 0.0.0.0:9080->9080/tcp, 5000/tcp, 49160/tcp, 0.0.0.0:9443->9443/tcp, 49170/tcp   torrent

Install Ubuntu

At Filesystem Setup for 120GB SSD:

  1. Leave bootloader partition alone

  2. I gave 20GB to / partition

  3. I gave 60GB to /home partition

  4. I left the rest as free space so it can be used later in this guide for our ZFS pool

Create user, add to sudo group, switch to user

adduser will

usermod -aG sudo will

su will

Configure host network interface

Get names of network interfaces

ip a

enp1s0f0 is for my host network enp1s0f1 is for my container network

Edit the existing YAML file

sudo vim /etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    enp1s0f0:
      addresses: [10.1.20.24/24]
      gateway4: 10.1.20.1
      nameservers:
        addresses:
        - 10.1.20.254
        - 10.1.20.253
        - 1.1.1.1
        search:
        - paw.blue
    enp1s0f1:
      dhcp4: false
# I don't think this bridge is necessary but I could be wrong
#  bridges:
#    br0:
#      interfaces: [enp1s0f1]
#      dhcp4: true

Apply changes

sudo netplan apply

Schedule script every boot to set the physical interface used for containers to be UP with PROMISC ON. This is necessary right now on Ubuntu 18.04 due to a bug documented here.

mkdir ~/scripts

echo "ip link set enp1s0f1 up && ip link set enp1s0f1 promisc on" > ~/scripts/enp1s0f1.sh

chmod +x ~/scripts/enp1s0f1.sh

The following command will write to crontab so that your script runs as root at boot.

This did not work with only the first sudo, so I threw a bunch of extra sudos in there to make it work. I don't know if they are all necessary.

sudo crontab -u root -l | { sudo cat; sudo echo "@reboot /home/will/scripts/enp1s0f1.sh"; } | sudo crontab -

Set unused interfaces down

I have extra physical interfaces that I'm not using right now, so I'm shutting them down.

sudo ip link set en131s0f0 down

sudo ip link set en131s0f1 down

Interface troubleshooting

You can do the following to remove an IP from an interface. For example, I accidentally assigned 10.1.20.24/24 to enp1s0f1 but I want that IP on enp1s0f0 instead:

sudo ip address del 10.1.20.24/24 dev enp1s0f1

You can do the following to restart an interface:

sudo ip link set enp1s0f1 down && sudo ip link set enp1s0f1 up

If you need to fix 127.0.0.53 being in resolv.conf (this happened to me):

sudo rm -f /etc/resolv.conf

Create your host upgrade script

vim ~/update.sh

#/bin/bash
time sudo apt update && sudo apt full-upgrade -y

Make update script executable:

chmod +x update.sh

Run script to update. You can run this whenever you want to update.

./update.sh

Install whatever common packages you use:

sudo apt install tree unrar ncdu -y

Install ZFS and import ZFS Pool from HBA controller

sudo apt install zfsutils-linux -y

zpool import tank

Import zpools at boot. Not necessary if you reference the disk by-id when doing initial import as discussed here.

sudo systemctl enable zfs-import-cache

Install and configure samba

This section is entirely optional. I like being able to access my entire media ZFS pool from Windows.

Install Samba as described here.

sudo apt install -y samba

sudo vim /etc/samba/smb.conf

[tank]
   comment = ZFS Pool
   path = /tank
   read only = no
   guest only = no
   guest ok = no
   share modes = yes

Turn SMB sharing on:

sudo zfs set sharesmb=on tank

Create Samba user and define password

sudo pdbedit -a will

Find out who owns the media folder

ls -l /tank/media

will@ara:~$ ls -l /tank/media
total 499
drwxrwx--- 335 1420 1420    372 May 18 00:56 movies
-rwxrwx---   1 1420 1420 283648 Sep 18  2017 plexpy.db
drwxrwx---   5 1420 1420     16 Jan 25 19:03 radarr
drwxrwx---   2 1420 1420      2 Feb 18 01:36 security
drwxrwx---   5 1420 1420     16 Jan 25 19:01 sonarr
drwxrwx---  55 1420 1420     57 May  7 13:15 tv

I don't care about UID 1420, but I do want to name the GID 1420 group "media"

sudo groupadd media

sudo groupmod -g 1420 media

Observe changes:

will@ara:~$ ls -l /tank/media
total 499
drwxrwx--- 335 1420 media    372 May 18 00:56 movies
-rwxrwx---   1 1420 media 283648 Sep 18  2017 plexpy.db
drwxrwx---   5 1420 media     16 Jan 25 19:03 radarr
drwxrwx---   2 1420 media      2 Feb 18 01:36 security
drwxrwx---   5 1420 media     16 Jan 25 19:01 sonarr
drwxrwx---  55 1420 media     57 May  7 13:15 tv

Add myself to the media group and restart Samba

sudo adduser will media

sudo systemctl restart smbd nmbd

Configure LXC profile to use macvlan

Run lxc without typing sudo every time:

sudo setfacl -m u:will:rwx ~/.config/lxc

Create and edit the YAML file:

lxc profile create lxcnet

lxc profile edit lxcnet

config:
  environment.TZ: "America/Chicago"
  description: Creates macvlan bridge for LXC containers
  devices:
    eth0:
      name: eth0
      nictype: macvlan
      parent: enp1s0f1
      type: nic
  name: lxcnet
  used_by: []

Create ZFS partition on local SSD

Find disk/partition to be used

sudo fdisk -l

Check lxc version. On 18.04 it's 3.0.0 right now.

lxc info

Start configuration of LXC

sudo lxd init

will@ara:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]: lxd
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]:
Create a new ZFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sdg4
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new network bridge? (yes/no) [default=yes]: no
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: yes
Name of the existing bridge or host interface: enp1s0f1
Is this interface connected to your MAAS server? (yes/no) [default=yes]:
MAAS IPv4 subnet name for this interface (empty for no subnet):
MAAS IPv6 subnet name for this interface (empty for no subnet):
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes

config: {}
cluster: null
networks: []
storage_pools:
- config:
    source: /dev/sdg4
  description: ""
  name: lxd
  driver: zfs
profiles:
- config: {}
  description: ""
  devices:
    eth0:
      name: eth0
      nictype: macvlan
      parent: enp1s0f1
      type: nic
    root:
      path: /
      pool: lxd
      type: disk
  name: default

The outcome of these steps is that the network configuration from the lxcnet profile is copied to the default profile, and the default profile is populated with the ZFS pool information.

You can see this with the following:

lxc profile show default

will@ara:~$ lxc profile show default
config: {}
description: ""
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: enp1s0f1
    type: nic
  root:
    path: /
    pool: lxd
    type: disk
name: default
used_by: []

Increase file and inode limits

sudo vim /etc/sysctl.conf

fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576

sudo vim /etc/security/limits.conf

* soft nofile 100000
* hard nofile 100000

Now reboot:

sudo reboot

Verify host network, LXD network, and ZFS

Observe your primary network interface matches what you set in /etc/netplan/00-netcfg.yaml

ip a

Note that a pool has been created with datasets:

sudo zfs list

Create an LXC container for torrent/vpn

Note, this is an empty Ubuntu 16.04 container. I'm naming it 'torrent'

lxc launch ubuntu:xenial torrent

See that the container has started:

lxc list

Look for the MAC address in the container:

lxc config show --expanded torrent

In my case, I see the following:

volatile.eth0.hwaddr: 00:16:3e:e1:65:36

On my DHCP server, I create a new MAC reservation:

Name: torrent
IP: 10.1.20.12
MAC: 00:16:3e:e1:65:36

Enter the torrent container:

lxc exec torrent bash

Remove the dynamic IP so you can get the static one assigned

ip addr flush dev eth0

rm /var/lib/dhcp/dhclient.eth0.leases

dhclient -r; dhclient

You should see the DHCP-assigned static IP address:

ip a

Now exit the container

exit

Mount host directory into LXC container

Stop it if it's running:

lxc stop torrent

Make it privileged to avoid file ownership issues as noted here:

lxc config set torrent security.privileged true

Mount /tank/downloads to /downloads:

lxc config device add torrent downloads disk source=/tank/downloads path=/downloads

Allow Docker inside LXD container:

lxc config set torrent security.nesting true

Create non-root user and assign privileges

Start the container and enter it:

lxc start torrent

lxc exec torrent bash

Create user in the container and assign permissions:

adduser will

usermod -aG sudo will

groupadd media

adduser will media

usermod -u 1420 will

groupmod -g 1420 media

Log into user and create command so you can run sudo as documented here:

su will

vim ~/.bashrc

alias sudo='sudo -S'

Install the .bashrc:

source ~/.bashrc

Install docker in LXC container

Instructions pulled from here.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt update

sudo apt-get install -y docker-ce

Start it and enable it to start at boot:

sudo systemctl start docker

sudo systemctl enable docker

Let user do docker things without typing sudo:

sudo gpasswd -a will docker

sudo service docker restart

sudo systemctl enable docker

Leave and come back:

exit

su will

Install PIA VPN and torrent docker in LXC

Enter container if you aren't already in it:

lxc exec torrent bash

Prepare host (LXC container) and create torrent config directory:

mkdir -p ~/torrent/config/openvpn

mkdir ~/torrent/openvpn_all

cd ~/torrent/openvpn_all

wget https://www.privateinternetaccess.com/openvpn/openvpn.zip

sudo apt install unzip

unzip openvpn.zip

Copy only the key, cert, and ovpn file you want to use:

cp *.crt ~/torrent/config/openvpn

cp *.pem ~/torrent/config/openvpn

cp US\ Midwest.ovpn ~/torrent/config/openvpn

Create and run torrent docker:

docker run -d \
    --cap-add=NET_ADMIN \
    -p 9080:9080 \
    -p 9443:9443 \
    -p 8118:8118 \
    -p 3000:3000 \
    --name=torrent \
    -v /home/will/torrent/config:/config \
    -v /downloads:/downloads \
    -v /etc/localtime:/etc/localtime:ro \
    -e VPN_ENABLED=yes \
    -e VPN_USER=yourusername \
    -e VPN_PASS=yourpassword \
    -e VPN_PROV=pia \
    -e STRICT_PORT_FORWARD=no \
    -e ENABLE_PRIVOXY=no \
    -e ENABLE_FLOOD=both \
    -e ENABLE_AUTODL_IRSSI=yes \
    -e LAN_NETWORK=10.1.20.0/24 \
    -e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
    -e DEBUG=true \
    -e PHP_TZ=America/Chicago \
    -e UMASK=000 \
    -e PUID=1420 \
    -e PGID=1420 \
    --restart=always \
    binhex/arch-rtorrentvpn

If you need to add a flag on the fly to a running container, here's an example:

docker update --restart=always torrent

Enter a Docker container for troubleshooting:

docker exec -it torrent /bin/bash

Install and set up Docker on host

Install Docker in Ubuntu 18.04:

curl -fsSL test.docker.com | sh

Create macvlan for hosts

docker network create -d macvlan \
--subnet=10.1.20.0/24 \
--gateway=10.1.20.1  \
-o parent=enp1s0f1 mvdock0

Create and run Plex container:

docker run -id \
--name plex \
--network=mvdock0 \
--ip=10.1.20.11 \
-h plex \
-e VERSION=latest \
-e TZ="America/Chicago" \
-e ADVERTISE_IP="http://plex.paw.blue:32400/" \
-e PLEX_UID=1420 -e PLEX_GID=1420 \
-v /tank/plexdata/config:/config \
-v /tank/media/tv:/tv \
-v /tank/media/movies:/movies \
-v /tank/media/education:/education \
-v /tank/transcode:/transcode \
plexinc/pms-docker

Sonarr:

docker run -d \
--name sonarr \
--network=mvdock0 \
--ip=10.1.20.15 \
-p 8989:8989 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-e PUID=1420 -e PGID=1420 \
-e TZ=America/Chicago \
-e DEBUG=false \
-v /etc/localtime:/etc/localtime:ro \
-v /home/will/sonarr/config:/config \
-v /tank/downloads:/downloads \
-v /tank/media/tv:/tv \
--restart=always \
linuxserver/sonarr

Radarr:

docker run -d \
--name=radarr \
--network=mvdock0 \
--ip=10.1.20.16 \
-p 7878:7878 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-e PGID=1420 -e PUID=1420 \
-e TZ=America/Chicago \
-v /etc/localtime:/etc/localtime:ro \
-v /home/will/radarr/config:/config \
-v /tank/downloads:/downloads \
-v /tank/media/movies:/movies \
--restart=always \
linuxserver/radarr

Jackett:

docker run -d \
--name=jackett \
--network=mvdock0 \
--ip=10.1.20.17 \
-p 9117:9117 \
-e NAME_SERVERS=10.1.20.254,10.1.20.253,208.67.222.222,1.1.1.1 \
-v /home/will/jackett/config:/config \
-v /tank/downloads:/downloads \
-e PGID=1420 -e PUID=1420 \
-e TZ=America/Chicago \
-v /etc/localtime:/etc/localtime:ro \
--restart=always \
linuxserver/jackett

Docker Experimental

If you want to use an ipvlan instead of a macvlan in Ubuntu 18.04, you will have to start docker in experimental mode.

Enabling experimental mode:

dockerd --experimental=true

Examples of layer 2 and layer 3 ipvlan networks:

docker network create -d ipvlan \
   --subnet=10.1.20.0/24 \
   -o ipvlan_mode=l3 \
   -o parent=enp1s0f1 ipdock0
docker network create -d ipvlan \
   --experimental=true \
   --subnet=10.1.20.0/24 \
   --gateway=10.1.20.1  \
   -o ipvlan_mode=l2 ipvlan20

FAQ

Q: Why don't you just run the vpn/torrent docker container on the host?

A: I tried to do that with the network=mvdock0 and ip flags, but it wasn't working. I think it has something to do with the way the VPN is influencing the network connection. By making the vpn/torrent docker container use the "host" network and having the "host" actually be an LXC container, I can still ensure this traffic passes through enp1s0f1 on the physical host.

Q: Why not do all LXC or all Docker? Why mix and match?

A: First, so I could learn both. Second, Docker is so easy to get my applications up and running. Third, my inspiration for this project was this amazing post by Jason Bayton, and I loved the idea of hosting LXC containers in ZFS. As it turns out, I only hosted one. But perhaps more soon!

Summary

This took me many hours to assemble, as I had very little LXC or Docker experience before setting out on this journey. I'm sure people will point out many, many flaws in this tutorial. Please comment so that I can fix them!

If this guide helped you, please consider a small crypto donation!

BTC: 3CvPzK2Sjcoi8A5Nkxuh6Djo6xX7udgYa9

LTC: MQBTp7k2c8ec43RAeAkNUu6E4i97dPGA8s

DGB: Sj9Qwky7p2J1pKLwvBNFhpLXk6ATswsDsP