I'm a software consultant. That means I get to see lots of different projects, most with their own unique technology stack and required toolset. That also means that all those tools required to develop and run a project needs to be installed somewhere.

Some customers require me to use a computer provided by them in which case this isn't an issue - once I'm finished with the project I hand back the computer and forget all about which tools were installed. Some customers however either doesn't provide a computer or allow me to bring my own (something I do prefer since I've set up my computer just the way I want it). In this case, said tools may eventually clutter up my computer or even cause conflicts with each other.

I run Arch Linux. I do this because I like the control it gives me. Arch Linux is a rolling release distribution, this means that updates happen often and most package versions follow upstream release cadence. This is in contrast to something like Debian or Fedora which release a set of packages and then typically only release minor upgrades for security- and bugfixes until the next OS release which will contain major version upgrades.

Because most packages in Arch are updated shortly after a new version has been released upstream it may sometimes be difficult (or even impossible) to install an older version of something.

Let's take Java as an example. At the time of writing this you can install:

Java 8
Java 11
Java 17
Java 18

Working on a project that is using Java 13? Tough luck.

Now, I can always download OpenJDK myself and install it to /opt, but do that often enough and eventually /opt starts to become rather cluttered. Additionally, I may not remember which project depends on which version, making it risky to uninstall something. Not to mention the hassle of switching between different versions of Java for different projects.

Running Projects in Isolation

So, to the point of this post. I decided I wanted to split up my computer into separate, isolated parts - one for me and my own stuff and one for each customer or project.

To do this I'm going to use containers. Most of you probably think of Docker when you hear the word container and while Docker has its uses it's built for running a single application and I want a complete environment - possibly to run Docker containers in - so I'm not going to use that.

Now, I could go with a full-blown VM via KVM or Xen, install Arch Linux there and have a completely isolated operating system. This does feel a bit like overkill though so being Swedish I'm going to go the lagom route.

LXC

Docker was initially (pre version 0.9) built on LXC, which is what I'm going to use (well, sort of). LXC is a container system that can run multiple isolated Linux systems on the same host - much like Docker. Unlike Docker however these are complete Linux systems with an init system and their own services. You can essentially treat them as lightweight VMs. Another important difference between Docker and LXC is that LXC containers can be run unprivileged. This means that they are run with fake user and group IDs - i.e. root in a container is not root on the host system.

LXC is not the most straightforward system to administer though, so instead of using LXC directly I'm going to use LXD, which is built on top of LXC and essentially acts as a frontend.

LXD

LXD consists of two parts: The LXD daemon - called lxd, and the LXD client - confusingly called lxc. You start lxd via your init system and then use lxc to administer your containers.

Setting Up

I'm not going to go through all steps of setting up LXD here, there are better places for that which are hopefully kept up to date as well. Here's the installation instructions for Arch Linux which I followed. If you run something else, refer to the official documentation.

The rest of this post assumes that you've set up LXD to run unprivileged containers (which is the recommended way).

One thing worth noting here is that I ran into issues with the network for my containers. This turned out to be because LXD and Firewalld didn't really get along. There is a section on this in the official guide. I did not get that to work however so instead I created a new zone for LXD and put lxdbr0 in that.

$ firewall-cmd --permanent --new-zone=lxd
$ firewall-cmd --permanent --zone=lxd --add-forward
$ firewall-cmd --permanent --zone=lxd --set-target ACCEPT
$ firewall-cmd --permanent --zone=lxd --change-interface=lxdbr0 --permanent

If you're not using Firewalld this will probably not be an issue.

Creating Containers

Now, with LXD installed we can create our first container.

$ lxc launch images:archlinux archlinux

This will download the Arch Linux image and create a new container named archlinux. You can enter the new container using

$ lxc exec archlinux /bin/bash

This gives you a root shell that you can use to set up the VM, create a regular user and install whatever packages you need.

Accessing Host Files

Now that you have your container, you probably want to access some files from the host in the container. With LXD you do this by creating devices that mount a host directory inside the container.

Here's how you mount a host directory inside the container so that you can access your ssh key and configuration, let's mount my Documents directory inside the container at the same location as on the host. Let's call the mount raniz-documents:

$ lxc config device add archlinux raniz-documents disk source=/home/raniz/Documents path=/home/raniz/Documents

Now, let's try to create a file in the Documents directory.

$ lxc exec archlinux -- sudo -u raniz touch ~/Documents/test
touch: cannot touch '/home/raniz/Documents/test': Permission denied

That didn't work. Let's try doing it as root instead:

$ lxc exec archlinux -- touch /home/raniz/Documents/test
touch: cannot touch '/home/raniz/Documents/test': Permission denied

Still no luck. The "problem" here is that our container is unprivileged. If we execute a command that stalls for a while (i.e. sleep 10s) and then check the active processes from another terminal we can see which user the command is running as. Here's the output of $ ps -ef | grep "sleep 10s" while running $ lxc exec archlinux sleep 10s in another terminal:

raniz    3878475   20661  0 07:36 pts/2    00:00:00 lxc exec archlinux sleep 10s
root     3878484   22304  0 07:36 ?        00:00:00 /usr/bin/lxd forkexec archlinux /var/lib/lxd/containers /var/log/lxd/archlinux/lxc.conf  0 0 0 -- env HOME=/root USER=root LANG=C.UTF-8 TERM=tmux-256color PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -- cmd sleep 10s
100000   3878491 3878484  0 07:36 pts/1    00:00:00 sleep 10s

The first row contains my invocation of lxc exec archlinux sleep 10s. The second row is the command that the LXD daemon is executing to run the command in the container. And the third row is the actual command running inside the container.

Note that my command shows up as my user (raniz), the lxd command is running as root, but the command in the container is running as UID 100000, which is a user that doesn't exist on the host. If we run id inside the container it will tell us that our UID is 0 (root), but outside the container it is actually 100000. This explains why not even the root user of the container is allowed to create files in my Documents directory - because it's actually not root but rather user 100000, which has no permissions to my Documents directory. The same goes for the raniz user inside the container, but with effective UID 101000 instead.

To allow our user inside the container to access files on devices originating outside the container we have two options:

Create a group on the host with a GID that exists inside the container
Map the user on the host to the user inside the container

Creating a Group on the Host

This is pretty straightforward: we create a new group on the host with a GID that exists inside the container and grant this group the permissions we want.

In our Arch Linux container, the users group has GID 984, this means that its effective GID on the host will be 100984. If we create that group and allow it to write to the Documents directory we should be fine.

$ sudo groupadd -g 100984 lxdusers
$ sudo usermod -a -G lxdusers raniz

You can then either change group ownership of the directory to lxdusers, or you can use ACLs:

Changing ownership:

$ chown :lxdusers Documents

(if the above command doesn't work it's probably because your group memberships haven't updated in your current shell. Either log out and in again or prefix the command with sudo -u $USER to run it in a new shell).

Using ACLs:

$ setfacl -m "g:lxdusers:rwx" Documents

The raniz user can now create files in Documents inside our container:

$ lxc exec archlinux -- sudo -u raniz touch ~/Documents/test

Mapping the User or Group Inside the Container

The previous approach works, but produces somewhat messy results because files created inside the container will be owned by the effective UID and GID, so if we look at the permissions of the newly created file we get this:

$ stat -c "%A %U (%u), %G (%g)" ~/Documents/test
-rw-r--r-- UNKNOWN (101000), lxdusers (100984)

This now means that our host user doesn't really have ownership of files created inside the container. For me, that is a problem and the solution to that is to make the UID of the user inside the container be the same as that of the host user.

To do this we need to do two things:

Allow LXD to map users from the host to the container
Tell LXD how to map users from the host to the container

Granting LXD Permissions to Map Users and Groups

The files /etc/subuid and /etc/subgid which you probably touched when setting up LXD tells the kernel who is allowed to map which UIDs and GIDs. The information we put in here controls which effective UIDs and GIDs LXD are allowed to use.

To allow LXD to map our UID and our GID to the container we need to add them to this file:

$ echo "root:$(id -u):1" | sudo tee -a /etc/subuid
$ echo "root:$(id -g):1" | sudo tee -a /etc/subgid

This allows LXD to map our UID and GID (1000 and 984 in my case) in our containers. The next step is to tell LXD how to do this.

Telling LXD to Map Users and Groups

We can tell LXD to map an UID on the host to an UID inside the container and vice versa with the raw.idmap configuration key. It works like this:

uid <host UID> <container UID>
gid <host GID> <container GID>

For my Arch Linux container running on my Arch Linux host, they are the same both inside and outside the container (1000 and 984), but to generalize we can call id both inside and outside the container to get the correct values.

$ h_uid=$(id -u)
$ c_uid=$(lxc exec archlinux -- id -u raniz)
$ h_gid=$(id -g)
$ c_gid=$(lxc exec archlinux -- id -g raniz)
$ echo "uid ${h_uid} ${c_uid}\ngid ${h_gid} ${c_gid}" | lxc config set archlinux raw.idmap -
$ lxc restart archlinux

If we now create a new file in our Documents directory and inspect it, we'll get:

$ lxc exec archlinux -- sudo -u raniz touch ~/Documents/test2
$ stat -c "%A %U (%u), %G (%g)" ~/Documents/test2
-rw-r--r-- raniz (1000), users (984)

And this means that our user inside the container now has the same effective UID as our user on the host.

Now, this means that our container is a little less secure since the user inside the container has the exact same privileges as the user outside the container. With ACLs and a group on the host you get more fine-grained control over what the user inside the container can access.

This isn't a problem for me because I don't use containers for permission control but rather to sandbox project tooling and as a way to run specific Linux distributions if I need to. It's also a good way to quickly clean up once I'm finished with a project because then I can just delete the entire container and not worry about any leftover packages installed on my system.

Next Steps

Now that we're up and running and can manipulate the host filesystem, the next step is to make the container a bit more usable.

That is too much for this post though, so in the next one I will show how to enable graphical applications, sound, SSH access, DNS and make setting up new containers a bit more ergonomic with LXD profiles.