Custom Proxmox VE LXC Images

Using Linux Containers can significantly improve deployment times to make customized instances of a system. For instance, you may want to create 10 unique instances of a system for training purposes, but don’t want to run custom code on each when it starts to generate key material, assign users, etc. Using Linux Containers can make that simple, but unfortunately it’s not always so simple to create that custom image for deployment. This post is going to cover the start to finish customization of an image (in this case, using Kali Linux) from the base image to one that can deploy in a non-privileged virtualization platform (in this case, Proxmox VE). Let’s get started!

To start, you’re going to need an environment on which you can build the image. Ubuntu makes this pretty straightforward, so we’ll use that for this tutorial. If you don’t know how to setup an Ubuntu environment feel free to check out my guide on creating a dual-boot system to get started: Hermit’s Hardware Hacking Box. You can just stop after installing Ubuntu if you want to speed things up for this guide.

Building the Subsystem

Before we can begin building custom images we need to install the base components that will let us do everything. In particular, we need to install the lxd and lxc components (and their associated utilities):

user@system$ sudo apt update
user@system$ sudo apt -y install lxc lxctl lxc-templates lxc-utils
user@system$ sudo snap install lxd

That gets us the base components we need, but we’re going to need to setup the underlying infrastructure that will be used. That’s configured via ‘lxd’ as follows:

user@system$ sudo lxd init

A few notes on the selections here:

• Clustering shouldn’t be needed for a single system, so you can answer “no” to that (default)
• MAAS also shouldn’t be needed for a single system, so you can answer “no” here as well (default)
• For the network bridge just let lxd create it for you (e.g. “lxdbr0” for “lxd bridge 0”)
• For the storage pool don’t use a block device, and instead create a loop-backed pool (should be default) using ZFS… and make sure you make note of the pool name, as you’ll need that later on (for this example I’ll use “lxc-pool” as the name)
• Don’t enable network access (don’t worry, you can change this when you build containers from the image later)
• Enable automatic image updates (to make future customized image builds faster)
• Don’t display a YAML version of the configuration (unless you just really want to see it)

Once that is done you should have a plain but configured lxd subsystem to build upon.

Initial Build

To create our image we’re going to leverage the work that has been done by others and get a functional base Kali image.

user@system$ lxc launch images:kali/current/amd64 kali-plain

That command tells lxc to connect to the standard lxc image store and pull down the 64-bit current Kali Linux image, and launch it with the name “kali-plain”. Once that completes we can interact with the instance in several ways, but two in particular at this point.

The “lxc exec” command effectively runs commands inside the instance (as root) and sends the information back out to us, while the “lxc console” command lets us jump into the instance from a console perspective and operate natively by logging in. Each has their own benefits, but in the case of the console access we have to have an account on the system to start, so let’s configure that to begin.

user@system$ lxc exec kali-plain -- adduser kaliuser
user@system$ lxc exec kali-plain -- usermod -aG sudo kaliuser
user@system$ lxc exec kali-plain -- sed -i '1 i\TERM=xterm-256color' /home/kaliuser/.bashrc

Let’s look at each in a bit more depth. First and foremost, each of those commands starts with “lxc exec kali-plain –” which seems a bit odd. The “lxc exec” portion lets us pass commands directly to the instance specified next (“kali-plain” in this case), then the “–” tells the shell to consider everything else as not being a command option (useful if we want to pass a configuration in this execution, such as “apt –update”). Everything after that “–” is basically run in a root shell. So, that means that we…

• Add a new user (“kaliuser”)
• Add the new user to the “sudo” group (so they can run things as root)
• Insert a line at the start of the user’s .bashrc file to specify the user of 256 color terminal emulation (in short, make it pretty)

Now we can grab console access to continue!

user@system$ lxc console kali-plain

Note that you may have to press the Enter key to get things working, but then you can log in as the “kaliuser” user account with the password setup in the “adduser” command. Once you’ve done that, proceed to update/configure as needed/necessary, doing things such as (perhaps):

kaliuser@kali-plain$ sudo apt update
kaliuser@kali-plain$ sudo apt upgrade
kaliuser@kali-plain$ sudo apt install git
kaliuser@kali-plain$ git clone https://github.com/hermit-hacker/ctf-tools.git

Once you’ve finished all desired installations, upgrades, and configurations log out of the user and exit the console by pressing “CTRL+a” then the “q” key. It’s time to get to a stable state to prepare for image creation.

After exiting the console first shut down the container that’s running and take a snapshot:

user@system$ lxc stop kali-plain
user@system$ lxc snapshot kali-plain configured-state

The second command just creates a snapshot of “kali-plain” named “configured state” in case we need to restore to this later.

Prepping the Image

For the image to work properly, we need to make sure that it is an unprivileged image. There are two key aspects of this: (1) we need to capture the files and settings in the image without overlapping the valid users on the host system, and (2) we need to ensure that privileged actions like creating devices can’t originate from within the image. The first one is a big one, so let’s talk about that.

In short, there are certain well known or assumed id numbers associated with certain accounts and groups (e.g. “root” has UID of “0”). That needs to stay true inside the container, but it could conflict with the system outside the container! If a threat actor were to escape from a container where they had UID 0 and land on the host, they’d pick up root access… not a great option. The way this is avoided in Linux Containers is by shifting the id numbers to a high range where they either won’t conflict with a privileged account or will be likely not to have any conflict at all. The implementation of this involves namespaces (read more here if you’d like more detail), and we have to account for that when we build the image.

First, we need to get into the appropriate namespace for our lxd process, and specifically the mount namespace. We can do that using the “nsenter” command as follows:

user@system$ sudo nsenter -t $(cat /var/snap/lxd/common/lxd.pid) -m

The “-m” means to enter the mount namespace, and the “-t” specifies the target process. We could also do this just by entering the process ID directly, but the above is an easier route as the .pid file will always contain the appropriate process ID. There shouldn’t be any output when the command runs successfully.

Now we need to navigate to where the files are actually located. Remember that pool name (e.g. “lxc-pool”) you noted early on in the configuration? It’s time to bring it back and navigate!

root@system# cd /var/snap/lxd/common/lxd/storage-pools/lxc-pool/containers
root@system# cd kali-plain/rootfs

As an aside, if you were to visit this same location without having used nsenter you would see no contents there, because the mounting is only within this namespace.

We’re almost ready… the only thing left is that comment earlier about how unprivileged containers can’t create devices (amongst other things), so we need to capture this filesystem without those aspects that would cause such creation. As lxc uses a gzipped TAR file as the storage format, we can create one directly that will be acceptable.

root@system# tar --exclude=dev --exclude=sys --exclude=proc -czvf /home/user/kali-plain.tar.gz ./

Once that is complete just upload the kali-plain.tar.gz file to Proxmox as a template. You’re ready to create… with one final caveat. 🙂 As we manually created the unprivileged image anything created from this should have the “unprivileged” checkbox not set. Yeah, it doesn’t make a lot of sense at first glance, but that’s the way it works.

Until next time, good hunting!