Container Deep Diving: Part 2

Okay, so here we are in part 2 of the container post series.

At the end of part 1 we were able to identify the problems of just using chroot to achieve process isolation on a machine. With this post the goal is to have the same functionality - running bash with test-root as the new root directory with the same technologies as containers. Once that is running we will adress the problems of seeing all the network interfaces as well as still being able to kill arbitrary processes on the machine.

A new root dir

This time around the complexity will increase quite a bit.

While we will ultimately still use chroot the idea is to execute it inside a pre-secured environment. A container. So how can this secure environment be created?

Enter kernel NAMESPACES(7):

       A namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global re‐
       source.  Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes.  One use of namespaces is to  imple‐
       ment containers.
      
        ....
       The  following  table shows the namespace types available on Linux.  The second column of the table shows the flag value that is used to specify the namespace type in various APIs.
       The third column identifies the manual page that provides details on the namespace type.  The last column is a summary of the resources that are isolated by the namespace type.

       Namespace Flag            Page                  Isolates
       Cgroup    CLONE_NEWCGROUP cgroup_namespaces(7)  Cgroup root directory
       IPC       CLONE_NEWIPC    ipc_namespaces(7)     System V IPC, POSIX message queues
       Network   CLONE_NEWNET    network_namespaces(7) Network devices, stacks, ports, etc.
       Mount     CLONE_NEWNS     mount_namespaces(7)   Mount points
       PID       CLONE_NEWPID    pid_namespaces(7)     Process IDs
       Time      CLONE_NEWTIME   time_namespaces(7)    Boot and monotonic clocks
       User      CLONE_NEWUSER   user_namespaces(7)    User and group IDs
       UTS       CLONE_NEWUTS    uts_namespaces(7)     Hostname and NIS domain name

The namespace that we need for the intial step is Mounts. The first of all the namespaces that were introduced to Linux.

According to mount_namespaces(7) this namespace was introduced in Linux 2.4.19 while the fs/namespace.c file seems to be introduced in 2.4.12 by Al Viro and somehow made it into the changelog in 2.4.11. This happended more than 20 years ago and is the base for the modern container stack.

But how exactly can Mount namespaces help us out in restricting a process to a new root directory?

By using a technique called bind-mount. With bind-mounts it is possible to have a directory on the machine show up as a mount point. Basically a portal into a directory which we can then move into our secure environment before starting a chroot in it.

On the root namespace of the machine - every process in a modern linux system is inside a namespace - it is just a normal directory that can be inspected.

So how does this look in commands?:

# Precheck all the available mounts in the root namespace
findmnt

# Step 1: create test-root with bash binary and dependencies
cd /
mkdir -p test-root/{bin,proc,old-root}
cp /bin/bash test-root/bin/bash
cp -a /usr /lib /lib64 test-root

# Step 2: jump into a new namespaced environment -> our secure environment
unshare --mount

# Step 3: create a bind mount for test-root and mount process information
mount --bind test-root test-root


# Step 4: switch the root folder to test-root and keep the old root mounted at old-root
## also mount the process information via the procfs after switching the root
cd test-root
pivot_root . old-root
mount -t proc proc /proc

# Step 5: unmount the old root from the secure environment so that only the new root is available
## `--lazy` is needed as our /bin/bash process is still attached to the old mount
## if we made the namespace persistent and executed a process in it later, then the mount will not available in it
umount --lazy old-root

# Step 6: chroot into the new secure environment
exec chroot . /bin/bash

Here is the process as a small video:

Nice, the first step is done and we are on par with part 1. But now lets reap some benefits when we get to the unsolved isolation issues.

Processes

When creating the secure environment the binary that actually did the namespace switching is unshare.

In Step 2 it is called with --mount, which will only create a new Mount namespace. To further restrict the environment we can now add the PID namespace, which separates all the processes into a new namespace of their own.

At this point it is important to note that the information for PID and Mount namespaces are stored in separate structures inside the kernel. This applies to all namespaces and means that you can mix and match them as you desire, giving the environments the exact configurations that you want.

For our example this means that we can simply continue after the last command before entering the chroot.

To create the new PID namespace lets simply call unshare again but use the --pid flag this time.

Hm, so what is the issue here? Lets go digging.

UNSHARE(1)
    -p, --pid[=file]
           Unshare the PID namespace. If file is specified, then a persistent namespace is created by a bind mount. (Creation of a persistent PID namespace will fail if the --fork option
           is not also specified.)

           See also the --fork and --mount-proc options.
           
           ....
           
    -f, --fork
           Fork the specified program as a child process of unshare rather than running it directly. This is useful when creating a new PID namespace. Note that when unshare is waiting
           for the child process, then it ignores SIGINT and SIGTERM and does not forward any signals to the child. It is necessary to send signals to the child process.
           
           ....
           
    --mount-proc[=mountpoint]
           Just before running the program, mount the proc filesystem at mountpoint (default is /proc). This is useful when creating a new PID namespace. It also implies creating a new
           mount namespace since the /proc mount would otherwise mess up existing programs on the system. The new proc filesystem is explicitly mounted as private (with
           MS_PRIVATE|MS_REC).



--mount-proc is not relevant for us as we have already created a new Mount namespace and do not want another one. But more on that in a second. The fix is in the --fork flag.

So why is this?

Checking the man page of pid_namespaces(7) gives the answer:

   The namespace init process
       The  first  process  created  in  a  new namespace (i.e., the process created using clone(2) with the CLONE_NEWPID flag, or the first child created by a process after a call to un‐
       share(2) using the CLONE_NEWPID flag) has the PID 1, and is the "init" process for the namespace (see init(1)).  This process becomes the parent of any child processes that are or‐
       phaned because a process that resides in this PID namespace terminated (see below for further details).

The important part is or the first child created by a process after a call to unshare(2).

The first process that was called after unshare is ls. So ls will execute in the new namespace and then exit. That exit will trigger the new namespace to be deleted.

In order to get the correct behaviour we want bash to be forked when calling unshare, therefore making it our new parent process in the namespace.

So the correct way is to run unshare --pid --fork:

But at this point there is still one thing missing and that relates to the --mount-proc flag mentioned above.

When checking the active processes with ps -aux the following is printed on a Fedora Workstation:

-bash-5.1# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
0              1  0.7  0.8 171724 16900 ?        Ss   11:35   0:01 /usr/lib/systemd/systemd rhgb --switched-root --sys
 
...........
 
0            156  0.0  0.0      0     0 ?        I<   11:35   0:00 [ipv6_addrconf]
0            157  0.0  0.0      0     0 ?        I    11:35   0:00 [kworker/u8:8]
0            701  0.0  0.6 393636 13424 ?        Ssl  11:35   0:00 /usr/libexec/udisks2/udisksd
983          703  0.0  0.1  84648  3676 ?        S    11:35   0:00 /usr/sbin/chronyd -F 2
0            705  0.0  0.4 238196  8248 ?        Ssl  11:35   0:00 /usr/libexec/upowerd
0            706  0.0  0.7 255924 14712 ?        Ssl  11:35   0:00 /usr/sbin/abrtd -d -s
70           708  0.0  0.0   8516   356 ?        S    11:35   0:00 avahi-daemon: chroot helper
1000        1351  0.6  0.5 310964 10624 ?        Sl   11:35   0:01 ibus-daemon --panel disable -r --xim
1000        1353  0.1  3.5 581004 71036 ?        Ssl  11:35   0:00 /usr/libexec/gsd-xsettings
 
...........

0           1798  0.0  0.4  18684  9744 ?        Ss   11:37   0:00 /usr/lib/systemd/systemd-hostnamed
0           1822  0.0  0.2   8804  5684 ?        S    11:38   0:00 -bash
0           1848  0.0  0.2   8804  5720 ?        S    11:38   0:00 -bash
0           1877  0.2  0.2   8804  5680 ?        S    11:38   0:00 -bash
0           1904  0.2  0.2   8804  5656 ?        S    11:38   0:00 -bash
0           1933  0.0  0.0   5580   252 ?        S    11:39   0:00 unshare --pid --fork
0           1934  0.0  0.1   7380  3984 ?        S    11:39   0:00 -bash
0           1935  0.0  0.1   9888  2368 ?        R+   11:39   0:00 ps -aux

All processes are displayed.

Why does this happen? Because we mounted /procfs in the previous PID namespace.

Skipping the --mount-proc flag leaves us in the desired Mount namespace but also does not automatically mount the /procfs that belongs to our new PID namespace. Simply unmounting the old /procfs and mounting the new one with umount /proc && mount -t proc proc /proc will fix this and voila, ps -aux now prints:

-bash-5.1# ps -aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
0              1  0.0  0.2   7380  4088 ?        S    11:39   0:00 -bash
0              5  0.0  0.1   9888  2380 ?        R+   11:41   0:00 ps -aux

and we will not be able to kill arbitrary processes anymore.

Now there is only 1 thing left to address from part 1.

Network

A quick man unshare reveals:

       -n, --net[=file]
           Unshare the network namespace. If file is specified, then a persistent namespace is created by a bind mount.

And thats it. We are now in a completely isolated environment to run our program, notably without any network access.

Here are all the commands again:

# Step 1: create test-root with bash binary and dependencies
cd /
mkdir -p test-root/{bin,proc,old-root}
cp /bin/bash test-root/bin/bash
cp -a /usr /lib /lib64 test-root

# Step 2: jump into a new namespaced environment -> our secure environment
unshare --mount

# Step 3: create a bind mount for test-root
mount --bind test-root test-root

# Step 4: switch the root folder to test-root and keep the old root mounted at old-root
## also mount the process information via the procfs after switching the root
cd test-root
pivot_root . old-root
mount -t proc proc /proc

# Step 5: unmount the old root from the secure environment so that only the new root is available
## `--lazy` is needed as our /bin/bash process is still attached to the old mount
umount --lazy old-root

# Step 6: enter a new PID namespace
unshare --pid --fork

# Step 7: enter a new network namespace
unshare --net

# Step 8: chroot into the new secure environment
exec chroot . /bin/bash

Recap

Lets look at the environment we have now, a folder mounted as the root, a new process hierarchy and an isolated network:

Quite a few moving parts, but really elegant when they work together like this and become the modern container.

What now?

The current Linux Kernel 5.16 has 8 namespaces and this post only covered 3 of them. The User and Cgroup are especially important for modern containers.

While the first one can map unprivileged users from the root namespace to the root user inside a namespace, which is really important for a safe container, the Cgroup namespace can manage the amount of ressources the namespace will get (RAM and CPU for example). You should definitely try extending the example from this post with those two. Check man namespaces for all the documentation needed for this.

As far as this series of posts is concerned the basics are now covered. Part 3 is going to dig into the current ecosystem. Think podman vs docker.

Until then, happy hacking!.

Back to overview