Skip to content

How Docker Actually Works: I Rebuilt a Container From Scratch With Linux Commands

No Docker, just Linux: build a root filesystem with debootstrap, isolate it with unshare, wire it to the internet with a veth pair and cap it with cgroups.

Prefer to watch? This post is the written companion to the video above.

In Part 1 we landed on one line: a container is an image, plus namespaces, plus cgroups. The image is just files. Namespaces and cgroups are plain Linux features.

If that is really true, we should be able to build a container ourselves with nothing but Linux commands. No Docker, no runc. So that is what we will do: build a small Debian root filesystem, isolate it, give it its own network, run a Node.js server inside it, cap its CPU and memory, and curl it from the host.

All scripts are in the container-from-scratch repo on GitHub.

web page

Every script in this post comes from this repo, at commit 94194e3.

Cloudmash333/container-from-scratch on GitHub

Here is the plan. Each step adds one piece: files (debootstrap), a jail (chroot), isolation (unshare and namespaces), a network (a network namespace, a veth pair and NAT), mounts (/proc, /sys, /tmp) and a limit (a cgroup v2 group for CPU and memory).

The build plan. Every step is a plain Linux command, and every step matches something runc does for Docker.

Step 1: a root filesystem with debootstrap

A container needs its own files. Docker gets them from an image. We will make our own with debootstrap, a tool that downloads a minimal Debian system into a folder.

sudo apt install debootstrap
mkdir containers
sudo debootstrap --variant=minbase stable ./containers http://deb.debian.org/debian/

It needs root and a target folder. When it finishes, ./containers holds a full Linux tree: bin, lib, usr, etc and so on. Check it: the host's /etc/os-release says Ubuntu 24.04, but ./containers/etc/os-release says Debian. Two different Linux distributions on one disk, both using the same kernel.

Step 2: jail it with chroot

sudo chroot ./containers /bin/bash

Now / means ./containers. Every path the shell opens is resolved inside that folder. That is the "jail" from Part 1.

But it is a very bare jail. minbase has no ps, no ping, no editor. Install what we need:

apt-get install iputils-ping procps nano nodejs iproute2 -y

iputils-ping to test the network, procps for ps, nano to edit files, nodejs for our app, and iproute2 for the ip command we need for networking.

debootstrap downloads a minimal Debian root filesystem into a folder, and chroot makes that folder the shell's /.

chroot only changes the files the process sees. It still shares processes, network and hostname with the host. That is what namespaces fix next.

Step 3: isolate it with unshare

unshare starts a program in new namespaces. We ask for almost all of them:

sudo unshare --mount --pid --uts --net --ipc --cgroup --fork /bin/bash

Each flag maps to a namespace from Part 1: mount, PID, hostname (UTS), network, IPC and cgroup.

web page

The full list of Linux namespaces. Our unshare call uses six of them; Time and User are left out.

namespaces(7) on man7.org

Why --fork? A new PID namespace does not move the process that creates it. Only its children go inside. Without --fork, unshare creates the namespace and then runs bash itself, so the shell stays in the old PID namespace with its old PID. With --fork, unshare forks a child that execs bash, and that child is PID 1 in the new namespace. Watch this part (06:41)

Why unshare needs --fork: only children enter a new PID namespace, so the forked bash becomes PID 1 inside it.

Now try the network:

ping -c 3 google.com
# ping: google.com: Temporary failure in name resolution

That failure is good news. A brand new network namespace has no interfaces except a loopback that is down. It is fully cut off. We have to wire it up ourselves.

Processes can talk through IPC: System V shared memory, message queues and semaphores, or POSIX shared memory under /dev/shm. On the host, create a 64 MB shared memory segment:

ipcmk -M 64M      # Shared memory id: 1
ipcs -m           # host lists it

Inside our container (started with --ipc), ipcs -m shows nothing. Different IPC namespace, different set of segments. Clean up on the host with ipcrm -m 1. (The ID depends on what already exists on your machine; the first one is usually 0.) Watch this part (07:33)

An IPC namespace hides the host's shared memory segments: the host lists the 64 MB segment, the container lists nothing.
from video content

The same demo from the video: two processes sharing one 64 MB segment on the host, the commands to create, list and remove it, and the unshare --ipc shell that cannot see it.

Step 4: give it a network (veth pair + NAT)

We create a named network namespace and connect it to the host with a veth pair: two virtual network interfaces joined like the two ends of a cable. A packet that goes in one end comes out the other.

Here is the whole script:

create_network.sh repo, commit 94194e3
# 1. Create a new network namespace
ip netns add mynetworkns

# 2. Create the veth pair in the default namespace
ip link add systemeth type veth peer name containereth

# 3. Move one end of the veth pair (containereth) into the new namespace
ip link set containereth netns mynetworkns

# 4. Assign an IP and bring up the interface on the host (systemeth)
ip addr add 10.0.0.1/24 dev systemeth
ip link set systemeth up

# 5. Assign an IP and bring up the interface inside the new namespace (containereth)
ip netns exec mynetworkns ip addr add 10.0.0.2/24 dev containereth
ip netns exec mynetworkns ip link set containereth up

# 6. Bring up the loopback interface in the new namespace
ip netns exec mynetworkns ip link set lo up

# 7. Enable IP forwarding on the host
sysctl -w net.ipv4.ip_forward=1

# 8. Add a NAT rule so the namespace can access the internet
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE

# 9. Add a default route inside the namespace, pointing to the host's veth end
ip netns exec mynetworkns ip route add default via 10.0.0.1

# 10. Test connectivity from inside the new namespace
ip netns exec mynetworkns ping -c 3 8.8.8.8

What each part does, in order:

  1. ip netns add mynetworkns: a new, empty network namespace.
  2. ip link add systemeth type veth peer name containereth: the cable. Both ends start on the host.
  3. ip link set containereth netns mynetworkns: push one end into the namespace.
  4. The host end gets 10.0.0.1/24 and is brought up.
  5. The container end gets 10.0.0.2/24 and is brought up, plus the loopback lo.
  6. sysctl -w net.ipv4.ip_forward=1: let the host route packets that are not for itself.
  7. iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE: packets from 10.0.0.x leave the host looking like they came from the host's own IP (NAT), so replies find their way back.
  8. Default route inside the namespace: via 10.0.0.1. Anything not local goes through the host end of the cable.
  9. ping -c 3 8.8.8.8 from inside the namespace to test.
from video content

Step 4 as I drew it in the video: a packet leaves containereth (10.0.0.2) in mynetworkns and arrives at systemeth (10.0.0.1) on the host.

Run it and enter the namespace, jailed in our root filesystem:

sudo bash create_network.sh
sudo ip netns exec mynetworkns chroot ./containers /bin/bash

ping 8.8.8.8 now works. ping google.com still fails, because the container's /etc/resolv.conf was copied from the build machine and points at a resolver it cannot reach. Give it a public DNS server, and set a hostname while we are here (this only changes the container's UTS namespace):

echo 'nameserver 8.8.8.8' > /etc/resolv.conf
hostname cloudmash-container
ping -c 3 google.com    # 3 packets transmitted, 3 received

Step 5: mount /proc, /sys and /tmp

mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t tmpfs tmpfs /tmp

These three lines confused a lot of viewers, so here is what they do.

/proc is a virtual filesystem. Nothing in it is stored on disk. It is an interface between user space and the kernel. When you read a file in /proc, the kernel generates the content right then from its live data structures.

from video content

/proc from the video: a virtual filesystem, an interface to kernel data that is generated when you read it.

You can see it yourself. cat /proc/uptime twice and the numbers change. ls -l /proc/uptime says the size is 0, yet it printed data. ls /proc shows numbered folders: those are process IDs, created on the fly by the kernel.

from video content

What is inside /proc, and who answers when you read it: not files on disk, but views into kernel memory, generated on every read.

And here is why it matters for containers: procfs shows the processes of the PID namespace of whoever mounted it. Mount it inside the container and ps shows PIDs starting from 1, only the container's processes, while the host sees the same processes as 2873 and up.

The first word after -t is the filesystem type. The second (proc, sys, tmpfs) is just a label: you can write any name there and it still works. /sys (sysfs) does the same for devices and kernel objects, and tmpfs gives the container its own in-memory /tmp.

from video content

The mount command taken apart: the type, a label that can be any string, and the mount point.

/proc is a window into kernel data, not files on disk. The uptime values and PIDs shown are illustrative, not captured output.

Part 3: I answer your doubts on mounts, --fork and IPC. The video starts at the mounts chapter (0:39).

Try it: run an app inside

Create server.js inside the container (nano is installed now):

server.js reconstructed from the video demo, not in the repo
// Minimal server used in the video (reconstructed from the demo; not in the repo).
// Run inside the container:  node server.js   then from the host: curl 10.0.0.2:3000
const http = require("http");

http
  .createServer((req, res) => {
    res.end("Hello, this response is from container\n");
  })
  .listen(3000, "0.0.0.0", () => console.log("listening on :3000"));
node server.js

ip addr inside shows lo and containereth with 10.0.0.2. On the host, ip addr shows systemeth with 10.0.0.1. From the host:

curl 10.0.0.2:3000
# Hello, this response is from container

Our host just talked to a server living in its own filesystem, its own namespaces and its own network.

The finished container: curl on the host goes through systemeth and containereth and reaches node server.js inside.

Step 6: cap CPU and memory with a cgroup

On cgroup v2 a cgroup is a folder under /sys/fs/cgroup, limits are files inside it, and you join a process by writing its PID into cgroup.procs. Run this on the host, not in the container.

attach_cgroup.sh repo, commit 94194e3
# Setup Cgroups
# Go to HOST system (your main system,outside container), this setting you have to do in host system
cd /sys/fs/cgroup
sudo mkdir mycgroup
echo "+cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo "50000 100000" | sudo tee /sys/fs/cgroup/mycgroup/cpu.max
echo "500000000" | sudo tee /sys/fs/cgroup/mycgroup/memory.max

ps aux | grep "mynetworkns"  #Get process id of 2nd networkns process , because first is sudo process and last one is grep itself
echo '<your process id here>' > /sys/fs/cgroup/mycgroup/cgroup.procs # Add process ID of network namespace process here
#verify whether cgroup changed to mycgroup for mynetworkns process
cat /proc/<id of process>/cgroup # this should show something like 0::mycgroup

Line by line:

  • +cpu into cgroup.subtree_control turns on the CPU controller for child groups.
  • cpu.max = 50000 100000: 50,000 microseconds of CPU in every 100,000. That is 50% of one CPU.
  • memory.max = 500000000 bytes, about 500 MB. Above that the OOM killer steps in.
  • Find the PID with ps aux | grep mynetworkns. The first line is the sudo process and the last one is grep itself; take the one in between.
  • Write it into cgroup.procs, then verify with cat /proc/<pid>/cgroup. It should now show 0::/mycgroup.
A cgroup is a folder, limits are files, and membership is a PID written into cgroup.procs.

That is a full container: its own root filesystem, its own namespaces, its own network and a resource fence. Exactly what runc does for Docker, by hand.

Clean up

Exit the shells, then on the host run one command from the repo folder:

sudo bash revert.sh

revert.sh turns the cpu controller back off, removes mycgroup, and then calls delete_networkns.sh for you:

revert.sh repo, commit 94194e3
# On your host(outside container)
#Revert cgroup changes
echo "-cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
sudo rmdir /sys/fs/cgroup/mycgroup

#delete network namespace
bash delete_networkns.sh

delete_networkns.sh removes the route, the NAT rule, IP forwarding, the veth pair and the namespace:

delete_networkns.sh repo, commit 94194e3
# Remove the default route in the network namespace
ip netns exec mynetworkns ip route del default via 10.0.0.1

# Delete the masquerade rule from iptables
iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -j MASQUERADE

# Reset IP forwarding (if it was originally disabled)
# If you are unsure, it's safer to leave it as is or check its initial state.
sysctl -w net.ipv4.ip_forward=0

#remove ip addr from the host systemeth interface
ip addr del 10.0.0.1/24 dev systemeth

# Bring down and delete the systemeth veth interface
ip link set systemeth down
ip link del systemeth

# Delete the network namespace
ip netns del mynetworkns

Going deeper

The same steps, mapped to what Docker and runc do:

By handDocker / runc
debootstrap into ./containersdocker pull + unpacked layers
chroot ./containerspivot_root into the rootfs
unshare --mount --pid --uts --net --ipc --cgroup --forkrunc clone() with CLONE_NEW* flags
create_network.sh (veth + NAT)docker0 bridge + veth + iptables
echo 'nameserver ...' > /etc/resolv.confDocker writes resolv.conf for you
mount -t proc proc /procrunc mounts /proc, /sys, /dev
/sys/fs/cgroup/mycgroupdocker run --cpus --memory

What's next

If Part 1 was the why, this was the how. Star the repo, try it in a VM, and tell me in the comments which Linux internals you want next.

Once one container makes sense, the next question is how many of them fit into a real system. I cover how containers and Kubernetes fit into system design on the dev site.