How Docker Actually Works: I Rebuilt a Container From Scratch With Linux Commands
No Docker, just Linux: build a root filesystem with debootstrap, isolate it with unshare, wire it to the internet with a veth pair and cap it with cgroups.
Prefer to watch? This post is the written companion to the video above.
In Part 1 we landed on one line: a container is an image, plus namespaces, plus cgroups. The image is just files. Namespaces and cgroups are plain Linux features.
If that is really true, we should be able to build a container ourselves with nothing but Linux commands. No Docker, no runc. So that is what we will do: build a small Debian root filesystem, isolate it, give it its own network, run a Node.js server inside it, cap its CPU and memory, and curl it from the host.
All scripts are in the container-from-scratch repo on GitHub.
Every script in this post comes from this repo, at commit 94194e3.
Cloudmash333/container-from-scratch on GitHubHere is the plan. Each step adds one piece: files (debootstrap), a jail (chroot), isolation (unshare and namespaces), a network (a network namespace, a veth pair and NAT), mounts (/proc, /sys, /tmp) and a limit (a cgroup v2 group for CPU and memory).
Step 1: a root filesystem with debootstrap
A container needs its own files. Docker gets them from an image. We will make our own with debootstrap, a tool that downloads a minimal Debian system into a folder.
sudo apt install debootstrap
mkdir containers
sudo debootstrap --variant=minbase stable ./containers http://deb.debian.org/debian/
It needs root and a target folder. When it finishes, ./containers holds a full Linux tree: bin, lib, usr, etc and so on. Check it: the host's /etc/os-release says Ubuntu 24.04, but ./containers/etc/os-release says Debian. Two different Linux distributions on one disk, both using the same kernel.
Step 2: jail it with chroot
sudo chroot ./containers /bin/bash
Now / means ./containers. Every path the shell opens is resolved inside that folder. That is the "jail" from Part 1.
But it is a very bare jail. minbase has no ps, no ping, no editor. Install what we need:
apt-get install iputils-ping procps nano nodejs iproute2 -y
iputils-ping to test the network, procps for ps, nano to edit files, nodejs for our app, and iproute2 for the ip command we need for networking.
chroot only changes the files the process sees. It still shares processes, network and hostname with the host. That is what namespaces fix next.
Step 3: isolate it with unshare
unshare starts a program in new namespaces. We ask for almost all of them:
sudo unshare --mount --pid --uts --net --ipc --cgroup --fork /bin/bash
Each flag maps to a namespace from Part 1: mount, PID, hostname (UTS), network, IPC and cgroup.
The full list of Linux namespaces. Our unshare call uses six of them; Time and User are left out.
Why --fork? A new PID namespace does not move the process that creates it. Only its children go inside. Without --fork, unshare creates the namespace and then runs bash itself, so the shell stays in the old PID namespace with its old PID. With --fork, unshare forks a child that execs bash, and that child is PID 1 in the new namespace. Watch this part (06:41)
Now try the network:
ping -c 3 google.com
# ping: google.com: Temporary failure in name resolution
That failure is good news. A brand new network namespace has no interfaces except a loopback that is down. It is fully cut off. We have to wire it up ourselves.
Sidebar: the IPC namespace in action
Processes can talk through IPC: System V shared memory, message queues and semaphores, or POSIX shared memory under /dev/shm. On the host, create a 64 MB shared memory segment:
ipcmk -M 64M # Shared memory id: 1
ipcs -m # host lists it
Inside our container (started with --ipc), ipcs -m shows nothing. Different IPC namespace, different set of segments. Clean up on the host with ipcrm -m 1. (The ID depends on what already exists on your machine; the first one is usually 0.) Watch this part (07:33)
The same demo from the video: two processes sharing one 64 MB segment on the host, the commands to create, list and remove it, and the unshare --ipc shell that cannot see it.
Step 4: give it a network (veth pair + NAT)
We create a named network namespace and connect it to the host with a veth pair: two virtual network interfaces joined like the two ends of a cable. A packet that goes in one end comes out the other.
Here is the whole script:
# 1. Create a new network namespace
ip netns add mynetworkns
# 2. Create the veth pair in the default namespace
ip link add systemeth type veth peer name containereth
# 3. Move one end of the veth pair (containereth) into the new namespace
ip link set containereth netns mynetworkns
# 4. Assign an IP and bring up the interface on the host (systemeth)
ip addr add 10.0.0.1/24 dev systemeth
ip link set systemeth up
# 5. Assign an IP and bring up the interface inside the new namespace (containereth)
ip netns exec mynetworkns ip addr add 10.0.0.2/24 dev containereth
ip netns exec mynetworkns ip link set containereth up
# 6. Bring up the loopback interface in the new namespace
ip netns exec mynetworkns ip link set lo up
# 7. Enable IP forwarding on the host
sysctl -w net.ipv4.ip_forward=1
# 8. Add a NAT rule so the namespace can access the internet
iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE
# 9. Add a default route inside the namespace, pointing to the host's veth end
ip netns exec mynetworkns ip route add default via 10.0.0.1
# 10. Test connectivity from inside the new namespace
ip netns exec mynetworkns ping -c 3 8.8.8.8 What each part does, in order:
ip netns add mynetworkns: a new, empty network namespace.ip link add systemeth type veth peer name containereth: the cable. Both ends start on the host.ip link set containereth netns mynetworkns: push one end into the namespace.- The host end gets
10.0.0.1/24and is brought up. - The container end gets
10.0.0.2/24and is brought up, plus the loopbacklo. sysctl -w net.ipv4.ip_forward=1: let the host route packets that are not for itself.iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -j MASQUERADE: packets from 10.0.0.x leave the host looking like they came from the host's own IP (NAT), so replies find their way back.- Default route inside the namespace:
via 10.0.0.1. Anything not local goes through the host end of the cable. ping -c 3 8.8.8.8from inside the namespace to test.
Step 4 as I drew it in the video: a packet leaves containereth (10.0.0.2) in mynetworkns and arrives at systemeth (10.0.0.1) on the host.
Run it and enter the namespace, jailed in our root filesystem:
sudo bash create_network.sh
sudo ip netns exec mynetworkns chroot ./containers /bin/bash
ping 8.8.8.8 now works. ping google.com still fails, because the container's /etc/resolv.conf was copied from the build machine and points at a resolver it cannot reach. Give it a public DNS server, and set a hostname while we are here (this only changes the container's UTS namespace):
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
hostname cloudmash-container
ping -c 3 google.com # 3 packets transmitted, 3 received
Step 5: mount /proc, /sys and /tmp
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t tmpfs tmpfs /tmp
These three lines confused a lot of viewers, so here is what they do.
/proc is a virtual filesystem. Nothing in it is stored on disk. It is an interface between user space and the kernel. When you read a file in /proc, the kernel generates the content right then from its live data structures.
/proc from the video: a virtual filesystem, an interface to kernel data that is generated when you read it.
You can see it yourself. cat /proc/uptime twice and the numbers change. ls -l /proc/uptime says the size is 0, yet it printed data. ls /proc shows numbered folders: those are process IDs, created on the fly by the kernel.
What is inside /proc, and who answers when you read it: not files on disk, but views into kernel memory, generated on every read.
And here is why it matters for containers: procfs shows the processes of the PID namespace of whoever mounted it. Mount it inside the container and ps shows PIDs starting from 1, only the container's processes, while the host sees the same processes as 2873 and up.
The first word after -t is the filesystem type. The second (proc, sys, tmpfs) is just a label: you can write any name there and it still works. /sys (sysfs) does the same for devices and kernel objects, and tmpfs gives the container its own in-memory /tmp.
The mount command taken apart: the type, a label that can be any string, and the mount point.
Part 3: I answer your doubts on mounts, --fork and IPC. The video starts at the mounts chapter (0:39).
Try it: run an app inside
Create server.js inside the container (nano is installed now):
// Minimal server used in the video (reconstructed from the demo; not in the repo).
// Run inside the container: node server.js then from the host: curl 10.0.0.2:3000
const http = require("http");
http
.createServer((req, res) => {
res.end("Hello, this response is from container\n");
})
.listen(3000, "0.0.0.0", () => console.log("listening on :3000")); node server.js
ip addr inside shows lo and containereth with 10.0.0.2. On the host, ip addr shows systemeth with 10.0.0.1. From the host:
curl 10.0.0.2:3000
# Hello, this response is from container
Our host just talked to a server living in its own filesystem, its own namespaces and its own network.
Step 6: cap CPU and memory with a cgroup
On cgroup v2 a cgroup is a folder under /sys/fs/cgroup, limits are files inside it, and you join a process by writing its PID into cgroup.procs. Run this on the host, not in the container.
# Setup Cgroups
# Go to HOST system (your main system,outside container), this setting you have to do in host system
cd /sys/fs/cgroup
sudo mkdir mycgroup
echo "+cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
echo "50000 100000" | sudo tee /sys/fs/cgroup/mycgroup/cpu.max
echo "500000000" | sudo tee /sys/fs/cgroup/mycgroup/memory.max
ps aux | grep "mynetworkns" #Get process id of 2nd networkns process , because first is sudo process and last one is grep itself
echo '<your process id here>' > /sys/fs/cgroup/mycgroup/cgroup.procs # Add process ID of network namespace process here
#verify whether cgroup changed to mycgroup for mynetworkns process
cat /proc/<id of process>/cgroup # this should show something like 0::mycgroup Line by line:
+cpuintocgroup.subtree_controlturns on the CPU controller for child groups.cpu.max=50000 100000: 50,000 microseconds of CPU in every 100,000. That is 50% of one CPU.memory.max=500000000bytes, about 500 MB. Above that the OOM killer steps in.- Find the PID with
ps aux | grep mynetworkns. The first line is thesudoprocess and the last one isgrepitself; take the one in between. - Write it into
cgroup.procs, then verify withcat /proc/<pid>/cgroup. It should now show0::/mycgroup.
That is a full container: its own root filesystem, its own namespaces, its own network and a resource fence. Exactly what runc does for Docker, by hand.
Clean up
Exit the shells, then on the host run one command from the repo folder:
sudo bash revert.sh
revert.sh turns the cpu controller back off, removes mycgroup, and then calls delete_networkns.sh for you:
# On your host(outside container)
#Revert cgroup changes
echo "-cpu" | sudo tee /sys/fs/cgroup/cgroup.subtree_control
sudo rmdir /sys/fs/cgroup/mycgroup
#delete network namespace
bash delete_networkns.sh delete_networkns.sh removes the route, the NAT rule, IP forwarding, the veth pair and the namespace:
# Remove the default route in the network namespace
ip netns exec mynetworkns ip route del default via 10.0.0.1
# Delete the masquerade rule from iptables
iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -j MASQUERADE
# Reset IP forwarding (if it was originally disabled)
# If you are unsure, it's safer to leave it as is or check its initial state.
sysctl -w net.ipv4.ip_forward=0
#remove ip addr from the host systemeth interface
ip addr del 10.0.0.1/24 dev systemeth
# Bring down and delete the systemeth veth interface
ip link set systemeth down
ip link del systemeth
# Delete the network namespace
ip netns del mynetworkns Going deeper
The same steps, mapped to what Docker and runc do:
| By hand | Docker / runc |
|---|---|
debootstrap into ./containers | docker pull + unpacked layers |
chroot ./containers | pivot_root into the rootfs |
unshare --mount --pid --uts --net --ipc --cgroup --fork | runc clone() with CLONE_NEW* flags |
create_network.sh (veth + NAT) | docker0 bridge + veth + iptables |
echo 'nameserver ...' > /etc/resolv.conf | Docker writes resolv.conf for you |
mount -t proc proc /proc | runc mounts /proc, /sys, /dev |
/sys/fs/cgroup/mycgroup | docker run --cpus --memory |
What's next
If Part 1 was the why, this was the how. Star the repo, try it in a VM, and tell me in the comments which Linux internals you want next.
Once one container makes sense, the next question is how many of them fit into a real system. I cover how containers and Kubernetes fit into system design on the dev site.