Rebuild the image ensuring absolute symlinks or correct relative paths. Scenario 2: Disk space leak from dangling snapshot links Sometimes, the parent link remains even after the child snapshot is deleted, preventing garbage collection.
# Get container PID crictl inspect <container> | grep pid nsenter -t <pid> -m bash Inside, check for broken symlinks find / -type l -xtype l 2>/dev/null cri file system tools link
crictl images # Lists images with their IDs and sizes crictl inspect <container-id> # Shows detailed mount points and layer paths crictl imagefsinfo # Reports filesystem usage for image storage The inspect command reveals the rootfs path—a symbolic link that points to the container’s writable layer. For example: Rebuild the image ensuring absolute symlinks or correct
crio-status info | grep -A 10 "storage" crio-status containers --id <id> # Shows container rootfs path The keyword "link" in the context of CRI file system tools refers to two distinct but related concepts: filesystem links (ln) and layer links (parent pointers) . Symbolic Links vs. Hard Links in Container Storage | Feature | Symbolic Link (symlink) | Hard Link | |---------|------------------------|------------| | Cross-filesystem | Yes | No | | Points to inode or path | Path | Inode | | Break if target deleted | Yes (dangling link) | No (file persists) | | Used in CRI for | Config file references, log paths | Deduplication of identical layers | For example: crio-status info | grep -A 10
This article explores the relationship between CRI-compliant runtimes (containerd and CRI-O), the filesystem tools that manipulate container storage, and how the humble link (both symbolic and hard) functions as the architectural glue holding container layers together. Before diving into tools and links, we must establish a baseline. The CRI is a Kubernetes API that kubelet uses to communicate with container runtimes. It abstracts the runtime implementation, allowing Kubernetes to work with Docker (via dockershim, now deprecated), containerd, CRI-O, and others.
Introduction: The Hidden Complexity of Container Filesystems In the world of containerized applications, the storage layer is often treated as a black box. Developers run docker run or kubectl apply , and somehow, the files appear. But beneath the surface lies a sophisticated ecosystem of snapshots, layers, and mount points. For those managing Kubernetes clusters using the Container Runtime Interface (CRI), understanding CRI file system tools and the critical role of the link (symbolic or hard link) is not just an advanced skill—it is a necessity for debugging, performance tuning, and disaster recovery.
/var/lib/containers/storage/overlay/<layer-id>/merged -> /var/lib/containers/storage/overlay/<layer-id>/../<parent-id>/merged Scenario 1: "No such file or directory" inside a container Even though the file exists in the image, the container cannot see it. This is often due to a broken symbolic link in a lower layer .