Mdthbs

A follow-up from this question.

My further reading on Docker storage drivers revealed that the overlay driver merges all the image layers into lower layers using a hard link implementation which cause excessive inode utilization. Can someone explain this? As far as I know, creating hard links does not create a new inode.

OverlayFS is a union filesystem, and there are two storage drivers at the Docker level that make use of it: the original/older version named overlay and the newer version named overlay2. In OverlayFS, there is a lower-level directory which is exposed as read-only. On top of this directory is the upper-level directory, which allows read-write access. Each of these directories is called a layer. The combined view of both the lower-level and upper-level directories is presented as a single unit, called the 'merged' directory.

The newer overlay2 storage driver natively supports up to 128 such layers. The older overlay driver is only able to work with two layers at a time. Since most Docker images are built using multiple layers, this limitation is fairly significant. To work around this limitation, each layer is implemented as a separate directory that simulates a complete image.

To examine the differences on my test system, I pulled the 'ubuntu' image from Docker Hub and examined the differences in directory structure between the overlay2 and overlay drivers:

[root@testvm1 overlay2]$ ls */diff

4864f14e58c1d6d5e7904449882b9369c0c0d5e1347b8d6faa7f40dafcc9d231/diff:

run



4abcfa714b4de6a7f1dd092070b1e109e8650a7a9f9900b6d4c3a7ca441b8780/diff:

var



a58c4e78232ff36b2903ecaab2ec288a092e6fc55a694e5e2d7822bf98d2c214/diff:

bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var



c3f1a237c46ed330a2fd05ab2a0b6dcc17ad08686bd8dc49ecfada8d85b93a00/diff:

etc  sbin  usr  var





[root@testvm1 overlay]# ls */root/

001311c618ad7b94d4dc9586f26e421906e7ebf5c28996463a355abcdcd501bf/root/:

bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var



048f81f400f7d74f969c4fdaff6553c782d12c04890ad869d75313505c868fbc/root/:

bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var



8060f0c647f24050e1a4bff71096ffdf9665bff26e6187add87ecb8a18532af9/root/:

bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var



fbdef944657234468ee55b12c7910aa495d13936417f9eb905cdc39a40fb5361/root/:

bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

In the overlay representation, each layer simulates a complete image, while the overlay2 layers only contain the exact differences between layers. In the overlay driver's approach, hard links are used as a way to save space between the different layers. However, this method is still not perfect, and new inodes are required when the image data contains special files such as symbolic links and character devices. This can quickly add up to large number of inodes.

The inode distribution betwen the overlay2 and overlay drivers on my test system are as shown below.

[root@testvm1 overlay2]$ du --inodes -s *

8       4864f14e58c1d6d5e7904449882b9369c0c0d5e1347b8d6faa7f40dafcc9d231

27      4abcfa714b4de6a7f1dd092070b1e109e8650a7a9f9900b6d4c3a7ca441b8780

3311    a58c4e78232ff36b2903ecaab2ec288a092e6fc55a694e5e2d7822bf98d2c214

1       backingFsBlockDev

25      c3f1a237c46ed330a2fd05ab2a0b6dcc17ad08686bd8dc49ecfada8d85b93a00

5       l



[root@testvm1 overlay]# du --inodes -s *

3298    001311c618ad7b94d4dc9586f26e421906e7ebf5c28996463a355abcdcd501bf

783     048f81f400f7d74f969c4fdaff6553c782d12c04890ad869d75313505c868fbc

768     8060f0c647f24050e1a4bff71096ffdf9665bff26e6187add87ecb8a18532af9

765     fbdef944657234468ee55b12c7910aa495d13936417f9eb905cdc39a40fb5361

The total count of inodes on overlay2 comes to 3378 on my system. Using overlay, this count goes up to 5615. This value is considering a single image and with no containers running, so a large system with a number of docker containers and images could quickly hit the inode limit imposed by the backing filesystem (XFS or EXT4, where the /var/lib/docker/overlay directory is located).

Due to this reason, the newer overlay2 storage driver is currently the recommended option for most new installations. The overlay driver is deprecated as of Docker v18.09 and is expected to be removed in a future release.