Cannot start docker after kernel upgrading

There is a problem with Linux kernel 3.16 (Debian Jessie) interacting with new versions of Docker (17.x). Sometimes a server running these versions of kernel and Docker Engine could stuck and don't answer on the users commands. The only way to solve this problem (server hanging) is rebooting the server.
To resolve this issue completely you should upgrade the kernel image from 3.16 to 4.X using jessie-backports.

1. Add a backports repository to sources list:

sudo bash -c 'echo "deb http://httpredir.debian.org/debian jessie-backports main contrib non-free" >  /etc/apt/sources.list.d/backports.list'

2. Update your cache

sudo apt-get update

3. Find out a necessary versions of packages
Basically you need to install two packages - linux-image-4.x.x.bpo.x-{arch} and linux-image-amd64. You can check available versions of these packages using the commands:

apt-cache policy linux-image-amd64

apt-cache search linux-image-4

Sample output:

root@server:~# apt-cache policy linux-image-amd64

linux-image-amd64:

  Installed: 3.16+63

  Candidate: 3.16+63

  Version table:

  4.9+80~bpo8+1 800

        200 http://httpredir.debian.org/debian/ jessie-backports/main amd64 Packages

        100 /var/lib/dpkg/status

 *** 3.16+63 800

        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages

        500 http://cdn.debian.net/debian/ jessie/main amd64 Packages

root@silver-cta1.semrush.net:~# apt-cache search linux-image-4

linux-headers-4.9.0-0.bpo.3-amd64 - Header files for Linux 4.9.0-0.bpo.3-amd64 linux-headers-4.9.0-0.bpo.3-rt-amd64 - Header files for Linux 4.9.0-0.bpo.3-rt-amd64 linux-image-4.9.0-0.bpo.3-amd64 - Linux 4.9 for 64-bit PCs ---------------------------output omitted-------------------------------------

4. Install kernel packages:

sudo apt-get install linux-image-amd64=4.9+80~bpo8+1 linux-image-4.9.0-0.bpo.3-amd64

5. Now you need to reboot the server.

After that check currently version of the kernel using dpkg or uname -a. If upgrade was successful you will see correct new package versions. Now start docker process

root@server:~# systemctl start docker

Job for docker.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

Oops, what's going on?

journalctl -u docker

Oct 13 08:06:39 silver-cta3.semrush.net systemd[1]: docker.service start request repeated too quickly, refusing to start.

Oct 13 08:06:39 silver-cta3.semrush.net systemd[1]: Failed to start Docker Application Container Engine.

Oct 13 08:06:39 silver-cta3.semrush.net systemd[1]: Unit docker.service entered failed state.

Oct 13 08:13:46 silver-cta3.semrush.net systemd[1]: Starting Docker Application Container Engine...

Oct 13 08:13:46 silver-cta3.semrush.net docker[26423]: Command "daemon" is deprecated, and will be removed in Docker 17.12. Please run `dockerd` directly.

Oct 13 08:13:46 silver-cta3.semrush.net docker[26423]: time="2017-10-13T08:13:46.065379664-04:00" level=info msg="libcontainerd: new containerd process, pid: 26438"

Oct 13 08:13:47 silver-cta3.semrush.net docker[26423]: time="2017-10-13T08:13:47.066742860-04:00" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: %!s(<nil>). Deleting synchronously"

Oct 13 08:13:47 silver-cta3.semrush.net docker[26423]: time="2017-10-13T08:13:47.068803075-04:00" level=error msg="[graphdriver] prior storage driver aufs failed: driver not supported"

Oct 13 08:13:47 silver-cta3.semrush.net docker[26423]: Error starting daemon: error initializing graphdriver: driver not supported

Oct 13 08:13:47 silver-cta3.semrush.net systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE

We see an error message: error msg="[graphdriver] prior storage driver aufs failed: driver not supported". Hm, let me check is aufs is supported file systems by the kernel?

root@silver-cta1.semrush.net:~# docker info | grep "Storage Driver"

Storage Driver: aufs

root@silver-cta1.semrush.net:~# grep -E 'aufs|overlay' /proc/filesystems

nodev overlay

The kernel doesn't support docker storage driver. Kernel 3.16 supports aufs, but 4.X does not, BUT it supports now overlayfs, which is used by Docker too.

Next step can be dangerous for your existing images or containers. We are building docker images on deploy stage from base images downloaded from a docker hubs, so for us this operation is safe, because at any moment of rime we can build new images and create new containers. But I should note that you don't miss your files.

Execute the following command:

mv /var/lib/docker/aufs/ /var/lib/docker/aufs.old

And start the docker service. Service should start without problems. Now docker engine will use overlay2 as its default storage driver.

Written on October 13, 2017