How to Monitor a Docker Container using PRTG

Note Note: An exposed and unsecured port could lead to a major security issue! Whoever is able to connect to the docker socket could easily gain full control on the system – meaning root access.

Therefore it is really important to handle these configurations with care. The measurement we are going to take is to secure the remote access by using TLS certificates. You can read more about this in the docker docs. A guide on the PRTG Docker Container Sensor can be found here.

https://www.cybus.io/learn/docker-container-monitoring-using-prtg/

Certificate Generation for Docker Deamon

First of all, we need to create the certificates. There are basically two options for doing this:

The first being to use your public company certificate authority. You can buy it from DigiCert, GlobalSign…Or even you can create your own free SSL certificate using Let’s Encrypt.
The second way is to create the certificates locally (self-certificate)

We are going to use the first option, which means all certificates are going to be public and signed by a CA. It’s more secure and reliable. You can choose the first option for testing and learning purpose.

We assume you got the certificates from a CA and copy the certificate files into /etc/docker/.ssl directory. Most CAs (Certificate Authority) provide certificates in PEM format in Base64 ASCII encoded files.

ca.pem: The intermediate certificate.
cert.pem: The server certificate.
key.pem: The private key

root@vm-1745618211:/etc/docker/.ssl# ll
total 24
drwxr-xr-x 2 root root 4096 Mar 28 01:26 ./
drwxr-xr-x 3 root root 4096 Mar 28 01:24 ../
-rwxr-xr-x 1 root root 5607 Mar 28 01:25 ca.pem*
-rwxr-xr-x 1 root root 1858 Mar 28 01:26 cert.pem*
-rwxr-xr-x 1 root root 1704 Mar 28 01:26 key.pem*

Docker Service Configuration

1. With all the necessary certificates in place, we have to expose them to the docker daemon. We can find the position of the responsible configuration file by checking the status of the docker service.

root@ub222:~# sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-03-26 22:55:54 EDT; 36min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 4542 (dockerd)
      Tasks: 10
     Memory: 25.2M
        CPU: 634ms
     CGroup: /system.slice/docker.service
             └─4425 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

2. To adjust the configuration, we are going to open the configuration using sudo privileges.

sudo nano /lib/systemd/system/docker.service

# Output
root@vm-1745618211:/# sudo nano /lib/systemd/system/docker.service
...
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutStartSec=0
RestartSec=2
Restart=always
...

3. Find the line starting with ExecStart=/usr/bin/dockerd -H fd:// and add the following content to it. Be sure to use the correct path for your ssl certificate files.

-H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/etc/docker/.ssl/ca.pem --tlscert=/etc/docker/.ssl/server-cert.pem --tlskey=/etc/docker/.ssl/server-key.pem

Once done, the line looks like this:

ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/etc/docker/.ssl/ca.pem --tlscert=/etc/docker/.ssl/server-cert.pem --tlskey=/etc/docker/.ssl/server-key.pem --containerd=/run/containerd/containerd.sock

4. Flush the changes and restart the docker service to take effect.

sudo systemctl daemon-reload
sudo systemctl restart docker

5. Now we can verify the changes did take effect by checking the docker daemon status:

sudo systemctl status docker.service

# Output
root@vm-1745618211:/# sudo systemctl status docker
docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-03-28 02:06:04 UTC; 14s ago
TriggeredBy:  docker.socket
       Docs: https://docs.docker.com
   Main PID: 7004 (dockerd)
      Tasks: 11
     Memory: 30.2M
     CGroup: /system.slice/docker.service
             7004 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376 --tlsverify=true --tlscacert=/etc/docker/.ssl/ca.pem --tlscert=/etc/docker/.ssl/cert.pem --tlskey=/etc/docker/.ssl/key.pem --containerd=/run/containerd/containerd.sock

6. Now we can use the Docker CLI to connect to the Docker Daemon using the specified port. The important part is to use –tlsverify=true as this tells the Docker CLI to use the certificates located in /etc/docker/.ssl directory.

docker -H 127.0.0.1:2376 --tlsverify=true version
docker -H 172.16.0.131:2376 --tlsverify=true version #Replace with your docker server's IP.

root@vm-1745618211:/# docker -H 127.0.0.1:2376 --tlsverify=true version
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:56 2023
 OS/Arch:           linux/amd64
 Context:           default
error during connect: Get "https://127.0.0.1:2376/v1.24/version": x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

Note Note: You can safely ignore the error because the certificate not cover the IP addesses.

root@vm-1745618211:/# docker -H 10.0.0.4:2376 --tlsverify=true version
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:46:56 2023
 OS/Arch:           linux/amd64
 Context:           default
error during connect: Get "https://10.0.0.4:2376/v1.24/version": x509: cannot validate certificate for 10.0.0.4 because it doesn't contain any IP SANs

7. Telnet to the Docker Deamon via port 2376 to verify it’s running and opening.

root@vm-1745618211:/# netstat -plant | grep 2376
tcp6       0      0 :::2376                 :::*                    LISTEN      7004/dockerd
tcp6       0      0 10.0.0.4:2376           123.27.21.137:64401     ESTABLISHED 7004/dockerd

Add docker sensor into PRTG

The last and final step is to add the docker sensor inside of PRTG. This should be fairly easy to accomplish by following the provided instructions from PRTG.

1. Open the PRTG web admin console | Add the docker device then add a sensor.

2. Search docker then select the docker sensor from the result.

3. Enter the certificate and the private key of the docker daemon that you’ve configured in the previous step. You can get it using the cat command.

root@vm-1745618211:/etc/docker/.ssl# cat key.pem
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC8udZQeWsUtIJ3
...
3wXO7YkEQzJoItZI4BCoEF+8msrXs1w2TWvSkwuR8USHCci9Jq1Cnf/BAoGBAJZv
8f95VIObYlHrlQSg5ARZU9P0lzNq+w0LE4oyYc44ee4XttrPkG7xSZ+qCzi9By8a
...
-----END PRIVATE KEY-----

root@vm-1745618211:/etc/docker/.ssl# cat cert.pem
-----BEGIN CERTIFICATE-----
MIIFLjCCBBagAwIBAgISA1ef7lcLvNgF00Rd3WUFDDkcMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
FGRvY2tlci5ib25iZW4zNjUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
...
t0nE+KVIr3rVpTjgYFkLBd3sri40QobwVfzd08EhPpIU06jKcHbCgoxFVfsWFqF4
v14IT1dCsDtjIZRfR/HcWbXpKKKBv8H6plNOE2rv6WrJJQ==
-----END CERTIFICATE-----

4. You should see the list of containers in the docker daemon. Select the containers that you want to monitor then click Create button.