How to setup Node Exporter, cAdvisor, Prometheus and Grafana (with Docker & Caddy)

Untested

Description

Node Exporter, cAdvisor, Prometheus and Grafana are monitoring services.

Create a DNS record

If you want to monitor your system from outside your network, you will need to create a record for Grafana (the rest don't need any).
To do this, go to cPanel and add a new record for Grafana (e.g. grafana.akdscloud.eu).

Caddy

grafana.akdscloud.eu {
    header Strict-Transport-Security max-age=31536000
    reverse_proxy localhost:3001
}

Folders

Create the folder structure and the required files.

mkdir -p ~/srv/services/monitoring
cd ~/srv/services/monitoring
mkdir -p prometheus/{config,data} grafana/data
sudo chown -R nobody:nogroup prometheus/data    # without this, prometheus won't work
touch docker-compose.yml .env prometheus/config/prometheus.yml

Docker

1. Add this to docker-compose.yml:

   services:
     cadvisor-pi:
       image: ghcr.io/google/cadvisor:latest
       container_name: cadvisor-pi
       restart: unless-stopped
       privileged: true
       ports:
         - "8080:8080"
       volumes:
         - /:/rootfs:ro
         - /var/run:/var/run:ro
         - /sys:/sys:ro
         - /var/lib/docker:/var/lib/docker:ro
     node-exporter:
       image: prom/node-exporter:latest
       container_name: node-exporter
       restart: unless-stopped
       network_mode: host
       pid: host
       command:
         - '--path.procfs=/host/proc'
         - '--path.sysfs=/host/sys'
         - '--path.rootfs=/rootfs'
         - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
       volumes:
         - /proc:/host/proc:ro
         - /sys:/host/sys:ro
         - /:/rootfs:ro
     prometheus:
       image: prom/prometheus:latest
       container_name: prometheus
       restart: unless-stopped
       ports:
         - "9090:9090"
       volumes:
         - ./prometheus/config:/etc/prometheus
         - ./prometheus/data:/prometheus
       command:
         - '--config.file=/etc/prometheus/prometheus.yml'
         - '--storage.tsdb.path=/prometheus'
         - '--web.console.libraries=/etc/prometheus/console_libraries'
         - '--web.console.templates=/etc/prometheus/consoles'
         - '--web.enable-lifecycle'
         - '--storage.tsdb.retention.time=90d'
     grafana:
       image: grafana/grafana:latest
       container_name: grafana
       restart: unless-stopped
       ports:
         - "127.0.0.1:3001:3000"
       volumes:
         - ./grafana/data:/var/lib/grafana
       environment:
         - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
         - GF_USERS_ALLOW_SIGN_UP=false
         - GF_SERVER_ROOT_URL=https://grafana.akdscloud.eu
       depends_on:
         - prometheus
   

2. Add this to .env:

   GRAFANA_ADMIN_PASSWORD=<your_string_password>
   

   You can generate a strong password with

   openssl rand -hex 20
   

3. Add this to prometheus.yml (in ./prometheus/config):

   global:
     scrape_interval: 15s
     scrape_timeout: 10s
     evaluation_interval: 15s
   
   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
         - targets: ['prometheus:9090']
   
     - job_name: 'cadvisor'
       static_configs:
         - targets: ['cadvisor:8080']
   
     - job_name: 'node'
       static_configs:
         - targets: ['node-exporter:9100']
   
     - job_name: 'caddy'
       metrics_path: /metrics
       static_configs:
         - targets: ['192.168.100.50:2019']
   
     - job_name: "node-exporter-pi"
       static_configs:
         - targets: ["192.168.100.10:9100"]
           labels:
             host: "RPi4"
   
     - job_name: "cadvisor-pi"
       static_configs:
         - targets: ["192.168.100.10:8080"]
           labels:
             host: "RPi4"
   

4. Download and start all services:

   docker compose up -d
   

5. If you created a DNS record for Grafana, restart Caddy to get an SSL certificate (details here). Otherwise, proceed to the next step.

Try it out

Check node-exporter

curl http://localhost:9100/metrics

If you see a huge text output, it’s working.

Check cAdvisor

curl http://localhost:8080/metrics

If you see a huge text output, it’s working.

Check Prometheus

See if you can access the Prometheus web interface at: http://your_machines_IP:9090/query
If you can, it works. If not, check the logs.

Check Grafana

See if you can access the Grafana web interface at: http://your_machines_IP:3001 or at the URL you set.
If you can, it works. If not, check the logs.

Troubleshooting

General issues

If there are issues, see the general troubleshooting guide.

No RAM usage

If docker stats show 0 RAM usage on the RPi, do this:

sudo nano /boot/firmware/cmdline.txt

Add these parameters to the end of the existing line (don't create a new line):

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Then reboot:

sudo reboot

After rebooting, docker stats should show actual memory usage values.

Configuration

Set Prometheus as a data source in Grafana

Connections → Data sources → Add new data source

Import dashboards

Dashboards → New → Import

Go to Grafana Dashboard and search for dashboards using keywords such as docker, raspberry pi, node exporter.
Copy the board's ID to your clipboard.
Go back to the import screen, paste the ID and click on Load. Select Prometheus as your data source if asked.
Enjoy you dashboard!
After that, try out other boards and see which ones you like best.

Additional Info

WARNING!

Prometheus merges metrics from different sources (mini-pc + rpi).
Use 2 Prometheus instances if you don't want merged metrics!