Monitoring serwera w Grafanie

1. Co powstanie

Cztery kontenery. Grafana tylko rysuje. Liczby zbiera Prometheus.

Na hoście potrzebujesz Dockera z wtyczką Compose, openssl i wolnego portu 9090. qBittorrent z Web UI na 127.0.0.1:8080 jest opcjonalny: bez niego reszta panelu działa, a kafelki torrentów zostaną puste.

Adresy w tej instrukcji są z tej sieci: LAN 192.168.1.10, WireGuard 10.10.0.1, dysk sda, karta enp1s0, tunel wg0. Jak u Ciebie jest inaczej, podmień je w Compose, w certyfikacie, w firewallu i w dwóch zapytaniach dashboardu (sekcja 6).

2. Katalogi

Jeden katalog stosu. Możesz go postawić obok innych usług albo osobno. Dalej przyjmuję /home/devops/docker.

mkdir -p /home/devops/docker/monitoring/grafana/provisioning/datasources \
  /home/devops/docker/monitoring/grafana/provisioning/dashboards \
  /home/devops/docker/monitoring/grafana/provisioning/alerting \
  /home/devops/docker/monitoring/grafana/dashboards \
  /home/devops/docker/monitoring/grafana/certs \
  /home/devops/docker/monitoring/textfile

Katalog textfile zostaw pusty. Skrypt sam go zapełni plikami .prom.

3. Prometheus

Zapisz jako /home/devops/docker/monitoring/prometheus.yml.

Prometheus jest w sieci Dockera, a node-exporter na sieci hosta (żeby widzieć prawdziwe karty enp1s0 i wg0). Dlatego cel scrapowania to host.docker.internal:9100, a nie nazwa kontenera. W Compose jest do tego extra_hosts.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    static_configs:
      - targets: ["host.docker.internal:9100"]

4. Skrypt zbierający

node-exporter nie umie wielkości katalogów, listy sesji ani API qBittorrenta. To robi skrypt. Zapisz go jako /home/devops/docker/monitoring/collect.sh i nadaj chmod 755.

Kontener ma network_mode: host, więc 127.0.0.1:8080 to qBittorrent na hoście, a dysk jest podmontowany jako /host. Dwie pętle: du co 15 minut (chodzenie po dysku jest ciężkie), prędkość torrentów co 15 sekund. Ten sam skrypt zapamiętuje hashe ukończonych torrentów i przez 30 minut wystawia metrykę qbit_download_just_finished. Przy pierwszym starcie tylko zapisuje to, co już jest ukończone, i nic nie zgłasza. Katalogi mniejsze niż 50 MiB są pomijane. Wynik to textfile/host.prom i textfile/qbit.prom. node-exporter czyta każdy plik kończący się na .prom.

#!/bin/sh
# Zbiera rozmiary katalogów i aktywne sesje logowania do node_exporter textfile.
OUT=/textfile/host.prom
mkdir -p /textfile

emit_dir() {
  kb=$1
  label=$2
  # Pomiń katalogi mniejsze niż 50 MiB.
  [ "$kb" -ge 51200 ] || return 0
  echo "host_directory_size_bytes{path=\"${label}\"} $((kb * 1024))"
}

write_metrics() {
  tmp="${OUT}.tmp"
  {
    echo "# HELP host_directory_size_bytes Disk space used by a directory."
    echo "# TYPE host_directory_size_bytes gauge"

    nice -n 19 du -x -k -d 1 /host/home 2>/dev/null | while read -r kb path; do
      [ "$path" = "/host/home" ] && continue
      name=${path##*/}
      emit_dir "$kb" "/home/${name}"
    done

    nice -n 19 du -x -k -d 1 /host/var 2>/dev/null | while read -r kb path; do
      [ "$path" = "/host/var" ] && continue
      name=${path##*/}
      [ "$name" = "lib" ] && continue
      emit_dir "$kb" "/var/${name}"
    done

    nice -n 19 du -x -k -d 1 /host/var/lib 2>/dev/null | while read -r kb path; do
      [ "$path" = "/host/var/lib" ] && continue
      name=${path##*/}
      emit_dir "$kb" "/var/lib/${name}"
    done

    for spec in "/host/usr /usr" "/host/opt /opt" "/host/snap /snap" "/host/root /root" "/host/boot /boot"; do
      set -- $spec
      if [ -d "$1" ]; then
        kb=$(nice -n 19 du -x -k -s "$1" 2>/dev/null | awk '{print $1}')
        if [ -n "$kb" ]; then
          emit_dir "$kb" "$2"
        fi
      fi
    done

    echo "# HELP host_logged_in_sessions Active login sessions."
    echo "# TYPE host_logged_in_sessions gauge"
    for f in /host/run/systemd/sessions/[0-9]*; do
      [ -f "$f" ] || continue
      state=$(sed -n 's/^STATE=//p' "$f" | head -n 1)
      active=$(sed -n 's/^ACTIVE=//p' "$f" | head -n 1)
      [ "$state" = "active" ] && [ "$active" = "1" ] || continue
      user=$(sed -n 's/^USER=//p' "$f" | head -n 1)
      from=$(sed -n 's/^REMOTE_HOST=//p' "$f" | head -n 1)
      service=$(sed -n 's/^SERVICE=//p' "$f" | head -n 1)
      [ -n "$user" ] || continue
      [ -n "$from" ] || from="local"
      [ -n "$service" ] || service="unknown"
      echo "host_logged_in_sessions{user=\"${user}\",from=\"${from}\",service=\"${service}\"} 1"
    done
  } > "$tmp"
  mv "$tmp" "$OUT"
  chmod 644 "$OUT"
}

count_state() {
  printf '%s' "$1" | grep -o "\"state\":\"$2\"" | wc -l | tr -d ' '
}

sum_states() {
  info=$1
  shift
  total=0
  for state in "$@"; do
    n=$(count_state "$info" "$state")
    total=$((total + n))
  done
  echo "$total"
}


prom_label() {
  printf '%s' "$1" | tr -d '"\\' | tr '\n\t' '  ' | cut -c1-80
}

qbit_complete_lines() {
  printf '%s' "$1" | tr '}' '\n' | while read -r obj; do
    hash=$(printf '%s' "$obj" | sed -n 's/.*"hash":"\([A-Fa-f0-9][A-Fa-f0-9]*\)".*/\1/p' | head -n 1)
    [ -n "$hash" ] || continue
    progress=$(printf '%s' "$obj" | sed -n 's/.*"progress":\([0-9.][0-9.]*\).*/\1/p' | head -n 1)
    awk -v p="${progress:-0}" 'BEGIN{exit !(p+0 >= 0.999)}' || continue
    name=$(printf '%s' "$obj" | sed -n 's/.*"name":"\([^"]*\)".*/\1/p' | head -n 1)
    [ -n "$name" ] || name="$hash"
    printf '%s\t%s\n' "$hash" "$(prom_label "$name")"
  done
}

qbit_finished_metrics() {
  info=$1
  seen=/textfile/qbit-finished.hashes
  recent=/textfile/qbit-recent.tsv
  current=/textfile/qbit-complete.now
  qbit_complete_lines "$info" > "$current"
  if [ ! -f "$seen" ]; then
    awk -F '\t' '{print $1}' "$current" | sort -u > "$seen"
    : > "$recent"
    return 0
  fi
  now=$(date +%s)
  expire=$((now + 1800))
  awk -F '\t' '{print $1}' "$current" | sort -u > /textfile/qbit-complete.ids
  sort -u "$seen" -o "$seen"
  comm -13 "$seen" /textfile/qbit-complete.ids | while read -r hash; do
    [ -n "$hash" ] || continue
    name=$(awk -F '\t' -v h="$hash" '$1==h {print substr($0, index($0, "\t")+1); exit}' "$current")
    [ -n "$name" ] || name="$hash"
    printf '%s\t%s\t%s\n' "$expire" "$hash" "$name" >> "$recent"
    echo "$hash" >> "$seen"
  done
  sort -u "$seen" -o "$seen"
  fresh=/textfile/qbit-recent.next
  : > "$fresh"
  if [ -f "$recent" ]; then
    while IFS="$(printf '\t')" read -r exp hash name; do
      [ -n "$hash" ] || continue
      [ "$exp" -gt "$now" ] 2>/dev/null || continue
      printf '%s\t%s\t%s\n' "$exp" "$hash" "$name" >> "$fresh"
      echo "qbit_download_just_finished{name=\"${name}\",hash=\"${hash}\"} 1"
    done < "$recent"
  fi
  mv "$fresh" "$recent"
}

write_qbit() {
  tmp="/textfile/qbit.prom.tmp"
  transfer=$(wget -qO- -T 5 http://127.0.0.1:8080/api/v2/transfer/info 2>/dev/null || true)
  info=$(wget -qO- -T 8 http://127.0.0.1:8080/api/v2/torrents/info 2>/dev/null || true)
  dl=$(printf '%s' "$transfer" | sed -n 's/.*"dl_info_speed":\([0-9][0-9]*\).*/\1/p' | head -n 1)
  up=$(printf '%s' "$transfer" | sed -n 's/.*"up_info_speed":\([0-9][0-9]*\).*/\1/p' | head -n 1)
  [ -n "$dl" ] || return 0
  [ -n "$up" ] || up=0
  downloading=$(sum_states "$info" downloading stalledDL forcedDL metaDL queuedDL checkingDL allocating)
  seeding=$(sum_states "$info" uploading stalledUP forcedUP queuedUP checkingUP)
  paused=$(sum_states "$info" pausedDL pausedUP stoppedDL stoppedUP)
  errors=$(sum_states "$info" error missingFiles)
  finished=$(qbit_finished_metrics "$info")
  {
    echo "# HELP qbit_transfer_bytes_per_second qBittorrent transfer speed."
    echo "# TYPE qbit_transfer_bytes_per_second gauge"
    echo "qbit_transfer_bytes_per_second{direction=\"download\"} $dl"
    echo "qbit_transfer_bytes_per_second{direction=\"upload\"} $up"
    echo "# HELP qbit_torrents Torrents by state."
    echo "# TYPE qbit_torrents gauge"
    echo "qbit_torrents{state=\"pobieranie\"} $downloading"
    echo "qbit_torrents{state=\"seed\"} $seeding"
    echo "qbit_torrents{state=\"pauza\"} $paused"
    echo "qbit_torrents{state=\"blad\"} $errors"
    if [ -n "$finished" ]; then
      echo "# HELP qbit_download_just_finished Torrent that finished in the last 30 minutes."
      echo "# TYPE qbit_download_just_finished gauge"
      printf '%s\n' "$finished"
    fi
  } > "$tmp"
  mv "$tmp" /textfile/qbit.prom
  chmod 644 /textfile/qbit.prom
}

(
  while true; do
    write_metrics
    sleep 900
  done
) &

while true; do
  write_qbit
  sleep 15
done

chmod 755 /home/devops/docker/monitoring/collect.sh

5. Pliki Grafany

Grafana przy starcie sama wczytuje źródło danych i dashboard. Nie trzeba klikać tego w interfejsie.

/home/devops/docker/monitoring/grafana/provisioning/datasources/prometheus.yml

uid: prometheus musi być taki sam jak w dashboardzie. Adres to nazwa kontenera w sieci Compose.

apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s

/home/devops/docker/monitoring/grafana/provisioning/dashboards/dashboards.yml

apiVersion: 1

providers:
  - name: Serwer
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards

6. Dashboard

Zapisz jako /home/devops/docker/monitoring/grafana/dashboards/serwer.json. To cały ekran: CPU, RAM, dysk i IO, osobno sieć LAN i VPN, temperatura, qBittorrent, usługi, katalogi, zalogowani.

Dwa miejsca zależne od sprzętu, gdyby nazwy się różniły: device="sda" (dysk), device="enp1s0" (LAN) i device="wg0" (VPN). Nazwy sprawdzisz przez lsblk i ip -br link.

{
  "annotations": {
    "list": []
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "id": 1,
      "type": "stat",
      "title": "CPU",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
          "legendFormat": "CPU"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 70
              },
              {
                "color": "red",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto"
      }
    },
    {
      "id": 2,
      "type": "stat",
      "title": "RAM",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100",
          "legendFormat": "RAM"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 75
              },
              {
                "color": "red",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto"
      }
    },
    {
      "id": 3,
      "type": "stat",
      "title": "Dysk /",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/\",fstype=~\"ext4|xfs\"} / node_filesystem_size_bytes{mountpoint=\"/\",fstype=~\"ext4|xfs\"}) * 100",
          "legendFormat": "Dysk"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 75
              },
              {
                "color": "red",
                "value": 90
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto"
      }
    },
    {
      "id": 4,
      "type": "stat",
      "title": "Load",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 12,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "node_load1",
          "legendFormat": "load1"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "decimals": 2,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 3
              },
              {
                "color": "red",
                "value": 4
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto"
      }
    },
    {
      "id": 5,
      "type": "stat",
      "title": "Uptime",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 16,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "time() - node_boot_time_seconds",
          "legendFormat": "uptime"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "text",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "none",
        "graphMode": "none",
        "justifyMode": "auto"
      }
    },
    {
      "id": 6,
      "type": "stat",
      "title": "Sesje",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 20,
        "y": 0
      },
      "targets": [
        {
          "refId": "A",
          "expr": "sum(max by (user, from, service) (host_logged_in_sessions)) or vector(0)",
          "legendFormat": "sesje"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "text",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "auto",
        "textMode": "auto",
        "colorMode": "none",
        "graphMode": "none",
        "justifyMode": "auto"
      }
    },
    {
      "id": 7,
      "type": "timeseries",
      "title": "CPU",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 4
      },
      "targets": [
        {
          "refId": "A",
          "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
          "legendFormat": "użycie"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 15,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "single",
          "sort": "none"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 8,
      "type": "timeseries",
      "title": "RAM",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 4
      },
      "targets": [
        {
          "refId": "A",
          "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes",
          "legendFormat": "zajęte"
        },
        {
          "refId": "B",
          "expr": "node_memory_MemAvailable_bytes",
          "legendFormat": "wolne"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "bytes",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 15,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 9,
      "type": "timeseries",
      "title": "Dysk /",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 12
      },
      "targets": [
        {
          "refId": "A",
          "expr": "node_filesystem_size_bytes{mountpoint=\"/\",fstype=~\"ext4|xfs\"} - node_filesystem_avail_bytes{mountpoint=\"/\",fstype=~\"ext4|xfs\"}",
          "legendFormat": "zajęte"
        },
        {
          "refId": "B",
          "expr": "node_filesystem_avail_bytes{mountpoint=\"/\",fstype=~\"ext4|xfs\"}",
          "legendFormat": "wolne"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "bytes",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 15,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 15,
      "type": "timeseries",
      "title": "Dysk IO",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 12
      },
      "targets": [
        {
          "refId": "A",
          "expr": "rate(node_disk_read_bytes_total{device=\"sda\"}[5m])",
          "legendFormat": "odczyt"
        },
        {
          "refId": "B",
          "expr": "rate(node_disk_written_bytes_total{device=\"sda\"}[5m])",
          "legendFormat": "zapis"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "Bps",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 10,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 10,
      "type": "timeseries",
      "title": "Sieć LAN",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 20
      },
      "targets": [
        {
          "refId": "A",
          "expr": "rate(node_network_receive_bytes_total{device=\"enp1s0\"}[5m])",
          "legendFormat": "odbiór"
        },
        {
          "refId": "B",
          "expr": "rate(node_network_transmit_bytes_total{device=\"enp1s0\"}[5m])",
          "legendFormat": "wysyłanie"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "Bps",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 10,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 16,
      "type": "timeseries",
      "title": "Sieć VPN",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 20
      },
      "targets": [
        {
          "refId": "A",
          "expr": "rate(node_network_receive_bytes_total{device=\"wg0\"}[5m])",
          "legendFormat": "odbiór"
        },
        {
          "refId": "B",
          "expr": "rate(node_network_transmit_bytes_total{device=\"wg0\"}[5m])",
          "legendFormat": "wysyłanie"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "Bps",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 10,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 17,
      "type": "timeseries",
      "title": "Temperatura CPU",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 8,
        "x": 0,
        "y": 28
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max(node_hwmon_temp_celsius{chip=\"platform_coretemp_0\"})",
          "legendFormat": "CPU"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "celsius",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 10,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 18,
      "type": "timeseries",
      "title": "qBittorrent",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 10,
        "x": 8,
        "y": 28
      },
      "targets": [
        {
          "refId": "A",
          "expr": "qbit_transfer_bytes_per_second{direction=\"download\"}",
          "legendFormat": "pobieranie"
        },
        {
          "refId": "B",
          "expr": "qbit_transfer_bytes_per_second{direction=\"upload\"}",
          "legendFormat": "wysyłanie"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "Bps",
          "custom": {
            "drawStyle": "line",
            "lineWidth": 2,
            "fillOpacity": 10,
            "showPoints": "never",
            "spanNulls": false
          }
        },
        "overrides": []
      },
      "options": {
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        },
        "legend": {
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        }
      }
    },
    {
      "id": 19,
      "type": "stat",
      "title": "Torrenty",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 8,
        "w": 6,
        "x": 18,
        "y": 28
      },
      "targets": [
        {
          "refId": "A",
          "expr": "qbit_torrents{state=\"pobieranie\"}",
          "legendFormat": "pobieranie"
        },
        {
          "refId": "B",
          "expr": "qbit_torrents{state=\"seed\"}",
          "legendFormat": "seed"
        },
        {
          "refId": "C",
          "expr": "qbit_torrents{state=\"pauza\"}",
          "legendFormat": "pauza"
        },
        {
          "refId": "D",
          "expr": "qbit_torrents{state=\"blad\"}",
          "legendFormat": "błąd"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "decimals": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "text",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "orientation": "horizontal",
        "textMode": "auto",
        "colorMode": "none",
        "graphMode": "none",
        "justifyMode": "auto"
      }
    },
    {
      "id": 11,
      "type": "table",
      "title": "Usługi",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 10,
        "w": 12,
        "x": 0,
        "y": 36
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (name, state) (node_systemd_unit_state) == 1",
          "format": "table",
          "instant": true
        }
      ],
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto",
            "filterable": false
          }
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "Stan"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "type": "color-text"
                }
              },
              {
                "id": "mappings",
                "value": [
                  {
                    "type": "value",
                    "options": {
                      "active": {
                        "color": "green",
                        "index": 0,
                        "text": "działa"
                      }
                    }
                  },
                  {
                    "type": "value",
                    "options": {
                      "failed": {
                        "color": "red",
                        "index": 1,
                        "text": "błąd"
                      }
                    }
                  },
                  {
                    "type": "value",
                    "options": {
                      "inactive": {
                        "color": "orange",
                        "index": 2,
                        "text": "wyłączona"
                      }
                    }
                  },
                  {
                    "type": "value",
                    "options": {
                      "activating": {
                        "color": "yellow",
                        "index": 3,
                        "text": "startuje"
                      }
                    }
                  }
                ]
              }
            ]
          }
        ]
      },
      "options": {
        "showHeader": true,
        "cellHeight": "sm",
        "footer": {
          "show": false,
          "reducer": [
            "sum"
          ],
          "countRows": false
        }
      },
      "transformations": [
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true,
              "Value": true,
              "__name__": true,
              "instance": true,
              "job": true,
              "type": true
            },
            "renameByName": {
              "name": "Usługa",
              "state": "Stan"
            }
          }
        },
        {
          "id": "sortBy",
          "options": {
            "sort": [
              {
                "field": "Usługa",
                "desc": false
              }
            ]
          }
        }
      ]
    },
    {
      "id": 12,
      "type": "bargauge",
      "title": "Największe katalogi",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 10,
        "w": 12,
        "x": 12,
        "y": 36
      },
      "targets": [
        {
          "refId": "A",
          "expr": "topk(12, max by (path) (host_directory_size_bytes))",
          "legendFormat": "{{path}}",
          "instant": true
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "bytes",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "blue",
                "value": null
              }
            ]
          }
        },
        "overrides": []
      },
      "options": {
        "orientation": "horizontal",
        "displayMode": "gradient",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showUnfilled": false,
        "valueMode": "color",
        "minVizWidth": 0,
        "minVizHeight": 10
      }
    },
    {
      "id": 13,
      "type": "table",
      "title": "Zalogowani",
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 46
      },
      "targets": [
        {
          "refId": "A",
          "expr": "max by (user, from, service) (host_logged_in_sessions) == 1",
          "format": "table",
          "instant": true
        }
      ],
      "fieldConfig": {
        "defaults": {
          "custom": {
            "align": "auto"
          }
        },
        "overrides": []
      },
      "options": {
        "showHeader": true,
        "cellHeight": "sm",
        "footer": {
          "show": false,
          "reducer": [
            "sum"
          ],
          "countRows": false
        }
      },
      "transformations": [
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true,
              "Value": true,
              "__name__": true,
              "instance": true,
              "job": true
            },
            "renameByName": {
              "user": "Użytkownik",
              "from": "Skąd",
              "service": "Przez"
            }
          }
        }
      ]
    }
  ],
  "refresh": "30s",
  "schemaVersion": 39,
  "tags": [
    "serwer"
  ],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "Europe/Warsaw",
  "title": "Serwer",
  "uid": "serwer-home",
  "version": 1
}

7. Certyfikat

Lokalny certyfikat na 10 lat, z oboma adresami IP. UID 472 to użytkownik wewnątrz obrazu Grafany. Bez tego kontener nie przeczyta klucza.

openssl req -x509 -newkey rsa:2048 -sha256 -days 3650 -nodes \
  -keyout /home/devops/docker/monitoring/grafana/certs/key.pem \
  -out /home/devops/docker/monitoring/grafana/certs/cert.pem \
  -subj "/CN=192.168.1.10" \
  -addext "subjectAltName=IP:192.168.1.10,IP:10.10.0.1,IP:127.0.0.1,DNS:localhost"
sudo chown 472:472 /home/devops/docker/monitoring/grafana/certs/*.pem
sudo chmod 644 /home/devops/docker/monitoring/grafana/certs/cert.pem
sudo chmod 640 /home/devops/docker/monitoring/grafana/certs/key.pem

8. docker-compose.yml

Zapisz jako /home/devops/docker/docker-compose.yml, jeśli stawiasz sam monitoring. Hasło ZMIEN_MNIE podmień przed startem (openssl rand -hex 12). Grafana zapisze je przy pierwszym utworzeniu wolumenu. Późniejsza edycja tej linii hasła już nie zmieni.

Jeśli w tym pliku masz już inne usługi, nie kasuj ich. Wklej cztery usługi prometheus, node-exporter, host-extras, grafana oraz wolumeny prometheus-data i grafana-data. Sieć nazwij tak, jak reszta stosu, i tę samą nazwę wpisz przy Prometheusie i Grafanie. node-exporter i host-extras zostają z network_mode: host i nie wchodzą do tej sieci.

Porty Grafany są przywiązane do konkretnych adresów, nie do 0.0.0.0. Prometheus i eksporter nie mają portów wystawionych do sieci domowej.

Lista usług w unit-include to jednostki systemd bez końcowego znaku dolara. Compose zjada $, więc regex kończy się na .service i tyle. Nową aplikację dopisujesz w tym nawiasie, na przykład mojaapp.

services:
  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    networks:
      - monitoring
    depends_on:
      - node-exporter

  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: unless-stopped
    user: "0:0"
    pid: host
    network_mode: host
    command:
      - --path.procfs=/host/proc
      - --path.sysfs=/host/sys
      - --path.rootfs=/rootfs
      - --web.listen-address=:9100
      - --collector.systemd
      - --collector.systemd.unit-include=^(nginx|docker|smbd|nmbd|plexmediaserver|ssh|cron|baseauth|battleship|ppl|quizhub|wheel|qbittorrent-nox@bb)\.service
      - --collector.textfile.directory=/textfile
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
      - /run/systemd/private:/run/systemd/private
      - ./monitoring/textfile:/textfile:ro

  host-extras:
    image: alpine:latest
    container_name: host-extras
    restart: unless-stopped
    network_mode: host
    entrypoint: ["/bin/sh", "/collect.sh"]
    volumes:
      - /:/host:ro
      - ./monitoring/collect.sh:/collect.sh:ro
      - ./monitoring/textfile:/textfile

  grafana:
    image: grafana/grafana:11.4.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "127.0.0.1:9090:3000"
      - "192.168.1.10:9090:3000"
      - "10.10.0.1:9090:3000"
    environment:
      TZ: Europe/Warsaw
      GF_DEFAULT_TIMEZONE: Europe/Warsaw
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: "ZMIEN_MNIE"
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_ANALYTICS_REPORTING_ENABLED: "false"
      GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
      GF_SECURITY_DISABLE_GRAVATAR: "true"
      GF_SMTP_ENABLED: "true"
      GF_SMTP_HOST: smtp.gmail.com:587
      GF_SMTP_USER: pepisboczek@gmail.com
      GF_SMTP_PASSWORD: "ZMIEN_MNIE"
      GF_SMTP_FROM_ADDRESS: pepisboczek@gmail.com
      GF_SMTP_FROM_NAME: Serwer
      GF_SMTP_STARTTLS_POLICY: MandatoryStartTLS
      GF_USERS_DEFAULT_THEME: dark
      GF_SERVER_PROTOCOL: https
      GF_SERVER_CERT_FILE: /etc/grafana/certs/cert.pem
      GF_SERVER_CERT_KEY: /etc/grafana/certs/key.pem
      GF_SECURITY_COOKIE_SECURE: "true"
      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/serwer.json
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana/provisioning:/etc/grafana/provisioning:ro
      - ./monitoring/grafana/dashboards:/var/lib/grafana/dashboards:ro
      - ./monitoring/grafana/certs:/etc/grafana/certs:ro
    networks:
      - monitoring
    depends_on:
      - prometheus

volumes:
  prometheus-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

Alerty mailowe

Grafana sama wysyła maila, kiedy reguła jest spełniona przez podany czas. Nadawca to konto Gmail, odbiorca boguslaw.boczkowski@gmail.com. Hasło w Compose to hasło aplikacji Gmail (16 znaków z myślnikami albo bez), nie zwykłe hasło do skrzynki. W pliku zostawiasz ZMIEN_MNIE i podmieniasz je przed startem. Port 587, STARTTLS.

Folder alertów nazywa się Alerty. Nie może nazywać się tak samo jak dashboard (Serwer), bo Grafana wtedy nie wstanie.

Ten sam problem powtarza mail co 4 godziny, dopóki warunek trwa. Po ustaniu Grafana wysyła maila, że alert się skończył. Pierwszy start skryptu zapisuje hashe torrentów, które już są ukończone, do textfile/qbit-finished.hashes i ich nie zgłasza. Zgłoszenie dotyczy tylko nowego ukończenia i wisi 30 minut.

/home/devops/docker/monitoring/grafana/provisioning/alerting/rules.yml

apiVersion: 1
groups:
  - orgId: 1
    name: serwer
    folder: Alerty
    interval: 1m
    rules:
      - uid: disk_free_20gb
        title: Mało wolnego miejsca
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: 'node_filesystem_avail_bytes{mountpoint="/",fstype=~"ext4|xfs"}'
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 21474836480
                    type: lt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        executionErrorState: Error
        for: 5m
        annotations:
          summary: "Na dysku systemowym zostało mniej niż 20 GB."
        isPaused: false

      - uid: service_down
        title: Usługa nie działa
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: 'node_systemd_unit_state{state="active"}'
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 1
                    type: lt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        executionErrorState: Error
        for: 3m
        annotations:
          summary: "Usługa {{ $labels.name }} nie jest w stanie active."
        isPaused: false

      - uid: cpu_high
        title: Wysokie zużycie CPU
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: '(100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100))'
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 90
                    type: gt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        executionErrorState: Error
        for: 15m
        annotations:
          summary: "CPU powyżej 90% przez 15 minut."
        isPaused: false

      - uid: temp_high
        title: Wysoka temperatura CPU
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: 'max(node_hwmon_temp_celsius{chip="platform_coretemp_0"})'
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 85
                    type: gt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        executionErrorState: Error
        for: 5m
        annotations:
          summary: "Temperatura CPU powyżej 85°C."
        isPaused: false

      - uid: ram_low
        title: Mało wolnego RAM
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: node_memory_MemAvailable_bytes
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 536870912
                    type: lt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: NoData
        executionErrorState: Error
        for: 10m
        annotations:
          summary: "Wolnego RAM zostało mniej niż 512 MB."
        isPaused: false

      - uid: exporter_down
        title: Brak metryk z serwera
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: 'up{job="node"}'
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 1
                    type: lt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: Alerting
        executionErrorState: Alerting
        for: 2m
        annotations:
          summary: "Prometheus nie zbiera metryk hosta."
        isPaused: false

      - uid: torrent_done
        title: Torrent pobrany
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              datasource:
                type: prometheus
                uid: prometheus
              editorMode: code
              expr: qbit_download_just_finished
              instant: true
              intervalMs: 1000
              maxDataPoints: 43200
              refId: A
          - refId: C
            relativeTimeRange:
              from: 0
              to: 0
            datasourceUid: __expr__
            model:
              conditions:
                - evaluator:
                    params:
                      - 0
                    type: gt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    type: last
                    params: []
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: A
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: threshold
        noDataState: OK
        executionErrorState: OK
        for: 1m
        annotations:
          summary: "Ukończono pobieranie: {{ $labels.name }}."
        isPaused: false

/home/devops/docker/monitoring/grafana/provisioning/alerting/contact.yml

apiVersion: 1
contactPoints:
  - orgId: 1
    name: email-serwer
    receivers:
      - uid: email_serwer
        type: email
        settings:
          addresses: boguslaw.boczkowski@gmail.com
          singleEmail: true

/home/devops/docker/monitoring/grafana/provisioning/alerting/policies.yml

apiVersion: 1
policies:
  - orgId: 1
    receiver: email-serwer
    group_by:
      - grafana_folder
      - alertname
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

Po starcie: Alerting, Contact points, email-serwer, Test. Ma przyjść mail z tematem o teście.

9. Firewall

Bind na prywatny adres nie zatrzyma przekierowania portu na routerze. Skrypt puszcza 9090 tylko z domu (192.168.1.0/24), WireGuarda (10.10.0.0/24) i localhosta. Port 9100 (surowe metryki, bez strony) tylko z localhosta i z sieci Dockera. IPv6 na obu portach jest odcięte.

Zapisz jako /home/devops/docker/monitoring/lan-only-9090.sh i chmod 755.

#!/bin/sh
# Port 9090 tylko z LAN, WireGuard i localhosta.
set -eu

iptables -N GRAFANA-LAN 2>/dev/null || true
iptables -F GRAFANA-LAN
iptables -A GRAFANA-LAN -s 192.168.1.0/24 -j ACCEPT
iptables -A GRAFANA-LAN -s 10.10.0.0/24 -j ACCEPT
iptables -A GRAFANA-LAN -s 127.0.0.0/8 -j ACCEPT
iptables -A GRAFANA-LAN -j DROP

if ! iptables -C INPUT -p tcp --dport 9090 -j GRAFANA-LAN 2>/dev/null; then
  iptables -I INPUT 1 -p tcp --dport 9090 -j GRAFANA-LAN
fi

if ! ip6tables -C INPUT -p tcp --dport 9090 -j DROP 2>/dev/null; then
  ip6tables -I INPUT 1 -p tcp --dport 9090 -j DROP
fi

# node_exporter (9100) tylko z localhosta i sieci Dockera, nie z LAN ani z internetu.
iptables -N NODEEXP-LOCAL 2>/dev/null || true
iptables -F NODEEXP-LOCAL
iptables -A NODEEXP-LOCAL -s 127.0.0.0/8 -j ACCEPT
iptables -A NODEEXP-LOCAL -s 172.16.0.0/12 -j ACCEPT
iptables -A NODEEXP-LOCAL -j DROP
if ! iptables -C INPUT -p tcp --dport 9100 -j NODEEXP-LOCAL 2>/dev/null; then
  iptables -I INPUT 1 -p tcp --dport 9100 -j NODEEXP-LOCAL
fi
if ! ip6tables -C INPUT -p tcp --dport 9100 -j DROP 2>/dev/null; then
  ip6tables -I INPUT 1 -p tcp --dport 9100 -j DROP
fi

Żeby reguły wróciły po restarcie, zapisz /etc/systemd/system/grafana-lan-only.service:

[Unit]
Description=Restrict Grafana port 9090 to the home LAN and WireGuard
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/home/devops/docker/monitoring/lan-only-9090.sh

[Install]
WantedBy=multi-user.target

sudo chmod 755 /home/devops/docker/monitoring/lan-only-9090.sh
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-lan-only.service

10. Start i sprawdzenie

cd /home/devops/docker
docker compose up -d
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

Grafana na liście portów ma mieć tylko 127.0.0.1:9090, 192.168.1.10:9090 i 10.10.0.1:9090. Prometheus bez opublikowanego portu.

curl -sk -o /dev/null -w 'grafana %{http_code}\n' https://192.168.1.10:9090/api/health
docker exec prometheus wget -qO- http://127.0.0.1:9090/api/v1/targets
cat /home/devops/docker/monitoring/textfile/qbit.prom

• grafana 200 oznacza, że HTTPS wstaje.
• W targetach Prometheusa prometheus i node mają być up.
• qbit.prom pojawia się po kilku sekundach. Pusty albo brak pliku: Web UI qBittorrenta nie słucha na 127.0.0.1:8080 albo wymaga logowania także z localhosta.
• W przeglądarce: https://192.168.1.10:9090, zaakceptuj certyfikat, zaloguj się jako admin hasłem z Compose. Dashboard nazywa się Serwer.
• W Grafanie, folder Alerty: siedem reguł. Test kontaktu email-serwer kończy się mailem.

Gdy panel jest pusty albo krzywy

• Usługi puste, w logu dial unix /run/systemd/private: brak montowania tego gniazda w node-exporterze.
• Sieć pokazuje eth0 zamiast enp1s0: node-exporter nie ma network_mode: host.
• Przeglądarka nie łączy się po HTTPS: w logu Grafany ma być protocol=https. Sprawdź ścieżki certyfikatu i właściciela 472.
• Z domu nie wchodzi, z VPN wchodzi: w firewallu nie ma sieci 192.168.1.0/24 albo port jest zbindowany tylko na adres VPN.
• Katalogi puste przez pierwsze minuty: du jeszcze liczy. Odśwież za chwilę.