️ PMM Disaster Recovery

If you like this project, consider supporting me on Buy Me a Coffee ☕️

tags:

• databases
• sysadmin

• linux

  PMM Disaster Recovery — Full Production Recovery Cheatsheet

Percona Monitoring and Management (PMM) is an open-source database monitoring, management, and observability solution by Percona. It bundles Prometheus, Grafana, VictoriaMetrics, ClickHouse, and PostgreSQL (QAN) into a single Docker-based appliance for MySQL, PostgreSQL, MongoDB, and ProxySQL.

Common use cases / Типичные сценарии: Database performance monitoring, query analytics (QAN), slow query analysis, replication monitoring, backup monitoring, security threat detection.

Status / Статус: Actively maintained by Percona (current: PMM 3.x). Alternatives: Datadog (commercial SaaS), Grafana Cloud (managed), Zabbix (general-purpose), New Relic (APM).

Default port / Порт по умолчанию: 443/tcp (HTTPS)

📚 Table of Contents

• Architecture
• Installation & Configuration
• Core Management
• Sysadmin Operations
• Security
• Backup & Restore
• Troubleshooting & Tools
• Production Runbooks
• Disaster Recovery Case Study
• Logrotate Configuration
• Additional Notes
• Official Documentation

Architecture

PMM Internal Components

Service Dependency Chain

PostgreSQL → Grafana datasource init → PMM plugin init → PMM UI available

[!IMPORTANT] If PostgreSQL fails, the entire PMM UI cascade will fail. Grafana depends on PostgreSQL for datasource configuration. Если PostgreSQL не стартует, весь каскад PMM UI выйдет из строя.

Internal Paths

Default Ports

Installation & Configuration

Volume Strategy

[!WARNING] Always use named volumes, never bind mounts. Bind mounts cause data corruption if the compose directory is copied. Всегда используйте именованные тома, никогда bind mounts.

Production docker-compose.yml

/opt/pmm/docker-compose.yml

version: '3.8'

services:
  pmm-server:
    image: percona/pmm-server:3.5.0
    container_name: pmm-server
    restart: always
    ports:
      - "443:443"
    environment:
      GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: pmm-app,pmm-qan-app-panel
      PMM_QAN_API2__SERVICE_TYPE: none
    volumes:
      - pmm-data:/srv

volumes:
  pmm-data:

Start Services

docker compose up -d       # Start PMM / Запустить PMM
docker compose ps          # Check status / Проверить статус
docker compose logs -f     # Follow logs / Следить за логами

PMM User ID

docker run --rm percona/pmm-server:3 id pmm  # Check UID / Проверить UID
# Output: uid=1000(pmm) gid=1000(pmm)

[!NOTE] PMM runs as UID 1000. All /srv data must be owned by 1000:1000.

Core Management

Internal Service Control

docker exec pmm-server supervisorctl status              # List services / Список сервисов
docker exec pmm-server supervisorctl restart all          # Restart all / Перезапустить все
docker exec pmm-server supervisorctl restart postgresql   # Restart PostgreSQL
docker exec pmm-server supervisorctl restart grafana      # Restart Grafana

Health Checks

curl -k https://localhost/pmm/server-status                # PMM status / Статус PMM
curl -k https://localhost/grafana/api/health               # Grafana health / Здоровье Grafana
curl -k -I https://localhost/pmm-ui/graph/                 # PMM UI check / Проверка PMM UI

Sysadmin Operations

Log Locations

docker exec pmm-server tail -f /srv/logs/grafana.log      # Follow Grafana log / Логи Grafana
docker exec pmm-server tail -f /srv/logs/postgresql.log    # Follow PG log / Логи PostgreSQL

Docker Volume Inspection

docker volume ls                                           # List volumes / Список томов
docker volume inspect pmm-data                             # Inspect PMM volume / Инспекция тома
docker inspect pmm-server | jq '.[0].Mounts'               # Check mounts / Монтирования

Shell Access

docker exec -it pmm-server bash                            # Enter container / Войти в контейнер
mount | grep srv                                           # Check mounts / Точки монтирования
ls -lah /srv/postgres14                                    # Check PG data / Данные PostgreSQL

Network & Firewall

sudo ufw allow 443/tcp                                    # UFW: allow PMM / Разрешить PMM
sudo firewall-cmd --permanent --add-port=443/tcp && sudo firewall-cmd --reload  # firewalld

Security

Grafana Password Reset

docker exec pmm-server grafana-cli \
  --config /etc/grafana/grafana.ini \
  --homepath /usr/share/grafana \
  admin reset-admin-password '<PASSWORD>'                  # Reset password / Сброс пароля

[!CAUTION] Change the default PMM admin password immediately after deployment. Смените пароль по умолчанию сразу после развёртывания.

Unsigned Plugins

# Required in docker-compose.yml
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: pmm-app,pmm-qan-app-panel

Backup & Restore

Full Backup

docker run --rm \
  -v pmm-data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/pmm-backup-$(date +%Y%m%d_%H%M%S).tar.gz /data

Restore

docker compose down                                        # Stop PMM / Остановить PMM

docker run --rm \
  -v pmm-data:/data \
  -v $(pwd):/backup \
  alpine sh -c 'cd / && tar xzf /backup/pmm-backup-<TIMESTAMP>.tar.gz'

docker compose up -d                                       # Start PMM / Запустить PMM

Automated Backup Script

/usr/local/bin/pmm-backup.sh

#!/bin/bash
BACKUP_DIR="/backup/pmm"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/pmm-backup-$TIMESTAMP.tar.gz"
mkdir -p "$BACKUP_DIR"

if docker run --rm \
  -v pmm-data:/data \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf "/backup/pmm-backup-$TIMESTAMP.tar.gz" /data; then

  if [ -s "$BACKUP_FILE" ] && tar -tzf "$BACKUP_FILE" >/dev/null 2>&1; then
    find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete     # Retain 7 days / Хранить 7 дней
    echo "✅ PMM backup successful: $BACKUP_FILE"
  else
    echo "❌ Backup validation failed. Archive is empty or corrupt."
    exit 1
  fi
else
  echo "❌ Docker tar command failed."
  exit 1
fi

Cron:

0 3 * * * /usr/local/bin/pmm-backup.sh >> /var/log/pmm-backup.log 2>&1

Troubleshooting & Tools

Diagnostic Commands

docker logs pmm-server | grep -Ei 'postgres|grafana|error|panic|wal|permission'  # Find errors / Поиск ошибок
docker exec pmm-server supervisorctl status                                       # Service status / Статус сервисов
docker volume inspect pmm-data                                                    # Verify volume / Проверка тома

Common Issues

Production Runbooks

Runbook: Full PMM Emergency Recovery

[!CAUTION] This involves destructive operations on PostgreSQL WAL. Ensure backups exist first. Убедитесь в наличии бэкапов перед началом.

1. Stop PMM / Остановить PMM:

   docker compose down

2. Fix ownership / Исправить владельца:

   docker run --rm -v pmm-data:/data alpine sh -c '
   chown -R 1000:1000 /data/postgres14
   chown -R 1000:1000 /data/grafana
   chown -R 1000:1000 /data/prometheus
   '

3. Fix permissions / Исправить права:

   docker run --rm -v pmm-data:/data alpine sh -c '
   chmod 700 /data/postgres14
   chmod 644 /data/postgres14/postgresql.conf
   chmod 644 /data/postgres14/pg_hba.conf
   '

4. Remove stale PID / Удалить PID:

   docker run --rm -v pmm-data:/data alpine sh -c '
   rm -f /data/postgres14/postmaster.pid
   rm -f /data/postgres14/postmaster.opts
   '

5. Reset WAL (if needed) / Сброс WAL:

   docker run --rm -u 1000 \
   -v pmm-data:/data \
   postgres:14 bash -c 'cd /data/postgres14 && pg_resetwal -f .'

6. Start PMM / Запустить PMM:

   docker compose up -d

7. Reset Grafana password / Сброс пароля Grafana:

   docker exec pmm-server grafana-cli \
     --config /etc/grafana/grafana.ini \
     --homepath /usr/share/grafana \
     admin reset-admin-password '<PASSWORD>'

8. Verify / Проверить:

   curl -k https://localhost/pmm/server-status
   curl -k https://localhost/grafana/api/health
   curl -k -I https://localhost/pmm-ui/graph/
   docker exec pmm-server supervisorctl status

Runbook: PostgreSQL WAL Recovery

[!WARNING] pg_resetwal may cause data inconsistency. Use only when PostgreSQL refuses to start.

1. Reset WAL:

   docker run --rm -u 1000 -v pmm-data:/data \
   postgres:14 bash -c 'cd /data/postgres14 && pg_resetwal -f .'

2. Remove WAL files (if reset fails):

   docker run --rm -v pmm-data:/data alpine sh -c 'rm -rf /data/postgres14/pg_wal/*'

3. Full PostgreSQL removal (last resort):

   [!CAUTION] Deletes all QAN data. Prometheus metrics are preserved.

   docker run --rm -v pmm-data:/data alpine sh -c 'rm -rf /data/postgres14'

Runbook: Volume Migration (Bind → Named)

1. Stop deployment:

   docker compose down

2. Create named volume:

   docker volume create pmm-data

3. Copy data:

   docker run --rm -v $(pwd)/data:/source:ro -v pmm-data:/dest \
   alpine sh -c 'cp -a /source/. /dest/'

4. Update docker-compose.yml from ./data:/srv to pmm-data:/srv
5. Start and verify:

   docker compose up -d
   docker exec pmm-server supervisorctl status

Recovery Timeline

Disaster Recovery Case Study

What Happened

PMM 3.x was running in Docker Compose with ~33 GB Prometheus metrics, active Grafana dashboards, and QAN data.

A second deployment was started from a copied directory using bind mounts (./data:/srv). Both deployments wrote to the same PostgreSQL data simultaneously.

Root Cause

Two PMM instances accessing the same PostgreSQL data directory caused WAL corruption, invalid checkpoints, and cascading PMM failure.

Symptoms

PostgreSQL:  PANIC: could not locate valid checkpoint record
Grafana:     {"code":7,"error":"Access denied"}
PMM UI:      404 page not found

Resolution

Recovered using the Emergency Recovery runbook above. Migrated to named volumes.

Lessons Learned

1. Named volumes > bind mounts — isolated per project / Изолированы по проекту
2. PMM runs as UID 1000 — all /srv data must be 1000:1000
3. QAN is secondary — Prometheus metrics are the priority
4. PMM UI depends on Grafana plugins — unsigned plugins must be allowed
5. WAL corruption is common after dirty shutdowns
6. docker volume inspect should always be step #1

Logrotate Configuration

[!NOTE] PMM logs are inside Docker at /srv/logs/. Use logrotate on host only if logs are mounted out.

/etc/logrotate.d/pmm-server

/opt/pmm/logs/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Additional Notes

Validation Checklist

• [ ] curl -k https://localhost/pmm/server-status → 200
• [ ] curl -k https://localhost/grafana/api/health → OK
• [ ] curl -k -I https://localhost/pmm-ui/graph/ → 200
• [ ] supervisorctl status → all RUNNING
• [ ] Named volumes used (not bind mounts)
• [ ] Admin password changed from default
• [ ] Backup cron configured

Production Tips

• Use named Docker volumes for all PMM data
• Configure daily backups with retention
• Monitor disk space on PMM host
• Set up alerting for PMM failures
• Test disaster recovery periodically

Official Documentation

• Percona PMM Docs: https://docs.percona.com/percona-monitoring-and-management/
• PMM Docker Setup: https://docs.percona.com/percona-monitoring-and-management/setting-up/server/docker.html
• PMM Backup & Restore: https://docs.percona.com/percona-monitoring-and-management/how-to/backup.html
• PMM GitHub: https://github.com/percona/pmm
• Percona Forum: https://forums.percona.com/
• PostgreSQL pg_resetwal: https://www.postgresql.org/docs/14/app-pgresetwal.html