infra/@dev-prod-rf.server/management/README.md

Management Guide: @infra-dev-prod-rf

Quick Reference for Operating 7 Docker Stacks
Status: PRODUCTION ACTIVE
Last Updated: 2025-11-09

Quick Start

Access Server

# From DEV-PRO (91.218.142.168)
ssh root@45.144.177.147

# Direct SSH (if available)
ssh root@45.144.177.147
# Password: mjrtY*q5SVlt (backup only)

Health Check

# Quick status
docker ps --format "table {{.Names}}\t{{.Status}}"

# Resource usage
docker stats --no-stream

# Server resources
free -h && df -h

Check seller1.ru (PRODUCTION)

# Service status
docker ps | grep seller1-prod

# Resource usage
docker stats seller1-prod-web seller1-prod-db

# Website availability
curl -I https://seller1.ru

# Logs
cd /opt/seller1-prod-stack && docker compose logs --tail 50

Stack Management

Stack Locations

/opt/pro-0kt-stack/              # Infrastructure (16 containers)
/opt/seller1-prod-stack/         # Production (4 containers) 🔴
/opt/seller1-dev-stack/          # Development (4 containers)
/opt/varbase-stack/              # Demo (3 containers)
/opt/commerce-stack/             # Demo (3 containers)
/opt/myshop-stack/               # Demo (3 containers)
/opt/python-stack/               # Apps (2 containers)

Common Commands

# View stack status
cd /opt/<stack-name>
docker compose ps

# View stack configuration
docker compose config

# Start/Stop stack
docker compose up -d
docker compose down

# Restart all services in stack
docker compose restart

# Restart specific service
docker compose restart <service-name>

# Update images and restart
docker compose pull
docker compose up -d

Docker Commands Reference

View Containers

# All containers with status
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# All containers (including stopped)
docker ps -a

# Container details
docker inspect <container-name>

# Running processes
docker top <container-name>

View Logs

# Last 100 lines
docker compose logs <service> --tail 100

# Follow logs (real-time)
docker compose logs -f <service>

# Direct container logs
docker logs <container-name>

# With timestamps
docker logs --timestamps <container-name>

# Last 1 hour
docker logs --since 1h <container-name>

# Filter by pattern
docker logs <container-name> | grep ERROR

Resource Monitoring

# Real-time stats
docker stats

# Without streaming
docker stats --no-stream

# Specific containers
docker stats seller1-prod-web seller1-prod-db

# Disk usage
docker system df

# Remove unused data
docker system prune -f
docker system prune -a -f --volumes  # ⚠️ Destructive!

Container Operations

# Start container
docker start <container-name>

# Stop container
docker stop <container-name>

# Restart container
docker restart <container-name>

# Kill container (force stop)
docker kill <container-name>

# Remove container
docker rm <container-name>

# Execute command in container
docker exec <container-name> <command>

# Interactive shell
docker exec -it <container-name> bash

Service Management

Infrastructure Stack (pro-0kt-stack)

cd /opt/pro-0kt-stack

# Status
docker compose ps

# Restart specific services
docker compose restart pro-0kt-nginx         # Reverse proxy
docker compose restart pro-0kt-postgres      # Shared database
docker compose restart pro-0kt-redis         # Shared cache
docker compose restart pro-0kt-authelia      # OAuth
docker compose restart pro-0kt-homarr        # Dashboard

# Logs
docker compose logs -f pro-0kt-nginx
docker compose logs -f pro-0kt-postgres

Production Stack (seller1-prod-stack) 🔴

cd /opt/seller1-prod-stack

# Status
docker compose ps

# Check production web
docker compose logs -f seller1-prod-web

# Check database
docker compose logs -f seller1-prod-db
docker exec seller1-prod-db pg_isready

# Restart (CONFIRM FIRST!)
docker compose restart seller1-prod-web

# DO NOT do without confirmation:
# docker compose down
# docker compose restart seller1-prod-db

Development Stacks

# seller1-dev-stack
cd /opt/seller1-dev-stack
docker compose ps
docker compose logs seller1-dev-web

# varbase-stack
cd /opt/varbase-stack
docker compose ps

# commerce-stack
cd /opt/commerce-stack
docker compose ps

# myshop-stack
cd /opt/myshop-stack
docker compose ps

# python-stack
cd /opt/python-stack
docker compose ps

Network Management

View Networks

# List all networks
docker network ls

# Inspect network
docker network inspect <network-name>

# Check NPM connections
docker network inspect pro-0kt-stack_pro-0kt-network | grep pro-0kt-nginx

NPM Network Connectivity

Verify NPM is connected to all networks:

docker network inspect pro-0kt-stack_pro-0kt-network | grep pro-0kt-nginx
docker network inspect seller1-prod-stack_seller1-prod-network | grep pro-0kt-nginx
docker network inspect seller1-dev-stack_seller1-dev-network | grep pro-0kt-nginx
docker network inspect myshop-stack_myshop-network | grep pro-0kt-nginx
docker network inspect commerce-stack_commerce-network | grep pro-0kt-nginx
docker network inspect varbase-stack_varbase-network | grep pro-0kt-nginx
docker network inspect python-stack_python-network | grep pro-0kt-nginx

Reconnect NPM if Needed

docker network connect <network-name> pro-0kt-nginx
docker exec pro-0kt-nginx nginx -s reload

Database Management

PostgreSQL

# Shared PostgreSQL (infrastructure)
docker exec pro-0kt-postgres psql -U postgres -d postgres -c "\l"

# seller1-prod database
docker exec seller1-prod-db psql -U postgres -d seller1_prod -c "SELECT version();"

# Health check
docker exec seller1-prod-db pg_isready

# Backup
docker exec seller1-prod-db pg_dumpall -U postgres > backup.sql

# Restore
docker exec -i seller1-prod-db psql -U postgres < backup.sql

MariaDB

# Shared MariaDB (infrastructure)
docker exec pro-0kt-mariadb mysql -u root -p$MYSQL_ROOT_PASSWORD -e "SHOW DATABASES;"

# Health check
docker exec pro-0kt-mariadb mysqladmin ping -u root -p$MYSQL_ROOT_PASSWORD

# Backup
docker exec pro-0kt-mariadb mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > backup.sql

# Restore
docker exec -i pro-0kt-mariadb mysql -u root -p$MYSQL_ROOT_PASSWORD < backup.sql

Redis

# Shared Redis
docker exec pro-0kt-redis redis-cli ping
docker exec pro-0kt-redis redis-cli info

# Check memory
docker exec pro-0kt-redis redis-cli info memory

# Clear cache
docker exec pro-0kt-redis redis-cli FLUSHALL

Storage Management

View Data Locations

# Show all persistent data
ls -lah /mnt/data/

# Show sizes
du -sh /mnt/data/* | sort -h

# Largest directories
du -sh /mnt/data/* | sort -rh | head -10

Disk Usage Monitoring

# Total disk usage
df -h

# Inodes usage
df -i

# Docker disk usage
docker system df

# Container-specific data
du -sh /mnt/data/seller1-prod/
du -sh /mnt/data/varbase/

Cleanup (Carefully!)

# Remove unused docker resources
docker system prune -f

# Remove unused images
docker image prune -f

# Remove stopped containers
docker container prune -f

# ⚠️ DANGEROUS: Remove ALL unused (including volumes)
docker system prune -a -f --volumes

# Clean old backups
find /var/backups/daily/ -mtime +7 -delete
find /var/backups/weekly/ -mtime +30 -delete

Backup & Recovery

Automated Backups

# Run daily backup script
/opt/scripts/backup-critical.sh

# Check backup status
ls -lah /var/backups/daily/
ls -lah /var/backups/weekly/
ls -lah /var/backups/monthly/

Manual Database Backup

# All PostgreSQL databases
docker exec pro-0kt-postgres pg_dumpall -U postgres > /var/backups/postgres-full-$(date +%Y%m%d).sql

# Specific database
docker exec seller1-prod-db pg_dump -U postgres seller1_prod > /var/backups/seller1-$(date +%Y%m%d).sql

# All MariaDB databases
docker exec pro-0kt-mariadb mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /var/backups/mariadb-full-$(date +%Y%m%d).sql

Volume Backup

# Backup seller1-prod data
tar czf /var/backups/seller1-prod-$(date +%Y%m%d).tar.gz /mnt/data/seller1-prod/

# Backup all data
tar czf /var/backups/all-data-$(date +%Y%m%d).tar.gz /mnt/data/

Restore Database

# PostgreSQL restore
docker exec -i seller1-prod-db psql -U postgres < backup.sql

# MariaDB restore
docker exec -i pro-0kt-mariadb mysql -u root -p$MYSQL_ROOT_PASSWORD < backup.sql

# Volume restore
cd / && tar xzf /var/backups/seller1-prod-backup.tar.gz

Monitoring & Health Checks

System Health

# Overall system status
uptime
free -h
df -h
ps aux | grep docker

# Docker daemon status
docker version
docker info

# Container health
docker ps --format "table {{.Names}}\t{{.Status}}"

Service Monitoring

Dashboard: https://home.0kt.ru (Homarr)
Uptime Monitoring: https://status.0kt.ru (Uptime Kuma)
Docker UI: https://portainer.0kt.ru (Portainer)

Health Check Script

# Run health check
/opt/scripts/health-check.sh

# View health check logs
tail -f /var/log/health-checks/latest.log

Monitor Production Service

# seller1.ru website check
curl -I https://seller1.ru

# Backend API check
curl -I https://pro.seller1.ru

# DNS resolution
nslookup seller1.ru

# All production containers
docker ps | grep seller1-prod
docker stats seller1-prod-*

SSL Certificate Management

View Current Certificates

# List certificate locations
ls -la /mnt/data/nginx/letsencrypt/live/

# Check certificate expiration
openssl x509 -in /mnt/data/nginx/letsencrypt/live/seller1.ru/cert.pem -noout -dates

# Check all certificates
for domain in /mnt/data/nginx/letsencrypt/live/*/; do
  echo "Domain: $(basename $domain)"
  openssl x509 -in "$domain/cert.pem" -noout -dates
done

SSL Certificate Renewal

# Automated renewal (via NPM)
# NPM auto-renews 30 days before expiration

# Manual renewal in NPM Admin
# https://npm.0kt.ru or http://localhost:81
# Edit proxy host > SSL > Request New SSL Certificate

# Force NPM renewal
docker compose -f /opt/pro-0kt-stack/docker-compose.yml restart pro-0kt-nginx

Adding New Domain with SSL

  1. Configure DNS A record → 45.144.177.147
  2. Wait for DNS propagation (10-60 minutes)
  3. Login to NPM Admin: http://localhost:81 (via SSH tunnel)
  4. Add Proxy Host:
    - Domain: newdomain.tld
    - Forward Host: container-name
    - Forward Port: service-port
    - Enable: Websockets Support, Block Common Exploits
  5. Request SSL Certificate:
    - Force SSL: Yes
    - HTTP/2 Support: Yes
  6. Test: https://newdomain.tld

Troubleshooting

Container Won't Start

# View logs
docker logs <container-name>

# Extended logs
cd /opt/<stack> && docker compose logs <service> --tail 200

# Validate configuration
docker compose config

# Check .env file
cat /opt/<stack>/.env

# Try restart
docker compose down && docker compose up -d

502 Bad Gateway

# Check backend container
docker ps | grep <container>

# Check container logs
docker logs <container> --tail 100

# Verify network connectivity
docker network inspect <network-name> | grep <container>

# Reconnect to network if needed
docker network connect <network> <container>

# Reload NPM
docker exec pro-0kt-nginx nginx -s reload

Database Connection Refused

# PostgreSQL: Check if running
docker ps | grep postgres

# Test connection
docker exec seller1-prod-db pg_isready

# Check credentials
cat /opt/<stack>/.env | grep POSTGRES

# MariaDB: Check if running
docker ps | grep mariadb

# Test connection
docker exec pro-0kt-mariadb mysqladmin ping -u root -p$MYSQL_ROOT_PASSWORD

Out of Disk Space

# Check usage
df -h
du -sh /mnt/data/* | sort -h

# Clean Docker
docker system prune -f
docker image prune -f

# Remove old backups
find /var/backups/ -mtime +30 -delete

# Check container logs (they can grow)
docker system df

High Memory Usage

# Check container memory
docker stats --no-stream

# Check system memory
free -h

# Kill memory hogs
docker kill <container-name>

# Remove unused images
docker image prune -a -f

# Restart Docker daemon
systemctl restart docker

Network Issues

# Check all networks
docker network ls

# Inspect specific network
docker network inspect <network-name>

# Check connectivity between containers
docker exec <container1> ping <container2>

# Reconnect containers to network
docker network connect <network-name> <container-name>

# DNS resolution
docker exec <container> nslookup <hostname>

Updates & Maintenance

Update Container Images

# Check for updates
cd /opt/<stack>
docker compose pull

# Apply updates (restart containers)
docker compose up -d

# Check new version
docker compose ps

Update System Packages

# Update package list
apt update

# Upgrade packages
apt upgrade

# Upgrade full (including kernel)
apt full-upgrade

# Clean up
apt autoremove
apt autoclean

Rotate Logs

# Docker logs auto-rotate (3x10MB per container)
# Manual rotation:
docker logs <container-name> --timestamps > container-backup.log

# Clear logs
truncate -s 0 /var/lib/docker/containers/*/*-json.log

# System logs
journalctl --vacuum-time=7d

Emergency Procedures

Website Down (seller1.ru)

  1. Check Service Status:
    bash docker ps | grep seller1-prod curl -I https://seller1.ru

  2. Check Logs:
    bash cd /opt/seller1-prod-stack docker compose logs --tail 100

  3. Check Database:
    bash docker exec seller1-prod-db pg_isready docker compose logs seller1-prod-db

  4. Restart Services (if needed):
    bash docker compose restart seller1-prod-web docker compose restart seller1-prod-db

  5. Check NPM (reverse proxy):
    bash docker ps | grep pro-0kt-nginx docker logs pro-0kt-nginx --tail 50

  6. Final Test:
    bash curl -I https://seller1.ru

All Domains Down

  1. Check NPM Status:
    bash docker ps | grep pro-0kt-nginx docker logs pro-0kt-nginx --tail 100

  2. Check Network:
    bash docker network ls docker network inspect pro-0kt-stack_pro-0kt-network

  3. Restart NPM:
    bash cd /opt/pro-0kt-stack docker compose restart pro-0kt-nginx docker exec pro-0kt-nginx nginx -s reload

  4. Reconnect Networks:
    bash /opt/scripts/npm-networks.sh

  5. Test Domains:
    bash curl -I https://home.0kt.ru curl -I https://seller1.ru

Out of Memory

  1. Check Memory Usage:
    bash free -h docker stats --no-stream

  2. Identify Memory Hogs:
    bash docker stats --no-stream | sort -k 4 -rh

  3. Kill Non-Critical Containers:
    bash docker kill <container-name>

  4. Restart Docker:
    bash systemctl restart docker

  5. Monitor Recovery:
    bash watch -n 1 free -h

Out of Disk Space

  1. Check Disk Usage:
    bash df -h du -sh /mnt/data/* | sort -rh

  2. Clean Docker:
    bash docker system prune -f docker image prune -a -f

  3. Remove Old Backups:
    bash find /var/backups/daily/ -mtime +7 -delete

  4. Check Largest Directories:
    bash du -sh /mnt/data/* | sort -rh

Dashboards:
- https://home.0kt.ru - Homarr (entry point)
- https://status.0kt.ru - Uptime Kuma (monitoring)
- https://portainer.0kt.ru - Portainer (Docker UI)

Utilities:
- https://git.0kt.ru - Gitea (repositories)
- https://pgadmin.0kt.ru - PgAdmin (PostgreSQL)
- https://n8n.0kt.ru - N8N (workflows)
- https://jupyter.0kt.ru - Jupyter (notebooks)
- https://mailhog.0kt.ru - Mailhog (email testing)

Production:
- https://seller1.ru - Main production site 🔴

Admin Access:
- NPM Admin: http://localhost:81 (SSH tunnel)
- Authelia: http://localhost:9091 (SSH tunnel)

Document Version: 1.0.0
Last Updated: 2025-11-09
Author: Infrastructure Team