0 Fork 0

Files

T

matitos ac92dd5ac4 Observability: Prometheus metrics, log panel, runbook

- GET /api/metrics renders Prometheus text (equity, P&L, per-strategy
  equity/pnl/allocation/halted, info) with no client library.
- In-memory ring log handler (app/logs.py) exposed at GET /api/logs and a
  dashboard "Server logs" panel; Runtime mirrors bot events into the logger.
- docs/runbook.md covers deploys, the kill switch, resetting paper state and
  going live. Adds tests for metrics rendering and the log ring.

2026-09-22 17:10:38 +00:00

2.1 KiB

Raw Permalink Blame History

Operations runbook

Practical steps for running the deployed bot. Host paths are examples — adjust to the deploy host.

Find the stack

docker compose ps
docker compose logs --tail=200 backend
docker compose logs -f frontend

The backend exposes GET /api/metrics (Prometheus text format) for the existing Prometheus/Grafana stack; scrape it alongside the other services. Useful series: crypto_running, crypto_equity, crypto_total_pnl, crypto_open_positions, crypto_strategy_equity, crypto_strategy_pnl, crypto_strategy_halted, crypto_info.

Stop / kill trading

Dashboard: the Kill all button (or Stop all) flattens and halts.
API: curl -X POST -H "X-API-Token: $TOKEN" https://<host>/api/bot/kill
Hard stop: docker compose stop backend stops the process entirely. It does not close open positions — use the API kill first if you care.

Something is wrong

docker compose ps — is the service healthy?
docker compose logs --tail=200 backend — look for error/Traceback.
In the dashboard, open Server logs (shows the in-memory ring of recent log lines) and Activity (trades, halts, errors).
curl https://<host>/api/live-check — if live is blocked, the reasons are listed.

A deploy went out but the UI looks stale

The deploy job fails if the frontend container's build stamp does not match the commit; if you see stale UI and the job passed, hard-refresh or open incognito. The footer shows client vs server versions — they must match.

Reset paper state

docker compose stop backend
# remove the SQLite file on the host mount
rm -f <CRYPTO_DATA_DIR>/trader.db
docker compose start backend

Going live (do this deliberately)

Rehearse the kill switch in paper mode.
Set MAX_DAILY_LOSS, MAX_DRAWDOWN_PCT and/or MAX_TRADES_PER_DAY, and MAX_POSITION_PCT < 1.0.
Set MIN_LIVE_BALANCE and add exchange credentials.
Set LIVE_CONFIRMATION="I UNDERSTAND THE RISK".
Deploy, confirm /api/live-check returns ok: true, then start with a small balance and watch it.

2.1 KiB Raw Permalink Blame History