Observability: Prometheus metrics, log panel, runbook
- GET /api/metrics renders Prometheus text (equity, P&L, per-strategy equity/pnl/allocation/halted, info) with no client library. - In-memory ring log handler (app/logs.py) exposed at GET /api/logs and a dashboard "Server logs" panel; Runtime mirrors bot events into the logger. - docs/runbook.md covers deploys, the kill switch, resetting paper state and going live. Adds tests for metrics rendering and the log ring.
This commit is contained in:
1 parent
c605a0f316
commit
ac92dd5ac4
12 files changed
+462
-1
No files matched your search
@@ -0,0 +1,59 @@
|
||||
# Operations runbook
|
||||
|
||||
Practical steps for running the deployed bot. Host paths are examples — adjust
|
||||
to the deploy host.
|
||||
|
||||
## Find the stack
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs --tail=200 backend
|
||||
docker compose logs -f frontend
|
||||
```
|
||||
|
||||
The backend exposes `GET /api/metrics` (Prometheus text format) for the existing
|
||||
Prometheus/Grafana stack; scrape it alongside the other services. Useful series:
|
||||
`crypto_running`, `crypto_equity`, `crypto_total_pnl`, `crypto_open_positions`,
|
||||
`crypto_strategy_equity`, `crypto_strategy_pnl`, `crypto_strategy_halted`,
|
||||
`crypto_info`.
|
||||
|
||||
## Stop / kill trading
|
||||
|
||||
- **Dashboard:** the **Kill all** button (or Stop all) flattens and halts.
|
||||
- **API:** `curl -X POST -H "X-API-Token: $TOKEN" https://<host>/api/bot/kill`
|
||||
- **Hard stop:** `docker compose stop backend` stops the process entirely. It
|
||||
does **not** close open positions — use the API kill first if you care.
|
||||
|
||||
## Something is wrong
|
||||
|
||||
1. `docker compose ps` — is the service `healthy`?
|
||||
2. `docker compose logs --tail=200 backend` — look for `error`/`Traceback`.
|
||||
3. In the dashboard, open **Server logs** (shows the in-memory ring of recent
|
||||
log lines) and **Activity** (trades, halts, errors).
|
||||
4. `curl https://<host>/api/live-check` — if live is blocked, the reasons are
|
||||
listed.
|
||||
|
||||
## A deploy went out but the UI looks stale
|
||||
|
||||
The deploy job fails if the frontend container's build stamp does not match the
|
||||
commit; if you see stale UI and the job passed, hard-refresh or open
|
||||
incognito. The footer shows `client` vs `server` versions — they must match.
|
||||
|
||||
## Reset paper state
|
||||
|
||||
```bash
|
||||
docker compose stop backend
|
||||
# remove the SQLite file on the host mount
|
||||
rm -f <CRYPTO_DATA_DIR>/trader.db
|
||||
docker compose start backend
|
||||
```
|
||||
|
||||
## Going live (do this deliberately)
|
||||
|
||||
1. Rehearse the kill switch in paper mode.
|
||||
2. Set `MAX_DAILY_LOSS`, `MAX_DRAWDOWN_PCT` and/or `MAX_TRADES_PER_DAY`, and
|
||||
`MAX_POSITION_PCT < 1.0`.
|
||||
3. Set `MIN_LIVE_BALANCE` and add exchange credentials.
|
||||
4. Set `LIVE_CONFIRMATION="I UNDERSTAND THE RISK"`.
|
||||
5. Deploy, confirm `/api/live-check` returns `ok: true`, then start with a
|
||||
small balance and watch it.
|
||||
Reference in new issue
Block a user