matitos
ac92dd5ac4
- GET /api/metrics renders Prometheus text (equity, P&L, per-strategy equity/pnl/allocation/halted, info) with no client library. - In-memory ring log handler (app/logs.py) exposed at GET /api/logs and a dashboard "Server logs" panel; Runtime mirrors bot events into the logger. - docs/runbook.md covers deploys, the kill switch, resetting paper state and going live. Adds tests for metrics rendering and the log ring.
60 lines
2.1 KiB
Markdown
60 lines
2.1 KiB
Markdown
# Operations runbook
|
|
|
|
Practical steps for running the deployed bot. Host paths are examples — adjust
|
|
to the deploy host.
|
|
|
|
## Find the stack
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose logs --tail=200 backend
|
|
docker compose logs -f frontend
|
|
```
|
|
|
|
The backend exposes `GET /api/metrics` (Prometheus text format) for the existing
|
|
Prometheus/Grafana stack; scrape it alongside the other services. Useful series:
|
|
`crypto_running`, `crypto_equity`, `crypto_total_pnl`, `crypto_open_positions`,
|
|
`crypto_strategy_equity`, `crypto_strategy_pnl`, `crypto_strategy_halted`,
|
|
`crypto_info`.
|
|
|
|
## Stop / kill trading
|
|
|
|
- **Dashboard:** the **Kill all** button (or Stop all) flattens and halts.
|
|
- **API:** `curl -X POST -H "X-API-Token: $TOKEN" https://<host>/api/bot/kill`
|
|
- **Hard stop:** `docker compose stop backend` stops the process entirely. It
|
|
does **not** close open positions — use the API kill first if you care.
|
|
|
|
## Something is wrong
|
|
|
|
1. `docker compose ps` — is the service `healthy`?
|
|
2. `docker compose logs --tail=200 backend` — look for `error`/`Traceback`.
|
|
3. In the dashboard, open **Server logs** (shows the in-memory ring of recent
|
|
log lines) and **Activity** (trades, halts, errors).
|
|
4. `curl https://<host>/api/live-check` — if live is blocked, the reasons are
|
|
listed.
|
|
|
|
## A deploy went out but the UI looks stale
|
|
|
|
The deploy job fails if the frontend container's build stamp does not match the
|
|
commit; if you see stale UI and the job passed, hard-refresh or open
|
|
incognito. The footer shows `client` vs `server` versions — they must match.
|
|
|
|
## Reset paper state
|
|
|
|
```bash
|
|
docker compose stop backend
|
|
# remove the SQLite file on the host mount
|
|
rm -f <CRYPTO_DATA_DIR>/trader.db
|
|
docker compose start backend
|
|
```
|
|
|
|
## Going live (do this deliberately)
|
|
|
|
1. Rehearse the kill switch in paper mode.
|
|
2. Set `MAX_DAILY_LOSS`, `MAX_DRAWDOWN_PCT` and/or `MAX_TRADES_PER_DAY`, and
|
|
`MAX_POSITION_PCT < 1.0`.
|
|
3. Set `MIN_LIVE_BALANCE` and add exchange credentials.
|
|
4. Set `LIVE_CONFIRMATION="I UNDERSTAND THE RISK"`.
|
|
5. Deploy, confirm `/api/live-check` returns `ok: true`, then start with a
|
|
small balance and watch it.
|