# Events, webhooks and telemetry

> Read moatbox events over the API, receive them as signed webhooks, and export metrics, logs and run traces to an OpenTelemetry collector.

moatbox records an event when a space changes state, a run starts or ends, or an agent's action waits for approval. You can get those events three ways:

- read them from the events API,
- have moatbox POST each one to your webhook,
- export them, with metrics and run traces, to an OpenTelemetry collector.

All three read the same event records, so an event carries the same `id` everywhere.

## The event object

```json
{
  "id": "evt_6f1c0a7e2b8d4c55a0e3d1f2b4c6a8e0",
  "type": "space.ready",
  "seq": 1042,
  "timestamp": "2026-11-08T10:00:00.123456Z",
  "team": {"id": "1b4e28ba-2fa1-11d2-883f-0016d3cca427"},
  "space": {"id": "9f3c2e1a-4b5d-4c6e-8f70-1a2b3c4d5e6f", "name": "web"},
  "data": {"previous_state": "sleeping", "woke_in_ms": 2140}
}
```

| Field | Meaning |
|---|---|
| `id` | `evt_` plus 32 hex characters. It stays the same across retries and redeliveries; use it to drop duplicates. |
| `type` | One of the types below. |
| `seq` | A number that grows with each event. Deliveries can arrive out of order; sort by `seq`. |
| `timestamp` | When the event happened, UTC. |
| `team` | The team's `id`. |
| `space` | The space's `id` and `name` (its slug), or `null` for a team event. |
| `data` | Fields that depend on the type. |

## Event types

| Type | When | `data` |
|---|---|---|
| `space.created` | A space was created, forked, made from a snapshot, or cloned from a share code. | `from` (`new`, `fork`, `snapshot`, `share`), `parent`, `size` |
| `space.ready` | A space is up: after create, after waking, or after an operator resumed it. | `previous_state`, `woke_in_ms` after a wake, `reason: sleep_failed` when a sleep was rolled back |
| `space.sleeping` | A space went to sleep: idle, by hand, its awake limit, the spend cap, or a wallet that ran out. Disk is kept, and memory unless the snapshot failed. | `previous_state` |
| `space.stopped` | An operator suspended a space; it cannot wake until resumed. | `previous_state`, `reason` |
| `space.error` | A space stopped answering health checks (`reason: not_responding`) or could not sleep (`reason: sleep_failed`). | `reason`, `since`, `error` |
| `space.recovered` | A space answers again after `space.error`. | `down_seconds` |
| `space.deleted` | A space moved to the trash. You can restore it until `purge_at`. | `previous_state`, `purge_at`, `by` |
| `space.restored` | A space came back from the trash. | `state`, `by` |
| `snapshot.saved` | Someone saved a named snapshot. | `snapshot_id`, `name`, `by` |
| `space.disk_low` | The disk crossed 85%. | `used`, `total` (bytes), `percent` |
| `space.disk_full` | The disk crossed 95%. Writes, sleep and fork can fail. | `used`, `total`, `percent` |
| `team.spend_cap_reached` | Spend beyond the plan reached the cap; every space sleeps until the next period or a higher cap. `space` is `null`. | `cap_cents`, `period_start` |
| `space.ai_cap_reached` | A space or team AI spend cap (per UTC day or month) was reached; the AI gateway refuses that space's model requests. | `scope` (`space`, `team`), `window` (`day`, `month`), `cap_usd`, `spent_usd`, `message` |
| `space.ai_key_refused` | A model provider refused an AI profile's key (HTTP 401 or 403). | `provider`, `account`, `profile`, `status`, `message` |
| `run.started` | An agent run, or an interactive SSH session (shell, file transfer, port forward), started. One-off SSH commands appear only in the audit log. | `run_id`, `agent` |
| `run.finished` | A run ended with exit code 0. | `run_id`, `exit` |
| `run.failed` | A run ended with a non-zero exit code or an error. | `run_id`, `error`, and `exit` when the run exited |
| `approval.requested` | The gateway held a risky action for a person to approve. | `run_id`, `entry_id`, `action` |
| `approval.decided` | A held action was approved, denied, undone or abandoned. | `run_id`, `entry_id`, `action`, `decision`, `by` |
| `webhook.test` | Only from the test button or the test route. It never appears in the event stream. | `message`, `webhook`, `sent_by` |

`GET /api/v1/teams/{t}/webhooks` returns this catalogue, minus `webhook.test`, as `event_types`, with an example `data` for each.

## Read events from the API

```bash
curl "https://app.moatbox.dev/api/v1/teams/<team-id>/events?after=1000&n=100" \
  -H "Authorization: Bearer mbx_…"
```

The route needs `team.read` and returns `{"events": [...]}`.

| Query | Meaning |
|---|---|
| `after` | Only events with `seq` above this, oldest first. Use it to page forward. |
| `before` | Only events with `seq` below this. |
| `type` | One event type. An unknown type gets `400`. |
| `space` | One space, by slug. |
| `n` | How many, 1 to 200. Defaults to 50. |

Without `after`, the newest events come first.

A key limited to spaces or environments cannot read team routes, this one included.

## Webhooks

### Who can manage them

Team owners and admins manage webhooks, in the dashboard or with an API key that holds `webhook.manage`. Reading webhooks and their deliveries also works with `team.read`. The key's creator must be an owner or admin either way. A team can have at most 10 webhooks.

### Create a webhook

```bash
curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/webhooks \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"name": "ops", "url": "https://hooks.example.com/moatbox", "events": ["run.failed", "approval.requested"]}'
```

- `name`: one line, 1 to 80 characters.
- `url`: `https://` only, at most 2000 characters, no user, password or `#fragment`. At save time, every address the host resolves to must be public: moatbox refuses private, loopback, link-local, carrier-grade NAT, documentation and NAT64 ranges. On each connection moatbox resolves the host again and connects only to a public address.
- `events`: at least one event type.

The `201` response holds `webhook` and `secret`. moatbox shows the secret only here. It looks like `whsec_` followed by base64 of 32 random bytes.

`PATCH /api/v1/teams/{t}/webhooks/{id}` takes any of `name`, `url`, `events` and `enabled`. The secret stays the same.

### What moatbox sends

Each delivery is a `POST` with the event object as a JSON body and these headers:

| Header | Value |
|---|---|
| `Content-Type` | `application/json` |
| `User-Agent` | `moatbox-webhooks/1` |
| `webhook-id` | The event's `id` |
| `webhook-timestamp` | Unix seconds when moatbox made this attempt |
| `webhook-signature` | `v1,<signature>`, space-separated when two secrets are active |
| `Moatbox-Event-Type` | The event's `type` |
| `Moatbox-Attempt` | The attempt number, from 1 |

### Verify the signature

The signature is HMAC-SHA256 over `<webhook-id>.<webhook-timestamp>.<raw body>`, keyed with the base64-decoded part of the secret after `whsec_`, then base64-encoded.

Check it against the raw request body, before any JSON parsing. Reject old timestamps to block replays; pick a window that suits you, such as 5 minutes.

```javascript
import crypto from "node:crypto";

export function verify(secret, headers, rawBody) {
  const id = headers["webhook-id"];
  const ts = headers["webhook-timestamp"];
  if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) return false;
  const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64");
  const expected = crypto.createHmac("sha256", key).update(`${id}.${ts}.${rawBody}`).digest();
  return headers["webhook-signature"].split(" ").some((sig) => {
    const [version, value] = sig.split(",");
    if (version !== "v1" || !value) return false;
    const got = Buffer.from(value, "base64");
    return got.length === expected.length && crypto.timingSafeEqual(got, expected);
  });
}
```

```python
import base64, hashlib, hmac, time

def verify(secret: str, headers: dict, raw_body: bytes) -> bool:
    msg_id = headers["webhook-id"]
    ts = headers["webhook-timestamp"]
    if abs(time.time() - int(ts)) > 300:
        return False
    key = base64.b64decode(secret.removeprefix("whsec_"))
    expected = hmac.new(key, f"{msg_id}.{ts}.".encode() + raw_body, hashlib.sha256).digest()
    for sig in headers["webhook-signature"].split(" "):
        version, _, value = sig.partition(",")
        if version == "v1" and hmac.compare_digest(base64.b64decode(value), expected):
            return True
    return False
```

### Rotate the secret

```bash
curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/webhooks/<webhook-id>/rotate \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"overlap_hours": 24}'
```

The response holds the new `secret` and `previous_valid_until`. During the overlap (0 to 72 hours, 24 by default) moatbox signs each delivery with both secrets and sends both signatures in `webhook-signature`. Deploy the new secret to your receiver before the overlap ends. With `overlap_hours: 0` the old secret stops at once.

### Delivery and retries

For each event, moatbox queues a delivery to every webhook that is on and subscribed to its type. A delivery can arrive more than once; drop duplicates by `id`. A delivery succeeds when your endpoint answers 2xx within 10 seconds. moatbox does not follow redirects; a 3xx counts as a failure.

On failure moatbox retries up to 11 more times, 12 attempts in all. The waits between attempts are 30 s, 2 min, 5 min, 15 min, 30 min, 1 h, 2 h, 3 h, 4 h, 5 h and 8 h, each varied by up to 10%. The last attempt comes about 24 hours after the first. moatbox sends up to 16 deliveries at once, so events can arrive out of order.

moatbox turns the webhook off at the first failed attempt after 24 hours in which every delivery to the endpoint failed, and emails the team's owners and admins. Pending deliveries wait, and moatbox sends them as soon as you turn the webhook back on (`PATCH` with `{"enabled": true}`). moatbox queues nothing for events that happen while the webhook is off. A delivery that used all 12 attempts stays `failed`; send it again with redeliver.

The webhook object reports `health`: `new`, `ok`, `failing` or `disabled`, with `failing_since`, `last_success_at`, `last_failure_at`, `delivered_24h`, `failed_24h` and `pending`.

### Delivery log, redelivery and tests

moatbox keeps delivery records for 30 days.

| Method | Path | Action | What it does |
|---|---|---|---|
| GET | `/api/v1/teams/{t}/webhooks` | `team.read` or `webhook.manage` | List webhooks, the limit, and `event_types` |
| POST | `/api/v1/teams/{t}/webhooks` | `webhook.manage` | Create one |
| GET | `/api/v1/teams/{t}/webhooks/{id}` | `team.read` or `webhook.manage` | Read one |
| PATCH | `/api/v1/teams/{t}/webhooks/{id}` | `webhook.manage` | Change it |
| DELETE | `/api/v1/teams/{t}/webhooks/{id}` | `webhook.manage` | Delete it |
| POST | `/api/v1/teams/{t}/webhooks/{id}/rotate` | `webhook.manage` | New secret |
| POST | `/api/v1/teams/{t}/webhooks/{id}/test` | `webhook.manage` | Send `webhook.test` now |
| GET | `/api/v1/teams/{t}/webhooks/{id}/deliveries` | `team.read` or `webhook.manage` | The delivery log |
| GET | `/api/v1/teams/{t}/webhooks/{id}/deliveries/{d}` | `team.read` or `webhook.manage` | One delivery with every attempt |
| POST | `/api/v1/teams/{t}/webhooks/{id}/deliveries/{d}/redeliver` | `webhook.manage` | One more attempt now |

- The log takes `?state=` (`pending`, `delivered` or `failed`), `?before=<delivery-id>` and `?n=` (1 to 200, default 50).
- One delivery returns its exact `payload` and its `attempts`, each with `n`, `at`, `status` or `error`, `ms`, the first 1 KiB of your `response`, and `manual`.
- A test sends one attempt and no retries. It returns `delivery_id` and `attempt`.
- A redelivery makes one attempt and returns `attempt`. When it succeeds, moatbox marks the delivery delivered.

## OpenTelemetry export

moatbox can push metrics, events and run traces to your OpenTelemetry collector over OTLP/HTTP with the JSON encoding. A team can have up to 3 endpoints. The same people and key actions that manage webhooks manage telemetry endpoints.

```bash
curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/telemetry \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"name": "collector", "url": "https://otlp.example.com/otlp", "headers": {"Authorization": "Basic …"}}'
```

- `url` is the collector's base URL. moatbox adds `/v1/metrics`, `/v1/logs` and `/v1/traces`, and refuses a URL that already ends in one of them. The public-address rules for webhooks apply here too.
- `headers` holds up to 20 headers. Each value is one line of at most 4096 bytes. moatbox refuses `Host`, `Content-Type`, `Content-Length`, `Content-Encoding`, `Transfer-Encoding`, `Connection` and `User-Agent`. moatbox encrypts the values and never returns them; the endpoint lists only header names.
- Saving runs a test export (an empty `/v1/logs` request) and returns `endpoint` and `test` (`ok`, `status`, `message`).
- A new endpoint receives events from the moment you add it, not older ones.

| Method | Path | Action |
|---|---|---|
| GET | `/api/v1/teams/{t}/telemetry` | `team.read` or `webhook.manage` |
| POST | `/api/v1/teams/{t}/telemetry` | `webhook.manage` |
| PATCH | `/api/v1/teams/{t}/telemetry/{id}` | `webhook.manage` |
| DELETE | `/api/v1/teams/{t}/telemetry/{id}` | `webhook.manage` |
| POST | `/api/v1/teams/{t}/telemetry/{id}/test` | `webhook.manage` |

`PATCH` takes any of `name`, `url`, `headers` and `enabled`. A `headers` object replaces the saved set, so list every header you want to keep; send `null` as a value to keep that header's saved value. The list shows `last_export_at`, `last_error`, `last_error_at` and `retry_at` per endpoint.

### Metrics

Once a minute, moatbox sends gauges for each space with monitoring on:

| Metric | Unit |
|---|---|
| `moatbox.space.cpu.utilization` | `1` |
| `moatbox.space.memory.usage` | `By` |
| `moatbox.space.memory.utilization` | `1` |
| `moatbox.space.filesystem.usage` | `By` |
| `moatbox.space.filesystem.utilization` | `1` |
| `moatbox.space.io.read` | `By/s` |
| `moatbox.space.io.write` | `By/s` |
| `moatbox.team.spaces.active` | `{space}` |

Resource attributes: `service.name=moatbox`, `moatbox.team.id`, and for a space `service.instance.id`, `moatbox.space.id`, `moatbox.space.name` and `moatbox.space.size`. After an outage moatbox resends up to 1 hour of metrics, in time order.

### Events as logs

Each event becomes a log record with `eventName` set to the event type and these attributes: `event.name`, `moatbox.event.id`, `moatbox.event.seq`, `moatbox.data.<key>` for each `data` field, and for a space event `moatbox.space.id` and `moatbox.space.name`.

| Severity | Types |
|---|---|
| `ERROR` (17) | `space.error`, `run.failed` |
| `WARN` (13) | `space.disk_low`, `space.disk_full`, `team.spend_cap_reached`, `space.stopped`, `space.ai_cap_reached`, `space.ai_key_refused` |
| `INFO` (9) | Every other type |

### Runs as traces

When a run ends, moatbox exports it as one trace of at most 1000 spans:

- a root span `run <agent>` with `moatbox.run.id`, `moatbox.run.agent` and `moatbox.run.exit`, plus `gen_ai.request.model`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens` when the agent reported them,
- a child span `tool <name>` per tool call the agent reported, with `gen_ai.tool.name` and `moatbox.tool.target`,
- a span `approval <action>` per held action, from held to decided, with `moatbox.approval.entry_id`, `moatbox.approval.decision` and `moatbox.approval.by`,
- a zero-length span per gateway action, with `moatbox.gateway.action` and `moatbox.gateway.summary`.

If moatbox cannot read the run's timeline, the run gets no trace.

Under zero data retention (see [Privacy](https://docs.moatbox.dev/privacy.md)) the spans keep names, times and outcomes but drop `moatbox.tool.target` and `moatbox.gateway.summary`.

### Retries

moatbox retries an export that gets `429`, `502`, `503`, `504` or no answer. It waits as long as `Retry-After` asks, 30 seconds when the header is missing, and at most 10 minutes. Other errors land in `last_error` and moatbox moves on.