Events, webhooks and telemetry

moatbox records an event when a space changes state, a run starts or ends, or an agent's action waits for approval. You can get those events three ways:

• read them from the events API,
• have moatbox POST each one to your webhook,
• export them, with metrics and run traces, to an OpenTelemetry collector.

All three read the same event records, so an event carries the same id everywhere.

The event object

{
  "id": "evt_6f1c0a7e2b8d4c55a0e3d1f2b4c6a8e0",
  "type": "space.ready",
  "seq": 1042,
  "timestamp": "2026-11-08T10:00:00.123456Z",
  "team": {"id": "1b4e28ba-2fa1-11d2-883f-0016d3cca427"},
  "space": {"id": "9f3c2e1a-4b5d-4c6e-8f70-1a2b3c4d5e6f", "name": "web"},
  "data": {"previous_state": "sleeping", "woke_in_ms": 2140}
}

Event types

GET /api/v1/teams/{t}/webhooks returns this catalogue, minus webhook.test, as event_types, with an example data for each.

Read events from the API

curl "https://app.moatbox.dev/api/v1/teams/<team-id>/events?after=1000&n=100" \
  -H "Authorization: Bearer mbx_…"

The route needs team.read and returns {"events": [...]}.

Without after, the newest events come first.

A key limited to spaces or environments cannot read team routes, this one included.

Webhooks

Who can manage them

Team owners and admins manage webhooks, in the dashboard or with an API key that holds webhook.manage. Reading webhooks and their deliveries also works with team.read. The key's creator must be an owner or admin either way. A team can have at most 10 webhooks.

Create a webhook

curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/webhooks \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"name": "ops", "url": "https://hooks.example.com/moatbox", "events": ["run.failed", "approval.requested"]}'

• name: one line, 1 to 80 characters.
• url: https:// only, at most 2000 characters, no user, password or #fragment. At save time, every address the host resolves to must be public: moatbox refuses private, loopback, link-local, carrier-grade NAT, documentation and NAT64 ranges. On each connection moatbox resolves the host again and connects only to a public address.
• events: at least one event type.

The 201 response holds webhook and secret. moatbox shows the secret only here. It looks like whsec_ followed by base64 of 32 random bytes.

PATCH /api/v1/teams/{t}/webhooks/{id} takes any of name, url, events and enabled. The secret stays the same.

What moatbox sends

Each delivery is a POST with the event object as a JSON body and these headers:

Verify the signature

The signature is HMAC-SHA256 over <webhook-id>.<webhook-timestamp>.<raw body>, keyed with the base64-decoded part of the secret after whsec_, then base64-encoded.

Check it against the raw request body, before any JSON parsing. Reject old timestamps to block replays; pick a window that suits you, such as 5 minutes.

import crypto from "node:crypto";

export function verify(secret, headers, rawBody) {
  const id = headers["webhook-id"];
  const ts = headers["webhook-timestamp"];
  if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) return false;
  const key = Buffer.from(secret.replace(/^whsec_/, ""), "base64");
  const expected = crypto.createHmac("sha256", key).update(`${id}.${ts}.${rawBody}`).digest();
  return headers["webhook-signature"].split(" ").some((sig) => {
    const [version, value] = sig.split(",");
    if (version !== "v1" || !value) return false;
    const got = Buffer.from(value, "base64");
    return got.length === expected.length && crypto.timingSafeEqual(got, expected);
  });
}

import base64, hashlib, hmac, time

def verify(secret: str, headers: dict, raw_body: bytes) -> bool:
    msg_id = headers["webhook-id"]
    ts = headers["webhook-timestamp"]
    if abs(time.time() - int(ts)) > 300:
        return False
    key = base64.b64decode(secret.removeprefix("whsec_"))
    expected = hmac.new(key, f"{msg_id}.{ts}.".encode() + raw_body, hashlib.sha256).digest()
    for sig in headers["webhook-signature"].split(" "):
        version, _, value = sig.partition(",")
        if version == "v1" and hmac.compare_digest(base64.b64decode(value), expected):
            return True
    return False

Rotate the secret

curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/webhooks/<webhook-id>/rotate \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"overlap_hours": 24}'

The response holds the new secret and previous_valid_until. During the overlap (0 to 72 hours, 24 by default) moatbox signs each delivery with both secrets and sends both signatures in webhook-signature. Deploy the new secret to your receiver before the overlap ends. With overlap_hours: 0 the old secret stops at once.

Delivery and retries

For each event, moatbox queues a delivery to every webhook that is on and subscribed to its type. A delivery can arrive more than once; drop duplicates by id. A delivery succeeds when your endpoint answers 2xx within 10 seconds. moatbox does not follow redirects; a 3xx counts as a failure.

On failure moatbox retries up to 11 more times, 12 attempts in all. The waits between attempts are 30 s, 2 min, 5 min, 15 min, 30 min, 1 h, 2 h, 3 h, 4 h, 5 h and 8 h, each varied by up to 10%. The last attempt comes about 24 hours after the first. moatbox sends up to 16 deliveries at once, so events can arrive out of order.

moatbox turns the webhook off at the first failed attempt after 24 hours in which every delivery to the endpoint failed, and emails the team's owners and admins. Pending deliveries wait, and moatbox sends them as soon as you turn the webhook back on (PATCH with {"enabled": true}). moatbox queues nothing for events that happen while the webhook is off. A delivery that used all 12 attempts stays failed; send it again with redeliver.

The webhook object reports health: new, ok, failing or disabled, with failing_since, last_success_at, last_failure_at, delivered_24h, failed_24h and pending.

Delivery log, redelivery and tests

moatbox keeps delivery records for 30 days.

• The log takes ?state= (pending, delivered or failed), ?before=<delivery-id> and ?n= (1 to 200, default 50).
• One delivery returns its exact payload and its attempts, each with n, at, status or error, ms, the first 1 KiB of your response, and manual.
• A test sends one attempt and no retries. It returns delivery_id and attempt.
• A redelivery makes one attempt and returns attempt. When it succeeds, moatbox marks the delivery delivered.

OpenTelemetry export

moatbox can push metrics, events and run traces to your OpenTelemetry collector over OTLP/HTTP with the JSON encoding. A team can have up to 3 endpoints. The same people and key actions that manage webhooks manage telemetry endpoints.

curl -X POST https://app.moatbox.dev/api/v1/teams/<team-id>/telemetry \
  -H "Authorization: Bearer mbx_…" \
  -H "Content-Type: application/json" \
  -d '{"name": "collector", "url": "https://otlp.example.com/otlp", "headers": {"Authorization": "Basic …"}}'

• url is the collector's base URL. moatbox adds /v1/metrics, /v1/logs and /v1/traces, and refuses a URL that already ends in one of them. The public-address rules for webhooks apply here too.
• headers holds up to 20 headers. Each value is one line of at most 4096 bytes. moatbox refuses Host, Content-Type, Content-Length, Content-Encoding, Transfer-Encoding, Connection and User-Agent. moatbox encrypts the values and never returns them; the endpoint lists only header names.
• Saving runs a test export (an empty /v1/logs request) and returns endpoint and test (ok, status, message).
• A new endpoint receives events from the moment you add it, not older ones.

PATCH takes any of name, url, headers and enabled. A headers object replaces the saved set, so list every header you want to keep; send null as a value to keep that header's saved value. The list shows last_export_at, last_error, last_error_at and retry_at per endpoint.

Metrics

Once a minute, moatbox sends gauges for each space with monitoring on:

Resource attributes: service.name=moatbox, moatbox.team.id, and for a space service.instance.id, moatbox.space.id, moatbox.space.name and moatbox.space.size. After an outage moatbox resends up to 1 hour of metrics, in time order.

Events as logs

Each event becomes a log record with eventName set to the event type and these attributes: event.name, moatbox.event.id, moatbox.event.seq, moatbox.data.<key> for each data field, and for a space event moatbox.space.id and moatbox.space.name.

Runs as traces

When a run ends, moatbox exports it as one trace of at most 1000 spans:

• a root span run <agent> with moatbox.run.id, moatbox.run.agent and moatbox.run.exit, plus gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens when the agent reported them,
• a child span tool <name> per tool call the agent reported, with gen_ai.tool.name and moatbox.tool.target,
• a span approval <action> per held action, from held to decided, with moatbox.approval.entry_id, moatbox.approval.decision and moatbox.approval.by,
• a zero-length span per gateway action, with moatbox.gateway.action and moatbox.gateway.summary.

If moatbox cannot read the run's timeline, the run gets no trace.

Under zero data retention (see Privacy) the spans keep names, times and outcomes but drop moatbox.tool.target and moatbox.gateway.summary.

Retries

moatbox retries an export that gets 429, 502, 503, 504 or no answer. It waits as long as Retry-After asks, 30 seconds when the header is missing, and at most 10 minutes. Other errors land in last_error and moatbox moves on.