# Release Health & Forensics

A release can pass every check and still break production; release health watches errors after each deploy and names the top suspect commit, for on-call engineers and service owners.

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flags: `release_health`, `forensics`.

Bumpkit closes the loop after a deploy. You tell it when a version went live, it counts production errors before and after, decides whether the release regressed, and names the commit that most likely caused it from the errors' stack frames. Separately, "what broke production" takes any error and stack trace and ranks the releases shipped before it.

| Feature | Flag | What it gives you |
|---------|------|-------------------|
| Release health | `release_health` | Deploy records, error counts, a verdict per deploy, suspect commits, regressions opened as incidents |
| What broke production | `forensics` | `POST /v1/incidents`, `bumpkit incidents`, the `what_broke` MCP tool, incident sources |

## Record deploys

Release health compares errors around the moment a version went live, so it needs a deploy record. Recording deploys needs `forensics` or `release_health`.

### From the CLI

Run `bumpkit deploy` from your deploy pipeline, in a checkout of the deployed commit:

```bash
bumpkit deploy --version v2.3.0 --environment production
```

By default it sends `HEAD` as the deployed commit, the `origin` remote as the repository, a tag on the commit as the version, and every commit since the previous tag with the lines each one changed. Bumpkit uses those changed lines to match errors to commits.

| Flag | Default | Meaning |
|------|---------|---------|
| `--sha` | `HEAD` | The deployed commit |
| `--version` | A tag on the commit | The version that went live, e.g. `v2.3.0` |
| `--environment` | `production` | Where it was deployed |
| `--repository` | The `origin` remote | `owner/name` |
| `--base` | The tag before the commit | The previous release, to send the commits this deploy shipped |
| `--status` | `success` | `success`, `failure` or `in_progress` |
| `--started-at` | none | When the deploy started (RFC 3339) |
| `--finished-at` | now | When the deploy finished (RFC 3339) |
| `--no-changes` | off | Don't send the commits and changed lines |

If there is no earlier tag, no commits are sent. Pass `--base` so errors can be attributed to individual commits.

### From the API

```bash
curl -X POST https://bumpkit.dev/v1/deployments \
  -H "x-api-key: $BUMPKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "repository": "acme/api",
    "sha": "4b7a1d2c9e3f",
    "environment": "production",
    "version": "v2.3.0",
    "status": "success",
    "commits": [
      {
        "sha": "4b7a1d2c9e3f",
        "subject": "Cache invoice totals",
        "files": [
          { "path": "billing/invoice.go", "hunks": [{ "start": 120, "end": 148, "context": "func Total" }] }
        ]
      }
    ]
  }'
```

| Field | Required | Notes |
|-------|----------|-------|
| `repository` | yes | `owner/name`, as your release checks record it |
| `sha` | yes | 7 to 40 hex characters |
| `environment` | no | Defaults to `production`; up to 64 characters |
| `version` | no | Up to 64 letters, digits and `. _ + - / @` |
| `status` | no | `success` (default), `failure` or `in_progress`. `succeeded`, `failed`, `error`, `started`, `pending` and `queued` are accepted as aliases |
| `deployed_at`, `started_at`, `finished_at` | no | RFC 3339. If `deployed_at` is missing, Bumpkit uses `finished_at` for a successful deploy, then `started_at`, then now |
| `commits` | no | Up to 500 commits, each with `sha`, `subject` and `files` (`path` and optional `hunks` with `start`, `end`, `context`) |

The endpoint returns `204 No Content`. Each recorded deploy is written to the [audit log](/docs/compliance#audit-log).

### From the GitHub App

If the [Bumpkit GitHub App](/docs/#install-the-github-app) is installed, GitHub `deployment_status` events are recorded as deploys automatically.

## Connect an error source

Bumpkit keeps error counts for 30 days. Connect at least one source; without one, every verdict is `unknown`.

### PostHog

Bumpkit polls PostHog for `$exception` events with a HogQL query.

1. Connect PostHog on the dashboard's **Integrations** page (see [Integrations](/docs/posthog)). Give the personal API key the `query:read` scope.
2. On the **Incidents** page, in the **Release health** card, set **PostHog errors for** to the repository (`owner/name`) and **PostHog environment** to the environment those errors come from (default `production`). Leave the repository empty to stop polling.

Bumpkit reads the version from `$app_version`, falling back to `$release`, and keeps only in-app stack frames when PostHog marks them. It polls every five minutes, two minutes behind real time, in one-hour slices, and catches up to 24 hours after a gap. When PostHog rate-limits or rejects the key, polling backs off (up to six hours) and the error shows on the Release health card.

### Sentry

1. In Sentry, create an internal integration with webhooks on, and an alert rule that sends to it.
2. On the dashboard's **Integrations** page, open the **Sentry** card, enter the repository (`owner/name`) and the integration's client secret, and click **Save error source**.
3. Bumpkit shows two URLs for the source. Point the Sentry webhook at one of them:

```text
https://bumpkit.dev/hooks/incidents/<token>   incidents, traced to a release (forensics)
https://bumpkit.dev/hooks/errors/<token>      error counts for release health
```

Bumpkit verifies the `Sentry-Hook-Signature` header (hex HMAC-SHA256 of the body with the client secret) and reads the `Sentry-Hook-Resource` header. It handles `event_alert` resources, and `issue` resources with action `created`; it ignores others with `204`. Sentry events sent to the incidents URL are also counted for release health.

### Signed webhook

Any tool can send errors with a signed webhook.

1. In the same **Production errors** card, enter the repository and leave **Sentry client secret** empty.
2. Click **Save error source**. Bumpkit shows the webhook secret once. Store it in your secret manager.
3. Sign each POST: put `sha256=` followed by the hex HMAC-SHA256 of the raw body, keyed with the secret, in the `X-Bumpkit-Signature` header.

```bash
body='{"events":[{"fingerprint":"TypeError-invoice-total","type":"TypeError","message":"Cannot read properties of undefined","environment":"production","version":"v2.3.0","frames":[{"file":"billing/invoice.go","function":"Total","line":131}],"count":4,"timestamp":"2026-10-21T09:30:00Z"}]}'
sig=$(printf '%s' "$body" | openssl dgst -sha256 -hmac "$BUMPKIT_ERRORS_SECRET" -hex | sed 's/^.* //')
curl -X POST "https://bumpkit.dev/hooks/errors/$TOKEN" \
  -H "Content-Type: application/json" \
  -H "X-Bumpkit-Signature: sha256=$sig" \
  -d "$body"
```

The errors endpoint takes one event object or `{"events": [...]}` with up to 500 events, and returns `202` with `{"recorded": n}`.

| Field | Notes |
|-------|-------|
| `fingerprint` | Groups occurrences of the same error |
| `type`, `message` | Joined as the error's title |
| `environment`, `version` | Where and in which release it happened |
| `frames` | Stack frames: `file`, `function`, `line` |
| `count` | Occurrences in this event, at least 1 (default 1) |
| `timestamp` | RFC 3339 |

Each event needs at least one of `type`, `message`, `fingerprint` or `frames`. The endpoint allows 600 requests a minute per source; beyond that it returns `429` with `Retry-After: 60`.

## Regression verdicts

For each deploy, Bumpkit compares production errors in the window after it went live with the window before. The window defaults to 60 minutes each side and stops at the next deploy. When the version label is on the errors, only errors tagged with this version (or untagged) count after the deploy. Each error gets a status:

| Status | Meaning |
|--------|---------|
| `new` | A clear rise in an error that wasn't seen before the deploy |
| `regressed` | A clear rise in an error that was already happening |
| `elevated` | A smaller rise worth watching |
| `emerging` | A handful of a new error, not yet a clear rise |

The release verdict is:

| Verdict | When |
|---------|------|
| `regressed` | The overall error rate clearly rose, or any error is `new` or `regressed` |
| `degraded` | The overall rate rose a little, or an error is `elevated` or `emerging` |
| `healthy` | Error rate held, or fewer than `min_events` errors after the deploy |
| `unknown` | No error source reports for the repository, or the after window is empty yet |

`min_events` is the fewest errors that can make a verdict, so a few stray errors never flag a release. Up to 10 error regressions are listed, most severe first. `partial: true` means the after window is still open.

### Settings

| Setting | Default | Range |
|---------|---------|-------|
| `window_minutes` | 60 | 15 to 1440 |
| `min_events` | 20 | 1 to 100000 |
| `posthog_repository` | none | `owner/name` |
| `posthog_environment` | `production` | One word |

Workspace owners and admins change them on the Incidents page or with the API:

```bash
curl -X PUT https://bumpkit.dev/v1/release-health/settings \
  -H "x-api-key: $BUMPKIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"window_minutes": 60, "min_events": 20, "posthog_repository": "acme/web", "posthog_environment": "production"}'
```

`GET /v1/release-health/settings` also returns `last_polled_at` and `last_poll_error`. Setting `posthog_repository` requires PostHog to be connected first.

## Suspect commits

For each regressed error, Bumpkit ranks the deploy's commits by how closely their changed lines match the error's stack trace. Suspects are shown with a likelihood (`high`, `medium` or `low`), not a percentage. Up to five suspects are listed, each with evidence such as `billing/invoice.go:131 (Total) → 4b7a1d2 changed lines 120–148 of billing/invoice.go.` If the deploy was recorded without commits, Bumpkit uses the files of the release check for the deployed commit (`attribution: release_files`).

## Regressions open incidents

When a deploy's verdict is `regressed`, Bumpkit:

- opens an incident titled `v2.3.0 regressed in production: <top error>` with high likelihood, linked to the release check for the deployed commit;
- posts a notice with the top suspect to your Slack, Discord and outbound webhook integrations;
- writes `incident.open` to the audit log.

When a later deploy brings that error back down, Bumpkit resolves the incident with that version and writes `incident.resolve`.

## Check a release

```bash
bumpkit release health v2.3.0
bumpkit release health v2.3.0 --environment staging --window 30 --json
```

It exits `1` when the release regressed, so you can use it as a post-deploy step. Flags: `--environment`, `--repository`, `--window` (minutes) and `--min-events`. The version can also be the deployed commit SHA.

The API equivalent is `GET /v1/releases/{version}/health` with the optional query parameters `repository`, `environment`, `window` and `min_events`. It returns `404 DEPLOYMENT_NOT_FOUND` if no successful deploy of that version is recorded.

```json
{
  "repository": "acme/api",
  "environment": "production",
  "version": "v2.3.0",
  "sha": "4b7a1d2c9e3f",
  "verdict": "regressed",
  "reason": "A new error appeared after v2.3.0: TypeError: Cannot read properties of undefined (42 in 60 minutes, p < 0.001).",
  "partial": false,
  "min_events": 20,
  "before": { "minutes": 60, "errors": 3, "per_hour": 3 },
  "after": { "minutes": 60, "errors": 45, "per_hour": 45 },
  "rate_ratio": 15,
  "regressions": [
    {
      "fingerprint": "TypeError-invoice-total",
      "status": "new",
      "before": 0,
      "after": 42,
      "suspects": [{ "sha": "4b7a1d2c9e3f", "likelihood": "high", "score": 5, "evidence": ["..."] }]
    }
  ],
  "attribution": "commits",
  "incident_id": "3f2c9a1e-8b7d-4c2a-9e1f-6a5b4c3d2e1f"
}
```

Agents use the `release_health` [MCP tool](/docs/mcp) with `version`, and optionally `repository`, `environment` and `window`.

## What broke production

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flag: `forensics`.

Give Bumpkit an error with its stack trace and it ranks the releases checked or deployed before it started.

```bash
bumpkit incidents report --error "$(cat trace.txt)" --since 2026-10-21T09:30:00Z
```

| Flag | Meaning |
|------|---------|
| `--error` | The error message and stack trace |
| `--file` | A file in the stack trace (repeatable) |
| `--repository` | Repository URL (default: the `origin` remote) |
| `--since` | When the error started (RFC 3339, default now) |
| `--lookback-days` | How far back to look for releases (1 to 90, default 14) |

The API is `POST /v1/incidents` with `repository`, `error`, `files`, `started_at` and `lookback_days`. Agents use the `what_broke` MCP tool with the same inputs.

Bumpkit reads the file paths and line numbers in the trace, skips third-party code, and ranks the releases in the lookback window by how strongly their changes and findings line up with the failing code. Suspects are shown with their likelihood, such as `Top suspect · high likelihood`, rather than a made-up percentage. If you record deploys for the repository, only releases that were actually deployed are ranked, timed from the deploy. Each suspect lists its evidence.

### Incident sources

The Sentry and signed webhook sources above also feed incidents. Send to `https://bumpkit.dev/hooks/incidents/<token>`; Bumpkit returns `202`, traces the error, and adds it to the **Incidents** page with its top suspects. The signed webhook body for incidents is:

| Field | Notes |
|-------|-------|
| `error` | The error with its stack trace |
| `files` | Files involved, if there is no trace |
| `title` | Defaults to the first line of `error` |
| `url` | A link back to the error |
| `id` | Your ID, used to deduplicate |
| `started_at` | RFC 3339 |

Send `error` or `files`. With codebase memory on (flag `codebase_memory`), a newly traced incident with a clear suspect is also saved as a memory on the failing path.

PagerDuty and Datadog incident sources are covered on [Integrations](/docs/integrations).

### Manage incidents

```bash
bumpkit incidents list --limit 50
bumpkit incidents resolve 3f2c9a1e-8b7d-4c2a-9e1f-6a5b4c3d2e1f --version v2.3.1 --sha 4b7a1d2
```

`GET /v1/incidents?limit=` lists recent incidents (up to 500) and `POST /v1/incidents/{id}/resolve` takes `{"version": "v2.3.1", "sha": "optional commit"}`. To plan a revert for an incident, see [Rollback & Time Machine](/docs/rollback).

See the [CLI Reference](/docs/cli) and [API Reference](/docs/api-reference) for every flag and field.