# Confidence Score & Leadership

"Is this release safe?" usually gets a gut-feel answer; the confidence score puts a number on it, measured against what happened to your past releases, for engineering leads and leadership who track release risk.

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flags: `confidence_score`, `leadership_dashboards`.

Bumpkit puts one number on every [gate decision](/docs/release-gate): how confident it is that the release won't need a revert, a hotfix or an incident. The score learns from what actually happened to your past releases. Leadership reports roll the same data up per repository and team.

| Feature | Flag | Where |
|---------|------|-------|
| Release confidence score | `confidence_score` | Gate responses, the **Confidence** tab on the dashboard's Policy page, `/v1/confidence`, `/v1/checks/{id}/confidence`, `/v1/risk`, the `bumpkit report` subcommands |
| Leadership dashboards | `leadership_dashboards` | The **Reports** tab on the dashboard's Insights page, `/v1/reports/leadership`, `/v1/reports/readiness`, `bumpkit report`, `bumpkit report readiness` |
| [Proven changes](#proven-changes) | `proven_changes` | The **Proven changes** tab on the dashboard's Insights page, `/v1/proven`, `/v1/proven/class`, `bumpkit proven`, the `proven_changes` MCP tool |
| [Agent scorecards](#agent-scorecards) | `agent_scorecards` | The **Reports** tab on the dashboard's Insights page, `/v1/scorecards`, `bumpkit scorecards` |

## The confidence score

With `confidence_score` on, every `POST /v1/gate` response carries a `confidence` object:

```json
{
  "confidence": {
    "check_id": "3f2c9a1e-0d4b-4c55-9e0a-7b1d2c3e4f50",
    "score": 82,
    "failure_probability": 0.18,
    "baseline": 95,
    "method": "workspace",
    "note": "Learned from this workspace's own release outcomes.",
    "factors": [
      { "key": "proven_high", "label": "Proven high-severity findings", "detail": "1 proven high finding", "value": 0.56, "points": -9 },
      { "key": "approvals", "label": "Human approvals", "detail": "1 human approval, on the gated commit", "value": 0.33, "points": 2 }
    ],
    "teams": ["@acme/payments"]
  }
}
```

| Field | Meaning |
|-------|---------|
| `score` | 100 minus the failure probability in percent |
| `failure_probability` | How likely the release is to need a revert, a hotfix or an incident |
| `baseline` | The score of a release with no risk factors at all |
| `method` | `workspace`, `global` or `heuristic` (see [How the score learns](#how-the-score-learns)) |
| `factors` | Each factor that moved the score, largest first. `points` is how many score points it moved: negative lowers confidence, positive raises it |

| `teams` | CODEOWNERS teams for the changed files, used by leadership reports |

Until Bumpkit has learned enough from your outcomes (`method` is `heuristic`), it shows no score: `score`, `failure_probability`, `baseline` and `points` are left out, the factors are still listed, and `note` says what's needed. The dashboard shows "No score yet".

Look up a gated release's score later:

```bash
bumpkit report score CHECK_ID
```

The API is `GET /v1/checks/{id}/confidence`.

### Factors

Each factor in `factors` names one thing about the release that moved the score:

| Key | Factor |
|-----|--------|
| `proven_high` | Proven high-severity findings |
| `inferred_high` | Inferred high-severity findings |
| `medium` | Medium-severity findings |
| `low` | Low-severity findings |
| `blast_radius` | Product areas the change reaches |
| `hotspot` | Changed files that often broke before |
| `untested` | Changed code without matching test changes |
| `test_evidence` | Tests changed with the code |
| `agent` | Commits carry coding agent trailers or authors |
| `approvals` | Human approvals on the gated commit |
| `churn` | Lines changed |

Findings you mark as false positives through [finding feedback](/docs/release-checks) count for less. Suppressed and `info` findings don't count.

### Outcomes

The model learns from outcomes recorded against release checks:

| Outcome | How it is recorded |
|---------|--------------------|
| `revert` | Automatically, when a gated commit's message says `This reverts commit <sha>` of an earlier checked release |
| `incident` | Automatically, from an [incident](/docs/release-health) whose top suspect is the release with high or medium likelihood |
| `hotfix` | Automatically, when that incident is resolved |
| `regression` | By hand |

Record any outcome yourself:

```bash
bumpkit report outcome 3f2c9a1e-0d4b-4c55-9e0a-7b1d2c3e4f50 --kind revert --note "Rolled back: checkout totals were wrong"
```

The API is `POST /v1/checks/{id}/outcomes` with `{"kind": "revert", "note": "optional"}`. The note is up to 1000 characters and kept in the audit record (`release.outcome`).

### How the score learns

A release counts once it has an outcome, or once enough time has passed without one. Bumpkit keeps learning from new outcomes on its own.

| Method | When |
|--------|------|
| `workspace` | Learned from your own release outcomes |
| `global` | Your workspace doesn't have enough outcomes yet and opted in to cross-workspace learning |
| `heuristic` | Not enough outcomes yet. No score is shown |

```bash
bumpkit report confidence
```

`GET /v1/confidence` shows the method in use and how well past scores matched what happened. Owners and admins can relearn now:

```bash
bumpkit report retrain
```

The API is `POST /v1/confidence/retrain`. Retraining is written to the audit log as `confidence.retrain`.

### Cross-workspace learning

Pooling is off by default. When an owner or admin turns it on, the workspace's outcomes help score every opted-in workspace, and it benefits from theirs.

```bash
bumpkit report pooling on
bumpkit report pooling off
```

The API is `PUT /v1/confidence/settings` with `{"pooling": true}`.

## Risk model

The risk table shows how often changes to each repository, area and file were followed by a failure. An area is the first two path segments, like `services/auth`.

```bash
bumpkit report risk --repo acme/api --scope area --limit 50
```

`--scope` is `file` (default), `area` or `repository`; `--limit` is up to 500. The API is `GET /v1/risk?repository=&scope=&limit=`, returning `changes`, `failures`, `risk` and `lift` (risk over the workspace base rate) per row.

## Leadership reports

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flag: `leadership_dashboards`.

```bash
bumpkit report --days 30
bumpkit report --repo acme/api --team @acme/payments
bumpkit report --from 2026-07-01 --to 2026-09-30 --csv > q3.csv
```

| Flag | Default | Meaning |
|------|---------|---------|
| `--days` | 90 | How many days back (up to 730) |
| `--from`, `--to` | none | A date range, `YYYY-MM-DD`; both are required |
| `--repo` | all | One repository |
| `--team` | all | One CODEOWNERS team |
| `--csv` | off | Print the weekly breakdown as CSV |

The report covers gated releases in the period:

| Section | Contents |
|---------|----------|
| `totals` | Releases, shipped, blocked, needing approval, overridden, failed, incidents, average confidence and average risk |
| `authorship` | Agent vs human releases, lines and incidents, and the agent share of code and of incidents |
| `trend` | Weekly releases, average confidence and average risk |
| `repositories`, `teams` | The same trend per repository and per CODEOWNERS team |

The API is `GET /v1/reports/leadership?days=&from=&to=&repository=&team=`, and `&format=csv` returns the CSV with the columns `week, repository, team, releases, shipped, blocked, needs_approval, overridden, failed, incidents, agent_releases, agent_lines, human_lines, agent_incidents, human_incidents, avg_confidence, avg_risk`. The dashboard's **Reports** page shows the same with time ranges and an **Export** menu.

### Authorship coverage

Coverage tells you how much of your authorship data you can trust as fact. Every gated release in the period lands in one of three buckets:

| Bucket | Meaning |
|--------|---------|
| Verified | A hook session, an org telemetry match or an agent bot author shows who wrote it, or agent hooks or org telemetry were watching and saw no agent |
| Inferred signals only | No verification, but declared or inferred evidence: a co-author trailer, a pull request footer, an agent branch name or the authorship model |
| No signals | No verification and no evidence at all |

The **Reports** page leads with a headline like `Authorship verified on 34% of 120 releases; 66% had no agent instrumentation.`, a weekly chart of the three buckets and, when releases go unverified, a link to turn on [Agent Telemetry](/docs/agent-telemetry). These are counts of stored releases, so they are facts, not estimates.

```bash
bumpkit report authorship-coverage --days 30
bumpkit report authorship-coverage --repo acme/api --from 2026-07-01 --to 2026-09-30
```

The API is `GET /v1/reports/authorship-coverage?days=&from=&to=&repository=` and the MCP tool is `report_authorship_coverage`. The leadership report and the readiness report carry the same counts.

### Authorship model

For releases with no verified signal, Bumpkit can name a likely writer, learned from your own verified releases. It doesn't read your code, commit bodies or pull request descriptions.

**When it names a writer.** Only when it has proven accurate for that agent on your verified releases. The release then shows the likely agent with how often the model was right, for example `Likely Claude Code · 88% (model correct on 88% of 1,240 verified releases)`. Otherwise no writer is named and no percentage is shown.

**What it never does.** A model guess is an **inferred** signal. It never verifies a release, never overrides a release that is verified or watched by hooks or telemetry, and never hard-blocks: like an agent branch name, it can only send a release for a person's review, as a `when: agent` approval rule does for any release not confirmed as written by a person. Agent scorecards and agent counts on release checks leave it out.

`bumpkit report authorship-coverage` and `report_authorship_coverage` show how accurate the model has been per writer and which writers it may name.

### Release readiness report

A report for executives and auditors on one repository, or every repository, over a period. It opens with a verdict (**Ready to release**, **Release with conditions** or **Hold releases**) and its confidence, then five exhibits:

| Exhibit | Contents |
|---------|----------|
| Releases and gate outcomes | Each release's gate outcome, gate runs and confidence score; overrides and failures after release |
| Blocked and needs-approval reasons | The policy rules that held releases back, with an example message for each |
| Authorship | Who wrote each release (AI agents or people) and who committed the code, with the measured authorship confidence where there is one, plus authorship coverage: how many releases were verified, had only declared or inferred signals, or had none |
| Risk findings | Findings by severity and the top proven high and medium findings, each with its confidence level (high, medium or low) |
| Approval latency | How long releases waited for a person: median, 90th percentile and longest, and the slowest releases |

A percentage appears only where it is measured. Counts read straight from stored records carry none because they are facts. When authorship confidence isn't measured yet, the report says so. The verdict carries no percentage.

```bash
bumpkit report readiness --repo acme/api --days 30 --format pdf -o readiness.pdf
bumpkit report readiness --from 2026-07-01 --to 2026-09-30 --format csv -o q3-readiness.csv
```

`--format` is `pdf` (default), `csv` or `json`. Without `-o`, PDF goes to `bumpkit-readiness.pdf` and CSV and JSON print to standard output. The API is `GET /v1/reports/readiness?days=&from=&to=&repository=&format=`, the MCP tool `report_readiness` returns the JSON, and the **Export** menu on the **Reports** page downloads the PDF, CSV or JSON for the period and repository on screen.

## Proven changes

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flag: `proven_changes`.

Proven changes learns which kinds of change, in which part of each repository, ship without problems. It only measures: the release gate doesn't change, and nothing skips review.

```bash
bumpkit proven --repository acme/api
bumpkit proven show apps/web ui --repository acme/api
```

**A class** is a repository, an area and a kind. One pull request can count toward several classes. The author isn't part of a class; each class shows the split.

- **Area:** a name from `areas:` in [`bumpkit.yml`](/docs/bumpkit-yml#areas). Otherwise the top-level directory, or two levels under the `apps`, `packages`, `services`, `cmd` and `internal` directories. Files at the top of the repository are in `root`.
- **Kind:** `tests`, `migrations`, `auth`, `api` and `ui` from the file paths, `deps` when a change only touches manifests and lockfiles, `docs` when it only touches documentation, and `code` for everything else.

**Shipped** means merged, or deployed when Bumpkit sees a deploy of that commit after the merge.

**A problem** is an outcome outside the author's control: a revert found in the diff, an incident, a hotfix or a regression. Commit messages never count. A later pull request counts as a revert when it undoes at least 80% of a change's lines in the same files.

**Clean** means shipped at least 14 days ago with no problem. Newer changes are pending. The window is the last 90 days.

| State | What it means |
|-------|---------------|
| Proven | 95% sure that fewer than 5% of these changes cause a problem. That takes 59 settled changes with no problem, or about 93 with one |
| Earning | No problems so far, and not enough settled changes yet, like `41 of about 59` |
| Paused | A change caused a problem, and the class doesn't have enough settled changes to clear it, or has no incident source yet. Open the class to see it |
| Excluded | The kind is on `proven.never`. `deps`, `migrations` and `auth` are on it by default |
| Not enough signal | No [incident source](/docs/release-health) is connected to the repository. No signal never reads as clean |

**The backtest** answers one question for each repository: if this had been on for the last 90 days, how many pull requests would have skipped review, and how many of those caused problems? It counts merged pull requests where Bumpkit itself asked for an approval and every class of the change was already proven at that moment, using only what was known then.

When your [data retention](/docs/data-handling) removes a change's file paths, Bumpkit says the class can't be computed instead of guessing. In retention modes that don't store code, Bumpkit can't compare lines, so it says it can't find reverts in the diff for those changes.

## Agent scorecards

> **Beta, early access.** A workspace without access gets `403 FEATURE_EARLY_ACCESS` and a link to request it. Flag: `agent_scorecards`.

Scorecards show how each coding agent's releases fared over a window.

```bash
bumpkit scorecards --days 30
```

`--days` is 1 to 365 (default 30). Per agent you get checks, ship, review and blocked counts, gated releases, how many needed approval or failed the gate, proven high findings, lines changed, the ship rate and the top finding kinds. The API is `GET /v1/scorecards?days=`, and agents use the `agent_scorecards` [MCP tool](/docs/mcp) with `days`.

See the [CLI Reference](/docs/cli) and [API Reference](/docs/api-reference) for every flag and field.