# Release process (/docs/internals/release-process)

Shipping is a single pipeline: **`release-train.yml`** runs on every push to
`main`, builds the runtime once, and promotes that identical wasm through the
networks. A second workflow, **`watch-mainnet-release.yml`**, watches the
chain and publishes release artifacts once the mainnet upgrade actually
executes.

## The release train [#the-release-train]

```
push to main
  └─ srtool build (once per train)
       └─ deploy devnet ───── smoke-check devnet
            └─ deploy testnet ── smoke-check testnet ── publish SDK rc to PyPI
                 └─ propose mainnet upgrade (multisig)
```

### Build once (srtool) [#build-once-srtool]

The train builds the runtime with [srtool](https://github.com/paritytech/srtool)
— a pinned Docker build environment that makes the wasm **byte-reproducible**.
The artifact (`subtensor.wasm` + digest) is uploaded once and reused by every
deploy job; promotion never rebuilds. Anyone can verify the hash independently
by running `scripts/srtool/build-srtool-image.sh` followed by
`scripts/srtool/run-srtool.sh` locally (see
[Repository scripts](/docs/internals/scripts)).

### The ship lever: `spec_version` [#the-ship-lever-spec_version]

Every deploy job first compares the built runtime's [`spec_version`](/code/runtime/src/lib.rs#L238)
(`runtime/src/lib.rs`) with what the target chain is running, and **no-ops
unless the build is newer**. Consequences:

* A merge without a `spec_version` bump builds and then does nothing — this
  is deliberate, and it's why the Spec Version Check on PRs exists (label
  `no-spec-version-bump` to opt out knowingly).
* Stale queued trains are harmless: they no-op at each guard.
* Rollback is not a concept here — you ship a *newer* fixed version.

### Promotion gates [#promotion-gates]

Human approval lives in GitHub **environment settings**, not workflow code:

| Stage   | Environment | Gate                                                                                         |
| ------- | ----------- | -------------------------------------------------------------------------------------------- |
| devnet  | `devnet`    | none — deploys automatically, then runs the devnet smoke suite                               |
| testnet | `testnet`   | environment reviewers (one-click approval), then smoke suite + SDK release candidate to PyPI |
| mainnet | `mainnet`   | environment reviewers **plus** the multisig ceremony below                                   |

Deploys to devnet and testnet are direct `setCode` transactions via the CI
deploy key. After the on-chain `spec_version` is verified, the train moves the
corresponding network mirror branch. That branch push triggers both Docker
publishers from the exact deployed commit, updating the matching `:devnet` or
`:testnet` tag in `ghcr.io/raofoundation/subtensor` and
`ghcr.io/raofoundation/subtensor-localnet`. It does not move either package's
`:latest` tag. The smoke suite then validates the live network while the
independent image builds run.

### The mainnet multisig ceremony [#the-mainnet-multisig-ceremony]

CI never upgrades mainnet directly. The train's final job submits a **multisig
proposal**: the CI key is one half of a 2-of-2 deployment multisig that holds
a `SudoUncheckedSetCode` proxy on the sudo key. The triumvirate then approves
the proposal 2-of-3 **out-of-band** — no GitHub credential can unilaterally
change the mainnet runtime. Until they sign, nothing happens on chain.

For rehearsal, the `mainnet-clone` PR label spins up a live clone of mainnet
running your runtime with a public endpoint
([mainnet clone testing](/docs/internals/mainnet-clone)).

## The release watcher [#the-release-watcher]

`watch-mainnet-release.yml` polls mainnet every 10 minutes. When the on-chain
`spec_version` matches the `main` branch and no GitHub release exists for it
yet, it cuts the release train's artifacts:

1. **GitHub release** `v<spec_version>`
2. **Production Docker images** tagged `v<spec_version>` and `latest`
3. **Python SDK + bittensor-core wheels** to PyPI (trusted publishing, with
   PEP 740 provenance attestations)
4. **Publishable Rust crates** to crates.io (auto-discovered; workspace crates
   are `publish = false` while they depend on the patched polkadot-sdk fork)
5. **Production website/docs** to Vercel

This ordering means the release always reflects what is *actually running on
mainnet*, not what was merged.

The watcher explicitly dispatches `docker.yml` and `docker-localnet.yml`
because releases created with the default `GITHUB_TOKEN` do not emit another
workflow event. The production build publishes both the immutable release tag
and `:latest`. The localnet build publishes the immutable release tag in its
separate package; localnet `:latest` remains an alias for the current `:main`
branch image, while `:devnet` and `:testnet` follow their respective branches.

## Hotfixes [#hotfixes]

The same pipeline applies: land the fix on `main` with a `spec_version` bump
and let the train promote it. There is no side channel that skips devnet and
testnet validation.