Internals
Release process
How a merge to main becomes a runtime upgrade on devnet, testnet, and mainnet, and how the release train's artifacts are published.
Shipping is a single pipeline: release-train.yml runs on every push to
main, builds the runtime once, and promotes that identical wasm through the
networks. A second workflow, watch-mainnet-release.yml, watches the
chain and publishes release artifacts once the mainnet upgrade actually
executes.
The release train
push to main
└─ srtool build (once per train)
└─ deploy devnet ───── smoke-check devnet
└─ deploy testnet ── smoke-check testnet ── publish SDK rc to PyPI
└─ propose mainnet upgrade (multisig)Build once (srtool)
The train builds the runtime with srtool
— a pinned Docker build environment that makes the wasm byte-reproducible.
The artifact (subtensor.wasm + digest) is uploaded once and reused by every
deploy job; promotion never rebuilds. Anyone can verify the hash independently
by running scripts/srtool/build-srtool-image.sh followed by
scripts/srtool/run-srtool.sh locally (see
Repository scripts).
The ship lever: spec_version
Every deploy job first compares the built runtime's spec_version
(runtime/src/lib.rs) with what the target chain is running, and no-ops
unless the build is newer. Consequences:
- A merge without a
spec_versionbump builds and then does nothing — this is deliberate, and it's why the Spec Version Check on PRs exists (labelno-spec-version-bumpto opt out knowingly). - Stale queued trains are harmless: they no-op at each guard.
- Rollback is not a concept here — you ship a newer fixed version.
Promotion gates
Human approval lives in GitHub environment settings, not workflow code:
| Stage | Environment | Gate |
|---|---|---|
| devnet | devnet | none — deploys automatically, then runs the devnet smoke suite |
| testnet | testnet | environment reviewers (one-click approval), then smoke suite + SDK release candidate to PyPI |
| mainnet | mainnet | environment reviewers plus the multisig ceremony below |
Deploys to devnet and testnet are direct setCode transactions via the CI
deploy key. After the on-chain spec_version is verified, the train moves the
corresponding network mirror branch. That branch push triggers both Docker
publishers from the exact deployed commit, updating the matching :devnet or
:testnet tag in ghcr.io/raofoundation/subtensor and
ghcr.io/raofoundation/subtensor-localnet. It does not move either package's
:latest tag. The smoke suite then validates the live network while the
independent image builds run.
The mainnet multisig ceremony
CI never upgrades mainnet directly. The train's final job submits a multisig
proposal: the CI key is one half of a 2-of-2 deployment multisig that holds
a SudoUncheckedSetCode proxy on the sudo key. The triumvirate then approves
the proposal 2-of-3 out-of-band — no GitHub credential can unilaterally
change the mainnet runtime. Until they sign, nothing happens on chain.
For rehearsal, the mainnet-clone PR label spins up a live clone of mainnet
running your runtime with a public endpoint
(mainnet clone testing).
The release watcher
watch-mainnet-release.yml polls mainnet every 10 minutes. When the on-chain
spec_version matches the main branch and no GitHub release exists for it
yet, it cuts the release train's artifacts:
- GitHub release
v<spec_version> - Production Docker images tagged
v<spec_version>andlatest - Python SDK + bittensor-core wheels to PyPI (trusted publishing, with PEP 740 provenance attestations)
- Publishable Rust crates to crates.io (auto-discovered; workspace crates
are
publish = falsewhile they depend on the patched polkadot-sdk fork) - Production website/docs to Vercel
This ordering means the release always reflects what is actually running on mainnet, not what was merged.
The watcher explicitly dispatches docker.yml and docker-localnet.yml
because releases created with the default GITHUB_TOKEN do not emit another
workflow event. The production build publishes both the immutable release tag
and :latest. The localnet build publishes the immutable release tag in its
separate package; localnet :latest remains an alias for the current :main
branch image, while :devnet and :testnet follow their respective branches.
Hotfixes
The same pipeline applies: land the fix on main with a spec_version bump
and let the train promote it. There is no side channel that skips devnet and
testnet validation.