Internals

Release process

How a merge to main becomes a runtime upgrade on devnet, testnet, and mainnet, and how the release train's artifacts are published.

View as Markdown

Shipping is a single pipeline: release-train.yml runs on every push to main, builds the runtime once, and promotes that identical wasm through the networks. A second workflow, watch-mainnet-release.yml, watches the chain and publishes release artifacts once the mainnet upgrade actually executes.

The release train

push to main
  └─ srtool build (once per train)
       └─ deploy devnet ───── smoke-check devnet
            └─ deploy testnet ── smoke-check testnet ── publish SDK rc to PyPI
                 └─ propose mainnet upgrade (multisig)

Build once (srtool)

The train builds the runtime with srtool — a pinned Docker build environment that makes the wasm byte-reproducible. The artifact (subtensor.wasm + digest) is uploaded once and reused by every deploy job; promotion never rebuilds. Anyone can verify the hash independently by running scripts/srtool/build-srtool-image.sh followed by scripts/srtool/run-srtool.sh locally (see Repository scripts).

The ship lever: spec_version

Every deploy job first compares the built runtime's spec_version (runtime/src/lib.rs) with what the target chain is running, and no-ops unless the build is newer. Consequences:

  • A merge without a spec_version bump builds and then does nothing — this is deliberate, and it's why the Spec Version Check on PRs exists (label no-spec-version-bump to opt out knowingly).
  • Stale queued trains are harmless: they no-op at each guard.
  • Rollback is not a concept here — you ship a newer fixed version.

Promotion gates

Human approval lives in GitHub environment settings, not workflow code:

StageEnvironmentGate
devnetdevnetnone — deploys automatically, then runs the devnet smoke suite
testnettestnetenvironment reviewers (one-click approval), then smoke suite + SDK release candidate to PyPI
mainnetmainnetenvironment reviewers plus the multisig ceremony below

Deploys to devnet and testnet are direct setCode transactions via the CI deploy key. After the on-chain spec_version is verified, the train moves the corresponding network mirror branch. That branch push triggers both Docker publishers from the exact deployed commit, updating the matching :devnet or :testnet tag in ghcr.io/raofoundation/subtensor and ghcr.io/raofoundation/subtensor-localnet. It does not move either package's :latest tag. The smoke suite then validates the live network while the independent image builds run.

The mainnet multisig ceremony

CI never upgrades mainnet directly. The train's final job submits a multisig proposal: the CI key is one half of a 2-of-2 deployment multisig that holds a SudoUncheckedSetCode proxy on the sudo key. The triumvirate then approves the proposal 2-of-3 out-of-band — no GitHub credential can unilaterally change the mainnet runtime. Until they sign, nothing happens on chain.

For rehearsal, the mainnet-clone PR label spins up a live clone of mainnet running your runtime with a public endpoint (mainnet clone testing).

The release watcher

watch-mainnet-release.yml polls mainnet every 10 minutes. When the on-chain spec_version matches the main branch and no GitHub release exists for it yet, it cuts the release train's artifacts:

  1. GitHub release v<spec_version>
  2. Production Docker images tagged v<spec_version> and latest
  3. Python SDK + bittensor-core wheels to PyPI (trusted publishing, with PEP 740 provenance attestations)
  4. Publishable Rust crates to crates.io (auto-discovered; workspace crates are publish = false while they depend on the patched polkadot-sdk fork)
  5. Production website/docs to Vercel

This ordering means the release always reflects what is actually running on mainnet, not what was merged.

The watcher explicitly dispatches docker.yml and docker-localnet.yml because releases created with the default GITHUB_TOKEN do not emit another workflow event. The production build publishes both the immutable release tag and :latest. The localnet build publishes the immutable release tag in its separate package; localnet :latest remains an alias for the current :main branch image, while :devnet and :testnet follow their respective branches.

Hotfixes

The same pipeline applies: land the fix on main with a spec_version bump and let the train promote it. There is no side channel that skips devnet and testnet validation.