Proof of Training

Inference is not the only kind of work AETRON verifies. Training a model is also proven, through a mechanism called Proof of Training (PoT).

The Challenge

Training runs for days or weeks, so re-running it for verification is impossible. There is also a substitution risk: a dishonest miner could download or buy a finished model and try to pass it off as its own work. The loss curve would look right, and probes would show progress, so naive checks would not catch it.

PoT has to prove that real training happened, that the miner owns the trajectory it claims, and it has to do so at a tiny fraction of the cost of the training itself.

Witnessed Checkpoints

The core mechanism is Witnessed Checkpoints, and it rests on one fact: a single training step (forward pass, backward pass, optimizer update) is deterministic. Given the same weights, the same optimizer state, and the same input batch, it reproduces bit for bit.

That property is not assumed, it was measured. In AETRON's testing the single step reproduced bit-identically across the numeric formats, batch sizes, attention backends and multi-GPU configurations a training task can declare. Non-determinism in a full training run comes from changing seeds, dataloader shuffling and other external state, not from the step itself. This is what turns a stochastic training problem into a deterministic one-step problem that a verifier can check cheaply.

What the miner commits before training starts

Before the first step, the miner writes a commitment to the trajectory it is about to produce:

• A hash of the initial weights together with the initial optimizer state.
• A commitment to the sample schedule: the ordered list of dataset indices that will be used at every step.
• A hash of the committed dataset, which fixes what data is available at all.
• The canonical execution spec: dtype, attention backend, dropout, optimizer, determinism flags and the target hardware class.

In the runtime this is the start_training_commitment extrinsic. It records the initial hash, the schedule commitment, the dataset hash, the canonical spec, the checkpoint root and the declared number of steps, and it is idempotent per training run, so a miner cannot quietly re-commit a different trajectory later.

What gets committed during training

At every step the miner produces a hash over the new weights plus the new optimizer state. Those per-step hashes are organized into a Merkle tree, and only the root reaches the chain. One root covers the whole run, regardless of how many steps it has.

How a checkpoint is challenged

1. The protocol picks a random step using verifiable randomness derived from chain state, so the miner cannot know in advance which step will be examined.
2. The miner is asked for the state just before that step: the weights, the optimizer state, the batch that was used, and Merkle proofs placing the surrounding hashes inside the committed root.
3. The verifier checks the commitment first. The supplied previous state must hash to the committed value, and the batch must be the one the committed schedule assigns to that step.
4. The verifier runs one training step on that state.
5. The result is compared against the committed hash for the step. A match means the step was honest, a mismatch means the trajectory was not produced the way it was claimed.

Verifiers do not publish a bare yes or no. They commit to their recomputed result first and reveal it later through reveal_witness_value, which carries the recomputed hash and a norm bucket bound to a salt and the verifier's own identity. The verdict comes from the largest cluster of matching values rather than from a majority of opinions, so a single verifier that reports a random hash simply falls outside the cluster and contributes nothing. For cross-architecture cases the miner declares the reference norm bucket for the challenged step through declare_witness_norm_bucket, and the winning cluster is compared against that instead of a bit-exact hash.

Randomness inside the step

Production training uses dropout, usually around 0.1, and dropout draws from the global RNG state. Two runs of the same step would then produce different masks, different gradients and different weights, which would make replay impossible even for an honest miner.

The canonical spec closes this by requiring an RNG seed policy that the miner must follow before every forward pass. The common form derives the seed per step from a committed base seed and the step index, so the verifier reconstructs the exact same seed from the step it was assigned. A policy that seeds once at the start also works, at the cost of carrying RNG state in the checkpoints. Testing confirmed the difference clearly: with a seed derived per step, repeated runs were bit-identical, and without a seed policy no two runs agreed. A task config that declares non-zero dropout without a seed policy is rejected.

Why substitution fails

Suppose a miner obtained a finished trajectory some other way and commits its hashes as its own. When a random step is challenged, there are two options.

The miner can supply the previous state from that borrowed trajectory. The commitment check passes, because those hashes are what it committed. But the batch is not free: the committed schedule fixes which sample belongs to that step. The verifier runs the step with the borrowed state and the scheduled batch. If the trajectory really was trained on the committed dataset with that schedule, the result matches, and that is honest training rather than substitution. If it was trained on anything else, the result does not match.

The alternative is to construct a previous state that happens to produce the claimed next hash. That means inverting an optimizer step through a model's backward pass, and it also fails the commitment check on the previous state, because the fabricated state does not hash to the committed value.

Detection does not rest on a hardness assumption the way a zero-knowledge proof does. It follows from the step being a deterministic function.

Partial cheating

An attacker who tampers with only a fraction of the steps is playing a sampling game against challenges it cannot predict. The odds move against it on both axes: the more steps it fakes, the more likely any single challenge lands on one, and the longer a run stays open to challenges, the less a small share of tampering helps.

The economics do the rest. A single detected substitution slashes the stake, so a strategy of being almost honest is not profitable even when the per-challenge odds look survivable.

Cost

Replaying one step is a forward pass, a backward pass and an optimizer update, which is seconds of work at most even for a full fine-tune. Against a long run checked at a hundred points, the verifier spends minutes where the training itself took hundreds of hours, on the order of 0.01 percent overhead.

Secondary Checks

Witnessed Checkpoints are the primary mechanism. Cheaper runtime checks run alongside them and need no GPU at all: the committed loss sequence is checked for monotonic decrease within a noise tolerance, for a rate of decrease consistent with the model and compute, for a loss that stops moving across many checkpoints, and for sudden jumps backwards. These do not prove training on their own, they catch obviously broken trajectories before anyone spends compute on a replay.

Backdoor Defense

There is a class of attack that passes every check above. An attacker who controls one stage of a pipeline trains a backdoor direction offline, then during honest collaborative training adds a small multiple of that direction to its own weights every so often. Each step is a genuine forward, backward and optimizer update, the pulse is applied in the same format as a normal update, and the committed hash replays exactly. The single-step check passes because nothing about the step was faked.

The effect is invisible in ordinary metrics. Validation loss stays where honest training would put it, and the model behaves safely until the trigger appears. Gensyn published this attack as "Backdoor in the Middle" with a 94 percent success rate and stated it planned to investigate countermeasures.

AETRON's answer combines several independent signals. Some are computed from the weight deltas each stage commits, and one reads the model's behavior on held-out probes rather than its weights, so an attacker tuning against one of them still faces the others. The detailed design and its calibration are not published.

Swarm Mode

Swarm Mode is the distributed form of training, where a model too large for one machine is split across several miners in a pipeline and each miner holds one stage. The backdoor defense stack exists because of it: pipeline stage injection is only possible when someone controls a stage rather than the whole model.

The specification is direct about scope. Swarm Mode is not part of the first phase, which covers single-miner training tasks up to roughly 70B parameters on one 8xH100 node, and where each miner holds the complete model there is no middle stage to attack. Swarm Mode is intended for the 200B to 1T parameter range, where a single node is no longer enough, and the defense stack is designed and validated against attack variants but not implemented in the first phase runtime.

Where to Go Next

• Proof of Intelligence: how training verification fits alongside inference verification.
• Execution Spec: the canonical execution parameters that make any replay reproducible.
• Attack Resistance: the attack classes PoI was tested against and the layer that closes each one.