Transit Fabric releases

Taken from the Transit Fabric source repository at commit 91be12e (91be12ef98b775c7f045124e6b8c686c15439650, main), file docs/specs/measurement.md. Links into the private source tree are shown as plain text. Raw Markdown · Red-team pack

Spec: measurement

Status: draft · Probe schedule and canary plans implemented in crates/bwos-core/src/probe.rs and probe_canary.rs · Vectors: testdata/vectors/probe_schedule.json · ADR-0039 (proposed) adds the sections marked with it. They are ahead of the code, and each says which Phase B task implements it (PB-06).

Canary plans (ADR-0051, ADR-0053, Task 14)

Implemented in crates/bwos-core/src/probe_canary.rs, re-exported from probe. Vectors: the canary object in probe_schedule.json and the canary-plan entry in canonical.json. Not scored. How canary carriage is paid is PB-38, and nothing in run_tempo reads a canary plan yet.

The always-on canary stream (ADR-0051 §1) is built in four parts:

What this replaces and what stays.

Allocation: a verifiable draw (ADR-0051 requirement 1, ADR-0053 M5)

probe::canary_allocation_deduped(seed, window, snapshot, caps, budget_bytes) -> CanaryAllocation and probe::canary_allocation(seed, window, snapshot, budget_bytes), which is the same with caps read from the snapshot's capacity_mbps. CanaryAllocation { window, pairs } maps (ingress, dst) to bytes for the window.

Inputs, all committed.

The draw.

  1. Eligible nodes. Snapshot nodes with caps[n] > 0 whose key is a valid route-table identifier of at most CANARY_MAX_KEY_LEN (64) bytes, as an ed25519 key in hex is. A key listed twice keeps its first UID.
  2. Bounded weight. w_n = min(caps[n], CANARY_NODE_WEIGHT_CAP_MBPS).
  3. Per-UID cap, water-filled (fix round 1, F4). UID totals W_u become min(W_u, C), where the level C solves C = c * Σ_u min(W_u, C) with c = CANARY_UID_CAP_PPM / 1e6, so no UID holds more than c of the capped total. With UID totals sorted descending, C is c_ppm * S_k / (1e6 - k * c_ppm) (u128, floor) for the smallest k whose next total W_(k+1) is at most it, S_k the sum of the totals after the first k. No cap binds when k = 0. With too few UIDs for any level (k * c_ppm reaching 1e6), every UID is cut to the smallest UID's total. A held UID's nodes are scaled to w_n * C / W_u (floor), and nodes scaled to 0 drop out. (The first version made one pass against the uncapped total, which left a UID with 90% of the raw weight 69%.)
  4. Too many nodes. If more than CANARY_MAX_PAIRS nodes remain, only the CANARY_MAX_PAIRS with the lowest ticket SHA-256("BWOS-v0/canary-alloc" || 0x00 || seed || snapshot_hash || u64be(window) || u32be(len(n)) || n), compared as 32-byte big-endian numbers, are kept. The ticket doesn't depend on weight, so the cut is the same lottery for every node.
  5. Ingress quota. q_n = budget_bytes * w_n / W' (u128, floor), where W' is the total weight after the cap and the cut.
  6. Draws per node. d_n = clamp(q_n / CANARY_CHUNK_BYTES, lo_n, CANARY_MAX_PAIRS_PER_NODE). The floor lo_n is CANARY_MIN_PAIRS_PER_NODE for a node with caps[n] >= CANARY_FLOOR_MIN_MBPS that is among the first CANARY_FLOOR_NODES_PER_UID nodes of its UID, ordered by weight descending, then key. It is 1 for any other node. If Σ d_n > CANARY_MAX_PAIRS (M), the draws are cut in proportion (fix round 1, F7). Each d_n becomes max(1, floor(d_n * M / Σ d)). What that leaves under M goes back one draw at a time by largest remainder d_n * M mod Σ d (then key), never above the old d_n, and anything over M from the minimum of 1 comes off the largest counts first (then key). So at 3_072 nodes, where the floors alone exceed M, every node keeps about its floor, and one big node can't collapse the rest to one draw.
  7. Each draw. Draw j < d_n of node n carries q_n / d_n bytes, plus 1 for j < q_n mod d_n. Its destination is a weighted pick by w among the eligible nodes of other UIDs:
    r   = u128be(SHA-256("BWOS-v0/canary-alloc" || 0x00 || seed || snapshot_hash || u64be(window) || u32be(len(n)) || n || u32be(j))[0..16])
    dst = the node whose weight interval, over nodes sorted by (uid, key) with n's UID left out, holds r mod (W' - W'_uid(n))
    
    W'_uid(n) is the weight of n's UID. A draw whose UID holds all the weight has no destination and is dropped, and so is a draw of 0 bytes.
  8. Per-pair cap. A pair (n, dst) sums its draws, capped at CANARY_PAIR_MAX_BYTES. Anything over the cap is dropped, not moved.

snapshot_hash = SHA-256(signing_bytes("snapshot-attestation", snapshot)): the bytes validators attest (signed::SignedSnapshot). n is the node's key as it appears in the snapshot. Modulo bias is accepted.

What the draw guarantees.

The budget (ADR-0051 §5). CANARY_BUDGET_BYTES_PER_WINDOW is 27_000_000_000 bytes, a starting value. That is 50 Mbps averaged over a 4_320 s window, fabric-wide. The budget is global, so the per-node rate falls as the fabric grows:

Fabric Ingress bytes per node per window Average ingress rate per node Whole stream, per month (600 windows)
100 equal nodes 270 MB 0.5 Mbps 16.2 TB
1,000 equal nodes 27 MB 50 kbps 16.2 TB

A node also carries the canaries routed through it as a relay, so its carried bytes are a few times its ingress bytes. A validator sources and sinks its stake share of the stream: a validator with 1% of the counted stake sends and receives 270 MB a window, about 162 GB a month each way, whatever the fabric's size. PB-38 sets the final budget.

Stake shares (ADR-0053 §4, OQ-105)

probe::canary_stake_shares(stakes, excluded_stake_hotkeys) -> BTreeMap<hotkey, ppm>.

A validator's share of pair p is probe::validator_pair_share(B_p, share) = B_p * min(share, 1e6) / 1e6 (floor), or 0 when that is below CANARY_MIN_FLOW_BYTES.

Plans

{"entries":[{"bytes":1200,"class":"latency","dst":"<node hex>","ingress":"<node hex>","start_ms":31337}],"v":0,"validator":"<ss58 hotkey>","window":1234}

probe::canary_plan(secret, allocation, stake_share_ppm, marginals) -> CanaryPlan. It leaves validator empty. The caller sets it to its SS58 hotkey before committing.

  1. Shaping input. marginals.effective(): the committed GlobalMarginals, with any empty or invalid part replaced by the default mix's part.
  2. Draws. Draw i is u64be(SHA-256("BWOS-v0/canary-draw" || 0x00 || secret || u64be(window) || u64be(i))[0..8]), with i counting up from 0 over the whole plan. Nobody without the 32-byte secret can compute timing or sizes before the reveal.
  3. Pairs in order. Pairs go in ascending (ingress, dst) order, each with share s = validator_pair_share(B_p, stake_share_ppm), skipping zeros. Flows are drawn until s is spent. Each flow takes exactly five draws, in this order: size bucket, size, class, slice, offset. A weighted pick by r over weights w_0..w_k takes r mod Σ w (u128) and walks the weights in order (sizes by bucket order, classes by name).
    • Size. Pick a size bucket b by weight, then the size lo + 1 + r mod (hi - lo), where hi is b's max_bytes and lo the previous bucket's (0 for the first). Raise it to at least CANARY_MIN_FLOW_BYTES and cut it to what's left. If what would be left is below CANARY_MIN_FLOW_BYTES, the flow takes all of it.
    • Entry limit. If CANARY_MAX_ENTRIES - (entries so far + 1) is at most the number of pairs still to spend after this one, the flow takes the pair's whole remainder, so every pair gets at least one entry within the limit. The allocation caps a pair at CANARY_PAIR_MAX_BYTES, which fits a u32.
    • Class. Pick one by the class weights.
    • Start. Pick a timing slice k by weight, then start_ms = k * L + r mod len_k. The window is CANARY_WINDOW_MS (4_320_000 ms), cut into slices of L = CANARY_WINDOW_MS / slices, with the remainder in the last (len_k = L, or what's left for the last).
  4. Order. Entries are sorted ascending by (start_ms, ingress, dst, class, bytes).

GlobalMarginals (canonical JSON, probe::GlobalMarginals):

{"classes":{"latency":500000},"sizes":[{"max_bytes":1500,"weight":250000}],"timing":[1,1],"v":0}

The default mix (GlobalMarginals::default_mix()), used when there is no data:

Part Default
sizes (max_bytes: weight ppm) 1_500: 250_000 · 16_000: 250_000 · 128_000: 200_000 · 1_000_000: 150_000 · 8_000_000: 100_000 · 64_000_000: 40_000 · 256_000_000: 10_000 (elephants)
classes latency: 500_000 · bulk: 300_000 · realtime: 200_000
timing uniform: one slice

The producer is a stub. Every validator uses GlobalMarginals::default_mix() until PB-38 or PB-39 adds a producer and commits the marginals with the window inputs. The default is a constant in the code, so all validators agree on it. The producer MUST build the marginals from validator-observed, fabric-wide counts only.

Commit and reveal (ADR-0051 requirement 2, finding 8)

The revealed object is a CanaryReveal, domain canary-plan:

{"plan":{"entries":[...],"v":0,"validator":"<ss58>","window":1234},"secret":"<64 lowercase hex>","v":0}

Injection: validator-run ends (ADR-0051 requirement 3)

Counted once (ADR-0051 requirement 4, finding 9)

Canary bytes count in one place only, the canary evidence PB-38 scores.

Residuals (T19)

Sims (Task 14)

They run the real draw and plan generator; nothing is scored until PB-38, so each measures what PB-38 will read. The customer flows in canary_favourer come from the plan generator itself, so they share its quirks; PB-38 needs an independent customer model (OQ-121).

Shared probe schedule

Every validator derives the same probe schedule for a window. It times the validator probe agent's link:, path:, route: and direct: samples. It does not time the canary stream, which follows the canary plans above:

seed      = SHA-256("BWOS-v0/probe-seed" || 0x00 || block_hash_32_bytes || u64be(window))
rand(i)   = u64be(SHA-256(seed || u64be(i))[0:8])
paths     = path IDs from the window's topology snapshot, sorted ascending (bytes)
for slot j in 0 .. N_PROBES-1:
    path_j   = paths[rand(2j)   mod len(paths)]
    offset_j = rand(2j+1) mod WINDOW_BLOCKS

block_hash is the hash of the first block of the window. Modulo bias is accepted (documented, negligible at these sizes).

Operators can compute this schedule, so a node can tell when a probe sample is due (T19). That is why it times only probe samples, which are measurements aggregated by stake-weighted median, and not the canary stream (§Canary plans). Active UDP canary datagrams (probe::run_udp_canary_probe) carry no magic prefix, window, sequence number or deterministic padding: every byte is fresh OS randomness and replies are matched by exact payload equality, so a node cannot recognize canaries by content ([SCR-6]). Echo responders echo any MIN_UDP_PROBE_BYTES..=MAX_UDP_PROBE_BYTES (20..=1400) byte datagram unchanged.

Path probes are keyed by path (Task 16a fix round 1, red team M4). A probe of an overlay path P is keyed route:<hex SHA-256 of P's node IDs joined by single spaces> (probe::route_path_key; node IDs are printable ASCII without whitespace, so the join is unambiguous, and the key fits the 256-byte limit at any path length). Mechanism 1 grading reads only these (scoring.md §Grading item 3).

Older helpers. The older helpers (validator::evaluate_solver_table*, the bwos CLI preview) still read path measurements keyed path:uid:<uid>[:<class>]:<src>:<dst>. Label-keyed results path:<table.solver>[:<class>]:<src>:<dst> are used only for a submission whose table.solver label is unique among the tempo's submissions ([SCR-4]). run_tempo reads neither.

Probe vantage ([T22], [T29], OQ-16)

Every measurement that scoring reads comes from a validator-run probe agent and reaches scoring only inside the validator-signed ProbeResultBatch of that validator (batch.validator), revealed against its commitment and aggregated across validators (§Results and commit-reveal). Nodes, operators and solvers report nothing: no node-reported, node-timestamped or solver-reported latency, loss, jitter or throughput is ever a scoring input. TempoInput::current_snapshot (whose link and direct latencies feed predicted gains, link failover and Q_n predictions) MUST be the previous tempo's TempoOutcome::next_snapshot, which validator::build_snapshot_and_demand builds from aggregated validator samples only.

Key Originated by What is timed
vantage:<n> any validator probe agent UDP canary echo exchange between the agent and node n's echo responder (probe::vantage_key). Under ADR-0039 this is a STAMP exchange inside the WireGuard session n opened to the agent (§STAMP probes and probe sessions)
link:<a>:<b> the validator's probe agent nearest a canary handed to a's overlay ingress addressed to b, forwarded by a to b, echoed by b back to the agent
direct:<src>:<dst> the validator's probe agent nearest src canary echo exchange with dst's public endpoint over the public internet (not the overlay)
path:...:<src>:<dst> the validator's probe agent nearest src canary handed to src's ingress, carried over the production route's hops to dst, echoed back
cap:<n> validator probe agent capacity test with a node-signed ack (§Capacity tests)
capteam:<n>:<b> each team member's probe agent its share of the team capacity test of n scheduled at block b, echoed (§Team capacity tests, ADR-0039)

The canary stream (§Canary plans) is not a probe-result key. A validator's probe agent drives its own revealed plan through a validator-run source node, and its validator-run sink node receives it. Both ends are the validator's, and so are the clocks. What reaches scoring, from PB-38, is only what validators measured at their own ends: delivered bytes, and hop digests checked under keys only they hold. It is aggregated across validators like the samples here. A node's own report of canary traffic is never an input.

Rules:

  1. Only the agent's clock. Each sample is a timing of packets the agent itself sent and received (random payloads, [SCR-6]). Validators MUST NOT derive a sample by subtracting another measurement that an endpoint can inflate (for example link = composite - vantage leg): an endpoint that delays the subtracted leg would make its link look faster. The agent-to-ingress leg therefore stays inside link: and path: samples as a conservative bias (links look slower, never faster). With this rule an endpoint or hop can only add delay or loss to a sample it forwards or echoes.
  2. Indistinguishable vantage and direct probes. Before ADR-0039, an agent SHOULD send vantage:<n> and direct:*:<n> probes to n from the same socket, interleaved, so n cannot delay one class without the other.
    • What ADR-0039 changes. vantage:<n> moves inside the probe session n opened (§STAMP probes and probe sessions), while direct: stays a random-payload echo over the public internet. The two can no longer share a socket, so this rule applies to neither.
    • What remains. n can tell them apart and delay direct: alone. That is the OQ-16 (a) residual, unchanged in size.

Residual ([T22], check of the gain formula gain = (direct_us - overlay_us) * 1e6 / direct_us): forwarding nodes on an overlay path can only make overlay_us larger, which lowers their own gain. But the destination dst of a pair echoes the direct: probe, which arrives outside the overlay and is therefore distinguishable from overlay traffic, so dst can inflate direct_us for pairs that end at itself; that raises the gain of every overlay route to dst and can make a route slower than the true direct path look positive (Mechanism 1 score and the [SCR-9] Mechanism 0 creditability check). It is bounded only by the Mechanism 1 trimmed mean and by dst having to delay its own vantage probes consistently. Still open (OQ-16).

STAMP probes and probe sessions (ADR-0039)

Implemented by Phase B Task 4Da (codec and vectors), Task 4Db (sessions, sender, reflector, latency_samples) and Task 18b (the source match with the collector session).

Sessions.

Packets.

Reflector (the node).

Sender (the validator's agent). These rules keep §Probe vantage rule 1, under which an endpoint can only add delay:

  1. Fresh sequence numbers. Every test packet's seq is fresh OS randomness. RFC 8762's incrementing counter isn't used, and a stateless reflector copies whatever it gets.

  2. Unpredictable timing. A sample sends STAMP_PACKETS_PER_SAMPLE (10) test packets. The gaps between them are exponential with mean STAMP_MEAN_GAP_MS (100), drawn from OS randomness and clamped to 10..=1_000 ms, so a node can't predict when the next test leaves.

  3. Matching replies. A reply counts only if all of these hold:

    • its sender_seq and ssid match a test this agent sent on this session;
    • it arrives after that test left and within STAMP_REPLY_DEADLINE_MS (2_000);
    • it's the first such reply. Later duplicates are ignored.

    A node therefore can't answer a test before it arrives.

  4. Only the agent's clock. stamp_rtt_us = T4 - T1, where T1 is the send time the agent recorded locally for that seq and T4 is the agent's receive time, both on the agent's monotonic clock. The echoed sender_timestamp, receive_timestamp and timestamp are the node's to set. They go to dashboards only and never into any scored value (§Self-reported telemetry). In particular, RTT is never corrected by timestamp - receive_timestamp, which the node could inflate to make itself look faster.

  5. Loss. A test with no counted reply by the deadline is lost.

Where STAMP is used, and where it isn't (controller ruling 4, red team RT8).

The agent's view of the session (Task 4Db).

Residuals.

Capacity tests

Capacity tests must be unpredictable. A node n with declared capacity is tested at block b when:

u64be(SHA-256("BWOS-v0/cap-test" || 0x00 || block_hash(b-1) || node_pubkey)[0:8]) mod CAP_TEST_PERIOD == 0

Operators learn of a test at most one block ahead (12 s on mainnet), which is too short to provision fake capacity. A validator runs at most CAP_TEST_MAX (default 2) concurrent capacity tests per block, pushing traffic up to the node's declared capacity for CAP_TEST_SECONDS (default 10 s) from the nearest probe agent and recording achieved throughput (throughput_mbps).

Default CAP_TEST_PERIOD = 900 blocks (~3 h), so each node is tested a few times a day.

ADR-0056 (2026-11-08, specified, not scored) will replace the team test's validator-pushed load with peer-sourced load and hidden validator slices, called about 3 times a day (§Peer-load capacity tests). Until it scores, the rules below stand.

ADR-0039 replaces the single-agent test with a team test (§Team capacity tests, Phase B Task 4Dc). It keeps the schedule above, the request header and the cap-test-ack below, but changes three things:

Task 4Dc implements the team test, its schedule and its crediting (§Team capacity tests, "Code"). Scoring uses it once the validator loop supplies TempoInput.block_hashes (Task 19b) and endpoint dedupe is on (§Probed endpoints, "Dedupe", stake trigger; Task 4Dc fix round 5, T124); otherwise the single-agent test with CAP_TEST_SECONDS (10 s) and the single-agent rule are what run.

Delivery proof ([SCR-3], RT2-3):

  1. The prober opens the connection and sends a CAP_TEST_HEADER_BYTES (72-byte) header: validator_pubkey (32) || u64be(window) || test_id (32), where test_id is fresh OS randomness (CAP_TEST_ID_BYTES).
  2. It streams uniformly random bytes and half-closes the connection.
  3. The node's sink replies with u16be(len) || canonical(CapTestAck) (len <= MAX_CAP_TEST_ACK_BYTES), signed with the node key under domain cap-test-ack:
    {"v":0,"node":"<node pubkey hex>","validator":"<validator pubkey hex>","window":1234,"test_id":"<32-byte hex>","digest":"<CapTestDigest hex>","bytes":16777216,"sig":"<ed25519 hex>"}
    
    digest is SHA-256("BWOS-v0/cap-test-stream" || 0x00 || stream) over every stream byte received after the header.
  4. The prober stops the clock when the ack arrives and accepts it only if it verifies under the key of the node under test (the <node> in cap:<node>) and binds exactly this test's validator, window, test_id, digest of the bytes sent and byte count (CapTestAck::verify_for). Then throughput_mbps = bytes * 8 / elapsed_us.

Sink availability (MED-INFRA-01): the reference sink (bwos-probe cap-sink) keeps accepting connections until one valid test completes or its overall deadline expires (-timeout-secs, default 4 h), so a scanner cannot use up the sink before the validator connects. It serves each connection on its own thread, closes a connection that has not sent the full header within 5 s, and serves at most 32 connections at once (4 per source IP). It sends no ack, and does not count the test, if the header validator is not a valid non-weak ed25519 key (or is not in the -validator allowlist, or the header window differs from -window) or the stream carries no bytes. The UDP echo responder (bwos-probe echo) likewise counts echoes per peer ip:port (at most -count each, optional -allow <ip>) and stops when one peer completes or at its deadline. The header is not authenticated (wire format unchanged), so a scanner that forges a well-formed header with stream bytes can still complete the one test.

A connect failure, timeout, short read, or missing, foreign or replayed ack is recorded as a failed sample (rtt_us = 0, loss_ppm = 1_000_000, throughput_mbps = 0), never omitted, so it counts in the rolling lower median of capacity-bonds.md. The prober puts the accepted acks in its ProbeResultBatch.cap_acks. When aggregating, every validator scores a cap:<node> sample that claims success (rtt_us > 0 or throughput_mbps > 0) as a failed sample unless the batch carries an ack that is validly signed by <node>, names the batch's validator and window, and has bytes * 8 >= throughput_mbps * rtt_us (ProbeResultBatch::to_metric_map). A sink that does not hold the tested node's key therefore earns that node nothing, so N node keys cannot share another operator's real sink, and an ack cannot be replayed into a test with a different test_id.

Known gap (OQ-18): one operator that installs all N of its node keys on the same box can still answer N sequential tests with one pipe. When endpoint metadata is supplied (TempoInput::node_diversity), scoring pools the verified capacity of a dedupe group (IPv4 /24, or IPv6 /32 for IPv6-only endpoints) to its largest single member (scoring.md §Endpoint dedupe, [T24]), so N keys on one box earn one pipe's capacity term. A group can earn more only through a simultaneous cohort test (below).

Team capacity tests (ADR-0039)

This is FlashFlow's method: a team of measurers, an echoed-data check, and background traffic credited up to a cap. We take the method, not the code.

Validator identity: one hotkey, keys with proof of possession (review R2, red team RT1, round 4 RT8). This applies to every ADR-0039 rule: team draws, the evidence quorum, endpoint aggregation, and the stake-weighted probe median (§Results and commit-reveal).

Quorum over recorded stake (red team RT5).

Two stake sources (review RR2).

Inputs scoring needs to recompute every team (reviews R1, RR1). TempoInput gains these fields, all filled by the validator loop (Task 19b), so every validator recomputes the same teams:

Which window a test belongs to. A test due at block b ends at block b + CAP_TEST_BLOCKS - 1. It is recorded in the batch of the window containing that block. So a test that starts in a window's last CAP_TEST_BLOCKS - 1 blocks counts in the next window, and none is dropped at a boundary.

A capteam:<n>:<b> sample counts only if all of these hold:

Unscheduled or foreign samples are refused. They count for nothing, neither as a pass nor as a failure.

Eligible validators. E(b) is the set of hotkeys in the validator set in force at b that meet all of these.

Which sets scoring trusts (Task 4Dc fix rounds 1-2, red team F6, N2, review R10; validator::verified_set_chain): the walk starts from TempoInput.prev_validator_set, the set in force at the previous window, which the validator loop takes from chain state (Task 19b) and which scoring trusts as given; with signed sets and no anchor, E(b) is empty. From the current trusted set, a set is taken only if signed::accept_next_set accepts it: a higher block at most MAX_SET_ADVANCE_BLOCKS ahead and not past the current set's not_after (Task 4Dc fix round 3, red team R6: an expired set accepts no successor, where nodes re-anchor), attested by a SIGNED_INPUT_QUORUM_PPM quorum of the current set's stake. So an unattested or self-signed set can neither anchor nor veto, and a set at or below the anchor's block is ignored. Two different sets the quorum accepts at one block mean it equivocated: both are refused, the current set stays, and later blocks are judged from it. The set in force at b is the last trusted one with block <= b, and only while b <= not_after; past it E(b) is empty.

Until signed validator sets exist (sim and localnet only), E(b) is:

each with an unlimited budget.

Test rate: the ramp (red team RT4, RT5).

R(n) = min( ceil(declared(n) * 1000 * (1e6 + CAP_TEST_HEADROOM_PPM) / 1e6),
            (2 * verified_prev(n) + CAP_RAMP_FLOOR_MBPS) * 1000 )              in kbps

Rates and budgets are in kbps (Task 4Dc fix round 1, OQ-91), so a small node's shares aren't rounded to whole Mbps: a budget of B Mbps is B * 1000 kbps (probe::test_rate_kbps, probe::budget_kbps).

Slots, order and per-UID cap (reviews R4, RR1; re-review breakage 2).

Team draw. The draw is stake-weighted, without replacement, in integers only, using ValidatorEntry.stake at b, over all of E(b) with k = min(CAP_TEAM_MAX_VALIDATORS, |E(b)|) (vectors: testdata/vectors/cap_team.json):

seed      = SHA-256("BWOS-v0/cap-team" || 0x00 || block_hash(b-1) || subject)
            subject = the node's 32-byte key (node test) or the 32-byte cohort_id (cohort test)
remaining = E(b) in SS58 hotkey order
for j in 0 .. k-1:
    total = Σ stake over remaining                                  (u128)
    r     = u128be(SHA-256(seed || u32be(j))[0:16]) mod total
    pick the first validator in remaining whose cumulative stake > r; remove it from remaining

Member rates, capped by the team's budgets (red team RT2). The team can push at most R_team = min(R, Σ_i cap_share_budget_mbps_i) over its members, in kbps. It is split by water-filling, which is deterministic: take members in ascending order of budget, then hotkey, and give each rate_i = min(budget_i, ceil(remaining_R / members_left)), then subtract it from remaining_R.

Each member's stream.

  1. Connection. The member's agent opens a capacity-test connection to the node's overlay address, inside the source-matched probe session the node opened to that agent (§STAMP probes and probe sessions). It sends the 72-byte request header as in §Capacity tests. Until Task 18b the reference agent (bwos-probe cap --team) connects to the node's echoing sink (cap-sink -echo) directly; that is a localnet and test harness only (below).

  2. Test id (review R5). The member declares the test in its batch's cap_teams: [{"node":"<hex>","block":b,"nonce":"<32-byte hex>"}], sorted and unique by (node, block), omitted when empty. The header's test_id is:

    SHA-256("BWOS-v0/cap-team-test" || 0x00 || node_key || u64be(b) || nonce)
    

    The ack must carry it.

  3. Start and rate. On first seeing block b, it streams for CAP_TEST_SECS at rate_i kbps.

  4. Content. The stream is a keystream only the member can regenerate, for example ChaCha20 under a fresh private key.

  5. Echo. The node's sink writes every stream byte back on the same connection, in order, as it receives it. Once the member has half-closed and the sink has echoed every byte, the sink sends u16be(len) || canonical(CapTestAck). The ack's digest and bytes cover every stream byte received.

  6. Echo check. The member splits the echoed stream into blocks of CAP_ECHO_BLOCK_BYTES (4_096) and checks each block with probability CAP_ECHO_CHECK_PPM (10_000), using its own OS randomness. It draws that randomness after the block was sent and never discloses it.

    • Why it's hard to fake. A node that corrupts or skips a fraction f of the blocks passes with probability (1 - CAP_ECHO_CHECK_PPM / 1e6)^(f * blocks), which is about 1% for f = 1e-4 over a 2.5 Gbps, 60 s share.
    • Why echo everything. The test then measures the lower of the node's downlink and uplink.
  7. Member sample. The member records capteam:<node>:<b>, with these fields:

    • rtt_us: elapsed time from the first stream byte sent to the last echoed byte received;
    • throughput_mbps: floor(echoed_bytes * 8 / rtt_us), informational;
    • loss_ppm: the share of checked blocks that failed;
    • jitter_us: 0.

    The ack goes in cap_acks. A connect failure, timeout or missing ack is a failed sample. A short echo is a failed sample (Task 4Dc fix round 1): the member records an echo-OK sample only when the echoed bytes equal the bytes it sent and ack.bytes, so scoring reads m_i from the ack.

Echo OK. A member's result is echo-OK when all of these hold:

Combining (red team RT3; round 4 RR1, RT1, RT5; round 5 A, D; Task 4Dc fix round 1, red team F1, F2, F9, OQ-91, OQ-93). Combining is pure and deterministic: probe::team_capacity_mbps(declared, members, S, background_mbps, background_in_team). members lists each member with rate_i > 0 that revealed a batch for the window (k' of them), with its hotkey, rate_i and result. S is the summed rates of the whole scheduled team, every member with rate_i > 0, revealed or not (CapAssignment::scheduled_kbps). All arithmetic is u128 in kbps, every division floors, 1e6 is the integer 1_000_000, and team is floored to whole Mbps once at the end.

m_i         = floor(8 * ack.bytes * 1000 / rtt_us) if echo-OK_i else 0     kbps, exact from the ack
                                                    (no sample in a revealed batch: 0)
c_i         = min(m_i, rate_i)                      capped share, so a member's ratio is at most 1
j           = the revealed member at index (k'-1)/2 when sorted ascending by c_i / rate_i,
              compared exactly as c_a * rate_b vs c_b * rate_a, ties to the lower hotkey   (the lower median)
S           = Σ rate_i over the whole scheduled team (never less than the revealed members' sum)
pushed      = min(declared, S)                      the share of it the node is judged on
median      = c_j * S / rate_j                      the k·lower_median term, generalised to unequal rates
measured    = min(Σ c_i, median)                    Σ over the revealed members, capped shares
bg          = min(background, measured * CAP_BACKGROUND_MAX_PPM / (1e6 - CAP_BACKGROUND_MAX_PPM))
team        = min(declared, measured + bg)   if background_in_team
            = min(declared, measured)        otherwise (before Task 12b, §Background traffic)
            then floored to Mbps
ratio_t     = min(1e6, (c_j * S + bg * rate_j) * 1e6 / (rate_j * pushed))   only if k' >= CAP_TEAM_MIN_VALIDATORS
a test with k' < 2, or pushed = 0, isn't a sample

Background traffic (controller ruling 2, red team RT4; round 4 RR2, RT3, RT4; round 5 B, C, F1, F3). As in FlashFlow, traffic the node keeps carrying during a test is credited, up to a cap, so a busy honest node isn't haircut. background_mbps(n) is never the node's own report. It comes from co-signed hop receipts with peers of other UIDs on production-table paths. Gated until PB-38 (Task 12b fix round 2, red team H1). probe::BACKGROUND_IN_TEAM is false: background is each stream's window average and lifts only ratio_t, never team, exactly as before Task 12b, and validator::tempo_team_stage enforces it whatever TempoInput holds (sim gate_holds_in_run_tempo, run through run_tempo in team mode). The receipt-run rule below is built and tested on the gate-on path (tempo_team_stage_gated, sims only) but scores nothing. Background in team will come only from canary or digest-verified carriage (PB-38, ADR-0051, ADR-0053), because counted hop bytes, even transit only, can be junk a node or its sybils push through assigned hops (red team H2, sim sybil_uid_background); only PB-38's change may turn the gate on.

Residuals.

Team mode MUST NOT run with endpoint dedupe off (Task 4Dc fix round 5, red team A1, review B4, T124). run_tempo uses team-mode capacity tests only when validator::tempo_endpoint_trigger is met; otherwise it runs the single-agent tests exactly as before, even with block_hashes supplied (test: the [T24] run_tempo test's team-mode block). With dedupe off nothing pools co-located keys: N keys on one box answer N sequential tests (OQ-18), and with one key per flood UID at a fleet's node size no per-UID schedule weight keeps a multi-node fleet above about 0.5 of its due tests at 4 flood UIDs or 0.3 at 8 (§"Per-UID fairness").

Team mode MUST NOT be enabled before PB-38 has landed and turned probe::BACKGROUND_IN_TEAM on, and only PB-38 may turn it on (Task 12b fix round 2, red team H1, OQ-106). Task 19b, which turns team mode on by filling block_hashes, has this as a precondition.

Team mode MUST NOT be enabled before Task 18b (red team F4, T105). Nothing committed ties a team stream to the endpoint validators see on the node's sessions: a sink box holding many node keys could answer the tests of nodes whose endpoints are in many /24s (about 16x for 20 keys behind one 10 Gbps box). The validator loop (Task 19b) MUST NOT fill TempoInput.block_hashes, which turns team mode on, until Task 18b runs team streams inside source-matched probe sessions with a per-validator member agent. cap --team --target is localnet and test only.

Code (Task 4Dc).

Cohort capacity tests ([T28], OQ-18, OQ-20)

A validator may test every live node of one dedupe group at the same time, as one cohort:

  1. Membership is validator-chosen. The cohort is the set of live nodes with verified capacity > 0 that the validator puts in one dedupe group (from the endpoint addresses it probes itself, never from operator self-reports), 2..=MAX_CAP_COHORT_NODES (16) nodes; zero-capacity keys bound into the /24 (burners) are left out. A larger group is tested in disjoint chunks of at most 16 nodes, each chunk its own cohort; scoring credits the best single chunk, never the sum of chunks (chunks tested at different times could be served one after another by one pipe). The cohort id is SHA-256("BWOS-v0/cap-cohort" || 0x00 || u32be(n) || node_1 || ... || node_n) over the 32-byte keys in ascending order (probe::cap_cohort_id). The group is due for a cohort test when capacity_test_due(prev_block_hash, cohort_id, CAP_TEST_PERIOD) holds (team cohorts are scheduled in scoring by probe::team_schedule, item 6; cohort streaming in bwos-probe is not implemented yet, roadmap M4-08; the library verifies and scores declared cohorts); its members are then not tested individually in that block, and a cohort test uses one of the validator's CAP_TEST_MAX test slots.

  2. Simultaneous streams. One probe agent opens all member streams at the same time. Member i's request header carries test_id_i = SHA-256("BWOS-v0/cap-cohort-test" || 0x00 || cohort_id || nonce || node_i) (probe::cap_cohort_test_id) with a fresh 32-byte validator nonce, so each member's node-signed cap-test-ack is bound to this cohort test. The agent records each member's cap:<node> sample as usual and span_us, the time from the first stream start to the last ack (or timeout).

  3. Validator-signed declaration. The batch lists the cohort in cap_cohorts (omitted when empty, so legacy encodings and signatures are unchanged): {"nonce":"<32-byte hex>","nodes":["<pubkey hex>",...],"span_us":<int>}, sorted and unique by (nodes, block) (block absent sorts first, so legacy batches order as before); a node is in at most one cohort per batch and block, and every member has a cap:<node> sample, or capteam:<node>:<b> in a team cohort (ProbeResultBatch::validate). A cap_teams entry may not repeat a team cohort member's (node, block), since that test's id comes from the cohort. Sample keys starting with capcohort: are rejected.

  4. Aggregate. ProbeResultBatch::to_metric_map adds capcohort:<node_1>,...,<node_n>:throughput_mbps = Σ bytes * 8 / span_us (floor; the key is the ascending membership joined with ,, probe::cap_cohort_key), summing the ack bytes of members whose sample completed within the span (0 < rtt_us <= span_us) and whose valid ack is signed by that member for this batch's validator and window with the member's cohort test_id; other members add 0. Total bytes over the common span, not the sum of per-member rates: one pipe serving the streams one after another (per-member rates C, C/2, C/3, C/4, sum 2.08 C) measures as C. The key is aggregated like every other key (stake-weighted lower median, MIN_VALIDATORS, stake quorum), so validators must have tested the same membership.

  5. History. TempoOutcome::cap_cohort_tests (membership key -> Mbps, probe::cap_cohort_members_key) is appended to TempoInput::cap_cohort_history; a cohort's verified aggregate is the lower median of its samples over the last CAP_COHORT_FRESH_ENTRIES (8) entries (validator::rolling_cohort_capacities; entries are windows with at least one aggregated cohort test). A cohort not re-tested in that span has no credit. In team mode a cohort needs at least CAP_COHORT_TEAM_MIN_SAMPLES (2) samples in the span (validator::rolling_cohort_capacities_min; Task 4Dc fix round 5, red team B1(b), T125): a team cohort sample is one team's result, and with one colluding seat on a cohort's only sample its lower median was the seat's figure, so the group, the max over its cohorts, took the best of many such lotteries (×1.5-2.0 at 5% of stake). The lower median of two is their minimum, so one seat never raises a group; with fewer than two the group falls back to its largest single. Residual: both samples with the colluder seated still give 1.5x in some runs (sim colocated_cohort_lottery, T125, OQ-99). A cohort falls due about once per 2.5 windows (CAP_TEST_PERIOD, 900 blocks), so by due-time randomness alone, with no attacker, about 4% of cohort-windows have no test in the last 8 (9 of 240 in the red team's round-3 re-check; 0.045 by due times in cgnat_cohort_turns), and in team mode, needing two, about 15% (0.151); while it has lapsed, a 15-operator carrier group is paid as its largest member, about 1/15 each, which costs such groups about 3.5% of their capacity term on average before team mode and about 14% in it (red team L1, T116(a), T125; OQ-95 proposes keeping credit until the cohort's next due test). A consolidated group (three pipes shrunk to one) falls to its new measurement once re-tests make up at least half of the samples in that span (two re-tests after a single earlier sample, up to four) instead of keeping the old aggregate for CAP_PENALTY_WINDOWS (red-team round 2). History is keyed by exact membership, and scoring applies any cohort whose members are all current positive-capacity members of the group (capacity-bonds.md), so a burner or a new member joining the /24 does not void an earlier result (red-team Pass-7 cohort_burner_griefer: 1.256x before, 1.000x after).

  6. Team cohorts (ADR-0039, Phase B Task 4Dc). Under team tests, a cohort is tested by one team. Design §4.2 covers this case: nodes sharing a /24 (including home users behind one CGNAT range, seen by validators as one public /24) or an IPv6 /32 get a simultaneous cohort test and share the measured aggregate.

    • Membership. It is fixed by the dedupe groups of the previous window's aggregated endpoints (§Probed endpoints). These are carried in TempoOutcome.endpoints and passed back as TempoInput.prev_endpoints, so membership is known before the window starts and is the same for every validator. Members are the group's nodes with verified capacity > 0 in the previous window's outcome. Operator reports are never used.

    • Groups above 16. The members are sorted by node key and cut into consecutive chunks of MAX_CAP_COHORT_NODES (16). A final chunk of one node isn't a cohort; that node is tested individually. Each chunk has its own cohort_id and is due on its own. Scoring credits the best chunk, as in item 1.

    • The team. When the cohort is due at block b, its team is drawn as in §Team capacity tests, as a tier-1 test, with the subject being the cohort_id. It takes each member validator's slot from b to b+4. Members of a cohort due at b aren't individually due at b.

    • Per-node rates, under one budget per member (round 4 RR4). A member's budget covers the whole cohort test: Σ_n rate_{i,n} <= budget_i.

      • Order. Nodes are taken in ascending order of their own slot_key = SHA-256("BWOS-v0/cap-slot" || 0x00 || block_hash(b-1) || node_key), so the node served last changes from test to test and can't be chosen.
      • Split. For each node n in turn, the team splits R(n) by the same water-filling over each member's remaining budget, budget_i − Σ rate_{i,m} over the nodes m before n, with members in ascending order of remaining budget, then hotkey. That gives rate_{i,n}.
      • Zero rates. A member with rate_{i,n} = 0 is dropped from node n's stream, as in a node test.
      • What each node is judged on. Each node's S_n = Σ_i rate_{i,n} over the scheduled members and pushed_n = min(declared_n, S_n). A node late in the order whose share the budgets ran out on (S_n < R(n)) is tested at less and judged against that, so it is never haircut; and its record counts in its ratio only, never in its measured (Task 4Dc fix round 1, red team F8), so a budget-starved test doesn't drag its credit down.
    • Streams. Each team member's agent opens all member streams at once, one inside each member node's source-matched probe session to that agent. Each stream runs at rate_{i,n} for CAP_TEST_SECS, with the member test_id_i of item 2 and the echo and echo check of §Team capacity tests. The batch's cap_cohorts entry gains block (b), omitted for legacy entries.

    • Each node's own result. Each node records capteam:<node>:<b> at each member validator. These samples also count as each node's individual team test: they are combined per node by team_capacity_mbps with that node's rates and background_mbps(n), under the same background rule as node tests, and enter its rolling history as (team, pushed_n, ratio_t).

    • The cohort's aggregate. Each member validator v declares the cohort in cap_cohorts. Its aggregate is computed as in item 4, but counting only intact echoed bytes:

      agg_v = Σ_n echoed_bytes(v, n) * 8 / span_v
      

      The cohort's result is team_capacity_mbps over the members v that revealed, with rate_v = Σ_n rate_{v,n}, S = Σ_n S_n over the whole scheduled team, declared = Σ_n declared_n and background = Σ_n background_mbps(n), in kbps with capped c_v as for a node. It isn't the stake-weighted median. It enters cap_cohort_history as in item 5.

    • Credit. The validator credits this measured aggregate through capacity-bonds.md §Endpoint dedupe, unchanged.

Until M4-08, bwos-probe emits batches with an empty cap_cohorts (no cohort streaming yet); scoring schedules team cohorts (item 6) and verifies and scores the cohorts validators declare.

How scoring uses it: capacity-bonds.md §Endpoint dedupe. Residuals: simultaneity and span_us are attested by the validator (as every throughput is); a group's credit is bounded by what one probe agent can push at once, and a group above 16 nodes by its best chunk; a node with real capacity placed in an honest group's /24 is pooled with it (co-location griefing, as in [T24]), but cannot reset the group's cohort credit. Two more, recorded under OQ-90 and T107 (red team F8, not redesigned in Task 4Dc): a group's credit is at most one team's budgets (10 Gbps at k = 4) and its best 16-node chunk, so a CGNAT /24 of 160 honest home nodes is credited as about 16 of them (about 0.1x); and chunks are cut by sorted node key, which keys are free to grind, so a planted key can push one honest member into a chunk of one, which isn't a cohort.

Peer-load capacity tests (ADR-0056)

Status: specified and simulated, not scored. Red-team revision 1 (2026-11-08) is folded in below.

Peers make the load, and a few hidden validator slices that the node can't pick out measure it. So validator cost per test doesn't depend on the node's size.

Calls: secret times. Each window, a node with a declaration has a coordinator, a validator drawn by stake from E(b) with the seed SHA-256("BWOS-v0/cap-peer-coord" || 0x00 || window_seed || node_pubkey). The coordinator commits a 32-byte plan_secret in its window manifest before the window's first block, and reveals it after the window ends, as Task 14's canary plan secrets are.

Slices (peer_load::slice_plan):

R     = probe::test_rate_kbps(declared, verified_prev)          the ADR-0039 ramp, headroom included
k     = max(CAP_PEER_HIDDEN_SLICES, ceil(R / CAP_PEER_SLICE_KBPS))
slice = ceil(R / k)                                             kbps, at most CAP_PEER_SLICE_KBPS
m     = min(CAP_PEER_HIDDEN_SLICES, k)                          hidden slices; k - m peer slices

Sink blinding. N never learns a slice's sink.

The draw: committed, secret until the reveal.

Eligible endpoints. A node is eligible as a source, R_out, sink or hidden relay at b when all of these hold:

A 100 Gb/s call needs 660 endpoints: at 8 per UID, at least 83 UIDs (OQ-137).

If the pool runs dry, the call runs with the slices that could be drawn. call_outcome takes slices_run = m + drawn, so the sample is at most slices_run * slice and a short pool never makes a call short.

Running a call.

Hidden slices. A hidden slice runs validator source agent → H_in → N → R_out → H_out → validator sink agent, at the slice rate.

Measurement and path proof.

The call outcome (peer_load::call_outcome). With n revealed hidden results h_i, each capped at slice:

n < CAP_PEER_HIDDEN_MIN (3)    →  void: not a sample, not a short call
short       = slices_run × upper_median(h) < min(declared, slices_run × slice) × (1e6 − CAP_TOLERANCE_PPM) / 1e6
sample      = 0 if short, else min(declared, usable, slices_run × floor(Σ h / n))

Retests, the bond and verified capacity (peer_load::bond_applies, strikes_apply, verified_kbps; capacity-bonds.md §Peer-load calls):

Source duty (peer_load::source_duty_charged). A source that sends less than its slice relieves the node it is drawn for. Lazy sources fabric-wide inflate every over-declarer by dilution(λ) (1.42x at λ = 0.3; sim capacity_lazy_sources).

usable (peer_load::usable_kbps, neighbour_term_kbps). Over N's neighbours j in the committed snapshot:

a_j    = min(avail_mbps of link (N, j) if present, 2 × verified_prev_j + CAP_RAMP_FLOOR_MBPS) × 1000
usable = min(S, 2 × (S − max_j a_j)),   S = Σ_j a_j

Option E was dropped in revision 1: sink blinding closes the egress gap without reading any endpoint record. Option W (committed send digests with a dead zone q, about 1.06x at q = 5%) is research (OQ-168).

Bounds, and what each sim gates:

Capacity tests of nodes behind NAT (interim, PB-48 fix 9, OQ-283)

Interim. This holds until team tests (Task 18b, OQ-99) or relay tests give a node behind NAT a test of its own. It exists so a home or router node can earn capacity on testnet. It changes nothing for a node whose sink can be reached. bwos_core::cap_dial has the encodings.

When. A plan-mode agent run with --cap-dial-listen <ip:port> takes dials. When a planned test is due (same start_block, stagger and group rules as a pushed test, §Live loop), the agent first tries to push to the node's sink, giving the connect CAP_PUSH_CONNECT_SECS (3 s). It asks for a dial only if that connect fails. Both happen in the one group test, so a node never has a pushed and a dialed test at the same time.

One start (PB-48 final verification F3). Once any member of a group test has asked for a dial, every stream of the group, pushed or dialed, starts at one instant: CAP_DIAL_HOLD_SECS (CAP_PUSH_CONNECT_SECS + CAP_DIAL_WAIT_SECS + 2 = 20 s) after the group started, past every push connect and the dial deadline. Each connection gets its request header as soon as it is made and is then held until that instant; one not made by then fails. So a node can't choose when its test runs by when it dials, and one back end can't serve a UID's nodes in staggered dial batches, each seeing the whole pipe.

The request. The agent sends one UDP datagram over the node's up probe session:

Only the session's WireGuard peer can send this, so the request carries the session's authentication. Before sending, the agent registers the token for the session's current source IP, valid for CAP_DIAL_WAIT_SECS (15 s).

The node. A node that holds its node key (--node-key: a no-root node, or a kernel node's bwos-node probe-session) takes the request from its probe peer and dials TCP:

It then sends CAP_DIAL_MAGIC ("BWD1") || token and serves the echoed test on that connection as cap-sink -echo does:

The node's limits:

Each dial's outcome is a CapDial log line. NodeStats counts dial_requested and dial_refused.

The agent's checks. A dialed connection counts only if all of these hold:

The token is then spent. Anything else is closed unanswered.

Admission (PB-48 final verification F1). On accept, before any read, the listener takes a connection only from a source IP holding an unexpired token that no connection has been taken on yet, and marks that token taken: one connection per token. A connection from any other IP is closed at once and takes no slot, so a flood from IPs with no token, however many, can't keep a due node's dial out. A connection whose hello then fails (garbage, late, or naming another token) gives its grant back, so the node can dial again on its token (release round, T324). It reads at most CAP_DIAL_MAX_HANDSHAKES (64) hellos at once, which only token holders reach. It keeps no state per IP beyond the tokens, which expire with their tests. ProbeServeStats counts dials_taken and dials_refused.

The test. The stream, the private byte checks, the early abort and the ack checks are the pushed test's (red team fix 3), so the test measures min(up, down). Its ceiling is min(declared, CAP_DIALED_CEILING_MBPS), and it is paced at its share of the agent's budget under that ceiling (§Live loop, "Rates"). The CapTarget is the connection's source IP with dialed: true and its paced_mbps.

Credit. Under PB-48's capacity rules, a dialed cap: sample needs its ack bytes, its target and CAP_MIN_REPORTERS like any other. Its throughput and pace are then capped at CAP_DIALED_CEILING_MBPS (100, proposed), and it pools by its target's /24 (/48) within its UID like a pushed one (F3). The node's credit is min(up, down, ceiling). Data-plane and uptime rules are unchanged.

Residuals.

Secure sampling (ADR-0038)

Status: specified and spiked, not scored (hop_digest::HOP_DIGESTS_SCORED is false). Implemented in crates/bwos-core/src/hop_digest.rs; vectors in testdata/vectors/hop_digest.json. How hops commit digests and how loss is attributed is in receipts.md §Hop digests.

The ingress and egress of a flow measure loss and delay on its real traffic by sampling packets that relays can't pick out. No byte is added to the packet.

Invariant bytes (hop_digest::invariant_bytes(packet) -> Option<Vec<u8>>). For a packet of at least 40 bytes whose version nibble is 6: the 40-byte IPv6 header with the traffic class (the low 4 bits of byte 0 and the high 4 bits of byte 1) and the hop limit (byte 7) set to zero, followed by the first 32 bytes of payload, or all of it if shorter. Anything else gives None and is never sampled or recorded. These are the bytes no relay changes: relays decrement the hop limit, and ECN may rewrite the traffic class.

Hop digest. hop_digest(inv) = SHA-256("BWOS-v0/hop-digest" || 0x00 || inv)[0..8]. Any hop can compute it. Two packets share a digest only when their invariant bytes match, or by a 2^-64 collision.

Sampling decision (hop_digest::sampled(key, invariant_bytes, rate_ppm) -> bool):

tag     = HMAC-SHA-256(key, "BWOS-v0/packet-sample" || 0x00 || hop_digest(inv))
sampled = u32be(tag[0..4]) < rate_ppm * 4294          (rate_ppm capped at 1_000_000)

hop_digest::digest_sampled(key, digest, rate_ppm) is the same test from the digest alone. The HMAC runs over the digest of the invariant bytes rather than the raw bytes, so anyone holding a revealed key and a signed list of digests can check which digests were sampled. A relay without the key sees a uniform 32-bit value either way. rate_ppm * 4294 is floor(2^32 / 10^6), so the realised rate is rate_ppm less 0.0225% of itself (10_000 ppm samples at 9_997.75 ppm). The default rate is SAMPLE_RATE_PPM (10_000, 1%).

Epochs. A window's time is cut into epochs of SAMPLE_EPOCH_MS (1_000 ms), numbered 1 ..= SAMPLE_EPOCHS_PER_WINDOW (8_192, enough for a window running about 1.9 times its nominal 72 minutes). Epoch 0 is the anchor and is never sampled. A node counts epochs on its own monotonic clock from the start of the window as it learned it, so nodes' epochs are not aligned: a packet can sit in epoch e at one hop and e + 1 at the next, and clock skew moves the boundaries further. SAMPLE_REVEAL_DELAY_MS only gives a late hop time to publish; it doesn't align anything. Until validators can match epochs across nodes deterministically (OQ-63, T74), attribution has to be run over a span of epochs rather than one, and nothing here is scored.

Keys (a backwards hash chain, as in TESLA):

K_N = SHA-256("BWOS-v0/sample-key" || 0x00 || mask_key || u64be(window))      N = SAMPLE_EPOCHS_PER_WINDOW
K_e = SHA-256("BWOS-v0/sample-key-chain" || 0x00 || K_{e+1})                   e = N-1 .. 0

mask_key is the flow's 32-byte key shared by its ingress and egress (datapath.md, "Who holds mask_key"). The egress can therefore compute every key of the window in advance (red team RT5, T72); deriving the keys from a secret only the ingress holds is the planned fix. Packets of epoch e are sampled under K_e (hop_digest::sample_keys). mask_key is never revealed, and revealing K_e reveals K_0 .. K_{e-1} but no later key.

SampleKeyReveal (domain sample-key-reveal, signed by the ingress's node key):

{"v":0,"window":1234,"epoch":17,"key":"<64 lowercase hex>","ingress":"<ingress node pubkey hex>","sig":"<ed25519 hex>"}

What the ends do. The ingress records the digest of every packet it sends on the flow, and the egress of every packet it receives, each in its own HopDigestBatch. After the reveal, anyone can filter the ingress's batch with digest_sampled under K_e to get the epoch's sampled set: end-to-end loss is the fraction the egress's batch lacks, and per-link loss is attribute_loss (receipts.md). Delay sampling (timestamps on sampled digests at the ends) is follow-up work and is not in these objects.

Why relays can't favour sampled packets. While a packet crosses, its epoch key is secret, so whether it is sampled is a uniform coin to the relay, independent of anything it can see (sim sample_favourer). After the reveal it knows, but its batch is already signed and published, so it can't edit what it claimed to have received. It can still leave out digests of packets it dropped, without knowing the key; that moves the loss to the link behind it, never off the link (T67).

Measurement work (ADR-0055; proposed, gated off)

This section defines what counts as a validator's measurement work, the bytes scoring.md §Measurement work pays for. Only bytes that a node signs for count, so a validator that copies the revealed inputs and measures nothing has none (T175). Nothing here is scored until scoring::measurement_work::VALIDATOR_WORK_ENABLED turns on (ADR-0055 §8).

Results and commit-reveal

  1. During the window, each validator records probe samples (ProbeSample) and assembles a signed ProbeResultBatch (domain probe-result):

    {"v":0,"window":1234,"validator":"<validator pubkey hex>","samples":[{"key":"link:<a>:<b>","rtt_us":12000,"loss_ppm":0,"jitter_us":50,"throughput_mbps":1000}],"cap_acks":[<CapTestAck>,...],"hotkey_sig":"<sig hex>"}
    

    cap_acks (omitted when empty) holds the node-signed cap-test-acks backing the batch's cap:<node> samples, sorted and unique by (node, test_id), at most MAX_PROBE_SAMPLES. cap_cohorts (omitted when empty) lists the batch's simultaneous cohort capacity tests (§Cohort capacity tests). Two ADR-0039 fields are omitted when empty, so existing batches keep their bytes and signatures:

    • endpoints lists the public endpoint this validator saw for each node it probed (§Probed endpoints);
    • cap_teams declares the batch's team capacity tests with their nonces (§Team capacity tests).

    Two PB-45b fields are omitted when empty too (identity.md §Delivery over the probe session, OQ-200): consents, the node consents the validator held at the reveal cutoff, and forwarded, the operator objects nodes forwarded that it admitted at its read block, each as {"domain","node","object"}. Both are sorted and unique by canonical bytes (objects also by (domain, hash)), within MAX_REVEAL_CONSENTS, MAX_REVEAL_OBJECTS and together MAX_REVEAL_FORWARD_BYTES, and every entry must verify; validate refuses a batch otherwise. The window's consent and object sets are the unions of what passes the chain checks (a consent list as a whole: listed, per-UID quotas; an object list per UID: admitted, within the UID's quota, so one UID's bad entries drop only that UID's), and only those unions reach scoring, never a validator's own pending pool.

  2. At commit time, each validator commits hash("probe-result", batch) on chain.

  3. At reveal, the batch is served as canonical JSON at /v0/objects/probe-result/<sha256> from the validator's HTTP endpoint (ADR-0015). Validators drop batches that don't match their commitment, fail cryptographic verification (ProbeResultBatch::verify), or contain duplicate measurement keys within the same batch.

  4. Aggregation: for each measurement key, take the stake-weighted lower median across distinct revealed validators (at most one batch per batch.validator per window, and under ADR-0039 at most one per hotkey; scoring::stake_weighted_lower_median, validator::verify_and_aggregate_probes_weighted, [T21], OQ-15). Each value is weighted by its validator's stake (TempoInput::validator_stakes). Under ADR-0039 it is keyed by SS58 hotkey, and each batch's ed25519 key is mapped to its hotkey through TempoInput::attestation_keys (§Team capacity tests, "Validator identity"); a batch whose key maps to no hotkey isn't counted. Before Task 11a the map was keyed by the same ed25519 key as the probe commitments; entries are sorted by (value, stake) ascending and the result is the smallest value whose cumulative stake c satisfies 2 * c >= total stake (u128, no floats). Missing values are ignored; a key needs at least MIN_VALIDATORS (default 2) distinct validators with positive stake (when validator_stakes is non-empty, validators absent from it or with zero stake are dropped before the count) and a stake quorum: the reporting stake r must satisfy r * 1e6 >= total * PROBE_STAKE_QUORUM_PPM (default 333_333, about 1/3), where total is the stake of every validator in validator_stakes, revealed or not (the number of revealed validators when the map is empty). Under ADR-0039, once attestation_keys is non-empty, total is recorded_stake, the stake of hotkeys holding a valid attestation-key record. Below PROBE_PARTICIPATION_FLOOR_PPM of all stake, the window carries forward the previous window's metrics where no fresh aggregate reaches quorum, and charges no failures for them, for at most PROBE_CARRY_MAX_WINDOWS windows in any PROBE_CARRY_SPAN_WINDOWS (§Team capacity tests, "Quorum over recorded stake"). A few minimum-stake validators therefore cannot complete a median for a key no staked validator measured (e.g. a cap:<own node> result; red-team T21 round 2). An exact 50/50 stake split resolves to the lower value (lower median). With an empty stake map (or all-equal stakes) every validator weighs 1 and every aggregated value is the plain lower median (sort values, pick index (n-1)//2); keys reported by fewer than a third of the revealed validators are dropped. This count-based fallback is for the simulator and localnet only: on testnet (and later) validators MUST supply metagraph stakes in validator_stakes (roadmap M4-07), because with an empty map two minimum-stake sybils pass the count quorum whenever at most four honest validators reveal. A minimum-stake validator therefore cannot fail a link or capacity test by reporting rtt_us = 0 when higher-stake validators disagree; failing a measurement needs at least half of the reporting stake.

All validators score from the same aggregated set, so Yuma consensus does not clip honest validators for having a different vantage point.

Daemon inputs (M4-07). bwos-validator loads validator_stakes from --stakes-file / BWOS_VALIDATOR_STAKES_FILE. The file maps each validator's probe key, the ed25519 key in probe_commitments, to its metagraph stake. Under ADR-0039 the live loop (Task 19b) keys validator_stakes by SS58 hotkey, straight from the metagraph, and fills attestation_keys. The stakes file is then for sim and localnet only. A missing stakes file on testnet is logged as a warning, because the count-based quorum fallback is for sim/localnet only. node_diversity used to come from --diversity-file; that flag and BWOS_NODE_DIVERSITY_FILE are now refused at start-up (Task 4B fix round 1, red team F2). Under ADR-0039, node_diversity is built from the probed endpoints committed in validators' batches together with agreed outside evidence (§Probed endpoints, §Underlay, IXPs and shared-risk groups). Every validator then builds the same groups, which a per-validator file can't guarantee (Invariant 1). Phase B Task 4B implements this (§Probed endpoints, "Dedupe"; OQ-80). Trusted voucher issuers and vouchers come from --trusted-issuers / BWOS_TRUSTED_VOUCHER_ISSUERS and --vouchers-file; node consents from --consents-file, merged each tempo with the retained <state-dir>/consents-state.json (highest started seq per node, revocations kept past their not_after; identity.md §Sequence, revocation and moves). Every input file is size-capped and strictly parsed. A malformed input stops the validator from starting; it never runs with an empty default.

Live loop (Task 19b)

bwos-validator live --network local|test --netuid <n> --hotkey <file> --attestation-key <file> --state-dir <dir> (crates/bwos-validator/src/live.rs) is the validator's per-window loop. It serves its object store (--listen, receipts open to the window's live nodes), fetches what it lacks from --peer-validators, and takes no demo input and no stakes, diversity or input file (F1, M4-08): every scoring input below comes from the chain and from objects the chain's commitments name. On test, a read with no staked hotkey is an error, never the one-vote fallback (inputs::require_stake_source, T21). Any network but local and test is refused before anything else, and the connection is genesis-checked (mainnet refused).

Schedule of window w (S = w * 360; parameters in scoring.md §Parameters):

Block What happens
S - 1, read A(w) Metagraph and stakes (uid_hotkeys; validator_stakes keyed by SS58 hotkey, R2). From each manifest that covers w (identity.md "Read block", "Validity"): the binding, the capacity declaration and the attestation-key record with its commitment block (committed before S, so window_attestation_keys keeps it, RT2). Taken when the block is final.
S Canary seed stakes: permit holders' stake at the seed block (canary_seed_stakes, §Stake shares).
S + SOLVER_COMMIT_DEADLINE_BLOCKS, read D(w) Route tables of manifests covering w, for window current_snapshot.window + 1, each signed (sr25519) by its committing hotkey. commit_block is the commitment's block (OQ-129, OQ-107 (5) and (7)); a later commitment is no submission. prev_table is the table the UID committed in the previous D read, never the solver's word. The path-probe draw is taken here; a draw that fails is retried each pass until S + 36 + DRAW_GRACE_BLOCKS (19b verification, M-a).
S + OWN_COMMIT_OFFSET_BLOCKS This validator commits its manifest for w: its attest record (renewed with four windows to spare), its probe batch for w from --probe-results and, with --agent-results, its probe agent's results for w merged in (PB-48, "The probe agent's plan and results" below; signed by the attestation key: one key per hotkey, OQ-55 (b)) and its outside evidence for w from --evidence. With --probe-pool, the batch's consents and forwarded are this validator's reveal of what nodes forwarded over its probe sessions (PB-45b, identity.md §Delivery over the probe session), and the batch is signed again with the attestation key, or is a batch of the reveal alone when --probe-results has none. The record and the manifest are served at once; the batch and the evidence are withheld (commit).
S + 359, read B(w) = A(w + 1) Probe batches and outside evidence named by manifests made for w (manifest.window == w): each batch for window w and signed by a key the committing hotkey's record maps to it; each evidence object the manifest's own (ev.validator, ev.window, hash, verify; RT5, M2). probe_commitments is keyed by SS58 hotkey (the identity join).
S + 360 Withheld objects go into the store (reveal), first in each pass of the loop, before any chain read, fetch or commit (19b verification, M-b).
S + 360 + REVEAL_DELAY_BLOCKS Score w, publish, set weights. A probe batch for w named by a manifest admitted at B(w), from a validator-permit holder at A(w), that isn't here yet is asked for again every 2.5 s (PROBE_BATCH_REASK), at most PROBE_BATCH_REASKS (4) times (M-b; integration fix round, M2). Catch-up scoring and re-asks stop for a path-probe draw that is due (L1). The window's revealed consents (from permit holders' batches, accepted one at a time, H1) join its node_consents, which live takes from nothing else: no consent in its own store counts (M1), and the operator objects its revealed union admits against A(w) go into the store before A(w)'s manifests are read, so they count for w itself (PB-45b, OQ-261, cold-start fix). Windows are scored in order from the one after the last scored (fix rounds 1-2, F3). Scoring is all or nothing: the loop state is restored if a window fails, which is retried during the next window and given up only once that has passed. A restarted validator catches up the windows it missed, at most READ_LOOKBACK_WINDOWS back, rebuilding the carried state without submitting or publishing for them: only the last finished window's weights are submitted. Catch-up yields to the window's own commit at S + OWN_COMMIT_OFFSET_BLOCKS.

A read is the chain state at that block, by its hash: the metagraph and the commitment records are read at the same block hash and refused if they differ (OQ-222), so every honest validator reads the same commitments (Invariant 1). Each record carries the manifest hash and, from INLINE_BINDINGS_ACTIVE_FROM_BLOCK (0 on local and test; a mainnet block is an ADR-0049 activation, OQ-201 (a)), the inline binding of ADR-0058, which fills TempoInput::inline_bindings and takes precedence over the UID's manifest binding. A manifest is admitted only if it is canonical, verifies (CommitmentManifest::verify, the hotkey's sr25519 signature), hashes to exactly the commitment, and is the committing registered hotkey's for the netuid (S4, D16); anything else is dropped before scoring. An SS58 binding's hotkey_sig is checked with verify_sr25519_hotkey_sig (OQ-131 (b)). Objects are pulled (OQ-220, OQ-223): from this store, then at once from the peers and the committing hotkey's on-chain axon (no URL in the manifest), each by the hash the chain authenticates, within MAX_OBJECT_BYTES and the fetch timeouts. On test an axon must be a global unicast IP; a pull base is a bare http://<ip>:<port> (F5). A pulled copy is kept and served. Fetches are batched (fix rounds 1-3): a read fetches every manifest at once, and scoring every unfetched manifest of A(w), B(w) and D(w) and then every object they name, and each waits once for the whole batch within READ_FETCH_BUDGET or SCORE_FETCH_BUDGET. Requests are grouped by object (C1): one job per (domain, hash) races the peers and the axon of every hotkey whose commitment names that hash, returning on the first verified copy, so a hotkey that copies another's hash with a dead axon can't stop the owner's axon being asked. Failures are remembered per (domain, hash, source) for the window, except for slot objects (probe batches, outside evidence), which may be revealed late. Jobs are shared out by hotkey (H1): they start one hotkey at a time in turn, each hotkey holding at most MAX_FETCHES_PER_HOTKEY jobs and all of them at most MAX_IN_FLIGHT_FETCHES; a job that can't start yet waits in its queue until the budget ends, never dropped. One hotkey's tarpit costs only that hotkey's share, and one blackholed peer only its own timeout. A manifest not fetched by a read is kept and re-read until the scoring deadline. Each pass of the loop reveals first and commits next, so no fetch delays either (M-b). Route tables are fetched when D(w) is read and path probes are drawn (S + 36); the tables fetched then are recorded in D(w) (drawn_tables) and are the only ones graded (fix round 3, H2). A table withheld at the draw, so none of its paths was probed, and served later counts as no submission. A validator that missed the draw (down from S + 36 to S + 36 + DRAW_GRACE_BLOCKS, or its plan couldn't be written: drawn_tables is None, where a draw that fetched no table is an empty set) grades no table and holds Mechanism 1 back for the window instead of giving the first party what the winner earned (M-a, OQ-224 (a)); the full fix is quorum-named drawn sets (OQ-224 (b)). There are no uploads from outside: an object nobody serves counts as absent (OQ-211; the full fix is OQ-211 (a), quorum-fetched objects). A validator serves its own store as its axon (--axon), so its manifest and revealed probe batches reach every validator, not only its --peer-validators (F4). Which window's objects count from each read (OQ-132): bindings, declarations and records by covers; batches and evidence only from manifests made for the scored window, and only objects of that window.

Filled from the chain and the carried outcome.

The probe agent's plan and results (PB-48). With --agent-results <dir>, live and its validator's probe agent (bwos-probe serve --plan <state-dir>/probe-plan.json --results-out <dir>) exchange two files. Each is written whole and renamed into place, read once to at most MAX_PLAN_FILE_BYTES (16 MiB), and strictly parsed (bwos_core::probe_plan).

The plan (ProbePlan) is {"v", "window", "block", "now_block", "rank", "validator", "cap_test_period", "nodes", "cap_tests"}. live keeps it written every pass once A(w) is read, waking every block while it has an agent.

The agent's work.

The results (AgentResults) are <dir>/<w>.json, rewritten at most every 5 s while samples come in and at the window's end, and pruned 16 windows back: {"v", "window", "samples", "cap_acks", "cap_targets", "cap_cohorts"} (cap_cohorts omitted when empty).

What live takes. At its own commit, live reads <dir>/<w>.json, refuses it unless it validates and is for w, and merges it into the batch (probe_plan::merge_agent_results).

PB-48's capacity rules. They turn on in the first window where any revealed batch carries a CapTarget (validator::cap_rules_on), as every plan-mode agent's batch does.

Not filled, by design. block_hashes stays empty, so team mode stays off: Task 18b, the endpoint trigger and OQ-99 are its preconditions (T105, T124, T125). cap_team_history stays empty with it. ADR-0055 measurement work, ADR-0056 peer-load scoring and ADR-0057 geo pricing stay behind their gates. failure_challenges, hop_digest_batches, sample_key_reveals, verdict_history and known_failed_links wait for Task 16c, which isn't on main.

Weights. Each mechanism is judged on its own (validator::mechanism_refusal, validate_weights): a mechanism with unassigned unearned weight is held back, so with no first-party UID Mechanism 1 sets nothing, and Mechanism 0 never goes to the first-party UID (Task 15). The first party is pinned by its SS58 hotkey (--first-party-hotkey), resolved to its UID at A(w) every window, never by UID number: a UID freed by a deregistration goes to whoever registers next (fix round 3, T241). It must be registered and its coldkey must not be the subnet owner's (T129). Weights are committed timelocked when the subnet has commit-reveal on, and set directly otherwise; a direct set that errors is read back before any retry. The loop warns when a mechanism's last weights would age past last + activity_cutoff before the next window (refresh_deadline_block, R17).

Publishing. After scoring w the loop writes the next window's validator set, peer directory (window w + 1, block S(w + 1), entries from next_snapshot), snapshot and table into <state-dir>/signed-inputs/, with chain-block, demand-mbps-per-pair, demand.json and solver-hotkey; the directory carries the window's demand_mbps_per_pair (OQ-221) (the incumbent UID's hotkey on chain, OQ-126), and Task 11b's Publisher attests them through publish_attested. Before an input is written, (domain, latest_key, attestation hash) is written and synced to <state-dir>/attest-guard.json, and a different object for a recorded key is refused, across restarts too (T60). Every 30 s the loop re-merges peers' attestations and relays the incumbent's latest table-update (named in its current manifest, pulled by hash from its axon if no store has it) for Publisher::check_update. Publication objects whose window or set ended more than VALIDATOR_SET_LIFETIME_BLOCKS ago are pruned (ObjectStore::prune_publications, OQ-67), and manifests and the objects they name once no retained read has named them for two windows.

Uptime

A draining node (ADR-0048) is not exempt. Probes, canary flows and capacity tests scheduled during a drain are scored exactly as for any other node. A sample the node refuses because it is drained is a failure. Until OQ-100 is settled, vantage probes during a node's announced drain (its own signed node-action record) also count as failures. That stops a node draining between its canary and capacity-test slots while keeping uptime.

A_u (ppm) = probes passed / probes scheduled for the fleet's nodes in the window, capacity-weighted: A_u = Σ_n Σ_s cap(n) * w_s * pass_s / Σ_n Σ_s cap(n) * w_s over the fleet's nodes n with scoring capacity cap(n) > 0 and their samples s of evidence weight w_s (below; u128 sums, floor, probe::compute_uptime_weighted_ppm). A node with no scoring capacity adds no samples. Without capacity weighting a 100 Mbps fleet-mate dominated a 10 Gbps node's uptime: three 100 Mbps attacker UIDs failing links with it cost the fleet 0.801x (attackers 1.053x; five: 0.696x / 1.142x), and a genuine fault on the small node cost a 10.1 Gbps fleet 0.194x versus 0.996x for unbonding it (red-team round 3; after: victims 0.977–0.990x in that 10 Gbps + 100 Mbps configuration, attackers at most 1.001x, genuine fault 0.953x; sim small_fleet_mate_framer over six fleet mixes: attackers at most 1.001x, victim UID at least 0.917x, worst for 10 Gbps + 1 Gbps vs 3×1 Gbps attackers, who themselves drop to 0.979x). A probe passes if it completes (rtt_us > 0) with loss at or below PROBE_LOSS_MAX_PPM (default 50_000).

Missing measurements are failures, never passes (validator::node_probe_evidence, [SCR-2]). Per node:

Residual ([T29]): beyond the one-sided budget both ends pay, but a griefing contest with equal absolute losses still raises the relative share of the side holding more of the network ((K - L)/(N - 2L) > K/N iff K > N/2), and a victim with a small (sparse-schedule) honest degree loses uptime ratio faster than meshed sybils. Measured without vantage results: dense (8 honest peers per victim) attackers 0.877–0.971x for K ≤ 8 and up to 1.05x at K = 20; sparse (honest degree 2, K = 7–10 single-node sybil UIDs, m = 1–5 victims, sim sparse_victim_framer) attackers 0.997–1.070x, victims about 0.32–0.37x (red-team pre-fix 1.064–1.348x, victims 0.025–0.036x). With vantage results for everyone (Pass-7): K ≤ MAX_NEUTRAL_FAIL_UIDS sybils cannot harm a clean victim at all; K ≤ 7 attackers at most 1.000x; K = 8–10 (sybils at or above half the UIDs) attackers at most 1.043x; for K ≥ 3 victims still fall to about 0.33–0.60x (sybils pay the same per failure, so it is a costly contest, not a free one). A victim with fewer than MIN_CLEAN_PEER_UIDS clean peer UIDs keeps the Pass-6 treatment. Closing the majority case needs a per-UID cost or link probes peers cannot tell from other traffic.

Q_n is the lower median of per-outgoing-link quality over counted outgoing links (attributable passes, and failures or dodges charged to the node), with 0 for failed or dodged links; with no outgoing link samples it is 1e6 only if the node produced passing evidence (capacity test or incoming link), else 0.

Outside evidence (ADR-0039)

This is evidence from outside the network that a node's ASN, upstreams and IXP presence are real: RPKI, ASPA, PeeringDB, RIS, RouteViews and RIPE Atlas. Validators snapshot it, sign it and commit it. Scoring never reads it live (Invariant 1).

Object. OutsideEvidence, under domain outside-evidence:

{"v":0,"window":1234,"validator":"<SS58 hotkey>","source":"rpki","fetched_block":443000,
 "records":[{"roa_origin":{"prefix":"203.0.113.0/24","asn":64500,"max_length":24}}],"sig":"<sr25519 hex>"}

Tag names are pinned. EvidenceSource serialises as one of these strings: rpki, aspa, peering_db, ris, route_views and atlas. EvidenceRecord is externally tagged with these snake_case names and fields:

Tag Fields Allowed sources
roa_origin prefix, asn, max_length (u8) rpki
aspa_providers customer, providers (sorted, deduped u32) aspa
ix_presence asn, ix_id peering_db
upstream asn, upstream ris, route_views
origin prefix, asn: a prefix seen announced in public BGP with that origin ASN ris, route_views
atlas_rtt probe, target, rtt_us atlas
Source Vendored file the founder fetches Records the importer writes
RPKI rpki-client -j JSON: {"roas":[{"asn":13335,"prefix":"192.0.2.0/24","maxLength":24,"ta":"apnic","expires":...}],...} roa_origin { prefix, asn, max_length: maxLength } per roas entry. ta and expires are dropped. asn 0 (AS0) is dropped
ASPA the aspas array of the same rpki-client -j JSON: [{"customer_asid":64500,"expires":...,"ta":"...","providers":[64501,64502]}] aspa_providers { customer, providers }
PeeringDB PeeringDB API netixlan response: {"data":[{"asn":64500,"ix_id":64,"operational":true,"status":"ok",...}]} ix_presence { asn, ix_id } per entry with status == "ok" and operational != false
RIS a JSON array of RIPEstat data-call responses, each told apart by data_call_name. asn-neighbours (one per ASN): {"data_call_name":"asn-neighbours","data":{"resource":"64500","neighbours":[{"asn":64501,"type":"left",...}]}}. network-info (one per endpoint): {"data_call_name":"network-info","data":{"prefix":"203.0.113.0/24","asns":["64500"]}} upstream { asn: resource, upstream } per asn-neighbours neighbour with type == "left", and origin { prefix, asn } per network-info ASN. ASNs may be written "64500" or "AS64500". Other data calls are refused
RouteViews a JSON array of {"prefix":"<cidr>","as_path":[u32,...]} converted from a RouteViews MRT RIB dump with bgpdump -m and jq, with paths containing an AS_SET left out. The conversion command is kept with the fixture Per path, after collapsing repeated ASNs (prepends): origin { prefix, asn: last ASN }, and upstream { asn: last, upstream: the one before it } when the path has two or more ASNs. A path in which an ASN reappears after a different ASN (AS-path poisoning or a loop) gives no record (red team RT3, evidence::path_records, sim as_path_poisoner)
RIPE Atlas a RIPE Atlas ping measurement results JSON array: [{"prb_id":1001,"dst_addr":"203.0.113.7","min":12.345,...}] atlas_rtt { probe: prb_id, target: dst_addr, rtt_us: round(min * 1000) }. A result without min, or with min < 0, is dropped. Floats appear only here, at import. Scoring sees integers

Import.

bwos-validator evidence-import --source <s> --in <file> --out <file> --hotkey <file> --window <w> --fetched-block <b> [--scope <file>]

Commit, reveal and retention (review R3).

Usable evidence. evidence::usable_evidence(&input.evidence, input.window) is the only way scoring reads outside evidence. It keeps an object only when all of these hold:

Then, for each (validator, source), only the object with the highest window is kept. Ties go to the lower canonical hash (wire::hash_hex), then to the lower canonical bytes with sig, so the choice is total. The result is ordered by (validator, source).

Agreed records (review R2). Validators are identified by SS58 hotkey (§Team capacity tests, "Validator identity"). A record counts in scoring only when enough validators agree on it, using the same quorum as probe keys (§Results and commit-reveal, step 4):

The pure function evidence::agreed_records(usable, validator_stakes /* keyed by SS58 hotkey */) -> Vec<EvidenceRecord> returns them sorted and deduped. Everything below reads only agreed records, so one validator can't invent or delete a record.

RIPE Atlas isn't scored. atlas_rtt records may be imported and committed, but no Phase B score reads them (ADR-0057 would read anchors' records as placement sites only once the terms are verified, §Network position), because Atlas's terms for commercial use aren't verified (ADR-0039 §3, OQ-39).

Probed endpoints (ADR-0039)

What each validator commits.

Aggregated endpoint. For each node, take the hotkeys with positive stake that report it at a public address, and group their addresses by dedupe prefix: the IPv4 /24 or the IPv6 /IPV6_DEDUPE_PREFIX_LEN (/32), exactly the prefixes scoring::diversity_groups and dedupe_diverse_node_capacities group by (§Endpoint dedupe in scoring.md).

Dedupe (Task 4B fix rounds 1 and 3, red team F2, findings 1 and 3). TempoInput.node_diversity, the input to scoring.md §Endpoint dedupe, comes only from committed endpoints:

Residuals.

Underlay, IXPs and shared-risk groups (ADR-0039)

This section is what the snapshot builder (validator::build_snapshot_and_demand) and the diversity term use. Every rule reads only agreed records and aggregated endpoints. Phase B Task 4B implements it.

Reserved ASNs. These are never an underlay ASN or an upstream:

Underlay ASN (red team RT2). A node's underlay ASN A is the origin seen in public BGP for the longest announced prefix covering its own committed endpoint, kept only if RPKI-valid for that origin:

  1. Candidates. Take every agreed origin { prefix, asn } (from RIS or RouteViews) with a non-reserved asn whose prefix contains the endpoint. The address family must match.
  2. RPKI-valid only. Keep only the origins that are RPKI-valid for their own prefix: some agreed roa_origin has the same asn, a prefix covering the announced prefix, and max_length >= its length.
  3. Longest prefix. Of those, keep the ones with the greatest prefix length.
  4. Pick or refuse. If exactly one ASN remains, that is A. Otherwise (none, or several valid origins) A is none. This is the conservative reading.

Amendment to Task 4A's text (Task 4C fix round 1, red team RT4), for Task 4B. Steps 2 and 3 were the other way round: longest prefix first, then RPKI-valid. In that order, anyone who announces an RPKI-invalid more-specific covering a node's endpoint leaves the node with no underlay ASN, because the invalid more-specific wins the longest-prefix step and is then thrown out. Filtering RPKI-valid origins first means an invalid announcement changes nothing.

A ROA alone is never the source. A prefix holder can sign ROAs for fresh ASNs that announce nothing, and a ROA-based rule would let it give every node a unique network (sim roa_origin_shopper, Task 4B).

A node with no underlay ASN gets no upstreams, no ixps and no srg, so diversity_ppm = 0.

Ambiguous underlay (Task 4B; Task 4C red team residual). Step 4 gives none when two or more RPKI-valid origins share the longest prefix. The node is then treated exactly like a node with no evidence:

Validators who reach evidence agreement (T61: a third of recorded stake, at least 1/6 of all stake) can cause this. They get a fake origin and a matching fake roa_origin agreed for a victim's prefix. Agreed origins enter a scope at most MAX_AGREED_SCOPE_ORIGINS (4) per endpoint, lowest ASN first, so they choose low fake ASNs. The victim then loses at most DIVERSITY_WEIGHT_PPM + IXP_PRESENCE_WEIGHT_PPM of C_n, and only on its carried bytes. The attackers gain nothing directly (threat-model T82, sim underlay_ambiguity_griefer). With the same power they can instead give their own nodes a fake underlay with two upstreams and an IXP. That earns at most the saturated 1e6 + DIVERSITY_WEIGHT_PPM + IXP_PRESENCE_WEIGHT_PPM (T90, sim agreed_underlay_forger).

Implementation (Task 4B).

Home and CGNAT nodes (controller ruling 6). These nodes take the underlay ASN only from their own committed endpoint, by the rule above. Behind NAT or CGNAT, that endpoint and so A belong to the carrier. Such a node inherits the carrier's upstreams and IXPs, though its own access link is a single point of failure (threat-model T47). With diversity on the usage term only and a low weight, the effect is bounded. This is by design (ruling 6, confirmed in Task 4B fix round 2): a CGNAT node's aggregated endpoint is an address of the carrier's prefix (§Probed endpoints, "Quorum and majority per prefix"), so its underlay ASN, upstreams, IXPs and as: SRG label are the carrier's, and it shares the carrier's SRG with every other node behind that carrier.

Upstreams. upstreams(n) is the sorted, deduped set of U such that:

An upstream is "left of A in a public BGP path", which includes peers seen at IXPs as well as transit providers. Both give paths that don't depend on one provider. ASPA doesn't change this set or the count (red team RT9):

IXPs. ixps(n) is the sorted, deduped set of ix_id over agreed ix_presence { asn: A, ix_id }.

Shared-risk groups. srg(n) is a sorted, deduped list of labels:

A network with two or more neighbours gets no up: label. A faked second upstream (T38) therefore also removes the up: label the node should carry. Labels are strings so later sources can add kinds without a wire change.

Unknown is not independent (red team RT9). A node with no srg has unknown risk, not independent risk. A resilient route needs known SRGs on both sides (routing.md §Validation, sim roaless_backup).

In the snapshot. routing::Node carries upstreams: Vec<u32>, ixps: Vec<u32> (PeeringDB ix_id) and srg: Vec<String>, each sorted, deduped and omitted when empty, so existing snapshots keep their bytes (routing.md §Topology snapshot). The diversity term is scoring.md §Coverage.

Not covered in Phase B.

Network position (ADR-0057, proposed; red-team revision 1)

Not produced yet. This section feeds the locational price G_n (scoring.md §Locational price G_n), which is off (GEO_PRICING_ENABLED = false). PB-44 implements it.

Sites and the region table. REGION_TABLE (scoring.md §Parameters) lists regions and sites. It is published on ADR-0049's scheduled-change path.

Separation rule. Every two sites a, b of different regions MUST be at least (R_a + R_b) * GEO_LIGHT_KM_PER_MS / 2000 km apart by chord, the straight line through the Earth (geo::site_separation_km). The publishing tool checks it, and a table that fails is not published.

Figures. In its ProbeResultBatch, each validator reports geo_rtt:<n>:<site>. Its value is the minimum RTT (µs) over the window of every timing node n answered from that site's agent. That covers:

Rules for the figure:

Peer links from the endpoint. An honest node accepts a peer link (WireGuard handshake source, BGP session) from node m only if the source matches m's committed endpoint by geo::source_matches. The data plane of a node therefore runs through its endpoint box, and a back box elsewhere carries nothing that is credited (T196, geo_endpoint_front, geo_v6_prefix_front). PB-44 adds this to datapath.md and to the no-root and kernel nodes' peer checks.

Placement (geo::place). A site is in range of n when its figure is at most its radius. Node n is placed in region r when GEO_PLACE_MIN_SITES in-range sites of r are independent.

What a node can and can't do.

Residuals.

Self-reported telemetry (ADR-0039)

Anything a node, its operator or its router reports about itself is for dashboards only and is never a scoring input:

None of these is a TempoInput field, an input to any bwos_core::scoring function, or a field of a validator-signed object that scoring reads. A validator MAY collect and display them. What validators observe themselves is different, and may be scored: