Transit Fabric releases

Taken from the Transit Fabric source repository at commit 91be12e (91be12ef98b775c7f045124e6b8c686c15439650, main), file docs/specs/receipts.md. Links into the private source tree are shown as plain text. Raw Markdown · Red-team pack

Spec: receipts

Status: draft · Implemented in crates/bwos-core/src/receipts.rs

Traffic classes

Class Meaning Default weight
paid Traffic from a paying customer or a partner session W_PAID = 1_000_000 ppm
free Free-tier client traffic W_FREE = 250_000 ppm
canary Validator probe traffic W_CANARY = 100_000 ppm

The ingress node sets the class from the session token. Only the ingress and the egress know it (ADR-0037). Overlay links carry plain inner IPv6 with no class on the packet (datapath.md), so a relay sees no class, and hop receipts carry one unsplit total per hop. The class split of a hop's bytes comes only from the ends: client receipts give paid, and [T16] attestation gives free (scoring.md §Byte crediting). Hop receipts earn no canary credit. Crediting canaries from the validators' own schedule and results is follow-up work (OQ-17). Taking the class off the packet doesn't make canaries indistinguishable: source addresses, timing and size still leak (see the residual list in datapath.md).

Hop receipt

Every RECEIPT_INTERVAL (30 s), each pair of adjacent hops on an active path co-signs cumulative counters for that window:

{
  "v": 0,
  "window": 1234,
  "seq": 17,
  "up": "<upstream node pubkey hex>",
  "down": "<downstream node pubkey hex>",
  "total": 128024679,
  "block": 1234560,
  "sig_up": "<ed25519 hex>",
  "sig_down": "<ed25519 hex>"
}

Hop receipts from counters (PB-13, Task 12b)

A node's hop receipt total is what it counted, not a fixed number of bytes per packet or request (the VPN data plane below bills its client deltas, 4,096 bytes per request, and nothing else uses that).

Receipt runs (Task 12b, round 5 B)

For each capacity test of a node due at block b (measurement.md §Team capacity tests), the node's receipt batch for the window MUST keep, per stream (window, up, down) it is an end of, every co-signed receipt from before to after, not just the latest: before is the stream's last receipt with block <= b, and after its first after before with block >= b + CAP_TEST_BLOCKS - 1 (receipts::test_bracket). A stream with no before yet keeps its receipts from its first; one with no after yet, to its latest. Validators read background from these runs (measurement.md §Background traffic); a run with a missing receipt shows a wide interval and adds nothing.

Overlay data plane: hop-by-hop co-signing ([INCENTIVE-HOP-01])

The VPN data plane (bwos_core::vpn: /ingress-tunnel, /ingress-stream, /ingress-bond-burst → /relay-tunnel, /relay-stream → /exit-tunnel, /exit-stream) MUST produce one co-signed receipt per adjacent hop of the path it forwards over, so a route [N0, N1, …, Nk] can pass the chain check below. A single end-to-end (ingress, exit) receipt matches no hop of a relayed route and earns nothing.

Chain check

For a sampled path A → B → C → D, validators check that the receipts (A,B), (B,C) and (C,D) agree on their unsplit total within tolerance. A hop whose total exceeds both neighbors' by more than the tolerance is zeroed for that window segment (it inflated). A missing middle receipt zeroes that path for the window. A path that visits any node twice is zeroed the same way, so one receipt is never counted for two hops (verify_path_chain).

Verified hop bytes are then credited to nodes once per path (scoring.md §Byte crediting): each hop's credit is split across the active paths through it by each path's weight bytes, its own paid and free takes, a path slower than Mechanism 1's regret comparator by more than ROUTE_REGRET_TOLERANCE_PPM earns nothing, and nodes on one path that share an operator UID or an endpoint dedupe group are credited for that path at most once ([T23]).

Hop digests (ADR-0038)

Status: specified and spiked, not scored (hop_digest::HOP_DIGESTS_SCORED is false). Nothing below changes a weight until ADR-0038's incentive-red-team pass sets it true. Implemented in crates/bwos-core/src/hop_digest.rs; sampling and keys are in measurement.md §Secure sampling.

Hop digests show which packets each hop received, so sampled packets (measurement.md §Secure sampling) can be followed along a path. They don't replace hop receipts: pay gating stays on the co-signed totals above.

What a hop records. For every packet it accepts for transit, a hop computes hop_digest(invariant_bytes(packet)) (8 bytes, measurement.md). It records the digest when it accepts the packet, before its send, whether or not the send succeeds. A packet it refuses (no route, hop limit, source check, not assigned) is not recorded. A no-root node records in Node::on_packet with --hop-digest. A kernel node needs an eBPF hook (OQ-65).

HopDigestBatch (domain hop-digest-batch, signed by the hop's node key):

{"v":0,"window":1234,"node":"<node pubkey hex>","epoch":17,"digests":["00a1b2c3d4e5f607","..."],"sig":"<ed25519 hex>"}

Loss attribution (hop_digest::attribute_loss(ingress, sampled, hops, egress), pure). ingress is the ingress's name, point 0 of the path. sampled is the epoch's sampled digests: the ingress's own batch digests that pass digest_sampled under the revealed K_e. hops is each hop's digest set in path order, and egress the egress's. For each sampled digest the last point holding it is found, hops first and the egress last:

The result maps each point (the ingress or a hop) to its count, with zero counts left out; a count is keyed by the point at the upstream end of the link that lost it. Example: hop B holds 100 sampled digests and the egress 90 of them, giving {B: 10}. A hop that lacks a digest a later point holds has an incomplete batch, and that loss isn't charged upstream.

A count against point i is evidence about the link from point i to the next point, not about point i alone. A next hop that drops a packet and leaves its digest out of its batch, or withholds its batch, makes the loss look like point i's, and without per-packet acknowledgements the two can't be told apart (ADR-0038 §4, T67, T71). Anything that pays or penalises on attribution MUST charge the link, or corroborate the hop with validator vantage probes as link blame already is (measurement.md §Probe vantage, T29).

Source verification. A path's sampled delivery counts only digests that both the source's own batch and the egress's batch hold (hop_digest::verified_delivered). The source is the node owning the packet's source address, or, for VPN traffic, the ingress a client receipt names. A sampled packet at a hop whose digest the claimed source never committed was injected with a forged source. When digests are scored, such traffic earns nothing on any hop (ADR-0038 §5, T68).

Evidence. A failure challenge (Task 16c) may cite HopDigestBatch and SampleKeyReveal hashes as spoof or link-loss evidence. It is confirmed only if every cited object verifies, the reveals chain to the ingress's anchor, every cited batch passed the order rule, and the objects themselves show the failure. A claim citing no digests is never confirmed. Credit is in the reporter's own mechanism, and none goes to a reporter whose UID owns a node on the reported path. Spoof credit is 0 while HOP_DIGESTS_SCORED is false (T69).

Client receipts

Clients sign delivery receipts for paid sessions:

{"v":0,"window":1234,"client":"<client pubkey hex>","ingress":"<node pubkey hex>","egress":"<node pubkey hex>","bytes":987654,"class":"realtime","sig":"<ed25519 hex>"}

bytes MUST be >= 1, and client, ingress, and egress MUST be pairwise-distinct valid non-weak Ed25519 public keys. Validators credit a hop's unsplit total as paid only as far as client receipts cover it within tolerance, and only when ingress and egress are bound to distinct live operator UIDs (uid_in != uid_out). The rest of the total, including intra-UID self-dealing bytes where uid_in == uid_out, is credited as free. Client receipt bytes are cumulative within the window: validators keep the maximum per (window, client, ingress, egress) across all batches and never sum repeated receipts.

class is the session's route class, a routing::Route.class value (ADR-0037). It's omitted when empty, and empty means latency, the baseline class (receipts::DEFAULT_CLIENT_CLASS). So receipts signed without it keep their signing bytes. A non-empty class MUST be printable ASCII of at most MAX_ROUTE_ID_LEN = 128 bytes, as in a route table (routing.md). The ingress and egress learn the class from it; relays never see it. Scoring doesn't weight credit by class yet (scoring.md §Byte crediting 6).

Per-class counters at the ends (PB-13, Task 12b). A node counts its own flows by class, as an ingress or egress, and never transit (bwos_net::node::Stats.classes, datapath.md §Counters and receipts): bytes_out is the IPv6 packet bytes its stack sends from its node address (/128), bytes_in those it takes at it, keyed by the class of the flow's remote address. The node is told a flow's class by whatever holds the session (Node::set_class(remote, class)); a flow never classed is latency (DEFAULT_CLIENT_CLASS). Traffic on the link address (BGP, BFD) and the node's own probes are in no class. bwos_node::receipts::client_receipt_classes(stats) gives, per class with any bytes, bytes_in + bytes_out: both directions, as a client receipt's bytes counts the session. These counters are the node's own view and earn nothing by themselves; credit comes only from client-signed receipts. Where a no-root node learns a flow's class is OQ-103.

Traffic profile (Task 12b, R9)

receipts::TrafficProfile { window: u64, class_bytes: BTreeMap<String, u64> } is the bytes per class in one window's committed client receipts, from receipts::traffic_profile(window, receipts):

Task 14's canary plan follows its class mix.

Bandwidth vouchers (ADR-0019)

To authorize paid VPN exit and SDN sessions (W_PAID = 1_000_000 ppm), a registered VPN Provider or Fleet Aggregator signs a BandwidthVoucher (domain bandwidth-voucher):

{"v":0,"window_from":1,"window_to":48,"client":"<client pubkey hex>","issuer":"<provider pubkey hex>","exit_node":"<optional exit pubkey hex>","max_paid_bytes":1073741824,"cost_ppm":1000000,"alpha_credit_u64":100000,"protocol_burn_ppm":50000,"sig":"<ed25519 hex>"}

Per-window node state (OQ-13)

Exit and ingress daemons keep their replay tables per window: client_paid_bytes (1,024 entries), the exit's burst_stripe_seen, latest_client_receipts, latest_cosigned and receipt book, the relay role's relay_client_bytes, downstream_seq and downstream_paid_bytes (1,024 entries each, [INCENTIVE-HOP-01]), and the ingress's per-first-hop seq/cumulative bytes (exit_seq/exit_paid_bytes, keyed by the downstream node) and stored co-signed receipts. advance_window(new_window) clears them and drops vouchers whose window_to < new_window. Before clearing, it keeps the closed window's signed batch in closed_window_batch for a late upload. Windows never move backwards.

Submission

Nodes upload receipt batches (domain receipt-batch) to validator ingest endpoints (/v0/receipts) once per RECEIPT_UPLOAD_INTERVAL (default 5 min):

{"v":0,"window":1234,"node":"<node pubkey hex>","hop_receipts":[...],"client_receipts":[...],"sig":"<ed25519 hex>"}

Batches are signed by the node key. Validators keep receipts for RETENTION_WINDOWS (default 48).

Node side ([INCENTIVE-HOP-01]):

Validator side:

Validator ingest (ObjectStore):