EffectFence

EffectFence

Public register: the Retry-Safety Index lists which agent-payment implementations pay once when the answer is lost — verified safe, found & fixed (with time-to-fix), and how to get verified. Every row links to its proof.

Your swarm doesn't need more memory. It needs a causal fence around tool side effects.

Free: submit any client, facilitator, SDK or toolkit that moves money — yours or someone else's — and we read it and publish a verdict on the Retry-Safety Index at no cost. Findings come back with the mechanism, the file and line, and a failing test. You are counted, never named, until you ship a fix. [Submit for grading →](https://github.com/aurumflux20/hostile-facilitator/issues/new?template=submit.yml)

⚡ effectfence — THE STORM

1,000 attempts to charge order #777 ($49.00): concurrent racers + late retries…

ACTUAL EXECUTIONS : 1 ← the whole point served sealed receipt : 995 told to stand down : 4 elapsed : 22.33ms

💰 double-charges prevented this run: $48,951.00 ✅ ONE execution. Every other attempt was fenced, replayed, or refused.

Run the attack yourself:

cargo run --release --example storm

The multi-agent turf war

The storm above is one action, many racers. The harder case is *different* agents making *contradictory* decisions on the same production resource — the coordination failure now reported across production multi-agent systems (~a third of 2026 multi-agent incidents). Three autonomous SRE agents react to one latency spike:

cargo run --example turf_war

Agent A (autoscaler): scale node pool UP

ADMITTED. Fence leased cluster:prod-us-east-1 — executing kubectl scale up. DONE. EffectCert minted — verify() -> true

Agent B (cost-optimizer): scale the same pool DOWN REFUSED before kubectl ran: read-set for metrics:prod-us-east-1 is stale: B decided on seq 0, world is at seq 1.

Agent C (deploy-bot): roll the deployment BACK C's causal view is concurrent with A's: true -> escalated to a human instead of corrupting the cluster.

Actions proposed: 3 executed: 1 cluster: intended, not corrupted.

Each agent was individually correct for the state it read. Run concurrently without coordination, all three kubectl calls fire and the cluster ends in a state none of them intended — the $100M outage. The fence lets exactly one act, refuses the other two *before* their side effect runs, and tells each one why.

Nothing above is mocked — every call is the real crate API.

EffectFence is a causal concurrency fence for multi-agent tool calls. When more than one agent (or retry, or re-dispatch) can end up trying to run the same side-effecting operation — charge a card, send a payout, provision a resource — EffectFence guarantees exactly one attempt ever executes it: same-instant races are decided by an atomic compare-exchange reservation, and late duplicates get the recorded outcome replayed instead of running again. Every effect that does run gets a content-addressed certificate chained to whatever it was causally built on.

It ships as a Rust library (effectfence::fence) and as a stdio MCP server exposing three tools — fence_prepare, fence_commit, fence_abort — so agents can route side-effecting tool calls through the fence instead of racing each other directly.

The problem

In a multi-agent gateway, more than one caller can end up trying to run the same effect:

• Two agents independently decide "charge the customer for order #123" needs to happen — at the same instant.
• A supervisor times out waiting for a tool call and re-dispatches it while the original is still in flight.
• A retried or duplicated event triggers the same decision again, minutes after the first attempt already succeeded.

Naively, any of these double-runs the effect. Naively rejecting every duplicate with no memory of outcome is also wrong: if the first attempt crashed, the effect never runs at all, and a duplicate that arrives after success gets an error instead of the result it needs. EffectFence closes all of it with optimistic concurrency control (OCC) plus an intent ledger: attempts don't block each other, exactly one executes, and every other attempt learns what actually happened.

Architecture

Four pieces compose into the fencing protocol:

The intent ledger is what stops *duplicates*, not just races. Every effect carries an intent — a stable id for the logical action (e.g. "charge:order-123"). Attempts sharing an intent are the same action: the first is admitted and holds a lease; concurrent duplicates are told an attempt is in flight; duplicates arriving after success get the recorded certificate replayed verbatim; duplicates after a failure are fenced (the side effect may or may not have fired — that must be reconciled, not blindly retried) until explicitly cleared. Crashed holders lose their lease after a TTL so the action isn't stuck forever.

Vector clocks (VectorClock) track causal "happened-before" relationships across agents — one logical counter per agent, joined via elementwise-max merge, compared via a le partial order, with a concurrent check for genuinely unordered events and a stable SHA-256 digest for inclusion in certificates.

OCC read-sets (ReadSetEntry) record the causal dependencies a decision was based on: "when I decided to act, domain D was at sequence S." Both prepare_effect_fence and commit_effect_cert validate every entry against live state — if anything moved, the attempt is rejected as stale rather than allowed to act on outdated information.

CAS domain fencing is where same-instant races are decided. Each domain (a named contention scope, e.g. "order:123") has an AtomicU64 sequence counter. The *decision* is a single atomic compare_exchange — exactly one concurrent caller can win for any given expected sequence. (Precision note: the counter *lookup* sits behind a short mutex; only the race decision itself is lock-free. Ideas for a fully lock-free path are welcome.)

              ┌ intent gate ──────── already done? → Replay(recorded cert)   [do NOT run]

│ in flight / failed? → rejected [do NOT run] EffectRequest┤ ├ read-set check ───── dependency moved? → ReadSetStale [do NOT run] │ └ domain CAS ────────── lost the race? → DomainRace [do NOT run] │ └ Fresh(ticket) → run the effect → commit_effect_cert → EffectCert ↘ abort_effect (failed; fenced until reconciled)

Every committed effect becomes an EffectCert: a SHA-256 content hash over {intent, parent, domain, seq, tool, args, result, vector_clock, read_set, agent}, chained to a parent cert hash for causal lineage. Two certs with the same hash are, by definition, records of the same effect — EffectCert::verify() recomputes the hash and confirms it hasn't been tampered with or hand-built incorrectly.

Scope

This is a single-process fence with a durable ledger. Every admission, outcome and operator clear is appended to a JSON-lines file and fsync'd before the fence answers, so a restart of the proxy does not forget what already ran: a duplicate after the restart is replayed, not re-executed. Anything that was in flight when the process died is fenced as an unknown outcome on the next start — the holder is gone and nobody can say whether the effect fired — until an operator reconciles and clears it. (Library users: EffectFence::open(path, config); EffectFence::new() stays in-memory. The shipped binary picks $EFFECTFENCE_LEDGER, else ~/.local/state/effectfence/ .jsonl, and EFFECTFENCE_LEDGER=memory opts out.) Two things it deliberately does not do (yet):

• Multi-replica fencing. The ledger is one file, so it protects one gateway process (across its restarts), not several replicas at once. A horizontally scaled gateway needs the same intent/domain/read-set model backed by a shared store (e.g. SETNX+CAS in Redis, or an optimistic version column in Postgres) — the types here are meant to carry over directly to that backend.
• A holder that outlives its lease. An in-flight lease is claimable once it expires — that is what makes crash recovery possible — and nothing in an expiry tells the fence whether the holder died or is simply slower than FenceConfig::lease_ttl (60 s by default). An effect that runs past its lease without saying anything therefore has its claim taken over and runs a second time, which is the very thing this crate exists to stop. The remedy is one call: EffectFence::heartbeat(intent) while the effect is still running (about every third of the lease). wrap does this for you on every forwarded call; agents driving fence_prepare directly should call the fence_heartbeat tool. Stop beating and the lease lapses on schedule, so a genuinely dead holder is still recovered.
• Enforcement. The fence protects agents that route their effects through it; it cannot stop an agent that bypasses it entirely. Deploy it at the one choke point your agents share (the gateway process that owns the tools).

Memory is bounded: finished outcomes expire after a configurable TTL (FenceConfig::result_ttl, swept by EffectFence::sweep), queries never create tracking state, and domain counters are tiny and manually evictable (evict_domain).

Quickstart: library

use effectfence::fence::{

prepare_effect_fence, commit_effect_cert, Admission, EffectFence, EffectRequest, VectorClock, };

let fence = EffectFence::new();

let req = EffectRequest { intent: "charge:order-123".into(), // same action -> same intent, always parent: None, // hash of the cert this follows, if any domain: "order:123".into(), // contention scope tool: "charge_card".into(), args: serde_json::json!({"amount_cents": 1999}), read_set: vec![], // other domains this decision cross-checked agent: "agent-a".into(), known_clock: VectorClock::new(), };

match prepare_effect_fence(&fence, req)? { Admission::Fresh(prepared) => { // This attempt won. Actually run the charge... let cert = commit_effect_cert( &fence, prepared, serde_json::json!({"charge_id": "ch_123"}), )?; assert!(cert.verify()); // (on failure: abort_effect(&fence, prepared, "why") instead) } Admission::Replay(cert) => { // This exact action already ran -- use cert.result, charge nothing. } }

A concurrent duplicate of the same intent gets Err(FenceError::IntentInFlight); a same-instant race on the domain gets Err(FenceError::DomainRace); either way it must not run the effect.

Try it in 10 seconds (nothing installs, nothing real fires)

cargo install effectfence   # or: npx -y effectfence demo

effectfence demo

Twelve agents reach for one $49 charge at the same instant. You'll see it hit a built-in server raw — 12 duplicate charges — then the same twelve calls behind the fence: exactly 1. This binary is talking to itself, so no real call fires and you need no server of your own to see the point.

── Act 1: the raw server, no fence ──

DISTINCT effec