What a pipe saves your bot

Ten paired runs of three real checks, on Sunday, Oct 18, 2026. An agent reading the raw sources, versus the same agent reading the pipe response.

Machine-readable: benchmarks.json · benchmarks.md

Headline

• 92% fewer tokens for the same three checks: 9,137 tokens for the pipe responses, 116,827 for the raw sources. Measured with tiktoken o200k_base.
• A quiet check is 142 tokens, not 25,574 (99.4% fewer) when the SEC watch had nothing new.
• 27× smaller for news. Four RSS feeds were 90,175 tokens raw. The pipe’s top-20 result was 3,308.
• Deterministic and replayable. 30 of 30 pipe runs succeeded. All 20 SEC and news outputs matched an independent parse of the same sources, field for field. 29 of 29 completed replays reproduced the original receipt. Markets equities and the alert matched 10 of 10. Crypto prices matched 5 of 10 exactly and 10 of 10 within 0.05%, because the price moved in the second between the two fetches.

Per task

Means of 10 iterations. “Fewer” is tokens, not time.

Cost projection

Input tokens only, from the counts above. Sources, both fetched 2026-10-18: DeepSeek Flash pricing (input $0.15/M off-peak, $0.30/M peak, cache miss; output $0.60/M and $1.20/M) and Claude Sonnet 5 input price ($2/M, used only as a projection). The $0.00005/run figure is Keith’s infra estimate from the spec, not a measurement and not a customer price.

The quiet SEC check was $0.003836 raw versus $0.000071 for the pipe, including infra, at DeepSeek off-peak. Hourly, that is $2.76 versus $0.05 a month. At DeepSeek Flash prices the monthly dollar savings are under $10 per source, which is less than the $19 Builder plan, and hourly checks of these three sources are 2,160 runs a month, above the Free plan’s 1,000. The case is stronger on context size. The Sonnet 5 column is the pricier-model projection.

Quiet check

A second call of a diff pipe, with 0 new filings, returned 142 tokens (362 bytes: count 0, an empty filings list, plus run id and receipt). The baseline re-read of the raw EDGAR response plus the list of ids already seen was 25,574 tokens. The pipe read is 99.4% smaller. The news “nothing new” case was not measured: the workspace hit its daily fetch cap first.

Replay

33 replays were requested. 29 of 29 completed replays reported matched: true (the receipt was reproduced). 4 returned HTTP 429 and were not scored. 0 mismatched. Replay is POST /v1/runs/{id}/replay.

Method

Reading the raw pages

Grok 4.7 read the raw pages directly. Each prompt ran in a fresh context and was scored against the pipe output. 1 of 30 runs had an error: 1 wrong field out of 2,060 field checks. One headline came back with a curly apostrophe and a trailing space. There were no missing, extra, or hallucinated items, and no malformed JSON. A separate nothing-new check was 10 of 10 correct. Pipes: 30 of 30 runs matched an independent parse, and 29 of 29 replays reproduced the receipt.

Caveats for this check: small sample, few distinct payloads, one model, temperature not controllable, and no-code parsing enforced by instruction rather than a sandbox. The token counts above are from DeepSeek V4 Flash. This accuracy check is Grok 4.7, answered 19:35–19:46 UTC the same day. An earlier attempt to score DeepSeek stopped when the key returned HTTP 402 after 3 calls. Those 3 calls are not this rate. There is no general speed claim. A plain script is faster than a pipe. A pipe is smaller than handing the raw payloads to an LLM.

• Date: Sunday, Oct 18, 2026, 19:16–19:25 UTC (1:16–1:25 PM MDT). US equity markets were closed. Crypto prices were live.
• Where: one Linux box, Denver-region egress, workspace keithbot-mail-grokbot-com.
• Tasks: robodir-sec-filings with mode all (1 EDGAR full-text-search JSON request, 25 most recent unique filings). crypto-ai-news with mode all (CoinDesk, Cointelegraph, TechCrunch Robotics, The Robot Report; 20 newest headlines). markets-snapshot (3 Coinbase spot prices and 4 Yahoo chart responses; 7 symbols).
• Baseline: the box fetched the same URLs the pipe fetches, in parallel, and built one prompt: the task spec with the JSON schema, then every raw response verbatim. That prompt is what the token count calls “raw”.
• Pipe side: POST /v1/pipes/<name>/run immediately after the box fetch. Tokens are the full REST response body, envelope included, not just the useful fields. Each pipe was also called once over MCP. Those 3 calls succeeded, and content[0].text matched the REST size (574, 5,258, and 3,305 tokens). The full JSON-RPC body is about twice as large because it carries both content and structuredContent. Whether an agent pays for both depends on its MCP client.
• Tokenizer: tiktoken o200k_base, exact counts on the actual payloads. On the 2 prompts where DeepSeek’s own usage counter was available, it counted 6.4% and 6.8% more tokens than o200k. Dollar figures use o200k counts.
• Token-count model: DeepSeek V4 Flash, called as the Flux Router alias flux-fast. The response header x-flux-model: deepseek-v4-flash confirmed it on every call. DeepSeek states that this legacy name is served by DeepSeek-V4.1-Flash-1017. Temperature 0, default thinking mode. x-flux-summarization-applied: false on every call, so the model saw the full raw input. It was the only model the available key could reach. Scoring that model stopped when Flux returned HTTP 402 free_tier_exhausted after 3 baseline calls. Those calls are not the error check below.
• Error-check model: Grok 4.7, Sunday, Oct 18, 2026, 19:35–19:46 UTC. Each prompt was a fresh context. The model read the raw pages and composed the JSON. Answers were scored against the pipe output. Temperature could not be set. The no-code rule was the instruction, not a sandbox.
• Ground truth: a Python re-implementation of each pipe, run on the box’s own raw fetch, compared field by field with the pipe output.
• Latency, for context only: pipe median (p90) was 0.88 s (1.02) for SEC, 1.29 s (1.43) for news, and 0.88 s (0.94) for markets. The parallel raw fetch median was 0.11 s, 0.21 s, and 0.20 s. A script plus a parse under 4 ms is faster. The 3 completed LLM calls took 4.44 s, 4.51 s, and 18.97 s. That sample is too small to publish as a speed headline.

Caveats

1. The Grok 4.7 check is a small sample. 30 reads, few distinct payloads, one model. Temperature was not controllable. No-code parsing was an instruction, not a sandbox. 1 of 30 runs had an error (1 field of 2,060). The benefit is the 92% token reduction, plus byte-exact output and a replayable receipt.
2. LLM latency is 3 calls. News LLM latency and output tokens were not measured.
3. No general “N× faster” claim. A pipe call (0.9–1.3 s) is slower than a script fetching the raw sources (0.1–0.2 s). It beats an LLM only when the comparison is the LLM’s own time on those 3 calls, and that sample is too small to headline.
4. Dollar amounts are small at budget-model prices. With DeepSeek Flash, hourly checking saves about $0.13–$9.35 per source per month. The Sonnet 5 input-price projection saves about $2–$125 per source per month. Hourly checks of these 3 sources are 2,160 runs a month, above the Free plan’s 1,000 runs a month. The strongest argument is context size and a replayable receipt, not dollars.
5. The baseline is generous to the agent. It used the pipe author’s sources (one EDGAR search, not 41 Atom feeds), counted one prompt rather than a multi-turn tool loop, and fetched in parallel.
6. The pipe side could be smaller still. The REST body includes text, fetched_at, and _change on every record. The measured pipe tokens are the whole response.
7. Tokenizer. DeepSeek counted about 6–7% more tokens than o200k. Ratios hold roughly. Absolute tokens shift.
8. Sources were quiet. It was a Sunday: no new 8-Ks, few headlines, so raw payloads were identical across iterations. Pipe tokens varied only by a few tokens of timestamps.
9. List price, not the router’s charge. Flux’s actual charge on the calls that completed was about 6.7× DeepSeek’s list price. The table uses the published DeepSeek list price.
10. Infra cost is an estimate from the spec, not a measurement and not what a customer pays.

Run on 2026-10-18. Numbers on this page are copied from that run. What your bot saves.