{
 "title": "Asipipes agent benchmark: raw sources + LLM vs one pipe call",
 "test_date": "2026-10-18",
 "test_window_utc": "2026-10-18T19:16Z to 19:25Z (Sunday; equity markets closed, crypto live)",
 "tokenizer": "tiktoken o200k_base (exact counts on the actual payloads); DeepSeek's own usage counter reported 6.4-6.8% more tokens on the 2 prompts it measured",
 "model": {
  "name": "DeepSeek V4 Flash (served as 'deepseek-v4-flash' via Flux Router alias 'flux-fast'; DeepSeek bills this name as DeepSeek-V4.1-Flash-1017)",
  "temperature": 0,
  "llm_runs_completed": 3,
  "llm_runs_scored": 1,
  "status": "Token counts used this model. The key returned HTTP 402 free_tier_exhausted after 3 baseline calls, so DeepSeek was not scored for errors. The published accuracy check is the separate Grok 4.7 read in error_rate."
 },
 "prices": {
  "deepseek_flash_per_M_input_cache_miss": {
   "off_peak": 0.15,
   "peak": 0.3
  },
  "deepseek_flash_per_M_output": {
   "off_peak": 0.6,
   "peak": 1.2
  },
  "source": "https://api-docs.deepseek.com/quick_start/pricing/ (fetched 2026-10-18)",
  "claude_sonnet_5_per_M_input_projection": 2.0,
  "claude_source": "https://platform.claude.com/docs/en/about-claude/pricing (fetched 2026-10-18)",
  "asipipes_infra_per_run_estimate": 5e-05,
  "asipipes_infra_note": "Keith's own infra cost estimate from SPEC section 11.6, not a customer price"
 },
 "method": "Same raw sources each pipe fetches, fetched from one Linux box; tokens = o200k_base count of (task prompt + raw payloads) vs the full /v1 pipe response JSON the agent reads. 10 paired iterations per task. Pipe output checked against an independent Python re-implementation run on the box's own fetch. Costs = measured tokens x published prices; input tokens only unless noted.",
 "headline": {
  "token_reduction_all_three_tasks_pct": 92.2,
  "tokens_per_round_raw": 116827,
  "tokens_per_round_pipe": 9137.0,
  "quiet_check_sec_tokens": {
   "raw_reread_with_seen_list": 25574,
   "pipe_zero_new": 142,
   "reduction_pct": 99.4
  },
  "pipe_runs": 30,
  "pipe_failures": 0,
  "pipe_outputs_matching_independent_parse": "20/20 exact (SEC, news); markets equities+alerts 10/10 exact, crypto 5/10 exact and 10/10 within 0.05% (prices moved between the two fetches)",
  "replays_matched": "29/29 (4 more replay calls were rate-limited, HTTP 429)",
  "mcp_calls": "3/3 succeeded, same tokens as REST content",
  "error_rate_llm_baseline": "Grok 4.7, fresh context, scored against the pipe output: 1 of 30 runs had an error (1 wrong field of 2,060). See error_rate."
 },
 "error_rate": {
  "model": "Grok 4.7",
  "method": "The model read the raw pages directly. Each prompt ran in a fresh context. Answers were scored against the pipe output. The model composed the JSON itself.",
  "answered_utc": "2026-10-18T19:35Z to 19:46Z",
  "runs": 30,
  "runs_with_an_error": 1,
  "runs_correct": 29,
  "field_checks": 2060,
  "wrong_fields": 1,
  "wrong_field": "One news headline came back with a curly apostrophe and a trailing space.",
  "missing_items": 0,
  "extra_or_hallucinated_items": 0,
  "malformed_json": 0,
  "nothing_new_correct": "10 of 10",
  "pipes_matched_independent_parse": "30 of 30",
  "pipes_replays_reproduced_receipt": "29 of 29",
  "statement": "Pipes returns byte-exact, replayable output. 1 of 30 runs had an error (1 field of 2,060).",
  "main_benefit": "92% fewer tokens, plus determinism and receipts",
  "caveats": [
   "Small sample.",
   "Few distinct payloads.",
   "One model.",
   "Temperature was not controllable.",
   "No-code parsing was enforced by instruction, not by a sandbox."
  ]
 },
 "tasks": {
  "sec": {
   "description": "New 8-K filings, 41 robotics/AI companies (SEC EDGAR full-text search JSON, 1 request)",
   "pipe": "robodir-sec-filings",
   "iterations": 10,
   "raw_tokens": 24520,
   "pipe_tokens": 5258.0,
   "token_reduction_pct": 78.6,
   "pipe_latency_median_s": 0.88,
   "raw_fetch_median_s": 0.111,
   "input_cost_per_check_usd_deepseek_offpeak": {
    "raw": 0.003678,
    "pipe_incl_infra": 0.000839
   },
   "input_cost_per_month_hourly_usd_deepseek_offpeak": {
    "raw": 2.65,
    "pipe_incl_infra": 0.6
   },
   "input_cost_per_month_hourly_usd_sonnet5_projection": {
    "raw": 35.31,
    "pipe_incl_infra": 7.61
   },
   "pipe_success": "10/10",
   "llm_measured_n1": {
    "prompt_tokens": 26193,
    "completion_tokens": 5765,
    "latency_s": 18.97,
    "full_cost_usd_deepseek_offpeak": 0.007388,
    "scored": false
   }
  },
  "news": {
   "description": "Crypto + AI/robotics headlines (4 RSS feeds)",
   "pipe": "crypto-ai-news",
   "iterations": 10,
   "raw_tokens": 90175,
   "pipe_tokens": 3308.0,
   "token_reduction_pct": 96.3,
   "pipe_latency_median_s": 1.29,
   "raw_fetch_median_s": 0.213,
   "input_cost_per_check_usd_deepseek_offpeak": {
    "raw": 0.013526,
    "pipe_incl_infra": 0.000546
   },
   "input_cost_per_month_hourly_usd_deepseek_offpeak": {
    "raw": 9.74,
    "pipe_incl_infra": 0.39
   },
   "input_cost_per_month_hourly_usd_sonnet5_projection": {
    "raw": 129.85,
    "pipe_incl_infra": 4.8
   },
   "pipe_success": "10/10"
  },
  "markets": {
   "description": "Price snapshot BTC/ETH/SOL + MSTY/MSTR/WNTR/QQQ (7 JSON APIs)",
   "pipe": "markets-snapshot",
   "iterations": 10,
   "raw_tokens": 2132,
   "pipe_tokens": 571.0,
   "token_reduction_pct": 73.2,
   "pipe_latency_median_s": 0.88,
   "raw_fetch_median_s": 0.199,
   "input_cost_per_check_usd_deepseek_offpeak": {
    "raw": 0.00032,
    "pipe_incl_infra": 0.000136
   },
   "input_cost_per_month_hourly_usd_deepseek_offpeak": {
    "raw": 0.23,
    "pipe_incl_infra": 0.1
   },
   "input_cost_per_month_hourly_usd_sonnet5_projection": {
    "raw": 3.07,
    "pipe_incl_infra": 0.86
   },
   "pipe_success": "10/10",
   "llm_measured_n2": {
    "prompt_tokens": 2268,
    "completion_tokens": [
     602,
     449
    ],
    "latency_s": [
     4.44,
     4.51
    ],
    "scored_runs": 1,
    "scored_errors": 0
   }
  }
 },
 "caveats": [
  "Grok 4.7, fresh context, scored against the pipe output: 1 of 30 runs had an error (1 of 2,060 fields: a curly apostrophe and a trailing space on one headline). Nothing-new checks were 10 of 10. Pipes: 30 of 30 matched an independent parse; 29 of 29 replays reproduced the receipt. Small sample, few distinct payloads, one model, temperature not controllable, no-code rule was an instruction rather than a sandbox. DeepSeek was not scored; its key returned HTTP 402 after 3 calls.",
  "LLM latency is from 3 calls only.",
  "A hand-written script fetches and parses these sources faster than a pipe call (0.1-0.2 s vs 0.9-1.3 s); the time savings apply only against an LLM reading raw data.",
  "At DeepSeek Flash prices the monthly dollar savings are under $10 per source, smaller than a $19 Builder plan; with pricier models (projection at Claude Sonnet 5 input price) they are larger.",
  "Hourly checks of 3 sources = 2,160 runs/month, above the Free plan's 1,000."
 ],
 "reproduce": "/workspace/asipipes-bench (bench/run_bench.py, bench/pipes_extra.py, bench/analyze.py)"
}