Skip to content

Track token usage and cost of every LLM call - #136

Merged
anhnh2002 merged 3 commits into
mainfrom
feat/caw-cost-tracking
Oct 15, 2026
Merged

anhnh2002 merged 3 commits into
mainfrom
feat/caw-cost-tracking

Conversation

@anhnh2002

Copy link
Copy Markdown
Collaborator

Problem

In subscription mode (claude-code / codex), caw already returns cost_usd for every run, but each call overwrote backend.last_usage and only the incremental updater read it. A full codewiki generate never reported its tokens or cost. On top of that, Codex runs were almost always priced at $0 by caw.

Changes

  • UsageLedger (backend.py) records one entry per CLI process / agent run: phase, kind, target, model, seconds, tokens and cost. It is thread-safe for caw's MCP worker threads. Nested caw sub-module agents are separate processes that record themselves, so totals add up without double counting.
  • Both backends route usage through LLMBackend._record_usage, which still sets last_usage for the updater.
  • Phases: clustering, modules, parent_modules, overview, update.
  • metadata.json gets a usage block (totals, by_phase, unpriced_models, calls) and generation_info.provider.
  • CLI summary prints tokens used and an estimated cost (marked API-equivalent for subscription providers). Models with tokens but no price show as "cost unknown" instead of $0, with the caw pricing set hint for Codex.
  • caw pin moved to main, which includes Price codex runs on the default model and newer OpenAI models anhnh2002/caw#2: Codex is priced on the model it actually runs (-m or ~/.codex/config.toml), newer OpenAI models fall back to the opencode table, and gpt-6.1-sol is priced.
  • Docs: "Cost tracking" section in guides/providers.md.

Known gap

In API-key mode, pydantic-ai sub-module agents' usage isn't counted. Counting it would mean passing usage=ctx.usage, which also makes the parent and its sub-agents share one request limit.

Testing

  • New tests/test_usage_ledger.py (9 tests): summing, phases, unpriced detection, thread safety, CawBackend recording, metadata block, CLI output.
  • Full suite: 339 passed. 1 failure (test_clustering_skip) also fails on main.
  • End-to-end with claude-code (Haiku) on a 2-file repo: the CLI printed 125,446 tokens and $0.08, and metadata.json recorded the same.
  • Live Codex call through CawBackend with no main_model: resolved gpt-6.1-sol and priced it.

Subscription mode (claude-code / codex) already received cost_usd from
caw, but each call overwrote backend.last_usage and only the incremental
updater read it, so a full generate never reported what it cost.

- UsageLedger (backend.py) records one entry per CLI process / agent
  run, with phase, kind, target, model, seconds, tokens and cost. It is
  thread-safe for caw's MCP worker threads. Nested caw sub-module agents
  record themselves, so totals add up without double counting.
- Both backends route usage through LLMBackend._record_usage, which
  still sets last_usage for the updater.
- Phases: clustering, modules, parent_modules, overview, update.
- metadata.json gets a "usage" block (totals, by_phase,
  unpriced_models, calls) and generation_info.provider.
- `codewiki generate` prints tokens and the estimated cost, notes that
  subscription cost is API-equivalent, and flags models with tokens but
  no price instead of showing $0.
caw main now prices codex runs on the model they actually run (including
gpt-6.1-sol and newer OpenAI models), which the cost report relies on.
CI installs requirements.txt only, so importing caw failed collection.
Use a stand-in usage object and a stub backend; the CawBackend test is
skipped when caw is missing.
@anhnh2002
anhnh2002 merged commit 2b99143 into main Oct 15, 2026
2 checks passed
@anhnh2002
anhnh2002 deleted the feat/caw-cost-tracking branch October 15, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant