diff --git a/ROADMAP.md b/ROADMAP.md index 90cc39a..ada7bcf 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -454,7 +454,7 @@ allowances; the agent records document-driven denials and approval decisions. - [x] **10.4 — Cost attribution** per run, thread (session), node and explicit tenant. The price is now recorded, not guessed: every gateway publishes a `cost_usd` through the same `llm_output` envelope, the runtime's usage - callback accumulates it per node and writes it onto the `end` event, and + callback accumulates it per node and writes it onto the `end` or `error` event, and an `AgentNode` writes the per-call figure onto each `model` event. A backend that reports no price still falls back to a `RateCard` estimate, and `recorded_cost_usd` and `estimated_cost_usd` never mix — a recorded @@ -467,9 +467,10 @@ allowances; the agent records document-driven denials and approval decisions. `grapharc cost --tenant NAME` select whole labelled runs, refuse conflicting labels, and leave old unlabelled runs unassigned. Planning tokens without a recorded price are estimated or marked unpriced, including in tenant - totals. **Producer limit:** planner turns and failed node terminal events - still omit the provider price ([#138](https://github.com/CodeGraphContext/GraphARC/issues/138)); - a rate-card estimate is not a recorded bill. + totals. Planner turns retain the call-specific price and model from the + returned message envelope, including parse failures. Every node terminal + error path retains its own usage scope, excluding concurrent siblings' + spend. A rate-card estimate is not a recorded bill. Tenant labels provide attribution, not authorization. - [ ] **10.2 — Rollback** and versioned graph/prompt configs. - [ ] **10.5 — Alerting** on budget, failure, and verifier-drift. diff --git a/docs/deep-dive.md b/docs/deep-dive.md index cfba7da..da8b0c3 100644 --- a/docs/deep-dive.md +++ b/docs/deep-dive.md @@ -94,7 +94,7 @@ OpenRouter also carries routing: model-level `fallback_models` chains, provider **Cost ceilings are enforced, on both sides of a call.** A `SpendMeter` refuses before a call once the ceiling is reached, and after a call it charges the cost and *then* raises if that crossed the line — so overspend is bounded by the single call that crossed it rather than discovered a node later. The limit worth stating: a call whose cost the provider does not report cannot be charged, and those land in `unpriced_calls` rather than being guessed at. `unpriced_calls > 0` means the ceiling saw less than the whole bill. -Caveats each backend accepts openly. **Claude CLI:** `bind_tools` and `with_structured_output` raise `NotImplementedError` — the adapter implements neither, so what you get is LangChain's `BaseChatModel` default, and `claude -p` in the tool-free mode GraphARC drives it in offers nothing to implement them with. GraphARC also has no cache control on this path — the CLI decides and the usage envelope reports what it did — and calls spend subscription quota. **OpenRouter:** credit is reserved against `max_tokens`, so the default is deliberately modest. **OpenAI:** the API returns token counts and no price, so `cost_usd` is `None` and a dollar ceiling counts calls instead of enforcing — pass `price_per_million=` or price the trace afterwards with `observe.cost.RateCard`; token budgets are unaffected. **Ollama:** free by definition, so calls are charged `0.0` rather than counted as unpriced, and whether tool-calling works depends on the model you pulled rather than on the adapter. **All of them:** a reported per-call `cost_usd` is captured and budgeted against. Successful node events and agent model events carry it into the trace; planner turns and failed node terminal events currently omit it. See [ROADMAP.md](../ROADMAP.md) §10.4 for explicit tenant attribution and producer pricing limits. +Caveats each backend accepts openly. **Claude CLI:** `bind_tools` and `with_structured_output` raise `NotImplementedError` — the adapter implements neither, so what you get is LangChain's `BaseChatModel` default, and `claude -p` in the tool-free mode GraphARC drives it in offers nothing to implement them with. GraphARC also has no cache control on this path — the CLI decides and the usage envelope reports what it did — and calls spend subscription quota. **OpenRouter:** credit is reserved against `max_tokens`, so the default is deliberately modest. **OpenAI:** the API returns token counts and no price, so `cost_usd` is `None` and a dollar ceiling counts calls instead of enforcing — pass `price_per_million=` or price the trace afterwards with `observe.cost.RateCard`; token budgets are unaffected. **Ollama:** free by definition, so calls are charged `0.0` rather than counted as unpriced, and whether tool-calling works depends on the model you pulled rather than on the adapter. **All of them:** a reported per-call `cost_usd` is captured and budgeted against. Successful and failed node terminal events, planner turns and agent model events carry it into the trace. See [ROADMAP.md](../ROADMAP.md) §10.4 for explicit tenant attribution and producer pricing limits. ## Tools and the harness @@ -248,7 +248,7 @@ A stable system is not one that claims to have no edges — it is one whose edge - **`run_command` is not confined.** Argv-only and never a shell, but the child is an ordinary process with your privileges. - **`interrupt()` suspends but cannot be resumed.** LangGraph's native interrupt stops the graph and shows on `get_state`, and there is no supported resume path — resuming means passing a `Command` as *input*, which is closed by design. Use the session layer's approval gate for human-in-the-loop. - **Still unwrapped from LangGraph:** `retry_policy`, `cache_policy`, `durability`, subgraphs. `.inner` reaches them, but execution entry points there fail closed, so `.inner` is an inspection escape hatch and not a way to run the graph. -- **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Explicit tenants are supported by `attribute_tenant` and `grapharc cost`; legacy unlabelled runs are excluded from named tenant bills. Planner turns and failed node terminal events still omit the provider price, so their tokens need a rate card or remain unpriced. +- **Cost is recorded when a backend reports one, estimated when it does not.** Both gateways publish the provider's `cost_usd` through the same `llm_output` envelope, the runtime's usage callback writes it onto the node's `end` event, and an agent's `model` events carry the per-call breakdown. A backend that reports no price still falls back to a `RateCard` estimate, and the two figures stay apart — `recorded_cost_usd` is never a guess. Explicit tenants are supported by `attribute_tenant` and `grapharc cost`; legacy unlabelled runs are excluded from named tenant bills. Planner turns and all failed node terminal paths retain the provider price and model identity when reported; parallel node failures retain only their own token spend. - **A node's tokens are its own, not the run's movement while it ran.** Worth stating because it was the other way round: an `end` event carried the difference between two readings of the run's *shared* meter, so under fan-out the workers' windows overlapped and each was credited with its siblings' concurrent spend. Three workers costing 8 tokens each traced as 24/16/8, and `metrics` and `cost` agreed on 48 for 24 tokens of real work — doubling the estimated bill purely because the work ran in parallel. Attribution now comes from a per-node scope on the meter, so the same work costs the same serially and in parallel; a hand charge the usage callback never saw still lands on the node that made it. - **A planning round is an envelope, not a measurement.** A `round` event used to carry the planner's `tokens` and the round's `duration_ms`, both of which `metrics`, `cost` and `replay` add on top of node totals — and the planner's spend was already reported by its own `plan` event, so it was counted twice, and a round's duration encloses the plan plus every node it ran. Neither is on the event now; both are on its `state_delta` as `round_tokens` / `round_iterations` / `round_duration_ms`, where no reader sums them. `RoundRecord.iterations` also holds a figure now rather than always `0`. - **The Claude CLI backend is completion-only, and an agent node on it is *delegated* rather than governed.** The CLI has no tool-calling wire format, so GraphARC cannot run its own gated loop over it. Rather than refuse, `AgentNode` hands the whole loop to Claude Code's headless agent, in one of two named tiers. `allowlist`, the default, pre-approves exactly the Claude Code twins of the node's own registered tools (`read_file`→`Read`, … `run_command`→`Bash`), so one operator declaration governs both the governed loop and the delegated one — but the enforcement is Claude Code's own gating, not this graph's per-call policy, there are no per-tool trace events, and anything unlisted falls to headless default gating, which fails closed. `bypass`, explicit opt-in only, runs `bypassPermissions`: every tool Claude Code has, no checks at all. In either tier the calls are not confined by the sandbox executor and the token figure is the sub-agent's own rather than one GraphARC metered call by call; a `--max-tokens` the delegated path cannot enforce is refused rather than silently unapplied. The workspace boundary and the wall-clock ceiling still hold — the CLI runs in its own session, and the deadline kills the whole process group, not just the direct child. It warns on `DelegatedToolUseWarning` at construction, naming the tier, and marks every trace event `executor=delegated` with its `delegated_mode`; filter that warning to an error to get the old refusal back. Structured output still needs an OpenAI-wire backend: `openrouter`, `openai`, or a local `ollama`. @@ -256,7 +256,7 @@ A stable system is not one that claims to have no edges — it is one whose edge - **`.env` and `grapharc.toml` follow the same discovery rule: the working directory, and nowhere else.** Neither searches parent directories — a run must not be governed by a file you did not know about, and must not be *billed* to one either. **This is a behaviour change:** the credential loader used to walk up to `/`, so a `.env` in an ancestor directory (a `$HOME` one on a shared box, a client project one above a demo checkout) was picked up silently. If you relied on that, move the file into the directory you run from, `export` the variable, or pass `env_file=` to name it explicitly. A real environment variable still beats any file. - **`grapharc run` has no budget unless you give it one.** Set any of `--max-tokens`, `--max-iterations`, `--max-seconds`, or `--max-concurrency`; without them each dimension is unlimited and the gate admits a topology of any worst-case cost. -**Verified this pass:** `pytest` → green, 2,354 selected and 13 deselected (the live ones); `ruff check .` clean; all eight `grapharc demo` stages green, plus the `trace` / `metrics` / `viz` / `replay` tour against a freshly recorded demo trace; the wheel builds and imports all submodules in a clean virtualenv with `[all]`, and `0.1.8` on PyPI is that wheel. The counts are a snapshot, not a property of the project — `pytest` re-derives them in one command, which is the only reason they are quoted, and `tests/test_deep_dive.py` fails this line rather than letting it drift. +**Verified this pass:** `pytest` → green, 2,384 selected and 13 deselected (the live ones); `ruff check .` clean; all eight `grapharc demo` stages green, plus the `trace` / `metrics` / `viz` / `replay` tour against a freshly recorded demo trace; the wheel builds and imports all submodules in a clean virtualenv with `[all]`, and `0.1.8` on PyPI is that wheel. The counts are a snapshot, not a property of the project — `pytest` re-derives them in one command, which is the only reason they are quoted, and `tests/test_deep_dive.py` fails this line rather than letting it drift. [ROADMAP.md](../ROADMAP.md) tracks what is built and what is not, item by item. diff --git a/grapharc/planner/proposal.py b/grapharc/planner/proposal.py index 40954c4..7c953b8 100644 --- a/grapharc/planner/proposal.py +++ b/grapharc/planner/proposal.py @@ -690,10 +690,19 @@ def propose( error = "" proposal = self._stamp(proposal) + # Read this call's envelope, rather than a model's mutable last_usage: + # the latter can describe another concurrent call, or an earlier call + # when this one failed before producing a message. + metadata = raw_message.response_metadata if raw_message is not None else {} + cost = (metadata.get("token_usage") or {}).get("cost") + if cost is None: + cost = metadata.get("cost_usd") self._emit( ctx, duration_ms=(time.perf_counter() - started) * 1000, tokens=tokens, + cost_usd=cost, + model=metadata.get("model_name"), state_delta={ "structured": structured, "nodes": proposal.node_count() if proposal else 0, diff --git a/grapharc/runtime/graph.py b/grapharc/runtime/graph.py index 02b7c7a..e02155e 100644 --- a/grapharc/runtime/graph.py +++ b/grapharc/runtime/graph.py @@ -855,6 +855,7 @@ def _leave( usage: Any = None, ) -> Any: """Everything both node wrappers do after the node body returns.""" + measured = _usage_fields(ctx, tokens_before, usage) delta: dict[str, Any] | None = None if result is not None: try: @@ -864,7 +865,7 @@ def _leave( "error", duration_ms=duration_ms, error=str(err), - tokens=ctx.meter.tokens - tokens_before, + **measured, ) raise @@ -881,31 +882,15 @@ def _leave( "error", duration_ms=duration_ms, error=f"budget: {exc.reason}", - tokens=ctx.meter.tokens - tokens_before, + **measured, ) raise - # The provider's own price for every model call made inside this node, - # collected by the usage callback. Left None when no backend reported - # one, so `observe.cost` can keep a measured figure apart from a - # rate-card estimate instead of recording a guess as a fact. - models = getattr(usage, "models", ()) - # This node's own spend, from the meter's per-node scope, not the movement - # of the run's shared total while the node ran. Under fan-out those differ: - # the workers' windows overlap, so every worker used to be credited with - # each sibling's concurrent spend — three workers costing 8 tokens each - # traced as 24/16/8, and `metrics` and `cost` both reported 48 for 24 - # tokens of real work, doubling the estimated bill purely because the work - # ran in parallel. Falls back to the difference when no scope was open, so - # a caller driving `_leave` outside `charging()` still gets a figure. - spent = getattr(usage, "tokens", None) emit( "end", state_delta=delta, duration_ms=duration_ms, - tokens=ctx.meter.tokens - tokens_before if spent is None else spent, - cost_usd=getattr(usage, "cost_usd", None), - model=models[0] if len(models) == 1 else None, + **measured, ) return result @@ -935,7 +920,7 @@ async def awrapped(state: Any, config: RunnableConfig) -> Any: "error", duration_ms=(time.perf_counter() - t0) * 1000, error=repr(exc), - tokens=ctx.meter.tokens - tokens_before, + **_usage_fields(ctx, tokens_before, usage), ) raise return self._leave( @@ -974,7 +959,7 @@ def wrapped(state: Any, config: RunnableConfig) -> Any: "error", duration_ms=(time.perf_counter() - t0) * 1000, error=repr(exc), - tokens=ctx.meter.tokens - tokens_before, + **_usage_fields(ctx, tokens_before, usage), ) raise return self._leave( @@ -991,6 +976,22 @@ def wrapped(state: Any, config: RunnableConfig) -> Any: return wrapped +def _usage_fields(ctx: RunContext, tokens_before: int, usage: Any) -> dict[str, Any]: + """The same measured spend on every terminal event, successful or failed. + + A node's own scope excludes concurrent siblings' spend. The shared meter + difference is only a fallback for callers outside `charging()`. A missing + provider price stays missing; rate-card estimates belong to trace readers. + """ + spent = getattr(usage, "tokens", None) + models = getattr(usage, "models", ()) + return { + "tokens": ctx.meter.tokens - tokens_before if spent is None else spent, + "cost_usd": getattr(usage, "cost_usd", None), + "model": models[0] if len(models) == 1 else None, + } + + def topology_delta(arc: GraphARC) -> dict[str, Any]: """The declared shape of a graph, as one trace-event `state_delta`. diff --git a/tests/test_provider_price_events.py b/tests/test_provider_price_events.py new file mode 100644 index 0000000..296e278 --- /dev/null +++ b/tests/test_provider_price_events.py @@ -0,0 +1,243 @@ +"""Provider-reported prices survive planning and failed governed work (#138).""" + +from __future__ import annotations + +import asyncio + +import pytest +from langchain_core.runnables import RunnableLambda +from langgraph.types import Command + +from grapharc.observe import RateCard, TraceRecorder, attribute, attribute_tenant +from grapharc.observe.cost import attribute_thread +from grapharc.observe.metrics import summarize +from grapharc.planner import PlannerNode +from grapharc.runtime.budget import Budget, BudgetExceeded, BudgetMeter +from grapharc.runtime.graph import ( + END, + START, + GraphARC, + GraphRoutingError, + RunContext, + StateTypeError, + WritePermissionError, +) +from grapharc.runtime.state import GraphARCState +from grapharc.testing import ScriptedChatModel + + +class State(GraphARCState): + answer: str = "" + proposal: object = None + count: int = 0 + + +def context(): + return RunContext( + run_id="run", thread_id="thread", tenant="acme", graph="g", meter=BudgetMeter(Budget()) + ) + + +def assert_price(trace, price): + run = attribute(trace, "run") + assert run.recorded_cost_usd == pytest.approx(price) + assert attribute_thread(trace, "thread").recorded_cost_usd == pytest.approx(price) + assert attribute_tenant(trace, "acme").recorded_cost_usd == pytest.approx(price) + assert run.unpriced_tokens == 0 + assert run.tokens == summarize(trace, "run").tokens > 0 + + +@pytest.mark.parametrize("reply", ['{"nodes": []}', "not a proposal"]) +@pytest.mark.parametrize("price", [None, 0.0, 0.003]) +@pytest.mark.parametrize("structured", [False, True]) +def test_planner_records_the_call_price_even_when_parsing_fails(tmp_path, reply, price, structured): + class StructuredModel(ScriptedChatModel): + def with_structured_output(self, schema, **kwargs): + return RunnableLambda( + lambda messages: { + "raw": self.invoke(messages), + "parsed": None, + "parsing_error": ValueError("parse failed"), + } + ) + + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = StructuredModel(responses=[reply], cost_usd=price, model_name="acme/planner") + ctx = context() + result = PlannerNode(model, trace=trace, structured=structured).propose("plan", ctx) + assert result.ok == (reply.startswith("{")) + event, = trace.read_events() + assert event.phase == "plan" + assert event.cost_usd == price + assert ctx.meter.tokens == result.tokens == event.tokens > 0 + cost = attribute(trace, "run", rates=RateCard(default=2.0)) + if price is None: + assert cost.recorded_cost_usd is None + assert cost.estimated_cost_usd == pytest.approx(result.tokens / 1000 * 2) + else: + assert event.model == "acme/planner" + assert_price(trace, price) + assert cost.estimated_cost_usd is None + + +@pytest.mark.asyncio +@pytest.mark.parametrize("async_node", [False, True]) +@pytest.mark.parametrize("failure", ["body", "write", "type", "routing", "budget", "manual_budget"]) +async def test_all_failed_node_terminals_retain_provider_usage(tmp_path, async_node, failure): + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = ScriptedChatModel(responses=["done"], cost_usd=0.002, model_name="acme/worker") + graph = GraphARC(State, name="g", trace=trace) + + def finish(ctx): + if failure == "body": + raise ValueError("failed after spending") + if failure == "write": + return {"undeclared": True} + if failure == "type": + return {"count": "not an integer"} + if failure == "routing": + return Command(goto="unknown") + if failure == "manual_budget": + ctx.meter.charge_tokens(100) + return {"answer": "done"} + + def body(state, ctx): + model.invoke("work") + return finish(ctx) + + async def abody(state, ctx): + await model.ainvoke("work") + return finish(ctx) + + graph.add_node("worker", abody if async_node else body, writes={"answer", "count"}) + graph.add_edge(START, "worker") + graph.add_edge("worker", END) + compiled = graph.compile() + budget = Budget(max_tokens=1 if failure == "budget" else 50) + expected = { + "body": ValueError, + "write": WritePermissionError, + "type": StateTypeError, + "routing": GraphRoutingError, + "budget": BudgetExceeded, + "manual_budget": BudgetExceeded, + }[failure] + with pytest.raises(expected): + await compiled.ainvoke( + {}, run_id="run", thread_id="thread", tenant="acme", budget=budget + ) + terminal, = [e for e in trace.read_events() if e.phase == "error"] + assert terminal.cost_usd == pytest.approx(0.002) + assert terminal.model == "acme/worker" + assert terminal.tokens == compiled.last_run.meter.tokens > 0 + assert_price(trace, 0.002) + + +def test_nested_planner_price_and_tokens_are_counted_once(tmp_path): + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = ScriptedChatModel(responses=['{"nodes": []}'], cost_usd=0.003) + planner = PlannerNode(model, trace=trace, structured=False) + graph = GraphARC(State, name="g", trace=trace) + graph.add_node("planner", lambda state, ctx: {"proposal": planner.propose("plan", ctx)}, + writes={"proposal"}) + graph.add_edge(START, "planner") + graph.add_edge("planner", END) + compiled = graph.compile() + compiled.invoke({}, run_id="run", thread_id="thread", tenant="acme") + plan, = [e for e in trace.read_events() if e.phase == "plan"] + end, = [e for e in trace.read_events() if e.phase == "end"] + assert plan.cost_usd == end.cost_usd == pytest.approx(0.003) + assert plan.tokens == end.tokens == compiled.last_run.meter.tokens + assert_price(trace, 0.003) + + +def test_nested_planner_budget_failure_preserves_price_and_stops_more_calls(tmp_path): + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = ScriptedChatModel(responses=['{"nodes": []}'] * 2, cost_usd=0.003) + planner = PlannerNode(model, trace=trace, structured=False) + graph = GraphARC(State, name="g", trace=trace) + + def body(state, ctx): + planner.propose("plan", ctx) + planner.propose("more", ctx) + return {} + + graph.add_node("planner", body, writes=set()) + graph.add_edge(START, "planner") + graph.add_edge("planner", END) + with pytest.raises(BudgetExceeded): + graph.compile().invoke({}, run_id="run", thread_id="thread", tenant="acme", + budget=Budget(max_tokens=1)) + assert model.call_count == 1 + assert_price(trace, 0.003) + + +@pytest.mark.asyncio +async def test_failed_parallel_node_is_not_billed_for_its_siblings(tmp_path): + trace = TraceRecorder(tmp_path / "trace.jsonl") + graph = GraphARC(State, name="g", trace=trace) + first = asyncio.Event() + second = asyncio.Event() + models = [ScriptedChatModel(responses=["done"], cost_usd=p) for p in (0.002, 0.005)] + + async def bad(state): + await models[0].ainvoke("first") + first.set() + await second.wait() + raise ValueError("failed after sibling spent") + + async def good(state): + await first.wait() + await models[1].ainvoke("second") + second.set() + return {} + + for name, body in [("bad", bad), ("good", good)]: + graph.add_node(name, body, writes=set()) + graph.add_edge(START, name) + graph.add_edge(name, END) + with pytest.raises(ValueError): + await graph.compile().ainvoke({}, run_id="run", thread_id="thread", tenant="acme") + terminal, = [e for e in trace.read_events() if e.phase == "error"] + assert terminal.tokens == models[0].last_usage["total_tokens"] + assert terminal.cost_usd == pytest.approx(0.002) + assert_price(trace, 0.007) + + +def test_a_failed_planner_call_cannot_reuse_an_earlier_calls_price(tmp_path): + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = ScriptedChatModel(responses=['{"nodes": []}'], cost_usd=0.003) + planner = PlannerNode(model, trace=trace, structured=False) + ctx = context() + assert planner.propose("plan", ctx).ok + assert not planner.propose("no more replies", ctx).ok + first, failed = trace.read_events() + assert first.cost_usd == pytest.approx(0.003) + assert failed.cost_usd is None + assert failed.model is None + assert failed.tokens == 0 + assert_price(trace, 0.003) + + +def test_unpriced_failed_node_remains_unpriced_unless_a_rate_is_supplied(tmp_path): + trace = TraceRecorder(tmp_path / "trace.jsonl") + model = ScriptedChatModel(responses=["done"]) + graph = GraphARC(State, name="g", trace=trace) + + def body(state): + model.invoke("work") + raise ValueError("unpriced failure") + + graph.add_node("worker", body, writes=set()) + graph.add_edge(START, "worker") + graph.add_edge("worker", END) + with pytest.raises(ValueError): + graph.compile().invoke({}, run_id="run") + terminal, = [e for e in trace.read_events() if e.phase == "error"] + assert terminal.cost_usd is None + cost = attribute(trace, "run") + assert cost.recorded_cost_usd is None + assert cost.unpriced_tokens == terminal.tokens > 0 + estimate = attribute(trace, "run", rates=RateCard(default=2.0)) + assert estimate.recorded_cost_usd is None + assert estimate.estimated_cost_usd == pytest.approx(terminal.tokens / 1000 * 2) diff --git a/tests/test_tenant_cost.py b/tests/test_tenant_cost.py index e4c3e7e..4d0123c 100644 --- a/tests/test_tenant_cost.py +++ b/tests/test_tenant_cost.py @@ -100,8 +100,9 @@ def test_error_events_keep_the_tenant_and_spend(tmp_path): cost = tenant_cost(trace, "alpha") assert cost.runs[0].errors == 1 assert cost.recorded_cost_usd == attribute(trace, "failed").recorded_cost_usd + assert cost.recorded_cost_usd == pytest.approx(0.002) assert cost.tokens > 0 - assert cost.unpriced_tokens == cost.tokens + assert cost.unpriced_tokens == 0 def test_tenants_partition_runs_without_double_counting_substeps(tmp_path): @@ -308,9 +309,10 @@ def test_saved_plan_execution_uses_the_current_explicit_tenant( assert {e.tenant for e in events if e.run_id in old_runs} == {"planning"} -def test_governed_planning_and_rounds_share_one_tenant_bill(tmp_path): +@pytest.mark.parametrize("price", [None, 0.003]) +def test_governed_planning_and_rounds_share_one_tenant_bill(tmp_path, price): trace = TraceRecorder(tmp_path / "trace.jsonl") - model = ScriptedChatModel(responses=plan_research.scripted_planner_replies(), cost_usd=0.003) + model = ScriptedChatModel(responses=plan_research.scripted_planner_replies(), cost_usd=price) loop = plan_research.build_loop(model, trace=trace) result = loop.run("explain elevated latency", run_id="loop", tenant="research") events = trace.read_events() @@ -328,14 +330,18 @@ def test_governed_planning_and_rounds_share_one_tenant_bill(tmp_path): assert any(r.executed for r in result.rounds) cost = tenant_cost(trace, "research", rates=RateCard(default=2.0)) assert [r.run_id for r in cost.runs] == ["loop"] - # Planner events currently carry tokens, not a provider price. That - # missing price must remain an estimate, even when the model has one. - assert cost.recorded_cost_usd is None assert cost.tokens == attribute(trace, "loop").tokens > 0 - assert cost.estimated_cost_usd == pytest.approx(cost.tokens * 2.0 / 1000) unpriced = tenant_cost(trace, "research") - assert unpriced.unpriced_tokens == unpriced.tokens - assert unpriced.complete is False + if price is None: + assert cost.recorded_cost_usd is None + assert cost.estimated_cost_usd == pytest.approx(cost.tokens * 2.0 / 1000) + assert unpriced.unpriced_tokens == unpriced.tokens + assert unpriced.complete is False + else: + assert cost.recorded_cost_usd == pytest.approx(model.call_count * price) + assert cost.estimated_cost_usd == 0.0 # only zero-token deterministic nodes + assert unpriced.unpriced_tokens == 0 + assert unpriced.complete is True def test_unlabelled_planner_spend_is_also_priced_or_marked_incomplete(tmp_path):