Add explicit zero-cost declarations and cost-source reporting to the roadmap
This commit is contained in:
@@ -8,9 +8,11 @@ Allow consumers to select execution observation once for an application-owned
|
|||||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||||
concurrent, and multi-engine calls using that operation's context should
|
concurrent, and multi-engine calls using that operation's context should
|
||||||
participate automatically. Include optional provider-reported cost alongside
|
participate automatically. Include optional provider-reported cost alongside
|
||||||
token usage, with built-in cost support initially limited to OpenRouter. Extend
|
token usage, with built-in cost decoding initially limited to OpenRouter and an
|
||||||
OpenRouter and vLLM token reporting with reasoning counts, documented cache reads
|
explicit backend zero-cost fallback. Distinguish reported and configured cost
|
||||||
and writes, and a shared derived token cache-hit ratio.
|
sources in individual and aggregate metadata. Extend OpenRouter and vLLM token
|
||||||
|
reporting with reasoning counts, documented cache reads and writes, and a shared
|
||||||
|
derived token cache-hit ratio.
|
||||||
|
|
||||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||||
could use this facility to report token usage for an entire command or pipeline.
|
could use this facility to report token usage for an entire command or pipeline.
|
||||||
@@ -31,8 +33,9 @@ PromptKit v0.24.0 already provides:
|
|||||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||||
coordinator-selected continuation runs.
|
coordinator-selected continuation runs.
|
||||||
|
|
||||||
The missing capabilities are contextual observer selection, provider-reported
|
The missing capabilities are contextual observer selection, cost accounting and
|
||||||
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
|
source metadata, an explicit zero-cost declaration, reasoning-token counts, and
|
||||||
|
a presence-aware cache-hit helper.
|
||||||
OpenRouter cache writes also need their documented nested response mapping;
|
OpenRouter cache writes also need their documented nested response mapping;
|
||||||
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
||||||
Applications currently have to enroll each request explicitly, and PromptKit
|
Applications currently have to enroll each request explicitly, and PromptKit
|
||||||
@@ -186,11 +189,12 @@ conventions. Preserve the existing shared behavior documented in the
|
|||||||
transient retries.
|
transient retries.
|
||||||
|
|
||||||
This selection supplies no default endpoint, model, API-key environment variable,
|
This selection supplies no default endpoint, model, API-key environment variable,
|
||||||
or sampling settings. It does not interpret monetary cost or provide a default
|
or sampling settings. It does not decode provider-reported monetary cost or
|
||||||
authentication-probe adapter. Without an explicit authentication adapter, report
|
provide a default authentication-probe adapter. The backend zero-cost declaration
|
||||||
the existing unsupported diagnostic outcome rather than assuming authentication
|
below remains available independently of provider selection. Without an explicit
|
||||||
is inapplicable. Existing reachability and inference diagnostics and explicit
|
authentication adapter, report the existing unsupported diagnostic outcome rather
|
||||||
authentication adapters remain available.
|
than assuming authentication is inapplicable. Existing reachability and inference
|
||||||
|
diagnostics and explicit authentication adapters remain available.
|
||||||
|
|
||||||
"Lowest common denominator" identifies the shared protocol foundation; it does
|
"Lowest common denominator" identifies the shared protocol foundation; it does
|
||||||
not remove existing optional features or promise every backend supports them.
|
not remove existing optional features or promise every backend supports them.
|
||||||
@@ -211,9 +215,11 @@ thinking, or translate model-specific controls based on the provider label.
|
|||||||
No separate generation protocol is needed; vLLM documents an
|
No separate generation protocol is needed; vLLM documents an
|
||||||
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
|
||||||
|
|
||||||
The token mappings are specified below. Monetary cost remains unknown, not zero,
|
The token mappings are specified below. vLLM supplies no provider-reported cost;
|
||||||
including when vLLM is locally hosted. This provider adds no price estimator,
|
cost remains unknown unless the backend explicitly declares the zero-cost
|
||||||
server administration, metrics collection, or automatic capability/version probe.
|
fallback below. Local hosting alone does not select that fallback. This provider
|
||||||
|
adds no price estimator, server administration, metrics collection, or automatic
|
||||||
|
capability/version probe.
|
||||||
|
|
||||||
### Smoke-Test Defaults And Explicit Overrides
|
### Smoke-Test Defaults And Explicit Overrides
|
||||||
|
|
||||||
@@ -419,7 +425,7 @@ For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
|
|||||||
are known and complete, even if cache writes are unknown. Output 840 with
|
are known and complete, even if cache writes are unknown. Output 840 with
|
||||||
reasoning 600 still represents 840 output tokens, not 1,440.
|
reasoning 600 still represents 840 output tokens, not 1,440.
|
||||||
|
|
||||||
## Optional Provider-Reported Cost
|
## Optional Cost Reporting
|
||||||
|
|
||||||
### Initial Provider Support
|
### Initial Provider Support
|
||||||
|
|
||||||
@@ -433,14 +439,54 @@ top-level charge; those describe different facts and could double-count cost.
|
|||||||
For the built-in client, select this interpretation through `provider: openrouter`,
|
For the built-in client, select this interpretation through `provider: openrouter`,
|
||||||
regardless of backend identity or namespace. Do not infer USD semantics from an
|
regardless of backend identity or namespace. Do not infer USD semantics from an
|
||||||
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
arbitrary backend's field named `cost`, or add endpoint discovery or verification.
|
||||||
Built-in cost interpretation is an extension unique to `openrouter` initially;
|
Built-in provider-reported cost interpretation is an extension unique to
|
||||||
`openai-compatible` and `vllm` leave cost unknown even if a response contains
|
`openrouter` initially; `openai-compatible` and `vllm` do not interpret
|
||||||
`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains
|
`usage.cost`. Without an applicable zero-cost declaration, their built-in cost
|
||||||
provider-neutral after the model-client boundary supplies interpreted USD facts.
|
remains unknown. No separate provider cost-decoding selector is needed.
|
||||||
|
Aggregation remains provider-neutral after the model-client boundary supplies
|
||||||
|
interpreted USD facts.
|
||||||
|
|
||||||
Use the cost included in the normal completion response. No additional usage
|
Use the cost included in the normal completion response. No additional usage
|
||||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||||
|
|
||||||
|
### Explicit Backend Zero-Cost Declaration
|
||||||
|
|
||||||
|
Add optional Boolean backend metadata `zero_cost`, independent of `provider`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
promptkit:
|
||||||
|
backends:
|
||||||
|
- id: local-inference
|
||||||
|
provider: vllm
|
||||||
|
endpoint: http://localhost:8000/v1
|
||||||
|
zero_cost: true
|
||||||
|
```
|
||||||
|
|
||||||
|
For each dispatched attempt, use a usable provider/model-client-reported cost
|
||||||
|
first, including an explicitly reported zero. If no usable reported cost is
|
||||||
|
available and the effective backend has `zero_cost: true`, supply USD zero with
|
||||||
|
a configured-zero source. Omitted, null, or false leaves the fallback disabled;
|
||||||
|
it does not suppress a reported cost. Apply this policy to built-in and injected
|
||||||
|
client execution using the selected backend, regardless of its provider type.
|
||||||
|
|
||||||
|
The fallback also applies to dispatched attempts that fail, are canceled, or
|
||||||
|
produce no decodable response. It is an operator declaration, not a provider
|
||||||
|
observation. Non-dispatched work contributes no cost. Do not infer this setting
|
||||||
|
from a local address, provider name, token counts, or missing credentials, and
|
||||||
|
do not install it automatically on any backend. Without a selected backend
|
||||||
|
declaration, no configured-zero fallback applies.
|
||||||
|
|
||||||
|
Resolve the field through ordinary final ConfigKit composition and inheritance;
|
||||||
|
include it in typed backend helpers, configuration capture, and backend
|
||||||
|
inspection. Retain the effective policy through prepared execution without
|
||||||
|
reloading configuration. It is local accounting metadata, not an outbound
|
||||||
|
request parameter. Preserve explicit/null/omitted source provenance in inspection
|
||||||
|
rather than manufacturing an explicit false value for the disabled fallback.
|
||||||
|
|
||||||
|
This supports an explicit zero declaration, not configurable input/output or
|
||||||
|
cache-read/write token rates, fixed nonzero request charges, pricing tables, or
|
||||||
|
cost estimation. It does not alter credentials, generation, or token reporting.
|
||||||
|
|
||||||
### Monetary Representation
|
### Monetary Representation
|
||||||
|
|
||||||
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
||||||
@@ -453,9 +499,10 @@ OpenRouter backend; no provider-specific interface is needed.
|
|||||||
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
||||||
remain token counts. For run results, execution events, and group summaries,
|
remain token counts. For run results, execution events, and group summaries,
|
||||||
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
||||||
`Complete bool`, following the existing observed-count conventions. Successful
|
`Complete bool`, following the existing observed-count conventions, plus the
|
||||||
run results should expose cost even when no observer was registered; observers
|
source metadata below. Successful run results should expose cost even when no
|
||||||
also provide per-attempt and terminal cost, including available facts on failure.
|
observer was registered; observers also provide per-attempt and terminal cost,
|
||||||
|
including available facts on failure.
|
||||||
|
|
||||||
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
||||||
addition, and the public API lightweight. Do not round to cents or another fixed
|
addition, and the public API lightweight. Do not round to cents or another fixed
|
||||||
@@ -465,42 +512,75 @@ exact financial ledgers, while fixed-scale integers would require a selected
|
|||||||
precision and rounding policy. Neither is warranted for this reporting feature.
|
precision and rounding policy. Neither is warranted for this reporting feature.
|
||||||
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
||||||
|
|
||||||
|
### Cost-Source Metadata
|
||||||
|
|
||||||
|
Give the presence-aware cost value a `Source` discriminator with these meanings:
|
||||||
|
|
||||||
|
- `provider`: a cost supplied by the provider or injected model-client response.
|
||||||
|
This records the reporting path, not independent verification of a charge.
|
||||||
|
- `configured_zero`: zero supplied by the explicit backend fallback.
|
||||||
|
- `mixed`: an aggregate includes both provider and configured-zero contributions.
|
||||||
|
- Empty: no known contribution has a source. This covers unknown cost and the
|
||||||
|
known, complete zero aggregate when no attempts were dispatched.
|
||||||
|
|
||||||
|
Derive aggregate source from the union of known contribution sources, including
|
||||||
|
zero-valued contributions. Unknown contributions supply no source but affect
|
||||||
|
completeness. A provider-only subtotal can therefore be incomplete without being
|
||||||
|
`mixed`. Carry this metadata consistently through attempt events, terminal
|
||||||
|
events, successful results, and group summaries. Reuse existing aggregation and
|
||||||
|
terminal deduplication rather than adding a parallel collector or cost history.
|
||||||
|
|
||||||
|
Consumers can distinguish reported charges, configured zeros, and mixed totals
|
||||||
|
without treating configured values as provider reports. Source metadata is
|
||||||
|
separate from configuration inspection and does not require provenance traversal
|
||||||
|
during execution.
|
||||||
|
|
||||||
### Presence, Aggregation, And Failures
|
### Presence, Aggregation, And Failures
|
||||||
|
|
||||||
- An absent or null cost is unknown. Explicit numeric zero is known zero.
|
- An absent or null provider cost is unknown unless the backend's explicit
|
||||||
A backend that reports no cost remains unknown even when a consumer knows it
|
zero-cost fallback supplies it. A reported numeric zero remains known zero
|
||||||
has no external charge.
|
with a provider source, even when the fallback is enabled.
|
||||||
- On a dispatched attempt, a usable reported cost is known and complete for
|
- On a dispatched attempt, a usable reported cost or configured-zero fallback
|
||||||
that attempt. Cost completeness is independent of token-field completeness.
|
is known and complete for that attempt. Cost completeness is independent of
|
||||||
|
token-field completeness and does not assert provider reporting.
|
||||||
- Sum known costs across dispatched attempts, including initial generations,
|
- Sum known costs across dispatched attempts, including initial generations,
|
||||||
retries, and repairs. `Known` means at least one attempt reported cost;
|
retries, and repairs. `Known` means at least one attempt has an available cost
|
||||||
`Complete` means every dispatched attempt did. Unknown values contribute no
|
from either source; `Complete` means every dispatched attempt does under this
|
||||||
amount, but make the total incomplete. With no dispatched attempts, follow
|
policy. Unknown values contribute no amount, but make the total incomplete.
|
||||||
the existing known, complete zero convention.
|
With no dispatched attempts, follow the existing known, complete zero
|
||||||
|
convention, with an empty source.
|
||||||
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
||||||
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
||||||
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
||||||
from OpenRouter mixed with unreported local-backend cost is incomplete.
|
from OpenRouter mixed with unreported local-backend cost is incomplete unless
|
||||||
|
those local attempts have an explicit configured-zero fallback. Combine
|
||||||
|
cost-source metadata as described above without changing token accounting.
|
||||||
- Keep valid cost observed from a decoded successful envelope when a later
|
- Keep valid cost observed from a decoded successful envelope when a later
|
||||||
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
||||||
error bodies continue to supply no usage. Cancellation or failure may leave
|
error bodies continue to supply no provider usage, but the configured-zero
|
||||||
incomplete reporting even when billable upstream work occurred.
|
fallback still applies to dispatched attempts. Without that fallback,
|
||||||
|
cancellation or failure may leave incomplete cost reporting even when billable
|
||||||
|
upstream work occurred.
|
||||||
- Treat an unusable optional cost value, including a nonnumeric value or one
|
- Treat an unusable optional cost value, including a nonnumeric value or one
|
||||||
not representable as a finite `float64`, as unknown rather than making an
|
not representable as a finite `float64`, as unknown rather than making an
|
||||||
otherwise valid generation fail. This does not relax existing response framing
|
otherwise valid generation fail; the zero-cost fallback can then apply. This
|
||||||
or content validation. An injected client likewise cannot supply a nonfinite
|
does not relax existing response framing or content validation. An injected
|
||||||
usable cost.
|
client likewise cannot supply a nonfinite usable cost.
|
||||||
|
|
||||||
Completeness describes coverage of reported values, not verification of the
|
Completeness describes coverage by available reported or explicitly configured
|
||||||
provider's billing. Preserve reported usable amounts without pricing inference
|
cost, not verification of the provider's billing or proof that every provider
|
||||||
or reconciliation.
|
reported a charge. Preserve usable reported amounts without pricing inference or
|
||||||
|
reconciliation.
|
||||||
|
|
||||||
For a successful run, result cost and terminal-event cost describe the same
|
For a successful run, result cost and terminal-event cost describe the same
|
||||||
dispatched attempts and must agree on value, presence, and completeness. Result
|
dispatched attempts and must agree on value, presence, completeness, and source.
|
||||||
cost must also be the same with observation enabled or disabled. For example,
|
Result cost must also be the same with observation enabled or disabled. For
|
||||||
an unreported failed attempt followed by a successful attempt reporting USD
|
example, an unreported failed attempt followed by a successful attempt reporting USD
|
||||||
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
|
0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider
|
||||||
accounting through repairs and ordinary or prepared execution.
|
source when no zero-cost fallback applies. Apply the same accounting through
|
||||||
|
repairs and ordinary or prepared execution. An invocation with fully reported
|
||||||
|
OpenRouter charges and configured-zero local calls can have a complete subtotal
|
||||||
|
with a mixed source, without claiming all providers reported costs.
|
||||||
|
|
||||||
Group cost completeness concerns the terminal observations included in its
|
Group cost completeness concerns the terminal observations included in its
|
||||||
subtotal. If two completed runs reported all their costs but a third observed
|
subtotal. If two completed runs reported all their costs but a third observed
|
||||||
@@ -523,7 +603,8 @@ application-specific concepts to PromptKit.
|
|||||||
Smoke diagnostics retain their separate report/evidence model and are not
|
Smoke diagnostics retain their separate report/evidence model and are not
|
||||||
enrolled in ordinary execution observation by this change. Reasoning-text
|
enrolled in ordinary execution observation by this change. Reasoning-text
|
||||||
capture, changes to reasoning or caching request behavior, additional built-in
|
capture, changes to reasoning or caching request behavior, additional built-in
|
||||||
cost providers, multiple currencies, currency conversion, pricing estimates,
|
cost providers, configurable token-rate estimation, fixed nonzero request
|
||||||
|
pricing, multiple currencies, currency conversion, pricing estimates,
|
||||||
billing reconciliation, persistent history, and logging infrastructure are
|
billing reconciliation, persistent history, and logging infrastructure are
|
||||||
outside this scope.
|
outside this scope.
|
||||||
|
|
||||||
@@ -537,6 +618,11 @@ outside this scope.
|
|||||||
- Add `provider` to the backend configuration model, typed source helpers,
|
- Add `provider` to the backend configuration model, typed source helpers,
|
||||||
capture, inspection, and execution plumbing. Update the bundled OpenRouter
|
capture, inspection, and execution plumbing. Update the bundled OpenRouter
|
||||||
backend to select it explicitly. Retain the selection through prepared runs.
|
backend to select it explicitly. Retain the selection through prepared runs.
|
||||||
|
- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and
|
||||||
|
prepared-execution plumbing. Apply the fallback outside provider-specific
|
||||||
|
decoding so it works with built-in and injected clients. Verify composition
|
||||||
|
and inheritance, omitted/null/false behavior, explicit overrides, provider
|
||||||
|
independence, retained inspection provenance, and absence from provider bodies.
|
||||||
- Consolidate provider-specific response and authentication-probe behavior in
|
- Consolidate provider-specific response and authentication-probe behavior in
|
||||||
`internal/llm`; keep coordination and accounting provider-neutral. Cover
|
`internal/llm`; keep coordination and accounting provider-neutral. Cover
|
||||||
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
||||||
@@ -561,7 +647,8 @@ outside this scope.
|
|||||||
recognized acceptance, rejection, unrecognized responses, provider defaults,
|
recognized acceptance, rejection, unrecognized responses, provider defaults,
|
||||||
and explicit kind/endpoint overrides, including `none` and adapter selection
|
and explicit kind/endpoint overrides, including `none` and adapter selection
|
||||||
without a provider declaration. Verify that ordinary generation performs no
|
without a provider declaration. Verify that ordinary generation performs no
|
||||||
diagnostic requests and leaves monetary cost unknown.
|
diagnostic requests and leaves monetary cost unknown unless the backend
|
||||||
|
explicitly declares the zero-cost fallback.
|
||||||
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
|
- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache
|
||||||
details, including responses representative of servers without detail
|
details, including responses representative of servers without detail
|
||||||
reporting enabled. Verify `created_cache_tokens` mapping independently of
|
reporting enabled. Verify `created_cache_tokens` mapping independently of
|
||||||
@@ -575,6 +662,16 @@ outside this scope.
|
|||||||
conversions, successful run results, attempt and terminal events, and existing
|
conversions, successful run results, attempt and terminal events, and existing
|
||||||
group summaries. Keep result cost available without registering an observer.
|
group summaries. Keep result cost available without registering an observer.
|
||||||
Preserve existing token fields and accounting.
|
Preserve existing token fields and accounting.
|
||||||
|
- Carry cost-source metadata through results, events, conversions, and group
|
||||||
|
snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs,
|
||||||
|
including zero-valued reported contributions and empty/non-dispatched cases.
|
||||||
|
Verify source union and terminal deduplication independently of cost value.
|
||||||
|
- Verify that usable reported cost wins over `zero_cost: true`, and missing,
|
||||||
|
null, or unusable reported cost permits fallback. Cover dispatched failures,
|
||||||
|
cancellation, undecodable responses, retries, repairs, ordinary/prepared
|
||||||
|
execution, and observer-enabled/disabled results. Verify that mixed OpenRouter
|
||||||
|
and declared zero-cost local calls can produce complete cost totals while
|
||||||
|
token completeness remains independent.
|
||||||
- Verify that successful-result cost agrees with terminal-event cost and is
|
- Verify that successful-result cost agrees with terminal-event cost and is
|
||||||
unchanged when observation is disabled, including the unreported-retry example
|
unchanged when observation is disabled, including the unreported-retry example
|
||||||
above. Verify group snapshots with fully reported completed runs alongside a
|
above. Verify group snapshots with fully reported completed runs alongside a
|
||||||
@@ -595,15 +692,18 @@ outside this scope.
|
|||||||
turn telemetry into an exact-decimal guarantee.
|
turn telemetry into an exact-decimal guarantee.
|
||||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||||
aggregation example to the consumer observation guide, including cost
|
aggregation example to the consumer observation guide, including cost
|
||||||
availability, units, and completeness. Document the reasoning/cache fields and
|
availability, units, completeness, and source. Explain that configured zeros
|
||||||
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
|
can make cost complete without provider reporting. Document the reasoning/cache
|
||||||
without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter
|
fields and cache-hit helper in their owning GoDoc and illustrate input/output
|
||||||
cost decoding in the
|
breakdowns without double-counting. Document OpenRouter/vLLM token decoding and
|
||||||
|
OpenRouter cost decoding in the
|
||||||
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
[integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||||
and update internal observation and model-client documentation for changed flows.
|
and update internal observation and model-client documentation for changed flows.
|
||||||
Document `provider`, its smoke-test defaults and override semantics in the
|
Document `provider`, its smoke-test defaults and override semantics in the
|
||||||
framework format reference and relevant public GoDoc; update inspection and
|
framework format reference and relevant public GoDoc; update inspection and
|
||||||
diagnostic documentation with the implemented behavior. Include the diagnostic
|
diagnostic documentation with the implemented behavior. Document `zero_cost`
|
||||||
|
and its fallback precedence, inspection behavior, and prepared lifetime in the
|
||||||
|
relevant format, GoDoc, and consumer owners. Include the diagnostic
|
||||||
migration in release notes. Include vLLM usage availability, server reporting
|
migration in release notes. Include vLLM usage availability, server reporting
|
||||||
settings, the meaning of cache creation, and the limits of its authentication
|
settings, the meaning of cache creation, and the limits of its authentication
|
||||||
diagnostic in the relevant consumer and integration documentation.
|
diagnostic in the relevant consumer and integration documentation.
|
||||||
@@ -616,12 +716,13 @@ Organize the eventual implementation plan around two workstreams:
|
|||||||
and nested execution boundaries and the existing group collector.
|
and nested execution boundaries and the existing group collector.
|
||||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||||
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
|
smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token
|
||||||
extensions and provider-reported cost through decoding, accounting, public
|
extensions, provider-reported cost, and configured-zero fallback through
|
||||||
results, events, and group summaries, plus the derived cache-hit helper.
|
decoding, accounting, public results, events, and group summaries, plus the
|
||||||
Include the corresponding configuration and API documentation.
|
derived cache-hit helper. Include the corresponding configuration and API
|
||||||
|
documentation.
|
||||||
|
|
||||||
The second area is more extensive than the context helper. Keep backend provider
|
The second area is more extensive than the context helper. Keep backend provider
|
||||||
configuration, diagnostic consolidation, token/cache reporting, and cost
|
configuration, diagnostic consolidation, token/cache reporting, and cost/source
|
||||||
accounting in separately reviewable implementation units rather than one large
|
accounting in separately reviewable implementation units rather than one large
|
||||||
implementation prompt.
|
implementation prompt.
|
||||||
Avoid coupling delivery of contextual observation to provider cost support.
|
Avoid coupling delivery of contextual observation to provider cost support.
|
||||||
@@ -637,8 +738,9 @@ contract.
|
|||||||
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
||||||
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
||||||
overflowing value fail decoding of the entire response. Decode the optional
|
overflowing value fail decoding of the entire response. Decode the optional
|
||||||
field separately so an unusable cost becomes unknown without changing existing
|
field separately so an unusable provider cost becomes unknown before applying
|
||||||
framing or content validation. This needs no general numeric-conversion layer.
|
any configured-zero fallback, without changing existing framing or content
|
||||||
|
validation. This needs no general numeric-conversion layer.
|
||||||
|
|
||||||
Use the completion criteria above to select focused offline checks for each
|
Use the completion criteria above to select focused offline checks for each
|
||||||
unit. This roadmap owns target behavior; a separate implementation plan owns
|
unit. This roadmap owns target behavior; a separate implementation plan owns
|
||||||
|
|||||||
Reference in New Issue
Block a user