Metadata-Version: 2.4
Name: os3-compare
Version: 0.1.0
Summary: OS3 comparison SDK (RL-04): send a request through a preference experiment, show both answers blind, record the end user's vote.
License: Proprietary
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: httpx<1,>=0.27

# os3-compare

The optional comparison SDK for OS3 preference experiments (RL-04). httpx only.

```bash
pip install https://api.os3robotics.com/downloads/os3_compare-0.1.0-py3-none-any.whl  # or ./compare_sdk
```

Ordinary gateway traffic never changes. For requests you want your users to compare, your
backend calls `Client.compare(experiment_id, request)` instead of `/v1/chat/completions`:

- Most requests come back with one answer (`compared` is false and `completion` is the usual
  chat completion).
- For the share of requests the experiment compares, OS3 asked both your current model and the
  experiment's candidate. You get the two answers blind, as `a` and `b` in random order, with
  the question to ask and the five outcomes the user may pick: `a`, `b`, `tie`, `both_bad`,
  `cannot_judge`, plus optional reasons (`accuracy`, `completeness`, `clarity`, `relevance`,
  `other`).
- Pass the user's pick back with `Client.feedback(c.id, c.feedback_token, outcome, reason)`.
  When the user writes the answer they wanted (say both were wrong), pass it as `correction`
  (not with `cannot_judge`). A comparison takes one vote from its end user; a replay is
  refused with 409.

```python
from os3_compare import Client

client = Client(api_key="os3_gw_...", base_url="https://gateway.os3robotics.com/v1")
c = client.compare(experiment_id, {"model": "gpt-6-luna", "messages": messages}, user="u-42")
if c.compared:
    show(c.question, c.a["message"], c.b["message"], c.outcomes)
    client.feedback(c.id, c.feedback_token, "b", reason="completeness")
else:
    show(c.completion["choices"][0]["message"])
```

Later, when you learn how the request turned out (a ticket resolved, a sale made), report it
with `Client.outcome(c.id, "resolved", 1, observed_at)`. OS3 attributes it to what the user was
shown (both answers, or your current model's alone) in `GET /outcomes`. Name the outcome as
your task's contract declares it (`outcomes`: name, `binary` or `numeric`, and the window in
days after the request within which it counts). The same report again is not counted twice.

The same calls without the SDK (the gateway's OS3 extension, `chat` scope):

```
POST /v1/comparisons       {"experiment_id", "request", "user"?}
POST /v1/feedback          {"comparison_id", "token", "outcome", "reason"?, "correction"?}
POST /v1/feedback/outcomes {"request_id" | "trace_id", "outcome", "value", "observed_at"}
```

`/v1/feedback/outcomes` takes any gateway request's id (its response's `x-os3-request-id`), not
only a comparison's.

The key is a gateway key of the experiment's workspace. Keep `feedback_token` on your backend,
or hand it only to the session that shows the comparison. Your team can vote on the same
comparisons in the portal (*Data › Traces › Comparisons*); every vote and correction is
exported with its provenance by `GET /preference/experiments/{id}/votes` (`os3_platform/review/preference.py`).
