# Backtest data

The [Python SDK](https://uselayer.sh/docs/sdk.md) backtests on order-book history you supply. This page describes the format that history takes (the SDK's **events**) and how `import_events()` turns files you already have into it.

Everything runs on your machine. Layer never sees your files.

## Import a file

```python
from uselayer import Client, import_events

data = import_events(
    "ticks.csv",
    venue="kalshi",
    columns={"time": "ts", "market": "ticker", "bid": "yes_bid", "bid_size": "yes_bid_qty",
             "ask": "yes_ask", "ask_size": "yes_ask_qty"},
    price_scale=0.01,  # the file has cents
)
print(data.report.summary())
Client(mode="backtest", books=data).replay(on_book)
```

`import_events()` reads these formats. It guesses the format from the file when you leave `format=` out.

| `format=` | What it reads |
|---|---|
| `csv`, `parquet` | One row per event. `columns=` maps the SDK's field names to yours. Parquet needs `pip install 'uselayer[parquet]'` |
| `jsonl` | The SDK's own format, one event per line, as `save_events()` writes it |
| `polymarket_us` | Messages from Polymarket US's markets WebSocket (`marketData`, `trade`, `marketDataLite`) |
| `polymarket` | Messages from polymarket.com's market channel (`book`, `price_change`, `last_trade_price`, `tick_size_change`, `market_resolved`) |
| `kalshi` | Messages from Kalshi's WebSocket (`orderbook_snapshot`, `orderbook_delta`, `trade`, `market_lifecycle_v2`) |
| `pmxt` | PMXT's hourly Polymarket order-book Parquet files, both of their schemas |

For the three WebSocket formats, put one message per line. You can wrap each one as `{"received_at": "<when you got it>", "message": <the message>}` to keep your receive time.

To join consecutive files, such as hours of PMXT, pass a list: `import_events(["hour1.parquet", "hour2.parquet"], markets=[...])`.

### Options

| Option | What it does |
|---|---|
| `venue`, `market` | For CSV and Parquet: the venue and market of every row, when the file has no column for them |
| `columns` | For CSV and Parquet: `{sdk_field: your_column}` (fields below) |
| `kind` | What every row is, when the file has no `kind` column |
| `price_scale` | Multiply prices by this, e.g. `0.01` for cents |
| `markets` | Keep only these markets. Required for `pmxt`, whose hours hold every Polymarket market: pass outcome token ids or condition ids |
| `tokens` | For `polymarket` and `pmxt`: `{token_id: (market, "yes" or "no")}` names a market and folds its NO token into the YES book. Without it, each token is its own market, with the token as YES |
| `tick_size` | The tick to check prices against: one number, or `{market: tick}` |
| `max_gap_s` | Report silences longer than this. Default: 10 times the data's median spacing, and at least 60 s |
| `strict` | `True` (the default) raises when the checker finds errors. `False` imports anyway; read `data.report` |

## The checker

Every import is checked before a backtest can use it, so a bad file can't quietly give wrong results.

| Problem | Severity | Means |
|---|---|---|
| `crossed` | error | A book's best bid is at or above its best ask |
| `impossible` | error | A price not between $0 and $1, or a size that isn't a positive number. Often cents read as dollars |
| `off_tick` | error | A price that isn't a multiple of the market's tick |
| `unreadable` | error | A row or line that can't be read: no time, bad JSON, an unknown kind |
| `sequence_gap` | error | Kalshi's message numbers skipped, so messages were lost and the book is wrong until the next snapshot |
| `gap` | warning | A silence much longer than the data's usual spacing, or a gap a recorder marked |
| `out_of_order` | warning | A row received (or stamped) earlier than the row before it for the same market |
| `stale_book` | warning | A full book re-sent with an old time, after newer data. It is skipped |
| `no_starting_book` | warning | A level change before the market's first full book. A replay skips it |
| `repaired` | warning | Polymarket book levels removed because the venue's own best bid and ask said they were gone |

Errors raise `VenueError` with code `bad_data`. Its message holds the summary, and `raw` holds the full report. Warnings never stop an import.

One token from a real PMXT hour (`polymarket_orderbook_2026-07-21T04.parquet`) reports:

```text
29766 events, 1 markets, 2026-07-21T04:00:00.095000+00:00 → 2026-07-21T04:59:59.952000+00:00.
warning gap ×1: no events for 2172 s, from 2026-07-21T04:09:00.161000+00:00 to 2026-07-21T04:45:11.982000+00:00.
warning no_starting_book ×6985: a level change for 1554… comes before any full book. (row 7139984)
warning repaired ×139: removed 139 book level(s) that the venue's own best bid/ask showed were already gone.
warning stale_book ×10: a re-sent book from 2026-07-21T04:06:28.100000+00:00 arrived after newer data; skipped. (row 7149344)
```

The 36-minute gap is a stretch missing from that hour's file. Through a gap the checker finds, a replay keeps the last book, so read results from that stretch with care. Through a recorder's `gap` event (below), the market has no book until the next one.

You can check any events yourself, for example ones you built in code: `check_events(events).summary()`.

## The event format

Every event is a JSON object with a `kind`. Prices are dollars from 0 to 1 for the market's **YES** side. NO is the mirror: a NO bid at 40¢ is a YES ask at 60¢. Times are ISO 8601 in UTC.

- `as_of` is always the **venue's** time, except on a recorder `gap`.
- `received_at` is when your machine got the event, if it was recorded. It is optional.
- A backtest replays events in `as_of` order.

### `book`: a full order book

```json
{"kind": "book", "venue": "kalshi", "market": "KXFED-26SEP-T4.25",
 "bids": [{"price": 0.41, "size": 120}], "asks": [{"price": 0.43, "size": 80}],
 "as_of": "2026-09-01T14:00:00Z", "received_at": null, "source": "recorded"}
```

`bids` run best first (highest), and `asks` run best first (lowest). `size` is in contracts.

### `book_change`: one level changing

```json
{"kind": "book_change", "venue": "polymarket_us", "market": "aec-nfl-sample-2026-09-07",
 "book_side": "ask", "price": 0.56, "size": 40, "as_of": "2026-09-07T17:00:01Z"}
```

`size` is the level's new total. `0` removes the level. A backtest applies each change to the market's last book and replays the result as a book. Kalshi sends changes as amounts to add or subtract, and the importer turns those into totals.

### `trade`: a trade the venue printed

```json
{"kind": "trade", "venue": "kalshi", "market": "KXSAMPLE-26SEP21-T50", "price": 0.45, "size": 10,
 "trade_id": "t-1", "aggressor": "buy", "as_of": "2026-09-21T15:00:02.150Z"}
```

`aggressor` is what the taker did to YES: `buy` means they took the asks, `sell` means they hit the bids. It is `null` when unknown.

### `status`: the market opened, paused, closed or halted

```json
{"kind": "status", "venue": "kalshi", "market": "KXSAMPLE-26SEP21-T50", "status": "paused", "as_of": "2026-09-21T15:30:00Z"}
```

### `resolution`: how the market settled

```json
{"kind": "resolution", "venue": "kalshi", "market": "KXSAMPLE-26SEP21-T50", "outcome": "no", "as_of": "2026-09-21T16:00:00Z"}
```

`outcome` is `yes`, `no` or `void` (refunded).

### `gap`: the recorder was disconnected

```json
{"kind": "gap", "origin": "recorder", "venue": "polymarket_us", "market": "aec-nfl-sample-2026-09-07",
 "as_of": "2026-09-07T17:00:00Z", "until": "2026-09-07T17:00:12Z", "reason": "disconnected: ConnectionClosedError: …"}
```

`record_stream()` writes one for each market while it was disconnected. Both times are your machine's clock, since the venue sent nothing. In a backtest the market has no book from `as_of` until the next book after the gap.

### CSV and Parquet fields

Each row is one event. Map your columns to these fields with `columns=`, or name your columns after them.

| Field | Used for |
|---|---|
| `time` | The venue's time (or `received_at` alone, if the file has only that) |
| `received_at` | When you received it |
| `venue`, `market` | Which market (or pass `venue=` / `market=`) |
| `kind` | `book`, `book_change`, `trade`, `status` or `resolution`. Guessed from which fields are filled when left out |
| `bids`, `asks` | A full book: JSON levels, `[[price, size], ...]` |
| `bid`, `bid_size`, `ask`, `ask_size` | A top-of-book row, read as a one-level book |
| `book_side`, `price`, `size` | A level change (`bid` or `ask`; `buy` and `sell` work too) |
| `price`, `size`, `trade_id`, `aggressor` | A trade |
| `status` | A status row |
| `outcome` | A resolution row |

Times can be ISO 8601 strings or Unix numbers in seconds, milliseconds, microseconds or nanoseconds (the unit is read from the size of the number). A time without a zone is read as UTC.

## Venue notes

- **polymarket.com:** each outcome token has its own book. Pass `tokens=` to fold both tokens into one market. Old books re-sent on reconnects are skipped. Some level removals never reach the stream, so the importer drops levels that are better than the best bid and ask the venue stamps on each change. On PMXT's files this brings rebuilt books from 88–99% to about 99% agreement with the venue's own best bid and ask, and stops them crossing.
- **PMXT:** the files hold only condition ids and token ids, not slugs or results. PMXT's first schema (Feb to mid-April 2026) has no venue time, so its receive time is used.
- **Kalshi:** books list bids on both sides. A NO bid at p is a YES ask at 1 − p. A snapshot has no time of its own, so it takes the time Kalshi sent it.
- **Polymarket US:** every `marketData` message is the full book. A settlement comes only in `marketDataLite`'s `settlementPx`.
