Jianxin Shen
Systems

Trading Infrastructure

Most of my working life has gone into one question every trading system has to answer: what actually happened? A strategy says what position it wants, the broker says what was executed, and the books say what was recorded. Live failures tend to start in the gaps between those three records; the work is giving each one a single owner and reconciling them honestly while the market keeps moving.

This page is about ideas and method, not any employer's system. Its examples are constructed: the replay runs on made-up data with no account connection and illustrates its own rules, not production performance.

Switch a rule off →

What you can try

Which rule keeps the books true?

A constructed twelve-event session with five of the rules below as switches. Turn one off and watch, event by event, where the system's books and the broker's records part ways.

  • A repeated request, a fill delivered twice, a lost notification and a cancel that loses a race.
  • See both checks: whether the books agree with the broker, and whether the broker holds what the strategy wants.
  • Every state is computed in the browser from downloadable inputs and the same calculation module.

Three records that want to blur together

A strategy holds a target: the position it wants. An order asks for a trade, and the broker's records say what was actually executed. The books learn about fills through notifications and queries, and they can lag, miss one or hear the same one twice. Each record is true about something different. Code that lets one stand in for another eventually trades on a position that does not exist.

Constructed example · the replay's numbers

Three records of one position

A strategy wants two units. The broker has filled one. Its notification never arrived.

Target · owned by the PMS2

What the strategy wants to hold.

1

Still working. An unfinished order. Normal, not an error.

Execution · confirmed by the broker1

What actually traded, fill F1.

1

Missing fill. An error. Needs evidence, not a guess.

Books · built from received fills0

What the system has recorded.

Reconcile Compare the books with the broker at the same cutoff. A difference there is an error: fetch the fill history, record F1 once by its ID, compare again. Compare target with the books to see what is still to do.

The same three numbers drive the order replay linked above. Code that lets the target stand in for execution, or execution stand in for the books, trades on a position that does not exist.

So each record needs one owner. The portfolio side owns what a position means; the order side owns the broker mechanics: identities, acknowledgements, partial fills, cancels and rejects. In the usual vocabulary these are the PMS and the OMS, and they are kept apart because they change for different reasons.

A generic execution system

Who owns what

Intent travels down one path; updates come back up another. Each box names the state it alone may change.

Strategyowns decisions: the target it wants
One queue, one executorowns the order of events: one decision at a time
PMSowns what positions mean: targets, holdings from fills, the protection each holding needs
OMSowns order mechanics: internal and broker IDs, acknowledgements, partial fills, cancels, rejects, retries
Brokerowns execution: the record that confirms what traded
A generic layout for teaching, not the structure of any particular system. The PMS and OMS are separate because position meaning and broker mechanics change for different reasons; the single queue and the single update reader keep the order of events deterministic.

Orders and fills

The judgements I hold come in two groups, each followed by a constructed example. They are general enough to apply to most execution systems and specific enough to argue with. The first group is about an order's life: who names it, what counts as filled and what may be retried.

  • State ownershipMost live trading failures are state ownership failures: two parts of the system each believed they knew the position, and only one of them was right.
  • Intent firstAn order has an internal identity before the broker acknowledges it. Without one, a timeout leaves the system unable to say whether the order exists.
  • Fills, not intentQuantities come only from fills. Sending an exit does not make a position flat, and a system that treats it as flat will act as if it no longer holds what it still holds.
  • Partial fillsPartial fills are first-class state. They change exposure, the remaining quantity and every protective order attached to the position.
  • RetriesAn upstream retry must not become a duplicate order, so intent carries an idempotency key. Rejects are classified, and only the clearly retryable ones are retried, a bounded number of times.
  • BacktestBacktest and live trading share one interface. The venue changes; the logic does not.

Constructed example

One order, step by step

Choose a scenario and step through it. The row of states shows where the order is; the two panels show what each side of the system believes at that moment.

Scenario
  1. Planned
  2. Sent
  3. Working
  4. Partly filled
  5. Filled
  6. Cancel requested
  7. Cancelled
  8. Rejected
Step 1

OMS sees

PMS sees

Judgement

Partial fill, then cancel the rest

  1. The target becomes an order. The strategy wants +2. The PMS turns the gap between target and holding into an order plan, and the OMS gives the order its own identity before anything leaves the building. OMS: O-1 · buy 2 · no broker ID yet. PMS: target 2 · holding 0.
  2. Sent, not yet acknowledged. The order is in transit. If the connection drops now, O-1 is still the name the system can ask about. OMS: O-1 · in transit. PMS: target 2 · holding 0.
  3. The broker acknowledges. The broker's own ID arrives with its first update. From here on, cancels must use it. OMS: O-1 ↔ B-77 · working · 0 of 2. PMS: target 2 · holding 0.
  4. One unit fills. A partial fill is its own state. Exposure, the remaining quantity and the stop that protects the position all change now, not when the order finishes. OMS: B-77 · working · 1 of 2 · fill F-1. PMS: holding 1 from F-1 · stop sized to 1.
  5. The rest is cancelled, or is it?. The strategy no longer wants the second unit. A cancel request goes out, and until the broker confirms, the order is still working and could still fill. OMS: B-77 · cancel requested · 1 still working. PMS: holding 1 · still exposed to one more fill.
  6. The broker confirms. Only now is the remainder gone. The holding is 1, the target was 2, and that difference is an unfinished order, not an error in the books. OMS: B-77 · cancelled · 1 of 2 filled. PMS: target 2 · holding 1 · nothing working.

A reject, retried once

  1. A market order is planned. The PMS asks for +1 at market. The OMS names it and records the request's idempotency key. OMS: O-2 · buy 1 · key K-9. PMS: target 1 · holding 0.
  2. Sent. The order leaves for the broker. OMS: O-2 · in transit. PMS: target 1 · holding 0.
  3. Rejected: no price available. The reject is classified. This one is transient and known to be safe to retry; most rejects are neither. OMS: O-2 · rejected · transient. PMS: target 1 · holding 0.
  4. One bounded retry. The retry is a new order with a new identity, tied to the same request. If the upstream sends the request again, the key stops a third order from appearing. OMS: O-3 · retry of K-9 · in transit. PMS: target 1 · holding 0.
  5. Filled. The holding comes from the fill. Had the retry failed too, the next step would have been an alert to a person, not another attempt. OMS: O-3 · filled · fill F-2. PMS: target 1 · holding 1 from F-2.

Exit sent is not flat

  1. A holding with its protection. The position holds 1. A target and a stop are both working; whichever fills cancels the other. OMS: target T-4 working · stop S-4 working. PMS: holding 1 · protected.
  2. The target partly fills. Half of the target fills. Both exits keep working, and the stop must shrink to what is still held. OMS: T-4 · 0.5 of 1 · S-4 resized to 0.5. PMS: holding 0.5.
  3. An exit for the rest is sent. The strategy closes the position early and sends an exit. It is tempting to mark the position flat here. OMS: exit X-5 · working. PMS: holding 0.5 · not flat.
  4. The exit fills. Now the holding is 0, because a fill says so. OMS: X-5 · filled · fill F-3. PMS: holding 0 from fills.
  5. Leftover protection. The target and stop are still working with nothing left to protect. Orphaned protection is cancelled, and the cancel waits for confirmation like any other. OMS: T-4, S-4 · cancel requested. PMS: holding 0 · orphan protection.
  6. Clean. Nothing is held and nothing is working. Holding and protection agree again. OMS: T-4, S-4 · cancelled. PMS: holding 0 · protection 0.
A constructed example with made-up identifiers. It shows the judgements on this page as behaviour; it is not a trace from any system and leaves out risk checks, market data and timing.

Reconciliation

The second group is about checking: what the system compares while trading continues, and what it does when two records disagree.

  • ProtectionProtection is reconciled too. Every holding needs a stop that matches it: a holding without one is unprotected, a stop without a holding is an orphan, and a stop for the wrong size is a mismatch.
  • ReconciliationReconciliation is a live feature, not an end-of-day job. It compares more than positions: working orders on both sides, fills received against fills applied, and orders the system placed against orders someone else placed.
  • Foreign ordersAn order the system did not create is shown to a person. It is neither claimed nor ignored.

Three comparisons

What reconciliation compares

Not one end-of-day check, but three comparisons that run while trading continues.

Target↔Holding from fills

The difference is what is still to do. If no order is working for it, the position has drifted, and a person looks.

Holding↔Protection
Stop workingNo stop
HoldingOKunprotected
Flatorphan stopOK

A stop for the wrong size is a mismatch. Unprotected and mismatched positions go to a person; an orphan stop can be cancelled, and the cancel waits for confirmation.

Books↔Broker records

Compared at the same cutoff, covering positions, working orders on both sides and fills received against fills applied. A missing fill is found in the broker's fill history and recorded once by its ID; an order the system did not create is shown, not claimed.

Amber marks an outcome that needs review. Equal quantities alone do not prove every underlying record is right; each comparison covers only what it reads.

Where the examples come from

Every example on this site is constructed or run locally, so that a reader can inspect it. Each one isolates a judgement from the two groups above.

The same habit elsewhere

The other two projects on this site came out of the same habit: name the record, give it one owner, and check it against the source that can confirm it. Battuta keeps what is inside a take apart from what is needed to play it. Withmere keeps what happened, what it believes and what it has been given to be in separate records, with one history that is only appended to.

What I want to examine next

Examine what changes when fills continue to arrive during reconciliation, rather than pausing activity at the checkpoint.

Technical details, assumptions and verification

The judgements above are stated as mine; this page does not describe or demonstrate any employer's system. The Lab session uses one account and one instrument, with no prices, risk checks or rejects, acknowledgement on send and a fixed sequence of market events; only the five rules change. Reconciliation compares positions at a cutoff; equal quantities alone do not establish that every underlying record is correct.

Type
Execution state, portfolio state and reconciliation
Domains
trading infra / execution / portfolio state / reconciliation / observability

Available material

  • A browser-computed session with five switchable rules, from a repeated request to a cancel that loses a race.
  • Downloadable constructed inputs and the same JavaScript calculation module used by the page.
  • Regression checks for each rule switched off alone, a compensating combination, determinism and all 32 rule sets.
  • A recorded local Kafka and SQLite experiment on counting one fill once across consumer crashes, described in its essay.

The session computes each state from its inputs and the rules chosen. The downloadable inputs and JavaScript module use the same calculation as the page.

  1. target 2
  2. repeated request
  3. fill F1
  4. F1 delivered again
  5. rest fills, one notification lost
  6. reconciliation
  7. target 0
  8. part of the exit fills
  9. cancel the rest
  10. it fills before the cancel lands
  11. final reconciliation

One rule off at a time

all rules on
books 0, broker 0, at most 2 held
retry key off
books and broker agree, but 4 held for a target of 2
count each fill once off
F1 counted twice; the exit sells a unit never held: books 0, broker −1
reconcile from history off
a phantom working order; books −1, broker 0
fills, not intent off
books marked flat on sending the exit; books −2, broker 0
cancel confirmation off
a late fill dropped, then repaired from history

Derive every state from the inputs and the rules. With every rule on, books and broker agree at each reconciliation and the broker never holds more than the target.