Building an Automated Prediction Market Trading System

How I built a data recorder, research harness, and live execution system for Kalshi's 15-minute Bitcoin contracts, and what three months of data showed about whether there was an edge.

Jacob Kobrick June – September 2026

Introduction

Kalshi lists a Bitcoin contract every 15 minutes that asks a single question: will BTC be above a given strike price when the window closes? It resolves on the CF Benchmarks Real-Time Index (BRTI), a per-second Bitcoin price built from several major exchanges, averaged over the last 60 seconds before close. The contracts trade on a normal limit order book in one-cent ticks. How I got interested was a bit roundabout. I kept seeing fake "AI crypto trading bots" promoted on X and Reddit that were really just credential and secret loggers, built to quietly send your exchange API keys or wallet secrets to whoever wrote them. There were a lot of them, and none made money for anyone except the people running the scam. Seeing so many made me curious about the honest version of the question, and I believed that an actual edge probably did exist somewhere in these markets, because of how the displayed price on Kalshi's UI was calculated and the delays I noticed on that end. Kalshi's 15-minute contracts seemed like a good place to look, since the horizon is so short and the settlement rule is completely public. If people were mispricing them, I figured it was something I could actually measure.

I didn't have a strategy in mind when I started, and I didn't want to commit to one before I had data, so the first thing I built was a recorder. The rest came later. There's a backtesting harness that replays recorded markets at prices I could realistically have gotten, a paper-trading engine, and eventually a live executor with a stack of risk checks in front of it. Most of the ideas I tested along the way didn't work. This write-up covers how the system fits together, how I evaluated the models, and what happened once it was trading real money.

1System design

Everything runs on one small Linux VPS as three systemd services. Two of them are recorders: one writes a row per second for each open contract, and the other stores the BRTI value once a second, both into SQLite. The third is the trading service. It's a single asyncio process (FastAPI serves the dashboard) that keeps the current order book in memory, runs the strategy on every tick, logs each decision, and sends real orders when live mode is turned on.

Figure 1. How data moves through the system. The recorders run separately from the trading service, so stopping or restarting trading never leaves a hole in the research data. Strategy settings live in a SQLite table that the trading service rereads every second, which lets me change a parameter without restarting anything.

I went with SQLite instead of Postgres because each recorder is the only process writing to its tables, and I could copy a database straight to my laptop for analysis. The one catch was that SQLite's online backup never finishes while a recorder is writing every second, so I take snapshots by pausing the recorder briefly and copying the file.

Every signal the strategy produces goes into a paper ledger, regardless of whether a live order was sent or filled. I wanted the strategy's track record to stay continuous even when the account ran short on funds or an order got rejected. It also meant I could compare what the strategy wanted to do against what actually executed, and that comparison is how I confirmed that live fills matched the prices my backtests assumed.

Settlement comes from polling the REST API rather than from the WebSocket. When a new contract opens, Kalshi's feed stops sending updates for the previous one, so the settlement message for a position I'm still holding can just never show up. Polling is slower, but it doesn't miss anything.

Live trading stays off unless I deliberately turn it on. Before each order, the executor checks that two separate environment flags both allow live trading, that the exchange reports itself open, that the balance covers the order with room to spare, and that the order fits under the per-order, per-session, and session-loss limits. If any one of those checks fails, nothing gets sent.

2Model evaluation

Over the first few weeks I trained three XGBoost models. The most developed one estimated the probability that a contract would settle YES, using per-second features from both the Kalshi market and the BRTI index. I trained it on 962 contracts from June 7 to 17 and held out the next 458 (June 18 to 23). Nothing about the held-out contracts was used during training, hyperparameter choices included, and the isotonic calibration step was fit on the training data only.

AUC measures how often a model gives a higher probability to a contract that settled YES than to one that settled NO, where 0.5 is a coin flip and 1.0 is perfect. On the held-out set the model scored an AUC of 0.905. That looks strong until you compare it with the obvious baseline. If you take each contract's last traded price and read it as a probability, you get 0.907 on exactly the same rows.

Figure 2. ROC curves on the held-out set (320,494 per-second observations from 458 contracts). The dashed curve uses the market's last price as the score, with no model involved.

**Table 1.** AUC by time remaining on the same held-out set. Rows from the same contract are highly correlated, so the effective sample size is much closer to 458 than to these row counts.
Time to close	Rows	Model	Market
0–3 min	62,440	0.989	0.988
3–7.5 min	121,417	0.929	0.929
7.5–15 min	136,637	0.799	0.803
All	320,494	0.905	0.907

In other words, the model hadn't found anything the market didn't already know. Most of its AUC comes from the fact that outcomes get easier to call as a contract approaches expiry, and the price captures that just as well. Table 1 shows the two staying within a few thousandths of each other at every horizon. For trading, the question that matters is whether the model disagrees with the price in the cases where the price turns out to be wrong, and by enough to cover the spread and fees. To test that, I chose one trading rule using only the training period and ran it a single time on the held-out period, buying at the ask and paying Kalshi's fees.

**Table 2.** Approaches in the order I tried them. Return is net profit after fees divided by the total cost of the positions taken. Backtests buy at the ask and hold to settlement, and each one's trading rule was fixed before its out-of-sample window was scored. Model-row returns are approximate because they're derived from per-contract results. Live rows come from actual exchange fills, and the revised row also shows return on the account balance at the start of the period, which is what Figure 3 plots.
Approach	Evaluation	Return on capital
XGBoost classifier, market data only58 features	Held-out 1,220 contracts, May 25 – Jun 7AUC 0.86	−4.9%1,077 trades
XGBoost regressor on net return80 features	Held-out ~1.0M rowsIC 0.009	No signalbest iteration = 0
XGBoost classifier, market + index27 features, isotonic calibration	Held-out 458 contracts, Jun 18–23AUC 0.905 (market 0.907)	≈ −4%137 trades; ≈ +19% in-sample
Rule-based signal, first version	Live, Jun 9 – Jul 9	−3.7%668 fills
Rule-based signal, revised	Live, Jul 13 – Aug 31	+25.3%on starting balance; +1.8% per dollar deployed; 564 fills

The third model is a good illustration of how easy it is to fool yourself with this kind of data. With an operating point tuned on the training period it returned about 19% on capital in-sample, and then lost about 4% on contracts it had never seen. After that I retired all three models. The strategy that eventually went live is a much simpler rule built around how the contracts settle. Every change to it was tested on a walk-forward split first, and then again on a later window of data that I had deliberately never looked at.

The models still turned out to be useful, just not as strategies. They became the bar any new idea had to clear, since a signal that couldn't beat the market price on its own wasn't worth backtesting.

3Live results and system performance

The revised rule started trading real money on July 13. Figure 3 shows the cumulative return through the end of August, calculated from the fill cost and fees that Kalshi reported on each order.

Figure 3. Cumulative net return from July 13 to August 31, 2026, as a percentage of the account balance at the start of the period. There were 564 filled orders across 50 trading days, and 35 of those days finished positive. Position sizes went up roughly tenfold on July 25 (dashed line), which is why the daily swings get larger after that. The +1.8% in Table 2 is lower because it divides by the total cost of every position, and the same money gets reused many times a day.

Moving from paper trading to real orders through Kalshi's API went better than I expected. Before going live I had estimated the fee formula by working backward from a handful of test trades, trying to pin down how the fee changed with position size and contract price. The actual fees came in about 11% lower than my estimate, and fills at full size landed slightly better than the prices my backtests assumed, so slippage cost less than I had budgeted for.

Most of the bad days in late July and early August were days when Bitcoin moved steadily in one direction for hours, which this rule doesn't handle well. In replay I tried a few ways of limiting the damage, including a daily stop-loss, blocking certain hours, and exiting early when a position turned against me. None of them helped enough to justify the extra complexity. They tended to cut into the good days about as much as they protected the bad ones, so I left the rule alone.

**Table 3.** Operational measurements from recorded data. Order latency runs from the local log entry written just before the request to the creation time the exchange reports, so it includes network time and any clock difference between the two machines.
Measure	Value
Index ticks recorded, Aug 10 – Sep 27	4.15M
Per-second market rows, same window	3.83M
Contracts covered, same window	4,440
Live orders sent, Jun – Sep	1,578
Failed orders	82 (5.2%)
Order latency, median / p90 / p99	0.58 / 0.97 / 1.07 s
Settlement reconstruction error, mean	$0.004

I did think about latency up front. I picked a VPS in Chicago on purpose because it's close to where Kalshi's API is hosted, and a ping from the server to the API comes back in about a millisecond. Since the network round trip is that short, most of the order latency in Table 3 is probably clock difference between the two machines or time spent on the exchange's side, though I can't separate those from the data I have. Beyond picking the location, I didn't spend much time optimizing for speed. The strategy reacts to things that play out over minutes, and when I studied the recorded order book, it took around ten seconds to reprice after a large index move. At that pace, shaving a few hundred milliseconds off order submission wouldn't have changed much. The more important check was whether fills at full size came in at the prices the backtests assumed, and they did. Of the 82 orders that failed, none failed because of price slippage.

4Engineering notes

Reconstructing settlement before trusting a backtest: A backtest has to know what each contract will settle at, and that depends on exactly how Kalshi averages the index. Before trusting any backtest, I rebuilt the settlement value from my own recorded ticks and compared it with Kalshi's published value on 332 contracts. The average error was under half a cent, and the worst was 28 cents. Getting there meant tracking down a parsing bug that read UTC timestamps as local time and shifted every window by four hours. Along the way I also found that unsettled markets store their result as an empty string instead of NULL, so they were passing a NOT NULL filter and getting counted as NO outcomes.
Connections that die silently: On August 11 the market recorder's WebSocket stopped delivering data without raising an error or closing, and the process sat idle for about 28 hours before I caught it. The trading service already restarts its feeds when they go quiet (after 8 seconds for the index and 60 for the order book). The recorder had a watchdog as well, but it watched for trade messages, and trades on these contracts can be sparse enough that it was never a reliable signal. That data is gone for good. What I took from it is that it's better to monitor what a process is actually writing than whether its connection looks healthy.
Win rate only means something next to entry price: The live rule wins most of its trades, and that number is easy to misread. On a binary contract, the price you pay is effectively the win rate you need to break even. If you buy at 80¢, a win pays you 20¢ and a loss costs you 80¢, so you have to win four times out of five just to break even, and a little more than that once fees are included. A strategy that wins 85% of the time at an average entry of 80¢ makes money, but the same 85% at an average entry of 88¢ loses it. The edge is the gap between the win rate and the average entry price, and in practice that gap is only a few percentage points, which is why a losing week can happen even when almost every trade wins. It's also why every return on this page is weighted by the money actually in each position. Early on I tracked an average return per trade with every trade counted equally, and during one losing week that number looked excellent because the small trades were mostly winners and the big ones were the losers.

5Takeaways

The machine learning experiments were a little humbling. I spent weeks building features, training models, and working through calibration, only to discover that the market's own price was slightly better at predicting the outcome. An AUC of 0.905 looks a lot less exciting with 0.907 sitting next to it. Still, I'm glad I went through the process, because it made me much more skeptical of results that look impressive in isolation.

I also underestimated how much of the project would have nothing to do with finding a trading strategy. Collecting reliable data, reconstructing settlement, accounting for fees, and making sure the live system behaved like the backtester took up most of the work.

The system is still running, and there's plenty I could improve. I'm not sure how much of the strategy's performance will hold up over a longer period, especially under different market conditions. Lately the edge seems to be getting smaller, and my guess is that other automated traders have started picking up on the same patterns, though I haven't been able to confirm that. Either way, I ended up with a much better understanding of how these markets work, a dataset I can keep experimenting with, and a system that has been genuinely fun to build. That's a pretty good outcome for something that started with me getting annoyed at crypto bot scams on X and Reddit.