The GPT weather forecaster
An AI bot combining rule-based temperature models, GPT-4o, and two weather APIs, betting daily temperature markets. Lifetime score: −$138, plus the API bill.
The hypothesisWeather is the most model-able thing prediction markets trade. Blend numerical forecasts with an LLM's synthesis, revise predictions as forecasts update, and bet the divergence from market prices.
The test
A daemon generated daily temperature predictions from rule-based models plus GPT, cross-referenced two independent weather data sources, placed bets against market prices, and revised its calls as new forecast runs arrived. Every prediction and revision was stored; every bet settled at resolution. It ran for months across dozens of city markets — long enough to measure everything.
The result
Net lifetime: −$138, before counting the LLM API costs (which, in its first incarnation, ran unmetered at a 10-minute cadence — the first retirement was as much about the bill as the P&L). One genuine finding emerged from the wreckage: the slice of predictions where the system's own confidence was ≥95% ran a 95.7% win rate at +8.4% ROI on paper over 346 trades — the model knew when it knew. But that slice never earned real size before the underlying market series was restructured out of existence, cutting the input feed permanently.
Conclusion
The market's weather prices were better than our blended model nearly everywhere the model dared to disagree. The salvageable lesson is about confidence gating, not forecasting: an AI system's self-reported certainty carried real signal, while its median prediction carried none. That principle — let the model classify its own certainty, never let it forecast — outlived the bot and now governs every LLM we still run.