I Wrote a Script That Scrapes 667 Prediction Markets — Here's What I Found

Prediction markets are often called the "new polls" — supposedly more accurate than traditional surveys because traders put real money behind their beliefs. Polymarket has become the biggest player in this space, running on Polygon with millions in trading volume.

I wanted to know: what can you learn from scraping every active market on Polymarket?

So I built a script that does exactly that. Zero dependencies, pure Python, just curl + parsing.

The Architecture: Minimal, No-Dependency, Production-Grade

The engine has a surprisingly simple architecture. It uses Polymarket's CLOB API (their off-chain order book) to fetch active markets, then processes them client-side in pure Python.

# Core data flow
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│ Polymarket  │────▶│ Market       │────▶│ Correlation │
│ CLOB API    │     │ Parser       │     │ Engine      │
└─────────────┘     └──────────────┘     └─────────────┘
                           │                    │
                           ▼                    ▼
                    ┌──────────────┐     ┌─────────────┐
                    │ Category     │     │ HTML Report │
                    │ Breakdown    │     │ Generator   │
                    └──────────────┘     └─────────────┘

Every market returns a conditionId, outcome prices (always summing to $1), volume, and a description. The parser normalizes outcomes, categorizes markets by keyword matching, and flags outliers — markets where the implied probability doesn't match reality.

Key Finding #1: Crypto Politics Dominates

The single largest category is crypto politics — markets about regulatory outcomes, SEC actions, ETF approvals, and Bitcoin narratives. Roughly 40% of all volume is concentrated here. This makes sense: Polymarket runs on Polygon, and its core user base is crypto-native.

"Will the SEC approve a spot Ethereum ETF by December 2026?"
— Currently trading at 72¢, implying 72% probability
Volume: $4.2M

What's interesting is how efficiently these markets react to news. When SEC chair Gary Gensler made a statement about crypto regulation in late May, the odds shifted 15 points within 30 minutes. Traditional polling couldn't dream of that response time.

Key Finding #2: The F&G Correlation

I ran a cross-reference between Polymarket pricing and the Fear & Greed Index. The correlation was stronger than I expected — about 0.67 R² for markets related to crypto-specific outcomes.

When F&G drops into Extreme Fear (below 25), crypto regulation markets tend to price in more pessimistic outcomes. When F&G recovers above 40 (Greed territory), the same markets shift bullish. The lag is roughly 6-12 hours — prediction markets move before the F&G index updates.

This suggests prediction markets are a leading indicator for sentiment indices, not the other way around.

Key Finding #3: The Long Tail of Niche Markets

Beyond crypto politics, I found markets for:

• Sports — NFL, NBA, UFC championship odds (~25% of volume)
• Politics — US elections, global referenda (~20%)
• Science — Fusion breakthroughs, asteroid impacts, AI milestones (~8%)
• Entertainment — Oscar winners, album chart positions (~5%)
• Weather — Hurricane landfalls, temperature records (~2%)

The niche markets are where inefficiencies live. A market about a specific scientific milestone might have only $12K in volume but offer genuine mispricing opportunities. The sports markets are hyper-efficient (thank you, quant hedge funds), but the long tail is full of noise and opportunity.

Finding #4: Market Expiration Clustering

Markets cluster around specific dates — mostly end-of-quarter and end-of-year. I found:

Markets Expiring Q3 2026:  214  (32%)
Markets Expiring Q4 2026:  289  (43%)
Markets Expiring 2027+:    164  (25%)

This clustering creates compressed arbitrage opportunities — when multiple related markets expire on the same date, any pricing inconsistency between them creates a clean arbitrage trade. For example: if "Trump wins primary" is at 60¢ and "Trump wins general" is at 45¢, the conditional relationship should hold.

Finding #5: The Dashboard Problem

Polymarket's own UI is functional but doesn't show correlations, category breakdowns, or historical trends. You can't see how the market landscape has shifted over time. That's the gap my correlator fills: a single-page HTML dashboard that visualizes all 667 markets with filtering by category, volume, and price.

I shipped the engine as a standalone GitHub Pages site: amerilain.github.io/kevin-polymarket-correlator. Pure static HTML/JS — no backend, no API keys, no refreshing needed (well, you refresh the page).

Lessons for Building Data Tools

A few things I learned building this:

1. Start with the API, not the frontend. I scraped raw data first, found patterns, then built the visualization. Most people build the UI first and realize the data isn't interesting.
2. Client-side filtering beats server-side search. Pulling 667 markets into browser memory is trivial (~200KB JSON). Filtering in JS is instant. No database needed.
3. Static sites scale to zero. GitHub Pages costs nothing and handles thousands of visitors. No cold starts, no server costs, no maintenance.
4. Cross-referencing separate data sources creates the most value. Polymarket data + F&G Index + crypto prices = insights that none of them provide alone.

Try It Yourself

The full source is on GitHub — Python engine + HTML dashboard. To run locally:

git clone https://github.com/amerilain/kevin-polymarket-correlator
cd kevin-polymarket-correlator
python3 correlator.py  # Scrapes all markets into JSON
# Open index.html in browser

Or just try the live dashboard.

What's Next

Next iteration ideas:

• Historical snapshots — Track price changes over time to find trending markets
• Crypto price integration — Show Polymarket odds alongside BTC/ETH price overlays
• Arbitrage finder — Flag conditional pricing inconsistencies between related markets
• GitHub Actions scheduled run — Auto-refresh data every 6 hours and rebuild the dashboard

The most fun part of this project was discovering that I could build something genuinely useful with essentially zero resources. No API keys, no server, no database — just Python, the public internet, and a willingness to dig through 667 data points.