Our data comes from a long list of catalogue websites. Some load content as you scroll. Some hide it behind cookie banners. Some tuck the useful parts inside page scripts. Before this, the answer was a hand-written scraper per site, each returning data in its own shape, each breaking without telling anyone.
The problem with hand-written scrapers
Three things hurt us. They were slow to build, so new sources waited. They failed quietly, so stale data looked like fresh data. And every one of them returned a different structure, so whatever sat downstream had to understand all of them.
I'd say the last one was the real cost. You can live with slow and fragile if the output is consistent. You can't build anything clean on top of a different shape for every site.
One output format: Markdown
The decision that made the rest work was mine to make, and it was picking a single output. Every scraper, for every site, turns each page into one Markdown file. Each file starts with a short header saying where it came from and when it was fetched. Headings stay headings, lists stay lists, tables stay tables.
Why Markdown and not JSON? Because we didn't know every question people would ask of the data. JSON forces you to decide the fields up front. Markdown keeps the page's structure and text, it's readable by a person, and language models handle it well. Structure can come later, for the cases that need it.
An agent writes the scraper
When we add a website, we hand it to the Hermes agent and it writes a Playwright scraper for that site. The scraper doesn't start from nothing. It sits on a shared library that handles the boring, repeated parts:
- accepting or dismissing cookie banners
- scrolling to trigger lazy-loaded content
- pulling data out of page scripts, JSON-LD and meta tags as well as the visible HTML
- turning HTML into Markdown with the source header
- retries and rate limits, so we stay polite to the sites we read
Because the library does the hard, repeated work, each site's scraper stays short. That matters twice: it's quicker for the agent to get right, and it's quicker for a person to review.
A few small services instead of one big one
I split the platform into services that each do one job:
- API gateway (FastAPI) with Firebase sign-in and API keys for other systems.
- Scheduler (APScheduler) that runs each source on its own timetable.
- Runner that executes the Playwright scripts.
- Worker that orchestrates a run and uploads the Markdown to Amazon S3.
- Streamer that sends live logs over server-sent events to a React dashboard (Vite, TypeScript, Zustand and React Query), so you can watch a run as it happens.
- Repo listener that pulls in scraper code the moment it's pushed to Bitbucket. Nobody deploys scrapers by hand.
Nginx sits in front, and a paginated streaming API serves the Markdown to whatever needs it next, including a converter that turns it into structured data for the cases where we do want fields.
Then a second agent answers questions
The Markdown feeds a second agent. You ask it a question in plain language, it answers from the files. You don't write SQL or learn how each website lays out its data.
What I learned
- Standardise the output before you standardise anything else. We still have one scraper per site. What we stopped having is one format per site.
- Let the agent write the part that varies. The shared library is code we own and test. The per-site scraper is the small, site-specific bit, and that's where an agent saves the most time.
- Make runs visible. Live logs and a dashboard turned "the scraper broke last week" into "the scraper broke this morning, here's the page".
Stack
Python, Playwright, FastAPI, APScheduler, React with Vite and TypeScript, MongoDB, Amazon S3, Docker Compose, Nginx, Hermes agent.
My role: platform design and delivery. Site names and volumes are left out on purpose.