A Linux container wakes every hour, scrapes 40 Smyths Toys product pages, diffs against the previous run, and commits the results to this repo. This page is rendered from those commits.
A single-purpose product watcher. Every hour it pulls the same 40 Smyths Toys product pages, parses each into structured fields (name, price, availability, image, category), and compares the result against the previous run.
Anything that moved — a price change, a stockout, a rename — shows up in the changes feed. Pages that throw an hCaptcha are solved by a free vision model and the puzzle is saved with its verdict. A run that fails becomes a row in the timeline with the error tail attached.
No database, no server: the scraper commits JSON files to this repo, GitHub Pages serves them, and the browser renders them.
tick.js) orchestrates: it ingests finished runs, diffs them, updates this page, and launches a fresh scrape container once an hour when one is due; a Claude Code agent reviews health separately and repairs code on failure.Tor daemon up, Xvfb virtual display up, then a self-check verifies Tor control auth + NEWNYM works (refuses to start if not), then patchright launches a real headed Chrome.SIGNAL NEWNYM to Tor to rotate the exit circuit, then retry.data/latest-changes.json.data/timeline.json, refresh data/run-summary.json, commit + push.Reverse-chronological. Newest run at top. Click any row to see captcha counts, Tor rotations, duration, and (on failures) the stderr tail.
Field-level diff against the previous run's snapshot (data/runs/<prev>.ndjson), matched by SKU. The first run has nothing to compare to, so everything is new; after that, only what actually moved — price, stock, name, image, category — appears here. Price moves are coloured down / up.
A challenged Tor exit gets an hCaptcha “select in order” doodle puzzle. The vision model reads the legend, locates each target, and clicks them in order. The coloured rings below mark where it clicked; the verdict says whether it cleared.
tor --hash-password salts randomly), so every SIGNAL NEWNYM failed auth (515) and a flagged exit could never be left. Fixed three ways: the hash is generated at build time so it always matches, rotateExit() now checks for a real 250 OK instead of assuming success, and the container verifies rotation auth at boot and refuses to start otherwise.503/429 under load.→The solver chains Gemini, OpenRouter Qwen3-VL, Groq and Nvidia and rotates on rate-limit or 5xx. If every free tier is saturated, the lane rotates the Tor exit instead — a fresh exit is often not challenged at all.patchright (a Chromium fork) strips the CDP automation signals Imperva fingerprints.Xvfb, a virtual display. Nothing reaches a physical screen, which is what lets it run in a container with no GUI.SIGNAL NEWNYM on consecutive failures.page.screenshot({ scale: 'css' }) puts the solver and the page on one coordinate system.apt-get mirror error.→The Dockerfile is idempotent; rerunning cleared it.Two distinct things. Resilience is how a single run keeps itself going; it runs inside the container. Self-healing is what happens around the runs — recovery and code repair — and lives on the host, not in the container.
SIGNAL NEWNYM on consecutive failures or a flagged exit.This is ordinary fault tolerance, not healing — it just gets one scrape finished.
What runs where: the container is the scrape plus its resilience. The host holds the schedule, the diff/timeline, the dashboard, and the repair agent — none of which ships to a deployment of the container.
Fixes, improvements and reviews the Claude Code agent has made, newest first. Each maps to a commit in the repos.
This is a demo, so the integration points still being agreed use sensible placeholders. None of them touch the scrape engine — they are where it reads input and writes output.
| Integration point | What the demo uses |
|---|---|
| Output schema | a factory JSON shape in transform.js — swap the mapper for the real field contract |
| URL list source | a fixed file of 40 URLs; the job already takes a list, so this is just where the list is read from |
| Trigger | a local 15-minute cron; in production a worker, queue or scheduler calls the job instead |
| Runtime | a Docker linux/amd64 image — the same image runs anywhere that env runs |
| Output sink | NDJSON committed to this repo; point it at a bucket, a database or a callback |
Most of the moving parts are optional and switch off with a single env var — no code change:
SOLVE_ENABLED=0 — turn the captcha solver off entirely (a challenged exit is simply rotated away).USE_TOR=0 — run direct, no Tor, no rotation.SOLVE_PAID_FALLBACK=0 — never call a paid solver (default; set to 1 with a key to enable).CAPTCHA_DUMP_DIR — stop saving puzzle screenshots.Once the schema, list source, trigger, runtime and output sink are confirmed, only that input/output layer changes.
The single-shot engine: patchright runner, Tor wiring, vision-LLM captcha solver, Dockerfile. Request access from Philippos.
This site and its orchestration: the 15-minute tick, the diff and timeline scripts, and the rendered dashboard.
The one-shot precursor: a single backfill of 58,605 UK Smyths products. This project is the recurring-watcher follow-up, scoped to 40 URLs to prove change detection.
The broader scraping toolkit this engine was extracted from — multi-lane orchestration and other targets beyond Smyths.