EXITO Supply Collector — a scraper that still tries on sites it's never seen
PythonPlaywrightCustomTkinterPyInstaller
Context
Store operators spend a large part of every day copying product data out of supplier sites. The supplier list keeps growing and no two sites are structured alike.
Problem
Writing a bespoke crawler per supplier means a developer is needed every time a supplier is added. Under that model the operator can never run it themselves.
Decisions
Never come back empty-handed from an unknown site
When no registered crawler matches, a GenericCrawler walks structured data (JSON-LD), then OpenGraph, then meta tags, then the DOM. It isn't perfect, but the answer is never just 'this site is unsupported'.
Made adding a supplier a registration, not a development task
Subclass `BaseCrawler`, implement `can_handle` and `collect`, register it with `CrawlerRegistry`. Site-specific selectors and credentials live in configuration rather than code.
Packaged it for someone who doesn't write Python
I didn't stop at a CLI. It has a CustomTkinter MVC interface and ships as a PyInstaller executable, because a tool only gets used if a non-technical operator can open it and start working.
Rewrote image paths so the output actually opens
Image references inside collected HTML are rewritten to point at the downloaded local files, so the output survives the source site changing or blocking hotlinks.
What I owned
- Architecture
- Crawler engineering
- Desktop UI
- Packaging and distribution
Actual deliverables
- Distributable desktop executable (PyInstaller)
- Per-supplier crawler plugins plus a generic fallback crawler
- Excel, JSON and CSV export adapters
Honestly — what I didn't get done
The GenericCrawler loses accuracy on sites with poor structured data. Those ultimately need a dedicated plugin — which is exactly why the plugin structure came first.
If you need this kind of work
Describe the thing you're currently stuck on. I'll reply by first separating what I can take on from what I can't.