All projects
Overview
A scraping engine built to collect Moroccan company data at scale. It runs as a distributed system of workers on a self-hosted server and has processed more than 2 million records.
Because long scraping runs fail in many ways, the design focuses on staying alive and staying clean: workers restart automatically and duplicate records are filtered before storage.
What I built
- Engineered a distributed scraping system processing 2M+ records on a self-hosted auto-healing server.
- Built a live dashboard with KPIs, database metrics, thread control and real-time job monitoring.
- Designed a fault-tolerant pipeline with auto-restart, deduplication and scalable PostgreSQL storage.
Design and architecture
- Workers: Node.js and Puppeteer, with thread control and automatic restart on failure.
- Monitoring: a live dashboard streamed over WebSocket showing KPIs, database metrics and job progress.
- Storage: PostgreSQL through Prisma, with deduplication keeping the dataset clean.