WebScraper Engine Morocco
A high-performance web scraping engine that extracted 1.5M Moroccan company records with real-time dashboard and transactional data integrity.
Overview
WebScraper Engine Morocco is a large-scale scraping solution built with Node.js, Puppeteer, Cherrio, Prisma, and PostgreSQL. It includes a real-time dashboard powered by Express.js, EJS templates, and WebSocket communication. The system uses multi-threaded scraping for performance, automated dynamic content extraction with Puppeteer and Cherrio, and transactional integrity through Prisma and PostgreSQL. With more than 1.5 million company records processed, the engine demonstrates robustness, scalability, and efficiency. The repository remains private due to sensitive data, but technical details, architecture diagrams, and anonymized screenshots are available.
Achievements
- •Scraped and processed over 1.5 million Moroccan company records
- •Built a high-speed, multi-threaded scraping architecture
- •Developed a real-time dashboard with EJS templates and WebSocket updates
- •Automated dynamic content extraction with Puppeteer and Cheerio
- •Ensured transactional integrity and deduplication with Prisma + PostgreSQL
- •Designed a modular and extensible architecture for future scaling