Data & scraping · June 2026

Allegro Parser — marketplace price & offer monitoring

Production system for daily monitoring of products, offers and sellers on the allegro.pl marketplace. A headless backend: data collection, anti-bot bypass, distributed processing and a REST API for competitor price analytics.

What was built:

DataDome anti-bot bypass via ephemeral Camoufox (antidetect Firefox) + rotating ISP proxies
Data extracted from embedded JSON blocks (not HTML scraping); one page load = all offers
PostgreSQL: 8 tables, 23 migrations, monthly partitioning of history (auto-create), bulk insert/upsert
Pipeline: Discovery → History → Offer-sync → Enrichment → Competitor tracking
Distributed processing: Celery + RabbitMQ, worker pools with queue priorities, Celery Beat, Flower
REST API on FastAPI (task triggers, data snapshots, webhook notifications, Swagger UI)
Redis for proxy state and ban-tracking, structlog, Docker (dev + prod)

24-hour stress test: 8,560 requests, 85% success, 0 permanent IP bans, 160,053 offers collected (6 ISP IPs in parallel).

Role: architecture and full-cycle backend development — data collection, anti-bot, data engineering, API.

Project screens

Screen 1
Screen 2
Screen 3
Screen 4