Key responsibilities include:

• Designing scalable scraping pipelines
• Managing high-concurrency scraping workers
• Queue orchestration (Redis / RabbitMQ / Kafka or similar)
• Rate-limit management
• Proxy rotation systems (residential / datacenter proxies)
• Handling anti-bot protections and browser fingerprinting
• CAPTCHA / bot protection handling
• Session and cookie lifecycle management
• Browser automation using Puppeteer / Headless Chrome / similar frameworks
• Building distributed scraping architectures
• Job scheduling, retry logic, and fault tolerance
• Data normalization, validation, and storage pipelines
• Monitoring scraping health, success rates, and failure patterns
• Operating and scaling Linux cloud infrastructure (AWS)

• Deep practical experience with AI coding assistants such as Claude Code, Cursor, Copilot, etc.
• Ability to use AI tools to generate, review, and optimize code
• Leveraging LLMs to assist with scraping logic, debugging, and system design
• Building workflows where AI tools are integrated into the development process
• Using AI for log analysis, debugging complex scraping failures, and improving system reliability