Project Description
Title:
Senior Web Scraping & Data Engineering Developer for Content Intelligence Platform
Project Overview:
I am building a content‑aggregation and insights pipeline that collects updates, posts, and signals from major online platforms and public sources. The output will support a monthly insights publication and a long‑term analytics framework. I am seeking an experienced developer from a cost‑efficient region with strong skills in web scraping, browser automation, data engineering, and API integration.
Key Responsibilities:
Build and maintain scraping workflows for platforms such as LinkedIn, X/Twitter, Reddit, Substack, Facebook, Instagram, and other public sources
Implement browser-based automation using tools like Playwright, Puppeteer, or Selenium
Use lightweight scrapers where appropriate to reduce compute usage
Create profile-based, organization-based, and keyword-based scraping approaches
Optimize concurrency, session handling, and anti-blocking strategies
Produce structured data outputs (JSON/CSV/DB) for downstream analytics
Implement delta scraping to capture only new content efficiently
Integrate with Apify Actors or custom Node.js/Python scripts
Maintain reliability as platforms evolve or change their anti-bot systems
Required Skills:
Strong experience with Python or Node.js
Expertise in browser automation frameworks such as Playwright, Puppeteer, or Selenium
Experience with Apify, Scrapy, or similar scraping frameworks
Knowledge of proxy rotation, session management, and anti-detection techniques
Ability to design scalable, maintainable scraping pipelines
Familiarity with REST APIs, OAuth, and data normalization
Experience with MongoDB, PostgreSQL, or cloud storage solutions
Preferred Experience:
Prior work scraping platforms with strong anti-bot protections
Experience building content aggregation, monitoring, or intelligence systems
Familiarity with summarization, tagging, or enrichment workflows (nice to have)
Engagement Details:
Long-term engagement (3–12 months)
Weekly deliverables
Competitive compensation based on experience
Must be available for periodic calls in US Eastern Time evenings
To Apply:
Please share:
Examples of similar scraping or automation projects
Your preferred tech stack
Your experience with browser automation
Your availability and hourly/monthly rate