Software Engineer (Web Crawler Development) - Scrapy
Job Description
Key Skills Required
Master these to land this role
Want to know if you're a match for this job?
At Zyte, we specialize in enabling businesses to extract data for innovation and growth. Founded in 2010, we are a globally distributed team of over 250 employees across 28 countries, dedicated to providing powerful tools for web data collection, formatting, and delivery at scale.
Our platform powers crawls of over 3 billion pages monthly, helping thousands of organizations make data-driven decisions. Zyte is seeking Software Engineers to join our Delivery team, focusing on developing web crawlers using Scrapy, our open-source flagship project.
Key Responsibilities:
- Design, develop, and maintain Scrapy web crawlers.
- Utilize the Zyte platform and open-source projects for distributed information extraction, retrieval, and data processing.
- Address and resolve performance and scalability challenges in distributed crawling at scale.
- Contribute to debugging and fixing issues within open-source projects, including Scrapy.
Zyte’s platform and Delivery offerings have seen significant growth, and this role will be pivotal in driving future projects.
Requirements:
- 3+ years of software development experience in Python.
- Strong proficiency in Python.
- Familiarity with Linux/UNIX, HTTP, HTML, JavaScript, and networking concepts.
- Effective communication skills in written and spoken English.
- Availability for full-time work.
Bonus:
- Experience with Scrapy is highly advantageous.
- Knowledge of crawling, extraction, and data processing tools (e.g., Scrapy, NLTK, pandas, scikit-learn, mapreduce, NoSQL).
- Advanced proficiency in English.
Benefits:
As a new team member, you will:
- Join a self-motivated, progressive, and multicultural team.
- Enjoy the flexibility to work from anywhere.
- Attend conferences and collaborate with global team members.
- Work with cutting-edge open-source technologies.
- Receive 35 days of paid holidays.
How would you rate this job post?
See what other professionals think about this role.
Similar Opportunities
Full-Stack Engineer at ElevenLabs
ElevenLabs
United Kingdom
IrelandDesign Engineer (Product-Focused, Frontend-Leaning/UX) at ElevenLabs
ElevenLabs
United KingdomDublin
Senior Software Engineer, Developer Productivity
Webflow
Argentina
QA Engineer - Quality Assurance Specialist
Ciandt
BrazilMore Openings at Zyte
Explore Top Companies in this Space
Oxylabs
Proxy Networks & Web Data Acquisition SaaS / AI-Powered Web Scraping APIs & Data Extraction / Network Infrastructure Tools / Technical Search Engine Optimization (SEO) & Intelligence Platforms
Domes Resorts
Hospitality / Glamping / Sustainable Tourism / Luxury Experiences
Pitch
Technology / Software / Enterprise Solutions / Business Tools
Verint
Enterprise Software / Customer Experience / Analytics / AI
Zyte (formerly known globally as Scrapinghub) is the premier, enterprise-grade web data extraction platform, open-source proxy infrastructure pioneer, and AI-driven data harvesting powerhouse engineered to serve as the definitive, unblockable web intelligence layer for scaling digital marketplaces, e-commerce giants, and LLM training networks worldwide. As the original creators and primary contributors of Scrapy—the industry-standard open-source web crawling framework—the company completely eliminates the severe systemic friction of modern web data acquisition, where engineering teams constantly battle sophisticated anti-bot blocks, IP bans, cloudflare challenges, and broken layout formats, by deploying an adaptive, automated data pipeline infrastructure. Moving far beyond traditional, brittle script writing or manual data collection, Zyte unifies its signature Smart Proxy Manager (formerly Crawlera) for seamless automated rotation telemetry, its patented AI-driven Automatic Extraction API that derives clean structured datasets from product or news fields instantly without custom spiders, and its enterprise-tier fully managed data feed service into a single high-availability operational ecosystem. The platform empowers thousands of data-fueled organizations and Fortune 500 enterprises to safely scale e-commerce price monitoring, scrape complex search engine result pages (SERPs), and fuel generative AI models with audit-ready, legally compliant web telemetry across 116 countries. Under the hood, its sophisticated cloud-native core natively orchestrates high-concurrency browser rendering pipelines via Zyte API, robust JavaScript executing sandboxes, and advanced residential proxy routing arrays designed to sustain billions of monthly requests with elite, production-hardened precision. What sets Zyte apart is its uncompromising dedication to replacing fragile manual engineering overhead with absolute data assurance; by bridging the gap between performance-intensive anti-bot defense unblocking and highly dependable, structured data deliveries, the firm enables modern scaling enterprise ecosystems to eliminate engineering backlogs, protect pricing intelligence streams, and unlock continuous commercial scalability.
Safety First
- Never pay for a job application.
- Do not share sensitive bank info.
- Verify the client before starting work.