Tech Lead - Web Scraping and Crawling (Remote)
Purple Squirrel Consulting Services Verified employer
Actively hiring · Posted 18d ago · JEV261153147B6978
Job description
Manage individual projects priorities, deadlines, and deliverables
Gather and process raw data at scale (including writing scripts, web scraping, calling/create
APIs, etc.) from the web / internet
Develop frameworks for automating and maintaining constant flow of data from multiple
sources
Identify, analysis, design, and implement internal process improvements
Design and implement tooling upgrades to increase stability and data quality
Help team to fix issues that occur in test and production environments
Automate software development processes, including build, deploy, and test
Manage and guide the team members
REQUIRED QUALIFICATIONS:
4+ years of web crawling/ scraping experience is a must
Strong knowledge of scraping frameworks such as Scrapy, Beautiful Soup, HTQL, Jsoup, Web-
Harvest and others
Excellent verbal, written, and interpersonal communication skills in English
Good to have Experience of complex crawling (like captcha, Mobile OTP based crawling,
bypassing proxy)
Sound Knowledge in Bot Management Techniques
Experience in various data extraction methods (like data extraction from PDF Files, web
pages, etc)
Good understanding of HTML DOM, CSS, Javascript, and RESTful web services
Good to have understanding of AWS
Experience with Linux
Experience with Java / Python
Key skills
How to apply
- Sign in or create your free profile — takes about two minutes.
- Apply with one tap — your profile is shared directly with Purple Squirrel Consulting Services.
- Track your application and get updates by email and WhatsApp.