Senior Software Engineer (Python)
This senior software engineer role, facilitated by Jobgether, focuses on building large-scale data extraction and processing systems. The candidate will design distributed pipelines for web crawling, data normalization, and search indexing within cloud environments. Strong proficiency in Python, search technologies like ElasticSearch, and distributed frameworks such as Spark and Airflow is required. The role involves close collaboration with ML teams to integrate production-grade data architectures.
50k new jobs listed every day. Install TAL to find more jobs like this.
Experience
5+ years
Function
Engineering
Work mode
Remote, India
Company
Tier 3
What you will work on
This senior software engineer role, facilitated by Jobgether, focuses on building large-scale data extraction and processing systems. The candidate will design distributed pipelines for web crawling, data normalization, and search indexing within cloud environments. Strong proficiency in Python, search technologies like ElasticSearch, and distributed frameworks such as Spark and Airflow is required. The role involves close collaboration with ML teams to integrate production-grade data architectures.
TAL's take
The role is posted by a recruitment agency on behalf of an unidentified company, limiting the caliber score due to lack of direct brand transparency.
The JD is highly specific regarding the technical requirements for data ingestion, crawling, and pipeline development.
Must haves
- 5+ years professional experience in Python development
- Experience in web scraping and data pipeline systems
- Experience with REST APIs and processing structured/unstructured data
- Knowledge of search technologies like ElasticSearch/OpenSearch
- Hands-on experience with Apache Airflow and Spark
- Experience working in cloud environments like AWS or GCP
Tools and skills
Nice to have: kubernetes, spacy, hugging face, tensorflow, pytorch.
About the company
Jobgether is a recruitment platform acting as an agency on behalf of unidentified partner companies.