Responsibilities:
• Design & Implement Data ingestion and Data lakes-based solutions
using Big Data Technologies.
• The Tech Lead should be highly proficient in the use of Big Data / Open- Source Technologies and standard techniques of Data Integration,
Data Manipulation.
• Should be able to design and develop cost efficient and performant data pipelines in the cloud platform
• Create data environment to support our data analytics, reporting and data science teams
• Experience with integration of data from multiple data sources
• Knowledge of various Data Pipeline techniques and frameworks
• Performance optimization - need to monitor the complete process and apply necessary infrastructure changes to speed up the query execution.
• Efficient data ingestion - Discovering patterns in data sets with data mining techniques and using different data ingestion APIs and inject data into the data lake as per need.
The Role offers:
• Great opportunities to learn various tools and technologies used in a sophisticated data architecture within the Business Intelligence and Analytics Data Services
• Gives an opportunity to showcase candidates' strong analytical skills and problem-solving ability
• An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations
• Grow into a Technical architect role over a period
Essential Skills:
• 6+ Years hands on knowledge on SQL as well as SQL/NoSQL databases
• Proficient in programming languages such as Python, PySpark, Scala and Java
• Experience with Spark
• Working knowledge of XML, ETL, API and Web Services
• Experience with integration of data from multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming
• Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or GCP)
Essential Qualification:
• BE/Btech in Computer Science, Engineering or relevant field
