A Senior Analyst in ‘Applied intelligence (Data Engineer) –understands the importance of sound analytical decision-making, relationship of tasks to the overall project, and executes projects in the context of a business performance improvement initiative.
Duties and Responsibilities:
• Design and develop scalable solutions to store and retrieve a high volume of data on the cloud.
• Deliver project and customer success by meeting deadlines, managing expectations, and by delivering good quality solutions.
• Develop and maintain scalable data pipelines and build out API integrations to support continuing increases in data volume and complexity.
• Perform data analysis required to troubleshoot data-related issues and assist in the resolution of data issues.
• Design data integrations and data quality framework.
• Work closely with all business units and engineering teams to develop a strategy for long-term data platform architecture.
• Responsible to Ingest data from files, streams, and databases. Process the data with Hive, Hadoop, Spark.
• Responsible to design and develop distributed, high-volume, high-velocity multi-threaded event processing systems using Big Data technologies.
• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Hadoop, and Cloud computing, etc.
• Investigate, identify, and establish new tools and processes for data warehousing, data quality assurance, reporting, business intelligence, data governance, and data cataloging.
• Setup reliable data ingestion pipelines for new data sources and integrate them effectively with existing data sets.
• Develop and maintain optimal data pipelines on the cloud.
• Work on migrating on-prem data warehouses to the cloud platform.
• Assist in the development of reusable code and accelerators development.
• Training and mentoring other members of the team.
• Open to learning new technologies in a short duration.
Bachelor /master’s degree preferably in Computer Science, Computer Engineering, or related discipline
• Advanced SQL knowledge and experience of working with a variety of databases.
• Hands-on experience on Big Data tools and design and develop data pipeline on any Cloud Platform.
• Possess in-depth knowledge of various Design Patterns in Bigdata, Data Processing Patterns (Batch/ NRT/RT processing) & capable of providing design & architecture of typical business problems.
• Hands-on experience in working with programming languages such as Python.
• Solid experience building and optimizing big data pipelines, architecture, and data sets.
• Experience with Big Data technologies such as Hadoop, Kafka, NoSQL databases
• Familiarity with AWS/GCP cloud deployment models
• Self-starter, challenger, and analytical thinker
• Experience in working on cloud data migration projects.
• Working experience in one of the cloud platform GCP/AWS/ Azure is a must
• Working experience with Big Query/ Snowflake/ Redshift (at least one of them)
• Experience in developing data pipelines using tools such as Spark, Python, AWS Glue, Azure Data Factory
• Hands-on experience in Dataflow, Apache Beam, Spark, Hadoop, Kafka, BigQuery, Data Catalogue, Airflow, etc.
• Dynamic who demonstrates a very positive attitude
• Strong Analytical and Communication Skills
• Excellent written and verbal communication skills
• Must have 4 to 6 years of expereince in Data enginnering domain.