Senior Analyst - Data Engineer
A Senior Analyst in Applied intelligence (Data Engineer) -understands the importance of sound analytical decision-making, relationship of tasks to the overall project, and executes projects in the context of a business performance improvement initiative.
Duties and Responsibilities:
Design and develop scalable solutions to store and retrieve a high volume of data on the cloud.
Deliver project and customer success by meeting deadlines, managing expectations, and by delivering good quality solutions.
Develop and maintain scalable data pipelines and build out API integrations to support continuing increases in data volume and complexity.
Perform data analysis required to troubleshoot data-related issues and assist in the resolution of data issues.
Design data integrations and data quality framework.
Work closely with all business units and engineering teams to develop a strategy for long-term data platform architecture.
Responsible to Ingest data from files, streams, and databases. Process the data with Hive, Hadoop, Spark.
Responsible to design and develop distributed, high-volume, high-velocity multi-threaded event processing systems using Big Data technologies.
Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies Apache Spark, Hadoop, and Cloud computing, etc.
Investigate, identify, and establish new tools and processes for data warehousing, data quality assurance, reporting, business intelligence, data governance, and data cataloging.
Setup reliable data ingestion pipelines for new data sources and integrate them effectively with existing data sets.
Develop and maintain optimal data pipelines on the cloud.
Work on migrating on-prem data warehouses to the cloud platform.
Assist in the development of reusable code and accelerators development.
Training and mentoring other members of the team.
Open to learning new technologies in a short duration.
Bachelor/master s degree preferably in Computer Science, Computer Engineering, or related discipline
Advanced SQL knowledge and experience of working with a variety of databases.
Hands-on experience on Big Data tools and design and develop data pipeline on any Cloud Platform.
Possess in-depth knowledge of various Design Patterns in Bigdata, Data Processing Patterns (Batch/NRT/RT processing) & capable of providing design & architecture of typical business problems.
Hands-on experience in working with programming languages such as Python.
Solid experience building and optimizing big data pipelines, architecture, and data sets.
Experience with Big Data technologies such as Hadoop, Kafka, NoSQL databases
Familiarity with AWS/GCP cloud deployment models
Self-starter, challenger, and analytical thinker
Experience in working on cloud data migration projects.
Working experience in one of the cloud platform GCP/AWS/Azure is a must
Working experience with Big Query/ Snowflake/Redshift (at least one of them)
Experience in developing data pipelines using tools such as Spark, Python, AWS Glue, Azure Data Factory
Hands-on experience in Dataflow, Apache Beam, Spark, Hadoop, Kafka, BigQuery, Data Catalogue, Airflow, etc.
Dynamic who demonstrates a very positive attitude
Strong Analytical and Communication Skills
Excellent written and verbal communication skills
Must have 4 to 6 years of expereince in Data enginnering domain