Sr Data Engineer, Data Governance

Job description

Opportunity:

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data team. In this role, you will be instrumental in designing, building, and maintaining robust and scalable data pipelines and infrastructure that power critical data-driven initiatives across our organization. You will work with vast datasets, cutting-edge technologies, and collaborate closely with AI researchers, other data engineers, data scientists, machine learning engineers, and product teams to deliver insights that shape the future of our products and user experiences.

What You'll Do:

Build and Optimize Data Infrastructure:

* Develop, construct, test, and maintain large-scale data ingest architecture consisting of diverse cloud-based services (messaging, storage, Kubernetes, persistent data store, serverless functions, etc).
* Create tooling like SDK, APIs to enable user self-service.
* Contribute to the design and evolution of our core data platform,ensuring its scalability, reliability, and cost-effectiveness.
* Implement robust monitoring, alerting, and logging solutions for data pipelines and infrastructure to proactively identify and resolve issues.

Design and Implement Scalable Data Pipelines:

* Design and implement highly reliable and efficient ETL/ELT processes to ingest, transform, and load data from diverse sources (e.g., real-time events, third-party APIs, rich media datasets) into our data lake and data warehouses.
* Utilize distributed data processing frameworks like Spark or similar to handle large scale data volumes with high throughput and low latency.