Senior Data Engineering Developer II

Summary

We are seeking a highly skilled and experienced Senior Data Engineering Developer II to join our growing team. In this role, you will be a key contributor to the design, development, and maintenance of our data infrastructure. You will lead projects, mentor junior engineers, and play a crucial role in shaping our data strategy. This is an excellent opportunity for a driven and results-oriented individual to make a significant impact on a dynamic and innovative team. 

Key Responsibilities

• Design, develop, and maintain highly scalable, robust, and fault-tolerant data processing systems. 
• Lead and manage data engineering projects with minimal supervision, assessing the impact on existing data infrastructure and implementing new data structures as needed.
• Design and deploy AI-driven data transformation processes and intelligent data quality frameworks that proactively identify, flag, and suggest remediation for data anomalies, drift, and inconsistencies, automating critical validation and cleansing tasks.
• Develop and implement AI-powered metadata management and data cataloging systems to automate data discovery, lineage tracking, and semantic understanding, significantly simplifying data governance and accessibility for data engineers and AI practitioners.
• Engineer sophisticated data solutions, including feature stores and optimized data access layers, specifically designed to accelerate the development, training, and deployment of advanced AI and machine learning models, enabling impactful AI outcomes.
• Utilize machine learning models to optimize large-scale data storage solutions, incorporating AI-driven data lifecycle management, predictive tiering, and automated optimization strategies for enhanced query performance and cost efficiency.
• Implement intelligent monitoring and alerting systems leveraging AI/ML for predictive failure detection, anomaly identification in pipeline performance, and automated root cause analysis, minimizing downtime and simplifying operational troubleshooting.
• Contribute to the strategic design and evolution of data architectures, advocating for and implementing AI-driven optimizations that enhance scalability, efficiency, and resilience of data platforms supporting AI initiatives.
• Enhance CI/CD pipelines for data infrastructure and applications through AI-driven testing, automated code review suggestions, and intelligent deployment strategies, aiming to accelerate delivery cycles and reduce manual intervention.
• Design, develop, and champion the adoption of novel AI-powered tools, frameworks, or extensions that directly automate, simplify, and revolutionize traditional data engineering tasks and operational workflows.

Critical Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a similarly rigorous technical field, alternatively demonstrating equivalent extensive professional experience.
• 5-7 yrs within senior data engineering or closely related data-focused roles.
• Proficient in database technologies, including database processing and performance tuning.
• Strong experience designing efficient data models and understanding Data Warehouse design patterns for maximum scalability.
• Advanced skills in transforming raw data into curated, actionable data elements for downstream consumption. 
• Showcase substantial hands-on experience leveraging cloud-native data services within major cloud environments, specifically including Google Cloud Platform (GCP) services such as BigQuery, Dataflow, Dataproc, Cloud Storage, and Pub/Sub.
• Possess practical, demonstrable experience in implementing and managing data warehousing solutions, applying dimensional modeling techniques (e.g., Star Schema, Snowflake Schema), and designing effective data marts.
• Display exceptional analytical acumen and sophisticated problem-solving capabilities, coupled with an unwavering commitment to data accuracy and meticulous attention to detail in system design and implementation.

Preferred Qualifications

• Relevant certifications (e.g., AWS Certified Data Analytics – Specialty, Google Cloud Certified Professional Data Engineer). 
• Certifications relevant to cloud data engineering or architecture, particularly Google Cloud Professional Data Engineer or Google Cloud Professional Cloud Architect.
  
• Prior successful experience working within dynamic Agile or Scrum development methodologies, contributing effectively to iterative development and rapid delivery cycles.
  
• Demonstrate practical familiarity with MLOps (Machine Learning Operations) principles, practices, and associated tooling for streamlining the machine learning model lifecycle management.
  
• Possess a strong understanding of data streaming paradigms and experience implementing real-time data processing architectures using technologies like Kafka Streams.
  
• Demonstrate proven experience in performance optimization and strategic cost management specifically for cloud-based big data and data processing solutions.
  
• Possess excellent interpersonal, written, and verbal communication skills, effectively conveying complex technical information to both technical and non-technical stakeholders.
  
• Demonstrated ability to effectively process, manage, and integrate data residing in various formats, including structured (e.g., CSV, Parquet, ORC), semi-structured (e.g., JSON, XML), and potentially unstructured data sources within large-scale data pipelines.
• Experience in architecting and implementing intelligent, adaptive data pipelines leveraging AI/ML techniques for automated performance tuning, anomaly detection, and predictive resource allocation, thereby simplifying operational management and enhancing pipeline resilience.
• Prior experience in partnering with AI/ML specialists to co-design and implement robust data infrastructure, feature stores, and model deployment pipelines that streamline the end-to-end lifecycle of AI solutions, making AI development more efficient.