Role Overview DescriptionThe Senior Data Engineer will work collaboratively with Data Science and Analytics teams to build scalable, robust, and efficient data pipelines to enable data-driven decision making.Responsible for designing data architectures, building/maintaining data pipelines, and integrating and transforming data from various sources to maximize accessibility and usability.Roles & ResponsibilitiesDesign reliable and scalable data architectures and pipelines for streaming and batch dataDevelop complex ETL processes to extract, transform and load data from various sources to destinationsWork on data quality monitoring and optimization processesCollaborate with data scientists and analyst to understand their data and analytics needsQualifications5+ years of experience as a Data EngineerExpert level knowledge of SQL, Python and PySparkExperience in Airflow is preferredStrong software engineering fundamentals including systems design, algorithmic thinking and complexity analysis
Job Title
Edstem Technologies | Senior PySpark Data Engineer