Data Engineer

An exciting opportunity awaits for you to join a forward-thinking government department in Brisbane as a Senior Data Engineer. This role places you at the heart of a transformative data modernisation programme, where your expertise will directly influence the development of an enterprise data platform supporting large-scale ICT transformation.

Roles and Responsibilities

• Design, implement, and maintain robust, scalable, and secure data pipelines and architectures using Microsoft Fabric and related tools within hybrid cloud environments.
• Manage complex data ingestion, transformation, and storage processes in Fabric Lakehouses or Delta Lakes by utilising PySpark, PySQL, and delta format to optimise availability for analytics.
• Apply advanced SQL development techniques for database management, integration tasks, query optimisation, and performance tuning across multiple platforms.
• Monitor and troubleshoot data pipelines proactively to ensure high levels of availability, reliability, and system performance for critical business operations.
• Implement comprehensive data quality management practices to identify issues early, resolve them efficiently, and ensure compliance with governance standards.
• Prepare and normalise large datasets to support statistical analysis and machine learning models that enable advanced analytics and informed decision-making.
• Identify opportunities to enhance analytical capabilities by optimising existing models and improving workflows for greater efficiency.
• Provide mentorship and training to colleagues on best practices in data engineering tools, techniques, and methodologies.
• Create detailed technical documentation covering architecture designs, operational processes, troubleshooting guides, and user access management protocols.
• Support DevOps practices such as continuous integration and delivery (CI/CD) to streamline solution deployments while enhancing platform reliability.

The ideal Candidate

• Demonstrated experience of at least three years in data engineering roles involving Azure Synapse, complex data pipelines, lakehouse architectures, and SQL-based tools such as SSIS or SSRS.
• Proven ability to design scalable solutions within hybrid cloud environments using advanced preparation and transformation techniques tailored for large datasets.
• Expertise in building robust pipelines for ingestion and transformation using Azure frameworks alongside proficiency with PySpark or PySQL.
• Strong command of SQL development including advanced database management skills relevant to integration projects across diverse platforms.
• Familiarity with preparing datasets for statistical analysis or machine learning applications through effective normalisation strategies.
• Solid understanding of implementing rigorous data quality management processes that ensure reliable outcomes aligned with governance requirements.
• Experience supporting business transitions into new analytics patterns—particularly those involving Microsoft Fabric—is highly regarded but not mandatory.
• A collaborative approach when working with stakeholders or project teams during periods of change or transition between project phases.
• Adaptability in responding positively to shifting operational demands within hybrid project-to-operations environments.
• Possession of Microsoft Azure Data Engineer Associate certification or equivalent is highly desirable; additional certifications in SQL or related fields are considered advantageous.

Aboriginal and Torres Strait Islander Peoples are encouraged to apply.
To apply, please click "Apply" or contact Richard Lynch on 07 3032 2295 or richard.lynch@robertwalters.com.au for a confidential discussion.