We are looking for an experienced Data Engineer with strong Databricks expertise to design, build and maintain scalable data solutions across a modern data environment.
The successful candidate will be responsible for developing robust data pipelines, integrating data from multiple sources, transforming and preparing data for analytics and downstream consumption, and working closely with architects, analysts and data teams to ensure reliable, high-quality data delivery.
This role requires strong hands-on experience with Databricks, data engineering, data integration and modern cloud data platforms.
What you'll do:
- Design, develop and maintain scalable data pipelines using Databricks.
- Build and optimise ETL/ELT processes for ingesting, transforming and delivering data.
- Integrate data from multiple internal and external sources.
- Develop reliable and reusable data processing frameworks.
- Work with structured, semi-structured and unstructured data.
- Build and maintain data models and curated data layers for analytics and reporting.
- Ensure data quality, integrity, consistency and availability across data pipelines.
- Monitor and optimise data pipeline performance and processing efficiency.
- Troubleshoot data pipeline failures, integration issues and performance bottlenecks.
- Work closely with Data Architects, Data Analysts, Data Scientists and other engineering teams.
- Support the implementation of data governance, security and access-control standards.
- Document data pipelines, transformations, integrations and technical solutions.
- Participate in solution design and architecture discussions.
- Support the migration and modernisation of legacy data workloads onto Databricks where required.
- Ensure engineering solutions follow agreed development, testing and deployment standards.
Your Expertise:
- Proven experience working as a Data Engineer within enterprise or complex data environments.
- Strong hands-on experience with Databricks.
- Strong experience building and maintaining ETL/ELT data pipelines.
- Experience working with Apache Spark / PySpark.
- Strong SQL skills.
- Experience with Python for data engineering and automation.
- Good understanding of data modelling and data warehouse concepts.
- Experience integrating data from APIs, databases, files and other data platforms.
- Experience working with large datasets and distributed data processing.
- Strong understanding of data quality, data validation and data transformation principles.
- Experience with version control and modern software development practices.
- Strong analytical and problem-solving skills.
- Ability to work closely with technical and business stakeholders.
Databricks Experience
The successful candidate should have practical experience with several of the following:
- Databricks notebooks and workflows
- Databricks Jobs / Workflows
- Delta Lake
- Delta Tables
- Medallion Architecture – Bronze, Silver and Gold layers
- Unity Catalog
- Databricks SQL
- PySpark / Spark SQL
- Data ingestion and transformation within Databricks
- Performance optimisation and cluster configuration
- Databricks security and access controls
- CI/CD deployment of Databricks solutions
- Data quality and monitoring within the Databricks environment
Advantageous Experience:
- Experience with Azure Databricks and the wider Microsoft Azure data ecosystem.
- Azure Data Factory.
- Azure Data Lake Storage.
- Microsoft Fabric.
- Azure Synapse Analytics.
- Experience with AWS or other cloud data platforms.
- Kafka or other streaming technologies.
- Experience with real-time or near-real-time data pipelines.
- Exposure to data governance and metadata management.
- Databricks certifications would be advantageous.
- Experience working within Agile delivery environments.
Qualifications
- Relevant degree or diploma in Computer Science, Information Technology, Data Engineering, Software Engineering or a related field.
- Relevant Databricks, cloud or data engineering certifications would be advantageous.
Ideal Candidate
The ideal candidate is a hands-on Data Engineer who is comfortable working within a modern Databricks environment and can take ownership of data pipelines from ingestion through to consumption.
They should understand not only how to build data solutions, but also how to make them scalable, reliable, maintainable and performant. The candidate should be comfortable collaborating with architects and wider data teams while still being technically strong enough to independently develop and troubleshoot complex data engineering solutions.
Other information applicable to the opportunity:
- 12-month Contract position
- Location: Johannesburg
Why work for us?
Want to work for an organization that solves complex real-world problems with innovative software solutions? At iOCO, we believe anything is possible with modern technology, software, and development expertise. We are continuously pushing the boundaries of innovative solutions across multiple industries using an array of technologies.?
You will be part of a consultancy, working with some of the most knowledgeable minds in the industry on interesting solutions across different business domains.?
Our culture of continuous learning will ensure that you will have all the opportunities, tools, and support to hone and grow your craft.?
By joining IOCO you will have an open invitation to developer inspiring forums. A place where you will be able to connect and learn from and with your peers by sharing ideas, experiences, practices, and solutions.?
iOCO is an equal opportunity employer with an obligation to achieve its own unique EE objectives in the context of Employment Equity targets. Therefore, our employment strategy gives primary preference to previously disadvantaged individuals or groups.