Salary: £? - ? per year Requirements:

Strong hands-on experience with Databricks, PySpark, AWS, SQL, and Python.
Databricks experience with notebooks, scheduled jobs, cluster configuration, driver and executor logs, Spark UI, and performance troubleshooting.
Experience with Spark SQL, Delta Lake, Parquet, and Hive metastore tables, including schemas, partitions, and the relationship between table metadata and underlying files.
Hands-on AWS experience, particularly with S3, IAM, and CloudWatch; understanding of role-based access, KMS encryption, and diagnosing data access failures.
Working knowledge of Git, code reviews, CI/CD, and controlled production deployments, including testing, rollback, and release validation.
Advantageous: experience with AWS Glue, Lambda, Step Functions, Linux, shell scripts, Terraform, GitLab, or Jenkins.

Responsibilities:

Run and maintain production data pipelines on Databricks and AWS.
Keep scheduled data processing reliable by resolving incidents, applying fixes, and improving data services performance.
Support and maintain production data pipelines, including incident investigation, safe recovery, root cause analysis, and permanent remediation.
Own incidents from investigation through recovery and closure; diagnose issues across Python, PySpark, SQL, Databricks jobs, and AWS integrations.
Provide clear progress updates and escalate issues in a timely manner.

Technologies:

AWS
AWS Glue
Lambda
CI/CD
CloudWatch
Databricks
Git
GitLab
Hive
IAM
Support
Jenkins
Linux
Python
PySpark
SQL
Spark
Terraform
UX UI Design
Cloud

More:

We are seeking a Data Engineer for a large-scale government project as part of a global IT transformation. This is a six-month contract based in Leeds, with a hybrid working model: primarily working from home, with travel to our Leeds office one to two days per month. The role combines L2/L3 production support with data engineering work and is due to start as soon as possible in October 2026. The rate is listed as £ per day, Inside IR35.

last updated 44 week of 2026

Data Engineer (Databricks)

Start with your email