Lead Data Engineer, Analytics and Information Management (AIM) Project (Ref: CON-18-038)

Tender description

Who Speclialist Work With The Lead Data Engineer will be joining a dedicated Scrum Team responsible for delivering the data platform. The Scrum Team is co-located and currently consists of a Scrum Master, 5 Data Engineers, a Data Architect and a Solution Architect. The Scrum team is part of a broader project team and also works closely with experts from source (data) systems, as well as users of the initial deliverables. The intention is to continue to expand the team organically, bringing in knowledge and experience, whilst maintaining agile values and allowing the team to self organise. What Specialists Work On The Lead Data Engineer will lead the team in: • Building data and Metadata processing using established frameworks. • Performing testing to validate that data is correctly processed and metadata is correctly maintained. • Designing and modelling, ensuring the alignment of semantic models to the FCA Conceptual Models. • Ensuring/demonstrating efficient code execution in the target environment. • Configuring the code orchestration, ensuring monitoring and error handling function in accordance with the FCA designs. • Managing developed artefacts into the FCA repository. Where the Work Takes Place 25 The North Colonnade, Canary Wharf, London EC14 5HS Early Market Engagement Working Arrangements The specialist will be expected to work on-site at the FCA's head office five days a week - though working from home may be permitted where possible. Being on-site will allow a greater level of collaboration with colleagues and allow the individual to attend face-to-face meetings as required. Expenses will only be granted in extraordinary circumstance. As no travel is expected as part of this role, we do not anticipate granting any expenses. Work Location 25 The North Colonnade, Canary Wharf, London EC14 5HS Security Clearance Additional T&Cs Skills & Experience Experience with AWS. Experience with Cloudrea Hadoop Distribution (CDH 5). Implementing data ingestion/presentation/semantics data layers as guided by BA/Data Governance/Conceptual Data models. Using data ingestion/data transformation frameworks and assisting in the development of new frameworks/or framework variants. Skills in data modelling (both structured and unstructured data). Skills in metadata repositories and technologies (Hive / Cloudera Navigator). Skills in data acquisition (landing, ingestion and metadata) of various data types including Salesforce, XML and Relational data. Skills in data manipulation: Java, Scala, Python executing within a Spark environment, orchestrated by Oozie/Talend. Experience with HDFS/Hive/AWS/Sentry Policies/Oozie/ZooKeeper. Experience with Big Data self-describing formats (JSON/Parquet/ORC etc., and when to use them). Experience with Spark clusters, both permanent (Cloudera) and transitory clusters Altus/EMR. Data presentation via Sentry/Impala to visualisation technologies e.g. Tableau / SpotFire. Nice to Haves Data management and analytics. Data discovery processes. Familiar with Scrum and working in an Agile team. Generation of data catalogues/models from the metadata repository and exposing that to various user communities. Skills in Search technologies (Drill/Lucene/Solr). Talend. Developing technical specifications corresponding with architecture, based on business and functional analysis. Capturing and enhancing metadata and using the metadata to generate a Data Catalogue (and the mechanism to expose that to the user community). Definition of Security Policies for the data implemented and the implementation of those policies in Sentry providing Authorisation controls at the required granularity. Experience in large scale distributed processing architectures e.g. enterprising caching, low latency data driven platforms (Enterprise Data Warehouse preferably). Expertise in fault tolerant systems in AWS including Clustering & multi AZ deployments. Working knowledge of setting up & running Big Data clusters (e.g. Hadoop, Spark, Impala) supported by various Databases, object stores (S3) on AWS. Understanding of configuration management and Devops technologies (e.g. GitLab /Jenkins ). BI/Data Prep/Visualisation/ETL tools; Zeppelin, SpotFire, Tableau, StreamSets for large scale dashboards, analysis and reporting. Knowledge of the various flavours of SQL. No. of Specialists to Evaluate 10 Cultural Fit Criteria Work as a team with our organisation and other suppliers Be transparent and collaborative when making decisions Have a no-blame culture and encourage people to learn from their mistakes Take responsibility for their work Share knowledge and experience with other team members Challenge the status quo Be comfortable standing up for their discipline Assessment Method Evaluation Weighting Technical competence 50% Cultural fit 20% Price 30% Questions from Suppliers Budget range