Data Engineering- Data Services and Analytics

Tender description

Why the Work is Being Done Data Services and Analytics is a centralised service within the Home Office's Digital Data and Technology's Enterprise Services, and is at the forefront of efforts to re-imagine the departments relationship with data. It leads the way in professionalising our approach to the acquisition and analysis of data, promoting dynamic insight-led and evidence based decision making tools and techniques. The Home Office Data Engineering Team is an operationally critical capability delivering regular ingests of multiple data sources to downstream products and services. As the front end capability in DSA it is highly visible and the one service upon which all other downstream services rely. Problem to Be Solved Data Services and Analytics is in the process of implementing a corporate technology strategy that will see existing 'on premises' platforms migrate to the Cloud. The key challenge is maintaining continuity of service whilst designing, implementing and operating new bulk and fast data pipelines to serve downstream consumers and provide enhanced capability. New capability is being built around AWS Cloud services and bespoke java based platforms. The Home Office is therefore looking for a supplier with knowledge of transitioning to and operating ETL in commodity Cloud-based environments whilst being able to support existing pipelines based primarily on SAS-Data Integration Studio and ActiveEon. Who Are the Users The users are either: Data Scientists and Data Analysts within the Home Office who explore and use data to provide analysis, insight, reporting and data products for their customers, to support business operations, policy making and decision making. OR End Business Users elsewhere in Government who use DSA Data Products. Early Market Engagement Work Already Done Development of the current 'on premises' capability to ingest and process data sets to business requirements. Construction of a new Cloud based common landing zone for all data consumed by the DSA. Orchestration, prioritisation and automation of data ingest. Development of a new Cloud based "fast data" streaming capability performing matching, ETL and data transformation for consumption. Existing Team The existing Team comprises eleven data engineers made up of three Home Office Permanent Civil Servants, two Graduates supplied to the Home Office through an Academy programme and six from a contracting partner. Current Phase Beta Work Location The work will be carried out from a Home Office location primarily in Croydon, as required on other Home Office and Supplier sites. Occasional working from home may be permitted with prior agreement from the Data Manager and in line with current Home Office policy. Working Arrangments The supplier is expected to provide a team of specialists to deliver the outcomes. The Supplier must be flexible and make working with a mixed team across multiple sites effective. Typically we expect resources to be on site 5 days a week. The team will work from the principal location as required Monday to Friday 0900-1700, although some out of hours support may be required occasionally. Travel to regional sites may be required, expenses incurred from travel outside of the M25/Greater London area will be subject to Home Office Travel (no Subsistence) Policy. Security Clearance All resources must hold, or be prepared to undergo, SC security clearance to operate on the project. (with willingness to acquire resources with NPPV3 and DV clearance). Additional T&Cs The work will be progressed by a number of Statements of Work which may be fixed price, T and M or Capped T and M as appropriate. Standard DOS framework and call off terms and conditions will apply. Resources cannot charge for more than 100 working days in any consecutive six months or more than 200 working days in any consecutive year. Skills & Experience Have comprehensive experience of ETL data processing across multiple datasets including expertise in Scheduling, Orchestration, Prioritisation and automation of data pipelines. Have expertise in the use of Structured Query language (SQL). Be able to demonstrate experience in change data capture approaches to simplifying data movements within data pipelines. Expertise in the development, troubleshooting and maintenance of Python code. Demonstrable experience in loading and managing large volumes of data including both structured/unstructured sources in many formats including (but not exclusively) JSON, XML, SQL Server, Oracle, Access and MySQL. Demonstrable experience of enabling data governance criteria for Data access, deletion/removal and general compliance. Experience in loading streamed data sources and related Cloud based technologies. Expertise in the development, troubleshooting and maintenance of Java Code (required to support core platforms). Experience in developing and maintaining ETL with SAS Data integration Studio tooling. Nice to Haves Maintain and develop Visual Basic scripting. Maintain and use ActiveEon (Scheduler). Use Postgres databases. Demonstrate capability in Pivotal GreenPlum MPP database. Manage data in an AWS Cloud environment. Demonstrate experience of operating in the Public Sector environment. Demonstrate expertise in the bulk and incremental data load to Cloud native services including simple cloud storage and structured/unstructured data repositories (Graph DB, RElational DB, Data warehousing, Data indexing/search). SC clearance already in place. Have experience in Kafka message queuing systems. Have experience in containerization technologies such as Docker. No. of Suppliers to Evaluate 4 Proposal Criteria Approach and methodology Technical skills and experience Service Model Mobilisation Plan A plan for optimising resources. Identification of risks and dependencies. Value for money Cultural Fit Criteria Work collaboratively with all teams in DSA to ensure smooth handoffs in an environment of continuous improvements and changes. Ability to function effectively and collaborate in a multi-supplier environmant. Work successfully in complex Public Sector organisations. Approach to proactive issue management, problem resolution and improving ways of working. Focus on achieving value for money in all activities. Blend rapidly and efficiently into existing methodology and constantly deliver continual improvements to raise delivery standard. Payment Approach Capped time and materials Assessment Method Case study Work history Reference Presentation Evaluation Weighting Technical competence 60% Cultural fit 5% Price 35% Questions from Suppliers No questions have been answered yet Budget range any submitted day rates within the contract must not exceed £1,000 (exc vat). the buyer does not anticipate sfia level 6 or 7 resources to be used for delivery of this contract. should the parties exceptionally agree any sfia level 6 or 7 resources for limited use (e.g client workshops), the day rates for such resources will remain under the cap of £1,000 (exc vat). responses must include maximum billed day rate in sfia rate card submission form.