DAC053 - Data Engineering Support

Tender description

Why the Work is Being Done The Data Analytics Competency Centre has the capability to obtain, load and analyse data sets from Home Office, other Government departments and Commercial partners to provide aggregated, more accurate and timely data to users. Current overall data volume is 21 Tb, from around 20 sources and is expected to double in size over the coming 12 months with data deltas and new sources. Problem to Be Solved The current capability has reached a level of maturity, and project now needs to build the capability to :- • Improve current loads and capability from existing data • Build new capabilities for new data sets • Migrate other existing HO services into the DACC • Migrate the DACC capability from Beta to LIVE • Review the technologies in use with a view to future proofing the capability and moving away from commercial products to more Open Source vendors. • Knowledge transfer plan. Who Are the Users Users are principally Data Analysts and Scientists within the Home Office. Customers range across Home Office departments and output from the DACC are supplied to other parts of Government. Early Market Engagement Work Already Done Work completed so far includes :- • Choice of software component which are now under review for replacement • Development of the current capability to ingest and process data sets to business requirements. • Orchestration, Prioritisation and automation of data ingest. • Development of a core set of skills, knowledge and capability within the Home Office and its contracting partners. Existing Team Existing team is composed of 2 Home Office employees and a flexible group supplied by a contracting partner. Current Phase Alpha Work Location Supplier will be based at the DACC Lab, Lunar House, Wellesley Road, Croydon. Occasional travel to regional sites may be required, expenses incurred from travel outside of the M25/Greater London will be subject to Home Office Travel and Subsistence Policy. Working Arrangments The supplier is expected to stand up a team made of up of the specialist skills required to deliver the outcomes. We expect the supplier to be flexible to our needs and make working with a mixed team across multiple sites effective. In general we would expect resources to be on site 5 days a week, although some out of hours support may be required from time to time. Security Clearance All staff must hold or be prepared to undergo SC Security Clearance to operate on this project, (with willingness for some resources to undergo NPPV3 and DV clearance) Additional T&Cs • The work will be progressed by a number of SoWs which may be fixed priced, Capped T&M or T&M as appropriate. • Standard DOS framework and call-off terms and conditions. Skills & Experience Comprehensive experience of ETL data processing Comprehensive experience of using MPP database(s) Expertise in the use of Structured Query Language (SQL) Expertise in Orchestration, Prioritisation and automation (some home built capability) Expertise in the development, troubleshooting and maintenance of Python code Comprehensive capability to deal with structured and unstructured data. In many formats including (but not exclusively) JSON, XML, SQL Server, Oracle, Access, MySQL Proven ability to provide support within a fast paced operational environment Demonstrable experience of improving efficiency and optimising processing cycles Demonstrable experience in loading and managing large volumes of data Demonstrable experience of enabling data governance criteria for Data access, deletion/removal and general compliance. Nice to Haves Capability to maintain and develop Visual Basic scripting Capability to maintain and use ActivEon (Scheduler) Expertise in the use of Postgres databases Expertise in Pivotal GreenPlum MPP database Experience in developing and maintaining ETL with SAS Data Integration Experience of operating in a cloud hosted environment Experience in loading streamed data sources and the related technologies. Experience of operating in Public sector environment No. of Suppliers to Evaluate 3 Proposal Criteria Approach and methodology Technical skills and experience Service model Mobilisation plan A plan for optimising resources Identification of risks and dependencies Cultural Fit Criteria Ability to function effectively and collaborate in a multi-supplier environment Working successfully in complex public sector organisations Approach to proactive issue management, problem resolution and improving ways of working Payment Approach Time and materials Assessment Method Written proposal Reference Presentation Evaluation Weighting Technical competence 65% Cultural fit 5% Price 30% Questions from Suppliers Budget range