DfE Data Engineering Delivery Capability

Tender description

Summary of the work We require a development partner who will be able to provide the technical capability on digital and technical projects as part of the DfE’s Data Transformation Portfolio. You will provide expertise and specialism to boost capacity and enhance capability, working alongside our in-house digital and data teams supporting delivery. Expected Contract Length 2 Years Latest start date Monday 11 April 2022 Budget Range Up to £4,500,000 ex VAT Why the Work is Being Done Our Data Directorate is undergoing a transformation programme to cultivate our in-house skills. This will help us achieve our strategic goal to offer ‘Data as a Service’, modernising our data collection services and laying foundation for good data governance practices; all of which will allow delivery of high-quality data into our estate enabling greater insight into the sector as well as drive data driven policy and operational decision-making. Problem to Be Solved The DfE Data Directorate is looking for a supplier to support a range of user-led data and digital projects, including both existing and new projects, from inception through to live. The supplier will work with blended teams, using agile to deliver outcomes based on user needs and, where applicable, government service standards. The supplier should have a good grounding in large, complex data delivery programmes covering data ingestion, cleansing, transformation and data matching, as well as strong experience in digital delivery to GDS standards. The data pipeline projects will predominantly require: Data Engineers skilled in Informatica DEI and Azure Data Factory/data bricks; QA automation testers (Skills: Azure, Specflow, Rest Sharp, C#), Cloud Infrastructure Engineers who are skilled in Azure and Data Analysts with data visualisation skills. The Digital Projects will predominantly require: Software Developers skilled in C#. net but with some capabilities required in REACT, JAVA; QA automation testers QA automation testers (Skills: ETL Testing, Specflow, Rest Sharp, C#), cloud infrastructure engineers skilled in Azure and DevOps engineers. Typical projects include supporting DfE's policy and operational response to the pandemic and recovery by bringing more attendance data rapidly into the department to fill critical gaps. Who Are the Users This contract will cover a range of data services for which we have a wide range of internal and external users who contribute to and consume our data pipelines and Digital services. Internal users include policy teams, Departmental analysts and corporate services access our data through analytical data stores and Bi reports. Other Internal Users include: Analysts, Policy team, Operations and Service Delivery teams. e.g. As a data collector working in Data Engineering and Operations division, I need to setup data collection in line with department policy External Users including: Academies and Multi Academy Trusts Schools Local Authorities (Education & Children's Services) e.g. As an academy I need to be able to submit and validate my school census return and check my data Work Already Done There are a number of projects and programmes in flight, that are at various stages of the delivery lifecycle requiring data engineering, software engineering, test engineering and Infrastructure resources. These range from the improvement of existing developed systems to transformation leading to new systems. Examples of completed and ongoing work includes • Search for schools and colleges to compare https://www.compare-school-performance.service.gov.uk/ • Get Information about Schools https://get-information-schools.service.gov.uk/ • Get Information about Pupils Existing Team The supplier will work with blended project teams comprising of permanent civil servants and other supplier teams across multiple strands of work for the data engineering service delivery. Typically, within agile delivery teams, working with service owners, product managers, delivery managers, business analysts, user researchers, architects, data and software engineers, Infrastructure Engineers, QA testers and users. You may collaborate or work with other suppliers who are providing capabilities as part of a team. Our multi-disciplinary agile teams stretch across 9 squads and are typically sized between 8 and 10. Current Phase Not applicable Skills & Experience • Technical agile delivery experience in end-to-end design, development, test and deployment of complex CI/CD data pipelines, using Informatica DEI, ADF & Databricks • Experience and skills in delivering new cloud data platforms, covering ingestion, transformation, validation, cleansing, matching and storage in a complex, high risk environment, to tight timescales. • Experience and skills in developing full stack digital services using C#.net, REST API build and integration and SQL/non-SQL databases in a cloud environment. • Experience and skills in delivering large digital and data programmes successfully, for example to GDS guidelines, progressing successfully through different phases of the delivery lifecycle including governance and assurance processes • Experience and skills in delivering automated CI/CD integrated data and software testing in large, complex, data intensive organisations within central government or regulated sector using Informatica DEI and ADF tools. • Experience and skills in delivering data lakes, data warehouses, streaming data ingestion and processing, integration with SQL/NoSQL databases, structured, semi-structured, and unstructured data sources, temporal data and time-series analysis • Experience and skills in delivering multi-strand digital and data programmes, setting up DevOps and DataOps capability and CI/CD pipelines • Experience and skills in using Infrastructure as Code (IaC) - YAML pipelines and setting up test automation from scratch within a complex data organisation Nice to Haves • Awareness and understanding of the Education sector • Awareness and understanding of educational datasets Work Location Manchester (and additionally Newcastle, Sheffield, Darlington, Coventry, London and Leeds ) Working Arrangments For any given SoW the work is likely to be based in Manchester, Darlington, Coventry, Sheffield or London some research will be required in other locations across England. Travel may be required to DfE offices in these locations. Primary base will be detailed in each Statement of Work. Expenses will not form part of the cost unless the supplier is asked to work from a secondary location during the execution of the work relating to the SoW. e.g. SoW primary base is Darlington, no expenses will be paid for Darlington. It's expected that this contract will predominantly sit within IR35. Security Clearance Proposed team members will have been subject to Baseline Personnel Security Standards checks. However higher levels of Security Clearance maybe required dependent on work agreed. Suppliers will also be required to complete a supplier security assurance form to ensure they meet the required standard for DfE e.g. Cyber Essentials Additional T&Cs Standard Framework and Call Off Terms and Conditions. Expenses must be pre-agreed and comply with CCS Travel and Subsistence Policy. Any expenses shall be submitted in line with DfE standard T&S policy. Primary work location stated in SoW will not attract expenses. Contract and Vendor Management will form a key part of governance and suppliers will be expected to complete a security questionnaire. Suppliers must provide sufficient guarantees to meet the requirements of GDPR in line with Procurement Policy Note 03/17 Changes to Data Protection Legislation & General Data Protection Regulation No. of Suppliers to Evaluate 5 Proposal Criteria • Within the last three years to have experience in end-to-end design, development, and agile delivery of digital products/services to high quality standards (in a digital, data and technology driven environment). • Within the last three years have experience in delivery of new data platforms and data migration projects in a complex and high risk environment. • Within the last three years have demonstrable experience in user-centred design services (user research, content design and interaction design and business analysis) • Proven experience of using a range of agile tools e.g. Azure Dev Ops and techniques including scrum, working in multi-team agile programmes • Evidence of coaching, building capability and upskilling of resources • Manage full product lifecycles including business analysis, user research, design, delivery, and the continuous improvement of one or more transactional services or platforms including rapid prototyping and delivering at pace • Evidence of demonstrable experience within the last three years in working on high-profile complex, multi-stakeholder digital, data and technology programmes • Evidence of establishing an Agile delivery capability in both data and software delivery lifecycles • Experience of working with an organisation at the beginning of an agile culture change • Experience of transferring knowledge to permanent staff within a client organisation Cultural Fit Criteria • How will your organisation support educational attainment relevant to the contract, including training schemes that address skills gaps and result in recognised qualifications • Describe how your organisation encourages diverse representation of under-represented groups in the workforce, e.g. Women, Black, Asian and Minority Ethnic, Disabled, LGBTQ+ representation and how you monitor and measure this Payment Approach Capped time and materials Assessment Method • Work history • Reference • Presentation Evaluation Weighting Technical competence 50% Cultural fit 10% Price 40% Questions from Suppliers 1. Who is the current incumbent? There is no incumbent supplier. This is a new contract through the DOS5 framework. 2. Regarding "Nice to have skills" - 'Awareness and understanding of the Education sector, Awareness and understanding of educational datasets.' Will you exclude suppliers with no past experience in the educational sector, but technically competent to deliver ? Suppliers will not be excluded if they have no past experience of the educational sector, but this will be taken into account when evaluating competing bids. 3. Is this a Time and Material project to a Fixed bid? No, the contract will have a fixed upper limit but we will be using a capped time and materials costing methodology in each Statement of Work. 4. Does this project support an offshore delivery model ? No, the resource must be UK based. 5. How many DDaT roles do you expect to fulfil? The number of roles will be project dependent and therefore the supplier must be able to flex up and down as needed. Estimated up to 40 at the maximum level. 6. To what extent is remote working within UK acceptable? There is likely to be a mix of remote and office working. Each Statement of Work will set out the specific working arrangements. 7. Can offshore resource be considered for development given the security requirement? No. Resource must be UK based. 8. Could DfE provide a view of the initial team required at inception, and how they intend to call off additional resources / adjust the team composition over time under call-off? This will be discussed in detail at the initial meeting with the successful supplier. There's an expectation that the successful supplier will be able to flex resource up and down as required. 9. Would alternates be considered to C# net from an API build? Would you accept platform based informatica/IBM API solutions? We are unable to answer this question as it lacks enough context. 10. Technical Agile Delivery experience – what is the weighting on the engagement, is it more with Dev-Ops or is it capabilities in product (DEI, ADF & Databricks)? We are unable to answer this question as it lacks enough context. 11. Can you expand on what you mean in regards to a “high risk environment” in question 2? We are unable to answer this question as it lacks enough context. 12. Can you clarify what you mean in terms of “Multi – strand digital and data programmes”. What would “strands” be in this context? We are unable to answer this question as it lacks enough context and background.