Learner Data Service - Engineering Resource - DfE

Tender description

Summary of the work We require a development partner who will provide the technical capability on the Learner Data Service data and technical projects as part of the DfE’s Data Transformation Portfolio. You will provide expertise and specialism to boost capacity and enhance capability, working alongside our in-house digital and data teams supporting delivery. Expected Contract Length 1year Latest start date Wednesday 1 March 2023 Budget Range £7,500,000.00 (incl. VAT) Why the Work is Being Done The Department has a cross cutting priority to embed data and digital throughout our work starting with improving data access, transparency, and quality to inform policy, operational delivery, and analytical activity. To support delivery of this priority we need to implement a ‘single thread’ approach for supply of data and that there are set patterns and approaches for managing operational and analytical data in relation to the education sector. Problem to Be Solved Establishing the core capabilities for operational and analytical data management and production. Heading up the build and engineering of the core capabilities utilising the Department’s annual process for collecting, matching, processing, validating, and checking the Department’s business critical datasets on learner participation and achievement, children, and early years as a use case. Who Are the Users As an analyst, I need data that details learners and their attainment so that I can evaluate a school or colleges performance and assess their learners’ achievements. As the head of data ownership and sharing, I need to limit the sharing and visibility of personal identifiable data so that we protect learners’ identities and only share when there is a permitted purpose. As a data operations engineer, I need access to data that shows where a learner is registered for education, and their characteristics so that I can undertake further data production activities to enable the calculation of funding for schools and colleges at the provider, and learner level (e.g. pupil premium and national funding formula). As a data operations engineer, I need to ensure the National Pupil Database is maintained so that we can share data within the department, across government, the sector and to the wider public domain. Early Market Engagement N/A Work Already Done Work is currently underway to establish and implement 4 assets by March 2023 – an operational data store, a matching service, a reference data store, and an analytical data store. These are being developed for the scope of primary education level learner and achievement and attainment data. Following this activity, we will continue to iterate on these assets and ensure the inclusion of secondary, 16-18- and 16–24-year-old learner and achievement and attainment data. Existing Team Our build-test-deploy delivery model is underpinned by an agile at scale build capability. Existing teams are comprised of multi-disciplinary agile delivery skills. These teams are involved in service development for providing new and legacy replacement data services. There are 10 teams with a typical team size is up to 10 with a skill mix such as product managers, delivery managers, scrum masters, business analysts, design professionals (data analysts and data modellers), data engineers, and test engineers. Solutions and data architects support cross programme delivery through setting strategic design direction as well as supporting build team decision making. Current Phase Beta Skills & Experience • Has previously undertaken data analysis, data engineering and test engineering for learner data services and/or products in the Department including the National Pupil Data Achievement and Attainment Data Tables (NPDAAT). • Technical agile delivery experience in end-to-end design, development, test and deployment of complex CI/CD data pipelines, using Informatica DEI, ADF & Databricks. • Experience and skills in delivering new cloud data platforms, covering ingestion, transformation, validation, cleansing, matching and storage in a complex, high risk environment, to tight timescales • Experience and skills in developing full stack digital services using C#.net, REST API build and integration and SQL/non-SQL databases in a cloud environment. • Experience and skills in delivering large digital and data programmes successfully, for example to GDS guidelines, progressing successfully through different phases of the delivery lifecycle including governance and assurance processes • Experience and skills in delivering automated CI/CD integrated data and software testing in large, complex, data intensive organisations within central government or regulated sector using Informatica DEI and ADF tools. • Experience and skills in delivering data lakes, data warehouses, streaming data ingestion and processing, integration with SQL/NoSQL databases, structured, semi-structured, and unstructured data sources, temporal data and time-series analysis • Experience and skills in working well in large organisations, across organisational boundaries, ideally in government and with multi-agency departments, delivering domain driven service design/architecture Nice to Haves • Awareness and understanding of the Education sector or Equivalent sector • Awareness and understanding of educational datasets or Equivalent sector Work Location Manchester (and additionally Newcastle, Sheffield, Darlington, Coventry, London and Leeds ) Working Arrangments This will be a 5 day per week (Monday to Friday) commitment and detailed working arrangements, including team location, size and make-up, will be contained within each statement of work. Day to day delivery will be governed by standard agile delivery techniques. These will be via the scrum approach and the standard agile ceremonies will be undertaken. These will be monitored by the Delivery Manager and the Product Owner will be responsible for prioritising backlogs in refinement sessions. Peer reviews by technical leads will also occur regularly to ensure fit to best practice and engineering standards. Security Clearance Proposed team members will have been subject to Baseline Personnel Security Standards checks. However higher levels of Security Clearance maybe required dependent on work agreed. Suppliers will also be required to complete a supplier security assurance form to ensure they meet the required standard for DfE e.g. Cyber Essentials Additional T&Cs Standard Framework and Call Off Terms and Conditions. Expenses must be pre-agreed and comply with CCS Travel and Subsistence Policy. Any expenses shall be submitted in line with DfE standard T&S policy if appropriate. Primary work location stated in SoW will not attract expenses. Contract and Vendor Management will form a key part of governance and suppliers will be expected to complete cyber security questionnaire and Financial Viability Assessment. Suppliers must provide sufficient guarantees to meet the requirements of GDPR in line with Procurement Policy Note 03/17 Changes to Data Protection Legislation & General Data Protection Regulation No. of Suppliers to Evaluate 5 Proposal Criteria • Within the last three years to have experience in end-to-end design, development, and agile delivery of data products/services to high quality standards (in a digital, data and technology driven environment) • Within the last three years have experience in engineering of new data platforms and data migration projects in a complex and high-risk environment • Within the last three years have demonstrable experience in developing data products (data analysis, data engineering, test engineering) • Proven experience of using a range of engineering tools e.g. Informatica DEI, ADF & Databricks • Evidence of coaching, building capability and upskilling of resources • Manage full build test deploy lifecycles including data analysis, data engineering and test engineering, and continuous improvement of one or more transactional data services or platforms including delivering at pace • Evidence of demonstrable experience within the last three years in working on high-profile complex, multi-stakeholder digital, data and technology programmes • Evidence of establishing an Agile delivery capability in both data and software delivery lifecycles • Experience of working with an organisation at the beginning of an agile culture change • Experience of transferring knowledge to permanent staff within a client organisation Cultural Fit Criteria • Demonstrate ability to work as a team and collaborate effectively with our organisation and other suppliers. • Demonstrate approach to knowledge and skills transfer to build in-house capabilities and ensure effective knowledge management and transfer within agreed timescales. • How will your organisation support educational attainment relevant to the contract, including training schemes that address skills gaps and result in recognised qualifications (e.g. , Apprenticeships, Industry placements, Traineeships) (5%) • Describe how your organisation encourages diverse representation of under-represented groups in the workforce, e.g. Women, Black, Asian and Minority Ethnic, Disabled, LGBTQ+ and how you monitor and measure this (5%) Payment Approach Capped time and materials Assessment Method • Case study • Work history • Reference • Presentation Evaluation Weighting Technical competence 50% Cultural fit 15% Price 35% Questions from Suppliers 1. If there is an incumbent supplier, who is this? Yes, there is an incumbent supplier. The supplier is Transform. 2. Is there an incumbent supplier who has delivered the Alpha?Is this contract meant to pick up from the incumbent has left or is the new supplier meant to be working alongside the incumbent and other resources? Yes, there is an incumbent supplier who has delivered Alpha, and part of the Beta work. This contract is to pick up from the incumbent supplier. 3. We note your requirement of:Has previously undertaken data analysis, data engineering and test engineering for learner data services and/or products in the Department including the National Pupil Data Achievement and Attainment Data Tables (NPDAAT).1. Is there an incumbent / are there existing providers who have been providing these / similar services?2. Could you please help us understand your requirement. We’ve previously undertaken data analysis, data engineering and test engineering in DfE and with similar organisations. However, your requirement makes it appear as if specific experience in Learner Data Services is required. Could you explain why? Yes, there are incumbent suppliers who have been providing these / similar services. This is a specific programme of work where knowledge of the learner data domain and data products handling learner data within the Department for Education is essential. 4. Can you give an idea of approximate number of resources required for the this work. Between 60 and 90. 5. You have release 3 x DOS tenders (Engineering, Delivery and Architecture) all require essetnial expereince for “undertaken data analysis, data engineering and test engineering for learner data services and/or products in the Department including the National Pupil Data Achievement and Attainment Data Tables (NPDAAT)”So are these closed off procurements for only suppliers that have worked within learner data services? This is a specific programme of work where knowledge of the learner data domain and data products handling learner data within the Department for Education is essential 6. Is the intention to spend £7.5 million within Fy23/24? Is their a minimum spend commitment? There is no minimum spend commitment 7. Is Informatica DEI a ‘must have’ technology or is it just being used as an example alongside ADF and Databricks. It is a must have technology.