PU-20-0089 ONS Matching System

Tender description

Why the Work is Being Done The ONS is aiming to publish outputs from the 2021 Census within a year of collection. To achieve this, we must have completed a matching exercise within an 8-week timeframe. The matching exercise itself involves matching Census data with a follow-up coverage survey (e.g. by name, sex, date of birth, address etc.) to identify if we have over or under-counted different parts of the population, which influences the final statistical outputs we produce. Problem to Be Solved We require a capability to enable the matching of records from two datasets and assure that the quality of the matches meet strict targets. The immediate need relating to successful delivery of the 2021 Census is for a solution to be ready for testing in March 2020 to provide confidence that we will be able to complete a matching exercise on Census datasets within an 8-week timeframe during 2021. A secondary need relates to potential re-use of the capability for wider ONS business needs beyond the Census with other datasets. Who Are the Users As a clerical matcher, I need to compare two or more records that have been marked as potential matches by an automated algorithm, so that I can determine if they really are the same person or not. Early Market Engagement The ONS conducted an early market engagement, hosting two webinars with the opportunity for suppliers to ask questions about the procurement. Links to the recordings for the webinars, a write-up of the questions and answers as well as a summary of the topics discussed can be found on the ONS InTend site: https://in-tendhost.co.uk/ons/ Registration is required, but free. Once complete, please search ‘Clerical Matching’ within the ‘Current Opportunities’ area and register your interest. We will confirm you are registered on the DOS3 Framework prior to enabling access to the documents. Work Already Done For Census, the ONS methodology team have developed an automated matching algorithm that has been assured, including by external panels, to provide confidence we can match 91% of records automatically. We need to use this algorithm for the automated matching step for Census. Generally, this either means integrating the algorithm into a matching system or the ONS running the automated matching and the system utilising the output (we would need to agree the interface/format specification for this). For wider ONS use (outside of Census), different algorithms may be used and/or we could utilise existing/off-the-shelf automated matching within a system/product. Existing Team The ONS has a data integration team that the supplier will be working with. This team consists of experts in record matching with a deep knowledge of the matching problems we need to solve and the quality targets that need to be met. Current Phase Not started Work Location Office for National Statistics Segensworth Road Titchfield Fareham Hampshire PO15 5RR Working Arrangments The expectation is that some time on site will be required, to work with the users, testing and understanding requirements. It is generally not a problem for the supplier to work from their own location if we can schedule some face-to-face time. Face-to-face time would primarily be at our Titchfield site with some limited time needed in Newport. See https://www.ons.gov.uk/aboutus/contactus/officelocations Security Clearance There may be some SC and BC (BPSS) clearance required, we also utilise CTC (counter terrorism check) for some roles. Generally staff accessing Census data will need to be SC cleared. Additional T&Cs Skills & Experience Expert knowledge of data matching processes and algorithms (6 points) Experience of storing/retrieving/displaying large volumes of multi-page images/scanned documents (PDFs) (3 points) Evidence of working with large volumes of data (the Census is circa 60 million records) (4 points) Experience with fuzzy-match based searching (4 points) Evidence of developing/configuring/optimising user interfaces for displaying data records (5 points) Experience of integrating simple case and workflow management (3 points) Evidence of support for data capture as part of a user workflow (e.g. storing case notes against decisions) (3 points) Experience of systems able to scale to meet increases in concurrent usage (4 points) Evidence of working with both Welsh (or international) and English data and typefaces (4 points) Experience of working with highly sensitive data and the associated assurance requirements (4 points) Knowledge of the latest NCSC guidance and principles (2 points) Experience of public, private or hybrid cloud hosting architecture & solutions (5 points) Experience integrating solutions with other systems and architecture (4 points) Evidence of applying the best practice principles of the Government Service Standard (4 points) Evidence of gathering user needs in accordance with the Government Service Standard (4 points) Evidence in user centred design (4 points) Proven record of effectively using agile methods to design, build and deliver (4 points) Experience developing production-ready solutions incrementally (3 points) Experience running and supporting an operational system (5 points) Experience of logging, auditing and monitoring of system and user activity in accordance with agreed metrics (4 points) Nice to Haves Applying user research insights to product development (1 point) Experience of system migration/install (e.g. to/from Cloud to private data centre) (2 points) Knowledge of Apache Hadoop based stacks (HDFS, Hue, Hive, Impala, Spark) (2 points) Knowledge of Python and PySpark (1 points) Experience of VMWare/VSphere (1 point) Knowledge of working with geospatial data (2 point) Experience working with APIs (1 point) Experience of different data formats (e.g. CSV, JSON, Avro, Parquet) (1 point) No. of Suppliers to Evaluate 5 Proposal Criteria How the approach or solution will meet the needs of our users (4 points) How the approach or solution meets our goals and targets (4 points) Estimated timeframes for the work (4 points) Identification of risks and dependencies and proposed approaches to managing them (4 points) Value for money (4 points) Cultural Fit Criteria Be transparent and collaborative when making decisions (4 points) Have a no-blame culture and encourage people to learn from their mistakes (4 points) Take responsibility for their work (4 points) Challenge the status quo (4 points) Be comfortable standing up for their discipline (4 points) Can work with clients with low technical expertise (2 points) Payment Approach Capped time and materials Assessment Method Written proposal Case study Work history Reference Presentation Evaluation Weighting Technical competence 70% Cultural fit 10% Price 20% Questions from Suppliers Budget range