AWS Data Transformation Pipeline Service for Court Judgment documents

Tender description

Summary of the work Design, implement, test and document an AWS data-transformation infrastructure and pipeline service to securely connect our systems. The service will retrieve, transform and send data according to business rules. Initial delivery is an MVP Beta for one data type (Court Judgments in MS Word), to be operational by April 2022. Expected Contract Length Three months with the possibility of extension to a maximum of 6 months. Latest start date Monday 17 January 2022 Budget Range Up to £90k to include travel and all other expenses. We have checked the role requirements (to the best of our knowledge) using the assessment tool found at https://www.gov.uk/guidance/check-employment-status-for-tax and the determination for the role(s) as advertised is that the intermediaries legislation does not apply to this engagement (out of scope). Why the Work is Being Done To support delivery of a new, cloud-based Data Transformation Service aiming to deliver secure, robust, automated and extensible ‘pipelines’ for routing data between internal systems and services. Specified data transformations and tools will be embedded into the pipeline as needed. This outcome initially supports an organisational commitment to receive, process and publish Court Judgment documents by April 2022. The supplier will deliver a Minimum Viable Product (MVP) as defined by The National Archives (TNA). Court judgments are sent to TNA as MS Word documents. The MVP Data Transformation Service must securely retrieve these from a specified cloud location, transform them to XML / LegalDocML using a parser application (created by another TNA project) and route them to our publication service. The MVP should be designed with a view to future extensibility. Further refinements on the product roadmap include filtering data based on business rules, handling additional document types, applying additional and more sophisticated transformations and routing data to and from other TNA services. Problem to Be Solved Specify, design and deliver an event-driven, automated workflow for Court Judgment documents as a step towards creating a fully featured Data Transformation Service for TNA. The MVP should identify receipt of a new Court Judgment by our Transfer service; retrieve and process the document using custom-built parser software to create an XML version; repackage the data; and deliver it to our Publication service. The Data Transformation Service must operate securely across TNA’s cloud environments and service boundaries. The MVP will handle only the most common form of Court Judgments, which are individual Word documents intended for publication. Error handling is required, e.g. to identify more complex Judgments and route these for manual processing. This MVP service should be designed with a view to adding further scenarios in the future. The supplier will work closely with TNA staff to ensure their design and deployment is consistent with our architecture and technology stack. The service should initially be deployed on its own AWS infrastructure at TNA, with the potential to share an environment with other cloud-based services in the future. The service must meet NCSC standards for system security and GDS service standards. Any web components must meet WCAG 2.1 Accessibility standards. Who Are the Users Internal stakeholders of the Data Transformation Service are TNA Service Owners responsible for the operation of other services (initially, Transfer and Publication) that either produce or consume data and rely on the Data Transformation Service to supply that data in a format suitable for use. External indirect stakeholders of the Data Transformation service are Court Service staff, who expect that Court Judgments sent to TNA will be published in line with a one-hour target. This requires a robust, automated pipeline to identify when a new judgement has been received, process it accurately and make it available for publication quickly. Future stakeholders will include Service Owners for our Preservation, Cataloguing and Access services, which will need to send and receive data and other types of public records. Early Market Engagement We have researched available data transformation tools and identified Talend Open Studio as suitable for our requirements for creating transformation scenarios (packaged as executable jar files) for both the immediate need and our longer-term roadmap. Work Already Done We have carried out a Discovery phase and are currently concluding the Alpha phase for the Transformation Engine to evaluate feasibility and identify our preferred technology stack. As part of the Alpha phase, prototypes have been implemented using event-based consumers written in Scala, a cloud-based Apache Kafka queue and transformations designed using Talend Open Studio. A separate project is currently creating a parser to convert Court Judgments received as MS Word documents to XML in LegalDocML format. The parser code will be executed by the Transformation Service. Existing Team The supplier’s team will deliver the Data Transformation Service. The National Archives staff will include a Product Manager with an understanding of user and stakeholder needs and an Agile Delivery Manager. The team that produced the Alpha prototype (Data Engineer, Technical Architect and Senior Developer) will be available and must be consulted about the supplier’s proposed approach and fit with our technology stack (e.g. Configuration of AWS environments, use of messaging for communication between services, proposals for the data pipeline). Internal users will be available to test that the service routes data correctly and processes it accurately. Current Phase Alpha Skills & Experience • Strong experience in building and configuring flexible data pipeline services • Excellent experience of deploying and optimising operational data processing services in AWS • Excellent experience of building, configuring, and securing AWS environments, including infrastructure configuration management (e.g. Terraform) • Experience of building automated event-based architectures using message queuing technologies • Experience of delivering Continuous Integration/Continuous Deployment pipelines including integration testing methodologies • Track record of delivering highly secure services and meeting OWASP and CIS web application security best practices • An iterative, Agile approach to delivery • Ability to produce clear technical documentation Nice to Haves • Track record of delivering cloud services that conform to the Government Service Standard • Knowledge of meeting GDS service assessment criteria Work Location Supplier to work remotely. Occasional attendance on-site at The National Archives, Kew, Richmond, Surrey TW9 4DU may be required. Working Arrangments The National Archives’ staff will be available during UK 9am-5pm. The supplier’s team is expected to work during these hours with scope for flexibility with mutual agreement. The supplier will provide their own equipment and technology and will be given access to our organisational GitHub, Jira and Slack resources as appropriate. Regular meetings will take place via Microsoft Teams with Slack available. The supplier will apply an Agile approach to scope, plan, and deliver work incrementally, with regular communication. The supplier will conduct regular, frequent product review or ‘show and tell’ sessions to demonstrate progress and seek feedback. Security Clearance Baseline clearance will be required (BPSS) No. of Suppliers to Evaluate 4 Proposal Criteria • Experience of building and configuring flexible data pipeline services in AWS. • Evidence of how highly secure services would be delivered to meet OWASP and CIS application security best practices • Experience of producing clear technical documentation and ensuring effective handover to enable the service to be maintained operationally • Team structure, including the relevance of the team members' skills and experience. Cultural Fit Criteria • Communicate clearly and openly, including working collaboratively with our product and delivery managers, technical specialists, and users to ensure the product meets the needs of its users • Work in an open and transparent way, communicating actively, sharing work in progress, and involving others as you go • Explain how you will engage and use your expertise to work effectively and constructively challenge our team • Describe how you will support positive working relationships throughout the life of the contract Payment Approach Capped time and materials Assessment Method Presentation Evaluation Weighting Technical competence 70% Cultural fit 10% Price 20% Questions from Suppliers 1. How did you arrive the budget of 90k ? We estimated the amount of work we think is necessary to deliver the outcome, taking into account past experience, the length of the engagement, the progress we have made so far with court judgments, the in-house team contribution, our experience of the market in terms of day rates, and a contingency factor.