Transfer of government digital records (files and metadata) from SharePoint Online to The National Archives' AWS infrastructure - Alpha prototype(s)

Tender description

About adding context and requirements test Work done so far This problem has been examined in previous research and a discovery phase. A 'Cloud to TDR' Discovery took place in Autumn/Winter 2022. The focus of the Discovery was to understand what users would expect from a cloud to TNA transfer via TDR by understanding their needs and wants. Within the user interviews, the Discovery also explored how Government Departments were creating and managing their records within cloud environments (primarily M365 SharePoint and Google Workspace), and the implications of this for transfer; and through interviews with stakeholders, key technical considerations were explored. Please see attached supporting reference documents: Cloud to TDR Discovery overview, and export of Miro board; Cloud to Cloud Spike; and Cloud transfer - Research Paper PY. Which phase the project is in Alpha Existing team The supplier’s team will deliver the proposed solution(s) and Cloud Transfer Prototype(s). The National Archives will provide access to a Product Manager with an understanding of user and stakeholder needs, an Agile Delivery Manager, and a member of Digital Archiving data team with previous experience of technical issues regarding cloud transfer. Members of the TDR Agile product development team and TNA's technical architect will also be available and must be consulted about the supplier’s proposed approach. Address where the work will be done Remote. For information, The National Archives is based at Kew, Richmond, Surrey TW9 4DU. Working arrangements The National Archives’ staff will be available during UK 9am-5pm. The supplier’s team is expected to work during these hours with scope for flexibility with mutual agreement. The TDR agile development team work in sprints and it is expected that the suppliers team will work to the same sprint cadence. The supplier will provide their own equipment and technology and will be given access to our organisational GitHub, Jira, Slack and AWS resources as appropriate. Regular meetings will take place via Microsoft Teams with Slack available. The supplier will apply an Agile approach to scope, plan, and deliver work incrementally, with regular, frequent and open communication. The supplier will conduct regular, frequent product/stakeholder review (and optionally, also ‘show and tell’) sessions to demonstrate progress and seek feedback. Provide more information about your security requirements: Baseline Personnel Security Standard (BPSS) Latest start date 2023-09-28 Enter the expected contract length: 3 months Write the term or acronym: TNA Write the term or acronym: TDR Explain the term or acronym: The National Archives Explain the term or acronym: Transfer Digital Records Service Are you prepared to show your budget details?: Yes Indicative maximum: 120000 Indicative minimum: 90000 Confirm if you require a contracted out service or supply of resource Contracted out service: the off-payroll rules do not apply Summary of work The National Archives receives digital records from government and preserves these for the benefit of future generations of citizens, researchers and policy makers. These government records are increasingly created and managed using cloud information management tools, such as SharePoint Online. We require the Supplier to design and prototype solution(s) for transferring these digital records (digital files and metadata) from the cloud-based systems used by government departments to the National Archives. Transferred records are to be received by TNA's Transfer Digital Records (TDR) service. This is an in-house service, built on AWS infrastructure. The initial solution(s) and prototype(s) will focus on transfer from SharePoint Online instances to TNA. Optionally, the supplier could also consider transfer from other Information Management tools or from cloud storage (such as S3). The solution(s) proposed must provide a secure, usable, highly automated method of transfer which ensures the integrity of the digital records (files and metadata) throughout the transfer process. PLEASE NOTE: this will be a 1 stage process. Suppliers will be shortlisted for presentation and interview based on the Stage 1 submissions ONLY. Dates for Stage 2 have been added ONLY because the system requires this. We will not be asking for further written proposals; the presentation/interview will essentially count as Stage 2. Please also note that our preferred interview dates are 4 and 5 September 2023. The system may demand that shortlisted suppliers submit a Stage 2 proposal - if so, those shortlisted suppliers can simply resubmit their stage 1 response. Where the supplied staff will work No specific location (for example they can work remotely) Who the organisation using the products or services is The National Archives Why the work is being done The National Archives has developed a new service for the transfer of born-digital records to TNA. Transfer Digital Records (TDR), released in public beta in April 2023, helps UK Government departments (and other ‘Public Record Bodies’) upload, prepare and transfer digital public records to TNA for permanent preservation (and if applicable access), through a web browser, and within an AWS environment. This helps departments transfer records quickly and easily, helping them comply with the Public Records Act. TDR accepts upload of records from a network drive environment. An increasing number of users are now using cloud ecosystems, mostly SharePoint Online. The National Archives wishes to develop a service that enables transfer of digital records from these environments into TDR. This affects not just the transfer of records currently being created but also a significant volume of legacy digital records that have been migrated to cloud systems. Records cannot yet be sent directly from M365 SharePoint to TDR, so transfer from SharePoint currently requires transferring bodies to download/export their records from SharePoint to a network drive environment in order to follow the standard upload process. This can result in loss of metadata which damages record integrity. We wish to create a new method that is easy to use and enables transfer from SharePoint Online (or other cloud-based tools) without loss of metadata or integrity. The business problem you need to solve For this Alpha we are looking for proposed solution(s) for the transfer of records from departmental cloud-based information management systems to TNA. This includes designing the approach and delivering a working prototype / proof of concept which demonstrates secure transfer of a government department's digital records from SharePoint Online to The National Archives, via the Transfer Digital Records (TDR) service (hosted in AWS). The proposed solution(s) must provide a secure method of transfer which ensures the integrity of the records and captures all metadata which provides additional context to the record. It must follow the below principles. • able to identify user transferring records via UUID • securely access the records a user wants to transfer • user able to select which records they want to transfer • each transfer assigned a UUID • harvest metadata about the selected records and send this metadata to a database hosted on AWS (or upload it to an AWS S3 bucket in a form suitable for ingest into a database) • upload the selected digital files to an AWS s3 bucket • once the metadata harvesting and uploading processes is completed, trigger an AWS step function via AWS API Gateway, passing the transfer UUID to the step function as input As part of testing any prototype or approach we would also want to undertake performance and latency tests of high volume transfers to understand any size or volume limitations. The supplier will work closely with TNA staff to ensure their design and deployment is consistent with our architecture and technology stack. Preferably, software should be written in Scala or Java. Any infrastructure should be created using Terraform and hosted in TDR's AWS environment. For the Alpha prototype we are not seeking full integration with TDR but we expect the prototype to simulate this. Proposed solutions should align with TDR’s user administration, registration and compliance process. The service must meet NCSC standards for system security and GDS service standards. Any web components must meet WCAG 2.1 Accessibility standards. First user type: Public record bodies (mostly UK Government Departments) who need to transfer records to TNA. The work is usually undertaken by Knowledge & Information Management (KIM) teams, or by Records Managers. Enter more details about this user type: As a member of the KIM team of a public record body that needs to transfer digital records to TNA, I want to be able to easily send files and metadata from my SharePoint Online platform to the Transfer Digital Records service. As a Digital Archivist at TNA, I want to be able to receive records from a Government department's SharePoint and be confident that the files and metadata have retained their integrity, and that all relevant metadata to provide provenance and context to the records has been successfully captured.