SurfaceNet Collate Platform

Tender description

Summary of the work Using AWS the project will deliver a range of functionality including: ETL data flow engine with flexible rules driven quality control processes Data migration / conversion of existing data Metadata querying and production e.g. calibration and quality flags Delivering visualised data to consumers in near real time Early life support Expected Contract Length 2 years Latest start date Wednesday 30 May 2018 Budget Range In the region of £1,000,000.00 Why the Work is Being Done The current on premise system for the collection & processing of surface sensor data is reaching its end of life. The Met Office are looking to leverage Amazon cloud platform technology to provide a flexible, scalable and resilient data processing platform to meet customer requirements around timeliness and quality of data. Observations data is at the heart of everything that the Met Office undertakes including climate research and the initiation and validation of forecasting models. Raw observations contribute to the national archive, scientific research and international exchange of data and improved access to this data is a key project driver. Problem to Be Solved The project needs to be able to collect and disseminate observations data to allow existing systems to be retired while also meeting rules around quality and timeliness. Key Outcomes: Automated tools to improve UX and deliver efficiencies. Ingest raw data from a range of sources. Transform data through calibration and a flexible data QC rules engine. Load data for consumption in agreed API formats Handle comprehensive Metadata. Meet NFRs such as security, timeliness, reliability, and WMO/CAA etc. standards. Ensure Continuity of existing services and data. Use AWS tool sets where possible Who Are the Users Direct consumers of the data will include: Network managers. Forecasters. Quality control team. Scientists. Engineers. Commercial consumers. Each grouping will have some bespoke requirements and may have slightly different perspectives on the same core requirement or element this will drive a significant degree of user flexibility to be incorporated into the system design. Work Already Done A detailed requirements capture process has informed a high level system architecture. The concept has also undergone peer review from industry experts. Rules around processing metrological data are well defined with definition from the WMO and others. A Proof of Concept has been undertaken with AWS IOT team to demonstrate core functionality. The project sits as part of a wider program of work including a new data gateway and is underpinned by the subject matter experts within the observations department. Initial hardware testing has been undertaken on integration of the field infrastructure with the AWS platform. Existing Team The Met Office team includes: Project Manager. Product Owner. Solution Architect. Tech Lead. Business Analyst. Hardware Test Engineer. Met Office developer staff will also be assigned to ensure transfer of skills during development to ensure future operating capability. Timely and appropriately skilled resource to complement and support the supplier team as and when required will be made available in order to remove impediments or provide domain knowledge. Current Phase Beta Skills & Experience • Experience/evidence of working on AWS utilising PaaS based tools and technologies to implement modular solutions to provide ETL functionality. • Experience of working alongside a customer team to deliver an integrated platform (ensuring knowledge transfer). • Ability to use Agile methodologies while complying with mandated international (meteorological) standards (provided). • Experience of dealing with data with prescriptive data processing and timeliness rules. • Ability to surge resources to support agreed project plan. • Experience of working with and designing scalable APIs for a wide variety of end users. • Utilise the most up to date IoT toolsets and understanding of IoT Security Foundation best practice. • Digital design skills with a focus on user experience. • Experience using Continuous Integration development approaches. Nice to Haves • Experience of delivering projects within the environmental monitoring sector ideally from a dispersed network of remote monitoring solutions. • A proven partnership with Amazon delivering AWS cloud hosted solutions, for large organisations. • Proven experience of working with government sector organisations to deliver projects with complex regulatory requirements including international standards and governing bodies. Work Location Met Office, Exeter, EX1 3PB. The majority of work is expected to take place on site to ensure robust knowledge transfer. Working Arrangments The supplier will deliver through Agile methods and will be able to work closely with Met Office teams. It should be noted that the project is required to meet a series of stringent regulatory requirements. The supplier will be expected to use the Met Office’s own systems such as JIRA (Confluence, Stories on Board) etc. Security Clearance Work on the Met Office premise or its IT environment will require SC. Additional T&Cs It is the Met Offices intention to hold the full IPR for all systems components and at the completion of the project all source code, business rules and documentation should be provided to the Met Office. No. of Suppliers to Evaluate 5 Proposal Criteria • Track record in similar applications • Experience with working with customer teams and integrating with existing platforms • Approach and methodology • How data latency can be managed effectively • Key risks and dependencies of your proposed strategy and mitigations you would put in place to manage them • Proposed Team structure and capabilities on a phase by phase basis • Organisational capacity to provide additional or flexible capability aligned to the scheduled deliverable • How knowledge will be imparted to ensure the MO can manage operation of the system into the future including examples of this from previous projects • Value for money Cultural Fit Criteria • Alignment to the GDS digital principles with a ‘user first’ focus Embedded agile alignment • A culture of openness and collaboration, a willingness and understanding of the principles involved in being a ‘strategic partner’ rather than a supplier • A flexible approach to communication, especially in the promotion of agile principles • A willingness to support co-location and face to face working • A cohesive approach to working with internal capabilities • Evidence of thought leadership in the creative and strategic space • Focus on open knowledge transfer to ensure future Met Office internal support Payment Approach Capped time and materials Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers 1. Reference: "The majority of work is expected to take place on site to ensure robust knowledge transfer". Can suppliers propose a hybrid approach in which the majority of the work takes place at the premises of the supplier and supplier's staff meet frequently with the Met Office's project team? No.The majority of the work is to take place on the MO premises in Exeter to allow pair programing and robust knowledge transfer. We expect the supplier to be embedded with our staff.Kind Regards 2. Apart from AWS, what other technology stack does the project wish to use e.g. Java, Python? The core technologies we use are:Java (in AWS Lambdas)Spring Boot for micro servicesAngularJS for web applicationsPython is also used when appropriate.