DPS Data Acquisition Preparation & Management (Emerging Technologies)

Tender description

Summary of the work Provide impact assessments Demand Management Delivery Management including testing & rollback capabilities Business Requirements Solution Design ETL (Extract-Transform-Load) Code Delivery Virtual databases code delivery Incident/problem management Change management Batch operation -support/recovery Event Management–Provide support when required HPI Management-Provide support when required Change Management–Evaluate change within SLA’s Capacity/performance management Knowledge management Expected Contract Length 16 months Latest start date Monday 8 February 2021 Budget Range £3,000,000 Why the Work is Being Done Data Platform Services (DPS) provides enterprise-wide data platform engineering capabilities that enable the acquisition, administration (management) and application (exploitation and consumption) of enterprise data. DPS provides the core platform capabilities and building blocks needed by: • CDIO Chief Data Office to implement enterprise data management (governance, models & standards), implement / manage data sharing and provide advanced analytics exploitation services. • CDIO Customer Groups to assemble, build and run line of business data services. • Data consumers (standard and advanced) within the wider HMRC Business to access and exploit data, within a supported self-serve framework. DPS will also provide the support offerings that enable HMRC to become a data-driven and insight-led organisation, drive data innovation and drive adoption of emerging technologies like AI (Artificial Intelligence) and ML (Machine Learning). Problem to Be Solved HMRC has a large and complex Data Analytics estate which has large number of data pipelines moving, messaging and transforming data between various data stores. DPS is building a logical data warehouse using data virtualisation to expose the legacy data warehouses and new data stores to wider HMRC data consumers. We design and build ETL/ELT and Data streaming services to ensure data is available in various data warehouses and lakes for consuming by different exploitation business services. Who Are the Users All HMRC customer groups (Customer Compliance, Customer Services, Customer Strategy and Tax Design, Corporate, VOA) need platform services they can consume for data on which they can build their specific line of business exploitation services. HMRC Chief Data Office needs platform services which enable Data Management, Governance and Data sharing. Other Government Departments rely on the Platform for receiving data for use within HMRC, and sharing HMRC data for their use. Existing Team HMRC and other suppliers Typically teams comprise of the following roles: Technical Project Manager Data Engineer (ETL/ELT, Fabric, Virtualisation) Scrum Master Solution Architect Product Owner Data Modellers Data Analysts Business Analysts QA Service Manager Current Phase Not applicable Skills & Experience • Please provide evidence to support each criterion below: • Experience working with and leveraging the benefits of AWS and Microsoft Azure Cloud hosted services • Experience working with ETL and ELT tools; namely Pentaho Data Integrator and/or Talend to design, create, deploy and operate enterprise services focusing on re-use, consistency and performance • Experience Data Modelling and creation of physical models conformant with Enterprise Data Model and organisation standards • Creating performant and resilient ETL/ELT jobs and Data Messaging pipelines for a large enterprise • Capability of Java and JavaScript development • Supporting 24x7 Data Acquisition and Data Preparation services • Experience working with Databases • Experience working with Data Messages Queueing technologies, preferably Kafka • Test driven development with Business Intelligence services including use of Continuous Integration environments • Automated and Performance testing of ETL / ELT and Data Messaging services • Experience of customer facing issue and incident resolution • Producing and maintaining technical documentation and a knowledge base • Working safely within our UK based secure engineering environments at SC level Nice to Haves • Please provide evidence to support each criterion below: • Experience using Metadata and information Management solutions • Experience using and administering Data Virtualisation solutions, preferably Denodo • Experience with Red Hat Linux • Experience working with large databases and data lakes • Experience working with public sector organisations Work Location The base locations where the work will take place are: Trinity Bridge House 2 Dearmans Place Salford Manchester UK M3 5BS Alexander House 21 Victoria Avenue Southend on Sea UK SS99 1AA Stratford Regional Centre 14 Westfield Avenue Stratford London UK E20 1HZ Working Arrangments The supplier will be required to co-locate with existing platform teams (mixed teams comprising internal staff and other contractors) 5 days per week. No expenses will be chargeable for travel or accommodation relating to agreed base locations. Where travel is necessary outside the base locations, this will be referenced in the Statement of Works with expenses agreed in line with HMRC travel policy. Current working practices dictate that work will be undertaken remotely. The supplier will be required to work with Chief Digital Information Office (CDIO) teams nationwide. The supplier will provide upskilling to HMRC staff to increase internal capability. Security Clearance Must have BPSS before employment commences and resources come on site. Must be SC eligible. SC process can start after employment commences. Clearance costs to be met by supplier. Additional T&Cs HMRC Mandatory Terms to be added No. of Suppliers to Evaluate 4 Proposal Criteria • Shortlisted suppliers will be asked to provide Case-Study, Written Proposal and Presentation. These will be scored using a selection of Essential Criteria outlined above to be provided with Proposal Scenario. • Below outlines the expected ways of working to be evidenced: • Demonstrate how you as a supplier have been fully accountable for delivery of a project or outcome and successfully delivered to specification, time and with value for money • Demonstrate how you as a supplier have been fully accountable for operating/running a service to agreed service levels and standards • Demonstrate as a supplier how you have taken remedial action where outcomes have not been to agreed standards • Agility - delivering iteratively and incrementally within a context of evolving requirements following SCRUM and Kanban approaches • Making knowledge capture and sharing part of day to day activities • Ensuring successful handover at any required stage of a project or a service • Delivering business value – making sure business requirements are understood, validated via positive challenge and delivered against while understanding the value being generated (i.e. business intent) • Innovative thinking – think outside the set of existing organisational constraints to arrive the most appropriate and efficient route to deliver desired outcomes • MVPs (Minimum Viable Product) – effective definition and delivery of true MVPs that shorten time to delivering business value • Reusability by design – ensuring solutions are formulated from reusable capabilities as well as ensuring new capabilities are reusable • Simplicity by design – a clear focus on producing and positively challenging solutions to be as simple as possible • Design for minimising build and run costs • Responsive mobilisation – ability to resource work and start delivering in 4-5 weeks • Enough capacity to ensure supplier can cope with DPS demand, in the specified locations Cultural Fit Criteria • Build inhouse skills 1.At DPS we want to grow our in-house skills: oHow would you support this? oWhat skills, time and expertise would you make available and where? • Evidence how you deliver in multi supplier environments adhering to DPS behaviours, including inclusivity and collaboration. Payment Approach Fixed price Assessment Method • Case study • Presentation Evaluation Weighting Technical competence 60% Cultural fit 10% Price 30% Questions from Suppliers 1. Is there an incumbent supplier? DPS operate a multi-supplier delivery environment, this opportunity is to supplement and enhance the current delivery arrangements. 2. What do you expect the split between the three different locations to be? DPS are flexible by project within the limitations of HMRC accommodation where capacity is available and needed. There may be some limitations due to Product Owner location, or individual security requirements for specific deliveries. 3. This looks similar to a procurement initiated in January ‘Data Platform Services – Data Acquisition, Preparation and Management’. Can you please clarify the main differences between the two services? his requirement has excluded our legacy technologies where demand is stable and met through existing contracts. Particularly on our new platforms we are seeing unprecedented demand exceeding previous estimates and need to respond to that, ensuring we have sufficient capacity and supplier diversity. 4. What is your preferred language for your code base? Which language would you prefer to continue to invest in? What kind of engineers are you looking to attract? HMRC uses wide range of tools and we build our systems in different tools and programming languages. As per the essential requirements of this bid, Java and JavaScript are required. Advanced SQL skills are very crucial as well. HMRC is keen to attract highly competent engineers to perform the roles of ETL/ELT processes and Reports and Dashboards developers, testers and administrators. 5. Could you confirm whether the requirement "Supporting 24x7 Data Acquisition and Data Preparation services" references providing a 24x7 managed service for Data acquisition and preparation or having tools and services available to users 24/7? This depends on the business requirements. It could be a mixture of both service availability and a managed 24/7 service. 6. Regarding "Supporting 24x7 Data Acquisition and Data Preparation services " – Is this L1/L2 and L3 support? Will out-of-hours be an on-call service or permanently staffed? Currently HMRC have no requirements to stand up any Data Acquisition services 24/7 but at any point this could be requested by any business area for their particular services and it could include L1-3 support, out of hours on call or permanently staffed. It does all depend on what the business requirements are. So capabilities and readiness to deliver these support models should be in place. 7. With regard to "Experience working with Databases" – Can you please name the database technologies that are relevant to the scope of this procurement? From Relational databases, it's mainly Oracle (on-prem and AWS RDS). From NoSQL databases, it's Amazon RedShift, Cloudera CDH (HDFS, Hive and HBase). 8. With regard to "Experience of customer facing issue and incident resolution" – Is the context of this procurement is the customer the public or internal teams/stakeholders? Primary project focus is internal stakeholders / customers. 9. With regard to "Experience using Metadata and Information Management solutions" – can you please name the tools/products that are particularly relevant to you? Mainly, experience of metadata management solutions is a nice-to-have criterion. However, experience with Informatica EDC and Axon is a plus. 10. With regard to "Experience with Red Hat Linux" – Are you looking for Red Hat Administration or development experience on Red Hat? Do you have a preference for Red Hat version? HMRC is looking for mainly for Development experience, preferably on version 7.x 11. With regard to "Experience working with large databases and data lakes" – Can you please say what HMRC would view as large in Tb or Pb? DPS owns Data Warehouses and lakes in order of 10's to 100's of TBs. 12. Just recently there was a presentation from HMRC's CIO and his team, explaining that HMRC now wanted to change and be open to new suppliers, as part of TSP. This was great news. However, this procurement appears to reflect the opposite. A complex piece of work with a wide range of resources across a range of locations. All with, no prior notice of coming to market and only 2 weeks to respond. This approach (for the reasons stated) indicates this work will likely go to incumbent- Lets see! Does the CIO know that this cuts across his presented approach please The nature and breadth of the services mean that the work touches many areas of HMRC’s IT infrastructure. The current climate has necessitated that we reach out to add to, and complement, current provision. The two weeks response time is automatically generated by the Digital Marketplace. 13. Please could the Authority explain how this requirement relates to the Digital Outcome and specialist opportunity of the same title opened and closed in January of this year. Is this an extension to those services or a new requirement? This requirement has excluded our legacy technologies where demand is stable and met through existing contracts. Particularly on our new platforms we are seeing unprecedented demand exceeding previous estimates and need to respond to that, ensuring we have sufficient capacity and supplier diversity. 14. Were these services awarded to a Supplier and if so, please could the Authority confirm which Supplier? This is not a rebid. This opportunity is in addition. 15. TechUK ran an event in October providing an overview of the Technology Sourcing Programme at which HMRC provided a summary of anticipated procurements, including a number for Data & Analytics services. Please could you confirm if this procurement is part of the TSP Programme. This is not part of the TSP procurement. 16. Can you please clarify if you both structured or non-structured databases are in consideration? Yes, structured, semi-structured and unstructured data are considered. 17. For Data Messages Queueing technologies – please can you clarify if your are looking at Cloud Native services ( i.e. AWS Kinesis) or you have zeroed in on Kafka? Are you looking to implement Messaging Queue services which supports a range of semantics? Currently, there is no demand to use Cloud-native queueing services. Suppliers should focus on Kafka-supporting range of semantics. 18. Can you please clarify if you wish to enhance the existing EDW or completely rearchitect? Does the supplier need to configure and set up Data Lake on a chosen cloud platform or only provide support on existing Data Lakes? Supplier might be needed to enhance DWs, but will not re-architect/re-design any data warehouse nor lakes. Suppliers might need, in some cases, to create staging/landing areas or so to enable the ETL jobs. Suppliers will work with existing Data Warehouses and Lakes. They will be source and/or target data stores for various ETL jobs.