Energy Statistics Data Management Solutions

Tender description

Why the Work is Being Done We need to deliver a step change to the data management and processing systems that support production of BEIS's energy statistics. This is the first phase of work to further understand requirements, develop solutions and deliver initial outcomes. We wish to find enthusiastic and skilled data and technical specialists to work alongside BEIS statisticians and system experts in a 3 month project (Feb-Apr 2019). Problem to Be Solved Much of our data on energy is stored and manipulated in linked Excel spreadsheets that have grown organically over time. This is inefficient and carries risk of error. We wish to move to a modern data warehousing solution, with automated (yet transparent) reproducible processes for managing, assuring and analysing our data assets. We need to develop solid and scalable data standards, models, dictionaries, governance and architecture to enable high quality data management with solutions for permissioning, audit, revisions, versioning and linking. Who Are the Users As a statistician I need to: - spend less time processing data so I can spend more time using it and gaining insight. - have harmonised data definitions, standards and governance so that I am confident in linking and using data to provide high quality outputs - be able to easily discover and update my data, control access and audit revisions so that I can adhere to data security and am confident of the 'truth' - have reproducible and transparent processes for validating and manipulating data so that I am confident of the quality of my statistics. Early Market Engagement Work Already Done An internal scoping phase has been carried out to identify the problems we are aiming to address, the benefits to be realised, initial requirements and some small scale testing of solutions. There is an existing cloud based internal secure infrastructure in which new data warehousing and processing solutions can be developed. There is also similar work taking place in other governments that we would want to learn from (Reproducible Analytical Pipelines). We want to focus this phase of work on storing and handling data in a structured warehouse, e.g. using MS SQL Server Management Studio and R. Existing Team You will be working closely with internal staff in the Analyst Directorate in BEIS. Primarily the Energy Statistics team and the Data Analytics team who have formed a joint working group for this project. Within Energy Stats there are ~15 individuals involved in the Energy Balances statistics (published monthly, quarterly and annually) that have the detailed knowledge of the data in scope and the business rules to be incorporated into the agreed solution . The Data Analytics team host the cloud based data analysts system (CBAS) upon which development work is carried out, and include IT specialists. Current Phase Alpha Work Location The Analyst Directorate are located at 1 Victoria Street, London. It is expected that you will be co-located for at least 3 days per week throughout the duration of this project. Working Arrangments It is imperative that you work closely and collaboratively with the internal team. The knowledge and expertise around the data and statistics reside in individuals and internal documentation. Development work will need to be on the CBAS platform which can only be access through BEIS IT systems. Knowledge transfer into existing teams is a necessary outcome of this work. We therefore expect you to be co-located with the team at least 3 days per week and work collaboratively through digital tools. We do not expect to pay travel and subsistence expenses (these should be included in total costs/ day rates) Security Clearance Basic security clearance is required (can be arranged although this could delay work starting). Additional T&Cs Skills & Experience Previous experience of designing databases and new data and technical architecture solutions that meet user / business needs Strong knowledge and understanding, and a track record of delivering automated data pipelines and producing robust and reproducible analysis Experience in working with complex, messy & interlinked datasets Previous experience of using MS SQL 2012 and above Previous experience of using SQL Server Integration Services Previous experience of using version control software and knowledge management solutions such as gitlab Ability to work closely and collaboratively with internal data and technical experts Ability to clearly communicate key criteria, trade-offs and risks to underpin business and development decisions Transfer skills and documentation for business as usual operations and future development Previous experience of using scripting languages such as Python, R. Previous experience of MS Excel and MS Access 2010 or later Nice to Haves A track record of delivering data management systems for government Knowledge of energy data Broader technical skills around javascript, PHP Ability to train and embed knowledge and skills into internal teams Knowledge of using geographic or spatial data with databases Previous experience of Linux OS, such as Ubuntu 16 (or later) or similar Previous experience of using R Previous experience of working with PostgresSQL or another open source SQL database No. of Suppliers to Evaluate 3 Proposal Criteria Approach and methodology Team composition and skills Technical solution Time frame Value for money Knowledge transfer Project management Risk management Cultural Fit Criteria Work as part of the Energy Stats and Data Analytics team Be transparent and collaborative when designing solutions and making decisions Take responsibility for driving and delivering work Ensure knowledge sharing and transfer into BEIS Work with data experts with a range of technical expertise Payment Approach Capped time and materials Assessment Method Written proposal Case study Work history Reference Evaluation Weighting Technical competence 50% Cultural fit 20% Price 30% Questions from Suppliers Budget range maximum day rates of £800 p/d (including any expenses & vat). expected total budget for ~3 months project is £180-200k, depending on team composition and skills.