Big Data: Data pipeline

Tender description

Why the Work is Being Done As part of the Museum's Digital Strategy the Big Data team deliver analysis and recommendations to departments on ways to understand our visitors to; improve the visitor experience through digital products, look for opportunities to increase revenue, support staff by providing clear, accurate and interactive dashboards. We require a data pipeline to consolidate our information, speed up our processes and visualize our findings to be complete within 9 - 12 months. Problem to Be Solved Combine Museum and external data sources into a Data pipeline. This includes, Audio Guide, Visitor Counting, Website, Social, Online Shop and external reference data like London tourist information and weather. Who Are the Users As a Data Scientist I need to analyse multiple data sources at once so that i can quickly analyse and visualize my findings to make recommendations to the business Early Market Engagement N/A Work Already Done From November 2016 to January 2017 the Big Data team ran an ‘Alpha’ Stage. The pilot successfully built a minimum viable product of the Big Data pipeline bringing three data sources; visitor counting, audio guide visit data and Wi-Fi data together in a centralised data warehouse and facilitated at speed analysis. The Azure Products include; Microsoft Azure SQL Server, Microsoft Azure Blob Storage, Microsoft Power BI, Microsoft Azure Machine Learning Studio. The data was brought together within Microsoft Azure SQL Server using a star schema. All personal identifiable data was anonymized by a GUID. Existing Team The main contacts are two departments in the Museum. The Digital and Publishing team (main users) and the Information Services Team. • Senior Product Manager: Big Data (Digital and Publishing), • Data Scientist (Digital and Publishing), • Junior Project Manager (Digital and Publishing), • Digital Data Analyst (Digital and Publishing), • Head of Business Solutions Delivery (Information Services), • Database Administrator (Information Services), • Analyst Programmer (Information Services), Current Phase Beta Work Location The British Museum, Great Russell Street, London WC1B 3DG Working Arrangments Onsite and offsite working. We envisage that the phases of work will not require extensive time onsite, perhaps 3 days per data source. Should you choose to include travel expenses those should be included in the cost proposal Security Clearance Additional T&Cs The chosen supplier will be expected to sign a non-disclosure agreement due to the sensitive nature of some Museum data. However the Museum is open to discussion as a use case and would act as a reference for the supplier. Skills & Experience 2 or more years’ experience working with clients on Microsoft Azure Software products Specific experience of working with the products for this project which are; Azure SQL Server , Azure Blob Storage, Microsoft Power BI, Machine Learning Demonstrable experience of ETL, data modelling and data warehousing Demonstrable experience of Transact SQL (stored procedure queries) Demonstrable knowledge of data acquisition including, APIs, FTPs and other collection solutions e.g. BCP utility (bulk copy) Demonstrable knowledge of data schemas (esp. star schemas) Demonstrable experience of general processes to import, clean, anonymize and insert in SQL server Excellent stakeholder engagement, to liaise with multiple Museum teams and to understand existing Museum systems and data Demonstrate how to support and maintain the pipeline for the beta phase Nice to Haves General stored proc layout experience Big data (Clickstream/Social Media) experience Experience of building a data warehouse for the purpose of a single customer view or closed loop marketing No. of Suppliers to Evaluate 3 Proposal Criteria Methodology or approach to delivery and support Technical capability Timeframe proposal Risks, assumptions and challenges identified Added value experience in Big Data solutions Estimated timeframes for the work Value for money Team structure Cultural Fit Criteria Share knowledge and skills Be transparent and collaborative when making decisions Take responsibility for their work Can work with clients with low technical expertise Work in partnership with the Museum Payment Approach Capped time and materials Assessment Method Written proposal Evaluation Weighting Technical competence 50% Cultural fit 15% Price 35% Questions from Suppliers Budget range