RM1043.8-1-Aberdeenshire Council - Provision and Development of a Centralised Data Platform

Tender description

About adding context and requirements test Pre-market engagement None Work done so far Provision of data platform aligns with the Digital, Data and Technology theme of the Future Aberdeenshire Transformation programme of work. This project forms part of the wider data strategy work that has been undertaken. Data governance tasks such as data dictionaries, data cataloguing and defining data owners and stewards is currently in progress. Which phase the project is in Discovery Existing team There is an existing centralised Data and analytics team with 3 team members, there are a number of data analysts across the Council, within the services completing traditional analytical tasks with their respective service data. Our internal data infrastructure is managed by our in-house Data and Server team who sit within our IT Service. Address where the work will be done Aberdeenshire Council Headquarters, Woodhill house, Westburn Rd, Aberdeen AB16 5GB Working arrangements The supplier staff will keep Aberdeenshire Council Data and Analytics team hours in order to maximise collaboration, efficiencies and knowledge transfer. Remote working, communicating with Aberdeenshire Council staff via Microsoft teams meetings, calls and messages. No scope for expenses to be provided as part of this project. Provide more information about your security requirements: Baseline Personnel Security Standard (BPSS) Provide more information about your security requirements (optional): For those members of the chosen supplier requiring direct access to the live data, Aberdeenshire Council would require that they undertake a basic DBS check. DBS checks will be at the supplier’s own expense. We would require the chosen supplier to hold Cyber Essentials Plus. Suppliers will be required to comply with Aberdeenshire Council's Security Compliance Standard. Latest start date 2024-09-02 Enter the expected contract length: 2 years Extension period: 1 year Special terms and conditions All suppliers are obliged to provide sufficient guarantees to implement appropriate technical and organisational measures, so that the processing of Council data meets the requirements of GDPR and ensures the protection of the rights of data subjects. Suppliers will be expected to comply with Aberdeenshire Council's Security Compliance Standard, which will shared with shortlisted suppliers. Write the term or acronym: CRM Write the term or acronym: Data hub Explain the term or acronym: Customer Relationship Management Explain the term or acronym: Internal name for central data platform project Are you prepared to show your budget details?: Yes Indicative maximum: 250000 Confirm if you require a contracted out service or supply of resource Contracted out service: the off-payroll rules do not apply Summary of work Aberdeenshire council (The Council) has a requirement for a centralised, cloud-based, data platform, internally referred to as the data hub. The data hub project is fundamentally about changing our relationship with our data, improving our analytical reporting capability, and creating quick and easy access to our organisational data. There must be capability to bring together data from a large and complex legacy data analytics estate in use at the Council. The project has three key goals: Data Hub: The development and implementation of a, Microsoft-based, centralised cloud data analytical platform to unify and improve our data in an analytic capacity to with within our existing azure infrastructure. The proposed solution should be deployable within our existing Azure tenant, using components such as Data Factory, Data Warehouse/OneLake with full compatibility with PowerBI, our chosen BI solution. The proposed solution should also be future proof to enable innovative technologies such as AI to be seamlessly embedded at a later stage. Advanced data analysis: Improve our analytical reporting capabilities through the ingesting of data from critical systems in relation to defined use cases, transformation of this data to improve quality and usability, matching of relatable datasets to provide insight into our data through developed visual advanced analytical dashboards. Specific use cases have been identified to focus the project on specific datasets and outcomes. There will be a requirement for the chosen supplier to deliver these use cases through advanced analytics. These use cases are Single View of the Citizen, Detailed Performance Analysis, Detailed CRM Analysis and Finance forecasts and Incorporating Partnership Data. Single View of the Citizen: The Council collects vast amounts of data in relation to our citizens and the services we provide them. However, this data is held in multiple legacy systems across the organisation with little to no capability in our current data infrastructure to access and match this data at an organisational level. This creates several problems for staff requiring this data to conduct business as usual, leading to time-consuming, repetitive, manual inefficiencies. There is a requirement to load data from our key legacy systems into the Data Hub and provide an analytical dashboard to present staff with all relevant information on our citizens. Data matching must be undertaken to ensure that the data from each system is related to the chosen citizen and any data quality issues must be resolved to ensure only the most current and correct data is displayed. This will allow the council to create real-time insight into our citizen data, understand trends and predict and resolve issues before they occur. Detailed Performance Analysis: Our current performance analysis is a very manual and time-consuming process; limitations prevent the council from gaining valuable insight into our performance data. We require the data hub to collate data directly related to our performance indicators from source systems, streamlining current manual data collection and transformation tasks. Data must be displayed to gain insight in trends within our historical data to predict future performance. Detailed CRM Analysis: Our current CRM data is held across multiple systems, this requires manual and time-consuming data matching to gain any insight into the data. Data quality issues across the systems also prevent the council from gaining valuable insight into our CRM data and considerable amounts of repetitive work is required to resolve these issues to gain any insight into the data. We wish to combine the data from these systems and automate the transformation of the data to prepare it for analytical use. This will allow us to identify historical trends, highlight areas of demand and improve our interactions with our citizens. Finance Forecasts: We require detailed and advanced analysis into our finance data to obtain a comprehensive understanding of financial trends, risks, and opportunities within our finance data held across multiple legacy systems. Using predictive analytics, we must be able to derive actionable insights and identify potential overspends within budgeted financial data. Incorporating Partnership Data: As a member of the Community Planning Partnership, the Council works closely with partner organisations deliver the best quality of life to the citizens of Aberdeenshire. There is a requirement to work with these organisations to access the data they make available on a regular, automated basis. This will allow the council to combine this external data with internal data to deliver shared priorities across the partnership. Partnership approach: To ensure the council gets maximum value from this project, there must be knowledge transfer and training provided to internal staff to use the data platform and analytical reports effectively, with documentation and guidance. Where the supplied staff will work No specific location (for example they can work remotely) Who the organisation using the products or services is Aberdeenshire Council Why the work is being done Aberdeenshire council (The Council) is in the process of the development of a comprehensive data strategy under the Future Aberdeenshire transformation programme. The programme seeks to deliver transformation at an organisational level through five key themes, this project falls under the digital, data and technology theme. As part of the wider data strategy, the Council has a need for business intelligence at an organisational level to be able to carry out the wholesale transformation we need to make. The provision of a centralised, cloud-based, data platform forms part of the wider data strategy. The Council is data rich but has work to do to make the most of its raw data. We can use this data to forecast our business needs, understand our performance, identify, and resolve issues and set priorities – so having better, well-managed, trusted, and easy-to-use data will help us provide the best services possible to those who rely on us. In terms of technology, our current data infrastructure doesn’t allow us to exploit our data efficiently or effectively. We have a vast data landscape with multiple legacy systems, which means that we are currently highly manual and often use varied processes and reporting software to extract and transform our data into useable information. The data hub will bring together data from a large and complex legacy data analytics estate in use at the Council. The extraction of data from our legacy systems is a fully manual process and a lack of consistency in recording, interpreting, and analysing data has created a problem across the organisation with how data from comparable systems can be used, in an analytical capacity, to discover insights and improve delivery. As a result, duplicate records are held with varying degrees of consistency and accuracy. This means that information used by and about our services can be incorrect and unreliable, without time-consuming manual input to transform the data. The Council seeks to improve our approach to data to enable better citizen facing services, improve support to decision making for policy and operational matters, and provide insights to data that can create value-add or efficiencies in services. use this data to forecast our business needs, understand our performance, identify, and resolve issues and set priorities. Having better, well-managed, trusted, and easy-to-use data will help us provide the best services possible to those who rely on us. The council wants to improve our ability to use and reuse our data, this will in turn improve our relationship with our data and provide critical advanced insight into the environment we operate in and the way we operate within it. Through the data hub the council wishes to understand the needs of our citizens better and the impact our services have on them. We require the data hub to be developed and established, with data from our critical datasets delivering value by the end of the financial year. With continued value and data incorporated delivered over the remainder of the project timeline. The business problem you need to solve Our current data analytics landscape is extremely manual and time-consuming. We require the removal of manual, time-consuming repetitive data extraction and transformation from multiple legacy systems used across the services. There is a requirement to provide a central repository for quick, cloud-based, access to good quality council-wide data held across the services, removing the dependencies on direct interaction with multiple legacy systems to allow for comparable data matching to provide relevant users with access to relevant council data to aid policy and strategic work, enabling data-driven decisions. None of our legacy systems currently share data and there is a lack of consistency in recording, interpreting, and analysing data. The lack of capability to consistently share data across services has created challenges where the matching of data for shared goals is concerned. The solution must integrate with these legacy systems to extract data on a consistent basis to allow analytical reports and dashboards driven by the data hub to be built on the most up-to-date version of the data as possible, promoting an organisational level single version of the truth. Automation of the extraction of data from these internal systems into the Data Hub, with the capability to incorporate the various external sources of data we report on and benchmark against, is a critical requirement for this project. The development and implementation of the data platform is expected to remove data silos, which are preventing data being effectively used across services and on an organisational level. Achieving an organisational approach to data analytics through awareness and access to organisational data, allowing for relevant data held across the individual services to be used in conjunction will enable and promote, fully informed, real-time, data-driven decision making. There is a risk of resistance to data-sharing across the services. Services are conscious there are data quality issues within the data held within their legacy systems but do not have the resource or technical capability to resolve these issues at a system level. Key stakeholder buy-in and an ability to work with users with a limited technical understanding will be critical to resolving these issues and accessing service level data. Through initiatives such as single view of the customer and performance dashboards, combining multiple data sources, from multiple legacy systems held by the services to present a cohesive and comprehensive view of our citizens and the performance of our services, through extraction of the data from legacy systems and storage in a cloud-based data platform, will improve efficiencies across the organisation and remove many manual tasks. Through the extraction and transformation of data from legacy systems into the data platform, there is a requirement to resolve data quality issues in extracted legacy system data, which currently require a considerable amount of manual and repetitive correction from data analysts to prepare data to be used in an analytical capacity, while there is an understanding of our data quality issues, services do not have the technical capability to resolve these issues. This creates the need for any analytical work to involve manual transformation of exported data to improve data quality, this work is repetitive and time-consuming. The data platform must also prepare our data to be used with emerging technologies such as AI, further removing manual and time-consuming tasks, allowing us to future proof our data delivery with advanced analytics and predictive analytics, allowing us to make better use of our data at an organisational level. First user type: Data Analysts First user type: Performance leads First user type: Council officers First user type: Data and analytics team leader First user type: Azure administrators First user type: Policy makers First user type: Senior leadership First user type: Chief officers First user type: Elected members Enter more details about this user type: Data Analysts across the organisation need to access data from our multiple legacy systems and various sources and ensure it is in a usable format. They require instant and continuous access to this data, in its most up-to-date availability to produce analytical reports for their various service requirements. Access to data from systems managed by other services will allow analysts to produce detailed reports and visualisation on relatable data to provide a clear and full picture to the effectiveness of the services we provide to our citizens. Data analysts require access to good quality data to ensure that their focus can be on delivering insights into our data as opposed to time-consuming and manual transformation of the data to prepare it for analytical use. Enter more details about this user type: View live dashboards, which are fed by constant streams of data from our various systems, to gain insights to council performance in relation to defined metrics and performance indicators. The ability to filter dashboards to view historic data and highlight potential emerging trends in our performance data will allow performance leads to make data-driven decisions and address potential issues before they arise. Enter more details about this user type: View live dashboards, which are fed by constant streams of data from our various systems, to obtain a single view of our citizens, combining data from multiple sources to visualise all data we hold on a citizen. This will allow for efficiencies in multiple processes, through the removal of manual data matching and sourcing of relatable data from other services and teams. Confidence in the data platform’s ability to present real-time data ensures that any decision made with citizen data is done so well-informed. Council officers should also have access to relevant data sources, through managed-access levels, to support a self-service first approach to data analytics within the council. Enter more details about this user type: Must be involved with the development and delivery of the proposed solution with elevated privileges to azure components of the solution to gain understanding of configuration of solution. Should be able to understand the developed solution and be able to work closely with the chosen supplier during design and build. This feeds into the need for knowledge sharing throughout the project. Enter more details about this user type: Azure administrators must be involved in the development of this solution to understand the resources enabled and their configuration. Defined roles to grant and revoke access to specific datasets must be granted to these users to allow for management of the solution beyond the end of the project. Enter more details about this user type: Policy makers must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable policy makers to make data-driven decisions. Enter more details about this user type: Senior leadership must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Senior leadership to make data-driven decisions. Enter more details about this user type: Chief officers must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Chief officers to make data-driven decisions Enter more details about this user type: Elected members must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. Having access to these interactive dashboards will allow elected members clear and concise access to relevant data to support decision making.