PySpark and Agile Technical Support

Tender description

Why the Work is Being Done With the UK due to leave the European Union in 2020, the focus on trade and investment statistics has never been higher. The ONS Trade and Investment Development Project was established in 2016 in order to meet the user needs of key stakeholders across government, particularly Department for International Trade (DIT). At the heart of this project is the need to develop and transition both new and existing processing systems on the office's strategic technology. Throughout 2020/21 ONS requires the services of a supplier who can provide a small team with expert technical skills in Cloudera and Amazon Web Services software development and Agile Project Management. This team will be integrated into an existing project team and work through an established product backlog to deliver system capability to a business as usual team who are responsible for the publication of new and improved trade and investment statistics for our users. This small team should consist of: 3 x Cloudera (essential) & AWS (desirable) software developers 1 x Agile Project Manager Problem to Be Solved With the UK leaving the EU in 2020, the government will be negotiating new bilateral trading partnerships across the globe. In addition to this, wider industry has a need to understand and monitor the levels and impact on current UK trade and investment. In order to meet these objectives effectively, we require more diverse and granular data and statistics and therefore the systems and capability to deliver these. The ONS strategic platform for statistical production is based on both PySpark by Cloudera and Amazon Web Services technology. Within Economic Statistics Group, where this project is based, there is currently a shortage of staff with expert skills required to take forward the development, test and production of statistical pipelines at pace. Who Are the Users As a development manager, I need to ensure the project team has expert skills in PySpark and Amazon Web Services software development and Agile Project Management, so that we can develop, test and productionise statistical transformation pipelines at pace. As a business as usual team member, I need to be able to utilise newly developed statistical pipelines to analyse and publish new trade and investment statistics, so that my users have the right insight to meet their needs. As a user of trade and investment statistics, I need to be able to have access to the latest trade and investment data from ONS in a timely manner, so that I can understand the current picture and use this to inform future policy direction. Early Market Engagement Not applicable Work Already Done The Trade and Investment Development Project has been in-flight since 2016 and produced a range of system, data and analysis outputs to date. The project has a well-established team comprising of Agile Project Managers, Product Owners, Technical Leads and specialist delivery teams all working from a defined product backlog. It has developed a Cloudera cluster, which is highly performant and requires programmes written in PySpark. Data has already been loaded onto this cluster. An efficient development cycle involving test and continuous integration already exists. It is governed by the wider Economic Statistics Transformation Programme (ESTP) via project and programme boards. Existing Team Rob Pike - Deputy Director, Global Trade & Investment (Project Executive) Luke James - Assistant Deputy Director, Global Trade & Investment (Senior Project Manager) Mel Gore - Global Trade & Investment (Product Manager) Leigh Skuse - Global Trade & Investment (Agile Project Manager) Remainder of the project team comprises of 5 distinct specialist delivery teams (circa 5 - 8 ppl) ranging across data analysis, methodological design and system development. Current Phase Live Work Location Office for National Statistics Government Buildings Cardiff Road Newport NP10 8XG Working Arrangments The supplier is expected to work alongside the existing project team, on site in Newport. The project is governed using agile principles and practices and therefore the supplier is expected to take a full role in these (incl. working in sprints, presenting at show and tells, contributing to retrospectives). The supplier will also be expected to work openly, taking part in team events where necessary and use digital collaborative tools to engage with the wider project team. Security Clearance Security Clearance (SC) is required for the 3 x PySpark & AWS developers Additional T&Cs Skills & Experience Developers – strong experience in development of relational database systems using the Cloudera stack (particularly Pyspark, Hive and Impala SQL) 20% Developers – able to produce technical documentation to support the development of production systems 5% Developers – experienced in development of Extract Transform and Load (ETL) pipelines 10% Developers – experience of developing continuous integration solutions (particularly in Jenkins) 3% Developers - experience in version control and software review using GIT 2% Agile Project Manager – strong experience of leading large multi-functional project teams within a statistical environment 20% Agile Project Manager – expert at planning complex cross-organisational project deliverables, including the management of stakeholders and dependencies 10% Agile Project Manager – expert in managing risk and removing blockers which could impede delivery 5% All team member – proficient in the use of Jira for work item management 5% All team members – experience of working across the full development lifecycle 5% All team members - awarness of good data security procedures and adherence to ONS secure information management policy 5% All team members - ability to work collaboratively with members in other technical roles including Business Analysts and Methodologists 10% Nice to Haves Developers – experience of data management 40% Agile Project Manager – certified Scrum Master (or similar) 40% Agile Project Manager – Prince2 Practitioner (or similar) 20% No. of Suppliers to Evaluate 3 Proposal Criteria How the approach or solution meets user needs How the approach or solution meets your organisation’s policy or goal Value for money Cultural Fit Criteria Be comfortable working at pace Works to overcome blockers without allowing them to derail progress Work as a team with our organisation and other suppliers Be transparent and collaborative Coach other team members including sharing technical knowledge and experience Works closely with stakeholders and takes collaborative decisions Payment Approach Time and materials Assessment Method Work history Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers No questions have been answered yet Budget range maximum day rate of £680 (plus vat) for each team member. total contract value for 2020/21 must not exceed £830k.