Automation Skills Injection

Tender description

Summary of the work The Met Office initiated an Application Lifecycle Management (ALM) project to improve the current delivery lifecycle. A new test automation blueprint has been developed. The Met Office is looking for a supplier to help review and deliver the blueprint whilst up-skilling and influencing the current delivery teams. Expected Contract Length 12 months Latest start date Supplier must be able to commence initial engagement activities 1 Dec 2016 Budget Range £200,000 to £400,000 in total. We have currently put “Time and Materials” as our price criteria. However, we are open to any suggestions from suppliers about the price criteria and are willing to review any alternatives. Rate card guide (excluding expenses): • Test Architect - up to £400 • Senior Automation Specialist – up to £350 • Automation Specialist – up to £300 Supplier to provide the resource augmentation price required to deliver the needs outlined in this briefing document. The Met Office will fund any new tooling and/or infrastructure. Why the Work is Being Done The testing capability within the Met Office is being transformed from a Quality Assurance (QA) to a Total Quality Engineering (TQE) model. This involves introducing new practices such as Behaviour Driven Development (BDD) and Specification by Example (SBE). These practices consider quality at the outset and support Continuous Integration which enables a ‘learn fast, learn often’ culture. The current test automation approach was found to be lacking and unable to fully support a Total Quality Engineering / Continuous Integration (CI) model. Problem to Be Solved A recent review identified the following challenges: • No common tooling strategy. • Duplicate tooling (e.g. 9 x Test Management tools). • Over reliance on unscripted testing (~60%). • Only ~20% of products have regression packs • Automation scripts not maintained post-delivery. • Emphasis on GUI automation. Addressing these challenges will: • Increase velocity. • Increase quality. • Increase repeatability. • Shorten feedback cycles (learn fast, learn often) • Reduce the overall time to market. • Reduce testing technical debt. • Drive a continuous improvement culture. • Link test artefacts and test execution results to requirements in a common tool. Who Are the Users Solution Engineering requires Supplier to: • Review test automation blueprint. • Establish and implement a common tooling solution (e.g. toolset, frameworks) • Generate scientific test data. • Provide skills injection (coaching). • Work with teams to deliver automated tests Delivery Team requires a tooling solution with ability to: • Stand-up and teardown a test environment. • Support various test phases and execution triggers (e.g. functional, non-functional, at code check-in, overnight). • Test at different levels of the technology stack (testing pyramid) and support BDD, SBE, TDD and CI. • Provide MI so that the Definition of Done can be assessed. Work Already Done The current test automation approach was found to be lacking and a new test automation blueprint has been developed. The blueprint includes an automation strategy and areas of testing technical debt to be addressed (e.g. unscripted regression testing). The blueprint recommends engaging a 3rd party to help implement the strategy and up-skill the current Met Office teams. Suggested team size is ~5: • 1 x Test Architect • n x Senior Automation Specialist(s) • n x Automation Specialist(s) The Met Office is in the process of purchasing OpenShift from Red Hat to manage some non-production environments and containerise applications. Existing Team • Delivery Teams – Delivery Manager, Technical Lead, Solution Engineers, Product Owner. • As a Service – Solution Architects, Test Architects, Product Managers, Technical Testers, PMO. • Automation capability to be provided by Supplier. Supplier to list roles. Current Phase Beta Skills & Experience • Experience implementing an enterprise wide test automation strategy in an Agile environment. • Delivering tooling solutions that support testing at different levels of the technology stack (testing pyramid) including data presented in graphical tiled format (e.g. weather maps). • Experience with solution engineering practices such as BDD, SBE, TDD, CI. • Working with projects during initiation translating strategic concepts and tooling architectures into practical implementations. • Experience coaching delivery teams. • Able to provide resource with practical experience in a large selection of automated tools and associated scripting languages to include Java. • Delivery Centres based within the EU/EEA. Nice to Haves • Able to provide a formal training capability. • Experience creating scientific test data (e.g. 4 dimensional geospatial weather data, GRIB2, NetCDF, CFNetCDF). • Able to offer additional services that the Met Office may decide to leverage at a future point (e.g. resource augmentation, fixed cost engagements). • Creating a culture of continuous improvement in order to constantly enhance testing principles, processes, techniques, tooling and frameworks. Work Location • Majority of engagements will be based in Exeter, UK. • Option to work remotely as appropriate. Working Arrangments Core team of ~3 to be based onsite with the potential for the other roles to work remotely. Supplier to rotate core team based onsite as appropriate. The Met Office is open to onshore / nearshore solutions. Security Clearance Please consider when responding, the necessity for all individuals to be willing and able to pass SC clearance if attending Exeter site. No. of Suppliers to Evaluate 5 Proposal Criteria • The proposed approach, methodology and experience. • How the approach or solution meets user needs • The proposed technical solution • The lead time to commence the engagement • The value for money of the proposal Cultural Fit Criteria • Work as a team with our organisation and other suppliers • Timeliness and effectiveness of communication • Willingness to engage as a partner • Openness and approach to knowledge transfer/training Payment Approach Time and materials Evaluation Weighting Technical competence 60% Cultural fit 15% Price 25% Questions from Suppliers 1. As part of the work already completed has a tool been selected or is there a short list of perspective tools. If yes can you share the tool or tools please. The automation blueprint identified a list of potential tools. The objective was to rationalise existing tools (e.g. Met Office currently uses 9 x Test Management tools) and standardise tooling frameworks. The initial set of tools identified are as follows although any future partner may wish to recommend alternatives:• SoapUI• Java• Protractor• Selenium• Appium• TestDroid• RiaTest• Ranorex• EggPlant• JMeter• Load Runner• Cucumber• Jira• Zephyr• Jenkins• Confluence• Bitbucket• Maven 2. How was the Blueprint developed? Was this done in house or withexternal assistance? The automation blueprint supports the automation pyramid (testing at all levels of the technology stack) and was produced in conjunction with both internal and external resource. 3. How many delivery teams are there and what is the size of each team and are they each working on similar technologies? We have ~16 squads each containing 6-8 engineers. The squads either work on front end, middleware, core capabilities or a combination of a three. A number of technologies are used however Java is the primary development language. 4. Is there a dedicated environment support team for deployments and managing day-to-day environment related issues and if so would this environment team provide support for implementation of new test tools? OpenShift (PaaS product from Red Hat) is currently being implemented within the Met Office. This technology will provide teams with the capability to build their own environments and install the necessary tooling. The Platforms team will provide OpenShift support and also support any environment not yet transitioned to OpenShift. 5. Are there any preferred testing tools specified in the test automation blueprint? The automation blueprint identified a list of potential tools. The objective was to rationalise existing tools (e.g. Met Office currently uses 9 x Test Management tools) and standardise tooling frameworks. The initial set of tools identified are as follows although any future partner may wish to recommend alternatives:• SoapUI• Java• Protractor• Selenium• Appium• TestDroid• RiaTest• Ranorex• EggPlant• JMeter• Load Runner• Cucumber• Jira• Zephyr• Jenkins• Confluence• Bitbucket• Maven 6. Is the scope of this work limited to test automation strategy implementation or improving overall testing delivery capabilities? The Met Office is looking for a supplier to help review and deliver the test automation strategy while at the same time up-skilling and influencing the current delivery teams. Improving the overall testing delivery capability is not part of this engagement and is being addressed as a workstream within the ALM programme. However, the supplier is welcome to contribute to any continuous improvement initiative across the solution engineering competency. 7. Are the Met Office planning to implement this new test automation strategy retrospectively or only for forthcoming projects? The plan is to implement the new test automation strategy as teams adopt the new way or working prescribed by the ALM programme. This will extend to new product developments, enhancements to the existing product set and when moving products from a legacy platform to an OpenShift container. 8. Does the tooling solutions for different levels of the technology stack include the Unit Testing level, or is this integration and above? The Met Office would like to automate at all levels of the technology stack following the industry standard automation pyramid approach. This covers Unit, Integration and System testing. 9. Are there any defined tools for implementing BDD/SBE/TDD? Some of our squads use Cucumber to help manage Gherkin scenarios but we intend to mature our capability and tooling in this space. Any tooling solution needs to be capable of supporting BDD/SBE/TDD and the broader context of how we gather and communicate requirements. 10. Would BDD/SBE/TDD be integrated into the Automation delivery? Eg. Specflow for C#, Cucumber Eclipse plugin for Java? The intention is that we will mature our capability in the BDD/SBE/TDD space which will then drive test automation. We currently use Cucumber to manage Gherkin scenarios and Java is the primary development language. 11. Would there be an expectation to train the BDD/SBE/TDD to other business areas such as Business Analysts? A separate workstream within the ALM Programme is responsible for training BDD/SBE/TDD. A supplier should be familiar with developing automated solutions that integrate with and support these practices/techniques. 12. What are the User Interfaces expected to be automated? The Met Office currently has around 1500 products which are delivered via GUIs, APIs, Apps, text (e.g. Email). We are currently looking to rationalise this product set however the focus of the engagement is to implement a automated solution which extends across all levels of the technology stack and also to upskill the current squads 13. Is the required language for automation set to JAVA or would it be possible to implemented other languages such as C#? Java is the primary language used in the Met Office; C# is not used. Appreciate that some tools use different languages (e.g. the Loadrunner scripting language is based on C) however, where there is a choice, Java should be selected. 14. Could we deploy our own automation frameworks developed for web and apis long as appropriate training was provided or would complete open source be preferred? The Met Office will be interested in considering any option however carefully consideration would need to be given to any specific licensing costs, the scalability of the framework to accommodate new technologies and the ability to support the framework in-house post engagement with the supplier. 15. What are the results reporting expectations of Automation? Test results should be published back into our core requirements management tooling so that a product owner / squad can determine if the user story has passed / met DoD. 16. What CI Implementation is currently used at the MET office? CI is limited due to tooling, skills and environment constraints. The ALM Programme is looking to address these items and any future supplier will be expected to help contribute in this area. 17. Are there expected outputs such as build radiators to be displayed for automation builds? Yes. The Met Office would like realtime MI so the squad can determine the exact status of any build, deployment, test set, user store, feature, epic. 18. Are the Automation engineers expected to develop build scripts for the environments ( setup and teardown) or is there DevOps support available? OpenShift (PaaS product from Red Hat) is currently being implemented within the Met Office. This technology will provide squads with the capability to build their own environments and install the necessary tooling. The Platforms team will provide OpenShift support although the squads will be expected to write their own environment build scripts. Anyone in the squad could write an environment build script which may or may not include the supplier. 19. Would we be able to review the existing Automation BluePrint before the proposal is delivered? We are unable to share this document at this point in the process. However, it will be made available to all suppliers who make the shortlist. 20. Could there be a consideration for remote regression testing from an onshore service ( overnight remote testing)? The Met Office is willing to consider all onsite / offsite proposals although the objective is to implement an automation strategy and up-skill the squads so that they can maintain and enhance the automated strategy post this engagement rather than simply outsource testing to a 3rd party. 21. Would any expected data be in a common format such as CSV or a Database Format (SQL)? The formats used include GRIB2, NetCDF, CFNetCDF, 4 dimensional geospatial weather data. Descriptions are available online. 22. Is the data manipulation expected to be automated as part of the test ( pre population of environment data before testing starts)? Any automated solution needs to address test data and how it should be injected. Some squads have developed utilities to help with test data creation although a supplier will need to assess the suitability of these utilities. 23. Where unscripted testing is used, could these SME’s be used as an information source for expected regression requirements? Yes. 24. Would anyone expected to be trained have a foundation knowledge of automation/coding? The team dynamic is mixed. Some squads have individuals who possess detailed automation / coding knowledge whilst other squads do not have any individuals with these skills.