Windows 10 Automated Testing

Tender description

Summary of the work Automated application testing Latest start date Monday 3 July 2017 Why the Work is Being Done The Trust has an ongoing programme to move users and applications to Windows 10 from XP. The Trust has a requirement to improve the quality, frequency and speed of regression testing of its applications to assist this programme. Work will need to be completed by 31st October 2017 Problem to Be Solved Create automated regression testing for applications Who Are the Users As a Testing Manager, I need to automate regression testing for applications so that I can increase the efficiency of the FIT / Windows 10 Programme Existing Team The Supplier will be working with the FIT (Windows 10) project team. They work closely with SCC who provide desktop equipment as a managed service to the Trust. Current Phase Discovery Skills & Experience • Demonstrate your company’s track record and experience in delivering Test Automation for web and non web applications? What ROI has been achieved in your previous projects? (max response 2 pages) • Demonstrate your company’s key skills and capabilities in Test Automation? In what way do you’re skills and capabilities contribute to the test assignment described? (max response 3 pages) Work Location Education Centre, York Road, London, SE1 7NJ Working Arrangments There will be key meetings where attendance on site in London is required but the majority of the work can be done remotely with virtual meetings. Resources can be a mix of on-shore and off-shore, however no data is to be transported off-shore. i.e. off-shore resources can login in remotely and create scripts etc. There will be no expenses available to the supplier. Additional T&Cs Supporting documentation can be found on the Delta eSourcing web site using access code TB9DJ2GADQ. The Delta system can be found at https://www.delta-esourcing.com/. If you experience any problems please contact the Delta Helpdesk on 0845 270 7050 or by online via their website for further assistance. Please note that you will need to register with Delta to access any information. No. of Suppliers to Evaluate 3 Proposal Criteria • What major challenges have you experienced in the delivery of previous Automation projects and how have you overcome them? (max response 2 pages) • Describe how you would implement an automated testing capability to provide the best VFM (Max 10 pages for a-g inclusive) a. Initial engagement and creation of an Automation test strategy • b. Engagement with key stakeholders in IT and the end user community • c. Knowledge Transfer and articles required to learn about the 50 applications • d. Analysis of the first 50 applications, capturing the key automation skills sets required • e. Selection of the optimal automation tools for a sustainable, agile automation framework and regression test pack to accommodate different types of applications • f. Selection of the automation tools in relation to the Trust’s requirement to transfer and implement the tools within the Trust • g. Outline plan for delivery including environments, devices, pcs, people resources (onsite and offsite), license agreements for software required to be supplied by the Trust to support delivery plan • Describe the minimum requirements, dependencies for the automation to be delivered. (max response 2 pages) • Describe approach and timeline for transitioning the ability for the Trust to maintain its automated testing of applications and the roles needed in GSTT to support application testing. (3 pages) • Give examples you are achieving for organisations of implementing Win 10 automated testing of applications aligned to regular releases from Microsoft of the Current Branch of Business builds. (2 pages) Cultural Fit Criteria • Work as a team with our organisation and other suppliers • Honesty and transparency about progress Payment Approach Fixed price Evaluation Weighting Technical competence 65% Cultural fit 5% Price 30% Questions from Suppliers 1. Given the "page limits" below & that the Digital Marketplace word limit for each question is 100 words, does the Trust have any experience of procuring services through the Digital Marketplace at all? "Demonstrate your company’s track record and experience in delivering Test Automation for web and non web applications? What ROI has been achieved in your previous projects? (max response 2 pages) Demonstrate your company’s key skills and capabilities in Test Automation? In what way do you’re skills and capabilities contribute to the test assignment described? (max response 3 pages)" Apologies. Clearly we did not understand the rules/limitations of the portal when we issued this requirement. Please see following responses to see suggested route for responses. 2. Supplier are only permitted to respond with 100 words max for questions in the essential and nice to have skill sets. Its not feasibly possible to provide 2-3 pages of answers as requested. How do wish to proceed? Please respond as completely as possible through the Digital Marketplace. Where you can not respond on the portal please put 'See Delta for full response' on your Digital Marketplace response and upload the full response onto the Delta portal. 3. Do you prefer for applications to be via Digital Marketplace (limited to an initial response of 100 words per Essential Requirement) or complete tender including supporting documentation via delta e-sourcing? Please respond as completely as possible through the Digital Marketplace & then upload a full document onto Delta as well 4. Could you also confirm if the opportunity can be applied for externally to the framework Please ensure you apply through the Digital Marketplace with any additional documents uploaded onto Delta 5. Page 3, 1.3 states "The current testing regime is described in the Faster IT Test Strategy (see supplied document in the procurement pack). " We don't seem to have this Strategy? Where can we find it? This document has now been uploaded onto the Delta portal 6. The requirement states a need to integrate data from at least one new source. Can you please advise if the Trust has an integration technology in place and if so, which one? If not, is the Trust expecting the successful supplier to provide one? GSTT use BIZ TALK Server 2009. Referred to as the TIE (Trust Integration Engine).This is the principle technology used to integrate the Trust applications though there are a limited number of applications that do not utilise the TIE and therefore use separate interfaces eg web services.This project does not include the delivery of new integration technology. 7. Could you us know the application types and browser types involved? For the none browser applications specifically what they are? > Most applications deployed throughout the Trust are 3rd party supplier products> Most applications are deployed by client and this can be a simple thin client with access to a UI up to a thick client including the database> Some 3rd party products have their own internal integration and communication technology> There are a few and important in-house developed products which are available via web> The official supported browsers used are IE 11 and MS Edge. Google Chrome is also used 8. It appears you require the first stage questions on the digital marketplace to be answered by 28th June. However, you also require the procurement document on the Delta portal to be completed. This requires much more detailed answers and evidence as well as binding pricing by the same date. At that point your digital marketplace entry suggests you will create a shortlist of three from which you will select a supplier to start work the following week? Is this correct? Yes the published deadlines, whilst being ambitious, are what the Trust is aiming for. 9. Can you provide an indication on your budget? Also, does the Trust have a preferred supplier in mind? The Trust expecting suppliers to bid competitively against the requirements listed.  The Trust can give a range of £80K to £160K. The Trust has no preferred supplier and prior experience of any framework supplier in this area of expertise so it is a completely open field. 10. Will the SAT tests and the Business process scenarios, be available to the engineers? What is the existing coverage of the user scenarios (we understand not all scenarios currently have a documented test case but perhaps if you could give a rough percentage)? Where they are available e.g. documented tests they will be made available. Approximately 5 %  of apps have documented test cases most have expert users from whom test cases would have to be captured. 11. How many/what types of devices are to be included ? (Including mobile) Tests from mobile devices (e.g. windows mobile. android or IOS phones) are out of scope. Tests to be executed on WIN 10 desktop and laptops via the LAN or WIFI from York Road and VPN (DSL line is available at York Road). 12. Should Data generation/management tools be included in the estimates where necessary? If you believe they are needed for you to meet the requirements please included.For good order sake as a minimum there is a requirement for a UI to upload scripts to a library (especially after change) select and initiate the scripts, report test results and debug the scriptsIf they are optional, feel free to add benefits and costs of these tools under question 12 and if supplier is selected the trust will discuss pro/cons of using the tools later. 13. In the RFP it states “Note not all applications have a test environment” : Would the project investigate how best to test these apps given no dedicated environment? No. If an application is to be tested without at test environment, then the same process as today will be followed. e.g use production with a test user and a test patient, this is currently undertaken manually today 14. Is Security Testing part of the scope? If yes, does Security Testing include Penetration Testing and Vulnerability analysis? Security testing is currently restricted to user authentication and does not include penetration or vulnerability 15. It is mentioned that the support of Test environments is split across the 3rd party suppliers and the Trust. Is Test environment set up and maintenance in scope of the RFP? Test environments management is out of scope 16. It is mentioned license agreements for software required to be supplied by the Trust to support delivery plan but in Response 8 , GSTT has assumed for the purposes of this automation project, the supplier will utilise its own licences.Can you confirm who will procure the licences? Licenses should be supplied by the supplier for the creation of the test automation framework, set up, production and testing of the scripts. GSTT will purchase the licenses for the implementation of the automation for GST full time staff to use once the automation has been delivered by the supplier. 17. Does GSTT have any test management tools like HP ALM? HP tools are not available, however the purchase and implementation of such test tools will be considered if the supplier is dependent upon them for the automation to be delivered to GSTT. Test tools need to be clearly shown as project dependencies. The Trust will purchase test tools to implement the Automation framework and scripts only. 18. For each of the applications to be tested is there an indication of the number of tests cases which exist for each? The identity and number of test cases per application is not available. We estimate approx. 5% of applications have documented test cases. 19. The brief mentions different hardware versions. Is there a list available of the Hardware versions and specifications or even a list of "core builds"? that should be tested Core Builds are currently Microsoft Windows 1511 and we will be upgrading to 1703. 20. Would the Trust consider tools and training that would enable the end users to be able to build and execute their own tests easily? The trust will consider tools and training of full time staff to maintain, update and execute the test scripts delivered. 21. Is there any test automation capability within the organisation at present? If so, what tools are currently used? Are there any ongoing licensing constraints? Are there any preferred tools/skillsets that need to be considered? Test automation does exist in GSTT within the development works stream where tools MS Test and Selenium Webdriver are used. This project has been established for the introduction of Automation into Applications Management team to cater for Deployments of WIN 10 upgrades/patches and BAU projects where no automation currently exists. 22. Will there be a requirement to integrate the chosen automation tool with AppTracker, e.g. reporting test results/managing test execution/raising defects? There is no requirement at this stage in the tender to integrate with Apps Tracker, however there is a requirement for a UI to upload scripts to a library (especially after change) select and initiate the scripts, report test results and debug the scripts. Recommendations can include integration with Apps Tracker. 23. Should updating test scripts to suit automated testing be included in the estimates where necessary? It is the intention of the project to move the maintenance of scripts in house. However support and maintenance to the scripts may be required and so this can be included in the tender response. 24. Is it anticipated that the test execution will be aligned with every Microsoft patch release (monthly)? Test execution to be aligned with the Microsoft current branch for business to be implemented across the Trust estate of desk tops and lap tops. 25. On average, for each of the 50 applications, how many “happy paths” need automating, and what is the average length in terms of user interactions? Is the expectation that all of these paths will be automated? There are approximately 5 to 10 critical processes per application that require to be automated. 26. Have any of these applications been automated in the past? No. The project is designed to deliver automation of regression tests for the first time. 27. Is there any current tooling in place within the trust? There are no test tools currently used for the purpose of Automation or standard test tools for the planning, preparation or execution of regression tests of WIN 10. Automation tools do exist for development projects which are outside the scope of this project. 28. What proportion of the delivered applications are only available over Citrix? For those applications, what is the expectation at to where these will be automated? (It’s pretty hard, from a functional perspective, to automate a Citrix session on a client machine -we’d normally put the tools on the Citrix server and run them from there) In the vast majority of cases clients will be tested directly from the Trust LAN or WIFI from York Road. 29. There are 50 defined P1 apps, only 45 are listed in the attached spreadsheet. Is this a partial list? The final list will be confirmed at the time of the contract. Suppliers should tender for 50 not 45. 30. The proposal talks about regular updates from MS – are there similar updates from the application vendors? Regular updates from Microsoft are not the same as updates from application vendors. The updates from Microsoft are to the operating system “current branch for business” distributed across the Trust’s desk tops and lap tops. 31. Is ongoing maintenance and support of automation assets to be considered in the proposal? Yes 32. Have GSTT performed any analysis to date on their test plans to determine ease of automation for various test scenarios? Yes, test cases do not exist for many applications in the scope for automation. 33. For the test scenarios proposed to be automated, is there an estimate on how long those take to execute manually? Yes, depending on the complexity of the application, some tests take a hours, others can take several days. 34. Are there any existing test automation licenses available? If yes, please can you provide the name of the tool and details for each license. No automation licenses are available. The supplier is expected to have and to use their own licenses for the delivery of the automation framework and test scripts. The Trust will procure licenses for the implementation of the automation. 35. Does GSTT have a test management product? There was reference to AppTracker, is this MigrationStudio AppTracker? Yes Migration Studio Apps Tracker is used currently. 36. Are you able to group how many applications are Desktop, Web and Citrix? Circa 75% Desktop, 20% Web and 5% Citrix 37. Is the supplier expected to also prepare the Manual Regression Test cases where they are not readily available? Yes 38. Are there any restrictions on the tools to be used for automation (opensource/licensed)? The supplier can propose the tools most suited to the test assignment. 39. Is Mobile Automation in scope? At present testing on mobile devices (Windows or android or phone mobile phones) is not in scope. 40. Suppliers are advised that questions 1 & 2 are posed to understand essential skills and experience. ... The responses of suppliers who do not score adequately enough in these 2 questions will be rejected and not evaluated further." - it would seem a lot of effort is required to complete questions 3-12 but suppliers might be rejected after Q1&2. Are you able to break down the tender into two phases and let suppliers who are successful in Q1&2 go on to phase 2? We would like a response to the whole requirement (so not broken into 2 phases) but do understand we are asking for a lot of work so would accept a response solely on the Digital Marketplace in by the deadline. We will then ask shortlisted suppliers to provide full written responses. We accept that some questions may be difficult to answer in this format which is due to our inexperience with this framework. 41. Have the application been migrated? Yes only applications that have been migrated to WIN 10 will be scoped for automated testing. 42. Are there any known issues or failed attempts in the past? No 43. Is testing on Windows XP required? No 44. Do you have access to the source code? In most instances the applications selected for test automation are 3rd party supplier products for which access is not granted to the source code. 45. Are these developed in-house or COTS? In most instances the applications selected for test automation are 3rd party supplier products. 46. Are these all desktop applications? Circa 75% Desktop, 20% Web and 5% Citrix 47. Do these communicate with databases or mainframes? All applications have connection to a database. There are no mainframe applications in scope of this project. 48. What is infrastructure? Some infrastructure is hosted by the 3rd party supplier, other applications are hosted on Trust Data Centres in London. 49. On what technologies have the standalone desktop applications (those are supposed to be tested) have been developed on Too many to list here 50. What is the technology stack for applications under test (e.g. Windows 10 and .net framework or Delphi’s framework etc.)? Base operating system is WIN 10 with a variety of application, database and middleware products. 51. Do we have test cases written or we need to map functionalities with various applications and prepare test cases for each application under test? Test cases do not exist for most applications . A knowledge transfer of the applications and the functionality will be required for the supplier to prepare the test cases. 52. UAT test scripts to be supplied by the apps owner? Are we referring automation test scripts here? (Refer document: FIT Test Strategy, pg#4, Section 3: Assumptions) User Acceptance Tests cases may exist in some limited instances. Where they are available test cases will be given to the supplier for the purpose of automation. 53. Is there any % of automation that exists for the current app? If yes, how much % is automated & what tools are used No 54. We assume that we need to do Non-functional testing - Non-functional Testing would be subject to feasibility and considering varieties of application along with its dependencies on network, it would require substantial amount of time initially to do feasibility study. Do we have to do Network, Performance and Security testing for all the applications? Correct , non functional (stress and performance) tests of application are out of scope. Basic security test are in scope. See list of features to be tested. 55. Can you please confirm that all apps are desktop based only (web or thick client). No mobile or responsive apps Circa 75% Desktop, 20% Web and 5% Citrix 56. Do you want to use AppTracker as Test Management Tool or you want us to propose another tool/s? Supplier can propose test tools suited to the test assignment 57. You have selected 50 application to automate. Can you let us know the number of applications for which the test cases are available and the number of test cases for each application? Very few applications have test cases prepared in advance for the automation. 58. How many applications are Citrix based and are they are hosted on single Citrix Server? Circa 75% Desktop, 20% Web and 5% Citrix. We have load balanced Citrix service. 59. We also need clarification on Security Feature. What do you want us to check on firewall and active directory security? Firewalls will not form part of the scope. This has been agreed following the publication of the tender. Where possible applications which utilise Active Directory security will be automated using the active directory login. 60. Looking at the features you want us to automate. We request more clarification on Network & Connectivity – are you looking for us to ensure applications work with the 4 categories listed (Application availability on Intranet, Application Interfaces/BIZ Talk, LAN/VPN/Citrix and WIFI) Yes all 4 where they apply. In most instances testing will be on client applications accessed via LAN and WIFI at York Road. 61. You mention both bronze and silver scenarios. Please confirm the numbers are fixed or not? We ask this because the number of features for bronze and silver are not the same They are fixed, subject to Firewalls being de-scoped from the tender. Silver has more features than Bronze. Please provide proposals accordingly for each service level.