NERC Data Labs Phase 4

Tender description

NERC JASMIN Data Labs development capital plan As part of the JASMINX proposal NERC has provided 120K to enable further developments of the NERC Data Labs project. NRC DataLabs underpins a number of UKRI projects supporting innovative use of NERC data combined with new computational and analytical methods using JASMIN as the host computing platform (Digital Environment and Digital solutions) . In this way, Data Labs leverage novel use of NERC's data assets and reduce the technical barriers to exploitation of JASMIN investment by the wider research community. Currently, NERC Data Labs provide a range of functionality to support research collaboration including:- • Secure sign on and allocation of users to individual "data labs" where development projects are carried out on JASMIN • Flexible data storage using GlusterFS including use of parallel data cluster services such as DASK and Spark • Use of the executable notebook technologies such as Jupyter, Zepplin and R • Hosting of lightweight applications such as RShiny to allow rapid collaboration among diverse research teams • Containerisation of application deployment (e.g. using Docker) so that technologies developed can be more easily moved to other cloud platforms as required. NERC Data Lab development will move the NERC Environmental Data Service toward the paradigm of data centres as data analytics service providers where the variety and volume of data available to researchers require them to have more flexible, cloud-based research environments where analytical methods are move to and run on-demand against the dynamic data resources within the data centre rather than repeatedly downloaded. Hence, the continued investment in Data Labs development is key to enable the NERC EDS to meet new demands of research for new data intensive and transdisciplinary research (e.g. as promoted by UKRI SPF calls). Profile of work 1. Provide a visual workflow engine interface to be able to build, share and publish workflows in DataLabs. Provide an editable visual representation of the workflow. This will be dependent on funds and may have to be a scoping exercise but we can learn from the DAFNI development 2. Improve the user interface so users can see their compute usage and clean up their resources once their jobs have completed 3. Provide tighter coupling and file level control (ACL's) between the Caringo petabyte object store and Datalabs. This will reduce the requirement for users to have a JASMIN account to access the object store and allow access to data stored in the Group Workspaces. This is dependent on JASMIN infrastructure resource. 4. Provide automated sign up to DataLabs 5. Provide dashboarding solutions to enable customisable front ends to Jupyter notebooks using tools such as Panel, Voila and Dash