High Performance Storage

Tender description

This project aims to expand the existing high performance storage (HPS) system used to store scientific data, diagnostic and surveillance analysis and to support the centralized sequencing services. Currently the existing high performance storage system relies on a single installation based on open source Lustre technology. The current Lustre installation has 300TB of usable capacity and it has been engineered following industry best practices, making the system high performing and very reliable. It is currently using approximately 65% of its usable capacity. The data growth trend indicates that at the current increasing rate this will reach capacity sometime in the next 8 to 12 months. For this reason new storage resources will be deployed using the same open-source high performance technologies based on Lustre. The Lustre architecture easily allows the expansion of the storage tier, although in this case it is more appropriate to implement a new separated Lustre system which will have a number of benefits detailed below in the related sections. Project Objectives • Install and commission the hardware and software for a new separate Lustre filesystem which can be presented to the high performance cluster. • Integrate new filesystem with the existing iRODS data management system, including integration with the scientific data catalogue and data curation procedures. • Provide increased overall reliability by having two separate Lustre file systems which will allow maintenance to be performed on each one independently without affecting the critical services they provide. • The new filesystem will be physically and logically separate from the current one allowing for the maximum data isolation, therefore providing an additional level of security when required. • Knowledge transfer and technical training on Lustre and iRODS (integration) open-source technologies. • Implement the Robinhood Lustre module to optimise the Lustre filesystems.