Met Office SPF ExCALIBUR EX20- 6: I/O & Storage

Tender description

<p>Successful Exascale simulations will require optimal flow of exabytes of data from long-term storage units, where the data are distributed in some way, through a multitude of increasingly smaller but faster storage devices until appropriately small packages of data (distributed in a completely different way to how they started out) reach the relevant processing entity which will have orders of magnitude smaller memory than the initial data set. Some of that data will need to be shared with other processing entities. Then at some point all those packages of data will need to journey back through the deep memory hierarchy to end up back in the long-term storage units. This aspect is amplified when ensembles of simulations are used for uncertainty quantification. This data flow can be broadly captured by the term I/O. The optimisation of I/O through all the different storage media is a huge challenge but one that must be addressed to make Exascale simulations viable. However, that optimisation is likely to vary depending on the precise details of each simulation and what storage is available, which might be a function of time (e.g. where resources are shared with other applications). It will also vary, potentially radically, when new storage media are added to a system or when the same application is run on a different machine. It is unreasonable to expect a user, no matter how expert that user is, to know how to optimally configure each simulation nor should they need to know how the data was written in order to be able to effectively interact with it. This activity will explore how this problem can be addressed, develop prototype implementations of potential solutions, demonstrate their effectiveness, and propose a roadmap for how use cases can adopt such solutions for practical large-scale applications. As an example, a potential solution might be a separation-of-concerns approach, implemented via some form of middleware, that separates the domain scientist from the detail of the storage media and allows either automatic or bespoke optimisations to be applied dependent on the target system architectures (akin to the separation-of-concerns approach being developed for alternative parallelisation paradigms).</p> Keywords: I/O and Storage