Tutorial on I/O data management with PDI
2025, September 30, 9:00-12:00
Maison de la Simulation, together with the Exa-DoST and Exa-DI projects of NumPEx organize a free Tutorial on I/O data management with PDI. This tutorial will be held online via Zoom.
Context
The increase in computational power goes hand in hand with an increase in the amount of data to manage. At Exascale, IO can easily become the main performance bottleneck. Several I/O libraries (FTI, HDF5, netCDF, etc) exist, providing efficient management of I/O data with different focuses respectively. However, the traditional way of tightly coupling the simulation code with external I/O libraries is less flexible and can have an important impact on the maintenance of the code base.
To address this issue of flexibility, we designed PDI, which is an abstraction layer with a plugin system that allows the use of external I/O libraries in a dynamic and flexible way. Users can choose different plugins, via a YAML configuration file, for the desired dataflow without recompiling the simulation code.
Content
This tutorial will present the main concepts of PDI with introductory lectures and incremental hands-on exercises. Several PDI plugins will be covered during the tutorial, including trace, hdf5, pycall, and user_code.
All hands-on exercises will be based on a simple 2D simulation written in C. Participants will learn how to integrate PDI into this example simulation code. They will also learn how to configure the YAML file in order to use different PDI plugins to manage the data generated by the simulation.
The main topics covered by hands-on exercises are:
- Instrument your code with PDI.
- Get the trace information with PDI.
- Use pycall plugin to generate partial images.
- Use hdf5 plugin to generate output files (one file per process, one file per timestep, one file for all timesteps).
- (If time allows) Use user_code plugin to compute in-process metrics on-the-fly.
- (If time allows) Demonstration of in-situ visualization using catalyst plugin.
Organization
Registration is free, but mandatory.
