Wang, Chen; Snir, Marc; Mohror, Kathryn (2020). High Performance Computing Application I/O Traces. In Lawrence Livermore National Laboratory (LLNL) Open Data Initiative. UC San Diego Library Digital Collections. https://doi.org/10.6075/J0Z899X4

The dataset comprises trace files from high performance computing (HPC) simulations. The trace files contain records of every I/O operation executed by a simulation application run, including I/O operations from HDF5, MPI-IO, and POSIX and all of the parameters supplied to those operations, e.g. file name, offset, and flags. The traces are generated by executing a simulation application that is linked with the Recorder tracing tool (https://github.com/uiuc-hpc/Recorder). The Recorder trace tool intercepts the I/O calls made by the application, records the I/O trace record, and then calls the intended I/O call so that the operation executes.

For each application execution, Recorder generates a set of trace files. The files are in a binary format (in this collection, .ift containing trace records and .mt containing trace metadata) that is optimized to reduce the overhead of trace collection on the application. The tool recorder2text.c in Recorder/tools can be used to parse and generate a human readable text file from the binary traces. There are also scripts in the Recorder/tools/reporter directory that can be used to perform different kinds of analysis on the traces, or can serve as examples of how to parse and analyze the binary trace data. The detailed documentation on how to read and visualize Recorder traces can be found at: https://github.com/uiuc-hpc/Recorder

For each application execution, Recorder generates a set of trace files. The files are in a binary format (in this collection, .ift containing trace records and .mt containing trace metadata) that is optimized to reduce the overhead of trace collection on the application. The tool recorder2text.c in Recorder/tools can be used to parse and generate a human readable text file from the binary traces. There are also scripts in the Recorder/tools/reporter directory that can be used to perform different kinds of analysis on the traces, or can serve as examples of how to parse and analyze the binary trace data. The detailed documentation on how to read and visualize Recorder traces can be found at: https://github.com/uiuc-hpc/Recorder

Version update. On 2021-02-02, trace files from 4 applications (HACC-IO, MILC-QCD, pF3D-IO, and VPIC-IO) were added to this dataset. In addition, runs from 1024 scale processes were added to the data for all 17 applications.

• 2020-07

• 2020

• Mohror, Kathryn
• Snir, Marc
• Wang, Chen

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under the DOE Early Career Research Program. This work was partially supported by NSF SHF grant 1763540 . This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-MI-811381).

• High Performance Computing (HPC) application I/O
• High performance computing (HPC) I/O library performance
• I/O performance

• data

• No linguistic content; Not applicable

Identifier: Chen Wang: https://orcid.org/0000-0001-9297-0415

Identifier: Kathryn Mohror: https://orcid.org/0000-0002-1366-1655

Identifier: Marc Snir: https://orcid.org/0000-0002-3504-2468

Doi: https://doi.org/10.6075/J0Z899X4

Primary associated publication

• Chen Wang, Jinghan Sun, Marc Snir, Kathryn Mohror, Elsa Gonsiorowski, "Recorder 2.0: Efficient Parallel I/O Tracing and Analysis," LLNL-CONF-80286, in the HPS Workshop at 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), New Orleans, LA, USA, 2020. https://doi.org/10.1109/IPDPSW50202.2020.00176

Software

• GitHub repository for Recorder tools: https://github.com/uiuc-hpc/Recorder

Creative Commons Attribution 4.0 International Public License

• Lawrence Livermore National Laboratory and University of Illinois at Urbana-Champaign

Under copyright (US)

Use: This work is available from the UC San Diego Library. This digital copy of the work is intended to support research, teaching, and private study.

Constraint(s) on Use: This work is protected by the U.S. Copyright Law (Title 17, U.S.C.). Use of this work beyond that allowed by "fair use" or any license applied to this work requires written permission of the copyright holder(s). Responsibility for obtaining permissions and any use and distribution of this work rests exclusively with the user and not the UC San Diego Library. Inquiries can be made to the UC San Diego Library program having custody of the work.

Research Data Curation Program, UC San Diego, La Jolla, 92093-0175 (https://lib.ucsd.edu/rdcp)

2023-06-06