SDRBench — Scientific Data Reduction Benchmark

SDRBench provides reference scientific datasets for benchmarking lossy and lossless data reduction (compression) techniques: climate, weather, cosmology, molecular dynamics, quantum chemistry, turbulence, combustion, X-ray imaging and fusion simulations from DOE laboratories and their partners.

This organization hosts an unmodified mirror of the SDRBench datasets, which are originally distributed by Argonne National Laboratory through Globus. Every file is byte-for-byte identical to the one in the original SDRBench archive (sha256 verified), and the mirror is kept in sync automatically.

pip install sdrbench

import sdrbench
sdrbench.list()                      # available datasets
nyx = sdrbench.dataset("nyx")
nyx.fields                           # field names
x = nyx["temperature"]               # numpy array, C order

• Website and documentation: https://sdrbench.github.io/
• Python package and sync pipeline: https://github.com/szcompressor/sdrbench
• Maintained by the SDRBench team (Argonne National Laboratory and collaborators)

Citation. Please cite SDRBench and the data provider named on each dataset card:

@inproceedings{zhao2020sdrbench,
  title     = {{SDRBench}: Scientific Data Reduction Benchmark for Lossy Compressors},
  author    = {Zhao, Kai and Di, Sheng and Liang, Xin and Li, Sihuan and Tao, Dingwen and Chen, Zizhong and Cappello, Franck},
  booktitle = {2020 IEEE International Conference on Big Data (Big Data)},
  pages     = {2716--2724},
  year      = {2020}
}