This talk was part of a larger session on Error Bounded Lossy Compression. It presents updates on LibPressio since the last annual meeting focusing on the automated configuration work completed as part of FRaZ and LibPressio-Opt as well as interface improvements.
Compression is commonly used in HPC applications to move and store data. Traditional losslesscompression, however, does not provide adequate compression of floating point data often found inscientific codes. Recently, researchers and scientists have turned to lossy compression techniquesthat approximate the original data rather than reproduce it in order to achieve desired levels ofcompression. Typical lossy compressors do not bound the errors introduced into the data, leading to the development of error bounded lossy compressors (EBLC). These tools provide the desired levelsof compression as mathematical guarantees on the errors introduced. The current state of EBLCleaves much to be desired. The existing EBLC all have different interfaces requiring codes to bechanged to adopt new techniques; EBLC have many more configuration options than theirpredecessors, making them more difficult to use; and EBLC typically bound quantities like pointwise errors rather than higher level metrics such as spectra, p-values, or test statistics thatscientists typically use. My dissertation aims to provide a uniform interface to compression and todevelop tools to allow application scientists to understand and apply EBLC. This canvas presentsthree groups of work: LibPressio, a standard interface for compression and analysis;FRaZ/LibPressio-Opt frameworks for the automated configuration of compressors using LibPressio;and work on tools for analyzing errors in particular domains
As time progresses, the volume of data surrounding machine learning and AI methods continues to grow from training, testing, and validation datasets to the models themselves. As the volume grows, there are increasing challenges in transporting and storing the data. Lossy compression techniques present the opportunity to drastically reduce the volume of data while maintaining or even improving upon the quality of the decisions made by AI. This talk presents a survey of novel research examining the effect of lossy compression on AI decision making in a variety of domains including medical data science and physics. We examine the effects of data ordering, error bounds, compression methodologies to make general recommendations from these and other areas regarding how to effectively leverage lossy compression in AI using the common LibPressio interface.
With ever-increasing volumes of scientific floating-point data being produced by high-performance computingapplications, significantly reducing scientific floating-point datasize is critical, and error-controlled lossy compressors have beendeveloped for years. None of the existing scientific floating-pointlossy data compressors, however, support effective fixed-ratiolossy compression. Yet fixed-ratio lossy compression for scientificfloating-point data not only compresses to the requested ratio butalso respects a user-specified error bound with higher fidelity. Inthis paper, we present FRaZ: a generic fixed-ratio lossy com-pression framework respecting user-specified error constraints.The contribution is twofold. (1) We develop an efficient iterativeapproach to accurately determine the appropriate error settingsfor different lossy compressors based on target compressionratios. (2) We perform a thorough performance and accuracyevaluation for our proposed fixed-ratio compression frameworkwith multiple state-of-the-art error-controlled lossy compressors,using several real-world scientific floating-point datasets fromdifferent domains. Experiments show that FRaZ effectively iden-tifies the optimum error setting in the entire error setting space ofany given lossy compressor. While fixed-ratio lossy compressionis slower than fixed-error compression, it provides an importantnew lossy compression technique for users of very large scientificfloating-point datasets.
This video provides a basic tutorial of how to install and use LibPressio – a generic abstraction for lossless and lossy compression for dense tensors. You can find LibPressio on github
Error Bounded Lossy Compression is a powerful technique that combines the power of lossy compression with the strong guarantees for the preservation of the original data. However, using error bounded lossy compression can be challenging. There is a proliferation of interfaces, different semantics, and difficult to understand relationships to user error bounds. This talk presents two tools — LibPressio and FRaZ — that help address these issues. LibPressio provides a usable abstraction across SZ, ZFP, MGARD, as well as traditional image compressors (JPEG, WEBP, PNG, TIFF, etc) and lossless compressors via BLOSC. FRaZ is a tool that uses LibPresssio to compress datasets according to a Fixed Compression Ratio with progress towards supporting arbitrary user error bounds. I present our recent work submitted to IPDPS and current work in progress.
Techniques for predicting error bounded lossy compression ratios from data distribution and compressor settings.
C++ templates are one of the most powerful, and probably least understood aspects of C++ programming. Trying to understand error messages from templates code can feel like staring into the abyss. In this talk, I introduce many of the various features and dive into applications of C++ templates that are used in the wild. I will cover several of the common pitfalls and design patterns that result from this feature of C++. I will also stare right into the heart of the abyss and explain why the error messages can be cryptic, why its a good thing, and how to begin to decipher those error messages.
This talk introduces Docker and why you should care about it. It highlights differences between Docker and VMs while clarifying common misconceptions. I talk about how to create docker containers and containerize applications.
Ever sit down to setup a set of computers and wonder, “why on earth do I have to do this manually on each and every node?” Writing scripts can solve the problem, but handling error is annoying and time consuming. It does not have to be that way. Configuration management tools such as Ansible, SaltStack, and Puppet take the pain out of managing small to large groups of computers. By the end of the presentation, You’ll learn some of the strengths and weaknesses of each so that you can decide which tool will fit your needs.