Expose NetCDF quantize and zstandard functions

[mauzey1]

Resolves #725

NetCDF variable quantization and Zstandard compression can now be enabled in CMOR using the functions cmor_set_quantize and cmor_set_zstandard.

Zstandard compression will only be applied to variables when deflate compression, which is used by default, is disabled. This is done to avoid errors caused by having two compression methods being applied.

[durack1]

@piotr-florek-mohc is there any chance you could pull the 3.9 release to test across the MOHC demo cases?

[durack1]

@mauzey1 apologies for the delay. This looks great to me, and you have a test in place. I have not tested the code, does the latest nightly here have these changes embedded?

[durack1]

I presume that the test suite will be run before merging, right?

[mauzey1]

@durack1 The nightly build will have the changes from this PR when it is merged into main.

I'm currently encountering an issue with the singleton test. It seems that adding the values for quantize and zstandard to the struct cmor_var_ are causing some kind of alignment error. I'll label this PR as a draft until it is fixed.

[durack1]

@wachsylon any chance you could take the CMOR 3.9.0 nightly for a test drive and provide any feedback before we get this released - hopefully this PR will be merged soon, enabling a nightly with the new functionality to test drive

[mauzey1]

@durack1 @wachsylon

I'm currently having issues with understanding why the macOS builds (x86 and arm64) are breaking on GitHub Actions. I was able to get macOS arm64 builds working in a CircleCI pipeline running macOS 14.3.1 with xcode 15.4.0 with this branch's changes. I was also able to get the builds working on my x86 Macbook running macOS 14.6.1 with xcode 15.4.0.

GitHub Action's macOS arm64 runner uses macOS 14.6.1 with xcode 15.4.0. I don't know why I'm getting errors since this configuration should be similar to the ones where I got working builds.

The specific error being that the values in the cmor_var_t struct seem to be misaligned by the addition of the variables zstandart_level, quantize_level, and quantize_nsd.

cmor/include/cmor.h

Lines 438 to 501 in 1e32b83

	typedef struct cmor_var_ {
	int self;
	int grid_id;
	int sign;
	int zfactor;
	int ref_table_id;
	int ref_var_id;
	int initialized;
	int error;
	int closed;
	int nc_var_id;
	int nc_zfactors[CMOR_MAX_VARIABLES];
	int nzfactor;
	int ntimes_written;
	double last_time_written;
	double last_time_bounds_written[2];
	int ntimes_written_coords[10];
	int associated_ids[10];
	int ntimes_written_associated[10];
	int time_nc_id;
	int time_bnds_nc_id;
	char id[CMOR_MAX_STRING];
	int ndims;
	int singleton_ids[CMOR_MAX_DIMENSIONS];
	int axes_ids[CMOR_MAX_DIMENSIONS];
	int original_order[CMOR_MAX_DIMENSIONS];
	char attributes_values_char[CMOR_MAX_ATTRIBUTES][CMOR_MAX_STRING];
	double attributes_values_num[CMOR_MAX_ATTRIBUTES];
	char attributes_type[CMOR_MAX_ATTRIBUTES]; /*stores attributes type */
	char attributes[CMOR_MAX_ATTRIBUTES][CMOR_MAX_STRING]; /*stores attributes names */
	int nattributes; /* number of attributes */
	char type;
	char itype;
	double missing;
	double omissing;
	double tolerance;
	float valid_min;
	float valid_max;
	float ok_min_mean_abs;
	float ok_max_mean_abs;
	char chunking_dimensions[CMOR_MAX_STRING];
	int shuffle;
	int deflate;
	int deflate_level;
	int zstandard_level;
	int quantize_mode;
	int quantize_nsd;
	int nomissing;
	char iunits[CMOR_MAX_STRING];
	char ounits[CMOR_MAX_STRING];
	int isbounds;
	int needsinit; /* need to be init or associated to file */
	int zaxis; /* for z vars, associated axis stored here */
	double *values;
	double first_time;
	double last_time;
	double first_bound;
	double last_bound;
	char base_path[CMOR_MAX_STRING];
	char current_path[CMOR_MAX_STRING];
	char suffix[CMOR_MAX_STRING];
	int suffix_has_date;
	char frequency[CMOR_MAX_STRING];
	} cmor_var_t;

I detected the errors in test_singletons.c where cmor_vars[var_id].ndims is set to -1, which is the value used in singleton_ids for unused IDs. If you tried looping through cmor_vars[var_id].singleton_ids using num_axes in place of cmor_vars[var_id].ndims in the for loop, you will see it going through -1 values until it reaches what appears to be the IDs in axes_ids.

cmor/Test/test_singletons.c

Lines 45 to 54 in 1e32b83

	// Find singleton dimension for bs550aer
	for(i = 0; i < cmor_vars[var_id].ndims; ++i){
	singleton_id = cmor_vars[var_id].singleton_ids[i];
	if(singleton_id != -1) {
	if(strcmp(cmor_axes[singleton_id].id, "wavelength") == 0)
	wavelength_found++;
	}
	}
	if(wavelength_found != 1)
	fail("error in singleton dimension");

I don't know what causes this issue nor why it only occurs in the GitHub Action builds and not anywhere else.

[mauzey1]

I have found the problem. It turned out that the build in the PR was using a branch (main?) that was different from the source branch that contained the changes. This caused the misaligned values when building tests using the source branch's headers while using the PR build's library.

I have fixed it by making the PR build use the source branch.