Demo use of dachar and daops for fixing data

[bouweandela]

It would be nice to demo how we could use the character/fixes defined by dachar to fix data with daops and then convert that from an xarray to iris cube for use in the ESMValCore.

Example fixes:

Fix standard name:

ESMValCore/esmvalcore/cmor/_fixes/cmip5/hadgem2_cc.py

Lines 34 to 55 in ea6618c

	class O2(Fix):
	"""Fixes for o2."""
	def fix_metadata(self, cubes):
	"""
	Fix standard and long names.
	Parameters
	----------
	cube: iris.cube.CubeList
	Returns
	-------
	iris.cube.CubeList
	"""
	std = 'mole_concentration_of_dissolved_molecular_oxygen_in_sea_water'
	long_name = 'Dissolved Oxygen Concentration'
	cubes[0].long_name = long_name
	cubes[0].standard_name = std
	return cubes

Fix fill value:

ESMValCore/esmvalcore/cmor/_fixes/cmip5/cesm1_bgc.py

Lines 8 to 26 in ea6618c

	class Gpp(Fix):
	"""Fixes for gpp variable."""
	def fix_data(self, cube):
	"""Fix data.
	Fix missing values.
	Parameters
	----------
	cube: iris.cube.Cube
	Returns
	-------
	iris.cube.Cube
	"""
	data = da.ma.masked_equal(cube.core_data(), 1.0e33)
	return cube.copy(data)

Fix hybrid sigma coordinate (more complicated, only if time permits):

ESMValCore/esmvalcore/cmor/_fixes/cmip6/cesm2.py

Lines 11 to 76 in ea6618c

	class Cl(Fix):
	"""Fixes for ``cl``."""
	def _fix_formula_terms(self, filepath, output_dir):
	"""Fix ``formula_terms`` attribute."""
	new_path = self.get_fixed_filepath(output_dir, filepath)
	copyfile(filepath, new_path)
	dataset = Dataset(new_path, mode='a')
	dataset.variables['lev'].formula_terms = 'p0: p0 a: a b: b ps: ps'
	dataset.variables['lev'].standard_name = (
	'atmosphere_hybrid_sigma_pressure_coordinate')
	dataset.close()
	return new_path
	def fix_data(self, cube):
	"""Fix data.
	Fixed ordering of vertical coordinate.
	Parameters
	----------
	cube: iris.cube.Cube
	Input cube to fix.
	Returns
	-------
	iris.cube.Cube
	"""
	(z_axis,) = cube.coord_dims(cube.coord(axis='Z', dim_coords=True))
	indices = [slice(None)] * cube.ndim
	indices[z_axis] = slice(None, None, -1)
	cube = cube[tuple(indices)]
	return cube
	def fix_file(self, filepath, output_dir):
	"""Fix hybrid pressure coordinate.
	Adds missing ``formula_terms`` attribute to file.
	Note
	----
	Fixing this with :mod:`iris` in ``fix_metadata`` or ``fix_data`` is
	**not** possible, since the bounds of the vertical coordinates ``a``
	and ``b`` are not present in the loaded :class:`iris.cube.CubeList`,
	even when :func:`iris.load_raw` is used.
	Parameters
	----------
	filepath : str
	Path to the original file.
	output_dir : str
	Path of the directory where the fixed file is saved to.
	Returns
	-------
	str
	Path to the fixed file.
	"""
	new_path = self._fix_formula_terms(filepath, output_dir)
	dataset = Dataset(new_path, mode='a')
	dataset.variables['a_bnds'][:] = dataset.variables['a_bnds'][::-1, :]
	dataset.variables['b_bnds'][:] = dataset.variables['b_bnds'][::-1, :]
	dataset.close()
	return new_path

The plan is that:

• @agstephens or one of his team members will prepare the character and fix .json files with dachar and possibly implement the fixes mentioned above in daops.
• Someone from @ESMValGroup/esmvaltool-coreteam (@jvegasbsc would you have time?) Will make a demo how this could be used to load the fixed data into the ESMValCore.

Related to #755

[jvegreg]

@jvegasbsc would you have time?

Yes, I can do it. It could save me lot's of time, so it is a good investment

[valeriupredoi]

way to go, Javi! 🍺

[agstephens]

@ellesmith88, tagging you on this issue as a collaboration point with ESMValTool.

[ellesmith88]

Hi @zklaus @nielsdrost @bouweandela @jvegasbsc @valeriupredoi and @agstephens I've been working on this on our end and have two PRs open with the json files and fix classes in dachar and the fixes implemented in daops: roocs/daops#40 and roocs/dachar#88.

This issue (roocs/daops#38) and my step by step comment should explain what I've done - I have looked at all fixes now though. Hopefully that makes sense but let me know if it doesn't. It would be great to get some feedback and see what you think.

[bouweandela]

We discussed this recently at an IS-ENES3 meeting with @agstephens @cehbrecht @nielsdrost @jvegasbsc @remi-kazeroni and @valeriupredoi. The conclusion was that we are still very interested in doing this and think it's a great idea, but @jvegasbsc indicated that he does not have time to work on this at the moment and no-one else volunteered either. We may need to find new funding to work on this further.

[github-actions]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 720 days of inactivity.

If this issue is still important to you, please comment below to remove the stale label. Otherwise, this issue will be automatically closed in 60 days.