# Changelog ## v2.15.0 ### Highlights Improvements for supporting CORDEX data including: - CMOR tables for CORDEX-CMIP6 (see [Pull request #3112](https://github.com/ESMValGroup/ESMValCore/pull/3112)) - Dataset fixes (see [Pull request #3033](https://github.com/ESMValGroup/ESMValCore/pull/3033), [Pull request #3110](https://github.com/ESMValGroup/ESMValCore/pull/3110), [Pull request #3032](https://github.com/ESMValGroup/ESMValCore/pull/3032)) - Preprocessor and regridding support (see [Pull request #3059](https://github.com/ESMValGroup/ESMValCore/pull/3059), [Pull request #3048](https://github.com/ESMValGroup/ESMValCore/pull/3048), [Pull request #3096](https://github.com/ESMValGroup/ESMValCore/pull/3096), [Pull request #3098](https://github.com/ESMValGroup/ESMValCore/pull/3098)) This release includes ### Backwards incompatible changes - Add `use_standard_grid` option for CORDEX datasets ([Pull request #3059](https://github.com/ESMValGroup/ESMValCore/pull/3059)) by [@bouweandela](https://github.com/bouweandela) - Remove configuration option `extra_facets_dir` and do not read extra facets from `~/.esmvaltool/extra_facets/` ([Pull request #3137](https://github.com/ESMValGroup/ESMValCore/pull/3137)) by [@schlunma](https://github.com/schlunma) ### Bug fixes - Make sure that histogram weights are Dask arrays when cube has lazy data ([Pull request #3038](https://github.com/ESMValGroup/ESMValCore/pull/3038)) by [@schlunma](https://github.com/schlunma) - Avoid a crash when running `esmvaltool config show` with Python 3.14 ([Pull request #3064](https://github.com/ESMValGroup/ESMValCore/pull/3064)) by [@bouweandela](https://github.com/bouweandela) - Remove accidentally added `esmvaltool config console` command ([Pull request #3066](https://github.com/ESMValGroup/ESMValCore/pull/3066)) by [@bouweandela](https://github.com/bouweandela) - Fix drs path for dir for cmip5 at CEDA JASMIN ([Pull request #3076](https://github.com/ESMValGroup/ESMValCore/pull/3076)) by [@valeriupredoi](https://github.com/valeriupredoi) - Ensure that input arrays to `scipy.interpolate.interp1d` functions are C-contiguous ([Pull request #3151](https://github.com/ESMValGroup/ESMValCore/pull/3151)) by [@schlunma](https://github.com/schlunma) - Avoid crash on missing session when caching fails with intake-esgf ([Pull request #3159](https://github.com/ESMValGroup/ESMValCore/pull/3159)) by [@bouweandela](https://github.com/bouweandela) ### CMOR standard - Update CMIP7 CMOR tables to DR-1.2.2.3-v1.0.6 ([Pull request #3070](https://github.com/ESMValGroup/ESMValCore/pull/3070)) by [@bouweandela](https://github.com/bouweandela) - Add CORDEX-CMIP6 support ([Pull request #3112](https://github.com/ESMValGroup/ESMValCore/pull/3112)) by [@bouweandela](https://github.com/bouweandela) ### Configuration - Add an HPC configuration file for SMHI (NSC) machine (Freja) ([Pull request #3028](https://github.com/ESMValGroup/ESMValCore/pull/3028)) by [@jlenh](https://github.com/jlenh) - Update path to CEDA-JASMIN esmeval gws (group workspace) ([Pull request #3041](https://github.com/ESMValGroup/ESMValCore/pull/3041)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add the option to only show the configuration for a single project ([Pull request #3020](https://github.com/ESMValGroup/ESMValCore/pull/3020)) by [@bouweandela](https://github.com/bouweandela) - Add support for specifying the output directory from the command line ([Pull request #3101](https://github.com/ESMValGroup/ESMValCore/pull/3101)) by [@lewisjared](https://github.com/lewisjared) ### Documentation - Fix a typo in ESMValCore ESGF data source documentation ([Pull request #3016](https://github.com/ESMValGroup/ESMValCore/pull/3016)) by [@bouweandela](https://github.com/bouweandela) - Fix the PDF build of the documentation ([Pull request #3068](https://github.com/ESMValGroup/ESMValCore/pull/3068)) by [@bouweandela](https://github.com/bouweandela) - Add `mip` to CMIP7 facets documentation page ([Pull request #3019](https://github.com/ESMValGroup/ESMValCore/pull/3019)) by [@bouweandela](https://github.com/bouweandela) - Update links to CMIP guidance ([Pull request #3075](https://github.com/ESMValGroup/ESMValCore/pull/3075)) by [@bouweandela](https://github.com/bouweandela) - Update release instructions related to Docker ([Pull request #3078](https://github.com/ESMValGroup/ESMValCore/pull/3078)) by [@bouweandela](https://github.com/bouweandela) - Update dataset GitHub issue template ([Pull request #3107](https://github.com/ESMValGroup/ESMValCore/pull/3107)) by [@bouweandela](https://github.com/bouweandela) - Update CMIP7 facets documentation page ([Pull request #3111](https://github.com/ESMValGroup/ESMValCore/pull/3111)) by [@bouweandela](https://github.com/bouweandela) ### Fixes for datasets - Expand ICON-XPP fix for new model version ([Pull request #3014](https://github.com/ESMValGroup/ESMValCore/pull/3014)) by [@schlunma](https://github.com/schlunma) - Modify ERA5 fixes to support data produced by the new GRIB to netcdf converter version of ECMWF ([Pull request #3030](https://github.com/ESMValGroup/ESMValCore/pull/3030)) by [@jlenh](https://github.com/jlenh) - Fix endianness of CORDEX CCLM4-8-17 data ([Pull request #3033](https://github.com/ESMValGroup/ESMValCore/pull/3033)) by [@bouweandela](https://github.com/bouweandela) - Avoid crash in fixes for time-independent CORDEX data ([Pull request #3032](https://github.com/ESMValGroup/ESMValCore/pull/3032)) by [@bouweandela](https://github.com/bouweandela) - Add support for freshly downloaded ERA5 hourly data ([Pull request #3087](https://github.com/ESMValGroup/ESMValCore/pull/3087)) by [@schlunma](https://github.com/schlunma) - Add support for variables `phcint` and `amoc` to ICON-XPP CMORizer ([Pull request #3025](https://github.com/ESMValGroup/ESMValCore/pull/3025)) by [@schlunma](https://github.com/schlunma) - Add support for reading ORAS5 data ([Pull request #2422](https://github.com/ESMValGroup/ESMValCore/pull/2422)) by [@jmalles](https://github.com/jmalles) - Add mesh fix for CMIP6 ICON-ESM-LR conservative regridding ([Pull request #3090](https://github.com/ESMValGroup/ESMValCore/pull/3090)) by [@alioacar](https://github.com/alioacar) - Fix CORDEX ALADIN53 tas coordinates ([Pull request #3110](https://github.com/ESMValGroup/ESMValCore/pull/3110)) by [@bouweandela](https://github.com/bouweandela) - CMIP6 CanESM CMOR fixes for variables with model level coordinates ([Pull request #3165](https://github.com/ESMValGroup/ESMValCore/pull/3165)) by [@k-a-webb](https://github.com/k-a-webb) ### Installation - Use pixi to manage development conda environments ([Pull request #3044](https://github.com/ESMValGroup/ESMValCore/pull/3044)) by [@bouweandela](https://github.com/bouweandela) - Run all tests on lockfile update ([Pull request #3119](https://github.com/ESMValGroup/ESMValCore/pull/3119)) by [@bouweandela](https://github.com/bouweandela) - Fix progress logging with Dask v2026 ([Pull request #3154](https://github.com/ESMValGroup/ESMValCore/pull/3154)) by [@bouweandela](https://github.com/bouweandela) - Open log files in append mode ([Pull request #3163](https://github.com/ESMValGroup/ESMValCore/pull/3163)) by [@bouweandela](https://github.com/bouweandela) ### Preprocessor - Drop range related attributes on load ([Pull request #3072](https://github.com/ESMValGroup/ESMValCore/pull/3072)) by [@bouweandela](https://github.com/bouweandela) - Drop `valid_range` correction from AIRS-2-0 fix ([Pull request #3153](https://github.com/ESMValGroup/ESMValCore/pull/3153)) by [@schlunma](https://github.com/schlunma) - Copy additional coordinates when regridding ([Pull request #3048](https://github.com/ESMValGroup/ESMValCore/pull/3048)) by [@bouweandela](https://github.com/bouweandela) - Add CORDEX domain target grids for regridding ([Pull request #3096](https://github.com/ESMValGroup/ESMValCore/pull/3096)) by [@ghossh](https://github.com/ghossh) ### Automatic testing - Avoid depending on the order of the configured project in a unit test ([Pull request #3023](https://github.com/ESMValGroup/ESMValCore/pull/3023)) by [@valeriupredoi](https://github.com/valeriupredoi) - Ensure CMOR tables are loaded and match configuration when running tests ([Pull request #3086](https://github.com/ESMValGroup/ESMValCore/pull/3086)) by [@bouweandela](https://github.com/bouweandela) - Update CircleCI Codecov orb version to 6.0.0 ([Pull request #3104](https://github.com/ESMValGroup/ESMValCore/pull/3104)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix pre-commit mypy error “TypeAliasType” not callable ([Pull request #3118](https://github.com/ESMValGroup/ESMValCore/pull/3118)) by [@flicj191](https://github.com/flicj191) & ([Pull request #3129](https://github.com/ESMValGroup/ESMValCore/pull/3129)) by [@bouweandela](https://github.com/bouweandela) ### Improvements - Improve code quality and test coverage ([Pull request #3029](https://github.com/ESMValGroup/ESMValCore/pull/3029)) by [@schlunma](https://github.com/schlunma) - Gracefully handle the case of missing facets required by `esmvalcore.io.local.LocalDataSource` ([Pull request #3021](https://github.com/ESMValGroup/ESMValCore/pull/3021)) by [@bouweandela](https://github.com/bouweandela) - Mention the branding suffix in the error message when no entry can be found in CMOR tables ([Pull request #3083](https://github.com/ESMValGroup/ESMValCore/pull/3083)) by [@bouweandela](https://github.com/bouweandela) - Nicer looking command line output ([Pull request #3079](https://github.com/ESMValGroup/ESMValCore/pull/3079)) by [@bouweandela](https://github.com/bouweandela) - Support automatic addition of supplementary variables for CORDEX ([Pull request #3098](https://github.com/ESMValGroup/ESMValCore/pull/3098)) by [@bouweandela](https://github.com/bouweandela) - Inform user about data downloads ([Pull request #3130](https://github.com/ESMValGroup/ESMValCore/pull/3130)) by [@schlunma](https://github.com/schlunma) ## v2.14.0 ### Highlights - Support for intake-esgf and new `esmvaltool config` commands have been added (See [Data sources](quickstart/configure.html.md#config-data-sources)) - A new way to configure the CMOR tables and the possibility to use data that is not described in a CMOR table have been implemented (see [CMOR table configuration](quickstart/configure.html.md#cmor-table-configuration)) - CMIP7 data is now supported (see [Pull request #2935](https://github.com/ESMValGroup/ESMValCore/pull/2935)) This release includes ### Backwards incompatible changes - Remove support for `~/.esmvaltool/config-user.yml` and `~/.esmvaltool/dask.yml` ([Pull request #2878](https://github.com/ESMValGroup/ESMValCore/pull/2878)) by [@bouweandela](https://github.com/bouweandela) - Remove deprecated ESMPy regridding schemes ([Pull request #2879](https://github.com/ESMValGroup/ESMValCore/pull/2879)) by [@bouweandela](https://github.com/bouweandela) - Add preliminary CMIP7 support ([Pull request #2935](https://github.com/ESMValGroup/ESMValCore/pull/2935)) by [@bouweandela](https://github.com/bouweandela) - Unify handling of group coordinates in temporal statistics preprocessors ([Pull request #2787](https://github.com/ESMValGroup/ESMValCore/pull/2787)) by [@schlunma](https://github.com/schlunma) - Make CMOR tables configurable through new configuration system and deprecate config-developer.yml ([Pull request #2946](https://github.com/ESMValGroup/ESMValCore/pull/2946)) by [@bouweandela](https://github.com/bouweandela) ### Deprecations - Move the `esmvalcore.local` and `esmvalcore.esgf` modules into the `esmvalcore.io` module ([Pull request #2911](https://github.com/ESMValGroup/ESMValCore/pull/2911)) by [@bouweandela](https://github.com/bouweandela) - Move preprocessor output filename template to new configuration ([Pull request #2923](https://github.com/ESMValGroup/ESMValCore/pull/2923)) by [@bouweandela](https://github.com/bouweandela) - Improve ana4MIPs support in the new configuration ([Pull request #2932](https://github.com/ESMValGroup/ESMValCore/pull/2932)) by [@bouweandela](https://github.com/bouweandela) - Make CMOR tables configurable through new configuration system and deprecate config-developer.yml ([Pull request #2946](https://github.com/ESMValGroup/ESMValCore/pull/2946)) by [@bouweandela](https://github.com/bouweandela) ### Bug fixes - Fix a bug where the mask_fillvalues preprocessor only recorded provenance for the first result ([Pull request #2904](https://github.com/ESMValGroup/ESMValCore/pull/2904)) by [@bouweandela](https://github.com/bouweandela) - Fixed behavior of `--search_data=quick` and `--search_data=complete` ([Pull request #2937](https://github.com/ESMValGroup/ESMValCore/pull/2937)) by [@schlunma](https://github.com/schlunma) - Two fixes in preprocessor documentation ([Pull request #2966](https://github.com/ESMValGroup/ESMValCore/pull/2966)) by [@LisaBock](https://github.com/LisaBock) - Do not overwrite provenance information written by `ProvenanceLogger` ([Pull request #2987](https://github.com/ESMValGroup/ESMValCore/pull/2987)) by [@schlunma](https://github.com/schlunma) - Avoid modifying supplementary variables ([Pull request #2992](https://github.com/ESMValGroup/ESMValCore/pull/2992)) by [@bouweandela](https://github.com/bouweandela) - Avoid issue with deepcopying/pickling IntakeESGFDatasets ([Pull request #2990](https://github.com/ESMValGroup/ESMValCore/pull/2990)) by [@bouweandela](https://github.com/bouweandela) - Repair fix_file ([Pull request #2993](https://github.com/ESMValGroup/ESMValCore/pull/2993)) by [@bouweandela](https://github.com/bouweandela) ### CMOR standard - Add CCI lst v3 cmor tables ([Pull request #2458](https://github.com/ESMValGroup/ESMValCore/pull/2458)) by [@morobking](https://github.com/morobking) - Add the option to ignore the `branding_suffix` for derived CMIP7 and custom variables ([Pull request #2982](https://github.com/ESMValGroup/ESMValCore/pull/2982)) by [@bouweandela](https://github.com/bouweandela) - Avoid hardcoded time coordinate names in CMOR check ([Pull request #2981](https://github.com/ESMValGroup/ESMValCore/pull/2981)) by [@bouweandela](https://github.com/bouweandela) ### Configuration - Avoid logging the configuration multiple times ([Pull request #2869](https://github.com/ESMValGroup/ESMValCore/pull/2869)) by [@bouweandela](https://github.com/bouweandela) - Add an interface for adding new data sources and add support for intake-esgf as a first example ([Pull request #2765](https://github.com/ESMValGroup/ESMValCore/pull/2765)) by [@bouweandela](https://github.com/bouweandela) - Add BSC data configuration ([Pull request #2907](https://github.com/ESMValGroup/ESMValCore/pull/2907)) by [@sloosvel](https://github.com/sloosvel) - Moved `dkrz-era5` data source in DKRZ default config to separate file ([Pull request #2996](https://github.com/ESMValGroup/ESMValCore/pull/2996)) by [@schlunma](https://github.com/schlunma) ### Computational performance improvements - Make sure all input arrays passed to `dask.array.apply_gufunc` are Dask arrays ([Pull request #2969](https://github.com/ESMValGroup/ESMValCore/pull/2969)) by [@schlunma](https://github.com/schlunma) ### Documentation - Fix indentation and add links in changelog for v2.13 ([Pull request #2862](https://github.com/ESMValGroup/ESMValCore/pull/2862)) by [@bouweandela](https://github.com/bouweandela) - Documentation layout to highlight 2 packages ([Pull request #2732](https://github.com/ESMValGroup/ESMValCore/pull/2732)) by [@flicj191](https://github.com/flicj191) - Document that we follow SPEC 0 ([Pull request #2885](https://github.com/ESMValGroup/ESMValCore/pull/2885)) by [@bouweandela](https://github.com/bouweandela) - Add missing preprocessor name rolling_window_statistics to example. ([Pull request #2890](https://github.com/ESMValGroup/ESMValCore/pull/2890)) by [@katjaweigel](https://github.com/katjaweigel) - Fix stable documentation build ([Pull request #2893](https://github.com/ESMValGroup/ESMValCore/pull/2893)) by [@bouweandela](https://github.com/bouweandela) - Move ESMValCore release documentation to How-to guide section ([Pull request #2889](https://github.com/ESMValGroup/ESMValCore/pull/2889)) by [@jlenh](https://github.com/jlenh) - Add ENES-RI Zenodo community ([Pull request #2894](https://github.com/ESMValGroup/ESMValCore/pull/2894)) by [@bouweandela](https://github.com/bouweandela) - Fix the name and version number on readthedocs ([Pull request #2926](https://github.com/ESMValGroup/ESMValCore/pull/2926)) by [@bouweandela](https://github.com/bouweandela) - Add updated release timeline doodle ([Pull request #2899](https://github.com/ESMValGroup/ESMValCore/pull/2899)) by [@jlenh](https://github.com/jlenh) - Fix tiny doc bug in (legacy) ESGF configuration ([Pull request #2945](https://github.com/ESMValGroup/ESMValCore/pull/2945)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin sphinx<9 ([Pull request #2949](https://github.com/ESMValGroup/ESMValCore/pull/2949)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add Matomo page view tracking for the documentation ([Pull request #2955](https://github.com/ESMValGroup/ESMValCore/pull/2955)) by [@bouweandela](https://github.com/bouweandela) - Add notes to `climate_statistics` docs about shifting of time coordinate ([Pull request #2967](https://github.com/ESMValGroup/ESMValCore/pull/2967)) by [@schlunma](https://github.com/schlunma) - Update documentation on derived variables with new CMOR tables configuration ([Pull request #2979](https://github.com/ESMValGroup/ESMValCore/pull/2979)) by [@schlunma](https://github.com/schlunma) - Update contact info ([Pull request #2985](https://github.com/ESMValGroup/ESMValCore/pull/2985)) by [@axel-lauer](https://github.com/axel-lauer) - Update contact email in documentation ([Pull request #3008](https://github.com/ESMValGroup/ESMValCore/pull/3008)) by [@axel-lauer](https://github.com/axel-lauer) ### Fixes for datasets - Fix ERA5 native6 fix to handle single monthly-averaged NetCDF files. ([Pull request #2512](https://github.com/ESMValGroup/ESMValCore/pull/2512)) by [@rbeucher](https://github.com/rbeucher) - Expand ICON extra facets ([Pull request #2965](https://github.com/ESMValGroup/ESMValCore/pull/2965)) by [@schlunma](https://github.com/schlunma) - Copy fixes for obs4MIPs dataset SSMI RSSv07r00 to RSS-v7 ([Pull request #2968](https://github.com/ESMValGroup/ESMValCore/pull/2968)) by [@bouweandela](https://github.com/bouweandela) - Expand ICON-XPP fix for new model version ([Pull request #3014](https://github.com/ESMValGroup/ESMValCore/pull/3014)) by [@schlunma](https://github.com/schlunma) ### Installation - Drop support for Python 3.11 ([Pull request #2905](https://github.com/ESMValGroup/ESMValCore/pull/2905)) by [@bouweandela](https://github.com/bouweandela) - Pin iris <3.14.1 to align with iris-esmf-regrid ([Pull request #2925](https://github.com/ESMValGroup/ESMValCore/pull/2925)) by [@bouweandela](https://github.com/bouweandela) - Update pyproject.toml with new license and license-files settings ([Pull request #2924](https://github.com/ESMValGroup/ESMValCore/pull/2924)) by [@bouweandela](https://github.com/bouweandela) - Support for Python 3.14.0 ([Pull request #2850](https://github.com/ESMValGroup/ESMValCore/pull/2850)) by [@valeriupredoi](https://github.com/valeriupredoi) - Make the tool compatible with numpy 2.4 ([Pull request #2941](https://github.com/ESMValGroup/ESMValCore/pull/2941)) by [@bouweandela](https://github.com/bouweandela) - Update PyPI publish action ([Pull request #2939](https://github.com/ESMValGroup/ESMValCore/pull/2939)) by [@bouweandela](https://github.com/bouweandela) - Unpin pandas ([Pull request #2961](https://github.com/ESMValGroup/ESMValCore/pull/2961)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Preprocessor - Extend preprocessor anomalies ([Pull request #2871](https://github.com/ESMValGroup/ESMValCore/pull/2871)) by [@axel-lauer](https://github.com/axel-lauer) ### Automatic testing - Ignore user configuration in configuration reload test ([Pull request #2903](https://github.com/ESMValGroup/ESMValCore/pull/2903)) by [@bouweandela](https://github.com/bouweandela) - Remove an outdated file that is part of the ESMValTool tests ([Pull request #2944](https://github.com/ESMValGroup/ESMValCore/pull/2944)) by [@bouweandela](https://github.com/bouweandela) - Improve the security of GitHub Action workflows ([Pull request #2952](https://github.com/ESMValGroup/ESMValCore/pull/2952)) by [@bouweandela](https://github.com/bouweandela) - Run s3 zarr tests only if s3 object store is online ([Pull request #2958](https://github.com/ESMValGroup/ESMValCore/pull/2958)) by [@valeriupredoi](https://github.com/valeriupredoi) - Correction to #2958 - add bucket to S3 url ([Pull request #2959](https://github.com/ESMValGroup/ESMValCore/pull/2959)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix CMOR test fails in OSX due to case discrepancy tables -> Tables ([Pull request #2975](https://github.com/ESMValGroup/ESMValCore/pull/2975)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Variable Derivation - Add derived variable lapserate ([Pull request #2872](https://github.com/ESMValGroup/ESMValCore/pull/2872)) by [@axel-lauer](https://github.com/axel-lauer) - Fix derivation for sea ice extent (siextent) ([Pull request #2648](https://github.com/ESMValGroup/ESMValCore/pull/2648)) by [@axel-lauer](https://github.com/axel-lauer) ### Improvements - Update logger to include esmvaltool logs ([Pull request #2886](https://github.com/ESMValGroup/ESMValCore/pull/2886)) by [@bouweandela](https://github.com/bouweandela) - Enable annotations ruff rule ([Pull request #2880](https://github.com/ESMValGroup/ESMValCore/pull/2880)) by [@bouweandela](https://github.com/bouweandela) - Fix issue with wildcards in timerange after introducing data sources ([Pull request #2900](https://github.com/ESMValGroup/ESMValCore/pull/2900)) by [@bouweandela](https://github.com/bouweandela) - Automatically sort `__all__` in Python modules ([Pull request #2913](https://github.com/ESMValGroup/ESMValCore/pull/2913)) by [@bouweandela](https://github.com/bouweandela) - Sort `__all__` in Python modules ([Pull request #2914](https://github.com/ESMValGroup/ESMValCore/pull/2914)) by [@bouweandela](https://github.com/bouweandela) - Better debug message when local data is not found within the requested timerange ([Pull request #2927](https://github.com/ESMValGroup/ESMValCore/pull/2927)) by [@bouweandela](https://github.com/bouweandela) - Add float to FacetValue and improve preprocessor filename template error message ([Pull request #2931](https://github.com/ESMValGroup/ESMValCore/pull/2931)) by [@bouweandela](https://github.com/bouweandela) - Removed unused code from `Dataset` class ([Pull request #2963](https://github.com/ESMValGroup/ESMValCore/pull/2963)) by [@schlunma](https://github.com/schlunma) - Smarter limit on preprocessor filename length ([Pull request #3011](https://github.com/ESMValGroup/ESMValCore/pull/3011)) by [@bouweandela](https://github.com/bouweandela) ## v2.13.0 ### Highlights - Extra facets are now more easily configurable with the new configuration system, as described in [Extra Facets](quickstart/configure.html.md#config-extra-facets). - Preprocessor [`esmvalcore.preprocessor.extract_surface_from_atm()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_surface_from_atm) allows to extract the surface values of a 3D atmospheric variable based on surface pressure. - Force the new configuration system through the environment variable ESMVALTOOL_CONFIG_DIR as described [here](quickstart/configure.html.md#config-yaml-files). - A set of ruff rules has been enabled in place of Prospector and related tools and Codacy, modernizing our development setup. Additional information is available in the section on [Code quality](contributing.html.md#code-quality) in our contribution guidelines. - An on-the-fly CMORizer for ICON-XPP is now available. - [Stale issues and pull requests](https://github.com/esMValGroup/esMValCore/issues?q=label%3AStale) will now be closed automatically following the stale action workflow. This release includes ### Backwards incompatible changes - Remove deprecated horizontal regridding schemes unstructured_nearest and linear_extrapolate ([Pull request #2743](https://github.com/ESMValGroup/ESMValCore/pull/2743)) by [@schlunma](https://github.com/schlunma) - unstructured_nearest: Please use the scheme nearest instead. This is an : exact replacement for data on unstructured grids. ESMValCore is now able to determine the most suitable regridding scheme based on the input data. - linear_extrapolate: Please use a generic scheme with reference: : iris.analysis:Linear and extrapolation_mode: extrapolate instead. This is an exact replacement, e.g.: ```yaml preprocessors: regrid_preprocessor: regrid: target_grid: 2.5x2.5 scheme: reference: iris.analysis:Linear extrapolation_mode: extrapolate ``` - Align custom lwp table with CMIP6 version ([Pull request #2791](https://github.com/ESMValGroup/ESMValCore/pull/2791)) by [@schlunma](https://github.com/schlunma) There is no way to restore the old behavior. Downstream code needs to be adapted. ### Deprecations - Make extra facets configurable using the new configuration system ([Pull request #2747](https://github.com/ESMValGroup/ESMValCore/pull/2747)) by [@schlunma](https://github.com/schlunma) See [Project-specific configuration](quickstart/configure.html.md#config-projects). Please use the configuration option extra_facets instead. For example, instead of: ```yaml # content of ~/.esmvaltool/extra_facets/cmip6-mapping.yml CMIP6-MODEL: Amon: tas: extra_facet_key: extra_facet_value ``` use the syntax described under [Extra Facets](quickstart/configure.html.md#config-extra-facets): ```yaml projects: CMIP6: extra_facets: CMIP6-MODEL: Amon: tas: extra_facet_key: extra_facet_value ``` ### CMOR standard - Fix units and values of tos variable in GISS-E2-1-G-CC esm-piControl ([Pull request #2689](https://github.com/ESMValGroup/ESMValCore/pull/2689)) by [@dhohn](https://github.com/dhohn) - Add entry for n2os variable ([Pull request #2724](https://github.com/ESMValGroup/ESMValCore/pull/2724)) by [@jlenh](https://github.com/jlenh) - Added custom CMOR tables for dpn2o and n2oflux ([Pull request #2751](https://github.com/ESMValGroup/ESMValCore/pull/2751)) by [@schlunma](https://github.com/schlunma) - Add UKESM1-0-LL exception to fix HadGEM3-GC31-LL ‘parent_time_units’ error ([Pull request #2782](https://github.com/ESMValGroup/ESMValCore/pull/2782)) by [@prosku](https://github.com/prosku) - Add custom CMOR table for above-ground biomass (agb) ([Pull request #2783](https://github.com/ESMValGroup/ESMValCore/pull/2783)) by [@axel-lauer](https://github.com/axel-lauer) ### Community - Added stale action ([Pull request #2759](https://github.com/ESMValGroup/ESMValCore/pull/2759)) by [@schlunma](https://github.com/schlunma) ### Documentation - Unpin upper sphinx but pin nbsphinx >=0.9.7 ([Pull request #2685](https://github.com/ESMValGroup/ESMValCore/pull/2685)) by [@valeriupredoi](https://github.com/valeriupredoi) - Explicitly export LC_ALL for sphinx builds ([Pull request #2708](https://github.com/ESMValGroup/ESMValCore/pull/2708)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove reference to psy-plot recipe in ESMValTool ([Pull request #2741](https://github.com/ESMValGroup/ESMValCore/pull/2741)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix Codecov badge in README ([Pull request #2754](https://github.com/ESMValGroup/ESMValCore/pull/2754)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove mention of Julia in documentation ahead of discontinued support in v2.13 ([Pull request #2819](https://github.com/ESMValGroup/ESMValCore/pull/2819)) by [@jlenh](https://github.com/jlenh) - Remove codacy badge ([Pull request #2822](https://github.com/ESMValGroup/ESMValCore/pull/2822)) by [@bouweandela](https://github.com/bouweandela) - Replace nbsphinx with myst-nb and remove pin on ipython ([Pull request #2838](https://github.com/ESMValGroup/ESMValCore/pull/2838)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Fixes for datasets - Fix for obs4MIPs dataset C3S-GTO-ECV-9-0 (toz) ([Pull request #2722](https://github.com/ESMValGroup/ESMValCore/pull/2722)) by [@axel-lauer](https://github.com/axel-lauer) - Add on-the-fly CMORizer for ICON-XPP ([Pull request #2659](https://github.com/ESMValGroup/ESMValCore/pull/2659)) by [@schlunma](https://github.com/schlunma) - Allow fix_file to return dataset objects ([Pull request #2579](https://github.com/ESMValGroup/ESMValCore/pull/2579)) by [@schlunma](https://github.com/schlunma) - Fix ocean region coordinate in msftmz dataset (CESM2) ([Pull request #1607](https://github.com/ESMValGroup/ESMValCore/pull/1607)) by [@dhohn](https://github.com/dhohn) ### Installation - Pin ipython<9.0 ([Pull request #2681](https://github.com/ESMValGroup/ESMValCore/pull/2681)) by [@valeriupredoi](https://github.com/valeriupredoi) - Set a temporary pin dask <2025.4.0 while Iris are looking for a solution ([Pull request #2720](https://github.com/ESMValGroup/ESMValCore/pull/2720)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove support for python 3.10 ([Pull request #2730](https://github.com/ESMValGroup/ESMValCore/pull/2730)) by [@valeriupredoi](https://github.com/valeriupredoi) - Unpin dask, pin iris, in light of solved #2716 (iris saver not working well with latest Dask API) ([Pull request #2726](https://github.com/ESMValGroup/ESMValCore/pull/2726)) by [@valeriupredoi](https://github.com/valeriupredoi) - Harmonize environment.yml with pyproject.yml: identical dependencies ([Pull request #2756](https://github.com/ESMValGroup/ESMValCore/pull/2756)) by [@valeriupredoi](https://github.com/valeriupredoi) - Install esmvaltool-sample-data(==0.0.4) from conda forge not PyPI ([Pull request #2795](https://github.com/ESMValGroup/ESMValCore/pull/2795)) by [@valeriupredoi](https://github.com/valeriupredoi) - Use `pip install --no-deps` to avoid overwriting packages from the conda environment ([Pull request #2829](https://github.com/ESMValGroup/ESMValCore/pull/2829)) by [@bouweandela](https://github.com/bouweandela) ### Iris - Fix test for upstream iris version ([Pull request #2752](https://github.com/ESMValGroup/ESMValCore/pull/2752)) by [@schlunma](https://github.com/schlunma) ### Preprocessor - Add unit conversion for air mass ([Pull request #2698](https://github.com/ESMValGroup/ESMValCore/pull/2698)) by [@LisaBock](https://github.com/LisaBock) - Add evaporation flux to special unit conversion ([Pull request #2691](https://github.com/ESMValGroup/ESMValCore/pull/2691)) by [@lukruh](https://github.com/lukruh) - Add preprocessor to extract surface values from 3D atmospheric variables ([Pull request #2641](https://github.com/ESMValGroup/ESMValCore/pull/2641)) by [@jlenh](https://github.com/jlenh) - Allow adding supplementary variables from a different project and different dataset ([Pull request #2755](https://github.com/ESMValGroup/ESMValCore/pull/2755)) by [@valeriupredoi](https://github.com/valeriupredoi) and ([Pull request #2771](https://github.com/ESMValGroup/ESMValCore/pull/2771)) by [@bouweandela](https://github.com/bouweandela) - Move concatenate preprocessor function to its own module ([Pull request #2766](https://github.com/ESMValGroup/ESMValCore/pull/2766)) by [@bouweandela](https://github.com/bouweandela) - Add preprocessor align_metadata ([Pull request #2789](https://github.com/ESMValGroup/ESMValCore/pull/2789)) by [@schlunma](https://github.com/schlunma) - Fix for add_ancillary_variable (#2820) ([Pull request #2825](https://github.com/ESMValGroup/ESMValCore/pull/2825)) by [@jlenh](https://github.com/jlenh) - Set correct calendar when reading file dates ([Pull request #2826](https://github.com/ESMValGroup/ESMValCore/pull/2826)) by [@bouweandela](https://github.com/bouweandela) ### Automatic testing - Add support for codacy-ruff ([Pull request #2683](https://github.com/ESMValGroup/ESMValCore/pull/2683)) by [@valeriupredoi](https://github.com/valeriupredoi) - Use –no-deps in upstream tests to avoid installation problems caused by upper pins ([Pull request #2710](https://github.com/ESMValGroup/ESMValCore/pull/2710)) by [@bouweandela](https://github.com/bouweandela) - Temporary pin dask !=2025.4.0 ([Pull request #2717](https://github.com/ESMValGroup/ESMValCore/pull/2717)) by [@valeriupredoi](https://github.com/valeriupredoi) - Explicitly call mamba env create with arguments in Circle CI configuration ([Pull request #2721](https://github.com/ESMValGroup/ESMValCore/pull/2721)) by [@valeriupredoi](https://github.com/valeriupredoi) - Avoid reading configuration from disk for every single test ([Pull request #2767](https://github.com/ESMValGroup/ESMValCore/pull/2767)) by [@bouweandela](https://github.com/bouweandela) - Do not assume that no warnings are raised during test_dataset_to_iris ([Pull request #2773](https://github.com/ESMValGroup/ESMValCore/pull/2773)) by [@schlunma](https://github.com/schlunma) - Update CircleCI orbs ([Pull request #2784](https://github.com/ESMValGroup/ESMValCore/pull/2784)) by [@bouweandela](https://github.com/bouweandela) - Zarr support (backend, in esmvalcore.preprocessor._io.py) ([Pull request #2785](https://github.com/ESMValGroup/ESMValCore/pull/2785)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update mamba and python (remove pins and allow for Python 3.13) conda-lock file creation Github Action - via ESMValTool_sample_data being a conda-forge package ([Pull request #2792](https://github.com/ESMValGroup/ESMValCore/pull/2792)) by [@valeriupredoi](https://github.com/valeriupredoi) - Automatically open pull requests to update GitHub Actions ([Pull request #2799](https://github.com/ESMValGroup/ESMValCore/pull/2799)) by [@bouweandela](https://github.com/bouweandela) - Use finer current version for pypa PyPI upload github action ([Pull request #2804](https://github.com/ESMValGroup/ESMValCore/pull/2804)) by [@valeriupredoi](https://github.com/valeriupredoi) - Change zarr exception test to GroupNotFoundError ([Pull request #2823](https://github.com/ESMValGroup/ESMValCore/pull/2823)) by [@jlenh](https://github.com/jlenh) - Remove failing test test_load_zarr_local_not_zarr_file due to zarr version changes ([Pull request #2824](https://github.com/ESMValGroup/ESMValCore/pull/2824)) by [@jlenh](https://github.com/jlenh) - Repair gha tests after pip install with no deps and add working reference to ipynb Jupyter notebooks in docs ([Pull request #2844](https://github.com/ESMValGroup/ESMValCore/pull/2844)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Variable Derivation - Add derivation of moisture flux into atmosphere ([Pull request #2697](https://github.com/ESMValGroup/ESMValCore/pull/2697)) by [@LisaBock](https://github.com/LisaBock) ### Improvements - Add ocean variable to the ACCESS Live CMORiser ([Pull request #2601](https://github.com/ESMValGroup/ESMValCore/pull/2601)) by [@rbeucher](https://github.com/rbeucher) - Remove accidentally added file ([Pull request #2701](https://github.com/ESMValGroup/ESMValCore/pull/2701)) by [@schlunma](https://github.com/schlunma) - Use netCDF4.Dataset to read start and end date from files ([Pull request #2728](https://github.com/ESMValGroup/ESMValCore/pull/2728)) by [@schlunma](https://github.com/schlunma) - Allow reading facets from filenames ([Pull request #2725](https://github.com/ESMValGroup/ESMValCore/pull/2725)) by [@schlunma](https://github.com/schlunma) - Always context managers when handling netCDF4.Dataset objects ([Pull request #2734](https://github.com/ESMValGroup/ESMValCore/pull/2734)) by [@schlunma](https://github.com/schlunma) - Enable more ruff rules ([Pull request #2715](https://github.com/ESMValGroup/ESMValCore/pull/2715)) by [@bouweandela](https://github.com/bouweandela) - Show external warnings only in debug log ([Pull request #2733](https://github.com/ESMValGroup/ESMValCore/pull/2733)) by [@schlunma](https://github.com/schlunma) - Fix sign of ERA5 rlut and rlutcs ([Pull request #2748](https://github.com/ESMValGroup/ESMValCore/pull/2748)) by [@schlunma](https://github.com/schlunma) - Using ESMVALTOOL_CONFIG_DIR will force the usage of new configuration system and ignore old configuration ([Pull request #2736](https://github.com/ESMValGroup/ESMValCore/pull/2736)) by [@schlunma](https://github.com/schlunma) - Improve error message if cubes do not overlap in time in multi_model_statistics with span=overlap ([Pull request #2762](https://github.com/ESMValGroup/ESMValCore/pull/2762)) by [@schlunma](https://github.com/schlunma) - In modules relevant for variable derivation: cleaned code, doc, and added type hints ([Pull request #2772](https://github.com/ESMValGroup/ESMValCore/pull/2772)) by [@schlunma](https://github.com/schlunma) - Added CFG.context/Session.context ([Pull request #2778](https://github.com/ESMValGroup/ESMValCore/pull/2778)) by [@schlunma](https://github.com/schlunma) - Use [https://esgf-node.ornl.gov/esgf-1-5-bridge](https://esgf-node.ornl.gov/esgf-1-5-bridge) for ESGF searches by default ([Pull request #2781](https://github.com/ESMValGroup/ESMValCore/pull/2781)) by [@bouweandela](https://github.com/bouweandela) - Pass chunks={} to Xarray dataset loader for Zarr stores ([Pull request #2794](https://github.com/ESMValGroup/ESMValCore/pull/2794)) by [@valeriupredoi](https://github.com/valeriupredoi) - Enable ruff rule that checks unused arguments ([Pull request #2809](https://github.com/ESMValGroup/ESMValCore/pull/2809)) by [@bouweandela](https://github.com/bouweandela) - Enable ruff rule to prevent print statements ([Pull request #2810](https://github.com/ESMValGroup/ESMValCore/pull/2810)) by [@bouweandela](https://github.com/bouweandela) - Enable ruff rule that enforces using items() to iterate over dict key/value pairs ([Pull request #2811](https://github.com/ESMValGroup/ESMValCore/pull/2811)) by [@bouweandela](https://github.com/bouweandela) - Remove use of prospector and related tools and Codacy ([Pull request #2818](https://github.com/ESMValGroup/ESMValCore/pull/2818)) by [@bouweandela](https://github.com/bouweandela) - Commenting out ORNL ESGF node in esgf_pyclient config ([Pull request #2806](https://github.com/ESMValGroup/ESMValCore/pull/2806)) by [@jlenh](https://github.com/jlenh) - Enable ruff rule that prevents overwriting loop variables ([Pull request #2813](https://github.com/ESMValGroup/ESMValCore/pull/2813)) by [@bouweandela](https://github.com/bouweandela) - Enable ruff rule that checks for commented out code ([Pull request #2808](https://github.com/ESMValGroup/ESMValCore/pull/2808)) by [@bouweandela](https://github.com/bouweandela) ## v2.12.0 ### Highlights - Preprocessor [`esmvalcore.preprocessor.extract_time()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_time) now allows to extract time blocks in each year by making parameters `start_year` and `end_year` optional. - A new way of [configuring the tool](quickstart/configure.html.md#config) has been developed. - Performance improvements: > - An iris-esmf-regrid scheme has been added to > preprocessor [`esmvalcore.preprocessor.regrid()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.regrid), which improves > the regridding of 2D grids and adds the capability to regrid UGRID meshes > out of the box. > - Data is now saved from one preprocessing task at the time when using the > distributed scheduler, in order to avoid running out of memory. > - A better default `num_workers` has been set when using more than one > `max_parallel_tasks` with an unconfigured threaded scheduler. - An on-the-fly cmorizer for ACCESS native data is now available. This release includes ### Backwards incompatible changes - Make derivation of total column ozone (toz) more flexible and add derivation of stratospheric and tropospheric column ozone ([Pull request #2509](https://github.com/ESMValGroup/ESMValCore/pull/2509)) by [@schlunma](https://github.com/schlunma) - The units of toz have been changed from `DU` to `m` to be consistent with the CMIP6 CMOR table. To restore the old behaviour, preprocessor [`esmvalcore.preprocessor.convert_units()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.convert_units) can be used to set the units back to `DU`. - Remove deprecated CMOR fix/check code ([Pull request #2552](https://github.com/ESMValGroup/ESMValCore/pull/2552)) by [@schlunma](https://github.com/schlunma) - CMOR fixes and checks have been clearly separated in v2.10.0, and the old code has now been removed. Use functions `esmvalcore.preprocessors.fix_metadata()`, `esmvalcore.preprocessors.fix_data()`, or [`esmvalcore.dataset.Dataset.load()`](api/esmvalcore.dataset.html.md#esmvalcore.dataset.Dataset.load) to fix data. Use functions [`esmvalcore.preprocessor.cmor_check_metadata()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.cmor_check_metadata), [`esmvalcore.preprocessor.cmor_check_data()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.cmor_check_data), or `esmvalcore.preprocessor.cmor_check()` to check data. - Remove deprecated statistical operators ([Pull request #2553](https://github.com/ESMValGroup/ESMValCore/pull/2553)) by [@schlunma](https://github.com/schlunma) - Old statistical operators that have been deprecated in v2.10.0 have now been removed. Please refer to [Statistical preprocessors](recipe/preprocessor.html.md#stat-preprocs) for a detailed description on how to use the operators. - Save all files in a task at the same time to avoid recomputing intermediate results ([Pull request #2522](https://github.com/ESMValGroup/ESMValCore/pull/2522)) by [@bouweandela](https://github.com/bouweandela) - The signature of the preprocessor function [`save()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.save) has changed. The function now accepts a `compute` argument that can be [`True`](https://docs.python.org/3/builtins/constants.html#True), in which case the return value will be [`None`](https://docs.python.org/3/builtins/constants.html#None) or [`False`](https://docs.python.org/3/builtins/constants.html#False), in which case the return value will be a [`Delayed`](https://docs.dask.org/en/stable/delayed-api.html#dask.delayed.Delayed) object that can be used to compute and save the data of the cube. ### Deprecations - Merge configuration object from multiple files (instead of one single file) ([Pull request #2448](https://github.com/ESMValGroup/ESMValCore/pull/2448)) by [@schlunma](https://github.com/schlunma) - The single configuration file `config-user.yml` has been deprecated in favour of configuration directories. By default, the directory `~/.config/esmvaltool` will be considered. To switch to the new format run: > ```bash > mkdir -p ~/.config/esmvaltool && mv ~/.esmvaltool/config-user.yml ~/.config/esmvaltool > ``` You can also specify the location of the configuration directory with the `--config_dir` flag. Please refer to [Configuration](quickstart/configure.html.md#config) for a detailed description on how to configure the tool. - Make Dask configurable in our configuration ([Pull request #2616](https://github.com/ESMValGroup/ESMValCore/pull/2616)) by [@schlunma](https://github.com/schlunma) - The old Dask configuration file that needed to be located at `~/.esmvaltool/dask.yml` is now deprecated. Please refer to [Dask configuration](quickstart/configure.html.md#config-dask) for a detailed description on how to configure Dask. ### Bug fixes - Concatenate by experiment before concatenating all input files ([Pull request #2343](https://github.com/ESMValGroup/ESMValCore/pull/2343)) by [@dhohn](https://github.com/dhohn) - Update CFG with configuration options given via command line ([Pull request #2595](https://github.com/ESMValGroup/ESMValCore/pull/2595)) by [@schlunma](https://github.com/schlunma) - Avoid a crash when there is a timeout when shutting down the Dask cluster ([Pull request #2580](https://github.com/ESMValGroup/ESMValCore/pull/2580)) by [@bouweandela](https://github.com/bouweandela) - More reliable datasets to recipe conversion ([Pull request #2472](https://github.com/ESMValGroup/ESMValCore/pull/2472)) by [@bouweandela](https://github.com/bouweandela) - Avoid mutating the input cubes when building the combined cube in preprocessor function `multi_model_statistics` ([Pull request #2564](https://github.com/ESMValGroup/ESMValCore/pull/2564)) by [@bouweandela](https://github.com/bouweandela) - Do not change function argument names when decorator preserve_float_dtype is used ([Pull request #2645](https://github.com/ESMValGroup/ESMValCore/pull/2645)) by [@schlunma](https://github.com/schlunma) - Always write target coordinates to source cube after regridding ([Pull request #2673](https://github.com/ESMValGroup/ESMValCore/pull/2673)) by [@schlunma](https://github.com/schlunma) ### CMOR standard - New custom variable for tos uncertainty ([Pull request #2470](https://github.com/ESMValGroup/ESMValCore/pull/2470)) by [@LisaBock](https://github.com/LisaBock) - Add prc fix for native6 ERA5 CMORization ([Pull request #2550](https://github.com/ESMValGroup/ESMValCore/pull/2550)) by [@malininae](https://github.com/malininae) - Added more variables to EMAC extra facets ([Pull request #2617](https://github.com/ESMValGroup/ESMValCore/pull/2617)) by [@schlunma](https://github.com/schlunma) - Adding custom tables for ETCCDI indices ([Pull request #2442](https://github.com/ESMValGroup/ESMValCore/pull/2442)) by [@malininae](https://github.com/malininae) ### Configuration - Add public Config.update_from_dirs() method ([Pull request #2538](https://github.com/ESMValGroup/ESMValCore/pull/2538)) by [@schlunma](https://github.com/schlunma) - Do not use Path objects as configuration dictionary keys to avoid errors in dask.config.merge ([Pull request #2578](https://github.com/ESMValGroup/ESMValCore/pull/2578)) by [@schlunma](https://github.com/schlunma) - Revise the user-config.yml to support updated MO user config requirements ([Pull request #2658](https://github.com/ESMValGroup/ESMValCore/pull/2658)) by [@ehogan](https://github.com/ehogan) ### Computational performance improvements - Add an iris-esmf-regrid based regridding scheme ([Pull request #2457](https://github.com/ESMValGroup/ESMValCore/pull/2457)) by [@bouweandela](https://github.com/bouweandela) - Miscellaneous lazy preprocessor improvements ([Pull request #2520](https://github.com/ESMValGroup/ESMValCore/pull/2520)) by [@bouweandela](https://github.com/bouweandela) - Only save data from one preprocessing task at a time with the Distributed scheduler ([Pull request #2610](https://github.com/ESMValGroup/ESMValCore/pull/2610)) by [@bouweandela](https://github.com/bouweandela) - Use better defaults when using `max_parallel_tasks` with an unconfigured threaded scheduler ([Pull request #2626](https://github.com/ESMValGroup/ESMValCore/pull/2626)) by [@bouweandela](https://github.com/bouweandela) - Fix OSX compatibility ([Pull request #2636](https://github.com/ESMValGroup/ESMValCore/pull/2636)) by [@bouweandela](https://github.com/bouweandela) - Make sure that supplementary variables and weights have same chunks as parent cube ([Pull request #2637](https://github.com/ESMValGroup/ESMValCore/pull/2637)) by [@schlunma](https://github.com/schlunma) ### Documentation - Ignore autosummary warning in documentation build ([Pull request #2480](https://github.com/ESMValGroup/ESMValCore/pull/2480)) by [@bouweandela](https://github.com/bouweandela) - Fix documentation build and broken link ([Pull request #2519](https://github.com/ESMValGroup/ESMValCore/pull/2519)) by [@bouweandela](https://github.com/bouweandela) - Dark mode compatible logo ([Pull request #2532](https://github.com/ESMValGroup/ESMValCore/pull/2532)) by [@lukruh](https://github.com/lukruh) - Add a pre-commit badge to README ([Pull request #2534](https://github.com/ESMValGroup/ESMValCore/pull/2534)) by [@valeriupredoi](https://github.com/valeriupredoi) - Retire Mambaforge ([Pull request #2556](https://github.com/ESMValGroup/ESMValCore/pull/2556)) by [@valeriupredoi](https://github.com/valeriupredoi) - Readthedocs configuration: temporary revert to miniconda before miniforge3 becomes available (remove Mambaforge) ([Pull request #2562](https://github.com/ESMValGroup/ESMValCore/pull/2562)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove Docker build badge in README ([Pull request #2565](https://github.com/ESMValGroup/ESMValCore/pull/2565)) by [@valeriupredoi](https://github.com/valeriupredoi) - Optimize documentation about Earth mover distance in distance_metric preprocessor ([Pull request #2423](https://github.com/ESMValGroup/ESMValCore/pull/2423)) by [@schlunma](https://github.com/schlunma) - Update the esmvaltool command welcome message ([Pull request #2635](https://github.com/ESMValGroup/ESMValCore/pull/2635)) by [@bouweandela](https://github.com/bouweandela) - Pin sphinx to < 8.2 ([Pull request #2671](https://github.com/ESMValGroup/ESMValCore/pull/2671)) by [@bouweandela](https://github.com/bouweandela) ### Fixes for datasets - Extra facets added for EMAC to map o3, tro3 and aps ([Pull request #2501](https://github.com/ESMValGroup/ESMValCore/pull/2501)) by [@FranziskaWinterstein](https://github.com/FranziskaWinterstein) - Fix for CMIP6 AWI-ESM-1-1-LR parent time units ([Pull request #2507](https://github.com/ESMValGroup/ESMValCore/pull/2507)) by [@brittaGrusdt](https://github.com/brittaGrusdt) - Use our own unit conversion function in our fixes ([Pull request #2560](https://github.com/ESMValGroup/ESMValCore/pull/2560)) by [@schlunma](https://github.com/schlunma) - Missing 2m height coordinate and monotonicity for tasmin in CESM2 and CESM2-WACCM ([Pull request #2574](https://github.com/ESMValGroup/ESMValCore/pull/2574)) by [@Karen-A-Garcia](https://github.com/Karen-A-Garcia) - Monotonicity fixes for Fgoals ([Pull request #2603](https://github.com/ESMValGroup/ESMValCore/pull/2603)) by [@Karen-A-Garcia](https://github.com/Karen-A-Garcia) - Expand Amon fix of FIO-ESM-2-0 (CMIP6) ([Pull request #2619](https://github.com/ESMValGroup/ESMValCore/pull/2619)) by [@schlunma](https://github.com/schlunma) - Added fixes for some 3D atmospheric variables of E3SM-1-1 (CMIP6) ([Pull request #2620](https://github.com/ESMValGroup/ESMValCore/pull/2620)) by [@schlunma](https://github.com/schlunma) - Correct incorrect time bounds in EMAC data ([Pull request #2621](https://github.com/ESMValGroup/ESMValCore/pull/2621)) by [@FranziskaWinterstein](https://github.com/FranziskaWinterstein) - Do not copy ICON horizontal grid every time it is used ([Pull request #2633](https://github.com/ESMValGroup/ESMValCore/pull/2633)) by [@schlunma](https://github.com/schlunma) - Fix oh for model: EC-Earth3-AerChem mip: AERMonZ ([Pull request #2634](https://github.com/ESMValGroup/ESMValCore/pull/2634)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update CMIP5 EC-EARTH pr fix ([Pull request #2666](https://github.com/ESMValGroup/ESMValCore/pull/2666)) by [@bouweandela](https://github.com/bouweandela) - Add a fix for differing index coord long names in NorESM2-MM and EC-Earth3-Veg-LR ([Pull request #2667](https://github.com/ESMValGroup/ESMValCore/pull/2667)) by [@bouweandela](https://github.com/bouweandela) ### Installation - Free esmpy of \` >=8.6.0\` pin and pin iris-grib >=0.20.0 ([Pull request #2542](https://github.com/ESMValGroup/ESMValCore/pull/2542)) by [@valeriupredoi](https://github.com/valeriupredoi) - Use pyproject.toml instead of setup.py/setup.cfg ([Pull request #2540](https://github.com/ESMValGroup/ESMValCore/pull/2540)) by [@bouweandela](https://github.com/bouweandela) - Use miniforge3 for our docker builds instead of mambaforge ([Pull request #2558](https://github.com/ESMValGroup/ESMValCore/pull/2558)) by [@valeriupredoi](https://github.com/valeriupredoi) - Support Python 3.13 ([Pull request #2566](https://github.com/ESMValGroup/ESMValCore/pull/2566)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin dask ([Pull request #2654](https://github.com/ESMValGroup/ESMValCore/pull/2654)) by [@sloosvel](https://github.com/sloosvel) ### Iris - Set iris.FUTURE flags in one place ([Pull request #2622](https://github.com/ESMValGroup/ESMValCore/pull/2622)) by [@bouweandela](https://github.com/bouweandela) - Silence Iris warnings in `area_statistics` preprocessor function ([Pull request #2625](https://github.com/ESMValGroup/ESMValCore/pull/2625)) by [@bouweandela](https://github.com/bouweandela) ### Preprocessor - Merge input cubes only once when computing lazy multimodel statistics ([Pull request #2518](https://github.com/ESMValGroup/ESMValCore/pull/2518)) by [@bouweandela](https://github.com/bouweandela) - Make start_year, end_year in extract_time optional to obtain time blocks in each year ([Pull request #2490](https://github.com/ESMValGroup/ESMValCore/pull/2490)) by [@malininae](https://github.com/malininae) - Adding hurs (relative humidity) derivation script ([Pull request #2397](https://github.com/ESMValGroup/ESMValCore/pull/2397)) by [@malininae](https://github.com/malininae) - Added cumulative sum preprocessor ([Pull request #2642](https://github.com/ESMValGroup/ESMValCore/pull/2642)) by [@schlunma](https://github.com/schlunma) - Unified ignoring of iris.warnings.IrisVagueMetadataWarning in preprocessors ([Pull request #2646](https://github.com/ESMValGroup/ESMValCore/pull/2646)) by [@schlunma](https://github.com/schlunma) - Raise error if weights are used with unweighted aggregator ([Pull request #2640](https://github.com/ESMValGroup/ESMValCore/pull/2640)) by [@schlunma](https://github.com/schlunma) - Allow using multi model statistics preprocessor on datasets without timerange ([Pull request #2644](https://github.com/ESMValGroup/ESMValCore/pull/2644)) by [@schlunma](https://github.com/schlunma) - Allow using output from multi_model_statistics or ensemble_statistics as reference for bias or distance_metric ([Pull request #2652](https://github.com/ESMValGroup/ESMValCore/pull/2652)) by [@schlunma](https://github.com/schlunma) - Add option to ignore horizontal coordinates if there are multiple when regridding ([Pull request #2672](https://github.com/ESMValGroup/ESMValCore/pull/2672)) by [@bouweandela](https://github.com/bouweandela) ### Observational and re-analysis dataset support - Add support for native ERA5 data in GRIB format ([Pull request #2178](https://github.com/ESMValGroup/ESMValCore/pull/2178)) by [@schlunma](https://github.com/schlunma) ### Automatic testing - Run a nightly test with the development version of dependencies ([Pull request #2478](https://github.com/ESMValGroup/ESMValCore/pull/2478)) by [@bouweandela](https://github.com/bouweandela) - Use ruff formatter and pre-commit ([Pull request #2524](https://github.com/ESMValGroup/ESMValCore/pull/2524)) by [@bouweandela](https://github.com/bouweandela) - Fix tests if deprecated ~/.esmvaltool/config-user.yml file is available ([Pull request #2543](https://github.com/ESMValGroup/ESMValCore/pull/2543)) by [@schlunma](https://github.com/schlunma) - Disable upstream tests on commits ([Pull request #2548](https://github.com/ESMValGroup/ESMValCore/pull/2548)) by [@bouweandela](https://github.com/bouweandela) - Disable collecting test coverage by default ([Pull request #2456](https://github.com/ESMValGroup/ESMValCore/pull/2456)) by [@bouweandela](https://github.com/bouweandela) - Enable ruff flake8-bugbear rule ([Pull request #2536](https://github.com/ESMValGroup/ESMValCore/pull/2536)) by [@bouweandela](https://github.com/bouweandela) - Pin mamba in conda lock creation github action ([Pull request #2561](https://github.com/ESMValGroup/ESMValCore/pull/2561)) by [@valeriupredoi](https://github.com/valeriupredoi) - [Numpy2] Support for numpy==2.0.0 (and pin iris >=3.11) ([Pull request #2395](https://github.com/ESMValGroup/ESMValCore/pull/2395)) by [@valeriupredoi](https://github.com/valeriupredoi) - Switch back to Python 3.12 for conda lock file creation due to mamba<2 pin ([Pull request #2606](https://github.com/ESMValGroup/ESMValCore/pull/2606)) by [@valeriupredoi](https://github.com/valeriupredoi) - Always ignore user’s configuration when running Dask tests ([Pull request #2624](https://github.com/ESMValGroup/ESMValCore/pull/2624)) by [@schlunma](https://github.com/schlunma) - Restrict runs of cron Github Actions on forks ([Pull request #2649](https://github.com/ESMValGroup/ESMValCore/pull/2649)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix test that loads realistic GRIB file ([Pull request #2665](https://github.com/ESMValGroup/ESMValCore/pull/2665)) by [@schlunma](https://github.com/schlunma) - Fix failing test with Dask 2025.2: ours issue not theirs ([Pull request #2663](https://github.com/ESMValGroup/ESMValCore/pull/2663)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Variable Derivation - Introduction of the variable prodlnox for EMAC ([Pull request #2499](https://github.com/ESMValGroup/ESMValCore/pull/2499)) by [@FranziskaWinterstein](https://github.com/FranziskaWinterstein) ### Improvements - On-the-fly cmoriser for ACCESS native data ([Pull request #2430](https://github.com/ESMValGroup/ESMValCore/pull/2430)) by [@rhaegar325](https://github.com/rhaegar325) - Fix CFF file ([Pull request #2476](https://github.com/ESMValGroup/ESMValCore/pull/2476)) by [@rbeucher](https://github.com/rbeucher) - Write settings.yml parameters in original order ([Pull request #2352](https://github.com/ESMValGroup/ESMValCore/pull/2352)) by [@enekomartinmartinez](https://github.com/enekomartinmartinez) - Fix access-mapping.yml extra_facets title ([Pull request #2485](https://github.com/ESMValGroup/ESMValCore/pull/2485)) by [@rhaegar325](https://github.com/rhaegar325) - Remove ability to log on to ESGF ([Pull request #2508](https://github.com/ESMValGroup/ESMValCore/pull/2508)) by [@bouweandela](https://github.com/bouweandela) - Disable automatic fixes by pre-commit.ci ([Pull request #2527](https://github.com/ESMValGroup/ESMValCore/pull/2527)) by [@bouweandela](https://github.com/bouweandela) - Ignore reformatting when viewing git blame ([Pull request #2539](https://github.com/ESMValGroup/ESMValCore/pull/2539)) by [@bouweandela](https://github.com/bouweandela) - Enable ruff pydocstyle linter rule ([Pull request #2547](https://github.com/ESMValGroup/ESMValCore/pull/2547)) by [@bouweandela](https://github.com/bouweandela) - Allows relative paths for diagnostic scripts. ([Pull request #2329](https://github.com/ESMValGroup/ESMValCore/pull/2329)) by [@rbeucher](https://github.com/rbeucher) - Fix 2593 Change log INFO to DEBUG ([Pull request #2600](https://github.com/ESMValGroup/ESMValCore/pull/2600)) by [@rbeucher](https://github.com/rbeucher) ## v2.11.1 ### Highlights This is a bugfix release which enables lazy computations in more preprocessors and allows installing the latests version of various dependencies, including Iris ([v3.11.0](https://github.com/SciTools/iris/releases/tag/v3.11.0)). This release includes ### Computational performance improvements - Optimize functions `mask_landsea()`, `mask_landseaice()` and `calculate_volume()` for lazy input ([Pull request #2515](https://github.com/ESMValGroup/ESMValCore/pull/2515)) by [@schlunma](https://github.com/schlunma) ### Installation - Remove support for Python 3.9 ([Pull request #2447](https://github.com/ESMValGroup/ESMValCore/pull/2447)) by [@valeriupredoi](https://github.com/valeriupredoi) - Switch to new iris >= 3.10.0 API ([Pull request #2500](https://github.com/ESMValGroup/ESMValCore/pull/2500)) by [@schlunma](https://github.com/schlunma) - Pin dask to avoid 2024.8.0 - problems with masked fill/missing values ([Pull request #2504](https://github.com/ESMValGroup/ESMValCore/pull/2504)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix rounding of Pandas datetimes in ICON CMORizer to allow installing latest Pandas version ([Pull request #2529](https://github.com/ESMValGroup/ESMValCore/pull/2529)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Automatic testing - Fix type hint for new mypy version ([Pull request #2497](https://github.com/ESMValGroup/ESMValCore/pull/2497)) by [@schlunma](https://github.com/schlunma) - Reformat datetime strings be in line with new `isodate==0.7.0` and actual ISO8601 and pin `isodate>=0.7.0` ([Pull request #2546](https://github.com/ESMValGroup/ESMValCore/pull/2546)) by [@valeriupredoi](https://github.com/valeriupredoi) ## v2.11.0 ### Highlights - Performance improvements have been made to many preprocessors: - Preprocessors [`esmvalcore.preprocessor.mask_landsea()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_landsea), [`esmvalcore.preprocessor.mask_landseaice()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_landseaice), [`esmvalcore.preprocessor.mask_glaciated()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_glaciated), [`esmvalcore.preprocessor.extract_levels()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_levels) are now lazy - Several new preprocessors have been added: - [`esmvalcore.preprocessor.local_solar_time()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.local_solar_time) - `esmvalcore.preprocessor.distance_metrics()` - [`esmvalcore.preprocessor.histogram()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.histogram) - NEW TREND: First time release manager shout-outs! - This is the first ESMValTool release managed by the Met Office! We want to shout this out - and for all future first time release managers to shout-out - to celebrate the growing, thriving ESMValTool community. This release includes ### Backwards incompatible changes - Allow contiguous representation of extracted regions ([Pull request #2230](https://github.com/ESMValGroup/ESMValCore/pull/2230)) by [@rebeccaherman1](https://github.com/rebeccaherman1) - The preprocessor function [`esmvalcore.preprocessor.extract_region()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_region) no longer automatically maps the extracted [`iris.cube.Cube`](https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.cube.html#iris.cube.Cube) to the 0-360 degrees longitude domain. If you need this behaviour, use `cube.intersection(longitude=(0., 360.))` in your Python code after extracting the region. There is no possibility to restore the previous behaviour from a recipe. - Use `iris.FUTURE.save_split_attrs = True` to remove iris warning ([Pull request #2398](https://github.com/ESMValGroup/ESMValCore/pull/2398)) by [@schlunma](https://github.com/schlunma) - Since [v3.8.0](https://scitools-iris.readthedocs.io/en/stable/whatsnew/3.8.html#v3-8-29-feb-2024), Iris explicitly distinguishes between local and global netCDF attributes. ESMValCore adopted this behavior with v2.11.0. With this change, attributes are written as local attributes by default, unless they already existed as global attributes or belong to a special list of global attributes (in which case attributes are written as global attributes). See [`iris.cube.CubeAttrsDict`](https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.cube.html#iris.cube.CubeAttrsDict) for details. ### Deprecations - Refactor regridding ([Pull request #2231](https://github.com/ESMValGroup/ESMValCore/pull/2231)) by [@schlunma](https://github.com/schlunma) - This PR deprecated two regridding schemes, which will be removed with ESMValCore v2.13.0: - `unstructured_nearest`: Please use the scheme `nearest` instead. This is an exact replacement for data on unstructured grids. ESMValCore is now able to determine the most suitable regridding scheme based on the input data. - `linear_extrapolate`: Please use a generic scheme with `reference: iris.analysis:Linear` and `extrapolation_mode: extrapolate` instead. - Allow deprecated regridding scheme `linear_extrapolate` in recipe checks ([Pull request #2324](https://github.com/ESMValGroup/ESMValCore/pull/2324)) by [@schlunma](https://github.com/schlunma) - Allow deprecated regridding scheme `unstructured_nearest` in recipe checks ([Pull request #2336](https://github.com/ESMValGroup/ESMValCore/pull/2336)) by [@schlunma](https://github.com/schlunma) ### Bug fixes - Do not overwrite facets from recipe with CMOR table facets for derived variables ([Pull request #2255](https://github.com/ESMValGroup/ESMValCore/pull/2255)) by [@bouweandela](https://github.com/bouweandela) - Fix error message in variable definition check ([Pull request #2313](https://github.com/ESMValGroup/ESMValCore/pull/2313)) by [@enekomartinmartinez](https://github.com/enekomartinmartinez) - Unify dtype handling of preprocessors ([Pull request #2393](https://github.com/ESMValGroup/ESMValCore/pull/2393)) by [@schlunma](https://github.com/schlunma) - Fix bug in `_rechunk_aux_factory_dependencies` ([Pull request #2428](https://github.com/ESMValGroup/ESMValCore/pull/2428)) by [@ehogan](https://github.com/ehogan) - Avoid loading entire files into memory when downloading from ESGF ([Pull request #2434](https://github.com/ESMValGroup/ESMValCore/pull/2434)) by [@bouweandela](https://github.com/bouweandela) - Preserve cube attribute global vs local when concatenating ([Pull request #2449](https://github.com/ESMValGroup/ESMValCore/pull/2449)) by [@bouweandela](https://github.com/bouweandela) ### CMOR standard - Also read default custom CMOR tables if custom location is specified ([Pull request #2279](https://github.com/ESMValGroup/ESMValCore/pull/2279)) by [@schlunma](https://github.com/schlunma) - Add custom CMOR table for total cloud water (tcw) ([Pull request #2277](https://github.com/ESMValGroup/ESMValCore/pull/2277)) by [@axel-lauer](https://github.com/axel-lauer) - Add height for sfcWindmax in MPI HighRes models ([Pull request #2292](https://github.com/ESMValGroup/ESMValCore/pull/2292)) by [@malininae](https://github.com/malininae) - Fixed `positive` attribute in custom rtnt table ([Pull request #2367](https://github.com/ESMValGroup/ESMValCore/pull/2367)) by [@schlunma](https://github.com/schlunma) - Fix `positive` attributes in custom CMOR variables ([Pull request #2380](https://github.com/ESMValGroup/ESMValCore/pull/2380)) by [@schlunma](https://github.com/schlunma) - Log CMOR check and generic fix output to separate file ([Pull request #2361](https://github.com/ESMValGroup/ESMValCore/pull/2361)) by [@schlunma](https://github.com/schlunma) ### Computational performance improvements - More lazy fixes and preprocessing functions ([Pull request #2325](https://github.com/ESMValGroup/ESMValCore/pull/2325)) by [@bouweandela](https://github.com/bouweandela) - Made preprocessors [`esmvalcore.preprocessor.mask_landsea()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_landsea), [`esmvalcore.preprocessor.mask_landseaice()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_landseaice) and [`esmvalcore.preprocessor.mask_glaciated()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_glaciated) lazy ([Pull request #2268](https://github.com/ESMValGroup/ESMValCore/pull/2268)) by [@joergbenke](https://github.com/joergbenke) - More lazy [`esmvalcore.preprocessor.extract_levels()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_levels) preprocessor function ([Pull request #2120](https://github.com/ESMValGroup/ESMValCore/pull/2120)) by [@bouweandela](https://github.com/bouweandela) - Use lazy weights for [`esmvalcore.preprocessor.climate_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.climate_statistics) and [`esmvalcore.preprocessor.axis_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.axis_statistics) ([Pull request #2346](https://github.com/ESMValGroup/ESMValCore/pull/2346)) by [@schlunma](https://github.com/schlunma) - Fixed potential memory leak in [`esmvalcore.preprocessor.local_solar_time()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.local_solar_time) ([Pull request #2356](https://github.com/ESMValGroup/ESMValCore/pull/2356)) by [@schlunma](https://github.com/schlunma) - Cache regridding weights if possible ([Pull request #2344](https://github.com/ESMValGroup/ESMValCore/pull/2344)) by [@schlunma](https://github.com/schlunma) - Implement lazy area weights ([Pull request #2354](https://github.com/ESMValGroup/ESMValCore/pull/2354)) by [@schlunma](https://github.com/schlunma) - Avoid large chunks in [`esmvalcore.preprocessor.climate_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.climate_statistics) preprocessor function with period=’full’ ([Pull request #2404](https://github.com/ESMValGroup/ESMValCore/pull/2404)) by [@bouweandela](https://github.com/bouweandela) - Load data only once for ESMPy regridders ([Pull request #2418](https://github.com/ESMValGroup/ESMValCore/pull/2418)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Use short links in changelog ([Pull request #2287](https://github.com/ESMValGroup/ESMValCore/pull/2287)) by [@bouweandela](https://github.com/bouweandela) - National Computing Infrastructure (NCI), Site specific configuration ([Pull request #2281](https://github.com/ESMValGroup/ESMValCore/pull/2281)) by [@rbeucher](https://github.com/rbeucher) - Update [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) doc with latest changes (new operators, etc.) ([Pull request #2321](https://github.com/ESMValGroup/ESMValCore/pull/2321)) by [@schlunma](https://github.com/schlunma) - Fix Codacy badge ([Pull request #2382](https://github.com/ESMValGroup/ESMValCore/pull/2382)) by [@bouweandela](https://github.com/bouweandela) - Change ‘mean’ to ‘percentile’ in doc strings of preprocessor statistics ([Pull request #2327](https://github.com/ESMValGroup/ESMValCore/pull/2327)) by [@lukruh](https://github.com/lukruh) - Fixed typo in doc about weighted statistics ([Pull request #2387](https://github.com/ESMValGroup/ESMValCore/pull/2387)) by [@schlunma](https://github.com/schlunma) ### Fixes for datasets - Fixing missing height 2m coordinates in GFDL-CM4 and KIOST-ESM ([Pull request #2294](https://github.com/ESMValGroup/ESMValCore/pull/2294)) by [@Karen-A-Garcia](https://github.com/Karen-A-Garcia) - Added fix for wrong units of `clt` for CIESM and FIO-ESM-2-0 ([Pull request #2330](https://github.com/ESMValGroup/ESMValCore/pull/2330)) by [@schlunma](https://github.com/schlunma) - Cmip6 gfdl_cm4: fix tas height fix to work for concatenated scenarios ([Pull request #2332](https://github.com/ESMValGroup/ESMValCore/pull/2332)) by [@mwjury](https://github.com/mwjury) - Cordex GERICS REMO2015 lon differences above 10e-4 ([Pull request #2334](https://github.com/ESMValGroup/ESMValCore/pull/2334)) by [@mwjury](https://github.com/mwjury) - Download ICON grid without locking ([Pull request #2359](https://github.com/ESMValGroup/ESMValCore/pull/2359)) by [@bouweandela](https://github.com/bouweandela) - Added ICON fixes for hfls and hfss ([Pull request #2360](https://github.com/ESMValGroup/ESMValCore/pull/2360)) by [@diegokam](https://github.com/diegokam) - Added ICON fix for rtnt ([Pull request #2366](https://github.com/ESMValGroup/ESMValCore/pull/2366)) by [@diegokam](https://github.com/diegokam) - Expanded ICON extra facets ([Pull request #2379](https://github.com/ESMValGroup/ESMValCore/pull/2379)) by [@schlunma](https://github.com/schlunma) - Add 10m height coordinate to SfcWind GFDL-CM4 instead of 2m height ([Pull request #2385](https://github.com/ESMValGroup/ESMValCore/pull/2385)) by [@Karen-A-Garcia](https://github.com/Karen-A-Garcia) - Cordex wrf381p: fix tas,tasmax,tasmin height ([Pull request #2333](https://github.com/ESMValGroup/ESMValCore/pull/2333)) by [@mwjury](https://github.com/mwjury) - Several minor fixes needed for marine BGC data. ([Pull request #2110](https://github.com/ESMValGroup/ESMValCore/pull/2110)) by [@ledm](https://github.com/ledm) ### Installation - Pin pandas yet again avoid new `2.2.1` as well ([Pull request #2353](https://github.com/ESMValGroup/ESMValCore/pull/2353)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update Iris pin to avoid using versions with memory issues ([Pull request #2408](https://github.com/ESMValGroup/ESMValCore/pull/2408)) by [@chrisbillowsMO](https://github.com/chrisbillowsMO) - Pin esmpy <8.6.0 ([Pull request #2402](https://github.com/ESMValGroup/ESMValCore/pull/2402)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin numpy<2.0.0 to avoid pulling 2.0.0rcX ([Pull request #2415](https://github.com/ESMValGroup/ESMValCore/pull/2415)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add support for Python=3.12 ([Pull request #2228](https://github.com/ESMValGroup/ESMValCore/pull/2228)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Preprocessor - New preprocessor: [`esmvalcore.preprocessor.local_solar_time()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.local_solar_time) ([Pull request #2258](https://github.com/ESMValGroup/ESMValCore/pull/2258)) by [@schlunma](https://github.com/schlunma) - Read derived variables from other MIP tables ([Pull request #2256](https://github.com/ESMValGroup/ESMValCore/pull/2256)) by [@bouweandela](https://github.com/bouweandela) - Added special unit conversion m -> DU for total column ozone (toz) ([Pull request #2270](https://github.com/ESMValGroup/ESMValCore/pull/2270)) by [@schlunma](https://github.com/schlunma) - Allow cubes as input for [`esmvalcore.preprocessor.bias()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.bias) preprocessor ([Pull request #2183](https://github.com/ESMValGroup/ESMValCore/pull/2183)) by [@schlunma](https://github.com/schlunma) - Add normalization with statistics to many statistics preprocessors ([Pull request #2189](https://github.com/ESMValGroup/ESMValCore/pull/2189)) by [@schlunma](https://github.com/schlunma) - Adding sfcWind derivation from uas and vas ([Pull request #2242](https://github.com/ESMValGroup/ESMValCore/pull/2242)) by [@malininae](https://github.com/malininae) - Update interval check in resample_hours ([Pull request #2362](https://github.com/ESMValGroup/ESMValCore/pull/2362)) by [@axel-lauer](https://github.com/axel-lauer) - Broadcast properly `cell_measures` when using [`esmvalcore.preprocessor.extract_shape()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_shape) with `decomposed: True` ([Pull request #2348](https://github.com/ESMValGroup/ESMValCore/pull/2348)) by [@sloosvel](https://github.com/sloosvel) - Compute volume from `cell_area` if available ([Pull request #2318](https://github.com/ESMValGroup/ESMValCore/pull/2318)) by [@enekomartinmartinez](https://github.com/enekomartinmartinez) - Do not expand wildcards for datasets of derived variables where not all input variables are available ([Pull request #2374](https://github.com/ESMValGroup/ESMValCore/pull/2374)) by [@schlunma](https://github.com/schlunma) - Modernize [`esmvalcore.preprocessor.regrid_time()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.regrid_time) and allow setting a common calendar for decadal, yearly, and monthly data ([Pull request #2311](https://github.com/ESMValGroup/ESMValCore/pull/2311)) by [@schlunma](https://github.com/schlunma) - Added unstructured linear regridding ([Pull request #2350](https://github.com/ESMValGroup/ESMValCore/pull/2350)) by [@schlunma](https://github.com/schlunma) - Add preprocessors `esmvalcore.preprocessor.distance_metrics()` and [`esmvalcore.preprocessor.histogram()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.histogram) ([Pull request #2299](https://github.com/ESMValGroup/ESMValCore/pull/2299)) by [@schlunma](https://github.com/schlunma) ### Automatic testing - Increase resources for testing installation from conda-forge ([Pull request #2297](https://github.com/ESMValGroup/ESMValCore/pull/2297)) by [@bouweandela](https://github.com/bouweandela) - Pin pandas to avoid broken `round` function ([Pull request #2305](https://github.com/ESMValGroup/ESMValCore/pull/2305)) by [@schlunma](https://github.com/schlunma) - Remove team reviewers from conda lock generation workflow in Github Actions ([Pull request #2307](https://github.com/ESMValGroup/ESMValCore/pull/2307)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove mocking from tests in `tests/unit/preprocessor/_regrid/test_extract_point.py` ([Pull request #2193](https://github.com/ESMValGroup/ESMValCore/pull/2193)) by [@ehogan](https://github.com/ehogan) - Pin `pytest-mypy` plugin to `>=0.10.3` comply with new `pytest==8` ([Pull request #2315](https://github.com/ESMValGroup/ESMValCore/pull/2315)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix regridding test for unstructured nearest regridding on OSX ([Pull request #2319](https://github.com/ESMValGroup/ESMValCore/pull/2319)) by [@schlunma](https://github.com/schlunma) - Fix flaky regrid test by clearing LRU cache after each test ([Pull request #2322](https://github.com/ESMValGroup/ESMValCore/pull/2322)) by [@valeriupredoi](https://github.com/valeriupredoi) - Xfail `tests/integration/cmor/_fixes/test_common.py::test_cl_hybrid_height_coord_fix_metadata` while Iris folk fix behaviour ([Pull request #2363](https://github.com/ESMValGroup/ESMValCore/pull/2363)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update codacy reporter orb to latest version ([Pull request #2388](https://github.com/ESMValGroup/ESMValCore/pull/2388)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add calls to `conda list` in Github Action test workflows to inspect environment ([Pull request #2391](https://github.com/ESMValGroup/ESMValCore/pull/2391)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin pandas yet again :panda_face: `test_icon` fails again with pandas=2.2.2 ([Pull request #2394](https://github.com/ESMValGroup/ESMValCore/pull/2394)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fixed units of cl test data (necessary since iris>=3.8.0) ([Pull request #2403](https://github.com/ESMValGroup/ESMValCore/pull/2403)) by [@schlunma](https://github.com/schlunma) ### Improvements - Show files of supplementary variables explicitly in log ([Pull request #2303](https://github.com/ESMValGroup/ESMValCore/pull/2303)) by [@schlunma](https://github.com/schlunma) - Remove warning about logging in to ESGF ([Pull request #2326](https://github.com/ESMValGroup/ESMValCore/pull/2326)) by [@bouweandela](https://github.com/bouweandela) - Do not read `~/.esmvaltool/config-user.yml` if `--config-file` is used ([Pull request #2309](https://github.com/ESMValGroup/ESMValCore/pull/2309)) by [@schlunma](https://github.com/schlunma) - Support loading ICON grid from ICON rootpath ([Pull request #2337](https://github.com/ESMValGroup/ESMValCore/pull/2337)) by [@schlunma](https://github.com/schlunma) - Handle warnings about invalid units for iris>=3.8 ([Pull request #2378](https://github.com/ESMValGroup/ESMValCore/pull/2378)) by [@schlunma](https://github.com/schlunma) - Added note on how to access `index.html` on remote server ([Pull request #2276](https://github.com/ESMValGroup/ESMValCore/pull/2276)) by [@schlunma](https://github.com/schlunma) - Remove custom fix for concatenation of aux factories now that bug in iris is solved ([Pull request #2392](https://github.com/ESMValGroup/ESMValCore/pull/2392)) by [@schlunma](https://github.com/schlunma) - Ignored iris warnings about global attributes ([Pull request #2400](https://github.com/ESMValGroup/ESMValCore/pull/2400)) by [@schlunma](https://github.com/schlunma) - Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir ([Pull request #2432](https://github.com/ESMValGroup/ESMValCore/pull/2432)) by [@alistairsellar](https://github.com/alistairsellar) ## v2.10.0 ### Highlights - All statistics preprocessors support the same operators and have a common [documentation](recipe/preprocessor.html.md#stat-preprocs). In addition, arbitrary keyword arguments for the statistical operation can be directly given to the preprocessor. - The output webpage generated by the tool now looks better and provides methods to select and filter the output. - Improved computational efficiency: - Automatic rechunking between preprocessor steps to keep the [graph size smaller](https://docs.dask.org/en/latest/best-practices.html#avoid-very-large-graphs) and the [chunk size optimal](https://blog.dask.org/2021/11/02/choosing-dask-chunk-sizes). - Reduce the size of the dask graph created by [`esmvalcore.preprocessor.anomalies()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.anomalies). - Preprocessors [`esmvalcore.preprocessor.mask_above_threshold()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_above_threshold), [`esmvalcore.preprocessor.mask_below_threshold()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_below_threshold), [`esmvalcore.preprocessor.mask_inside_range()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_inside_range), [`esmvalcore.preprocessor.mask_outside_range()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_outside_range) are now lazy. - Lazy coordinates bounds are no longer loaded into memory by the CMOR checks and fixes. This release includes ### Backwards incompatible changes - Remove the deprecated option `use_legacy_supplementaries` ([Pull request #2202](https://github.com/ESMValGroup/ESMValCore/pull/2202)) by [@bouweandela](https://github.com/bouweandela) - The recommended upgrade procedure is to remove `use_legacy_supplementaries` from config-user.yml (if it was there) and remove any mention of `fx_variables` from the recipe. If automatically defining the required supplementary variables does not work, define them in the variable or (`additional_`) `datasets` section as described in [Defining supplementary variables (ancillary variables and cell measures)](recipe/overview.html.md#supplementary-variables). - Use smarter (units-aware) weights ([Pull request #2139](https://github.com/ESMValGroup/ESMValCore/pull/2139)) by [@schlunma](https://github.com/schlunma) - Some preprocessors handle units better. For details, see the pull request. - Removed deprecated configuration option `offline` ([Pull request #2213](https://github.com/ESMValGroup/ESMValCore/pull/2213)) by [@schlunma](https://github.com/schlunma) - In [v2.8.0](#changelog-v2-8-0), we replaced the old `offline` configuration option. From this version on, it stops working. Please refer to [v2.8.0](#changelog-v2-8-0) for upgrade instructions. - Fix issue with CORDEX datasets requiring different dataset tags for downloads and fixes ([Pull request #2066](https://github.com/ESMValGroup/ESMValCore/pull/2066)) by [@ljoakim](https://github.com/ljoakim) - Due to the different facets for CORDEX datasets, there was an inconsistency in the fixing mechanism. This change requires changes to existing recipes that use CORDEX datasets. Please refer to the pull request for detailed update instructions. - For the following changes, no user change is necessary - Remove deprecated way of calling `read_cmor_tables()` ([Pull request #2201](https://github.com/ESMValGroup/ESMValCore/pull/2201)) by [@bouweandela](https://github.com/bouweandela) - Remove deprecated callback argument from preprocessor `load` function ([Pull request #2207](https://github.com/ESMValGroup/ESMValCore/pull/2207)) by [@bouweandela](https://github.com/bouweandela) - Remove deprecated preprocessor function cleanup ([Pull request #2215](https://github.com/ESMValGroup/ESMValCore/pull/2215)) by [@bouweandela](https://github.com/bouweandela) ### Deprecations - Clearly separate fixes and CMOR checks ([Pull request #2157](https://github.com/ESMValGroup/ESMValCore/pull/2157)) by [@schlunma](https://github.com/schlunma) - Added new operators for statistics preprocessor (e.g., `'percentile'`) and allowed passing additional arguments ([Pull request #2191](https://github.com/ESMValGroup/ESMValCore/pull/2191)) by [@schlunma](https://github.com/schlunma) - This harmonizes the operators for all statistics preprocessors. From this version, the new names can be used; the old arguments will stop working from version 2.12.0. Please refer to [Statistical preprocessors](recipe/preprocessor.html.md#stat-preprocs) for a detailed description. ### Bug fixes - Re-add correctly region-extracted cell measures and ancillary variables after [extract_region](recipe/preprocessor.html.md#extract-region) ([Pull request #2166](https://github.com/ESMValGroup/ESMValCore/pull/2166)) by [@valeriupredoi](https://github.com/valeriupredoi), [@schlunma](https://github.com/schlunma) - Fix sorting of datasets - Fix sorting of ensemble members in [`datasets_to_recipe()`](api/esmvalcore.dataset.html.md#esmvalcore.dataset.datasets_to_recipe) ([Pull request #2095](https://github.com/ESMValGroup/ESMValCore/pull/2095)) by [@bouweandela](https://github.com/bouweandela) - Fix a problem with sorting datasets that have a mix of facet types ([Pull request #2238](https://github.com/ESMValGroup/ESMValCore/pull/2238)) by [@bouweandela](https://github.com/bouweandela) - Avoid a crash if dataset has supplementary variables ([Pull request #2198](https://github.com/ESMValGroup/ESMValCore/pull/2198)) by [@bouweandela](https://github.com/bouweandela) ### CMOR standard - ERA5 on-the-fly CMORizer: changed sign of variables `evspsbl` and `evspsblpot` ([Pull request #2115](https://github.com/ESMValGroup/ESMValCore/pull/2115)) by [@katjaweigel](https://github.com/katjaweigel) - Add `ch4` surface custom cmor table entry ([Pull request #2168](https://github.com/ESMValGroup/ESMValCore/pull/2168)) by [@hb326](https://github.com/hb326) - Add CMIP3 institutes names used at NCI ([Pull request #2152](https://github.com/ESMValGroup/ESMValCore/pull/2152)) by [@rbeucher](https://github.com/rbeucher) - Added [`get_time_bounds()`](api/esmvalcore.cmor.html.md#esmvalcore.cmor.fixes.get_time_bounds) and [`get_next_month()`](api/esmvalcore.cmor.html.md#esmvalcore.cmor.fixes.get_next_month) to public API ([Pull request #2214](https://github.com/ESMValGroup/ESMValCore/pull/2214)) by [@schlunma](https://github.com/schlunma) - Improve concatenation checks - Relax concatenation checks for `--check_level=relax` and `--check_level=ignore` ([Pull request #2144](https://github.com/ESMValGroup/ESMValCore/pull/2144)) by [@sloosvel](https://github.com/sloosvel) - Fix `concatenate` preprocessor function ([Pull request #2240](https://github.com/ESMValGroup/ESMValCore/pull/2240)) by [@bouweandela](https://github.com/bouweandela) - Fix time overlap handling in concatenation ([Pull request #2247](https://github.com/ESMValGroup/ESMValCore/pull/2247)) by [@zklaus](https://github.com/zklaus) ### Computational performance improvements - Make [Minimum, maximum and interval masking](recipe/preprocessor.html.md#threshold-masking) preprocessors lazy ([Pull request #2169](https://github.com/ESMValGroup/ESMValCore/pull/2169)) by [@joergbenke](https://github.com/joergbenke) - Restored usage of numpy in \_mask_with_shp ([Pull request #2209](https://github.com/ESMValGroup/ESMValCore/pull/2209)) by [@joergbenke](https://github.com/joergbenke) - Do not realize lazy coordinate bounds in CMOR check ([Pull request #2146](https://github.com/ESMValGroup/ESMValCore/pull/2146)) by [@sloosvel](https://github.com/sloosvel) - Rechunk between preprocessor steps ([Pull request #2205](https://github.com/ESMValGroup/ESMValCore/pull/2205)) by [@bouweandela](https://github.com/bouweandela) - Reduce the size of the dask graph created by the `anomalies` preprocessor function ([Pull request #2200](https://github.com/ESMValGroup/ESMValCore/pull/2200)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Add reference to release v2.9.0 in the changelog ([Pull request #2130](https://github.com/ESMValGroup/ESMValCore/pull/2130)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Add merge instructions to release instructions ([Pull request #2131](https://github.com/ESMValGroup/ESMValCore/pull/2131)) by [@zklaus](https://github.com/zklaus) - Update mamba before building environment during Readthedocs build ([Pull request #2149](https://github.com/ESMValGroup/ESMValCore/pull/2149)) by [@valeriupredoi](https://github.com/valeriupredoi) - Ensure compatible zstandard and zstd versions for .conda support ([Pull request #2204](https://github.com/ESMValGroup/ESMValCore/pull/2204)) by [@zklaus](https://github.com/zklaus) - Remove outdated documentation ([Pull request #2210](https://github.com/ESMValGroup/ESMValCore/pull/2210)) by [@bouweandela](https://github.com/bouweandela) - Remove meercode badge from README because their API is broken ([Pull request #2224](https://github.com/ESMValGroup/ESMValCore/pull/2224)) by [@valeriupredoi](https://github.com/valeriupredoi) - Correct usage help text of version command ([Pull request #2232](https://github.com/ESMValGroup/ESMValCore/pull/2232)) by [@jfrost-mo](https://github.com/jfrost-mo) - Add `navigation_with_keys: False` to `html_theme_options` in Readthedocs `conf.py` ([Pull request #2245](https://github.com/ESMValGroup/ESMValCore/pull/2245)) by [@valeriupredoi](https://github.com/valeriupredoi) - Replace squarey badge with roundy shield for Anaconda sticker in README ([Pull request #2233](https://github.com/ESMValGroup/ESMValCore/pull/2233), [Pull request #2260](https://github.com/ESMValGroup/ESMValCore/pull/2260)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Fixes for datasets - Updated doc about fixes and added type hints to fix functions ([Pull request #2160](https://github.com/ESMValGroup/ESMValCore/pull/2160)) by [@schlunma](https://github.com/schlunma) ### Installation - Clean-up how pins are written in conda environment file ([Pull request #2125](https://github.com/ESMValGroup/ESMValCore/pull/2125)) by [@valeriupredoi](https://github.com/valeriupredoi) - Use importlib.metadata instead of deprecated pkg_resources ([Pull request #2096](https://github.com/ESMValGroup/ESMValCore/pull/2096)) by [@bouweandela](https://github.com/bouweandela) - Pin shapely to >=2.0 ([Pull request #2075](https://github.com/ESMValGroup/ESMValCore/pull/2075)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin Python to <3.12 in conda environment ([Pull request #2272](https://github.com/ESMValGroup/ESMValCore/pull/2272)) by [@bouweandela](https://github.com/bouweandela) ### Preprocessor - Improve preprocessor output sorting code ([Pull request #2111](https://github.com/ESMValGroup/ESMValCore/pull/2111)) by [@bouweandela](https://github.com/bouweandela) - Preprocess datasets in the same order as they are listed in the recipe ([Pull request #2103](https://github.com/ESMValGroup/ESMValCore/pull/2103)) by [@bouweandela](https://github.com/bouweandela) ### Automatic testing - [Github Actions] Compress all bash shell setters into one default option per workflow ([Pull request #2126](https://github.com/ESMValGroup/ESMValCore/pull/2126)) by [@valeriupredoi](https://github.com/valeriupredoi) - [Github Actions] Fix Monitor Tests Github Action ([Pull request #2135](https://github.com/ESMValGroup/ESMValCore/pull/2135)) by [@valeriupredoi](https://github.com/valeriupredoi) - [condalock] update conda lock file ([Pull request #2141](https://github.com/ESMValGroup/ESMValCore/pull/2141)) by [@valeriupredoi](https://github.com/valeriupredoi) - [Condalock] make sure mamba/conda are at latest version by forcing a pinned mamba install ([Pull request #2136](https://github.com/ESMValGroup/ESMValCore/pull/2136)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update code coverage orbs ([Pull request #2206](https://github.com/ESMValGroup/ESMValCore/pull/2206)) by [@bouweandela](https://github.com/bouweandela) - Revisit the comment-triggered Github Actions test ([Pull request #2243](https://github.com/ESMValGroup/ESMValCore/pull/2243)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove workflow that runs Github Actions tests from PR comment ([Pull request #2244](https://github.com/ESMValGroup/ESMValCore/pull/2244)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - Merge v2.9.x into main ([Pull request #2128](https://github.com/ESMValGroup/ESMValCore/pull/2128)) by [@schlunma](https://github.com/schlunma) - Fix typo in citation file ([Pull request #2182](https://github.com/ESMValGroup/ESMValCore/pull/2182)) by [@bouweandela](https://github.com/bouweandela) - Cleaned and extended function that extracts datetimes from paths ([Pull request #2181](https://github.com/ESMValGroup/ESMValCore/pull/2181)) by [@schlunma](https://github.com/schlunma) - Add file encoding (and some read modes) at open file step ([Pull request #2219](https://github.com/ESMValGroup/ESMValCore/pull/2219)) by [@valeriupredoi](https://github.com/valeriupredoi) - Check type of argument passed to `read_cmor_tables()` ([Pull request #2217](https://github.com/ESMValGroup/ESMValCore/pull/2217)) by [@valeriupredoi](https://github.com/valeriupredoi) - Dynamic HTML output for monitoring ([Pull request #2062](https://github.com/ESMValGroup/ESMValCore/pull/2062)) by [@bsolino](https://github.com/bsolino) - Use PyPI’s trusted publishers authentication ([Pull request #2269](https://github.com/ESMValGroup/ESMValCore/pull/2269)) by [@valeriupredoi](https://github.com/valeriupredoi) ## v2.9.0 ### Highlights It is now possible to use the [Dask distributed scheduler](https://docs.dask.org/en/latest/deploying.html), which can [significantly reduce the run-time of recipes](https://github.com/ESMValGroup/ESMValCore/pull/2049#pullrequestreview-1446279391). Configuration examples and advice are available in [our documentation](quickstart/configure.html.md#config-dask). More work on improving the computational performance is planned, so please share your experiences, good and bad, with this new feature in [Discussion #1763](https://github.com/ESMValGroup/ESMValCore/discussions/1763). This release includes ### Backwards incompatible changes - Remove deprecated configuration options ([Pull request #2056](https://github.com/ESMValGroup/ESMValCore/pull/2056)) by [@bouweandela](https://github.com/bouweandela) - The module `esmvalcore.experimental.config` has been removed. To upgrade, import the module from [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config) instead. - The module `esmvalcore._config` has been removed. To upgrade, use [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config) instead. - The methods `esmvalcore.config.Session.to_config_user` and `esmvalcore.config.Session.from_config_user` have been removed. To upgrade, use [`esmvalcore.config.Session`](api/esmvalcore.config.html.md#esmvalcore.config.Session) to access the configuration values directly. ### Bug fixes - Respect `ignore_warnings` settings from the project configuration in config-developer.yml in [`esmvalcore.dataset.Dataset.load()`](api/esmvalcore.dataset.html.md#esmvalcore.dataset.Dataset.load) ([Pull request #2046](https://github.com/ESMValGroup/ESMValCore/pull/2046)) by [@schlunma](https://github.com/schlunma) - Fixed usage of custom location for custom CMOR tables ([Pull request #2052](https://github.com/ESMValGroup/ESMValCore/pull/2052)) by [@schlunma](https://github.com/schlunma) - Fix issue with writing index.html when [running a recipe](quickstart/run.html.md#running) with `--resume-from` ([Pull request #2055](https://github.com/ESMValGroup/ESMValCore/pull/2055)) by [@bouweandela](https://github.com/bouweandela) - Fixed bug in ICON CMORizer that lead to shifted time coordinates ([Pull request #2038](https://github.com/ESMValGroup/ESMValCore/pull/2038)) by [@schlunma](https://github.com/schlunma) - Include `-` in allowed characters for bibtex references ([Pull request #2097](https://github.com/ESMValGroup/ESMValCore/pull/2097)) by [@alistairsellar](https://github.com/alistairsellar) - Do not raise an exception if the requested version of a file is not available for all matching files on ESGF ([Pull request #2105](https://github.com/ESMValGroup/ESMValCore/pull/2105)) by [@bouweandela](https://github.com/bouweandela) ### Computational performance improvements - Add support for [configuring Dask distributed](quickstart/configure.html.md#config-dask) ([Pull request #2049](https://github.com/ESMValGroup/ESMValCore/pull/2049), [Pull request #2122](https://github.com/ESMValGroup/ESMValCore/pull/2122)) by [@bouweandela](https://github.com/bouweandela) - Make [`esmvalcore.preprocessor.extract_levels()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_levels) lazy ([Pull request #1761](https://github.com/ESMValGroup/ESMValCore/pull/1761)) by [@bouweandela](https://github.com/bouweandela) - Lazy implementation of [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) and [`esmvalcore.preprocessor.ensemble_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.ensemble_statistics) ([Pull request #968](https://github.com/ESMValGroup/ESMValCore/pull/968) and [Pull request #2087](https://github.com/ESMValGroup/ESMValCore/pull/2087)) by [@Peter9192](https://github.com/Peter9192) - Avoid realizing data in preprocessor function [`esmvalcore.preprocessor.concatenate()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.concatenate) when cubes overlap ([Pull request #2109](https://github.com/ESMValGroup/ESMValCore/pull/2109)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Remove unneeded sphinxcontrib extension ([Pull request #2047](https://github.com/ESMValGroup/ESMValCore/pull/2047)) by [@valeriupredoi](https://github.com/valeriupredoi) - Show ESMValTool logo on [PyPI webpage](https://pypi.org/project/ESMValCore/) ([Pull request #2065](https://github.com/ESMValGroup/ESMValCore/pull/2065)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix gitter badge in README ([Pull request #2118](https://github.com/ESMValGroup/ESMValCore/pull/2118)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Add changelog for v2.9.0 ([Pull request #2088](https://github.com/ESMValGroup/ESMValCore/pull/2088) and [Pull request #2123](https://github.com/ESMValGroup/ESMValCore/pull/2123)) by [@bouweandela](https://github.com/bouweandela) ### Fixes for datasets - Pass the [`esmvalcore.config.Session`](api/esmvalcore.config.html.md#esmvalcore.config.Session) to fixes ([Pull request #1988](https://github.com/ESMValGroup/ESMValCore/pull/1988)) by [@schlunma](https://github.com/schlunma) - ICON: Allowed specifying vertical grid information in recipe ([Pull request #2067](https://github.com/ESMValGroup/ESMValCore/pull/2067)) by [@schlunma](https://github.com/schlunma) - Allow specifying `raw_units` for CESM2, EMAC, and ICON CMORizers ([Pull request #2043](https://github.com/ESMValGroup/ESMValCore/pull/2043)) by [@schlunma](https://github.com/schlunma) - ICON: allow specifying horizontal grid file in recipe/extra facets ([Pull request #2078](https://github.com/ESMValGroup/ESMValCore/pull/2078)) by [@schlunma](https://github.com/schlunma) - Fix tas/tos CMIP6: FIO, KACE, MIROC, IITM ([Pull request #2061](https://github.com/ESMValGroup/ESMValCore/pull/2061)) by [@pepcos](https://github.com/pepcos) - Add fix for EC-Earth3-Veg tos calendar ([Pull request #2100](https://github.com/ESMValGroup/ESMValCore/pull/2100)) by [@bouweandela](https://github.com/bouweandela) - Correct GISS-E2-1-G `tos` units ([Pull request #2099](https://github.com/ESMValGroup/ESMValCore/pull/2099)) by [@bouweandela](https://github.com/bouweandela) ### Installation - Drop support for Python 3.8 ([Pull request #2053](https://github.com/ESMValGroup/ESMValCore/pull/2053)) by [@bouweandela](https://github.com/bouweandela) - Add python 3.11 to Github Actions package (conda and PyPI) installation tests ([Pull request #2083](https://github.com/ESMValGroup/ESMValCore/pull/2083)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove `with_mypy` or `with-mypy` optional tool for prospector ([Pull request #2108](https://github.com/ESMValGroup/ESMValCore/pull/2108)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Preprocessor - Added `period='hourly'` for [`esmvalcore.preprocessor.climate_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.climate_statistics) and [`esmvalcore.preprocessor.anomalies()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.anomalies) ([Pull request #2068](https://github.com/ESMValGroup/ESMValCore/pull/2068)) by [@schlunma](https://github.com/schlunma) - Support IPCC AR6 regions in [`esmvalcore.preprocessor.extract_shape()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_shape) ([Pull request #2008](https://github.com/ESMValGroup/ESMValCore/pull/2008)) by [@schlunma](https://github.com/schlunma) ## v2.8.1 ### Highlights This release adds support for Python 3.11 and includes several bugfixes. This release includes: ### Bug fixes - Pin numpy !=1.24.3 ([Pull request #2011](https://github.com/ESMValGroup/ESMValCore/pull/2011)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix a bug in recording provenance for the `mask_multimodel` preprocessor ([Pull request #1984](https://github.com/ESMValGroup/ESMValCore/pull/1984)) by [@schlunma](https://github.com/schlunma) - Fix ICON hourly data rounding issues ([Pull request #2022](https://github.com/ESMValGroup/ESMValCore/pull/2022)) by [@BauerJul](https://github.com/BauerJul) - Use the default SSL context when using the `extract_location` preprocessor ([Pull request #2023](https://github.com/ESMValGroup/ESMValCore/pull/2023)) by [@ehogan](https://github.com/ehogan) - Make time-related CMOR fixes work with time dimensions time1, time2, time3 ([Pull request #1971](https://github.com/ESMValGroup/ESMValCore/pull/1971)) by [@schlunma](https://github.com/schlunma) - Always create a cache directory for storing ICON grid files ([Pull request #2030](https://github.com/ESMValGroup/ESMValCore/pull/2030)) by [@schlunma](https://github.com/schlunma) - Fixed altitude <–> pressure level conversion for masked arrays in the `extract_levels` preprocessor ([Pull request #1999](https://github.com/ESMValGroup/ESMValCore/pull/1999)) by [@schlunma](https://github.com/schlunma) - Allowed ignoring of scalar time coordinates in the `multi_model_statistics` preprocessor ([Pull request #1961](https://github.com/ESMValGroup/ESMValCore/pull/1961)) by [@schlunma](https://github.com/schlunma) ### Fixes for datasets - Add support for hourly ICON data ([Pull request #1990](https://github.com/ESMValGroup/ESMValCore/pull/1990)) by [@BauerJul](https://github.com/BauerJul) - Fix areacello in BCC-CSM2-MR ([Pull request #1993](https://github.com/ESMValGroup/ESMValCore/pull/1993)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### Installation - Add support for Python=3.11 ([Pull request #1832](https://github.com/ESMValGroup/ESMValCore/pull/1832)) by [@valeriupredoi](https://github.com/valeriupredoi) - Modernize conda lock file creation workflow with mamba, Mambaforge etc ([Pull request #2027](https://github.com/ESMValGroup/ESMValCore/pull/2027)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin libnetcdf!=4.9.1 ([Pull request #2072](https://github.com/ESMValGroup/ESMValCore/pull/2072)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### Documentation - Add changelog for v2.8.1 ([Pull request #2079](https://github.com/ESMValGroup/ESMValCore/pull/2079)) by [@bouweandela](https://github.com/bouweandela) ### Automatic testing - Use mocked geopy.geocoders.Nominatim to avoid ReadTimeoutError ([Pull request #2005](https://github.com/ESMValGroup/ESMValCore/pull/2005)) by [@schlunma](https://github.com/schlunma) - Update pre-commit hooks ([Pull request #2020](https://github.com/ESMValGroup/ESMValCore/pull/2020)) by [@bouweandela](https://github.com/bouweandela) ## v2.8.0 ### Highlights - ESMValCore now supports wildcards in recipes and offers improved support for ancillary variables and dataset versioning thanks to contributions by [@bouweandela](https://github.com/bouweandela). For details, see [Automatically populating a recipe with all available datasets](recipe/overview.html.md#dataset-wildcards) and [Defining supplementary variables](recipe/overview.html.md#supplementary-variables). - Support for CORDEX datasets in a rotated pole coordinate system has been added by [@sloosvel](https://github.com/sloosvel). - Native [ICON](quickstart/find_data.html.md#read-icon) output is now made UGRID-compliant on-the-fly to unlock the use of more sophisticated regridding algorithms, thanks to [@schlunma](https://github.com/schlunma). - The Python API has been extended with the addition of three modules: [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config), [`esmvalcore.dataset`](api/esmvalcore.dataset.html.md#module-esmvalcore.dataset), and `esmvalcore.local`, all these features courtesy of [@bouweandela](https://github.com/bouweandela). For details, see our new example [Example notebooks](example-notebooks.html.md). - The preprocessor [`multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) has been extended to support more use-cases thanks to contributions by [@schlunma](https://github.com/schlunma). For details, see [Multi-model statistics](recipe/preprocessor.html.md#multi-model-statistics). This release includes: ### Backwards incompatible changes Please read the descriptions of the linked pull requests for detailed upgrade instructions. - The algorithm for automatically defining the ancillary variables and cell measures has been improved ([Pull request #1609](https://github.com/ESMValGroup/ESMValCore/pull/1609)) by [@bouweandela](https://github.com/bouweandela). If this does not work as expected, more examples of how to adapt your recipes are given [here](https://github.com/ESMValGroup/ESMValCore/pull/1609#Backward-incompatible-changes) and in the corresponding sections of the [recipe documentation](recipe/overview.html.md#supplementary-variables) and the [preprocessor documentation](recipe/preprocessor.html.md#preprocessors-using-supplementary-variables). - Remove deprecated features scheduled for removal in v2.8.0 or earlier ([Pull request #1826](https://github.com/ESMValGroup/ESMValCore/pull/1826)) by [@schlunma](https://github.com/schlunma). Removed `esmvalcore.iris_helpers.var_name_constraint` (has been deprecated in v2.6.0; please use [`iris.NameConstraint`](https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.html#iris.NameConstraint) with the keyword argument `var_name` instead) and the option `always_use_ne_mask` for [`esmvalcore.preprocessor.mask_landsea()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_landsea) (has been deprecated in v2.5.0; the same behavior can now be achieved by specifying `supplementary_variables`. - No files will be found if a non-existent version of a dataset is specified ([Pull request #1835](https://github.com/ESMValGroup/ESMValCore/pull/1835)) by [@bouweandela](https://github.com/bouweandela). If a `version` of a dataset is specified in the recipe, the tool will now search for exactly that version, instead of simply using the latest version. Therefore, it is necessary to make sure that the version number in the directory tree matches with the version number in the recipe to find the files. - The default filename template for obs4MIPs has been updated to better match filenames used in this project in ([Pull request #1866](https://github.com/ESMValGroup/ESMValCore/pull/1866)) by [@bouweandela](https://github.com/bouweandela). This may cause issues if you are storing all the files for obs4MIPs in a directory with no subdirectories per dataset. ### Deprecations Please read the descriptions of the linked pull requests for detailed upgrade instructions. - Various configuration related options that are now available through [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config) have been deprecated ([Pull request #1769](https://github.com/ESMValGroup/ESMValCore/pull/1769)) by [@bouweandela](https://github.com/bouweandela). - The `fx_variables` preprocessor argument and related features have been deprecated ([Pull request #1609](https://github.com/ESMValGroup/ESMValCore/pull/1609)) by [@bouweandela](https://github.com/bouweandela). See [Pull request #1609#Deprecations](https://github.com/ESMValGroup/ESMValCore/pull/1609#Deprecations) for more information. - Combined `offline` and `always_search_esgf` into a single option `search_esgf` ([Pull request #1935](https://github.com/ESMValGroup/ESMValCore/pull/1935)) [@schlunma](https://github.com/schlunma). The configuration option/command line argument `offline` has been deprecated in favor of `search_esgf`. The previous `offline: true` is now `search_esgf: never` (the default); the previous `offline: false` is now `search_esgf: when_missing`. More details on how to adapt your workflow regarding these new options are given in [Pull request #1935](https://github.com/ESMValGroup/ESMValCore/pull/1935) and the [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html?highlight=search_esgf#user-configuration-file). - `esmvalcore.preprocessor.cleanup()` has been deprecated ([Pull request #1949](https://github.com/ESMValGroup/ESMValCore/pull/1949)) [@schlunma](https://github.com/schlunma). Please do not use this anymore in the recipe (it is not necessary). ### Python API - Support searching ESGF for a specific version of a file and add `esmvalcore.esgf.ESGFFile.facets` ([Pull request #1822](https://github.com/ESMValGroup/ESMValCore/pull/1822)) by [@bouweandela](https://github.com/bouweandela) - Fix issues with searching for files on ESGF ([Pull request #1863](https://github.com/ESMValGroup/ESMValCore/pull/1863)) by [@bouweandela](https://github.com/bouweandela) - Move the `esmvalcore.experimental.config` module to [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config) ([Pull request #1769](https://github.com/ESMValGroup/ESMValCore/pull/1769)) by [@bouweandela](https://github.com/bouweandela) - Add `esmvalcore.local`, a module to search data on the local filesystem ([Pull request ##1835](https://github.com/ESMValGroup/ESMValCore/pull/#1835)) by [@bouweandela](https://github.com/bouweandela) - Add [`esmvalcore.dataset`](api/esmvalcore.dataset.html.md#module-esmvalcore.dataset) module ([Pull request #1877](https://github.com/ESMValGroup/ESMValCore/pull/1877)) by [@bouweandela](https://github.com/bouweandela) ### Bug fixes - Import from [`esmvalcore.config`](api/esmvalcore.config.html.md#module-esmvalcore.config) in the `esmvalcore.experimental` module ([Pull request #1816](https://github.com/ESMValGroup/ESMValCore/pull/1816)) by [@bouweandela](https://github.com/bouweandela) - Added scalar coords of input cubes to output of esmpy_regrid ([Pull request #1811](https://github.com/ESMValGroup/ESMValCore/pull/1811)) by [@schlunma](https://github.com/schlunma) - Fix severe bug in [`esmvalcore.preprocessor.mask_fillvalues()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.mask_fillvalues) ([Pull request #1823](https://github.com/ESMValGroup/ESMValCore/pull/1823)) by [@schlunma](https://github.com/schlunma) - Fix LWP of ICON on-the-fly CMORizer ([Pull request #1839](https://github.com/ESMValGroup/ESMValCore/pull/1839)) by [@schlunma](https://github.com/schlunma) - Fixed issue in irregular regridding regarding scalar coordinates ([Pull request #1845](https://github.com/ESMValGroup/ESMValCore/pull/1845)) by [@schlunma](https://github.com/schlunma) - Update product attributes and metadata.yml with cube metadata before saving files ([Pull request #1837](https://github.com/ESMValGroup/ESMValCore/pull/1837)) by [@schlunma](https://github.com/schlunma) - Remove an extra space character from a filename ([Pull request #1883](https://github.com/ESMValGroup/ESMValCore/pull/1883)) by [@bouweandela](https://github.com/bouweandela) - Improve resilience of ESGF search ([Pull request #1869](https://github.com/ESMValGroup/ESMValCore/pull/1869)) by [@bouweandela](https://github.com/bouweandela) - Fix issue with no files found if timerange start/end differs in length ([Pull request #1880](https://github.com/ESMValGroup/ESMValCore/pull/1880)) by [@bouweandela](https://github.com/bouweandela) - Add driver and sub_experiment tags to generate dataset aliases ([Pull request #1886](https://github.com/ESMValGroup/ESMValCore/pull/1886)) by [@sloosvel](https://github.com/sloosvel) - Fixed time points of native CESM2 output ([Pull request #1772](https://github.com/ESMValGroup/ESMValCore/pull/1772)) by [@schlunma](https://github.com/schlunma) - Fix type hints for Python versions < 3.10 ([Pull request #1897](https://github.com/ESMValGroup/ESMValCore/pull/1897)) by [@bouweandela](https://github.com/bouweandela) - Fixed set_range_in_0_360 for dask arrays ([Pull request #1919](https://github.com/ESMValGroup/ESMValCore/pull/1919)) by [@schlunma](https://github.com/schlunma) - Made equalized attributes in concatenated cubes consistent across runs ([Pull request #1783](https://github.com/ESMValGroup/ESMValCore/pull/1783)) by [@schlunma](https://github.com/schlunma) - Fix issue with reading dates from files ([Pull request #1936](https://github.com/ESMValGroup/ESMValCore/pull/1936)) by [@bouweandela](https://github.com/bouweandela) - Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2 ([Pull request #1937](https://github.com/ESMValGroup/ESMValCore/pull/1937)) by [@bouweandela](https://github.com/bouweandela) - Fix issue where data was not loaded and saved ([Pull request #1962](https://github.com/ESMValGroup/ESMValCore/pull/1962)) by [@bouweandela](https://github.com/bouweandela) - Fix type hints for Python 3.8 ([Pull request #1795](https://github.com/ESMValGroup/ESMValCore/pull/1795)) by [@bouweandela](https://github.com/bouweandela) - Update the institute facet of the CSIRO-Mk3L-1-2 model ([Pull request #1966](https://github.com/ESMValGroup/ESMValCore/pull/1966)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Fixed race condition that may result in errors in `esmvalcore.preprocessor.cleanup()` ([Pull request #1949](https://github.com/ESMValGroup/ESMValCore/pull/1949)) by [@schlunma](https://github.com/schlunma) - Update notebook so it uses supplementaries instead of ancillaries ([Pull request #1945](https://github.com/ESMValGroup/ESMValCore/pull/1945)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Fix anaconda badge in README ([Pull request #1759](https://github.com/ESMValGroup/ESMValCore/pull/1759)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix mistake in the documentation of `esmvalcore.esgf.find_files` ([Pull request #1784](https://github.com/ESMValGroup/ESMValCore/pull/1784)) by [@bouweandela](https://github.com/bouweandela) - Support linking to “stable” ESMValTool version on readthedocs ([Pull request #1608](https://github.com/ESMValGroup/ESMValCore/pull/1608)) by [@bouweandela](https://github.com/bouweandela) - Updated ICON doc with information on usage of extract_levels preprocessor ([Pull request #1903](https://github.com/ESMValGroup/ESMValCore/pull/1903)) by [@schlunma](https://github.com/schlunma) - Add changelog for latest released version v2.7.1 ([Pull request #1905](https://github.com/ESMValGroup/ESMValCore/pull/1905)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update preprocessor.rst due to renaming of NCEP dataset to NCEP-NCAR-R1 ([Pull request #1908](https://github.com/ESMValGroup/ESMValCore/pull/1908)) by [@hb326](https://github.com/hb326) - Replace timerange nested lists in docs with overview table ([Pull request #1940](https://github.com/ESMValGroup/ESMValCore/pull/1940)) by [@zklaus](https://github.com/zklaus) - Updated section “backward compatibility” in contributing.rst ([Pull request #1918](https://github.com/ESMValGroup/ESMValCore/pull/1918)) by [@axel-lauer](https://github.com/axel-lauer) - Add link to ESMValTool release procedure steps ([Pull request #1957](https://github.com/ESMValGroup/ESMValCore/pull/1957)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Synchronize documentation table of contents with ESMValTool ([Pull request #1958](https://github.com/ESMValGroup/ESMValCore/pull/1958)) by [@bouweandela](https://github.com/bouweandela) ### Improvements - Support wildcards in the recipe and improve support for ancillary variables and dataset versioning ([Pull request #1609](https://github.com/ESMValGroup/ESMValCore/pull/1609)) by [@bouweandela](https://github.com/bouweandela). More details on how to adapt your recipes are given in the corresponding pull request description and in the corresponding sections of the [recipe documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/overview.html#defining-supplementary-variables-ancillary-variables-and-cell-measures) and the [preprocessor documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/recipe/preprocessor.html#preprocessors-using-supplementary-variables). - Create a session directory with suffix “-1”, “-2”, etc if it already exists ([Pull request #1818](https://github.com/ESMValGroup/ESMValCore/pull/1818)) by [@bouweandela](https://github.com/bouweandela) - Message for users when they use esmvaltool executable from esmvalcore only ([Pull request #1831](https://github.com/ESMValGroup/ESMValCore/pull/1831)) by [@valeriupredoi](https://github.com/valeriupredoi) - Order recipe output in index.html ([Pull request #1899](https://github.com/ESMValGroup/ESMValCore/pull/1899)) by [@bouweandela](https://github.com/bouweandela) - Improve reading facets from ESGF search results ([Pull request #1920](https://github.com/ESMValGroup/ESMValCore/pull/1920)) by [@bouweandela](https://github.com/bouweandela) ### Fixes for datasets - Fix rotated coordinate grids and tas and pr for CORDEX datasets ([Pull request #1765](https://github.com/ESMValGroup/ESMValCore/pull/1765)) by [@sloosvel](https://github.com/sloosvel) - Made ICON output UGRID-compliant (on-the-fly) ([Pull request #1664](https://github.com/ESMValGroup/ESMValCore/pull/1664)) by [@schlunma](https://github.com/schlunma) - Fix automatic download of ICON grid file and make ICON UGRIDization optional (default: true) ([Pull request #1922](https://github.com/ESMValGroup/ESMValCore/pull/1922)) by [@schlunma](https://github.com/schlunma) - Add siconc fixes for EC-Earth3-Veg and EC-Earth3-Veg-LR models ([Pull request #1771](https://github.com/ESMValGroup/ESMValCore/pull/1771)) by [@egalytska](https://github.com/egalytska) - Fix siconc in KIOST-ESM ([Pull request #1829](https://github.com/ESMValGroup/ESMValCore/pull/1829)) by [@LisaBock](https://github.com/LisaBock) - Extension of ERA5 CMORizer (variable cl) ([Pull request #1850](https://github.com/ESMValGroup/ESMValCore/pull/1850)) by [@axel-lauer](https://github.com/axel-lauer) - Add standard variable names for EMAC ([Pull request #1853](https://github.com/ESMValGroup/ESMValCore/pull/1853)) by [@FranziskaWinterstein](https://github.com/FranziskaWinterstein) - Fix for FGOALS-f3-L clt ([Pull request #1928](https://github.com/ESMValGroup/ESMValCore/pull/1928)) by [@LisaBock](https://github.com/LisaBock) ### Installation - Add all deps to the conda-forge environment and suppress installing and reinstalling deps with pip at readthedocs builds ([Pull request #1786](https://github.com/ESMValGroup/ESMValCore/pull/1786)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin netCDF4<1.6.1 ([Pull request #1805](https://github.com/ESMValGroup/ESMValCore/pull/1805)) by [@bouweandela](https://github.com/bouweandela) - Unpin NetCF4 ([Pull request #1814](https://github.com/ESMValGroup/ESMValCore/pull/1814)) by [@valeriupredoi](https://github.com/valeriupredoi) - Unpin flake8 ([Pull request #1820](https://github.com/ESMValGroup/ESMValCore/pull/1820)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add iris-esmf-regrid as a dependency ([Pull request #1809](https://github.com/ESMValGroup/ESMValCore/pull/1809)) by [@sloosvel](https://github.com/sloosvel) - Pin esmpy<8.4 ([Pull request #1871](https://github.com/ESMValGroup/ESMValCore/pull/1871)) by [@zklaus](https://github.com/zklaus) - Update esmpy import for ESMF v8.4.0 ([Pull request #1876](https://github.com/ESMValGroup/ESMValCore/pull/1876)) by [@bouweandela](https://github.com/bouweandela) ### Preprocessor - Allow [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) on cubes with arbitrary dimensions ([Pull request #1808](https://github.com/ESMValGroup/ESMValCore/pull/1808)) by [@schlunma](https://github.com/schlunma) - Smarter removal of coordinate metadata in [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) preprocessor ([Pull request #1813](https://github.com/ESMValGroup/ESMValCore/pull/1813)) by [@schlunma](https://github.com/schlunma) - Allowed usage of [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) on single cubes/products ([Pull request #1849](https://github.com/ESMValGroup/ESMValCore/pull/1849)) by [@schlunma](https://github.com/schlunma) - Allowed usage of [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) on cubes with identical `name()` and `units` (but e.g. different long_name) ([Pull request #1921](https://github.com/ESMValGroup/ESMValCore/pull/1921)) by [@schlunma](https://github.com/schlunma) - Allowed ignoring scalar coordinates in [`esmvalcore.preprocessor.multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) ([Pull request #1934](https://github.com/ESMValGroup/ESMValCore/pull/1934)) by [@schlunma](https://github.com/schlunma) - Refactored [`esmvalcore.preprocessor.regrid()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.regrid) and removed unnecessary code not needed anymore due to new iris version ([Pull request #1898](https://github.com/ESMValGroup/ESMValCore/pull/1898)) by [@schlunma](https://github.com/schlunma) - Do not realise coordinates during CMOR check ([Pull request #1912](https://github.com/ESMValGroup/ESMValCore/pull/1912)) by [@sloosvel](https://github.com/sloosvel) - Make [`esmvalcore.preprocessor.extract_volume()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_volume) work with closed and mixed intervals and allow nearest value selection ([Pull request #1930](https://github.com/ESMValGroup/ESMValCore/pull/1930)) by [@sloosvel](https://github.com/sloosvel) ### Release - Changelog for v2.8.0rc1 ([Pull request #1952](https://github.com/ESMValGroup/ESMValCore/pull/1952)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Increase version number for ESMValCore v2.8.0rc1 ([Pull request #1955](https://github.com/ESMValGroup/ESMValCore/pull/1955)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Changelog for v2.8.0rc2 ([Pull request #1959](https://github.com/ESMValGroup/ESMValCore/pull/1959)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Increase version number for ESMValCore v2.8.0rc2 ([Pull request #1973](https://github.com/ESMValGroup/ESMValCore/pull/1973)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Changelog for v2.8.0 ([Pull request #1978](https://github.com/ESMValGroup/ESMValCore/pull/1978)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Increase version number for ESMValCore v2.8.0 ([Pull request #1983](https://github.com/ESMValGroup/ESMValCore/pull/1983)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### Automatic testing - Set implicit optional to true in mypy config to avert side effects and test fails from new mypy version ([Pull request #1790](https://github.com/ESMValGroup/ESMValCore/pull/1790)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove duplicate implicit_optional = True line in `setup.cfg` ([Pull request #1791](https://github.com/ESMValGroup/ESMValCore/pull/1791)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix failing test due to missing sample data ([Pull request #1797](https://github.com/ESMValGroup/ESMValCore/pull/1797)) by [@bouweandela](https://github.com/bouweandela) - Remove outdated cmor_table facet from data finder tests ([Pull request #1798](https://github.com/ESMValGroup/ESMValCore/pull/1798)) by [@bouweandela](https://github.com/bouweandela) - Modernize tests for [`esmvalcore.preprocessor.save()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.save) ([Pull request #1799](https://github.com/ESMValGroup/ESMValCore/pull/1799)) by [@bouweandela](https://github.com/bouweandela) - No more sequential tests since SegFaults were not noticed anymore ([Pull request #1819](https://github.com/ESMValGroup/ESMValCore/pull/1819)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update pre-commit configuration ([Pull request #1821](https://github.com/ESMValGroup/ESMValCore/pull/1821)) by [@bouweandela](https://github.com/bouweandela) - Updated URL of ICON grid file used for testing ([Pull request #1914](https://github.com/ESMValGroup/ESMValCore/pull/1914)) by [@schlunma](https://github.com/schlunma) ### Variable Derivation - Add derivation of sea ice extent ([Pull request #1695](https://github.com/ESMValGroup/ESMValCore/pull/1695)) by [@sloosvel](https://github.com/sloosvel) ## v2.7.1 ### Highlights This is a bugfix release where we unpin cf-units to allow the latest iris=3.4.0 to be installed. It also includes an update to the default configuration used when searching the ESGF for files, to account for a recent change of the CEDA ESGF index node hostname. The changelog contains only changes that were made to the `main` branch. ### Installation - Set the version number on the development branches to one minor version more than the previous release ([Pull request #1854](https://github.com/ESMValGroup/ESMValCore/pull/1854)) by [@bouweandela](https://github.com/bouweandela) - Unpin cf-units ([Pull request #1770](https://github.com/ESMValGroup/ESMValCore/pull/1770)) by [@bouweandela](https://github.com/bouweandela) ### Bug fixes - Improve error handling if an esgf index node is offline ([Pull request #1834](https://github.com/ESMValGroup/ESMValCore/pull/1834)) by [@bouweandela](https://github.com/bouweandela) ### Automatic testing - Removed unnecessary test that fails with iris 3.4.0 ([Pull request #1846](https://github.com/ESMValGroup/ESMValCore/pull/1846)) by [@schlunma](https://github.com/schlunma) - Update CEDA ESGF index node hostname ([Pull request #1838](https://github.com/ESMValGroup/ESMValCore/pull/1838)) by [@valeriupredoi](https://github.com/valeriupredoi) ## v2.7.0 ### Highlights - We have a new preprocessor function called [‘rolling_window_statistics’](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/preprocessor.html#rolling-window-statistics) implemented by [@malininae](https://github.com/malininae) - We have improved the support for native models, refactored native model fixes by adding common base class NativeDatasetFix, changed default DRS for reading native ICON output, and added tests for input/output filenames for [ICON](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/find_data.html#icon) and [EMAC](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/find_data.html#emac) on-the-fly CMORizer, all these features courtesy of [@schlunma](https://github.com/schlunma) - Performance of preprocessor functions that use time dimensions has been sped up by **two orders of magnitude** thanks to contributions by [@bouweandela](https://github.com/bouweandela) This release includes: ### Backwards incompatible changes - Change default DRS for reading native ICON output ([Pull request #1705](https://github.com/ESMValGroup/ESMValCore/pull/1705)) by [@schlunma](https://github.com/schlunma) ### Bug fixes - Add support for regions stored as MultiPolygon to extract_shape preprocessor ([Pull request #1670](https://github.com/ESMValGroup/ESMValCore/pull/1670)) by [@bouweandela](https://github.com/bouweandela) - Fixed type annotations for Python 3.8 ([Pull request #1700](https://github.com/ESMValGroup/ESMValCore/pull/1700)) by [@schlunma](https://github.com/schlunma) - Core \_io.concatenate() may fail due to case when one of the cubes is scalar - this fixes that ([Pull request #1715](https://github.com/ESMValGroup/ESMValCore/pull/1715)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pick up esmvalcore badge instead of esmvaltool one in README ([Pull request #1749](https://github.com/ESMValGroup/ESMValCore/pull/1749)) by [@valeriupredoi](https://github.com/valeriupredoi) - Restore support for scalar cubes to time selection preprocessor functions ([Pull request #1750](https://github.com/ESMValGroup/ESMValCore/pull/1750)) by [@bouweandela](https://github.com/bouweandela) - Fix calculation of precipitation flux in EMAC on-the-fly CMORizer ([Pull request #1755](https://github.com/ESMValGroup/ESMValCore/pull/1755)) by [@schlunma](https://github.com/schlunma) ### Deprecations - Remove deprecation warning for regrid schemes already deprecated for v2.7.0 ([Pull request #1753](https://github.com/ESMValGroup/ESMValCore/pull/1753)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Documentation - Add Met Office Installation Method ([Pull request #1692](https://github.com/ESMValGroup/ESMValCore/pull/1692)) by [@mo-tgeddes](https://github.com/mo-tgeddes) - Add MO-paths to config file ([Pull request #1709](https://github.com/ESMValGroup/ESMValCore/pull/1709)) by [@mo-tgeddes](https://github.com/mo-tgeddes) - Update MO obs4MIPs paths in the user configuration file ([Pull request #1734](https://github.com/ESMValGroup/ESMValCore/pull/1734)) by [@mo-tgeddes](https://github.com/mo-tgeddes) - Update Making a release section of the documentation ([Pull request #1689](https://github.com/ESMValGroup/ESMValCore/pull/1689)) by [@sloosvel](https://github.com/sloosvel) - Added changelog for v2.7.0 ([Pull request #1746](https://github.com/ESMValGroup/ESMValCore/pull/1746)) by [@valeriupredoi](https://github.com/valeriupredoi) - update CITATION.cff file with 2.7.0 release info ([Pull request #1757](https://github.com/ESMValGroup/ESMValCore/pull/1757)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - New preprocessor function ‘rolling_window_statistics’ ([Pull request #1702](https://github.com/ESMValGroup/ESMValCore/pull/1702)) by [@malininae](https://github.com/malininae) - Remove pytest_flake8 plugin and use flake8 instead ([Pull request #1722](https://github.com/ESMValGroup/ESMValCore/pull/1722)) by [@valeriupredoi](https://github.com/valeriupredoi) - Added CESM2 CMORizer ([Pull request #1678](https://github.com/ESMValGroup/ESMValCore/pull/1678)) by [@schlunma](https://github.com/schlunma) - Speed up functions that use time dimension ([Pull request #1713](https://github.com/ESMValGroup/ESMValCore/pull/1713)) by [@bouweandela](https://github.com/bouweandela) - Modernize and minimize pylint configuration ([Pull request #1726](https://github.com/ESMValGroup/ESMValCore/pull/1726)) by [@bouweandela](https://github.com/bouweandela) ### Fixes for datasets - Refactored native model fixes by adding common base class NativeDatasetFix ([Pull request #1694](https://github.com/ESMValGroup/ESMValCore/pull/1694)) by [@schlunma](https://github.com/schlunma) ### Installation - Pin netCDF4 != 1.6.1 since that seems to throw a flurry of Segmentation Faults ([Pull request #1724](https://github.com/ESMValGroup/ESMValCore/pull/1724)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Automatic testing - Pin flake8<5.0.0 since Circle CI tests are failing copiously ([Pull request #1698](https://github.com/ESMValGroup/ESMValCore/pull/1698)) by [@valeriupredoi](https://github.com/valeriupredoi) - Added tests for input/output filenames for ICON and EMAC on-the-fly CMORizer ([Pull request #1718](https://github.com/ESMValGroup/ESMValCore/pull/1718)) by [@schlunma](https://github.com/schlunma) - Fix failed tests for Python<3.10 resulting from typing ([Pull request #1748](https://github.com/ESMValGroup/ESMValCore/pull/1748)) by [@schlunma](https://github.com/schlunma) ## v2.6.0 ### Highlights - A new set of CMOR fixes is now available in order to load native EMAC model output and CMORize it on the fly. For details, see [Supported native models: EMAC](quickstart/find_data.html.md#read-emac). - The version number of ESMValCore is now automatically generated using [setuptools_scm](https://github.com/pypa/setuptools_scm/#default-versioning-scheme), which extracts Python package versions from git metadata. This release includes ### Deprecations - Deprecate the function esmvalcore.var_name_constraint ([Pull request #1592](https://github.com/ESMValGroup/ESMValCore/pull/1592)) by [@schlunma](https://github.com/schlunma). This function is scheduled for removal in v2.8.0. Please use [`iris.NameConstraint`](https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.html#iris.NameConstraint) with the keyword argument var_name instead: this is an exact replacement. ### Bug fixes - Added start_year and end_year attributes to derived variables ([Pull request #1547](https://github.com/ESMValGroup/ESMValCore/pull/1547)) by [@schlunma](https://github.com/schlunma) - Show all results on recipe results webpage ([Pull request #1560](https://github.com/ESMValGroup/ESMValCore/pull/1560)) by [@bouweandela](https://github.com/bouweandela) - Regridding regular grids with similar coordinates ([Pull request #1567](https://github.com/ESMValGroup/ESMValCore/pull/1567)) by [@tomaslovato](https://github.com/tomaslovato) - Fix timerange wildcard search when deriving variables or downloading files ([Pull request #1562](https://github.com/ESMValGroup/ESMValCore/pull/1562)) by [@sloosvel](https://github.com/sloosvel) - Fix force_derivation bug ([Pull request #1627](https://github.com/ESMValGroup/ESMValCore/pull/1627)) by [@sloosvel](https://github.com/sloosvel) - Correct build-and-deploy-on-pypi action ([Pull request #1634](https://github.com/ESMValGroup/ESMValCore/pull/1634)) by [@sloosvel](https://github.com/sloosvel) - Apply clip_timerange to time dependent fx variables ([Pull request #1603](https://github.com/ESMValGroup/ESMValCore/pull/1603)) by [@sloosvel](https://github.com/sloosvel) - Correctly handle requests.exceptions.ConnectTimeout when an ESGF index node is offline ([Pull request #1638](https://github.com/ESMValGroup/ESMValCore/pull/1638)) by [@bouweandela](https://github.com/bouweandela) ### CMOR standard - Added custom CMOR tables used for EMAC CMORizer ([Pull request #1599](https://github.com/ESMValGroup/ESMValCore/pull/1599)) by [@schlunma](https://github.com/schlunma) - Extended ICON CMORizer ([Pull request #1549](https://github.com/ESMValGroup/ESMValCore/pull/1549)) by [@schlunma](https://github.com/schlunma) - Add CMOR check exception for a basin coord named sector ([Pull request #1612](https://github.com/ESMValGroup/ESMValCore/pull/1612)) by [@dhohn](https://github.com/dhohn) - Custom user-defined location for custom CMOR tables ([Pull request #1625](https://github.com/ESMValGroup/ESMValCore/pull/1625)) by [@schlunma](https://github.com/schlunma) ### Containerization - Remove update command in Dockerfile ([Pull request #1630](https://github.com/ESMValGroup/ESMValCore/pull/1630)) by [@sloosvel](https://github.com/sloosvel) ### Community - Add David Hohn to contributors’ list ([Pull request #1586](https://github.com/ESMValGroup/ESMValCore/pull/1586)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Documentation - [Github Actions Docs] Full explanation on how to use the GA test triggered by PR comment and added docs link for GA hosted runners ([Pull request #1553](https://github.com/ESMValGroup/ESMValCore/pull/1553)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update the command for building the documentation ([Pull request #1556](https://github.com/ESMValGroup/ESMValCore/pull/1556)) by [@bouweandela](https://github.com/bouweandela) - Update documentation on running the tool ([Pull request #1400](https://github.com/ESMValGroup/ESMValCore/pull/1400)) by [@bouweandela](https://github.com/bouweandela) - Add support for DKRZ-Levante ([Pull request #1558](https://github.com/ESMValGroup/ESMValCore/pull/1558)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Improved documentation on native dataset support ([Pull request #1559](https://github.com/ESMValGroup/ESMValCore/pull/1559)) by [@schlunma](https://github.com/schlunma) - Tweak extract_point preprocessor: explain what it returns if one point coord outside cube and add explicit test ([Pull request #1584](https://github.com/ESMValGroup/ESMValCore/pull/1584)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update CircleCI, readthedocs, and Docker configuration ([Pull request #1588](https://github.com/ESMValGroup/ESMValCore/pull/1588)) by [@bouweandela](https://github.com/bouweandela) - Remove support for Mistral in config-user.yml ([Pull request #1620](https://github.com/ESMValGroup/ESMValCore/pull/1620)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Add changelog for v2.6.0rc1 ([Pull request #1633](https://github.com/ESMValGroup/ESMValCore/pull/1633)) by [@sloosvel](https://github.com/sloosvel) - Add a note on transferring permissions to the release manager ([Pull request #1645](https://github.com/ESMValGroup/ESMValCore/pull/1645)) by [@bouweandela](https://github.com/bouweandela) - Add documentation on building and uploading Docker images ([Pull request #1644](https://github.com/ESMValGroup/ESMValCore/pull/1644)) by [@bouweandela](https://github.com/bouweandela) - Update documentation on ESMValTool module at DKRZ ([Pull request #1647](https://github.com/ESMValGroup/ESMValCore/pull/1647)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Expanded information on deprecations in changelog ([Pull request #1658](https://github.com/ESMValGroup/ESMValCore/pull/1658)) by [@schlunma](https://github.com/schlunma) ### Improvements - Removed trailing whitespace in custom CMOR tables ([Pull request #1564](https://github.com/ESMValGroup/ESMValCore/pull/1564)) by [@schlunma](https://github.com/schlunma) - Try searching multiple ESGF index nodes ([Pull request #1561](https://github.com/ESMValGroup/ESMValCore/pull/1561)) by [@bouweandela](https://github.com/bouweandela) - Add CMIP6 amoc derivation case and add a test ([Pull request #1577](https://github.com/ESMValGroup/ESMValCore/pull/1577)) by [@valeriupredoi](https://github.com/valeriupredoi) - Added EMAC CMORizer ([Pull request #1554](https://github.com/ESMValGroup/ESMValCore/pull/1554)) by [@schlunma](https://github.com/schlunma) - Improve performance of volume_statistics ([Pull request #1545](https://github.com/ESMValGroup/ESMValCore/pull/1545)) by [@sloosvel](https://github.com/sloosvel) ### Fixes for datasets - Fixes of ocean variables in multiple CMIP6 datasets ([Pull request #1566](https://github.com/ESMValGroup/ESMValCore/pull/1566)) by [@tomaslovato](https://github.com/tomaslovato) - Ensure lat/lon bounds in FGOALS-l3 atmos variables are contiguous ([Pull request #1571](https://github.com/ESMValGroup/ESMValCore/pull/1571)) by [@sloosvel](https://github.com/sloosvel) - Added AllVars fix for CMIP6’s ICON-ESM-LR ([Pull request #1582](https://github.com/ESMValGroup/ESMValCore/pull/1582)) by [@schlunma](https://github.com/schlunma) ### Installation - Removed package/meta.yml ([Pull request #1540](https://github.com/ESMValGroup/ESMValCore/pull/1540)) by [@schlunma](https://github.com/schlunma) - Pinned iris>=3.2.1 ([Pull request #1552](https://github.com/ESMValGroup/ESMValCore/pull/1552)) by [@schlunma](https://github.com/schlunma) - Use setuptools-scm to automatically generate the version number ([Pull request #1578](https://github.com/ESMValGroup/ESMValCore/pull/1578)) by [@bouweandela](https://github.com/bouweandela) - Pin cf-units to lower than 3.1.0 to temporarily avoid changes within new version related to calendars ([Pull request #1659](https://github.com/ESMValGroup/ESMValCore/pull/1659)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Preprocessor - Allowed special case for unit conversion of precipitation (kg m-2 s-1 <–> mm day-1) ([Pull request #1574](https://github.com/ESMValGroup/ESMValCore/pull/1574)) by [@schlunma](https://github.com/schlunma) - Add general extract_coordinate_points preprocessor ([Pull request #1581](https://github.com/ESMValGroup/ESMValCore/pull/1581)) by [@sloosvel](https://github.com/sloosvel) - Add preprocessor accumulate_coordinate ([Pull request #1281](https://github.com/ESMValGroup/ESMValCore/pull/1281)) by [@jvegreg](https://github.com/jvegreg) - Add axis_statistics and improve depth_integration ([Pull request #1589](https://github.com/ESMValGroup/ESMValCore/pull/1589)) by [@sloosvel](https://github.com/sloosvel) ### Release - Increase version number for ESMValCore v2.6.0rc1 ([Pull request #1632](https://github.com/ESMValGroup/ESMValCore/pull/1632)) by [@sloosvel](https://github.com/sloosvel) - Update changelog and version for 2.6rc3 ([Pull request #1646](https://github.com/ESMValGroup/ESMValCore/pull/1646)) by [@sloosvel](https://github.com/sloosvel) - Add changelog for rc4 ([Pull request #1662](https://github.com/ESMValGroup/ESMValCore/pull/1662)) by [@sloosvel](https://github.com/sloosvel) ### Automatic testing - Refresh CircleCI cache weekly ([Pull request #1597](https://github.com/ESMValGroup/ESMValCore/pull/1597)) by [@bouweandela](https://github.com/bouweandela) - Use correct cache restore key on CircleCI ([Pull request #1598](https://github.com/ESMValGroup/ESMValCore/pull/1598)) by [@bouweandela](https://github.com/bouweandela) - Install git and ssh before checking out code on CircleCI ([Pull request #1601](https://github.com/ESMValGroup/ESMValCore/pull/1601)) by [@bouweandela](https://github.com/bouweandela) - Fetch all history in Github Action tests ([Pull request #1622](https://github.com/ESMValGroup/ESMValCore/pull/1622)) by [@sloosvel](https://github.com/sloosvel) - Test Github Actions dashboard badge from meercode.io ([Pull request #1640](https://github.com/ESMValGroup/ESMValCore/pull/1640)) by [@valeriupredoi](https://github.com/valeriupredoi) - Improve esmvalcore.esgf unit test ([Pull request #1650](https://github.com/ESMValGroup/ESMValCore/pull/1650)) by [@bouweandela](https://github.com/bouweandela) ### Variable Derivation - Added derivation of hfns ([Pull request #1594](https://github.com/ESMValGroup/ESMValCore/pull/1594)) by [@schlunma](https://github.com/schlunma) ## v2.5.0 ### Highlights - The new preprocessor [`extract_location()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.extract_location) can extract arbitrary locations on the Earth using the [geopy](https://pypi.org/project/geopy/) package that connects to OpenStreetMap. For details, see [Extract location](recipe/preprocessor.html.md#extract-location). - Time ranges can now be extracted using the [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601). In addition, wildcards are allowed, which makes the time selection much more flexible. For details, see [Recipe section: Datasets](recipe/overview.html.md#datasets). - The new preprocessor [`ensemble_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.ensemble_statistics) can calculate arbitrary statistics over all ensemble members of a simulation. In addition, the preprocessor [`multi_model_statistics()`](api/esmvalcore.preprocessor.html.md#esmvalcore.preprocessor.multi_model_statistics) now accepts the keyword `groupy`, which allows the calculation of multi-model statistics over arbitrary multi-model ensembles. For details, see [Ensemble statistics](recipe/preprocessor.html.md#ensemble-statistics) and [Multi-model statistics](recipe/preprocessor.html.md#multi-model-statistics). This release includes ### Backwards incompatible changes - Update Cordex section in config-developer.yml ([Pull request #1303](https://github.com/ESMValGroup/ESMValCore/pull/1303)) by [@francesco-cmcc](https://github.com/francesco-cmcc). This changes the naming convention of ESMValCore’s output files from CORDEX dataset. This only affects recipes that use CORDEX data. Most likely, no changes in diagnostics are necessary; however, if code relies on the specific naming convention of files, it might need to be adapted. - Dropped Python 3.7 ([Pull request #1530](https://github.com/ESMValGroup/ESMValCore/pull/1530)) by [@schlunma](https://github.com/schlunma). ESMValCore v2.5.0 dropped support for Python 3.7. From now on Python >=3.8 is required to install ESMValCore. The main reason for this is that conda-forge dropped support for Python 3.7 for OSX and arm64 (more details are given [here](https://github.com/ESMValGroup/ESMValTool/issues/2584#issuecomment-1063853630)). ### Bug fixes - Fix extract_shape when fx vars are present ([Pull request #1403](https://github.com/ESMValGroup/ESMValCore/pull/1403)) by [@sloosvel](https://github.com/sloosvel) - Added support of extra_facets to fx variables added by the preprocessor ([Pull request #1399](https://github.com/ESMValGroup/ESMValCore/pull/1399)) by [@schlunma](https://github.com/schlunma) - Augmented input for derived variables with extra_facets ([Pull request #1412](https://github.com/ESMValGroup/ESMValCore/pull/1412)) by [@schlunma](https://github.com/schlunma) - Correctly use masked arrays after unstructured_nearest regridding ([Pull request #1414](https://github.com/ESMValGroup/ESMValCore/pull/1414)) by [@schlunma](https://github.com/schlunma) - Fixing the broken derivation script for XCH4 (and XCO2) ([Pull request #1428](https://github.com/ESMValGroup/ESMValCore/pull/1428)) by [@hb326](https://github.com/hb326) - Ignore .pymon-journal file in test discovery ([Pull request #1436](https://github.com/ESMValGroup/ESMValCore/pull/1436)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fixed bug that caused automatic download to fail in rare cases ([Pull request #1442](https://github.com/ESMValGroup/ESMValCore/pull/1442)) by [@schlunma](https://github.com/schlunma) - Add new JULIA_LOAD_PATH to diagnostic task test ([Pull request #1444](https://github.com/ESMValGroup/ESMValCore/pull/1444)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix provenance file permissions ([Pull request #1468](https://github.com/ESMValGroup/ESMValCore/pull/1468)) by [@bouweandela](https://github.com/bouweandela) - Fixed usage of statistics=std_dev option in multi-model statistics preprocessors ([Pull request #1478](https://github.com/ESMValGroup/ESMValCore/pull/1478)) by [@schlunma](https://github.com/schlunma) - Removed scalar coordinates p0 and ptop prior to merge in multi_model_statistics ([Pull request #1471](https://github.com/ESMValGroup/ESMValCore/pull/1471)) by [@axel-lauer](https://github.com/axel-lauer) - Added dataset and alias attributes to multi_model_statistics output ([Pull request #1483](https://github.com/ESMValGroup/ESMValCore/pull/1483)) by [@schlunma](https://github.com/schlunma) - Fixed issues with multi-model-statistics timeranges ([Pull request #1486](https://github.com/ESMValGroup/ESMValCore/pull/1486)) by [@schlunma](https://github.com/schlunma) - Fixed output messages for CMOR logging ([Pull request #1494](https://github.com/ESMValGroup/ESMValCore/pull/1494)) by [@schlunma](https://github.com/schlunma) - Fixed clip_timerange if only a single time point is extracted ([Pull request #1497](https://github.com/ESMValGroup/ESMValCore/pull/1497)) by [@schlunma](https://github.com/schlunma) - Fixed chunking in multi_model_statistics ([Pull request #1500](https://github.com/ESMValGroup/ESMValCore/pull/1500)) by [@schlunma](https://github.com/schlunma) - Fixed renaming of auxiliary coordinates in multi_model_statistics if coordinates are equal ([Pull request #1502](https://github.com/ESMValGroup/ESMValCore/pull/1502)) by [@schlunma](https://github.com/schlunma) - Fixed timerange selection for automatic downloads ([Pull request #1517](https://github.com/ESMValGroup/ESMValCore/pull/1517)) by [@schlunma](https://github.com/schlunma) - Fixed chunking in multi_model_statistics ([Pull request #1524](https://github.com/ESMValGroup/ESMValCore/pull/1524)) by [@schlunma](https://github.com/schlunma) ### Deprecations - Renamed vertical regridding schemes ([Pull request #1429](https://github.com/ESMValGroup/ESMValCore/pull/1429)) by [@schlunma](https://github.com/schlunma). Old regridding schemes are supported until v2.7.0. For details, see [Vertical interpolation schemes](recipe/preprocessor.html.md#vertical-interpolation-schemes). ### Documentation - Remove duplicate entries in changelog ([Pull request #1391](https://github.com/ESMValGroup/ESMValCore/pull/1391)) by [@zklaus](https://github.com/zklaus) - Documentation on how to use HPC central installations ([Pull request #1409](https://github.com/ESMValGroup/ESMValCore/pull/1409)) by [@valeriupredoi](https://github.com/valeriupredoi) - Correct brackets in preprocessor documentation for list of seasons ([Pull request #1420](https://github.com/ESMValGroup/ESMValCore/pull/1420)) by [@bouweandela](https://github.com/bouweandela) - Add Python=3.10 to package info, update Circle CI auto install and documentation for Python=3.10 ([Pull request #1432](https://github.com/ESMValGroup/ESMValCore/pull/1432)) by [@valeriupredoi](https://github.com/valeriupredoi) - Reverted unintentional change in .zenodo.json ([Pull request #1452](https://github.com/ESMValGroup/ESMValCore/pull/1452)) by [@schlunma](https://github.com/schlunma) - Synchronized config-user.yml with version from ESMValTool ([Pull request #1453](https://github.com/ESMValGroup/ESMValCore/pull/1453)) by [@schlunma](https://github.com/schlunma) - Solved issues in configuration files ([Pull request #1457](https://github.com/ESMValGroup/ESMValCore/pull/1457)) by [@schlunma](https://github.com/schlunma) - Add direct link to download conda lock file in the install documentation ([Pull request #1462](https://github.com/ESMValGroup/ESMValCore/pull/1462)) by [@valeriupredoi](https://github.com/valeriupredoi) - CITATION.cff fix and automatic validation of citation metadata ([Pull request #1467](https://github.com/ESMValGroup/ESMValCore/pull/1467)) by [@valeriupredoi](https://github.com/valeriupredoi) - Updated documentation on how to deprecate features ([Pull request #1426](https://github.com/ESMValGroup/ESMValCore/pull/1426)) by [@schlunma](https://github.com/schlunma) - Added reference hook to conda lock in documentation install section ([Pull request #1473](https://github.com/ESMValGroup/ESMValCore/pull/1473)) by [@valeriupredoi](https://github.com/valeriupredoi) - Increased ESMValCore version to 2.5.0rc1 ([Pull request #1477](https://github.com/ESMValGroup/ESMValCore/pull/1477)) by [@schlunma](https://github.com/schlunma) - Added changelog for v2.5.0 release ([Pull request #1476](https://github.com/ESMValGroup/ESMValCore/pull/1476)) by [@schlunma](https://github.com/schlunma) - Increased ESMValCore version to 2.5.0rc2 ([Pull request #1487](https://github.com/ESMValGroup/ESMValCore/pull/1487)) by [@schlunma](https://github.com/schlunma) - Added some authors to citation and zenodo files ([Pull request #1488](https://github.com/ESMValGroup/ESMValCore/pull/1488)) by [@SarahAlidoost](https://github.com/SarahAlidoost) - Restored scipy intersphinx mapping ([Pull request #1491](https://github.com/ESMValGroup/ESMValCore/pull/1491)) by [@schlunma](https://github.com/schlunma) - Increased ESMValCore version to 2.5.0rc3 ([Pull request #1504](https://github.com/ESMValGroup/ESMValCore/pull/1504)) by [@schlunma](https://github.com/schlunma) - Fix download instructions for the MSWEP dataset ([Pull request #1506](https://github.com/ESMValGroup/ESMValCore/pull/1506)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Documentation updated for the new cmorizer framework ([Pull request #1417](https://github.com/ESMValGroup/ESMValCore/pull/1417)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Added tests for duplicates in changelog and removed duplicates ([Pull request #1508](https://github.com/ESMValGroup/ESMValCore/pull/1508)) by [@schlunma](https://github.com/schlunma) - Increased ESMValCore version to 2.5.0rc4 ([Pull request #1519](https://github.com/ESMValGroup/ESMValCore/pull/1519)) by [@schlunma](https://github.com/schlunma) - Add Github Actions Test badge in README ([Pull request #1526](https://github.com/ESMValGroup/ESMValCore/pull/1526)) by [@valeriupredoi](https://github.com/valeriupredoi) - Increased ESMValCore version to 2.5.0rc5 ([Pull request #1529](https://github.com/ESMValGroup/ESMValCore/pull/1529)) by [@schlunma](https://github.com/schlunma) - Increased ESMValCore version to 2.5.0rc6 ([Pull request #1532](https://github.com/ESMValGroup/ESMValCore/pull/1532)) by [@schlunma](https://github.com/schlunma) ### Fixes for datasets - Added fix for AIRS v2.1 (obs4mips) ([Pull request #1472](https://github.com/ESMValGroup/ESMValCore/pull/1472)) by [@axel-lauer](https://github.com/axel-lauer) ### Preprocessor - Added bias preprocessor ([Pull request #1406](https://github.com/ESMValGroup/ESMValCore/pull/1406)) by [@schlunma](https://github.com/schlunma) - Improve error messages when a preprocessor is failing ([Pull request #1408](https://github.com/ESMValGroup/ESMValCore/pull/1408)) by [@schlunma](https://github.com/schlunma) - Added option to explicitly not use fx variables in preprocessors ([Pull request #1416](https://github.com/ESMValGroup/ESMValCore/pull/1416)) by [@schlunma](https://github.com/schlunma) - Add extract_location preprocessor to extract town, city, mountains etc - anything specifiable by a location ([Pull request #1251](https://github.com/ESMValGroup/ESMValCore/pull/1251)) by [@jvegreg](https://github.com/jvegreg) - Add ensemble statistics preprocessor and ‘groupby’ option for multimodel ([Pull request #673](https://github.com/ESMValGroup/ESMValCore/pull/673)) by [@sloosvel](https://github.com/sloosvel) - Generic regridding preprocessor ([Pull request #1448](https://github.com/ESMValGroup/ESMValCore/pull/1448)) by [@zklaus](https://github.com/zklaus) ### Automatic testing - Add pandas as dependency :panda_face: ([Pull request #1402](https://github.com/ESMValGroup/ESMValCore/pull/1402)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fixed tests for python 3.7 ([Pull request #1410](https://github.com/ESMValGroup/ESMValCore/pull/1410)) by [@schlunma](https://github.com/schlunma) - Remove accessing .xml() cube method from test ([Pull request #1419](https://github.com/ESMValGroup/ESMValCore/pull/1419)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove flag to use pip 2020 solver from Github Action pip install command on OSX ([Pull request #1357](https://github.com/ESMValGroup/ESMValCore/pull/1357)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add Python=3.10 to Github Actions and switch to Python=3.10 for the Github Action that builds the PyPi package ([Pull request #1430](https://github.com/ESMValGroup/ESMValCore/pull/1430)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin flake8<4 to keep getting relevant error traces when tests fail with FLAKE8 issues ([Pull request #1434](https://github.com/ESMValGroup/ESMValCore/pull/1434)) by [@valeriupredoi](https://github.com/valeriupredoi) - Implementing conda lock ([Pull request #1164](https://github.com/ESMValGroup/ESMValCore/pull/1164)) by [@valeriupredoi](https://github.com/valeriupredoi) - Relocate pytest-monitor outputted database .pymon so .pymon-journal file should not be looked for by pytest ([Pull request #1441](https://github.com/ESMValGroup/ESMValCore/pull/1441)) by [@valeriupredoi](https://github.com/valeriupredoi) - Switch to Mambaforge in Github Actions tests ([Pull request #1438](https://github.com/ESMValGroup/ESMValCore/pull/1438)) by [@valeriupredoi](https://github.com/valeriupredoi) - Turn off conda lock file creation on any push on main branch from Github Action test ([Pull request #1489](https://github.com/ESMValGroup/ESMValCore/pull/1489)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add DRS path test for IPSLCM files ([Pull request #1490](https://github.com/ESMValGroup/ESMValCore/pull/1490)) by [@senesis](https://github.com/senesis) - Add a test module that runs tests of iris I/O every time we notice serious bugs there ([Pull request #1510](https://github.com/ESMValGroup/ESMValCore/pull/1510)) by [@valeriupredoi](https://github.com/valeriupredoi) - [Github Actions] Trigger Github Actions tests (run-tests.yml workflow) from a comment in a PR ([Pull request #1520](https://github.com/ESMValGroup/ESMValCore/pull/1520)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update Linux condalock file (various pull requests) github-actions[bot] ### Installation - Move nested-lookup dependency to environment.yml to be installed from conda-forge instead of PyPi ([Pull request #1481](https://github.com/ESMValGroup/ESMValCore/pull/1481)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pinned iris ([Pull request #1511](https://github.com/ESMValGroup/ESMValCore/pull/1511)) by [@schlunma](https://github.com/schlunma) - Updated dependencies ([Pull request #1521](https://github.com/ESMValGroup/ESMValCore/pull/1521)) by [@schlunma](https://github.com/schlunma) - Pinned iris<3.2.0 ([Pull request #1525](https://github.com/ESMValGroup/ESMValCore/pull/1525)) by [@schlunma](https://github.com/schlunma) ### Improvements - Allow to load all files, first X years or last X years in an experiment ([Pull request #1133](https://github.com/ESMValGroup/ESMValCore/pull/1133)) by [@sloosvel](https://github.com/sloosvel) - Filter tasks earlier ([Pull request #1264](https://github.com/ESMValGroup/ESMValCore/pull/1264)) by [@jvegreg](https://github.com/jvegreg) - Added earlier validation for command line arguments ([Pull request #1435](https://github.com/ESMValGroup/ESMValCore/pull/1435)) by [@schlunma](https://github.com/schlunma) - Remove profile_diagnostic from diagnostic settings and increase test coverage of \_task.py ([Pull request #1404](https://github.com/ESMValGroup/ESMValCore/pull/1404)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add output2 to the product extra facet of CMIP5 data ([Pull request #1514](https://github.com/ESMValGroup/ESMValCore/pull/1514)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Speed up ESGF search ([Pull request #1512](https://github.com/ESMValGroup/ESMValCore/pull/1512)) by [@bouweandela](https://github.com/bouweandela) ## v2.4.0 ### Highlights - ESMValCore now has the ability to automatically download missing data from ESGF. For details, see [Data Retrieval](quickstart/find_data.html.md#data-retrieval). - ESMValCore now also can resume an earlier run. This is useful to reuse expensive preprocessor results. For details, see [Running](quickstart/run.html.md#running). This release includes ### Bug fixes - Crop on the ID-selected region(s) and not on the whole shapefile ([Pull request #1151](https://github.com/ESMValGroup/ESMValCore/pull/1151)) by [@stefsmeets](https://github.com/stefsmeets) - Add ‘comment’ to list of removed attributes ([Pull request #1244](https://github.com/ESMValGroup/ESMValCore/pull/1244)) by [@Peter9192](https://github.com/Peter9192) - Speed up multimodel statistics and fix bug in peak computation ([Pull request #1301](https://github.com/ESMValGroup/ESMValCore/pull/1301)) by [@bouweandela](https://github.com/bouweandela) - No longer make plots of provenance ([Pull request #1307](https://github.com/ESMValGroup/ESMValCore/pull/1307)) by [@bouweandela](https://github.com/bouweandela) - No longer embed provenance in output files ([Pull request #1306](https://github.com/ESMValGroup/ESMValCore/pull/1306)) by [@bouweandela](https://github.com/bouweandela) - Removed automatic addition of areacello to obs4mips datasets ([Pull request #1316](https://github.com/ESMValGroup/ESMValCore/pull/1316)) by [@schlunma](https://github.com/schlunma) - Pin docutils <0.17 to fix bullet lists on readthedocs ([Pull request #1320](https://github.com/ESMValGroup/ESMValCore/pull/1320)) by [@zklaus](https://github.com/zklaus) - Fix obs4MIPs capitalization ([Pull request #1328](https://github.com/ESMValGroup/ESMValCore/pull/1328)) by [@bouweandela](https://github.com/bouweandela) - Fix Python 3.7 tests ([Pull request #1330](https://github.com/ESMValGroup/ESMValCore/pull/1330)) by [@bouweandela](https://github.com/bouweandela) - Handle fx variables in extract_levels and some time operations ([Pull request #1269](https://github.com/ESMValGroup/ESMValCore/pull/1269)) by [@sloosvel](https://github.com/sloosvel) - Refactored mask regridding for irregular grids (fixes #772) ([Pull request #865](https://github.com/ESMValGroup/ESMValCore/pull/865)) by [@zklaus](https://github.com/zklaus) - Fix da.broadcast_to call when the fx cube has different shape than target data cube ([Pull request #1350](https://github.com/ESMValGroup/ESMValCore/pull/1350)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add tests for \_aggregate_time_fx ([Pull request #1354](https://github.com/ESMValGroup/ESMValCore/pull/1354)) by [@sloosvel](https://github.com/sloosvel) - Fix extra facets ([Pull request #1360](https://github.com/ESMValGroup/ESMValCore/pull/1360)) by [@bouweandela](https://github.com/bouweandela) - Pin pip!=21.3 to avoid pypa/pip#10573 with editable installs ([Pull request #1359](https://github.com/ESMValGroup/ESMValCore/pull/1359)) by [@zklaus](https://github.com/zklaus) - Add a custom date2num function to deal with changes in cftime ([Pull request #1373](https://github.com/ESMValGroup/ESMValCore/pull/1373)) by [@zklaus](https://github.com/zklaus) - Removed custom version of AtmosphereSigmaFactory ([Pull request #1382](https://github.com/ESMValGroup/ESMValCore/pull/1382)) by [@schlunma](https://github.com/schlunma) ### Deprecations - Remove write_netcdf and write_plots from config-user.yml ([Pull request #1300](https://github.com/ESMValGroup/ESMValCore/pull/1300)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Add link to plot directory in index.html ([Pull request #1256](https://github.com/ESMValGroup/ESMValCore/pull/1256)) by [@stefsmeets](https://github.com/stefsmeets) - Work around issue with yapf not following PEP8 ([Pull request #1277](https://github.com/ESMValGroup/ESMValCore/pull/1277)) by [@bouweandela](https://github.com/bouweandela) - Update the core development team ([Pull request #1278](https://github.com/ESMValGroup/ESMValCore/pull/1278)) by [@bouweandela](https://github.com/bouweandela) - Update the documentation of the provenance interface ([Pull request #1305](https://github.com/ESMValGroup/ESMValCore/pull/1305)) by [@bouweandela](https://github.com/bouweandela) - Update version number to first release candidate 2.4.0rc1 ([Pull request #1363](https://github.com/ESMValGroup/ESMValCore/pull/1363)) by [@zklaus](https://github.com/zklaus) - Update to new ESMValTool logo ([Pull request #1374](https://github.com/ESMValGroup/ESMValCore/pull/1374)) by [@zklaus](https://github.com/zklaus) - Update version number for third release candidate 2.4.0rc3 ([Pull request #1384](https://github.com/ESMValGroup/ESMValCore/pull/1384)) by [@zklaus](https://github.com/zklaus) - Update changelog for 2.4.0rc3 ([Pull request #1385](https://github.com/ESMValGroup/ESMValCore/pull/1385)) by [@zklaus](https://github.com/zklaus) - Update version number to final 2.4.0 release ([Pull request #1389](https://github.com/ESMValGroup/ESMValCore/pull/1389)) by [@zklaus](https://github.com/zklaus) - Update changelog for 2.4.0 ([Pull request #1366](https://github.com/ESMValGroup/ESMValCore/pull/1366)) by [@zklaus](https://github.com/zklaus) ### Fixes for datasets - Add fix for differing latitude coordinate between historical and ssp585 in MPI-ESM1-2-HR r2i1p1f1 ([Pull request #1292](https://github.com/ESMValGroup/ESMValCore/pull/1292)) by [@bouweandela](https://github.com/bouweandela) - Add fixes for time and latitude coordinate of EC-Earth3 r3i1p1f1 ([Pull request #1290](https://github.com/ESMValGroup/ESMValCore/pull/1290)) by [@bouweandela](https://github.com/bouweandela) - Apply latitude fix to all CCSM4 variables ([Pull request #1295](https://github.com/ESMValGroup/ESMValCore/pull/1295)) by [@bouweandela](https://github.com/bouweandela) - Fix lat and lon bounds for FGOALS-g3 mrsos ([Pull request #1289](https://github.com/ESMValGroup/ESMValCore/pull/1289)) by [@thomascrocker](https://github.com/thomascrocker) - Add grid fix for tos in fgoals-f3-l ([Pull request #1326](https://github.com/ESMValGroup/ESMValCore/pull/1326)) by [@sloosvel](https://github.com/sloosvel) - Add fix for CIESM pr ([Pull request #1344](https://github.com/ESMValGroup/ESMValCore/pull/1344)) by [@bouweandela](https://github.com/bouweandela) - Fix DRS for IPSLCM : split attribute ‘freq’ into : ‘out’ and ‘freq’ ([Pull request #1304](https://github.com/ESMValGroup/ESMValCore/pull/1304)) by [@senesis](https://github.com/senesis) ### CMOR standard - Remove history attribute from coords ([Pull request #1276](https://github.com/ESMValGroup/ESMValCore/pull/1276)) by [@jvegreg](https://github.com/jvegreg) - Increased flexibility of CMOR checks for datasets with generic alevel coordinates ([Pull request #1032](https://github.com/ESMValGroup/ESMValCore/pull/1032)) by [@schlunma](https://github.com/schlunma) - Automatically fix small deviations in vertical levels ([Pull request #1177](https://github.com/ESMValGroup/ESMValCore/pull/1177)) by [@bouweandela](https://github.com/bouweandela) - Adding standard names to the custom tables of the rlns and rsns variables ([Pull request #1386](https://github.com/ESMValGroup/ESMValCore/pull/1386)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### Preprocessor - Implemented fully lazy climate_statistics ([Pull request #1194](https://github.com/ESMValGroup/ESMValCore/pull/1194)) by [@schlunma](https://github.com/schlunma) - Run the multimodel statistics preprocessor last ([Pull request #1299](https://github.com/ESMValGroup/ESMValCore/pull/1299)) by [@bouweandela](https://github.com/bouweandela) ### Automatic testing - Improving test coverage for \_task.py ([Pull request #514](https://github.com/ESMValGroup/ESMValCore/pull/514)) by [@valeriupredoi](https://github.com/valeriupredoi) - Upload coverage to codecov ([Pull request #1190](https://github.com/ESMValGroup/ESMValCore/pull/1190)) by [@bouweandela](https://github.com/bouweandela) - Improve codecov status checks ([Pull request #1195](https://github.com/ESMValGroup/ESMValCore/pull/1195)) by [@bouweandela](https://github.com/bouweandela) - Fix curl install in CircleCI ([Pull request #1228](https://github.com/ESMValGroup/ESMValCore/pull/1228)) by [@jvegreg](https://github.com/jvegreg) - Drop support for Python 3.6 ([Pull request #1200](https://github.com/ESMValGroup/ESMValCore/pull/1200)) by [@valeriupredoi](https://github.com/valeriupredoi) - Allow more recent version of scipy ([Pull request #1182](https://github.com/ESMValGroup/ESMValCore/pull/1182)) by [@schlunma](https://github.com/schlunma) - Speed up conda build conda_build Circle test by using mamba solver via boa (and use it for Github Actions test too) ([Pull request #1243](https://github.com/ESMValGroup/ESMValCore/pull/1243)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix numpy deprecation warnings ([Pull request #1274](https://github.com/ESMValGroup/ESMValCore/pull/1274)) by [@bouweandela](https://github.com/bouweandela) - Unpin upper bound for iris (previously was at <3.0.4) ([Pull request #1275](https://github.com/ESMValGroup/ESMValCore/pull/1275)) by [@valeriupredoi](https://github.com/valeriupredoi) - Modernize conda_install test on Circle CI by installing from conda-forge with Python 3.9 and change install instructions in documentation ([Pull request #1280](https://github.com/ESMValGroup/ESMValCore/pull/1280)) by [@valeriupredoi](https://github.com/valeriupredoi) - Run a nightly Github Actions workflow to monitor tests memory per test (configurable for other metrics too) ([Pull request #1284](https://github.com/ESMValGroup/ESMValCore/pull/1284)) by [@valeriupredoi](https://github.com/valeriupredoi) - Speed up tests of tasks ([Pull request #1302](https://github.com/ESMValGroup/ESMValCore/pull/1302)) by [@bouweandela](https://github.com/bouweandela) - Fix upper case to lower case variables and functions for flake compliance in tests/unit/preprocessor/_regrid/test_extract_levels.py ([Pull request #1347](https://github.com/ESMValGroup/ESMValCore/pull/1347)) by [@valeriupredoi](https://github.com/valeriupredoi) - Cleaned up a bit Github Actions workflows ([Pull request #1345](https://github.com/ESMValGroup/ESMValCore/pull/1345)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update circleci jobs: renaming tests to more descriptive names and removing conda build test ([Pull request #1351](https://github.com/ESMValGroup/ESMValCore/pull/1351)) by [@zklaus](https://github.com/zklaus) - Pin iris to latest >=3.1.0 ([Pull request #1341](https://github.com/ESMValGroup/ESMValCore/pull/1341)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Installation - Pin esmpy to anything but 8.1.0 since that particular one changes the CPU affinity ([Pull request #1310](https://github.com/ESMValGroup/ESMValCore/pull/1310)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - Add a more friendly and useful message when using default config file ([Pull request #1233](https://github.com/ESMValGroup/ESMValCore/pull/1233)) by [@valeriupredoi](https://github.com/valeriupredoi) - Replace os.walk by glob.glob in data finder (only look for data in the specified locations) ([Pull request #1261](https://github.com/ESMValGroup/ESMValCore/pull/1261)) by [@bouweandela](https://github.com/bouweandela) - Machine-specific directories for auxiliary data in the config-user.yml file ([Pull request #1268](https://github.com/ESMValGroup/ESMValCore/pull/1268)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Add an option to download missing data from ESGF ([Pull request #1217](https://github.com/ESMValGroup/ESMValCore/pull/1217)) by [@bouweandela](https://github.com/bouweandela) - Speed up provenance recording ([Pull request #1327](https://github.com/ESMValGroup/ESMValCore/pull/1327)) by [@bouweandela](https://github.com/bouweandela) - Improve results web page ([Pull request #1332](https://github.com/ESMValGroup/ESMValCore/pull/1332)) by [@bouweandela](https://github.com/bouweandela) - Move institutes from config-developer.yml to default extra facets config and add wildcard support for extra facets ([Pull request #1259](https://github.com/ESMValGroup/ESMValCore/pull/1259)) by [@bouweandela](https://github.com/bouweandela) - Add support for reusing preprocessor output from previous runs ([Pull request #1321](https://github.com/ESMValGroup/ESMValCore/pull/1321)) by [@bouweandela](https://github.com/bouweandela) - Log fewer messages to screen and hide stack trace for known recipe errors ([Pull request #1296](https://github.com/ESMValGroup/ESMValCore/pull/1296)) by [@bouweandela](https://github.com/bouweandela) - Log ESMValCore and ESMValTool versions when running ([Pull request #1263](https://github.com/ESMValGroup/ESMValCore/pull/1263)) by [@jvegreg](https://github.com/jvegreg) - Add “grid” as a tag to the output file template for CMIP6 ([Pull request #1356](https://github.com/ESMValGroup/ESMValCore/pull/1356)) by [@zklaus](https://github.com/zklaus) - Implemented ICON project to read native ICON model output ([Pull request #1079](https://github.com/ESMValGroup/ESMValCore/pull/1079)) by [@bsolino](https://github.com/bsolino) ## v2.3.1 This release includes ### Bug fixes - Update config-user.yml template with correct drs entries for CEDA-JASMIN ([Pull request #1184](https://github.com/ESMValGroup/ESMValCore/pull/1184)) by [@valeriupredoi](https://github.com/valeriupredoi) - Enhancing MIROC5 fix for hfls and evspsbl ([Pull request #1192](https://github.com/ESMValGroup/ESMValCore/pull/1192)) by [@katjaweigel](https://github.com/katjaweigel) - Fix alignment of daily data with inconsistent calendars in multimodel statistics ([Pull request #1212](https://github.com/ESMValGroup/ESMValCore/pull/1212)) by [@Peter9192](https://github.com/Peter9192) - Pin cf-units, remove github actions test for Python 3.6 and fix test_access1_0 and test_access1_3 to use cf-units for comparisons ([Pull request #1197](https://github.com/ESMValGroup/ESMValCore/pull/1197)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fixed search for fx files when no `mip` is given ([Pull request #1216](https://github.com/ESMValGroup/ESMValCore/pull/1216)) by [@schlunma](https://github.com/schlunma) - Make sure climate statistics always returns original dtype ([Pull request #1237](https://github.com/ESMValGroup/ESMValCore/pull/1237)) by [@zklaus](https://github.com/zklaus) - Bugfix for regional regridding when non-integer range is passed ([Pull request #1231](https://github.com/ESMValGroup/ESMValCore/pull/1231)) by [@stefsmeets](https://github.com/stefsmeets) - Make sure area_statistics preprocessor always returns original dtype ([Pull request #1239](https://github.com/ESMValGroup/ESMValCore/pull/1239)) by [@zklaus](https://github.com/zklaus) - Add “.” (dot) as allowed separation character for the time range group ([Pull request #1248](https://github.com/ESMValGroup/ESMValCore/pull/1248)) by [@zklaus](https://github.com/zklaus) ### Documentation - Add a link to the instructions to use pre-installed versions on HPC clusters ([Pull request #1186](https://github.com/ESMValGroup/ESMValCore/pull/1186)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Bugfix release: set version to 2.3.1 ([Pull request #1253](https://github.com/ESMValGroup/ESMValCore/pull/1253)) by [@zklaus](https://github.com/zklaus) ### Fixes for datasets - Set circular attribute in MCM-UA-1-0 fix ([Pull request #1178](https://github.com/ESMValGroup/ESMValCore/pull/1178)) by [@sloosvel](https://github.com/sloosvel) - Fixed time coordinate of MIROC-ESM ([Pull request #1188](https://github.com/ESMValGroup/ESMValCore/pull/1188)) by [@schlunma](https://github.com/schlunma) ### Preprocessor - Filter warnings about collapsing multi-model dimension in multimodel statistics preprocessor function ([Pull request #1215](https://github.com/ESMValGroup/ESMValCore/pull/1215)) by [@bouweandela](https://github.com/bouweandela) - Remove fx variables before computing multimodel statistics ([Pull request #1220](https://github.com/ESMValGroup/ESMValCore/pull/1220)) by [@sloosvel](https://github.com/sloosvel) ### Installation - Pin lower bound for iris to 3.0.2 ([Pull request #1206](https://github.com/ESMValGroup/ESMValCore/pull/1206)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin iris<3.0.4 to ensure we still (sort of) support Python 3.6 ([Pull request #1252](https://github.com/ESMValGroup/ESMValCore/pull/1252)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - Add test to verify behaviour for scalar height coord for tas in multi-model ([Pull request #1209](https://github.com/ESMValGroup/ESMValCore/pull/1209)) by [@Peter9192](https://github.com/Peter9192) - Sort missing years in “No input data available for years” message ([Pull request #1225](https://github.com/ESMValGroup/ESMValCore/pull/1225)) by [@ledm](https://github.com/ledm) ## v2.3.0 This release includes ### Bug fixes - Extend preprocessor multi_model_statistics to handle data with “altitude” coordinate ([Pull request #1010](https://github.com/ESMValGroup/ESMValCore/pull/1010)) by [@axel-lauer](https://github.com/axel-lauer) - Remove scripts included with CMOR tables ([Pull request #1011](https://github.com/ESMValGroup/ESMValCore/pull/1011)) by [@bouweandela](https://github.com/bouweandela) - Avoid side effects in extract_season ([Pull request #1019](https://github.com/ESMValGroup/ESMValCore/pull/1019)) by [@jvegreg](https://github.com/jvegreg) - Use nearest scheme to avoid interpolation errors with masked data in regression test ([Pull request #1021](https://github.com/ESMValGroup/ESMValCore/pull/1021)) by [@stefsmeets](https://github.com/stefsmeets) - Move \_get_time_bounds from preprocessor._time to cmor.check to avoid circular import with cmor module ([Pull request #1037](https://github.com/ESMValGroup/ESMValCore/pull/1037)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix test that makes conda build fail ([Pull request #1046](https://github.com/ESMValGroup/ESMValCore/pull/1046)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix ‘positive’ attribute for rsns/rlns variables ([Pull request #1051](https://github.com/ESMValGroup/ESMValCore/pull/1051)) by [@lukasbrunner](https://github.com/lukasbrunner) - Added preprocessor mask_multimodel ([Pull request #767](https://github.com/ESMValGroup/ESMValCore/pull/767)) by [@schlunma](https://github.com/schlunma) - Fix bug when fixing bounds after fixing longitude values ([Pull request #1057](https://github.com/ESMValGroup/ESMValCore/pull/1057)) by [@sloosvel](https://github.com/sloosvel) - Run conda build parallel AND sequential tests ([Pull request #1065](https://github.com/ESMValGroup/ESMValCore/pull/1065)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add key to id_prop ([Pull request #1071](https://github.com/ESMValGroup/ESMValCore/pull/1071)) by [@lukasbrunner](https://github.com/lukasbrunner) - Fix bounds after reversing coordinate values ([Pull request #1061](https://github.com/ESMValGroup/ESMValCore/pull/1061)) by [@sloosvel](https://github.com/sloosvel) - Fixed –skip-nonexistent option ([Pull request #1093](https://github.com/ESMValGroup/ESMValCore/pull/1093)) by [@schlunma](https://github.com/schlunma) - Do not consider CMIP5 variable sit to be the same as sithick from CMIP6 ([Pull request #1033](https://github.com/ESMValGroup/ESMValCore/pull/1033)) by [@bouweandela](https://github.com/bouweandela) - Improve finding date range in filenames (enforces separators) ([Pull request #1145](https://github.com/ESMValGroup/ESMValCore/pull/1145)) by [@senesis](https://github.com/senesis) - Review fx handling ([Pull request #1147](https://github.com/ESMValGroup/ESMValCore/pull/1147)) by [@sloosvel](https://github.com/sloosvel) - Fix lru cache decorator with explicit call to method ([Pull request #1172](https://github.com/ESMValGroup/ESMValCore/pull/1172)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update \_volume.py ([Pull request #1174](https://github.com/ESMValGroup/ESMValCore/pull/1174)) by [@ledm](https://github.com/ledm) ### Deprecations ### Documentation - Final changelog for 2.3.0 ([Pull request #1163](https://github.com/ESMValGroup/ESMValCore/pull/1163)) by [@zklaus](https://github.com/zklaus) - Set version to 2.3.0 ([Pull request #1162](https://github.com/ESMValGroup/ESMValCore/pull/1162)) by [@zklaus](https://github.com/zklaus) - Fix documentation build ([Pull request #1006](https://github.com/ESMValGroup/ESMValCore/pull/1006)) by [@bouweandela](https://github.com/bouweandela) - Add labels required for linking from ESMValTool docs ([Pull request #1038](https://github.com/ESMValGroup/ESMValCore/pull/1038)) by [@bouweandela](https://github.com/bouweandela) - Update contribution guidelines ([Pull request #1047](https://github.com/ESMValGroup/ESMValCore/pull/1047)) by [@bouweandela](https://github.com/bouweandela) - Fix basestring references in documentation ([Pull request #1106](https://github.com/ESMValGroup/ESMValCore/pull/1106)) by [@jvegreg](https://github.com/jvegreg) - Updated references master to main ([Pull request #1132](https://github.com/ESMValGroup/ESMValCore/pull/1132)) by [@axel-lauer](https://github.com/axel-lauer) - Add instructions how to use the central installation at DKRZ-Mistral ([Pull request #1155](https://github.com/ESMValGroup/ESMValCore/pull/1155)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### Fixes for datasets - Added fixes for various CMIP5 datasets, variable cl (3-dim cloud fraction) ([Pull request #1017](https://github.com/ESMValGroup/ESMValCore/pull/1017)) by [@axel-lauer](https://github.com/axel-lauer) - Added fixes for hybrid level coordinates of CESM2 models ([Pull request #882](https://github.com/ESMValGroup/ESMValCore/pull/882)) by [@schlunma](https://github.com/schlunma) - Extending LWP fix for CMIP6 models ([Pull request #1049](https://github.com/ESMValGroup/ESMValCore/pull/1049)) by [@axel-lauer](https://github.com/axel-lauer) - Add fixes for the net & up radiation variables from ERA5 ([Pull request #1052](https://github.com/ESMValGroup/ESMValCore/pull/1052)) by [@lukasbrunner](https://github.com/lukasbrunner) - Add derived variable rsus ([Pull request #1053](https://github.com/ESMValGroup/ESMValCore/pull/1053)) by [@lukasbrunner](https://github.com/lukasbrunner) - Supported mip-level fixes ([Pull request #1095](https://github.com/ESMValGroup/ESMValCore/pull/1095)) by [@schlunma](https://github.com/schlunma) - Fix erroneous use of grid_latitude and grid_longitude and cleaned ocean grid fixes ([Pull request #1092](https://github.com/ESMValGroup/ESMValCore/pull/1092)) by [@schlunma](https://github.com/schlunma) - Fix for pr of miroc5 ([Pull request #1110](https://github.com/ESMValGroup/ESMValCore/pull/1110)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Ocean depth fix for cnrm_esm2_1, gfdl_esm4, ipsl_cm6a_lr datasets + mcm_ua_1_0 ([Pull request #1098](https://github.com/ESMValGroup/ESMValCore/pull/1098)) by [@tomaslovato](https://github.com/tomaslovato) - Fix for uas variable of the MCM_UA_1_0 dataset ([Pull request #1102](https://github.com/ESMValGroup/ESMValCore/pull/1102)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Fixes for sos and siconc of BCC models ([Pull request #1090](https://github.com/ESMValGroup/ESMValCore/pull/1090)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Run fgco2 fix for all CESM2 models ([Pull request #1108](https://github.com/ESMValGroup/ESMValCore/pull/1108)) by [@LisaBock](https://github.com/LisaBock) - Fixes for the siconc variable of CMIP6 models ([Pull request #1105](https://github.com/ESMValGroup/ESMValCore/pull/1105)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Fix wrong sign for land surface flux ([Pull request #1113](https://github.com/ESMValGroup/ESMValCore/pull/1113)) by [@LisaBock](https://github.com/LisaBock) - Fix for pr of EC_EARTH ([Pull request #1116](https://github.com/ESMValGroup/ESMValCore/pull/1116)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### CMOR standard - Format cmor related files ([Pull request #976](https://github.com/ESMValGroup/ESMValCore/pull/976)) by [@jvegreg](https://github.com/jvegreg) - Check presence of time bounds and guess them if needed ([Pull request #849](https://github.com/ESMValGroup/ESMValCore/pull/849)) by [@sloosvel](https://github.com/sloosvel) - Add custom variable “tasaga” ([Pull request #1118](https://github.com/ESMValGroup/ESMValCore/pull/1118)) by [@LisaBock](https://github.com/LisaBock) - Find files for CMIP6 DCPP startdates ([Pull request #771](https://github.com/ESMValGroup/ESMValCore/pull/771)) by [@sloosvel](https://github.com/sloosvel) ### Preprocessor - Update tests for multimodel statistics preprocessor ([Pull request #1023](https://github.com/ESMValGroup/ESMValCore/pull/1023)) by [@stefsmeets](https://github.com/stefsmeets) - Raise in extract_season and extract_month if result is None ([Pull request #1041](https://github.com/ESMValGroup/ESMValCore/pull/1041)) by [@jvegreg](https://github.com/jvegreg) - Allow selection of shapes in extract_shape ([Pull request #764](https://github.com/ESMValGroup/ESMValCore/pull/764)) by [@jvegreg](https://github.com/jvegreg) - Add option for regional regridding to regrid preprocessor ([Pull request #1034](https://github.com/ESMValGroup/ESMValCore/pull/1034)) by [@stefsmeets](https://github.com/stefsmeets) - Load fx variables as cube cell measures / ancillary variables ([Pull request #999](https://github.com/ESMValGroup/ESMValCore/pull/999)) by [@sloosvel](https://github.com/sloosvel) - Check horizontal grid before regridding ([Pull request #507](https://github.com/ESMValGroup/ESMValCore/pull/507)) by [@BenMGeo](https://github.com/BenMGeo) - Clip irregular grids ([Pull request #245](https://github.com/ESMValGroup/ESMValCore/pull/245)) by [@bouweandela](https://github.com/bouweandela) - Use native iris functions in multi-model statistics ([Pull request #1150](https://github.com/ESMValGroup/ESMValCore/pull/1150)) by [@Peter9192](https://github.com/Peter9192) ### Notebook API (experimental) ### Automatic testing - Report coverage for tests that run on any pull request ([Pull request #994](https://github.com/ESMValGroup/ESMValCore/pull/994)) by [@bouweandela](https://github.com/bouweandela) - Install ESMValTool sample data from PyPI ([Pull request #998](https://github.com/ESMValGroup/ESMValCore/pull/998)) by [@jvegreg](https://github.com/jvegreg) - Fix tests for multi-processing with spawn method (i.e. macOSX with Python>3.8) ([Pull request #1003](https://github.com/ESMValGroup/ESMValCore/pull/1003)) by [@bvreede](https://github.com/bvreede) - Switch to running the Github Action test workflow every 3 hours in single thread mode to observe if Segmentation Faults occur ([Pull request #1022](https://github.com/ESMValGroup/ESMValCore/pull/1022)) by [@valeriupredoi](https://github.com/valeriupredoi) - Revert to original Github Actions test workflow removing the 3-hourly test run with -n 1 ([Pull request #1025](https://github.com/ESMValGroup/ESMValCore/pull/1025)) by [@valeriupredoi](https://github.com/valeriupredoi) - Avoid stale cache for multimodel statistics regression tests ([Pull request #1030](https://github.com/ESMValGroup/ESMValCore/pull/1030)) by [@bouweandela](https://github.com/bouweandela) - Add newer Python versions in OSX to Github Actions ([Pull request #1035](https://github.com/ESMValGroup/ESMValCore/pull/1035)) by [@bvreede](https://github.com/bvreede) - Add tests for type annotations with mypy ([Pull request #1042](https://github.com/ESMValGroup/ESMValCore/pull/1042)) by [@stefsmeets](https://github.com/stefsmeets) - Run problematic cmor tests sequentially to avoid segmentation faults on CircleCI ([Pull request #1064](https://github.com/ESMValGroup/ESMValCore/pull/1064)) by [@valeriupredoi](https://github.com/valeriupredoi) - Test installation of esmvalcore from conda-forge ([Pull request #1075](https://github.com/ESMValGroup/ESMValCore/pull/1075)) by [@valeriupredoi](https://github.com/valeriupredoi) - Added additional test cases for integration tests of data_finder.py ([Pull request #1087](https://github.com/ESMValGroup/ESMValCore/pull/1087)) by [@schlunma](https://github.com/schlunma) - Pin cf-units and fix tests (cf-units>=2.1.5) ([Pull request #1140](https://github.com/ESMValGroup/ESMValCore/pull/1140)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix failing CircleCI tests ([Pull request #1167](https://github.com/ESMValGroup/ESMValCore/pull/1167)) by [@bouweandela](https://github.com/bouweandela) - Fix test failing due to fx files chosen differently on different OS’s ([Pull request #1169](https://github.com/ESMValGroup/ESMValCore/pull/1169)) by [@valeriupredoi](https://github.com/valeriupredoi) - Compare datetimes instead of strings in \_fixes/cmip5/test_access1_X.py ([Pull request #1173](https://github.com/ESMValGroup/ESMValCore/pull/1173)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin Python to 3.9 in environment.yml on CircleCI and skip mypy tests in conda build ([Pull request #1176](https://github.com/ESMValGroup/ESMValCore/pull/1176)) by [@bouweandela](https://github.com/bouweandela) ### Installation - Update yamale to version 3 ([Pull request #1059](https://github.com/ESMValGroup/ESMValCore/pull/1059)) by [@zklaus](https://github.com/zklaus) ### Improvements - Refactor diagnostics / tags management ([Pull request #939](https://github.com/ESMValGroup/ESMValCore/pull/939)) by [@stefsmeets](https://github.com/stefsmeets) - Support multiple paths in input_dir ([Pull request #1000](https://github.com/ESMValGroup/ESMValCore/pull/1000)) by [@jvegreg](https://github.com/jvegreg) - Generate HTML report with recipe output ([Pull request #991](https://github.com/ESMValGroup/ESMValCore/pull/991)) by [@stefsmeets](https://github.com/stefsmeets) - Add timeout to requests.get in \_citation.py ([Pull request #1091](https://github.com/ESMValGroup/ESMValCore/pull/1091)) by [@SarahAlidoost](https://github.com/SarahAlidoost) - Add SYNDA drs for CMIP5 and CMIP6 (closes #582) ([Pull request #583](https://github.com/ESMValGroup/ESMValCore/pull/583)) by [@zklaus](https://github.com/zklaus) - Add basic support for variable mappings ([Pull request #1124](https://github.com/ESMValGroup/ESMValCore/pull/1124)) by [@zklaus](https://github.com/zklaus) - Handle IPSL-CM6 ([Pull request #1153](https://github.com/ESMValGroup/ESMValCore/pull/1153)) by [@senesis](https://github.com/senesis) ## v2.2.0 ### Highlights ESMValCore is now using the recently released [Iris 3](https://scitools-iris.readthedocs.io/en/latest/whatsnew/3.0.html). We acknowledge that this change may impact your work, as Iris 3 introduces several changes that are not backward-compatible, but we think that moving forward is the best decision for the tool in the long term. This release is also the first one including support for downloading CMIP6 data using Synda and we have also started supporting Python 3.9. Give it a try! This release includes ### Bug fixes - Fix path settings for DKRZ/Mistral ([Pull request #852](https://github.com/ESMValGroup/ESMValCore/pull/852)) by [@bouweandela](https://github.com/bouweandela) - Change logic for calling the diagnostic script to avoid problems with scripts where the executable bit is accidentally set ([Pull request #877](https://github.com/ESMValGroup/ESMValCore/pull/877)) by [@bouweandela](https://github.com/bouweandela) - Fix overwriting in generic level check ([Pull request #886](https://github.com/ESMValGroup/ESMValCore/pull/886)) by [@sloosvel](https://github.com/sloosvel) - Add double quotes to script args in rerun screen message when using vprof profiling ([Pull request #897](https://github.com/ESMValGroup/ESMValCore/pull/897)) by [@valeriupredoi](https://github.com/valeriupredoi) - Simplify time handling in multi-model statistics preprocessor ([Pull request #685](https://github.com/ESMValGroup/ESMValCore/pull/685)) by [@Peter9192](https://github.com/Peter9192) - Fix links to Iris documentation ([Pull request #966](https://github.com/ESMValGroup/ESMValCore/pull/966)) by [@jvegreg](https://github.com/jvegreg) - Bugfix: Fix units for MSWEP data ([Pull request #986](https://github.com/ESMValGroup/ESMValCore/pull/986)) by [@stefsmeets](https://github.com/stefsmeets) ### Deprecations - Deprecate defining write_plots and write_netcdf in config-user file ([Pull request #808](https://github.com/ESMValGroup/ESMValCore/pull/808)) by [@bouweandela](https://github.com/bouweandela) ### Documentation - Fix numbering of steps in release instructions ([Pull request #838](https://github.com/ESMValGroup/ESMValCore/pull/838)) by [@bouweandela](https://github.com/bouweandela) - Add labels to changelogs of individual versions for easy reference ([Pull request #899](https://github.com/ESMValGroup/ESMValCore/pull/899)) by [@zklaus](https://github.com/zklaus) - Make CircleCI badge specific to main branch ([Pull request #902](https://github.com/ESMValGroup/ESMValCore/pull/902)) by [@bouweandela](https://github.com/bouweandela) - Fix docker build badge url ([Pull request #906](https://github.com/ESMValGroup/ESMValCore/pull/906)) by [@stefsmeets](https://github.com/stefsmeets) - Update github PR template ([Pull request #909](https://github.com/ESMValGroup/ESMValCore/pull/909)) by [@stefsmeets](https://github.com/stefsmeets) - Refer to ESMValTool GitHub discussions page in the error message ([Pull request #900](https://github.com/ESMValGroup/ESMValCore/pull/900)) by [@bouweandela](https://github.com/bouweandela) - Support automatically closing issues ([Pull request #922](https://github.com/ESMValGroup/ESMValCore/pull/922)) by [@bouweandela](https://github.com/bouweandela) - Fix checkboxes in PR template ([Pull request #931](https://github.com/ESMValGroup/ESMValCore/pull/931)) by [@stefsmeets](https://github.com/stefsmeets) - Change in config-user defaults and documentation with new location for esmeval OBS data on JASMIN ([Pull request #958](https://github.com/ESMValGroup/ESMValCore/pull/958)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update Core Team info ([Pull request #942](https://github.com/ESMValGroup/ESMValCore/pull/942)) by [@axel-lauer](https://github.com/axel-lauer) - Update iris documentation URL for sphinx ([Pull request #964](https://github.com/ESMValGroup/ESMValCore/pull/964)) by [@bouweandela](https://github.com/bouweandela) - Set version to 2.2.0 ([Pull request #977](https://github.com/ESMValGroup/ESMValCore/pull/977)) by [@jvegreg](https://github.com/jvegreg) - Add first draft of v2.2.0 changelog ([Pull request #983](https://github.com/ESMValGroup/ESMValCore/pull/983)) by [@jvegreg](https://github.com/jvegreg) - Add checkbox in PR template to assign labels ([Pull request #985](https://github.com/ESMValGroup/ESMValCore/pull/985)) by [@jvegreg](https://github.com/jvegreg) - Update install.rst ([Pull request #848](https://github.com/ESMValGroup/ESMValCore/pull/848)) by [@bascrezee](https://github.com/bascrezee) - Change the order of the publication steps ([Pull request #984](https://github.com/ESMValGroup/ESMValCore/pull/984)) by [@jvegreg](https://github.com/jvegreg) - Add instructions how to use esmvaltool from HPC central installations ([Pull request #841](https://github.com/ESMValGroup/ESMValCore/pull/841)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Fixes for datasets - Fixing unit for derived variable rsnstcsnorm to prevent overcorrection2 ([Pull request #846](https://github.com/ESMValGroup/ESMValCore/pull/846)) by [@katjaweigel](https://github.com/katjaweigel) - Cmip6 fix awi cm 1 1 mr ([Pull request #822](https://github.com/ESMValGroup/ESMValCore/pull/822)) by [@mwjury](https://github.com/mwjury) - Cmip6 fix ec earth3 veg ([Pull request #836](https://github.com/ESMValGroup/ESMValCore/pull/836)) by [@mwjury](https://github.com/mwjury) - Changed latitude longitude fix from Tas to AllVars. ([Pull request #916](https://github.com/ESMValGroup/ESMValCore/pull/916)) by [@katjaweigel](https://github.com/katjaweigel) - Fix for precipitation (pr) to use ERA5-Land cmorizer ([Pull request #879](https://github.com/ESMValGroup/ESMValCore/pull/879)) by [@katjaweigel](https://github.com/katjaweigel) - Cmip6 fix ec earth3 ([Pull request #837](https://github.com/ESMValGroup/ESMValCore/pull/837)) by [@mwjury](https://github.com/mwjury) - Cmip6_fix_fgoals_f3_l_Amon_time_bnds ([Pull request #831](https://github.com/ESMValGroup/ESMValCore/pull/831)) by [@mwjury](https://github.com/mwjury) - Fix for FGOALS-f3-L sftlf ([Pull request #667](https://github.com/ESMValGroup/ESMValCore/pull/667)) by [@mwjury](https://github.com/mwjury) - Improve ACCESS-CM2 and ACCESS-ESM1-5 fixes and add CIESM and CESM2-WACCM-FV2 fixes for cl, clw and cli ([Pull request #635](https://github.com/ESMValGroup/ESMValCore/pull/635)) by [@axel-lauer](https://github.com/axel-lauer) - Add fixes for cl, cli, clw and tas for several CMIP6 models ([Pull request #955](https://github.com/ESMValGroup/ESMValCore/pull/955)) by [@schlunma](https://github.com/schlunma) - Dataset fixes for MSWEP ([Pull request #969](https://github.com/ESMValGroup/ESMValCore/pull/969)) by [@stefsmeets](https://github.com/stefsmeets) - Dataset fixes for: ACCESS-ESM1-5, CanESM5, CanESM5 for carbon cycle ([Pull request #947](https://github.com/ESMValGroup/ESMValCore/pull/947)) by [@bettina-gier](https://github.com/bettina-gier) - Fixes for KIOST-ESM (CMIP6) ([Pull request #904](https://github.com/ESMValGroup/ESMValCore/pull/904)) by [@remi-kazeroni](https://github.com/remi-kazeroni) - Fixes for AWI-ESM-1-1-LR (CMIP6, piControl) ([Pull request #911](https://github.com/ESMValGroup/ESMValCore/pull/911)) by [@remi-kazeroni](https://github.com/remi-kazeroni) ### CMOR standard - CMOR check generic level coordinates in CMIP6 ([Pull request #598](https://github.com/ESMValGroup/ESMValCore/pull/598)) by [@sloosvel](https://github.com/sloosvel) - Update CMIP6 tables to 6.9.33 ([Pull request #919](https://github.com/ESMValGroup/ESMValCore/pull/919)) by [@jvegreg](https://github.com/jvegreg) - Adding custom variables for tas uncertainty ([Pull request #924](https://github.com/ESMValGroup/ESMValCore/pull/924)) by [@LisaBock](https://github.com/LisaBock) - Remove monotonicity coordinate check for unstructured grids ([Pull request #965](https://github.com/ESMValGroup/ESMValCore/pull/965)) by [@jvegreg](https://github.com/jvegreg) ### Preprocessor - Added clip_start_end_year preprocessor ([Pull request #796](https://github.com/ESMValGroup/ESMValCore/pull/796)) by [@schlunma](https://github.com/schlunma) - Add support for downloading CMIP6 data with Synda ([Pull request #699](https://github.com/ESMValGroup/ESMValCore/pull/699)) by [@bouweandela](https://github.com/bouweandela) - Add multimodel tests using real data ([Pull request #856](https://github.com/ESMValGroup/ESMValCore/pull/856)) by [@stefsmeets](https://github.com/stefsmeets) - Add plev/altitude conversion to extract_levels ([Pull request #892](https://github.com/ESMValGroup/ESMValCore/pull/892)) by [@axel-lauer](https://github.com/axel-lauer) - Add possibility of custom season extraction. ([Pull request #247](https://github.com/ESMValGroup/ESMValCore/pull/247)) by [@mwjury](https://github.com/mwjury) - Adding the ability to derive xch4 ([Pull request #783](https://github.com/ESMValGroup/ESMValCore/pull/783)) by [@hb326](https://github.com/hb326) - Add preprocessor function to resample time and compute x-hourly statistics ([Pull request #696](https://github.com/ESMValGroup/ESMValCore/pull/696)) by [@jvegreg](https://github.com/jvegreg) - Fix duplication in preprocessors DEFAULT_ORDER introduced in #696 ([Pull request #973](https://github.com/ESMValGroup/ESMValCore/pull/973)) by [@jvegreg](https://github.com/jvegreg) - Use consistent precision in multi-model statistics calculation and update reference data for tests ([Pull request #941](https://github.com/ESMValGroup/ESMValCore/pull/941)) by [@Peter9192](https://github.com/Peter9192) - Refactor multi-model statistics code to facilitate ensemble stats and lazy evaluation ([Pull request #949](https://github.com/ESMValGroup/ESMValCore/pull/949)) by [@Peter9192](https://github.com/Peter9192) - Add option to exclude input cubes in output of multimodel statistics to solve an issue introduced by #949 ([Pull request #978](https://github.com/ESMValGroup/ESMValCore/pull/978)) by [@Peter9192](https://github.com/Peter9192) ### Automatic testing - Pin cftime>=1.3.0 to have newer string formatting in and fix two tests ([Pull request #878](https://github.com/ESMValGroup/ESMValCore/pull/878)) by [@valeriupredoi](https://github.com/valeriupredoi) - Switched miniconda conda setup hooks for Github Actions workflows ([Pull request #873](https://github.com/ESMValGroup/ESMValCore/pull/873)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add test for latest version resolver ([Pull request #874](https://github.com/ESMValGroup/ESMValCore/pull/874)) by [@stefsmeets](https://github.com/stefsmeets) - Update codacy coverage reporter to fix coverage ([Pull request #905](https://github.com/ESMValGroup/ESMValCore/pull/905)) by [@nielsdrost](https://github.com/nielsdrost) - Avoid hardcoded year in tests and add improvement to plev test case ([Pull request #921](https://github.com/ESMValGroup/ESMValCore/pull/921)) by [@bouweandela](https://github.com/bouweandela) - Pin scipy to less than 1.6.0 until [Issue #927](https://github.com/ESMValGroup/ESMValCore/issues/927) gets resolved ([Pull request #928](https://github.com/ESMValGroup/ESMValCore/pull/928)) by [@valeriupredoi](https://github.com/valeriupredoi) - Github Actions: change time when conda install test runs ([Pull request #930](https://github.com/ESMValGroup/ESMValCore/pull/930)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove redundant test line from test_utils.py ([Pull request #935](https://github.com/ESMValGroup/ESMValCore/pull/935)) by [@valeriupredoi](https://github.com/valeriupredoi) - Removed netCDF4 package from integration tests of fixes ([Pull request #938](https://github.com/ESMValGroup/ESMValCore/pull/938)) by [@schlunma](https://github.com/schlunma) - Use new conda environment for installing ESMValCore in Docker containers ([Pull request #951](https://github.com/ESMValGroup/ESMValCore/pull/951)) by [@bouweandela](https://github.com/bouweandela) ### Notebook API (experimental) - Implement importable config object in experimental API submodule ([Pull request #868](https://github.com/ESMValGroup/ESMValCore/pull/868)) by [@stefsmeets](https://github.com/stefsmeets) - Add loading and running recipes to the notebook API ([Pull request #907](https://github.com/ESMValGroup/ESMValCore/pull/907)) by [@stefsmeets](https://github.com/stefsmeets) - Add displaying and loading of recipe output to the notebook API ([Pull request #957](https://github.com/ESMValGroup/ESMValCore/pull/957)) by [@stefsmeets](https://github.com/stefsmeets) - Add functionality to run single diagnostic task to notebook API ([Pull request #962](https://github.com/ESMValGroup/ESMValCore/pull/962)) by [@stefsmeets](https://github.com/stefsmeets) ### Improvements - Create CODEOWNERS file ([Pull request #809](https://github.com/ESMValGroup/ESMValCore/pull/809)) by [@jvegreg](https://github.com/jvegreg) - Remove code needed for Python <3.6 ([Pull request #844](https://github.com/ESMValGroup/ESMValCore/pull/844)) by [@bouweandela](https://github.com/bouweandela) - Add requests as a dependency ([Pull request #850](https://github.com/ESMValGroup/ESMValCore/pull/850)) by [@bouweandela](https://github.com/bouweandela) - Pin Python to less than 3.9 ([Pull request #870](https://github.com/ESMValGroup/ESMValCore/pull/870)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove numba dependency ([Pull request #880](https://github.com/ESMValGroup/ESMValCore/pull/880)) by [@schlunma](https://github.com/schlunma) - Add Listing and finding recipes to the experimental notebook API ([Pull request #901](https://github.com/ESMValGroup/ESMValCore/pull/901)) by [@stefsmeets](https://github.com/stefsmeets) - Skip variables that don’t have dataset or additional_dataset keys ([Pull request #860](https://github.com/ESMValGroup/ESMValCore/pull/860)) by [@valeriupredoi](https://github.com/valeriupredoi) - Refactor logging configuration ([Pull request #933](https://github.com/ESMValGroup/ESMValCore/pull/933)) by [@stefsmeets](https://github.com/stefsmeets) - Xco2 derivation ([Pull request #913](https://github.com/ESMValGroup/ESMValCore/pull/913)) by [@bettina-gier](https://github.com/bettina-gier) - Working environment for Python 3.9 (pin to !=3.9.0) ([Pull request #885](https://github.com/ESMValGroup/ESMValCore/pull/885)) by [@valeriupredoi](https://github.com/valeriupredoi) - Print source file when using config get_config_user command ([Pull request #960](https://github.com/ESMValGroup/ESMValCore/pull/960)) by [@valeriupredoi](https://github.com/valeriupredoi) - Switch to Iris 3 ([Pull request #819](https://github.com/ESMValGroup/ESMValCore/pull/819)) by [@stefsmeets](https://github.com/stefsmeets) - Refactor tasks ([Pull request #959](https://github.com/ESMValGroup/ESMValCore/pull/959)) by [@stefsmeets](https://github.com/stefsmeets) - Restore task summary in debug log after #959 ([Pull request #981](https://github.com/ESMValGroup/ESMValCore/pull/981)) by [@bouweandela](https://github.com/bouweandela) - Pin pre-commit hooks ([Pull request #974](https://github.com/ESMValGroup/ESMValCore/pull/974)) by [@stefsmeets](https://github.com/stefsmeets) - Improve error messages when data is missing ([Pull request #917](https://github.com/ESMValGroup/ESMValCore/pull/917)) by [@jvegreg](https://github.com/jvegreg) - Set remove_preproc_dir to false in default config-user ([Pull request #979](https://github.com/ESMValGroup/ESMValCore/pull/979)) by [@valeriupredoi](https://github.com/valeriupredoi) - Move fiona to be installed from conda forge ([Pull request #987](https://github.com/ESMValGroup/ESMValCore/pull/987)) by [@valeriupredoi](https://github.com/valeriupredoi) - Re-added fiona in setup.py ([Pull request #990](https://github.com/ESMValGroup/ESMValCore/pull/990)) by [@valeriupredoi](https://github.com/valeriupredoi) ## v2.1.0 This release includes ### Bug fixes - Set unit=1 if anomalies are standardized ([Pull request #727](https://github.com/ESMValGroup/ESMValCore/pull/727)) by [@bascrezee](https://github.com/bascrezee) - Fix crash for FGOALS-g2 variables without longitude coordinate ([Pull request #729](https://github.com/ESMValGroup/ESMValCore/pull/729)) by [@bouweandela](https://github.com/bouweandela) - Improve variable alias management ([Pull request #595](https://github.com/ESMValGroup/ESMValCore/pull/595)) by [@jvegreg](https://github.com/jvegreg) - Fix area_statistics fx files loading ([Pull request #798](https://github.com/ESMValGroup/ESMValCore/pull/798)) by [@jvegreg](https://github.com/jvegreg) - Fix units after derivation ([Pull request #754](https://github.com/ESMValGroup/ESMValCore/pull/754)) by [@schlunma](https://github.com/schlunma) ### Documentation - Update v2.0.0 release notes with final additions ([Pull request #722](https://github.com/ESMValGroup/ESMValCore/pull/722)) by [@bouweandela](https://github.com/bouweandela) - Update package description in setup.py ([Pull request #725](https://github.com/ESMValGroup/ESMValCore/pull/725)) by [@mattiarighi](https://github.com/mattiarighi) - Add installation instructions for pip installation ([Pull request #735](https://github.com/ESMValGroup/ESMValCore/pull/735)) by [@bouweandela](https://github.com/bouweandela) - Improve config-user documentation ([Pull request #740](https://github.com/ESMValGroup/ESMValCore/pull/740)) by [@bouweandela](https://github.com/bouweandela) - Update the zenodo file with contributors ([Pull request #807](https://github.com/ESMValGroup/ESMValCore/pull/807)) by [@valeriupredoi](https://github.com/valeriupredoi) - Improve command line run documentation ([Pull request #721](https://github.com/ESMValGroup/ESMValCore/pull/721)) by [@jvegreg](https://github.com/jvegreg) - Update the zenodo file with contributors (continued) ([Pull request #810](https://github.com/ESMValGroup/ESMValCore/pull/810)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - Reduce size of docker image ([Pull request #723](https://github.com/ESMValGroup/ESMValCore/pull/723)) by [@jvegreg](https://github.com/jvegreg) - Add ‘test’ extra to installation, used by docker development tag ([Pull request #733](https://github.com/ESMValGroup/ESMValCore/pull/733)) by [@bouweandela](https://github.com/bouweandela) - Correct dockerhub link ([Pull request #736](https://github.com/ESMValGroup/ESMValCore/pull/736)) by [@bouweandela](https://github.com/bouweandela) - Create action-install-from-pypi.yml ([Pull request #734](https://github.com/ESMValGroup/ESMValCore/pull/734)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add pre-commit for linting/formatting ([Pull request #766](https://github.com/ESMValGroup/ESMValCore/pull/766)) by [@stefsmeets](https://github.com/stefsmeets) - Run tests in parallel and when building conda package ([Pull request #745](https://github.com/ESMValGroup/ESMValCore/pull/745)) by [@bouweandela](https://github.com/bouweandela) - Readable exclude pattern for pre-commit ([Pull request #770](https://github.com/ESMValGroup/ESMValCore/pull/770)) by [@stefsmeets](https://github.com/stefsmeets) - Github Actions Tests ([Pull request #732](https://github.com/ESMValGroup/ESMValCore/pull/732)) by [@valeriupredoi](https://github.com/valeriupredoi) - Remove isort setup to fix formatting conflict with yapf ([Pull request #778](https://github.com/ESMValGroup/ESMValCore/pull/778)) by [@stefsmeets](https://github.com/stefsmeets) - Fix yapf-isort import formatting conflict (Fixes #777) ([Pull request #784](https://github.com/ESMValGroup/ESMValCore/pull/784)) by [@stefsmeets](https://github.com/stefsmeets) - Sorted output for esmvaltool recipes list ([Pull request #790](https://github.com/ESMValGroup/ESMValCore/pull/790)) by [@stefsmeets](https://github.com/stefsmeets) - Replace vmprof with vprof ([Pull request #780](https://github.com/ESMValGroup/ESMValCore/pull/780)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update CMIP6 tables to 6.9.32 ([Pull request #706](https://github.com/ESMValGroup/ESMValCore/pull/706)) by [@jvegreg](https://github.com/jvegreg) - Default config-user path now set in config-user read function ([Pull request #791](https://github.com/ESMValGroup/ESMValCore/pull/791)) by [@jvegreg](https://github.com/jvegreg) - Add custom variable lweGrace ([Pull request #692](https://github.com/ESMValGroup/ESMValCore/pull/692)) by [@bascrezee](https://github.com/bascrezee) - Create Github Actions workflow to build and deploy on Test PyPi and PyPi ([Pull request #820](https://github.com/ESMValGroup/ESMValCore/pull/820)) by [@valeriupredoi](https://github.com/valeriupredoi) - Build and publish the esmvalcore package to conda via Github Actions workflow ([Pull request #825](https://github.com/ESMValGroup/ESMValCore/pull/825)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Fixes for datasets - Fix cmip6 models ([Pull request #629](https://github.com/ESMValGroup/ESMValCore/pull/629)) by [@npgillett](https://github.com/npgillett) - Fix siconca variable in EC-Earth3 and EC-Earth3-Veg models in amip simulation ([Pull request #702](https://github.com/ESMValGroup/ESMValCore/pull/702)) by [@egalytska](https://github.com/egalytska) ### Preprocessor - Move cmor_check_data to early in preprocessing chain ([Pull request #743](https://github.com/ESMValGroup/ESMValCore/pull/743)) by [@bouweandela](https://github.com/bouweandela) - Add RMS iris analysis operator to statistics preprocessor functions ([Pull request #747](https://github.com/ESMValGroup/ESMValCore/pull/747)) by [@pcosbsc](https://github.com/pcosbsc) - Add surface chlorophyll concentration as a derived variable ([Pull request #720](https://github.com/ESMValGroup/ESMValCore/pull/720)) by [@sloosvel](https://github.com/sloosvel) - Use dask to reduce memory consumption of extract_levels for masked data ([Pull request #776](https://github.com/ESMValGroup/ESMValCore/pull/776)) by [@valeriupredoi](https://github.com/valeriupredoi) ## v2.0.0 This release includes ### Bug fixes - Fixed derivation of co2s ([Pull request #594](https://github.com/ESMValGroup/ESMValCore/pull/594)) by [@schlunma](https://github.com/schlunma) - Padding while cropping needs to stay within sane bounds for shapefiles that span the whole Earth ([Pull request #626](https://github.com/ESMValGroup/ESMValCore/pull/626)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fix concatenation of a single cube ([Pull request #655](https://github.com/ESMValGroup/ESMValCore/pull/655)) by [@bouweandela](https://github.com/bouweandela) - Fix mask fx dict handling not to fail if empty list in values ([Pull request #661](https://github.com/ESMValGroup/ESMValCore/pull/661)) by [@valeriupredoi](https://github.com/valeriupredoi) - Preserve metadata during anomalies computation when using iris cubes difference ([Pull request #652](https://github.com/ESMValGroup/ESMValCore/pull/652)) by [@valeriupredoi](https://github.com/valeriupredoi) - Avoid crashing when there is directory ‘esmvaltool’ in the current working directory ([Pull request #672](https://github.com/ESMValGroup/ESMValCore/pull/672)) by [@valeriupredoi](https://github.com/valeriupredoi) - Solve bug in ACCESS1 dataset fix for calendar. ([Pull request #671](https://github.com/ESMValGroup/ESMValCore/pull/671)) by [@Peter9192](https://github.com/Peter9192) - Fix the syntax for adding multiple ensemble members from the same dataset ([Pull request #678](https://github.com/ESMValGroup/ESMValCore/pull/678)) by [@SarahAlidoost](https://github.com/SarahAlidoost) - Fix bug that made preprocessor with fx files fail in rare cases ([Pull request #670](https://github.com/ESMValGroup/ESMValCore/pull/670)) by [@schlunma](https://github.com/schlunma) - Add support for string coordinates ([Pull request #657](https://github.com/ESMValGroup/ESMValCore/pull/657)) by [@jvegreg](https://github.com/jvegreg) - Fixed the shape extraction to account for wraparound shapefile coords ([Pull request #319](https://github.com/ESMValGroup/ESMValCore/pull/319)) by [@valeriupredoi](https://github.com/valeriupredoi) - Fixed bug in time weights calculation ([Pull request #695](https://github.com/ESMValGroup/ESMValCore/pull/695)) by [@schlunma](https://github.com/schlunma) - Fix diagnostic filter ([Pull request #713](https://github.com/ESMValGroup/ESMValCore/pull/713)) by [@jvegreg](https://github.com/jvegreg) ### Documentation - Add pandas as a requirement for building the documentation ([Pull request #607](https://github.com/ESMValGroup/ESMValCore/pull/607)) by [@bouweandela](https://github.com/bouweandela) - Document default order in which preprocessor functions are applied ([Pull request #633](https://github.com/ESMValGroup/ESMValCore/pull/633)) by [@bouweandela](https://github.com/bouweandela) - Add pointers about data loading and CF standards to documentation ([Pull request #571](https://github.com/ESMValGroup/ESMValCore/pull/571)) by [@valeriupredoi](https://github.com/valeriupredoi) - Config file populated with site-specific data paths examples ([Pull request #619](https://github.com/ESMValGroup/ESMValCore/pull/619)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update Codacy badges ([Pull request #643](https://github.com/ESMValGroup/ESMValCore/pull/643)) by [@bouweandela](https://github.com/bouweandela) - Update copyright info on readthedocs ([Pull request #668](https://github.com/ESMValGroup/ESMValCore/pull/668)) by [@bouweandela](https://github.com/bouweandela) - Updated references to documentation (now docs.esmvaltool.org) ([Pull request #675](https://github.com/ESMValGroup/ESMValCore/pull/675)) by [@axel-lauer](https://github.com/axel-lauer) - Add all European grants to Zenodo ([Pull request #680](https://github.com/ESMValGroup/ESMValCore/pull/680)) by [@bouweandela](https://github.com/bouweandela) - Update Sphinx to v3 or later ([Pull request #683](https://github.com/ESMValGroup/ESMValCore/pull/683)) by [@bouweandela](https://github.com/bouweandela) - Increase version to 2.0.0 and add release notes ([Pull request #691](https://github.com/ESMValGroup/ESMValCore/pull/691)) by [@bouweandela](https://github.com/bouweandela) - Update setup.py and README.md for use on PyPI ([Pull request #693](https://github.com/ESMValGroup/ESMValCore/pull/693)) by [@bouweandela](https://github.com/bouweandela) - Suggested Documentation changes ([Pull request #690](https://github.com/ESMValGroup/ESMValCore/pull/690)) by [@ssmithClimate](https://github.com/ssmithClimate) ### Improvements - Reduce the size of conda package ([Pull request #606](https://github.com/ESMValGroup/ESMValCore/pull/606)) by [@bouweandela](https://github.com/bouweandela) - Add a few unit tests for DiagnosticTask ([Pull request #613](https://github.com/ESMValGroup/ESMValCore/pull/613)) by [@bouweandela](https://github.com/bouweandela) - Make ncl or R tests not fail if package not installed ([Pull request #610](https://github.com/ESMValGroup/ESMValCore/pull/610)) by [@valeriupredoi](https://github.com/valeriupredoi) - Pin flake8<3.8.0 ([Pull request #623](https://github.com/ESMValGroup/ESMValCore/pull/623)) by [@valeriupredoi](https://github.com/valeriupredoi) - Log warnings for likely errors in provenance record ([Pull request #592](https://github.com/ESMValGroup/ESMValCore/pull/592)) by [@bouweandela](https://github.com/bouweandela) - Unpin flake8 ([Pull request #646](https://github.com/ESMValGroup/ESMValCore/pull/646)) by [@bouweandela](https://github.com/bouweandela) - More flexible native6 default DRS ([Pull request #645](https://github.com/ESMValGroup/ESMValCore/pull/645)) by [@bouweandela](https://github.com/bouweandela) - Try to use the same python for running diagnostics as for esmvaltool ([Pull request #656](https://github.com/ESMValGroup/ESMValCore/pull/656)) by [@bouweandela](https://github.com/bouweandela) - Fix test for lower python version and add note on lxml ([Pull request #659](https://github.com/ESMValGroup/ESMValCore/pull/659)) by [@valeriupredoi](https://github.com/valeriupredoi) - Added 1m deep average soil moisture variable ([Pull request #664](https://github.com/ESMValGroup/ESMValCore/pull/664)) by [@bascrezee](https://github.com/bascrezee) - Update docker recipe ([Pull request #603](https://github.com/ESMValGroup/ESMValCore/pull/603)) by [@jvegreg](https://github.com/jvegreg) - Improve command line interface ([Pull request #605](https://github.com/ESMValGroup/ESMValCore/pull/605)) by [@jvegreg](https://github.com/jvegreg) - Remove utils directory ([Pull request #697](https://github.com/ESMValGroup/ESMValCore/pull/697)) by [@bouweandela](https://github.com/bouweandela) - Avoid pytest version that crashes ([Pull request #707](https://github.com/ESMValGroup/ESMValCore/pull/707)) by [@bouweandela](https://github.com/bouweandela) - Options arg in read_config_user_file now optional ([Pull request #716](https://github.com/ESMValGroup/ESMValCore/pull/716)) by [@jvegreg](https://github.com/jvegreg) - Produce a readable warning if ancestors are a string instead of a list. ([Pull request #711](https://github.com/ESMValGroup/ESMValCore/pull/711)) by [@katjaweigel](https://github.com/katjaweigel) - Pin Yamale to v2 ([Pull request #718](https://github.com/ESMValGroup/ESMValCore/pull/718)) by [@bouweandela](https://github.com/bouweandela) - Expanded cmor public API ([Pull request #714](https://github.com/ESMValGroup/ESMValCore/pull/714)) by [@schlunma](https://github.com/schlunma) ### Fixes for datasets - Added various fixes for hybrid height coordinates ([Pull request #562](https://github.com/ESMValGroup/ESMValCore/pull/562)) by [@schlunma](https://github.com/schlunma) - Extended fix for cl-like variables of CESM2 models ([Pull request #604](https://github.com/ESMValGroup/ESMValCore/pull/604)) by [@schlunma](https://github.com/schlunma) - Added fix to convert “geopotential” to “geopotential height” for ERA5 ([Pull request #640](https://github.com/ESMValGroup/ESMValCore/pull/640)) by [@egalytska](https://github.com/egalytska) - Do not fix longitude values if they are too far from valid range ([Pull request #636](https://github.com/ESMValGroup/ESMValCore/pull/636)) by [@jvegreg](https://github.com/jvegreg) ### Preprocessor - Implemented concatenation of cubes with derived coordinates ([Pull request #546](https://github.com/ESMValGroup/ESMValCore/pull/546)) by [@schlunma](https://github.com/schlunma) - Fix derived variable ctotal calculation depending on project and standard name ([Pull request #620](https://github.com/ESMValGroup/ESMValCore/pull/620)) by [@valeriupredoi](https://github.com/valeriupredoi) - State of the art FX variables handling without preprocessing ([Pull request #557](https://github.com/ESMValGroup/ESMValCore/pull/557)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add max, min and std operators to multimodel ([Pull request #602](https://github.com/ESMValGroup/ESMValCore/pull/602)) by [@jvegreg](https://github.com/jvegreg) - Added preprocessor to extract amplitude of cycles ([Pull request #597](https://github.com/ESMValGroup/ESMValCore/pull/597)) by [@schlunma](https://github.com/schlunma) - Overhaul concatenation and allow for correct concatenation of multiple overlapping datasets ([Pull request #615](https://github.com/ESMValGroup/ESMValCore/pull/615)) by [@valeriupredoi](https://github.com/valeriupredoi) - Change volume stats to handle and output masked array result ([Pull request #618](https://github.com/ESMValGroup/ESMValCore/pull/618)) by [@valeriupredoi](https://github.com/valeriupredoi) - Area_weights for cordex in area_statistics ([Pull request #631](https://github.com/ESMValGroup/ESMValCore/pull/631)) by [@mwjury](https://github.com/mwjury) - Accept cubes as input in multimodel ([Pull request #637](https://github.com/ESMValGroup/ESMValCore/pull/637)) by [@sloosvel](https://github.com/sloosvel) - Make multimodel work correctly with yearly data ([Pull request #677](https://github.com/ESMValGroup/ESMValCore/pull/677)) by [@valeriupredoi](https://github.com/valeriupredoi) - Optimize time weights in time preprocessor for climate statistics ([Pull request #684](https://github.com/ESMValGroup/ESMValCore/pull/684)) by [@valeriupredoi](https://github.com/valeriupredoi) - Add percentiles to multi-model stats ([Pull request #679](https://github.com/ESMValGroup/ESMValCore/pull/679)) by [@Peter9192](https://github.com/Peter9192) ## v2.0.0b9 This release includes ### Bug fixes - Cast dtype float32 to output from zonal and meridional area preprocessors ([Pull request #581](https://github.com/ESMValGroup/ESMValCore/pull/581)) by [@valeriupredoi](https://github.com/valeriupredoi) ### Improvements - Unpin on Python<3.8 for conda package (run) ([Pull request #570](https://github.com/ESMValGroup/ESMValCore/pull/570)) by [@valeriupredoi](https://github.com/valeriupredoi) - Update pytest installation marker ([Pull request #572](https://github.com/ESMValGroup/ESMValCore/pull/572)) by [@bouweandela](https://github.com/bouweandela) - Remove vmrh2o ([Pull request #573](https://github.com/ESMValGroup/ESMValCore/pull/573)) by [@mattiarighi](https://github.com/mattiarighi) - Restructure documentation ([Pull request #575](https://github.com/ESMValGroup/ESMValCore/pull/575)) by [@bouweandela](https://github.com/bouweandela) - Fix mask in land variables for CCSM4 ([Pull request #579](https://github.com/ESMValGroup/ESMValCore/pull/579)) by [@zklaus](https://github.com/zklaus) - Fix derive scripts wrt required method ([Pull request #585](https://github.com/ESMValGroup/ESMValCore/pull/585)) by [@zklaus](https://github.com/zklaus) - Check coordinates do not have repeated standard names ([Pull request #558](https://github.com/ESMValGroup/ESMValCore/pull/558)) by [@jvegreg](https://github.com/jvegreg) - Added derivation script for co2s ([Pull request #587](https://github.com/ESMValGroup/ESMValCore/pull/587)) by [@schlunma](https://github.com/schlunma) - Adapted custom co2s table to match CMIP6 version ([Pull request #588](https://github.com/ESMValGroup/ESMValCore/pull/588)) by [@schlunma](https://github.com/schlunma) - Increase version to v2.0.0b9 ([Pull request #593](https://github.com/ESMValGroup/ESMValCore/pull/593)) by [@bouweandela](https://github.com/bouweandela) - Add a method to save citation information ([Pull request #402](https://github.com/ESMValGroup/ESMValCore/pull/402)) by [@SarahAlidoost](https://github.com/SarahAlidoost) For older releases, see the release notes on [https://github.com/ESMValGroup/ESMValCore/releases](https://github.com/ESMValGroup/ESMValCore/releases).