<a id="api-configuration"></a>

# Configuration

This section describes the [`config`](#module-esmvalcore.config) module.

## CFG

Configuration of ESMValCore/Tool is done via [`CFG`](#esmvalcore.config.CFG)
object:

```python
>>> from esmvalcore.config import CFG
>>> CFG
Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
        'compress_netcdf': False,
        'exit_on_warning': False,
        'log_level': 'info',
        'max_parallel_tasks': None,
        'output_dir': PosixPath('/home/user/esmvaltool_output'),
        'output_file_type': 'png',
        'profile_diagnostic': False,
        'remove_preproc_dir': True,
        'save_intermediary_cubes': False)
```

All configuration parameters are listed [here](../quickstart/configure.html.md#config-options).

[`CFG`](#esmvalcore.config.CFG) is essentially a python dictionary with a few
extra functions, similar to [`matplotlib.rcParams`](https://matplotlib.org/stable/api/matplotlib_configuration_api.html#matplotlib.rcParams).
This means that values can be updated like this:

```python
>>> CFG['output_dir'] = '~/esmvaltool_output'
>>> CFG['output_dir']
PosixPath('/home/user/esmvaltool_output')
```

Notice that [`CFG`](#esmvalcore.config.CFG) automatically converts the path
to an instance of [`pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) and expands the home directory.
All values entered into the config are validated to prevent mistakes, for
example, it will warn you if you make a typo in the key:

```python
>>> CFG['output_directory'] = '~/esmvaltool_output'
InvalidConfigParameter: `output_directory` is not a valid config parameter.
```

Or, if the value entered cannot be converted to the expected type:

```python
>>> CFG['max_parallel_tasks'] = '🐜'
InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
```

[`CFG`](#esmvalcore.config.CFG) is also flexible, so it tries to correct the
type of your input if possible:

```python
>>> CFG['max_parallel_tasks'] = '8'  # str
>>> type(CFG['max_parallel_tasks'])
int
```

It is also possible to temporarily use different configuration options using
the [`context()`](#esmvalcore.config.Config.context) context manager:

```python
>>> with CFG.context({"output_dir": "/path/to/output"}, log_level="debug"):
...     print(CFG["output_dir"])
...     print(CFG["log_level"])
PosixPath('/path/to/output')
debug
>>> print(CFG["output_dir"])
PosixPath('/home/user/esmvaltool_output')
>>> print(CFG["log_level"])
info
```

By default, the configuration is loaded from YAML files in the user’s home
directory at `~/.config/esmvaltool`.
If set, this can be overwritten with the `ESMVALTOOL_CONFIG_DIR` environment
variable.
Defaults for options that are not specified explicitly are listed [here](../quickstart/configure.html.md#config-options).
To reload the current configuration object according to these rules, use:

```python
>>> CFG.reload()
```

To load the configuration object from custom directories, use:

```python
>>> dirs = ['my/default/config', 'my/custom/config']
>>> CFG.load_from_dirs(dirs)
```

To update the existing configuration object from custom directories, use:

```python
>>> dirs = ['my/default/config', 'my/custom/config']
>>> CFG.update_from_dirs(dirs)
```

## Session

Recipes and diagnostics will be run in their own directories.
This behavior can be controlled via the [`Session`](#esmvalcore.config.Session)
object.
A [`Session`](#esmvalcore.config.Session) must always be initiated from the
global [`CFG`](#esmvalcore.config.CFG) object:

```python
>>> session = CFG.start_session(name='my_session')
```

A [`Session`](#esmvalcore.config.Session) is very similar to the config.
It is also a dictionary, and copies all the keys from the
[`CFG`](#esmvalcore.config.CFG) object.
At this moment, `session` is essentially a copy of
[`CFG`](#esmvalcore.config.CFG):

```python
>>> print(session == CFG)
True
>>> session['output_dir'] = '~/my_output_dir'
>>> print(session == CFG)  # False
False
```

A [`Session`](#esmvalcore.config.Session) also knows about the directories where
the data will stored.
The session name is used to prefix the directories.

```python
>>> session.session_dir
/home/user/my_output_dir/my_session_20201203_155821
>>> session.run_dir
/home/user/my_output_dir/my_session_20201203_155821/run
>>> session.work_dir
/home/user/my_output_dir/my_session_20201203_155821/work
>>> session.preproc_dir
/home/user/my_output_dir/my_session_20201203_155821/preproc
>>> session.plot_dir
/home/user/my_output_dir/my_session_20201203_155821/plots
```

Unlike the global configuration, of which only one can exist, multiple sessions
can be initiated from [`CFG`](#esmvalcore.config.CFG).

## API reference

Configuration module.

### esmvalcore.config.CFG

Global ESMValCore configuration object of type
[`esmvalcore.config.Config`](#esmvalcore.config.Config).

By default, this will be loaded from YAML files in the user configuration
directory (by default `~/.config/esmvaltool`, but this can be changed
with the `ESMVALTOOL_CONFIG_DIR` environment variable) similar to the way
[Dask handles configuration](https://docs.dask.org/en/stable/configuration.html).

**Classes:**

| [`Config`](#esmvalcore.config.Config)(\*args, \*\*kwargs)   | ESMValTool configuration object.                                     |
|-------------------------------------------------------------------------------|----------------------------------------------------------------------|
| [`Session`](#esmvalcore.config.Session)(config[, name])      | Container class for session configuration and directory information. |

### *class* esmvalcore.config.Config(\*args, \*\*kwargs)

ESMValTool configuration object.

Do not instantiate this class directly, but use
[`esmvalcore.config.CFG`](#esmvalcore.config.CFG) instead.

**Methods:**

| [`clear`](#esmvalcore.config.Config.clear)()                    | Clear contents of configuration object.                          |
|-----------------------------------------------------------------------------|------------------------------------------------------------------|
| [`context`](#esmvalcore.config.Config.context)([mapping])         | Set configuration options temporarily inside a `with` statement. |
| [`copy`](#esmvalcore.config.Config.copy)()                     | Copy the keys/values of this object to a dict.                   |
| [`load_from_dirs`](#esmvalcore.config.Config.load_from_dirs)(dirs)       | Clear and load configuration object from directories.            |
| [`nested_update`](#esmvalcore.config.Config.nested_update)(new_options) | Nested update of configuration object with another mapping.      |
| [`reload`](#esmvalcore.config.Config.reload)()                   | Clear and reload the configuration object.                       |
| [`start_session`](#esmvalcore.config.Config.start_session)(name)        | Start a new session from this configuration object.              |
| [`update_from_dirs`](#esmvalcore.config.Config.update_from_dirs)(dirs)     | Update configuration object from directories.                    |

#### clear() → [None](https://docs.python.org/3/builtins/constants.html#None)

Clear contents of configuration object.

* **Return type:**
  None

#### context(mapping: Mapping | [None](https://docs.python.org/3/builtins/constants.html#None) = None, \*\*kwargs: Any) → Generator[[None](https://docs.python.org/3/builtins/constants.html#None), [None](https://docs.python.org/3/builtins/constants.html#None), [None](https://docs.python.org/3/builtins/constants.html#None)]

Set configuration options temporarily inside a `with` statement.

This configuration will only be effective inside the context manager.

* **Parameters:**
  * **mapping** (*Mapping* *|* *None*) – Mapping with temporary configuration options.
  * **\*\*kwargs** (*Any*) – Temporary configuration options.
* **Return type:**
  Generator[None, None, None]

#### copy() → [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Copy the keys/values of this object to a dict.

* **Return type:**
  [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [*Any*](https://docs.python.org/3/library/typing.html#typing.Any)]

#### load_from_dirs(dirs: Iterable[[str](https://docs.python.org/3/builtins/stdtypes.html#str) | Path]) → [None](https://docs.python.org/3/builtins/constants.html#None)

Clear and load configuration object from directories.

This searches for all YAML files within the given directories and
merges them together using [`dask.config.collect()`](https://docs.dask.org/en/stable/configuration.html#dask.config.collect). Nested objects
are properly considered; see [`dask.config.update()`](https://docs.dask.org/en/stable/configuration.html#dask.config.update) for details.
Values in the latter directories are preferred to those in the former.

Options that are not explicitly specified via YAML files are set to the
[default values](../quickstart/configure.html.md#config-options).

#### NOTE
Just like [`dask.config.collect()`](https://docs.dask.org/en/stable/configuration.html#dask.config.collect), this silently ignores
non-existing directories.

* **Parameters:**
  **dirs** (*Iterable* *[*[*str*](https://docs.python.org/3/builtins/stdtypes.html#str) *|* *Path* *]*) – A list of directories to search for YAML configuration files.
* **Raises:**
  [**esmvalcore.exceptions.InvalidConfigParameter**](esmvalcore.exceptions.html.md#esmvalcore.exceptions.InvalidConfigParameter) – Invalid configuration option given.
* **Return type:**
  None

#### nested_update(new_options: Mapping) → [None](https://docs.python.org/3/builtins/constants.html#None)

Nested update of configuration object with another mapping.

Merge the existing configuration object with a new mapping using
[`dask.config.merge()`](https://docs.dask.org/en/stable/configuration.html#dask.config.merge)  (new values are preferred over old values).
Nested objects are properly considered; see [`dask.config.update()`](https://docs.dask.org/en/stable/configuration.html#dask.config.update)
for details.

* **Parameters:**
  **new_options** (*Mapping*) – New configuration options.
* **Raises:**
  [**esmvalcore.exceptions.InvalidConfigParameter**](esmvalcore.exceptions.html.md#esmvalcore.exceptions.InvalidConfigParameter) – Invalid configuration option given.
* **Return type:**
  None

#### reload() → [None](https://docs.python.org/3/builtins/constants.html#None)

Clear and reload the configuration object.

This will read all YAML files in the user configuration directory (by
default `~/.config/esmvaltool`, but this can be changed with the
`ESMVALTOOL_CONFIG_DIR` environment variable) and merges them
together using [`dask.config.collect()`](https://docs.dask.org/en/stable/configuration.html#dask.config.collect). Nested objects are properly
considered; see [`dask.config.update()`](https://docs.dask.org/en/stable/configuration.html#dask.config.update) for details.

Options that are not explicitly specified via YAML files are set to the
[default values](../quickstart/configure.html.md#config-options).

#### NOTE
If the user configuration directory does not exist, this will be
silently ignored.

* **Raises:**
  [**esmvalcore.exceptions.InvalidConfigParameter**](esmvalcore.exceptions.html.md#esmvalcore.exceptions.InvalidConfigParameter) – Invalid configuration option given.
* **Return type:**
  None

#### start_session(name: [str](https://docs.python.org/3/builtins/stdtypes.html#str)) → [Session](#esmvalcore.config.Session)

Start a new session from this configuration object.

* **Parameters:**
  **name** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name of the session.
* **Return type:**
  [Session](#esmvalcore.config.Session)

#### update_from_dirs(dirs: Iterable[[str](https://docs.python.org/3/builtins/stdtypes.html#str) | Path]) → [None](https://docs.python.org/3/builtins/constants.html#None)

Update configuration object from directories.

This will first search for all YAML files within the given directories
and merge them together using [`dask.config.collect()`](https://docs.dask.org/en/stable/configuration.html#dask.config.collect) (if identical
values are provided in multiple files, the value from the last file
will be used).  Then, the current configuration is merged with these
new configuration options using [`dask.config.merge()`](https://docs.dask.org/en/stable/configuration.html#dask.config.merge) (new values
are preferred over old values). Nested objects are properly considered;
see [`dask.config.update()`](https://docs.dask.org/en/stable/configuration.html#dask.config.update) for details.

#### NOTE
Just like [`dask.config.collect()`](https://docs.dask.org/en/stable/configuration.html#dask.config.collect), this silently ignores
non-existing directories.

* **Parameters:**
  **dirs** (*Iterable* *[*[*str*](https://docs.python.org/3/builtins/stdtypes.html#str) *|* *Path* *]*) – A list of directories to search for YAML configuration files.
* **Raises:**
  [**esmvalcore.exceptions.InvalidConfigParameter**](esmvalcore.exceptions.html.md#esmvalcore.exceptions.InvalidConfigParameter) – Invalid configuration option given.
* **Return type:**
  None

### *class* esmvalcore.config.Session(config: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict), name: [str](https://docs.python.org/3/builtins/stdtypes.html#str) = 'session')

Container class for session configuration and directory information.

Do not instantiate this class directly, but use
`CFG.start_session` instead.

* **Parameters:**
  * **config** ([*dict*](https://docs.python.org/3/builtins/stdtypes.html#dict)) – Dictionary with configuration settings.
  * **name** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name of the session to initialize, for example, the name of the
    recipe (default=’session’).

**Methods:**

| [`clear`](#esmvalcore.config.Session.clear)()                  | Clear contents of configuration object.                          |
|---------------------------------------------------------------------------|------------------------------------------------------------------|
| [`context`](#esmvalcore.config.Session.context)([mapping])       | Set configuration options temporarily inside a `with` statement. |
| [`copy`](#esmvalcore.config.Session.copy)()                   | Copy the keys/values of this object to a dict.                   |
| [`set_session_name`](#esmvalcore.config.Session.set_session_name)([name]) | Set the name for the session.                                    |

**Attributes:**

| [`cmor_log`](#esmvalcore.config.Session.cmor_log)                | Path to the log file with CMOR check messages.                                                                                               |
|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
| [`main_log`](#esmvalcore.config.Session.main_log)                | Path to the log file.                                                                                                                        |
| [`main_log_debug`](#esmvalcore.config.Session.main_log_debug)          | Path to the debug log file.                                                                                                                  |
| [`plot_dir`](#esmvalcore.config.Session.plot_dir)                | Directory with diagnostic script plot files.                                                                                                 |
| [`preproc_dir`](#esmvalcore.config.Session.preproc_dir)             | Directory with preprocessor output files.                                                                                                    |
| [`relative_cmor_log`](#esmvalcore.config.Session.relative_cmor_log)       | Relative path to the log file with CMOR check messages, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).        |
| [`relative_main_log`](#esmvalcore.config.Session.relative_main_log)       | Relative path to the log file, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).                                 |
| [`relative_main_log_debug`](#esmvalcore.config.Session.relative_main_log_debug) | Relative path to the debug log file, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).                           |
| [`relative_plot_dir`](#esmvalcore.config.Session.relative_plot_dir)       | Relative path to diagnostic script plot directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).             |
| [`relative_preproc_dir`](#esmvalcore.config.Session.relative_preproc_dir)    | Relative path to the preprocessor output directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).            |
| [`relative_run_dir`](#esmvalcore.config.Session.relative_run_dir)        | Relative path to the directory with information about the run, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir). |
| [`relative_work_dir`](#esmvalcore.config.Session.relative_work_dir)       | Relative path to diagnostic script output directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).           |
| [`run_dir`](#esmvalcore.config.Session.run_dir)                 | Directory containing information about the run.                                                                                              |
| [`session_dir`](#esmvalcore.config.Session.session_dir)             | Session directory.                                                                                                                           |
| [`work_dir`](#esmvalcore.config.Session.work_dir)                | Directory with diagnostic script output files.                                                                                               |

#### clear() → [None](https://docs.python.org/3/builtins/constants.html#None)

Clear contents of configuration object.

* **Return type:**
  None

#### *property* cmor_log

Path to the log file with CMOR check messages.

#### context(mapping: Mapping | [None](https://docs.python.org/3/builtins/constants.html#None) = None, \*\*kwargs: Any) → Generator[[None](https://docs.python.org/3/builtins/constants.html#None), [None](https://docs.python.org/3/builtins/constants.html#None), [None](https://docs.python.org/3/builtins/constants.html#None)]

Set configuration options temporarily inside a `with` statement.

This configuration will only be effective inside the context manager.

* **Parameters:**
  * **mapping** (*Mapping* *|* *None*) – Mapping with temporary configuration options.
  * **\*\*kwargs** (*Any*) – Temporary configuration options.
* **Return type:**
  Generator[None, None, None]

#### copy() → [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Any](https://docs.python.org/3/library/typing.html#typing.Any)]

Copy the keys/values of this object to a dict.

* **Return type:**
  [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [*Any*](https://docs.python.org/3/library/typing.html#typing.Any)]

#### *property* main_log

Path to the log file.

#### *property* main_log_debug

Path to the debug log file.

#### *property* plot_dir

Directory with diagnostic script plot files.

#### *property* preproc_dir

Directory with preprocessor output files.

#### relative_cmor_log *= PosixPath('run/cmor_log.txt')*

Relative path to the log file with CMOR check messages, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_main_log *= PosixPath('run/main_log.txt')*

Relative path to the log file, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_main_log_debug *= PosixPath('run/main_log_debug.txt')*

Relative path to the debug log file, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_plot_dir *= PosixPath('plots')*

Relative path to diagnostic script plot directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_preproc_dir *= PosixPath('preproc')*

Relative path to the preprocessor output directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_run_dir *= PosixPath('run')*

Relative path to the directory with information about the run, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### relative_work_dir *= PosixPath('work')*

Relative path to diagnostic script output directory, with respect to [`session_dir`](#esmvalcore.config.Session.session_dir).

#### *property* run_dir

Directory containing information about the run.

#### *property* session_dir

Session directory.

This is a uniquely named directory inside the [output directory](../quickstart/output.html.md#outputdata).

#### session_name *: [str](https://docs.python.org/3/builtins/stdtypes.html#str) | [None](https://docs.python.org/3/builtins/constants.html#None)*

#### set_session_name(name: [str](https://docs.python.org/3/builtins/stdtypes.html#str) = 'session') → [None](https://docs.python.org/3/builtins/constants.html#None)

Set the name for the session.

The name is used to name the session directory, e.g.
session_20201208_132800/. The date is suffixed automatically.

* **Parameters:**
  **name** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str))
* **Return type:**
  None

#### *property* work_dir

Directory with diagnostic script output files.