# Access data from any source

ESMValCore supports a modular system for reading data from various data sources.
In the future, this module may be extended with support for writing output data.

The interface is defined in the [`esmvalcore.io.protocol`](esmvalcore.io.protocol.html.md#module-esmvalcore.io.protocol) module and
the other modules here provide an implementation for a particular data source.

## esmvalcore.io

A modular system for reading input data from various sources.

An input data source can be defined in the configuration by using
[`esmvalcore.config.CFG`](esmvalcore.config.html.md#esmvalcore.config.CFG), for example:

```python
>>> from esmvalcore.config import CFG
>>> CFG["projects"]["CMIP6"]["data"]["local"] = {
        "type": "esmvalcore.io.local.LocalDataSource",
        "rootpath": "~/climate_data",
        "dirname_template": "{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}",
        "filename_template": "{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc",
    }
```

or as a [YAML configuration file](../quickstart/configure.html.md#config-overview):

```yaml
projects:
  CMIP6:
    data:
      local:
        type: "esmvalcore.io.local.LocalDataSource"
        rootpath: "~/climate_data"
        dirname_template: "{project}/{activity}/{institute}/{dataset}/{exp}/{ensemble}/{mip}/{short_name}/{grid}/{version}"
        filename_template: "{short_name}_{mip}_{dataset}_{exp}_{ensemble}_{grid}*.nc"
```

where `CMIP6` is a project, and `local` is a unique name describing the
data source. The data source type,
[`esmvalcore.io.local.LocalDataSource`](esmvalcore.io.local.html.md#esmvalcore.io.local.LocalDataSource), in the example above, needs to
implement the [`esmvalcore.io.protocol.DataSource`](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataSource) protocol. Any
remaining key-value pairs in the configuration, `rootpath`,
`dirname_template`, and `filename_template` in this example, are passed
as keyword arguments to the data source when it is created.

If there are multiple data sources configured for a project, deduplication of
search results happens based on the
[`esmvalcore.io.protocol.DataElement.name`](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataElement.name) attribute and the `"version"`
facet in [`esmvalcore.io.protocol.DataElement.facets`](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataElement.facets) of the data elements
provided by the data sources. If no `version` facet is specified in the
search, the latest version will be used. If there is a tie, the data element
provided by the data source with the lowest value of
[`esmvalcore.io.protocol.DataSource.priority`](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataSource.priority) is chosen.

**Functions:**

| [`load_data_sources`](#esmvalcore.io.load_data_sources)(session[, project])   | Get the list of available data sources.   |
|------------------------------------------------------------------------------------------|-------------------------------------------|

### esmvalcore.io.load_data_sources(session: [Session](esmvalcore.config.html.md#esmvalcore.config.Session), project: [str](https://docs.python.org/3/builtins/stdtypes.html#str) | [None](https://docs.python.org/3/builtins/constants.html#None) = None) → [list](https://docs.python.org/3/builtins/stdtypes.html#list)[[esmvalcore.io.protocol.DataSource](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataSource)]

Get the list of available data sources.

If no `priority` is configured for a data source, the default priority
of 1 is used.

* **Parameters:**
  * **session** ([*Session*](esmvalcore.config.html.md#esmvalcore.config.Session)) – The configuration.
  * **project** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str) *|* *None*) – If specified, only data sources for this project are returned.
* **Returns:**
  A list of available data sources.
* **Raises:**
  **ValueError:** – If the project or its settings are not found in the configuration.
* **Return type:**
  [list](https://docs.python.org/3/builtins/stdtypes.html#list)[[esmvalcore.io.protocol.DataSource](esmvalcore.io.protocol.html.md#esmvalcore.io.protocol.DataSource)]

### Submodules

* [esmvalcore.io.esgf](esmvalcore.io.esgf.html.md)
* [esmvalcore.io.intake_esgf](esmvalcore.io.intake_esgf.html.md)
* [esmvalcore.io.local](esmvalcore.io.local.html.md)
* [esmvalcore.io.protocol](esmvalcore.io.protocol.html.md)