# Discovering data

This notebook shows how to find out what data is available locally as well as on ESGF. It also shows how to download the data from ESGF.

```ipython3
from esmvalcore.config import CFG
from esmvalcore.dataset import Dataset
from esmvalcore.io.esgf import download
```

Configure ESMValCore so it always searches the ESGF for data. See [data sources](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/configure.html#config-data-sources) for more information.

```ipython3
CFG["search_data"] = "complete"
CFG.nested_update(
    {
        "projects": {
            "CMIP6": {
                "data": {
                    "esgf": {
                        "type": "esmvalcore.io.esgf.ESGFDataSource",
                        "download_dir": "~/climate_data",
                        "priority": 10,
                    },
                },
            },
        },
    },
)
```

We define a dataset template to search for all CMIP6 datasets that provide surface air temperature (tas) on a monthly resolution for the historical experiment. Note that ESMValCore uses its own names for the facets for a more uniform naming across different CMIP phases and other projects. The mapping to the facet names used on ESGF can be found in [esmvalcore.io.esgf.facets.FACETS](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/api/esmvalcore.io.esgf.html#esmvalcore.io.esgf.facets.FACETS).

```ipython3
dataset_template = Dataset(
    short_name="tas",
    mip="Amon",
    project="CMIP6",
    exp="historical",
    dataset="*",
    institute="*",
    ensemble="*",
    grid="*",
)
```

Next, we use the `Dataset.from_files` method to build a list of datasets from the available files. This may take a while as searching the ESGF for many files is a bit slow. Because the search results are cached for a [configurable duration](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/quickstart/configure.html#esgf-configuration), subsequent searches will be faster.

```ipython3
datasets = list(dataset_template.from_files())
print(f"Found {len(datasets)} datasets, showing the first 10:")
datasets[:10]
```

```myst-ansi
Found 778 datasets, showing the first 10:
```

Let’s look at the first dataset in more detail. We can print the facets describing the dataset:

```ipython3
dataset = datasets[0]
dataset
```

and see what files are available:

```ipython3
dataset.files
```

A single file can be downloaded using its `prepare` method:

```ipython3
dataset.files[0].prepare()
```

For downloading many files, the [esmvalcore.io.esgf.download](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/api/esmvalcore.io.esgf.html#esmvalcore.io.esgf.download) function is recommended because it will download the files in parallel. The ESMValCore will try to guess the fastest host and download from there. If it is not available for some reason, it will automatically fall back to the next host.

```ipython3
download(dataset.files)
```