en:papa-deimos

papa-deimos is an initiative of the ' Centro de Investigaciones del Mar y el Océano ' (CIMA, UBA-CONICET/IFAECI CNRS-IRD) shared with the ' Department of Atmospheric and Ocean Sciences ' (DCAO, UBA - FCEN) to start building and learning the data access and management of the near future. As climate databases gain higher spatial resolution, their file sizes grow rapidly. Current data management and analysis is becoming less and less effective and innovative solutions must be implemented.

This is a problem that affects the entire international climate community and the only possible solution is to bring the calculation and diagnosis of data where they reside. That is, providing the data storage server with ' in-situ ' calculation capabilities.

Description

Current procedure

For the management and analysis of climate data, the current procedure is as follows:

1. Preparation of the ' scientific question '
2. ' Search ' for data sources that may be useful in answering the question. A search is performed to find the required data, usually on remote servers. The data server, file format, period covered, etc. are determined.
3. ' Data download ': Once user has access to the data server, user proceeds to make a copy of the required data on its own resource via the Internet.
4. ' Analysis ' of data: In the ' local ' version of the data, own computational resources are used to perform statistical calculations and diagnoses
5. ' Figures ' results: In own resources, figures are made that show the results obtained.

Desired procedure

The ideal solution would be based on a single climate data server which would give access to all the climate data of interest (observations, modeling,...) and at the same time, with the necessary analysis capabilities.

• Searching for data and its possible access sources can be a difficult and laborious task. Currently there is the Earth System Grid Federation (ESGF) which guarantees universal and open access to climate data from global and regional modeling exercises. However, there is no similar platform for observational data. Furthermore, access to all that data that is not in some ESGF node (for example, experiments specific to a region, country, ...) becomes laborious and difficult.
• As databases grow, downloading over the Internet becomes more problematic, slow and difficult.
• Data storage requirements for study grow very quickly and require coordination at the institutional level, since with individual resources it becomes practically impossible and problems of duplication and redundancy are generated. Good coordination ensures optimal execution of financing resources.
• The calculation and diagnosis of climate data increasingly requires greater computational resources to which there is not always access. It also usually requires advanced programming skills, which not all data users have.
• Climate data may be in different formats, and/or be required in another format to be used by different tools (e.g. netCDF, GIS, ...)

papa-deimos attempts to provide this ' in-situ ' storage and calculation service to the CIMA/DCAO community, through which the researcher, teacher or student performs the calculations and only has to download the final figure or the file resulting from your diagnoses. The system has two components:

• ' papa ': 150 TB data server
• ' deimos ': computing node with 48 processors and 125 GB of RAM.

The user accesses the system via a web interface called Jupyter. This interface allows access through programming notebooks (python, R, ...). The user programs directly in the notebook, remotely accessing the data, avoiding potential data loss accidents.

Adapt.AR: federal resource

Currently, work is being done with the National Meteorological Service ([1]) and the National Water Institute (argentina.gob.ar/ina INA), to send a financing proposal within the Program of Scientific and Technological Facilities (FCT) of Strategic and Interinstitutional Interest, for the creation of a Regional Climate Data Center (Aadpt.AR) to provide 'on-site' access and diagnosis service at the Federal level. See more details in this presentation of the project made at CIMA/DCAO in October 2022. The implementation of Adapt.AR

papa-deimos is the prototype tool that helps us learn the necessary requirements of equipment with these characteristics.

The international climate community is beginning to implement these solutions with similar very high capacity systems. The equipment called JASMIN of the Met Office (United Kingdom meteorological service) serves as an example and reference.

The papa-deimos system could be established in part, thanks to funds from INSU - / LEFE

use

Steps to follow for the use of papa-deimos.

Existing databases

Table of data accessible in papa-deimos.

Technical specifications

Description técnica of papa-deimos.

Commissioning of the 'papa-deimos' system

Steps followed for the installation of papa-deimos.