Skip to content

imas_architecture.rst

Latest commit

 

History

History
345 lines (260 loc) · 16 KB

imas_architecture.rst

File metadata and controls

345 lines (260 loc) · 16 KB
Raw

IMAS-Python architecture

This document provides a brief overview of the components of IMAS-Python, grouped into different functional areas.

We don't aim to give detailed explanations of the code or the algorithms in it. These should be annotated in more detail in docstrings and inline comments.

Data Dictionary metadata

These classes are used to parse and represent IDS metadata from the Data Dictionary. Metadata objects are generated from a Data Dictionary XML and are (supposed to be) immutable.

Data Dictionary building and loading

The following submodules are responsible for building the Data Dictionary and loading DD definitions at runtime.

  • :py:mod:`imas.dd_helpers` handles building the IDSDef.zip file, containing all versions of the Data Dictionary since 3.22.0.
  • :py:mod:`imas.dd_zip` handles loading the Data Dictionary definitions at run time. These definitions can be loaded from an IDSDef.zip or from a custom XML file.

IDS nodes

The following submodules and classes represent IDS nodes.

Lazy instantiation

IDS nodes are instantiated only when needed. This is handled by IDSStructure.__getattr__. When a new IDS Structure is created, it initially doesn't have any IDS child nodes instantiated:

>>> import imas
>>> # Create an empty IDS
>>> cp = imas.IDSFactory().core_profiles()
>>> # Show which elements are already created:
>>> list(cp.__dict__)
['_lazy', '_children', '_parent', 'metadata', '__doc__', '_lazy_context']
>>> # When we request a child element, it is automatically created:
>>> cp.time
<IDSNumericArray (IDS:core_profiles, time, empty FLT_1D)>
>>> list(cp.__dict__)
['_lazy', '_children', '_parent', 'metadata', '__doc__', '_lazy_context',
 'time', '_toplevel']

This improves performance by creating fewer python objects: in most use cases, only a subset of the nodes in an IDS will be used. These use cases benefit a lot from lazy instantiation.

Lazy loading

:ref:`lazy loading` defers reading the data from the backend in a :py:meth:`~imas.db_entry.DBEntry.get` or :py:meth:`~imas.db_entry.DBEntry.get_slice` until the data is requested. This is handled in two places:

  1. IDSStructure.__getattr__ implements the lazy loading alongside the lazy instantiation. When a new element is created by lazy instantiation, it will call imas.db_entry_helpers._get_child to lazy load this element:
    • When the element is a data node (IDSPrimitive subclass), the data for this element is loaded from the backend.
    • When the element is another structure, nothing needs to be loaded from the backend. Instead, we store the context on the created IDSStructure and data loading is handled recursively when needed.
    • When the element is an Array of Structures, we also only store the context on the created IDSStructArray. Loading is handled as described in point 2.
  2. IDSStructArray._load implements the lazy loading of array of structures and their elements. This is triggered whenever an element is accessed (__getitem__) or the size of the Array of Structures is requested (__len__).

Creating and loading IDSs

Access Layer interfaces

MDSplus support

  • :py:mod:`imas.backends.imas_core.mdsplus_model` is responsible for creating MDSplus models. These models are specific to a DD version and are required when using the MDSplus backend for creating new Data Entries.

    .. seealso:: :ref:`MDSplus in IMAS-Python`

Versioning

IMAS-Python uses setuptools-scm for versioning. An IMAS-Python release has a corresponding tag (which sets the version). The imas._version module is generated by setuptools-scm and implements this logic for editable installs. This module is generated by setuptools-scm when building python packages.

Conversion between Data Dictionary versions

:py:mod:`imas.ids_convert` contains logic for converting an IDS between DD versions.

The :py:class:`~imas.ids_convert.DDVersionMap` class creates and contains mappings for an IDS between two Data Dictionary versions. It creates two mappings: one to be used when converting from the newer version of the two to the older version (new_to_old) and a map for the reverse (old_to_new). These mappings are of type :py:class:`~imas.ids_convert.NBCPathMap`. See its API documentation for more details.

:py:func:`~imas.ids_convert.convert_ids` is the main API method for converting IDSs between versions. It works as follows:

  • It builds a DDVersionMap between the two DD versions version and selects the correct NBCPathMap (new_to_old or old_to_new).
  • If needed, it creates a target IDS of the destination DD version.
  • It then uses the NBCPathMap to convert data and store it in the target IDS.

:py:class:`~imas.db_entry.DBEntry` can also handle automatic DD version conversion. It uses the same DDVersionMap and NBCPathMap as :py:func:`~imas.ids_convert.convert_ids`. When reading data from the backends, the NBCPathMap is used to translate between the old and the new DD version. See the implementation in :py:mod:`imas.backends.imas_core.db_entry_helpers`.

Miscelleneous

The following is a list of miscelleneous modules, which don't belong to any of the other categories on this page.

  • :py:mod:`imas.exception` contains all Exception classes that IMAS-Python may raise.

  • :py:mod:`imas.setup_logging` initializes a logging handler for IMAS-Python.

  • :py:mod:`imas.training` contains helper methods for making training data available.

  • :py:mod:`imas.util` contains useful utility methods. It is imported automatically.

    All methods requiring third party libraries (rich and scipy) are implemented in imas._util. This avoids importing these libraries immediately when a user imports imas (which can take a couple hundred milliseconds). Instead, this module is only loaded when a user needs this functionality.