visualdynamics.io.escdf

escdf

The Engineering Sciences Common Data Format, read and written by its specification.

ESCDF (github.com/sandialabs/Engineering-Sciences-Common-Data-Format, BSD-3-Clause) is one HDF5 file: metadata groups at the root — a geometry, a channel table, a set of parameters — and an activities group with one group per test or analysis, holding the results it produced and a parameters list naming the metadata it links to. Every dataset is a group stamped with the name of its specification, a descriptive name and a version, and carries one HDF5 dataset per property, each stamped with its type. What the types are is written in plain-text specification files, one per type, and those files are the standard: a reader that parses them reads every type there is, and a type added tomorrow is a file added tomorrow. The files ship here as package data (escdf_specifications/, with their license), and this module reads them the way the reference implementation does, from the grammar its documentation states — and was held to the reference's own files, both ways, as it was written (2026-09-30; PLAN.md "ESCDF: import and export first").

What this module is: the generic layer. Specification is a parsed file; Dataset is one group's worth of values against its specification, validated the way the reference validates — every property named, every shared dimension the same size wherever it appears, one choice of every either-or group, an enumeration honored — and File is the whole file with its activities. read and write move a File to and from disk on h5py. What it is not: the mapping from these to Visual Dynamics' own objects, which is escdf_objects.

The on-disk conventions, as measured from files the reference wrote: strings are variable-length UTF-8, a scalar string a zero-dimensional dataset; bytes are variable-length uint8; a variable_length property is a one-dimensional dataset of variable-length rows of its type; numbers are the type the specification names, u8 meaning eight bytes, so uint64, and c16 complex128; every property dataset carries a data_type attribute naming that type; every group carries _specification_name, _descriptive_name and _version (three integers); an activity group carries activity_name and activity_date (ISO 8601, UTC) and a parameters string dataset of the metadata names it links to; the file carries created_by and created_date. A choice group is written as whichever alternative was chosen, under the shared property name, and read back by which shape and type is found.

Classes:

Name	Description
Property	One property of a specification: its name, type, shape and options.
Specification	One parsed specification file.
Dataset	One dataset: a group's values against its specification.
Activity	One activity: a test or an analysis, its results and the metadata
File	A whole file: who made it and when, the metadata at its root, and

Functions:

Name	Description
valid_identifier	The format's name for a group: ASCII letters, digits and
specifications	Every type the vendored specification files define, by name,
write	Write a File as an ESCDF file, validating first.
read	Read an ESCDF file whole: every dataset against its

Classes

Property dataclass

Property(name: str, dtype: str, shape: tuple[str | int, ...] = (), optional: bool = False, variable_length: bool = False, enum: str | None = None, regex: str | None = None, choice: tuple[str, str] | None = None)

One property of a specification: its name, type, shape and options.

shape is the specification's own words: named dimensions ('num_nodes'), literal sizes (3), or () for a scalar. choice is (group, alternative) for a property in an either-or group, where exactly one alternative of the group is written.

Attributes:

Name	Type	Description
key	str	What the on-disk dataset is called: the name, shared by the

Attributes

key property

key: str

What the on-disk dataset is called: the name, shared by the alternatives of a choice group.

Specification dataclass

Specification(name: str, version: tuple[int, int, int], extends: str | None, properties: list[Property] = list(), enumerations: dict[str, list[str]] = dict(), doc: str = '', _parents: dict[str, Specification] = dict())

One parsed specification file.

Attributes: name: The type's name, the file's first word. version: Three integers from the first line's 'vX.Y.Z'. extends: The type this one inherits from, or None. properties: This file's own properties, in order. enumerations: Each enumeration's allowed values. doc: The prose between the header and the properties block.

Methods:

Name	Description
all_properties	Every property, inherited ones first, a child's redefinition
is_a	Whether this type is name or extends it.

Methods:

all_properties

all_properties() -> list[Property]

Every property, inherited ones first, a child's redefinition of a parent's name replacing the parent's (a data_type that narrows its enumeration, as response_spectrum does).

Source code in src/visualdynamics/io/escdf.py

def all_properties(self) -> list[Property]:
    """Every property, inherited ones first, a child's redefinition
    of a parent's name replacing the parent's (a `data_type` that
    narrows its enumeration, as `response_spectrum` does)."""
    chain: list[Specification] = []
    spec: Specification | None = self
    while spec is not None:
        chain.append(spec)
        spec = self._parents.get(spec.extends) if spec.extends else None
    merged: dict[str, list[Property]] = {}
    for spec in reversed(chain):
        own: dict[str, list[Property]] = {}
        for prop in spec.properties:
            own.setdefault(prop.name, []).append(prop)
        for name, props in own.items():
            merged[name] = props
    return [prop for props in merged.values() for prop in props]

is_a

is_a(name: str) -> bool

Whether this type is name or extends it.

Source code in src/visualdynamics/io/escdf.py

def is_a(self, name: str) -> bool:
    """Whether this type is `name` or extends it."""
    spec: Specification | None = self
    while spec is not None:
        if spec.name == name:
            return True
        spec = self._parents.get(spec.extends) if spec.extends else None
    return False

Dataset dataclass

Dataset(name: str, kind: str, descriptive_name: str = '', values: dict[str, Any] = dict(), version: tuple[int, int, int] | None = None, extras: dict[str, Any] = dict())

One dataset: a group's values against its specification.

Attributes: name: The group's name, a valid identifier. kind: The specification's name ('geometry', 'data', ...). descriptive_name: Free text, the reference's _descriptive_name. values: Property name to value. A string property is a str or an array of str; a bytes property a list of uint8 arrays; a variable-length property a list of arrays; numbers arrays. version: The specification version the group claims, or None to write the vendored specification's own. extras: Datasets in the group the specification does not name, kept as read so a re-export loses nothing.

Methods:

Name	Description
problems	What keeps this dataset from validating against its

Methods:

problems

problems() -> list[str]

What keeps this dataset from validating against its specification, as the reference would refuse it: a required property missing, a choice group with none or two alternatives chosen, a shared dimension of two sizes, a type the value cannot take, an enumeration or pattern not honored. Empty when valid; an unknown type is a problem of its own.

Returns:

Type	Description
list of str

Source code in src/visualdynamics/io/escdf.py

def problems(self) -> list[str]:
    """What keeps this dataset from validating against its
    specification, as the reference would refuse it: a required
    property missing, a choice group with none or two alternatives
    chosen, a shared dimension of two sizes, a type the value cannot
    take, an enumeration or pattern not honored. Empty when valid;
    an unknown type is a problem of its own.

    Returns
    -------
    list of str
    """
    spec = self.specification
    if spec is None:
        # a type nobody defined is kept as read, everything of it in
        # extras — the reference loads one as `unknown` and keeps its
        # properties too (measured 2026-09-30, at the root and in an
        # activity alike); values it has no specification for cannot
        # be checked, so it may carry none
        unknown = (f'{self.kind!r} is not a type the specifications define, '
                   'so it can carry nothing but extras')
        return [unknown] if self.values else []
    problems: list[str] = []
    props = spec.all_properties()
    enums = spec.all_enumerations()
    by_name: dict[str, list[Property]] = {}
    for prop in props:
        by_name.setdefault(prop.name, []).append(prop)
    dims: dict[str, int] = {}
    chosen: dict[str, list[str]] = {}
    for name, alternatives in by_name.items():
        value = self.values.get(name)
        if value is None:
            if all(p.optional for p in alternatives):
                continue
            if alternatives[0].choice is None:
                problems.append(f'{name} is required')
            continue
        matches = [p for p in alternatives if _fits(p, value)]
        if not matches:
            problems.append(f'{name}: {_describe(value)} does not fit '
                            + ' or '.join(_shape_text(p) for p in alternatives))
            continue
        prop = matches[0]
        if prop.choice is not None:
            chosen.setdefault(prop.choice[0], []).append(prop.choice[1])
        shape = _value_shape(value, prop)
        for axis, size in zip(prop.shape, shape):
            if isinstance(axis, str) and dims.setdefault(axis, size) != size:
                problems.append(f'{name}: {axis} is {size} here and '
                                f'{dims[axis]} elsewhere')
        if prop.enum is not None:
            allowed = set(enums.get(prop.enum, []))
            for item in np.asarray(value, dtype=object).ravel():
                if str(item) not in allowed:
                    problems.append(f'{name}: {item!r} is not one of '
                                    + ', '.join(sorted(allowed)))
                    break
        if prop.regex is not None:
            pattern = re.compile(prop.regex)
            for item in np.asarray(value, dtype=object).ravel():
                if not pattern.match(str(item)):
                    problems.append(f'{name}: {item!r} does not match '
                                    f'{prop.regex}')
                    break
    groups: dict[str, set[str]] = {}
    for prop in props:
        if prop.choice is not None:
            groups.setdefault(prop.choice[0], set()).add(prop.choice[1])
    for group, alternatives in groups.items():
        picked = set(chosen.get(group, []))
        # one alternative may cover several properties (even spacing
        # is a start and a step); every property of it must be there
        if not picked:
            if any(p.optional for p in props if p.choice and p.choice[0] == group):
                continue
            problems.append(f'choice {group}: none of '
                            + ', '.join(sorted(alternatives)) + ' is given')
        elif len(picked) > 1:
            problems.append(f'choice {group}: both ' + ' and '.join(sorted(picked)))
        else:
            alternative = next(iter(picked))
            wanted = [p.name for p in props
                      if p.choice == (group, alternative)]
            missing = [n for n in wanted if n not in self.values]
            if missing:
                problems.append(f'choice {group}/{alternative} also needs '
                                + ', '.join(missing))
    return problems

Activity dataclass

Activity(name: str, descriptive_name: str = '', date: datetime | None = None, links: list[str] = list(), data: dict[str, Dataset] = dict())

One activity: a test or an analysis, its results and the metadata it links to.

File dataclass

File(created_by: str = '', created_date: datetime | None = None, metadata: dict[str, Dataset] = dict(), activities: dict[str, Activity] = dict())

A whole file: who made it and when, the metadata at its root, and its activities.

Methods:

Name	Description
problems	Every dataset's problems, each prefixed with where it is, and

Methods:

problems

problems() -> list[str]

Every dataset's problems, each prefixed with where it is, and a link that names no metadata.

Source code in src/visualdynamics/io/escdf.py

def problems(self) -> list[str]:
    """Every dataset's problems, each prefixed with where it is, and
    a link that names no metadata."""
    out = [f'{name}: {p}' for name, ds in self.metadata.items()
           for p in ds.problems()]
    specs = specifications()
    for activity in self.activities.values():
        for link in activity.links:
            if link not in self.metadata:
                out.append(f'{activity.name} links to {link!r}, which is not '
                           'in the metadata')
        for name, ds in activity.data.items():
            out.extend(f'{activity.name}/{name}: {p}' for p in ds.problems())
            # the reference refuses a known type that is not a result
            # inside an activity (a parameter set belongs at the root,
            # linked); an unknown type it takes either place
            spec = specs.get(ds.kind)
            if spec is not None and not spec.is_a('activity_result'):
                out.append(f'{activity.name}/{name}: {ds.kind} is not an '
                           'activity result; it belongs in the metadata, linked')
    return out

Functions:

valid_identifier

valid_identifier(name: str, prefix: str = 'item_') -> str

The format's name for a group: ASCII letters, digits and underscores, starting with a letter — whitespace to underscores, anything else dropped, prefix in front if what is left does not start with a letter (the reference's own repair, so a name repaired here is the name it would give).

Parameters:

Name	Type	Description	Default
name	str	Any text.	required
prefix	str	What to put in front when the repaired name starts with a digit or is empty.	'item_'

Returns:

Type	Description
str

Source code in src/visualdynamics/io/escdf.py

def valid_identifier(name: str, prefix: str = 'item_') -> str:
    """The format's name for a group: ASCII letters, digits and
    underscores, starting with a letter — whitespace to underscores,
    anything else dropped, `prefix` in front if what is left does not
    start with a letter (the reference's own repair, so a name repaired
    here is the name it would give).

    Parameters
    ----------
    name : str
        Any text.
    prefix : str, default 'item_'
        What to put in front when the repaired name starts with a digit
        or is empty.

    Returns
    -------
    str
    """
    repaired = re.sub(r'[^a-zA-Z0-9_]', '', re.sub(r'\s+', '_', name))
    if not _IDENTIFIER.match(repaired):
        repaired = prefix + repaired
    return repaired

parse_specification

parse_specification(text: str) -> Specification

A specification file's text, parsed by the documented grammar.

The first line is 'name - vX.Y.Z', underlined; an 'extends: other' line names the parent; prose follows until a 'properties' header; each property line is 'name - type [- shape [- options]]', the fields split on ' - ', the shape comma-separated names or numbers or the word 'scalar', the options comma-separated from 'optional', 'variable_length', 'enum:', 'regex:' and 'or::'; an 'enumerations' block lists 'name - a, b, c'; a 'notes' or 'Examples' block ends it.

Parameters:

Name	Type	Description	Default
text	str	The file's contents.	required

Returns:

Type	Description
Specification

Source code in src/visualdynamics/io/escdf.py

def parse_specification(text: str) -> Specification:
    """A specification file's text, parsed by the documented grammar.

    The first line is 'name - vX.Y.Z', underlined; an 'extends: other'
    line names the parent; prose follows until a 'properties' header;
    each property line is 'name - type [- shape [- options]]', the
    fields split on ' - ', the shape comma-separated names or numbers or
    the word 'scalar', the options comma-separated from 'optional',
    'variable_length', 'enum:<name>', 'regex:<pattern>' and
    'or:<group>:<alternative>'; an 'enumerations' block lists
    'name - a, b, c'; a 'notes' or 'Examples' block ends it.

    Parameters
    ----------
    text : str
        The file's contents.

    Returns
    -------
    Specification
    """
    lines = text.splitlines()
    head = next(line for line in lines if line.strip())
    name, _, version = head.partition(' - ')
    numbers = tuple(int(v) for v in version.strip().lstrip('v').split('.'))
    spec = Specification(name.strip(), (numbers + (0, 0, 0))[:3], None)
    section = 'doc'
    doc: list[str] = []
    for i, line in enumerate(lines[1:]):
        stripped = line.strip()
        if set(stripped) == {'-'} and stripped:
            continue                                  # an underline
        lowered = stripped.lower()
        if lowered in ('properties', 'enumerations', 'notes', 'examples'):
            section = lowered
            continue
        if section == 'doc':
            if lowered.startswith('extends:'):
                parent = stripped.split(':', 1)[1].strip()
                spec.extends = None if parent.lower() == 'none' else parent
            elif stripped:
                doc.append(stripped)
        elif section == 'properties' and stripped:
            spec.properties.append(_parse_property(stripped))
        elif section == 'enumerations' and stripped:
            enum_name, _, values = stripped.partition(' - ')
            spec.enumerations[enum_name.strip()] = [
                v.strip() for v in values.split(',') if v.strip()]
    spec.doc = ' '.join(doc)
    return spec

specifications

specifications() -> dict[str, Specification]

Every type the vendored specification files define, by name, each knowing its parents.

Returns:

Type	Description
dict of str to Specification

Source code in src/visualdynamics/io/escdf.py

def specifications() -> dict[str, Specification]:
    """Every type the vendored specification files define, by name,
    each knowing its parents.

    Returns
    -------
    dict of str to Specification
    """
    global _SPECIFICATIONS
    if _SPECIFICATIONS is None:
        _SPECIFICATIONS = _package_specifications()
    return _SPECIFICATIONS

write

write(file: File, path: str | PathLike) -> None

Write a File as an ESCDF file, validating first.

Parameters:

Name	Type	Description	Default
file	File	What to write; file.problems() must be empty.	required
path	path - like	The file to write; replaced if it exists.	required

Source code in src/visualdynamics/io/escdf.py

def write(file: File, path: str | os.PathLike) -> None:
    """Write a `File` as an ESCDF file, validating first.

    Parameters
    ----------
    file : File
        What to write; `file.problems()` must be empty.
    path : path-like
        The file to write; replaced if it exists.
    """
    problems = file.problems()
    if problems:
        raise ValueError('not a valid ESCDF file:\n  ' + '\n  '.join(problems))
    with h5py.File(path, 'w') as h5:
        h5.attrs['created_by'] = file.created_by or 'unknown'
        h5.attrs['created_date'] = _iso(file.created_date)
        for dataset in file.metadata.values():
            _write_dataset(h5, dataset)
        activities = h5.create_group('activities')
        for activity in file.activities.values():
            group = activities.create_group(activity.name)
            group.attrs['activity_name'] = activity.descriptive_name
            group.attrs['activity_date'] = _iso(activity.date)
            links = group.create_dataset('parameters', (len(activity.links),),
                                         dtype=h5py.string_dtype('utf-8'))
            if activity.links:
                links[...] = np.asarray(activity.links, dtype=object)
            links.attrs['data_type'] = 'str'
            for dataset in activity.data.values():
                _write_dataset(group, dataset)

read

read(path: str | PathLike) -> File

Read an ESCDF file whole: every dataset against its specification, unknown types and undefined properties kept as they are.

Parameters:

Name	Type	Description	Default
path	path - like	The file.	required

Returns:

Type	Description
File

Source code in src/visualdynamics/io/escdf.py

def read(path: str | os.PathLike) -> File:
    """Read an ESCDF file whole: every dataset against its
    specification, unknown types and undefined properties kept as they
    are.

    Parameters
    ----------
    path : path-like
        The file.

    Returns
    -------
    File
    """
    file = File()
    with h5py.File(path, 'r') as h5:
        file.created_by = _text(h5.attrs.get('created_by', ''))
        file.created_date = _from_iso(h5.attrs.get('created_date', ''))
        for name, item in h5.items():
            if name == 'activities' or not isinstance(item, h5py.Group):
                continue
            file.metadata[name] = _read_dataset(item)
        for name, group in h5.get('activities', {}).items():
            if not isinstance(group, h5py.Group):
                continue
            activity = Activity(name, _text(group.attrs.get('activity_name', '')),
                                _from_iso(group.attrs.get('activity_date', '')))
            if 'parameters' in group:
                activity.links = [_text(v) for v in np.asarray(group['parameters'][()]).ravel()]
            for item_name, item in group.items():
                if isinstance(item, h5py.Group):
                    activity.data[item_name] = _read_dataset(item)
            file.activities[name] = activity
    return file

sniff

sniff(path: str | PathLike) -> bool

An HDF5 file with an activities group and the file's own creation stamps.

Source code in src/visualdynamics/io/escdf.py

def sniff(path: str | os.PathLike) -> bool:
    """An HDF5 file with an `activities` group and the file's own
    creation stamps."""
    try:
        with h5py.File(path, 'r') as h5:
            return 'activities' in h5 and 'created_by' in h5.attrs
    except (OSError, ValueError):
        return False

datasets_of

datasets_of(file: File, kind: str) -> Iterable[tuple[str | None, Dataset]]

Every dataset in the file that is kind or extends it, with the activity it belongs to (None for metadata), in file order.

Source code in src/visualdynamics/io/escdf.py

def datasets_of(file: File, kind: str) -> Iterable[tuple[str | None, Dataset]]:
    """Every dataset in the file that is `kind` or extends it, with the
    activity it belongs to (None for metadata), in file order."""
    specs = specifications()
    for dataset in file.metadata.values():
        spec = specs.get(dataset.kind)
        if spec is not None and spec.is_a(kind):
            yield None, dataset
    for activity in file.activities.values():
        for dataset in activity.data.values():
            spec = specs.get(dataset.kind)
            if spec is not None and spec.is_a(kind):
                yield activity.name, dataset