visualdynamics.core.run_log_files

run_log_files

Controller files: what a controller writes while a test runs, proposed for the run log, and the log's files in a report's data index.

A controller writes its records (a run's stream, its spectra, its settings) into an output folder. ControllerWatch watches one or more such folders and proposes each file that appears: it never attaches anything to the log itself. The log keeper accepts a proposal (accept), which writes an entry naming the file by SHA-256, with its format and what it holds, in the run that was current when the file appeared (place_at). A proposal not accepted leaves no trace in the log.

Only settled files are hashed. A file is settled when its size and modification time have stayed the same over settle_s seconds, seen on at least two looks at the folder; a file still being written would hash to a value its finished bytes do not have. A file that changes while it is hashed is not proposed and waits to settle again, and one removed before it settles is forgotten. Hashing is throttled to rate bytes a second, because the file sits on the controller's own disk or share and the next run is writing there too, and a stream can be tens of gigabytes. At a test's close everything seen and not yet hashed is hashed at once (ControllerWatch.archive).

Watched and polled. The folders are listed every poll_s seconds, whether or not anything says they changed, because a network share often sends no notice of a change; a front end that is told of a change (ControllerWatch.poke) only makes the next look come sooner. All of it, looking and hashing, runs on a thread of its own (ControllerWatch.start), which never holds up the program that writes the log: proposals are handed to a callback or to a queue that is safe to read from any thread.

Format and holdings. Each file is described as a data index describes a file (data_index.IndexEntry): its name, where it is, its size, its SHA-256, its format and what it holds, read through Visual Dynamics' own importers. A Rattlesnake run is read from its dimensions alone (rattlesnake.stream_summary), so a run of any size gives its channels, its sample rate and its stream's length without reading the stream; another format an importer recognizes is read whole when it is no larger than HOLDINGS_LIMIT; and the run, axis and level its name gives are added (runs.read_name). A format nothing recognizes is recorded by name, size and hash with the format 'unknown': it is never refused.

The stream and the counter. A Rattlesnake stream's length is compared with the time the run's entries counted (check_stream). The stream holds the ramps up and down, and any lower levels run inside the same run, so it is longer than the counter by however long those take; the comparison is within a stated tolerance, and a difference beyond it is a finding for the log keeper, never a refusal.

One data index. The log's files go into the project's data index through the index's own entry (DataIndex.add, the one the import and Project.index_file use), with the checksum the log recorded, so nothing is hashed twice and a report lists them beside every other file it rests on (index_run_log). The log's own file goes in with each test's head hash (RunLog.head) in what it holds, which anchors the chain inside the file to a record outside it.

Classes:

Name	Description
ControllerFilesError	A folder, a setting or a request the controller-file watch cannot
Place	Where in the log an entry goes: a test and a run by number, an
Stream	A Rattlesnake time stream's size, from its dimensions
Description	What a file is: its format ('unknown' when nothing recognizes
Proposal	A controller file proposed for the log.
WatchProblem	Something the watch could not do: the folder or file and why.
StreamCheck	A stream's length against the run's counter.
IndexedLog	What index_run_log put into a data index: the log's own entry,
ControllerWatch	A watch over one or more controller output folders.

Functions:

Name	Description
describe	What the file at path is and holds, read through Visual
place_at	The place in log that was current at when: the run started
counter_seconds	A counter as the log keeper typed it, in seconds: '1:02:03' (hours,
counted_seconds	(seconds, '') the counter ran in run run of test test, from the
check_stream	A Rattlesnake stream's length against the time its run counted.
accept	Write a proposed file into the log: an entry naming it by SHA-256,
index_run_log	Put a run log's files, and the log with each test's head hash, into

Classes

ControllerFilesError

Bases: ValueError

A folder, a setting or a request the controller-file watch cannot take; the message names the cause.

Place

Bases: NamedTuple

Where in the log an entry goes: a test and a run by number, an axis by name, any of them None.

Stream dataclass

Stream(key: str, sample_rate: float, samples: int, seconds: float, channels: int)

A Rattlesnake time stream's size, from its dimensions (rattlesnake.stream_summary): its key, sample rate in hertz, samples, length in seconds and channels.

Description dataclass

Description(format: str, holds: str, channels: tuple[str, ...] = (), run: dict[str, Any] = dict(), stream: Stream | None = None, notes: tuple[str, ...] = ())

What a file is: its format ('unknown' when nothing recognizes it), what it holds in words, its channels, the run identity its name gives (runs.read_name), a Rattlesnake time stream when it has one, and why anything was not read.

Proposal dataclass

Proposal(file: IndexEntry, folder: str, seen: datetime, channels: tuple[str, ...] = (), run: dict[str, Any] = dict(), stream: Stream | None = None, notes: tuple[str, ...] = (), archived: bool = False)

A controller file proposed for the log.

file is the file as a data index records it (IndexEntry, its source the run log), hashed once it settled. folder is the watched folder it appeared in and seen when it was first seen there, with its zone; the run current at seen is where it is proposed to go (place_at). archived marks a file hashed at a test's close rather than once it had settled. The rest is its Description.

Methods:

Name	Description
file_ref	The file as a log entry names it: name, SHA-256, size, format,

Attributes:

Name	Type	Description
name	str	The file's name.
path	str	Where the file was when it was hashed.
sha256	str	The file's SHA-256, as 64 lower-case hex digits.

Attributes

name property

name: str

The file's name.

path property

path: str

Where the file was when it was hashed.

sha256 property

sha256: str

The file's SHA-256, as 64 lower-case hex digits.

Methods:

file_ref

file_ref() -> FileRef

The file as a log entry names it: name, SHA-256, size, format, holdings and path.

Source code in src/visualdynamics/core/run_log_files.py

def file_ref(self) -> FileRef:
    """The file as a log entry names it: name, SHA-256, size, format,
    holdings and path."""
    return FileRef(self.file.name, self.file.sha256, self.file.size,
                   self.file.format, self.file.holds, self.file.path)

WatchProblem dataclass

WatchProblem(path: str, reason: str)

Something the watch could not do: the folder or file and why. The watch goes on; the problem is reported once until it clears.

StreamCheck dataclass

StreamCheck(ok: bool | None, message: str, tolerance_s: float, stream_s: float | None = None, sample_rate: float | None = None, counted_s: float | None = None)

A stream's length against the run's counter.

ok is True within the tolerance, False beyond it, and None when nothing was compared (no time stream, no run, no counter), which is never a pass. stream_s and sample_rate are the stream's, counted_s the counter's, tolerance_s the difference allowed, and message says what was found in words.

IndexedLog dataclass

IndexedLog(log: IndexEntry, files: tuple[IndexEntry, ...], missing: tuple[tuple[FileRef, str], ...])

What index_run_log put into a data index: the log's own entry, each file's entry, and each file not found, with why.

ControllerWatch

ControllerWatch(folders: str | PathLike | Sequence[str | PathLike], *, settle_s: float = SETTLE_S, poll_s: float = POLL_S, rate: float | None = RATE, limit: int = HOLDINGS_LIMIT, existing: bool = False, clock: Callable[[], float] = monotonic, now: Callable[[], datetime] = _local_now)

A watch over one or more controller output folders.

Each look (scan) lists the folders, notes each file that is new or has changed since the watch began, and hashes each one that has settled, returning a Proposal for it; what could not be done comes back as a WatchProblem. start runs the looks on a thread of the watch's own every poll_s seconds and hands what they find to callbacks, or to a queue read with take; stop ends it.

The files already in a folder when the watch begins are the records of earlier runs and are not proposed, unless existing is set; one of them that changes afterwards is.

Parameters:

Name	Type	Description	Default
folders	str, os.PathLike, or a sequence of them	The folders to watch. Each must exist when the watch begins.	required
settle_s	float	Seconds a file's size and modification time must stay the same before it is hashed.	SETTLE_S
poll_s	float	Seconds between looks when the watch runs on its thread.	POLL_S
rate	float or None	Bytes a second hashing reads at most; None for no limit.	RATE
limit	int	The largest file read whole for its holdings (describe).	HOLDINGS_LIMIT
existing	bool	Propose the files already in the folders as well.	False
clock	callable	Seconds on a monotonic clock, which settling is timed by.	monotonic
now	callable	The time with its zone, which a file is recorded as first seen at.	_local_now

Raises:

Type	Description
ControllerFilesError	When no folder is named, a folder does not exist or is a file, or a setting is not a number it can be.

Methods:

Name	Description
scan	Look at the folders once, and hash and propose each file that
archive	Look at the folders once more and hash every file seen and not
start	Run the looks on a thread of the watch's own, every poll_s
poke	Look now rather than at the next poll: what a notice that a
ask_archive	Have the thread hash every file not yet proposed at its next
stop	End the thread: a hash in progress is abandoned at its next
take	Everything the thread has found since the last call, when it

Attributes:

Name	Type	Description
pending	tuple[str, ...]	The files seen and not yet proposed, by path.
running	bool	Whether the watch's thread is running.

Source code in src/visualdynamics/core/run_log_files.py

def __init__(self, folders: str | os.PathLike
             | Sequence[str | os.PathLike], *,
             settle_s: float = SETTLE_S, poll_s: float = POLL_S,
             rate: float | None = RATE, limit: int = HOLDINGS_LIMIT,
             existing: bool = False,
             clock: Callable[[], float] = time.monotonic,
             now: Callable[[], datetime] = _local_now) -> None:
    if isinstance(folders, (str, os.PathLike)):
        folders = [folders]
    named = list(dict.fromkeys(os.path.abspath(os.fspath(folder))
                               for folder in folders))
    if not named:
        raise ControllerFilesError('name at least one controller output '
                                   'folder to watch')
    for folder in named:
        if os.path.isfile(folder):
            raise ControllerFilesError(f'{folder} is a file, not a '
                                       'folder')
        if not os.path.isdir(folder):
            raise ControllerFilesError(
                f'no folder at {folder}: a controller output folder is '
                'watched once it exists')
    self.folders: tuple[str, ...] = tuple(named)
    self.settle_s: float = _finite(settle_s, 'the settling time')
    self.poll_s: float = _finite(poll_s, 'the polling interval', positive=True)
    if rate is not None:
        _finite(rate, 'the hashing rate', positive=True)
    self.rate: float | None = None if rate is None else float(rate)
    self.limit: int = limit
    self._clock = clock
    self._now = now
    self._known: dict[str, tuple[int, int]] = {}
    self._pending: dict[str, _Seen] = {}
    self._failing: set[str] = set()
    self._looking = threading.Lock()
    self._stopping = threading.Event()
    self._wake = threading.Event()
    self._archive_asked = threading.Event()
    self._thread: threading.Thread | None = None
    self._queue: queue.Queue = queue.Queue()
    self._on_proposal: Callable[[Proposal], Any] | None = None
    self._on_problem: Callable[[WatchProblem], Any] | None = None
    if not existing:
        for folder in self.folders:
            for path, stat in self._listing(folder) or ():
                self._known[path] = _key(stat)

Attributes

pending property

pending: tuple[str, ...]

The files seen and not yet proposed, by path.

running property

running: bool

Whether the watch's thread is running.

Methods:

scan

scan() -> list[Proposal | WatchProblem]

Look at the folders once, and hash and propose each file that has settled.

Returns:

Type	Description
list of Proposal and WatchProblem	What this look found, in the order found.

Source code in src/visualdynamics/core/run_log_files.py

def scan(self) -> list[Proposal | WatchProblem]:
    """Look at the folders once, and hash and propose each file that
    has settled.

    Returns
    -------
    list of Proposal and WatchProblem
        What this look found, in the order found.
    """
    with self._looking:
        found: list[Proposal | WatchProblem] = list(self._look())
        for path in self._settled():
            found += self._hash(path, archived=False)
        return found

archive

archive() -> list[Proposal | WatchProblem]

Look at the folders once more and hash every file seen and not yet proposed, settled or not: what a test's close asks, so the files of its last run are named before it is archived. A file that changes while it is hashed is reported, not proposed.

Returns:

Type	Description
list of Proposal and WatchProblem	What was found, in the order found.

Source code in src/visualdynamics/core/run_log_files.py

def archive(self) -> list[Proposal | WatchProblem]:
    """Look at the folders once more and hash every file seen and not
    yet proposed, settled or not: what a test's close asks, so the
    files of its last run are named before it is archived. A file
    that changes while it is hashed is reported, not proposed.

    Returns
    -------
    list of Proposal and WatchProblem
        What was found, in the order found.
    """
    with self._looking:
        found: list[Proposal | WatchProblem] = list(self._look())
        for path in list(self._pending):
            found += self._hash(path, archived=True)
        return found

start

start(on_proposal: Callable[[Proposal], Any] | None = None, on_problem: Callable[[WatchProblem], Any] | None = None) -> None

Run the looks on a thread of the watch's own, every poll_s seconds, until stop.

The callbacks are called on that thread; a front end hands what they get to its own thread (a queued signal). Without them, what is found goes to a queue that take reads from any thread.

Parameters:

Name	Type	Description	Default
on_proposal	callable	Called with each Proposal.	None
on_problem	callable	Called with each WatchProblem.	None

Returns:

Type	Description
None

Raises:

Type	Description
ControllerFilesError	When the watch is running already.

Source code in src/visualdynamics/core/run_log_files.py

def start(self, on_proposal: Callable[[Proposal], Any] | None = None,
          on_problem: Callable[[WatchProblem], Any] | None = None
          ) -> None:
    """Run the looks on a thread of the watch's own, every `poll_s`
    seconds, until `stop`.

    The callbacks are called on that thread; a front end hands what
    they get to its own thread (a queued signal). Without them, what
    is found goes to a queue that `take` reads from any thread.

    Parameters
    ----------
    on_proposal : callable, optional
        Called with each `Proposal`.
    on_problem : callable, optional
        Called with each `WatchProblem`.

    Returns
    -------
    None

    Raises
    ------
    ControllerFilesError
        When the watch is running already.
    """
    if self.running:
        raise ControllerFilesError('the watch is running already')
    self._on_proposal, self._on_problem = on_proposal, on_problem
    self._stopping.clear()
    self._thread = threading.Thread(target=self._run, daemon=True,
                                    name='run-log-files')
    self._thread.start()

poke

poke() -> None

Look now rather than at the next poll: what a notice that a folder changed asks. Safe from any thread.

Source code in src/visualdynamics/core/run_log_files.py

def poke(self) -> None:
    """Look now rather than at the next poll: what a notice that a
    folder changed asks. Safe from any thread."""
    self._wake.set()

ask_archive

ask_archive() -> None

Have the thread hash every file not yet proposed at its next look, which comes at once (archive). Safe from any thread.

Source code in src/visualdynamics/core/run_log_files.py

def ask_archive(self) -> None:
    """Have the thread hash every file not yet proposed at its next
    look, which comes at once (`archive`). Safe from any thread."""
    self._archive_asked.set()
    self._wake.set()

stop

stop(timeout: float | None = 10.0) -> bool

End the thread: a hash in progress is abandoned at its next chunk. Safe to call when it is not running.

Parameters:

Name	Type	Description	Default
timeout	float or None	Seconds to wait for the thread to end; None to wait as long as it takes.	10.0

Returns:

Type	Description
bool	Whether the thread has ended.

Source code in src/visualdynamics/core/run_log_files.py

def stop(self, timeout: float | None = 10.0) -> bool:
    """End the thread: a hash in progress is abandoned at its next
    chunk. Safe to call when it is not running.

    Parameters
    ----------
    timeout : float or None, optional
        Seconds to wait for the thread to end; None to wait as long
        as it takes.

    Returns
    -------
    bool
        Whether the thread has ended.
    """
    self._stopping.set()
    self._wake.set()
    thread = self._thread
    if thread is not None and thread is not threading.current_thread():
        thread.join(timeout)
    ended = thread is None or not thread.is_alive()
    if ended:
        self._thread = None
        # the callbacks hold the front end's objects; let go of them
        self._on_proposal = self._on_problem = None
    return ended

take

take() -> list[Proposal | WatchProblem]

Everything the thread has found since the last call, when it runs without callbacks. Safe from any thread.

Returns:

Type	Description
list of Proposal and WatchProblem	In the order found.

Source code in src/visualdynamics/core/run_log_files.py

def take(self) -> list[Proposal | WatchProblem]:
    """Everything the thread has found since the last call, when it
    runs without callbacks. Safe from any thread.

    Returns
    -------
    list of Proposal and WatchProblem
        In the order found.
    """
    taken = []
    while True:
        try:
            taken.append(self._queue.get_nowait())
        except queue.Empty:
            return taken

Functions:

describe

describe(path: str | PathLike, *, limit: int = HOLDINGS_LIMIT) -> Description

What the file at path is and holds, read through Visual Dynamics' importers.

A Rattlesnake run gives its kind, channels, sample rate and time stream from its dimensions, whatever its size. Another format an importer recognizes is read whole when it is at most limit bytes, and its holdings are the data index's (project.holds). A file nothing recognizes has the format 'unknown'. The run, axis, level and start its name gives are added. Nothing here raises for a file that cannot be read: what was not read, and why, is in notes and in the holdings.

Parameters:

Name	Type	Description	Default
path	str or PathLike	The file.	required
limit	int	The largest file read whole for its holdings, in bytes.	HOLDINGS_LIMIT

Returns:

Type	Description
Description	Its format, holdings, channels, run identity and stream.

Source code in src/visualdynamics/core/run_log_files.py

def describe(path: str | os.PathLike, *,
             limit: int = HOLDINGS_LIMIT) -> Description:
    """What the file at `path` is and holds, read through Visual
    Dynamics' importers.

    A Rattlesnake run gives its kind, channels, sample rate and time
    stream from its dimensions, whatever its size. Another format an
    importer recognizes is read whole when it is at most `limit` bytes,
    and its holdings are the data index's (`project.holds`). A file
    nothing recognizes has the format 'unknown'. The run, axis, level
    and start its name gives are added. Nothing here raises for a file
    that cannot be read: what was not read, and why, is in `notes` and
    in the holdings.

    Parameters
    ----------
    path : str or os.PathLike
        The file.
    limit : int, optional
        The largest file read whole for its holdings, in bytes.

    Returns
    -------
    Description
        Its format, holdings, channels, run identity and stream.
    """
    from .. import io
    from ..io import rattlesnake
    from .runs import read_name

    path = os.fspath(path)
    name = os.path.basename(path)
    run, _refused = read_name(name)
    known = io.known_format(path)
    words: list[str] = []
    notes: list[str] = []
    channels: tuple[str, ...] = ()
    stream = None
    try:
        is_rattlesnake = known is not None and rattlesnake.sniff(path)
    except (OSError, ValueError):
        is_rattlesnake = False
    try:
        if known is None:
            notes.append('no importer recognizes its format; it is recorded '
                         'by name, size and SHA-256')
        elif is_rattlesnake:
            words, channels, stream = _rattlesnake(path)
        elif os.path.getsize(path) > limit:
            notes.append(f'larger than {limit} bytes, so it was not read for '
                         'its channels')
        else:
            words, channels = _imported(path)
    except Exception as error:  # noqa: BLE001 — any reader's failure is a note
        notes.append(f'its holdings could not be read ({error})')
    said = _name_words(run)
    if said:
        words.append(said)
    holds = '; '.join(words) or 'not read: ' + '; '.join(notes)
    return Description(known or UNKNOWN, holds, channels, dict(run), stream,
                       tuple(notes))

place_at

place_at(log: RunLog, when: datetime) -> Place

The place in log that was current at when: the run started last at or before it (its test, axis and run), or the test opened last at or before it when that came after, or no place.

A file that appears after a run stops still belongs to that run, since a controller writes its records as the run ends; one that appears after the next test opens belongs to that test.

Parameters:

Name	Type	Description	Default
log	RunLog	The log, read in its corrected view.	required
when	datetime	The time, with its zone.	required

Returns:

Type	Description
Place	The test, axis and run, any of them None.

Source code in src/visualdynamics/core/run_log_files.py

def place_at(log: RunLog, when: datetime) -> Place:
    """The place in `log` that was current at `when`: the run started
    last at or before it (its test, axis and run), or the test opened
    last at or before it when that came after, or no place.

    A file that appears after a run stops still belongs to that run,
    since a controller writes its records as the run ends; one that
    appears after the next test opens belongs to that test.

    Parameters
    ----------
    log : RunLog
        The log, read in its corrected view.
    when : datetime
        The time, with its zone.

    Returns
    -------
    Place
        The test, axis and run, any of them None.
    """
    best: Entry | None = None
    for entry in log.view():
        if entry.kind not in ('run_start', 'test_opened') \
                or entry.event_time > when:
            continue
        if best is None or entry.event_time >= best.event_time:
            best = entry
    if best is None:
        return Place(None, None, None)
    if best.kind == 'test_opened':
        return Place(best.test, None, None)
    return Place(best.test, best.axis, best.run)

counter_seconds

counter_seconds(text: str) -> float

A counter as the log keeper typed it, in seconds: '1:02:03' (hours, minutes, seconds), '12:30' (minutes, seconds), '750', '750 s', '12.5 min' or '1 h'.

Parameters:

Name	Type	Description	Default
text	str	The counter.	required

Returns:

Type	Description
float	Seconds.

Raises:

Type	Description
ControllerFilesError	When the text is none of those forms, or a clock's minutes or seconds are 60 or more; the message names the text.

Source code in src/visualdynamics/core/run_log_files.py

def counter_seconds(text: str) -> float:
    """A counter as the log keeper typed it, in seconds: '1:02:03' (hours,
    minutes, seconds), '12:30' (minutes, seconds), '750', '750 s',
    '12.5 min' or '1 h'.

    Parameters
    ----------
    text : str
        The counter.

    Returns
    -------
    float
        Seconds.

    Raises
    ------
    ControllerFilesError
        When the text is none of those forms, or a clock's minutes or
        seconds are 60 or more; the message names the text.
    """
    said = str(text).strip()
    clock = _CLOCK.fullmatch(said)
    if clock:
        hours, minutes, seconds = clock.groups()
        if (hours is not None and int(minutes) >= 60) or float(seconds) >= 60:
            raise ControllerFilesError(f'counter {text!r}: minutes and '
                                       'seconds on a clock run to 59')
        return (int(hours or 0) * 3600.0 + int(minutes) * 60.0
                + float(seconds))
    amount = _AMOUNT.fullmatch(said)
    if amount and amount.group(2).lower() in _UNITS:
        return float(amount.group(1)) * _UNITS[amount.group(2).lower()]
    raise ControllerFilesError(
        f'counter {text!r} is not a time: write it as 1:02:03, 12:30, '
        '750 s or 12.5 min')

counted_seconds

counted_seconds(log: RunLog, test: int, run: int) -> tuple[float | None, str]

(seconds, '') the counter ran in run run of test test, from the run's entries: the last counter recorded at a stop or abort less the counter at its start (0 when its start has none). (None, why) when it cannot be told.

Parameters:

Name	Type	Description	Default
log	RunLog	The log, read in its corrected view.	required
test	int	The test and run, by number.	required
run	int	The test and run, by number.	required

Returns:

Type	Description
tuple of (float or None, str)	The seconds counted, or None and the reason.

Source code in src/visualdynamics/core/run_log_files.py

def counted_seconds(log: RunLog, test: int, run: int
                    ) -> tuple[float | None, str]:
    """(seconds, '') the counter ran in run `run` of test `test`, from the
    run's entries: the last counter recorded at a stop or abort less the
    counter at its start (0 when its start has none). (None, why) when it
    cannot be told.

    Parameters
    ----------
    log : RunLog
        The log, read in its corrected view.
    test, run : int
        The test and run, by number.

    Returns
    -------
    tuple of (float or None, str)
        The seconds counted, or None and the reason.
    """
    start = 0.0
    end: float | None = None
    found = False
    for entry in log.view():
        if entry.test != test or entry.run != run:
            continue
        found = True
        counter = entry.fields.get('counter')
        if counter is None or entry.kind not in ('run_start', 'run_stop',
                                                 'abort'):
            continue
        try:
            seconds = counter_seconds(counter)
        except ControllerFilesError as error:
            return None, f'entry #{entry.id}: {error}'
        if entry.kind == 'run_start':
            start = seconds
        else:
            end = seconds
    if not found:
        return None, f'the log has no entries for run {run} of test {test}'
    if end is None:
        return None, (f'run {run} of test {test} has no counter recorded at '
                      'its stop')
    return end - start, ''

check_stream

check_stream(log: RunLog, proposal: Proposal, *, place: Place | None = None, tolerance_s: float = STREAM_TOLERANCE_S) -> StreamCheck

A Rattlesnake stream's length against the time its run counted.

The stream runs from the controller's start to its stop, ramps included, so it is expected to be somewhat longer than the counter. A difference of more than tolerance_s either way is a finding for the log keeper: shorter, the stream may not hold the whole run; longer, more ran than the counter and its ramps account for. Nothing is refused, and when nothing can be compared the check says so and is not a pass.

Parameters:

Name	Type	Description	Default
log	RunLog	The log the run's counter is read from (counted_seconds).	required
proposal	Proposal	The file, with its stream (describe).	required
place	Place	The run it belongs to; the run current when it was first seen (place_at) unless given.	None
tolerance_s	float	Seconds the stream may differ from the counter.	STREAM_TOLERANCE_S

Returns:

Type	Description
StreamCheck	What was compared and found.

Raises:

Type	Description
ControllerFilesError	When the tolerance is not a number of seconds, 0 or more.

Source code in src/visualdynamics/core/run_log_files.py

def check_stream(log: RunLog, proposal: Proposal, *,
                 place: Place | None = None,
                 tolerance_s: float = STREAM_TOLERANCE_S) -> StreamCheck:
    """A Rattlesnake stream's length against the time its run counted.

    The stream runs from the controller's start to its stop, ramps
    included, so it is expected to be somewhat longer than the counter.
    A difference of more than `tolerance_s` either way is a finding for
    the log keeper: shorter, the stream may not hold the whole run;
    longer, more ran than the counter and its ramps account for. Nothing
    is refused, and when nothing can be compared the check says so and
    is not a pass.

    Parameters
    ----------
    log : RunLog
        The log the run's counter is read from (`counted_seconds`).
    proposal : Proposal
        The file, with its stream (`describe`).
    place : Place, optional
        The run it belongs to; the run current when it was first seen
        (`place_at`) unless given.
    tolerance_s : float, optional
        Seconds the stream may differ from the counter.

    Returns
    -------
    StreamCheck
        What was compared and found.

    Raises
    ------
    ControllerFilesError
        When the tolerance is not a number of seconds, 0 or more.
    """
    tolerance = _finite(tolerance_s, 'the tolerance')
    name = proposal.name
    stream = proposal.stream
    if stream is None:
        return StreamCheck(None, f'{name} holds no time stream, so nothing '
                           'was compared with the counter', tolerance)
    where = place if place is not None else place_at(log, proposal.seen)
    rate, length = stream.sample_rate, stream.seconds
    if where.test is None or where.run is None:
        return StreamCheck(None, f'{name} is not in a run of the log, so its '
                           f'{length:.1f} s stream was not compared with a '
                           'counter', tolerance, length, rate)
    counted, why = counted_seconds(log, where.test, where.run)
    if counted is None:
        return StreamCheck(None, f'{name}: {why}, so its {length:.1f} s '
                           'stream was not compared with a counter',
                           tolerance, length, rate)
    held = (f'{name}: the stream holds {length:.1f} s at {rate:.6g} Hz '
            f'against {counted:.1f} s counted in run {where.run} of test '
            f'{where.test}')
    over = length - counted
    if over < -tolerance:
        return StreamCheck(False, f'{held}, {-over:.1f} s shorter, more than '
                           f'the {tolerance:g} s allowed: the stream may not '
                           'hold the whole run', tolerance, length, rate,
                           counted)
    if over > tolerance:
        return StreamCheck(False, f'{held}, {over:.1f} s longer, more than '
                           f'the {tolerance:g} s allowed for its ramps',
                           tolerance, length, rate, counted)
    return StreamCheck(True, f'{held}, within the {tolerance:g} s allowed',
                       tolerance, length, rate, counted)

accept

accept(log: RunLog, proposal: Proposal, *, author: str, place: Place | None = None, kind: str = 'note', text: str = '') -> Entry

Write a proposed file into the log: an entry naming it by SHA-256, with its size, format, holdings and path, at the time it was first seen, in the run current then.

Parameters:

Name	Type	Description	Default
log	RunLog	The log, open for writing.	required
proposal	Proposal	The file.	required
author	str	The log keeper's initials.	required
place	Place	Where it goes; the run current when it was first seen (place_at) unless given.	None
kind	str	The kind of entry, a note unless given.	'note'
text	str	Free text.	''

Returns:

Type	Description
Entry	The entry written.

Raises:

Type	Description
ControllerFilesError	When the log already names this file (the same name and SHA-256); the message names the entry.
RunLogError	For anything the log refuses (RunLog.append).

Source code in src/visualdynamics/core/run_log_files.py

def accept(log: RunLog, proposal: Proposal, *, author: str,
           place: Place | None = None, kind: str = 'note',
           text: str = '') -> Entry:
    """Write a proposed file into the log: an entry naming it by SHA-256,
    with its size, format, holdings and path, at the time it was first
    seen, in the run current then.

    Parameters
    ----------
    log : RunLog
        The log, open for writing.
    proposal : Proposal
        The file.
    author : str
        The log keeper's initials.
    place : Place, optional
        Where it goes; the run current when it was first seen
        (`place_at`) unless given.
    kind : str, optional
        The kind of entry, a note unless given.
    text : str, optional
        Free text.

    Returns
    -------
    Entry
        The entry written.

    Raises
    ------
    ControllerFilesError
        When the log already names this file (the same name and
        SHA-256); the message names the entry.
    RunLogError
        For anything the log refuses (`RunLog.append`).
    """
    for entry in log.entries:
        for ref in entry.files:
            if ref.sha256 == proposal.sha256 and ref.name == proposal.name:
                raise ControllerFilesError(f'{proposal.name} is in the log '
                                           f'already, entry #{entry.id}')
    where = place if place is not None else place_at(log, proposal.seen)
    return log.append(kind, author=author, text=text,
                      event_time=proposal.seen, test=where.test,
                      axis=where.axis, run=where.run,
                      files=[proposal.file_ref()])

index_run_log

index_run_log(index: DataIndex, log: RunLog, *, tests: Iterable[int | None] | None = None, search: Sequence[str | PathLike] = ()) -> IndexedLog

Put a run log's files, and the log with each test's head hash, into a data index (Project.data_index), so a report lists them.

Each file the log names is entered through the index's own entry (DataIndex.add) with the size and SHA-256 the log recorded, so it is not hashed again; the index's check before a report is issued (DataIndex.verify) hashes it and says whether it still matches. A file is looked for where it was when it was hashed, then by its name in each folder of search, then beside the log; a file found nowhere is listed as missing, since an index entry names a file that is there. The log's own file is entered as a run log holding each test's head (RunLog.head), replacing the run log's earlier entry for the same file, since the log grows.

Parameters:

Name	Type	Description	Default
index	DataIndex	The index, the project's data index in a project.	required
log	RunLog	The log.	required
tests	iterable of int or None	The tests whose files and heads go in (None for the entries between tests); every test, and the entries between tests, unless given.	None
search	sequence of str or os.PathLike	Folders a file is looked for in by its name.	()

Returns:

Type	Description
IndexedLog	The log's entry, each file's entry, and each file not found.

Raises:

Type	Description
ControllerFilesError	When a test named is not in the log.

Source code in src/visualdynamics/core/run_log_files.py

def index_run_log(index: DataIndex, log: RunLog, *,
                  tests: Iterable[int | None] | None = None,
                  search: Sequence[str | os.PathLike] = ()) -> IndexedLog:
    """Put a run log's files, and the log with each test's head hash, into
    a data index (`Project.data_index`), so a report lists them.

    Each file the log names is entered through the index's own entry
    (`DataIndex.add`) with the size and SHA-256 the log recorded, so it
    is not hashed again; the index's check before a report is issued
    (`DataIndex.verify`) hashes it and says whether it still matches. A
    file is looked for where it was when it was hashed, then by its name
    in each folder of `search`, then beside the log; a file found
    nowhere is listed as missing, since an index entry names a file
    that is there. The log's own file is entered as a run log holding
    each test's head (`RunLog.head`), replacing the run log's earlier
    entry for the same file, since the log grows.

    Parameters
    ----------
    index : DataIndex
        The index, the project's data index in a project.
    log : RunLog
        The log.
    tests : iterable of int or None, optional
        The tests whose files and heads go in (None for the entries
        between tests); every test, and the entries between tests, unless
        given.
    search : sequence of str or os.PathLike, optional
        Folders a file is looked for in by its name.

    Returns
    -------
    IndexedLog
        The log's entry, each file's entry, and each file not found.

    Raises
    ------
    ControllerFilesError
        When a test named is not in the log.
    """
    from .. import io

    if tests is None:
        chosen: list[int | None] = list(log.tests())
        if 'log' in log.chains():
            chosen.append(None)
    else:
        chosen = list(dict.fromkeys(tests))
        held = log.tests()
        for test in chosen:
            if test is not None and test not in held:
                raise ControllerFilesError(f'the log holds no test {test}')
    folders = [os.fspath(folder) for folder in search]
    folders.append(os.path.dirname(os.path.abspath(log.path)))
    added: list[IndexEntry] = []
    missing: list[tuple[FileRef, str]] = []
    done: set[tuple[str, str]] = set()
    for entry in log.view():
        if entry.test not in chosen:
            continue
        for ref in entry.files:
            if (ref.name, ref.sha256) in done:
                continue
            done.add((ref.name, ref.sha256))
            path = _found(ref, folders)
            if path is None:
                where = f'at {ref.path} or ' if ref.path else ''
                reason = (f'{ref.name} (entry #{entry.id}) was not found '
                          f'{where}by its name in {", ".join(folders)}')
                missing.append((ref, reason))
                continue
            size = ref.size if ref.size is not None else os.path.getsize(path)
            added.append(index.add(
                path, ref.holds or f'named in the run log, entry #{entry.id}',
                format=ref.format or io.format_of(path), source=SOURCE,
                checksum=(size, ref.sha256)))
    own = os.path.abspath(log.path)
    index.entries = [item for item in index.entries
                     if not (item.source == SOURCE and os.path.normcase(
                         item.path) == os.path.normcase(own))]
    entry = index.add(own, _heads(log, chosen), format=LOG_FORMAT,
                      source=SOURCE)
    return IndexedLog(entry, tuple(added), tuple(missing))