`silloncore` internals

The engine, server, and storage layer. The engine is the single source of truth for all read/query/export logic used by the CLI and sillonlab.

Engine

`silloncore.engine`

`get_project_context(engine, run_names=None)`

Fetches the context summary for the project or specific runs.

This function operates in two modes: "overview" (if no run names are provided) and "specific" (if targeted run names are provided). It returns pure data dictionaries intended for downstream processing, containing absolutely no UI elements.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_names`	`list[str]`	A list of specific run names or UUIDs to query. Defaults to None, which triggers the project-wide overview mode.	`None`

Returns:

Name Type Description

dict

A dictionary containing the query mode and the formatted run data. Format:

{
    "mode": "overview" | "specific",
    "runs": [
        {
            "id": str,
            "name": str,
            "timestamp": str,
            "param_count": int,
            "asset_count": int,
            "status": str,
            # ... additional keys in specific mode (runtime, language)
        },
        ...
    ]
}

`get_run_details(engine, run_names=None, params=None, meta=None, results=None)`

Universal API to fetch deep details for specific runs.

This function extracts parameters, metadata, and results for given runs. It understands the "%all%" wildcard string inside the specific query lists to return all items of that category.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_names`	`list[str]`	A list of run names or UUIDs to target. Defaults to None.	`None`
`params`	`list[str]`	A list of parameter keys to fetch, or `["%all%"]`. Defaults to None.	`None`
`meta`	`list[str]`	A list of metadata keys to fetch, or `["%all%"]`. Defaults to None.	`None`
`results`	`list[str]`	A list of result keys to fetch, or `["%all%"]`. Defaults to None.	`None`

Returns:

Name	Type	Description
`dict`	`dict`	A populated dictionary containing the requested tracking data. Keys (`parameter`, `metadata`, `result`, `artifacts`) will only be present if they were requested.

`add_metadata_to_runs(engine, run_names, notes=None, tags=None, metadata=None)`

Appends new notes, tags, or metadata keys to existing simulation runs.

This acts as the business logic layer: it validates the input data, delegates the heavy SQL updates to the database layer, and formats the response for the CLI or frontend API.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_names`	`list[str]`	A list of run names or UUIDs to update.	required
`notes`	`list[str]`	A list of textual notes to append. Defaults to None.	`None`
`tags`	`list[str]`	A list of tags to append. Defaults to None.	`None`
`metadata`	`dict`	Metadata key/value pairs to merge into the runs. Defaults to None.	`None`

Returns:

Name	Type	Description
`dict`	`dict`	A status dictionary containing the operation result, the count of updated runs, and lists of the added items.

`resolve_run_identifier(engine, token)`

Resolves a user token to a run's uuid: exact name, exact uuid, or prefix.

Lets users refer to a run by an unambiguous uuid prefix (e.g. a3f9) in addition to its full name or uuid. An exact name wins, then an exact uuid, then a uuid that uniquely starts with the token. Ambiguous or unknown tokens resolve to None.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`token`	`str`	The name, uuid, or uuid prefix.	required

Returns:

Type	Description
	str \| None: The matched run's uuid, or None.

`get_run_snapshot(engine, run_name)`

Fetches the complete snapshot of a single run, by name, uuid, or prefix.

This is the data backbone for the interfaces (sillonlab, GUI...): one call returns every tracked field of a run, ready to be cached and displayed. Artifacts are indexed by their result name for direct lookups. A unique uuid prefix is accepted in addition to the full name/uuid.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_name`	`str`	The name, uuid, or uuid prefix of the run to load.	required

Returns:

Type	Description
`dict`	dict \| None: The full run row (parameters, results, meta_data, tag, note, runtime, status, uuid, ...) with extra `artifacts`, `figures` and `analyses` keys mapping names to their linked rows, or None if the run does not exist.

`load_run_result(storage_root, snapshot, name)`

Loads one result value of a run, wherever it was stored.

Resolution order matches how the server stored the result: 1. A dataset saved in the run's HDF5 glob is read back from disk. 2. A saved artifact resolves to the path of its stored copy. 3. Anything else (plain value, external path) is returned as stored in the database.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder holding the `glob` and `artifact` directories of the project.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The result name to load.	required

Raises:

Type	Description
`LookupError`	If the run has no result or artifact with this name.

Returns:

Name	Type	Description
`Any`		The loaded result value.

`load_run_parameter(storage_root, snapshot, name)`

Loads one parameter value of a run, wherever it was stored.

Lightweight parameters are returned straight from the database. Heavy parameter arrays were offloaded to the glob parameter group at log time (the database only holds a marker), so they are read back from disk.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder holding the `glob` directory.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The parameter name to load.	required

Raises:

Type	Description
`LookupError`	If the run has no parameter with this name.

Returns:

Name	Type	Description
`Any`		The loaded parameter value.

`load_run_artifact(storage_root, snapshot, name)`

Resolves the on-disk location of a saved artifact of a run.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder holding the `artifact` directory.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The result name the artifact was logged under.	required

Raises:

Type	Description
`LookupError`	If the run has no artifact with this name.

Returns:

Name	Type	Description
`Path`	`Path`	The path to the artifact file, or to its folder if the artifact holds several files.

`load_run_source(storage_root, snapshot)`

Loads the main script source code recorded for a run.

The source normally lives in the run's HDF5 glob; some runs carry it as a plain metadata entry instead, which is used as a fallback.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder holding the `glob` directory.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required

Returns:

Type	Description
	str \| None: The recorded source code, or None if it was not tracked.

`match_cheap(entry, parameters=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None, storage_root=None)`

Tests the database-only (glob-free) criteria against a run.

Works on either a select_run_index row or a get_run_snapshot dict — it only reads fields present in both (parameters, meta_data, tag, date, hashes, columns, and the result/artifact/analysis names). This is the cheap first phase of a query; no HDF5 file is opened (the only exception is a value condition on a glob-stored big-array parameter, which is rare).

Args mirror the value/presence criteria documented on query_runs.

Returns:

Name	Type	Description
`bool`	`bool`	True if the run satisfies every cheap criterion.

`match_heavy(storage_root, snapshot, results=None, analyses=None)`

Tests result/analysis value conditions, reading the run's glob once.

All needed datasets are read in a single file open (read_glob_many); non-glob results (artifacts, plain DB values) fall back to load_run_result.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A full run snapshot from `get_run_snapshot`.	required
`results`	`dict`	Result value/predicate conditions.	`None`
`analyses`	`dict`	Analysis value/predicate conditions.	`None`

Returns:

Name	Type	Description
`bool`	`bool`	True if the run satisfies every heavy criterion.

`match_run(storage_root, snapshot, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)`

Tests all query criteria against an in-memory run snapshot.

Cheap criteria are checked first (so the glob is only touched when a run passes them and there are result/analysis value conditions). Used by RunCollection.where, which already holds full snapshots in memory.

`query_runs(engine, storage_root=None, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)`

Finds the runs matching a set of criteria, in two phases for speed.

Phase 1 (cheap, glob-free): a single bulk index fetch (select_run_index) is filtered in memory on the database-only criteria — parameter and metadata value/predicate conditions, generic column conditions (fields), date range (before/after), and all presence filters (has_parameter, has_metadata, has_result, has_analysis, has_artifact, has_tag).

Phase 2 (heavy, only when results/analyses value conditions are given): for each survivor of phase 1, its glob is opened once to evaluate the result/analysis value conditions. So globs are read only for runs that already passed the cheap filters, and never for pure cheap queries.

Value conditions (parameters, results, analyses, metadata, fields) are dicts mapping a name to an expected value (equality) or a callable predicate. Presence filters are lists of names. All criteria combine with logical AND; with no criteria every run is returned.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root; required when filtering on result or analysis values.	`None`
`parameters`	`dict`	Parameter value/predicate conditions.	`None`
`results`	`dict`	Result value/predicate conditions.	`None`
`analyses`	`dict`	Analysis value/predicate conditions.	`None`
`metadata`	`dict`	Metadata value/predicate conditions.	`None`
`fields`	`dict`	Top-level column value/predicate conditions.	`None`
`has_parameter`	`list[str]`	Parameter names that must exist.	`None`
`has_metadata`	`list[str]`	Metadata names that must exist.	`None`
`has_result`	`list[str]`	Result/artifact names that must exist.	`None`
`has_analysis`	`list[str]`	Analysis names that must exist.	`None`
`has_artifact`	`list[str]`	Artifact names that must exist.	`None`
`has_tag`	`list[str]`	Tags the run must have.	`None`
`before`	`datetime \| str`	Exclusive upper date bound (a `YYYY-MM-DD` string parses to midnight).	`None`
`after`	`datetime \| str`	Exclusive lower date bound.	`None`

Returns:

Type	Description
`list`	list[str]: The names of the matching runs.

`fetch_run_result(storage_root, snapshot, name, dest=None)`

Fetches a result or artifact of a run as a file on disk.

Artifacts are copied to the destination. Glob results are loaded and saved as a .npy file (or .txt for plain strings). This is the engine behind "grab this result and put it in my working directory".

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The result, artifact, or figure name to fetch.	required
`dest`	`str \| Path`	The destination directory. Defaults to the current working directory.	`None`

Raises:

Type	Description
`LookupError`	If the run has no result with this name.

Returns:

Name	Type	Description
`Path`	`Path`	The path of the fetched copy.

`get_run_sizes(storage_root, snapshot)`

Measures the storage footprint of every stored item of a run.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required

Returns:

Name	Type	Description
`dict`	`dict`	Mapping of item names to `{"kind": str, "bytes": int}` where kind is one of "result", "artifact", "figure", "analysis".

`export_run(storage_root, snapshot, dest=None, format='npz')`

Exports the stored data of a run into a portable file.

Formats

"npz": every glob result and analysis in one compressed numpy file.
"npy": a folder with one .npy file per result/analysis.
"hdf5": a standalone HDF5 file with the result and analysis groups, plus the parameters and run identity stored as root attributes (the closest thing to a zip of everything).

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`dest`	`str \| Path`	Output file (or folder for "npy"). Defaults to `<run_name>_export.<ext>` in the current directory.	`None`
`format`	`str`	One of "npz", "npy", "hdf5". Defaults to "npz".	`'npz'`

Raises:

Type	Description
`ValueError`	If the format is unknown.

Returns:

Name	Type	Description
`dict`	`dict`	`{"path": Path, "exported": [names], "skipped": [names]}`.

`build_run_report(engine, storage_root, snapshot)`

Assembles a self-contained, human- and machine-readable run summary.

Gathers everything needed to understand "what a run used and did" later: parameters, results (with sizes), metadata, tags, notes, figures with their data provenance, analyses, and execution context. Pure data — no files are written here.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required

Returns:

Name	Type	Description
`dict`	`dict`	A structured manifest of the run.

`export_run_report(engine, storage_root, snapshot, dest=None, with_data=False)`

Exports a run's context as a self-contained zip bundle.

The bundle answers "what did this run use and do" and is safe to archive or share. It contains: - manifest.json — the full structured report, - report.md — a human-readable summary, - source/main.py — the run's recorded entry script (if available), - data.hdf5 — all results and analyses (only when with_data=True).

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`dest`	`str \| Path`	Output zip path. Defaults to `<run_name>_report.zip` in the current directory.	`None`
`with_data`	`bool`	Also embed the run's data as HDF5. Defaults to False.	`False`

Returns:

Name	Type	Description
`Path`	`Path`	The path of the written zip bundle.

`prune_runs(engine, storage_root, run_names=None, before=None, keep_metadata=True)`

Frees disk space by deleting the stored data of selected runs.

Targets runs either by name/uuid (run_names) or by age (before). For each target, the glob/, artifact/ and figure/ folders are removed. With keep_metadata the database rows are preserved (so the run still shows up in queries, just without its heavy data); otherwise the rows are deleted too.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root.	required
`run_names`	`list[str]`	Run names or uuids to prune.	`None`
`before`	`datetime`	Prune runs created strictly before this datetime (run dates are stored as `%Y-%m-%d-%H:%M:%S`).	`None`
`keep_metadata`	`bool`	Keep the database rows. Defaults to True.	`True`

Returns:

Name	Type	Description
`dict`	`dict`	`{"status", "pruned": [names], "freed_bytes": int, "kept_metadata": bool}`. Status is "error" if no selector is given (refuses to prune all).

`delete_run(engine, storage_root, run_name)`

Permanently deletes a single run: its stored data and its database row.

Removes the run's glob/, artifact/ and figure/ folders and deletes the SimulationTable row along with its linked artifacts, figures, and analyses. This is the irreversible "I don't want this run anymore" action (unlike prune_runs(..., keep_metadata=True), which only frees disk).

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root.	required
`run_name`	`str`	The name or uuid of the run to delete.	required

Returns:

Name	Type	Description
`dict`	`dict`	`{"status": "success", "deleted": str, "freed_bytes": int}`, or `{"status": "error", "message": str}` if the run does not exist.

`rename_run(engine, run_name, new_name)`

Renames a run, rejecting the rename if the new name is already taken.

Storage is uuid-based, so this is a pure metadata update.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_name`	`str`	The current name, uuid, or uuid prefix of the run.	required
`new_name`	`str`	The desired new name.	required

Returns:

Name	Type	Description
`dict`	`dict`	`{"status": "success", "old": ..., "new": ...}`, or `{"status": "error", "message": ...}` if the run is missing or the new name clashes with an existing run.

`find_by_hash(engine, file_or_hash)`

Finds which run(s) own a file, by content hash.

If given an existing file path, the file is hashed (get_hash); otherwise the argument is treated as a hash directly. Looks the hash up across the stored figures and artifacts.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`file_or_hash`	`str \| Path`	A file path or a SHA-256 hash.	required

Returns:

Type	Description
`list`	list[dict]: One `{run_name, run_uuid, kind, name}` per match.

`find_children(engine, uuid)`

Finds runs that inherited/derived from a given run (reverse lineage).

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`uuid`	`str`	The uuid of the parent run.	required

Returns:

Type	Description
`list`	list[dict]: One `{"name", "uuid"}` per child run.

`load_run_figure(storage_root, snapshot, name)`

Resolves the on-disk location of a figure logged during a run.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder holding the `figure` directory.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The name the figure was logged under.	required

Raises:

Type	Description
`LookupError`	If the run has no figure with this name.

Returns:

Name	Type	Description
`Path`	`Path`	The path to the figure file.

`add_run_analysis(engine, storage_root, snapshot, name, data, info=None)`

Attaches post-processed data to an already finished run.

The data is written into the run's HDF5 glob under the analysis group and a row linking it to the run (with hash and free-form context) is inserted in the database, so the processed data can be reloaded later exactly like a native result.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The analysis name.	required
`data`	`Any`	The processed data to store (h5py compatible).	required
`info`	`dict`	Free-form context (inputs used, comment...).	`None`

Returns:

Name	Type	Description
`dict`	`dict`	The inserted analysis row.

`load_run_analysis(storage_root, snapshot, name)`

Loads back a post-processed analysis attached to a run.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The project storage root.	required
`snapshot`	`dict`	A run snapshot from `get_run_snapshot`.	required
`name`	`str`	The analysis name to load.	required

Raises:

Type	Description
`LookupError`	If no analysis with this name is stored for the run.

Returns:

Name	Type	Description
`Any`		The stored analysis data.

`compare(engine, run_id1, run_id2)`

Compares two simulation runs and returns their differences.

Instantiates two Reference objects from the database and utilizes the Diffref engine to calculate the exact changes in parameters, context, and source code between the original run and the target run.

Parameters:

Name	Type	Description	Default
`engine`	`Engine`	The active SQLAlchemy database engine.	required
`run_id1`	`str`	The identifier (name or UUID) of the baseline run.	required
`run_id2`	`str`	The identifier (name or UUID) of the target run to compare.	required

Returns:

Name Type Description

dict

A dictionary containing the diff mappings. Format:

{
    "status": "success",
    "diff_param": dict,       # Parameter added/removed/changed metrics
    "diff_status": tuple,     # Status difference (if any)
    "diff_runtime": tuple,    # Runtime difference (if any)
    "diff_source": str        # Unified text diff of the source code
}

Storage (glob)

`silloncore.glob`

`Glob`

Manages the HDF5 storage backend (Glob) for a simulation run.

This class handles creating, appending to, and closing the glob.hdf5 file for a specific run. It utilizes HDF5's SWMR (Single Writer Multiple Reader) mode to allow the server to write data while external tools simultaneously read it.

Attributes:

Name	Type	Description
`path`	`Path`	The directory path where the `glob.hdf5` file is stored.
`file`	`File`	The underlying HDF5 file object.
`results`	`list`	A queue of tuples containing `(name, result_object)` waiting to be committed to disk.

`save(name, result_object)`

Queues a result to be saved and calculates its hash.

Parameters:

Name	Type	Description	Default
`name`	`str`	The target dataset name for this result.	required
`result_object`	`Any`	The data to be saved (must be compatible with h5py).	required

Returns:

Name	Type	Description
`tuple`		A tuple containing: - str: The name of the result. - str: The computed SHA-256 hash of the result object.

`save_from_staging(name, staging_path, hsh=None)`

Claims a staged array into the glob without ever loading it.

The copy happens inside HDF5 (H5Ocopy), so the bytes go staging file -> glob file without passing through this process. Reading it into python first -- and then holding it in self.results until dump -- meant the daemon's memory grew with every large array a run logged, and the staging file was already deleted, so RAM held the only copy.

Written through immediately rather than queued: there is nothing to queue, the data is already on disk.

Parameters:

Name	Type	Description	Default
`name`	`str`	Target dataset name.	required
`staging_path`	`Path`	The client's staging file.	required
`hsh`	`str`	Digest computed client-side by `write_staging_array`.	`None`

Returns:

Name	Type	Description
`tuple`		`(pointer, sha256_hash)`.

`save_param(name, param_object)`

Queues a heavy parameter to be saved to the 'parameter' group.

Mirrors save but targets the parameter group so large input arrays live in the glob (out of the SQLite database) exactly like results.

Parameters:

Name	Type	Description	Default
`name`	`str`	The target dataset name for this parameter.	required
`param_object`	`Any`	The data to be saved (h5py-compatible).	required

Returns:

Name	Type	Description
`tuple`		`(name, sha256_hash)`.

`save_param_from_staging(name, staging_path, hsh=None)`

Claims a staged parameter array into the glob without loading it.

The parameter mirror of save_from_staging; see it for why this streams instead of reading.

`commit_result()`

Writes all queued results to the HDF5 file on disk.

Creates or requires the 'result' group and iterates through the results queue to create datasets. If a dataset name already exists, it deletes the old one to prevent errors before writing the new data. Forces a disk flush.

`commit_parameter()`

Writes all queued heavy parameters to the 'parameter' HDF5 group.

Mirrors commit_result: requires the 'parameter' group, overwrites any duplicate dataset, and flushes to disk.

`commit_source(source)`

Saves the main source code string into the HDF5 metadata group.

Parameters:

Name	Type	Description	Default
`source`	`str`	The raw source code of the simulation script.	required

`close()`

Safely closes the underlying HDF5 file.

`read_glob(storage_root, uuid, group, pointer)`

Reads a dataset from a run's HDF5 glob file under a given storage root.

Unlike read_data, this function does not rely on the current working directory: the caller provides the storage root holding the glob folder (usually the .sillon directory or the configured storage root).

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder containing the `glob` directory.	required
`uuid`	`str`	The unique identifier of the simulation run.	required
`group`	`str`	The HDF5 group name (e.g., 'metadata' or 'result').	required
`pointer`	`str`	The specific dataset name within the group.	required

Returns:

Name	Type	Description
`Any`		The extracted data, decoded to a string if it was stored as bytes, or `None` if the glob file or the dataset does not exist.

`read_glob_many(storage_root, uuid, datasets)`

Reads several datasets from a run's glob file in a single file open.

Used by the query engine so a run's glob is opened once for all of its result/analysis value conditions, rather than once per condition.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder containing the `glob` directory.	required
`uuid`	`str`	The unique identifier of the simulation run.	required
`datasets`	`Iterable[tuple[str, str]]`	`(group, pointer)` pairs to read.	required

Returns:

Name	Type	Description
`dict`		Mapping of `(group, pointer)` to its value (decoded if bytes), or `None` for any dataset (or whole file) that does not exist.

`append_glob(storage_root, uuid, group, pointer, data)`

Writes a dataset into a run's HDF5 glob after the run has ended.

Used for post-hoc data (e.g., the analysis group): the glob file is reopened in append mode, the dataset is (over)written, and the data hash is returned for database tracking. The glob folder is created if the run never stored heavy data.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder containing the `glob` directory.	required
`uuid`	`str`	The unique identifier of the simulation run.	required
`group`	`str`	The HDF5 group name (e.g., 'analysis').	required
`pointer`	`str`	The dataset name within the group.	required
`data`	`Any`	The data to store (must be compatible with h5py).	required

Returns:

Name	Type	Description
`str`		The SHA-256 hash of the stored data.

`glob_sizes(storage_root, uuid, group='result')`

Lists the on-disk byte size of every dataset of a glob group.

Parameters:

Name	Type	Description	Default
`storage_root`	`str \| Path`	The folder containing the `glob` directory.	required
`uuid`	`str`	The unique identifier of the simulation run.	required
`group`	`str`	The HDF5 group to inspect. Defaults to "result".	`'result'`

Returns:

Name	Type	Description
`dict`		A mapping of dataset names to their size in bytes. Empty if the glob file or the group does not exist.

`read_data(uuid, group, pointer)`

Retrieves specific data from a simulation's HDF5 glob file.

This function accesses the HDF5 file for a specific run and extracts the dataset located at the specified group and pointer. It assumes the current working directory is the project root. Automatically decodes byte strings to UTF-8.

Parameters:

Name	Type	Description	Default
`uuid`	`str`	The unique identifier of the simulation run.	required
`group`	`str`	The HDF5 group name (e.g., 'metadata' or 'result').	required
`pointer`	`str`	The specific dataset name within the group.	required

Raises:

Type	Description
`FileNotFoundError`	If the `glob.hdf5` file for the given UUID does not exist.
`KeyError`	If the specified group/pointer path is not found in the file.

Returns:

Name	Type	Description
`Any`		The extracted data, decoded to a string if it was stored as bytes, or `None` if the operation fails gracefully.

Project paths

`silloncore.project_paths`

Resolution of a sillon project's storage root and database engine.

A project keeps its data either directly under .sillon/ or under a custom storage root recorded in .sillon/config.toml. Both the CLI and sillonlab need to find that data, so the logic lives here once.

`resolve_storage_root(project_path)`

Returns the folder holding a project's glob, artifact and figure dirs.

Reads the storage root from .sillon/config.toml, defaulting to the .sillon directory itself when no custom root is configured.

Parameters:

Name	Type	Description	Default
`project_path`	`str \| Path`	The project root (folder containing `.sillon`).	required

Returns:

Name	Type	Description
`Path`	`Path`	The resolved storage root.

`resolve_engine(project_path)`

Connects to a project's SQLite database, wherever it was created.

Handles both the default layout (.sillon/database.sql) and layouts produced with a custom storage root.

Parameters:

Name	Type	Description	Default
`project_path`	`str \| Path`	The project root (folder containing `.sillon`).	required

Raises:

Type	Description
`FileNotFoundError`	If no database can be located for the project.

Returns:

Name	Type	Description
`Engine`		A SQLAlchemy engine connected to the project database.

Simulation & environment

`silloncore.simulation`

`ResultItem` `dataclass`

A standardized data structure representing the result of a simulation.

Attributes:

Name	Type	Description
`value`	`Any`	The actual result value, or a pointer/path to the result.
`is_artifact`	`bool`	Indicates if the result is a saved physical file (artifact). Defaults to False.
`hsh`	`str`	The SHA-256 hash of the result or artifact for integrity tracking. Defaults to "".

`FigureItem` `dataclass`

A standardized data structure representing a figure logged during a run.

Attributes:

Name	Type	Description
`value`	`str`	The storage pointer of the saved figure file.
`hsh`	`str`	The SHA-256 hash of the figure file.
`meta`	`dict`	Provenance metadata: which parameters/results were used to draw the figure (`used` key), a caption, the file format...

`SimHashes` `dataclass`

Stores hashes related to various components of a simulation run.

Attributes:

Name	Type	Description
`source`	`str`	Hash of the main script source code. Defaults to "".
`git`	`str`	Hash representing the current Git commit state. Defaults to "".
`project`	`str`	Hash representing the project state. Defaults to "".
`parameters`	`str`	Hash of the input parameters. Defaults to "".
`artifacts`	`str`	Hash of the saved artifacts. Defaults to "".
`result_out`	`str`	Hash of the standard output or result stream. Defaults to "".

`SingletonMeta`

Bases: type

A metaclass that implements the Singleton design pattern.

Ensures that only one instance of any class using this metaclass is ever created.

`SimulationDict`

A centralized registry to hold all active simulations.

All simulations are represented through a dictionary. To ensure the uniqueness of this registry across the application, it utilizes the Singleton pattern.

Attributes:

Name	Type	Description
`sim_dict`	`dict`	The internal dictionary storing `Simulation` objects by their `run_id`.

`add_sim(run_id, **kwargs)`

Adds a simulation to the registry.

Each simulation is characterized by a unique key (run_id). Once added, the simulations can be manipulated individually.

Parameters:

Name	Type	Description	Default
`run_id`	`str`	The unique identifier for the simulation run.	required
`**kwargs`		Keyword arguments passed directly to the `Simulation` constructor.	`{}`

`rm_sim(key)`

Removes a simulation from the registry based on its unique identifier.

Parameters:

Name	Type	Description	Default
`key`	`str`	The `run_id` of the simulation to remove.	required

`Simulation`

Represents a single, trackable simulation run.

Handles the collection of parameters, results, metadata, tags, and notes, as well as managing the underlying environment storage (artifacts and HDF5 glob).

Attributes:

Name	Type	Description
`run_id`	`str`	The unique identifier (UUID) for the run.
`run_name`	`str`	The human-readable name of the run.
`envh`	`RunEnvironmentHandler`	Handler for artifacts and glob files.
`glob`	`Glob`	The HDF5 interface for heavy data storage.
`platform`	`str`	Operating system or platform executing the run.
`project_name`	`str`	The name of the parent project.
`hostname`	`str`	The machine's hostname.
`organisation`	`str`	The organization running the project.
`author`	`str`	The user executing the run.
`runtime`	`str`	The execution duration of the run.
`status`	`str`	Current execution status (e.g., 'RUNNING').
`parameters`	`Dict[str, Any]`	Tracked input parameters.
`results`	`Dict[str, ResultItem]`	Tracked output metrics and artifacts.
`metadata`	`Dict[str, Any]`	Custom tracked metadata.
`tags`	`list`	User-defined tags.
`notes`	`list`	User-defined textual notes.
`hashes`	`SimHashes`	Object tracking hashes of various components.
`date`	`str`	Timestamp string of when the run was created.

`check_name()`

Verifies the run name exists, generating a random one if it is empty or None.

`set_date()`

Sets the creation date of the run based on the current system time.

`log_param(name, value)`

Logs a parameter key-value pair to the simulation state.

Lightweight, JSON-serializable values are stored inline in the database. Heavy parameter arrays arrive as a staging reference (__sillon_array_ref__) and are offloaded to the run's HDF5 glob parameter group, exactly like large results; the database then only keeps a small marker recording the pointer, shape and dtype.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the parameter.	required
`value`	`Any`	The value of the parameter, or a staging reference.	required

`add_metadata(name, value)`

Adds custom metadata to the simulation.

Checks the METADATA_TABLE to see if the metadata key requires special handling (e.g., source code). Otherwise, saves it standardly.

Parameters:

Name	Type	Description	Default
`name`	`str`	The metadata key or reserved namespace key.	required
`value`	`Any`	The metadata value or payload.	required

`add_note(name, value)`

Appends a textual note to the simulation run.

Parameters:

Name	Type	Description	Default
`name`	`str`	Unused mapping key (kept for API consistency).	required
`value`	`Any`	The note content.	required

`add_tag(name, value)`

Appends a tag string to the simulation run.

Parameters:

Name	Type	Description	Default
`name`	`str`	Unused mapping key (kept for API consistency).	required
`value`	`Any`	The tag string.	required

`log_result(name, value_dict)`

Saves a metric, dataset, or file artifact as a result.

This command handles multiple data flows: 1. Simple data logging (saves to HDF5 glob). 2. File artifact copying (copies a file and logs its DB pointer). 3. Simple DB logging without copying the artifact (if save_result is False).

Parameters:

Name	Type	Description	Default
`name`	`str`	The identifier name for the result.	required
`value_dict`	`dict`	A dictionary containing routing instructions. Expected keys include: - `value` (Any): The actual data to save into HDF5. - `path` (str): The file path if logging an external artifact. - `save_result` (bool): Whether to physically copy the artifact into the project directory (Defaults to False).	required

Note

You cannot provide path=None and save_result=False at the same time. This function currently relies on a globally defined environment handler and requires further testing for massive data dumps.

`log_figure(name, value_dict)`

Saves a tracked figure file with its provenance metadata.

The figure file (already rendered by the client, e.g. a matplotlib png) is copied into the run's figure storage, hashed, and kept with its metadata so the database can record which data produced it.

Parameters:

Name	Type	Description	Default
`name`	`str`	The identifier name for the figure.	required
`value_dict`	`dict`	A dictionary containing routing instructions. Expected keys: - `path` (str): The path of the rendered figure file. - `meta` (dict): Provenance metadata (`used` data names, caption, format...). - `cleanup` (bool): Whether the source file is a temporary staging file to delete after the copy (Defaults to False).	required

`get_param(name)`

Retrieves a tracked parameter by name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the parameter.	required

Returns:

Name	Type	Description
`Any`		The stored parameter value.

`get_result(name)`

Retrieves a tracked result object by name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the result.	required

Returns:

Name	Type	Description
`ResultItem`		The standard ResultItem object.

`get_metadata(name)`

Retrieves tracked metadata by name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The key for the metadata.	required

Returns:

Name	Type	Description
`Any`		The stored metadata payload.

`commit_source(simulation_obj, source)`

Commits the main script source to the simulation's HDF5 glob and hashes it.

Parameters:

Name	Type	Description	Default
`simulation_obj`	`Simulation`	The active simulation instance.	required
`source`	`str`	The raw source code string to be saved.	required

`commit_runtime(simulation_obj, runtime)`

Commits the runtime duration to the simulation object.

Parameters:

Name	Type	Description	Default
`simulation_obj`	`Simulation`	The active simulation instance.	required
`runtime`	`str \| float`	The calculated execution time of the run.	required

`commit_status(simulation_obj, status)`

Commits the execution status to the simulation object.

Parameters:

Name	Type	Description	Default
`simulation_obj`	`Simulation`	The active simulation instance.	required
`status`	`str`	The current status (e.g., "SUCCESS", "FAILED").	required

`commit_parent(simulation_obj, parent)`

Records a lineage edge: a run this run inherited/derived from.

Parameters:

Name	Type	Description	Default
`simulation_obj`	`Simulation`	The active simulation instance.	required
`parent`	`dict`	`{"uuid", "name", "params": [inherited param names]}`.	required

`silloncore.envhandler`

`ProjectEnvironmentHandler`

Manages the workspace environment for a sillon project.

This class handles the initialization and management of the .sillon directory, including the SQLite database, configuration files, and artifact/glob structures required for a project.

Attributes:

Name	Type	Description
`db_path`	`Path`	The file path to the SQLite database.
`sillon_dir`	`Path`	The file path to the `.sillon` directory.

`get_config()`

Retrieves the current project configuration.

Returns:

Name	Type	Description
`dict`		The configuration dictionary loaded from `config.toml`.

`get_engine()`

Retrieves the SQLAlchemy engine linked to the project database.

Returns:

Name	Type	Description
`Engine`		The active SQLAlchemy engine instance.

`get_sql_session()`

Retrieves the active database session.

Returns:

Name	Type	Description
`Session`		The SQLModel session for database interactions.

`get_project_path()`

Retrieves the base path of the project.

Returns:

Name	Type	Description
`Path`		The resolved path to the project root.

`commit_run(run)`

Saves a simulation run to the database and HDF5 glob.

Commits the HDF5 heavy results, closes the glob file safely, and inserts the structured simulation record into the SQLite database.

Parameters:

Name	Type	Description	Default
`run`		The simulation run object containing tracking data and a `.glob` instance.	required

Returns:

Name	Type	Description
`int`		The database ID of the inserted simulation record.

`RunEnvironmentHandler`

Manages the file storage structure for an individual simulation run.

This class isolates artifacts and glob (HDF5) files for a specific run based on its unique UUID, ensuring isolated storage per run.

Attributes:

Name	Type	Description
`uuid`	`str`	The unique identifier for the run.
`project_path`	`Path`	The root path of the parent project.
`artifact_path`	`Path`	The specific folder path for this run's artifacts.
`glob_path`	`Path`	The specific folder path for this run's HDF5 glob file.

`create_glob()`

Creates and returns an HDF5 Glob interface for this run.

Returns:

Name	Type	Description
`Glob`		An initialized Glob instance pointing to the run's specific `glob_path`.

`save_figure(file_path)`

Copies a rendered figure file into the run's figure storage.

Mirrors save_artifact but targets the figure directory, keeping tracked figures separate from generic artifacts.

Parameters:

Name	Type	Description	Default
`file_path`	`str \| Path`	The path to the rendered figure file.	required

Raises:

Type	Description
`FileNotFoundError`	If the provided `file_path` does not exist.

Returns:

Name	Type	Description
`tuple`		A tuple containing: - str: The figure storage pointer (folder uuid). - str: The calculated hash of the figure file.

`save_artifact(file_path)`

Copies an artifact (file or directory) into the run's artifact storage.

Calculates the hash of the source file, generates a new UUID for the artifact folder, and securely copies the content into the .sillon/artifact directory.

Parameters:

Name	Type	Description	Default
`file_path`	`str \| Path`	The path to the source file or directory to be saved.	required

Raises:

Type	Description
`FileNotFoundError`	If the provided `file_path` does not exist on disk.

Returns:

Name	Type	Description
`tuple`		A tuple containing: - str: The absolute path to the newly saved artifact directory. - str: The calculated hash of the source artifact.

Version control / diffing

`silloncore.versioncontrol`

`Reference`

Represents a loaded snapshot of a simulation run from the database.

This class facilitates quick lookups of a run's main elements without having to query the database multiple times. It fetches core data, artifacts, and the source code from the HDF5 glob.

Attributes:

Name	Type	Description
`name`	`str`	The specific name of the run.
`date`	`str`	The timestamp of the run.
`parameters`	`dict`	The tracked input parameters.
`results`	`dict`	The tracked output results and metrics.
`metadata`	`dict`	Custom tracking metadata.
`id`	`str`	The UUID of the run.
`runtime`	`str \| float`	The execution duration of the run.
`status`	`str`	The final execution status of the run.
`artifacts`	`Any`	Artifacts associated with the run, if any.
`source`	`str`	The raw main script source code fetched from the glob.

`Diffref`

Diffing engine based on built-in difflib to differentiate two run references.

This class extracts and compares the source code, parameters, and execution context between two loaded Reference objects.

Attributes:

Name	Type	Description
`ref1`	`Reference`	The base/original simulation reference.
`ref2`	`Reference`	The target/new simulation reference.
`diff_source`	`str`	The unified text diff of the source code.
`diff_parameters`	`dict`	A dictionary detailing added, removed, and changed parameters.
`diff_runtime`	`tuple \| None`	The before and after states of the runtime.
`diff_status`	`tuple \| None`	The before and after states of the status.

`get_diff_source()`

Computes the unified diff between the source codes of the two references.

The resulting unified diff string is generated using difflib and stored in the diff_source attribute.

`get_diff_parameters()`

Computes the set differences between the parameters of both references.

Identifies newly added keys, removed keys, and keys whose values have changed. The results are stored in the diff_parameters attribute formatted as a dictionary containing diff_data, diff_key_added, and diff_key_removed.

`get_diff_context()`

Computes the differences for basic metadata and execution context.

Calculates the differences between the runtime and status of the two references and stores them in their respective class attributes.

`simple_compare(value_old, value_new)`

Compares two generic values to format their differences.

Parameters:

Name	Type	Description	Default
`value_old`	`Any`	The original tracking value.	required
`value_new`	`Any`	The new tracking value.	required

Returns:

Type	Description
	tuple \| None: Returns `None` if the values are identical. Otherwise, returns a tuple containing `(value_old, value_new, "N/A")`.

silloncore internals

Engine

silloncore.engine

get_project_context(engine, run_names=None)

get_run_details(engine, run_names=None, params=None, meta=None, results=None)

add_metadata_to_runs(engine, run_names, notes=None, tags=None, metadata=None)

resolve_run_identifier(engine, token)

get_run_snapshot(engine, run_name)

load_run_result(storage_root, snapshot, name)

load_run_parameter(storage_root, snapshot, name)

load_run_artifact(storage_root, snapshot, name)

load_run_source(storage_root, snapshot)

match_cheap(entry, parameters=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None, storage_root=None)

match_heavy(storage_root, snapshot, results=None, analyses=None)

match_run(storage_root, snapshot, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)

query_runs(engine, storage_root=None, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)

fetch_run_result(storage_root, snapshot, name, dest=None)

get_run_sizes(storage_root, snapshot)

export_run(storage_root, snapshot, dest=None, format='npz')

build_run_report(engine, storage_root, snapshot)

export_run_report(engine, storage_root, snapshot, dest=None, with_data=False)

prune_runs(engine, storage_root, run_names=None, before=None, keep_metadata=True)

delete_run(engine, storage_root, run_name)

rename_run(engine, run_name, new_name)

find_by_hash(engine, file_or_hash)

find_children(engine, uuid)

load_run_figure(storage_root, snapshot, name)

add_run_analysis(engine, storage_root, snapshot, name, data, info=None)

load_run_analysis(storage_root, snapshot, name)

compare(engine, run_id1, run_id2)

Storage (glob)

silloncore.glob

Glob

save(name, result_object)

save_from_staging(name, staging_path, hsh=None)

save_param(name, param_object)

save_param_from_staging(name, staging_path, hsh=None)

commit_result()

commit_parameter()

commit_source(source)

close()

read_glob(storage_root, uuid, group, pointer)

read_glob_many(storage_root, uuid, datasets)

append_glob(storage_root, uuid, group, pointer, data)

glob_sizes(storage_root, uuid, group='result')

read_data(uuid, group, pointer)

Project paths

silloncore.project_paths

resolve_storage_root(project_path)

resolve_engine(project_path)

Simulation & environment

silloncore.simulation

ResultItem dataclass

FigureItem dataclass

SimHashes dataclass

SingletonMeta

SimulationDict

add_sim(run_id, **kwargs)

rm_sim(key)

Simulation

check_name()

set_date()

log_param(name, value)

add_metadata(name, value)

add_note(name, value)

add_tag(name, value)

log_result(name, value_dict)

log_figure(name, value_dict)

get_param(name)

get_result(name)

get_metadata(name)

commit_source(simulation_obj, source)

commit_runtime(simulation_obj, runtime)

commit_status(simulation_obj, status)

commit_parent(simulation_obj, parent)

silloncore.envhandler

ProjectEnvironmentHandler

get_config()

get_engine()

get_sql_session()

`silloncore` internals

`silloncore.engine`

`get_project_context(engine, run_names=None)`

`get_run_details(engine, run_names=None, params=None, meta=None, results=None)`

`add_metadata_to_runs(engine, run_names, notes=None, tags=None, metadata=None)`

`resolve_run_identifier(engine, token)`

`get_run_snapshot(engine, run_name)`

`load_run_result(storage_root, snapshot, name)`

`load_run_parameter(storage_root, snapshot, name)`

`load_run_artifact(storage_root, snapshot, name)`

`load_run_source(storage_root, snapshot)`

`match_cheap(entry, parameters=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None, storage_root=None)`

`match_heavy(storage_root, snapshot, results=None, analyses=None)`

`match_run(storage_root, snapshot, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)`

`query_runs(engine, storage_root=None, parameters=None, results=None, analyses=None, metadata=None, fields=None, has_parameter=None, has_metadata=None, has_result=None, has_analysis=None, has_artifact=None, has_tag=None, before=None, after=None)`

`fetch_run_result(storage_root, snapshot, name, dest=None)`

`get_run_sizes(storage_root, snapshot)`

`export_run(storage_root, snapshot, dest=None, format='npz')`

`build_run_report(engine, storage_root, snapshot)`

`export_run_report(engine, storage_root, snapshot, dest=None, with_data=False)`

`prune_runs(engine, storage_root, run_names=None, before=None, keep_metadata=True)`

`delete_run(engine, storage_root, run_name)`

`rename_run(engine, run_name, new_name)`

`find_by_hash(engine, file_or_hash)`

`find_children(engine, uuid)`

`load_run_figure(storage_root, snapshot, name)`

`add_run_analysis(engine, storage_root, snapshot, name, data, info=None)`

`load_run_analysis(storage_root, snapshot, name)`

`compare(engine, run_id1, run_id2)`

`silloncore.glob`

`Glob`

`save(name, result_object)`

`save_from_staging(name, staging_path, hsh=None)`

`save_param(name, param_object)`

`save_param_from_staging(name, staging_path, hsh=None)`

`commit_result()`

`commit_parameter()`

`commit_source(source)`

`close()`

`read_glob(storage_root, uuid, group, pointer)`

`read_glob_many(storage_root, uuid, datasets)`

`append_glob(storage_root, uuid, group, pointer, data)`

`glob_sizes(storage_root, uuid, group='result')`

`read_data(uuid, group, pointer)`

`silloncore.project_paths`

`resolve_storage_root(project_path)`

`resolve_engine(project_path)`

`silloncore.simulation`

`ResultItem` `dataclass`

`FigureItem` `dataclass`

`SimHashes` `dataclass`

`SingletonMeta`

`SimulationDict`

`add_sim(run_id, **kwargs)`

`rm_sim(key)`

`Simulation`

`check_name()`

`set_date()`

`log_param(name, value)`

`add_metadata(name, value)`

`add_note(name, value)`

`add_tag(name, value)`

`log_result(name, value_dict)`

`log_figure(name, value_dict)`

`get_param(name)`

`get_result(name)`

`get_metadata(name)`

`commit_source(simulation_obj, source)`

`commit_runtime(simulation_obj, runtime)`

`commit_status(simulation_obj, status)`

`commit_parent(simulation_obj, parent)`

`silloncore.envhandler`

`ProjectEnvironmentHandler`

`get_config()`

`get_engine()`

`get_sql_session()`

`get_project_path()`

`commit_run(run)`

`RunEnvironmentHandler`

`create_glob()`