Data

perun structures the data collected as a tree, with the root node containing the aggregated data of an indiviudal run of the application, and the nodes further down the tree contain the information of the indivdual compute nodes, devices and sensors. Sensors are meant as the individual values that can be collected from the distinct monitoring backends.

_images/data_structure.png

Each node in the data structure, once the raw data at the bottom has been processed, contain a set of summarized metrics based on the data that was collected by its sub-nodes, and a metadata dictionary with any information that could be obtained by the application, node, device or API.

Each node contains a list of metrics or stats, which represent the accumulated data. As well as metadata.

The nodeType attribute indiciates the type of object in the hierarchy this nodes represents. At the lowest level, the leafs of the tree, you would have individual “sensors”, values collected by a single device or interface. Higher up the tree, the data nodes represent groups of devices and computational nodes. The three bottom levels of the tree represent the hardware. Further up the three, data starts being acumulated by individual runs of the application, with “run” being a single execution of the application, a “multi_run” is the data from multiple runs when perun is run with the `--rounds N` option, and at the highest level, the root of the tree is the application itself.

Working with the HDF5 output

By default perun writes an HDF5 file next to your script (named after the application). It contains everything perun collected: the full data tree, per-device raw samples, computed metrics, metadata, and monitored regions. This page shows how to open that file and get the data out, either with plain h5py/pandas or with perun’s own command line utilities — no need to learn perun’s internal Python API first.

File layout

The file mirrors perun’s perun.data_model.data.DataNode tree one-to-one. Every node is an HDF5 group; the tree goes from the application at the root down to individual sensors at the leaves:

<app_name>/                      # APP        (group)
├── attrs: type, creation_dt, last_execution_dt, perun_version, ...
├── metrics/                     # aggregated Stats for the whole app
│   └── <METRIC_NAME>/           # e.g. ENERGY, RUNTIME, CO2, MONEY
│       └── attrs: type, unit, mag, dtype, sum, mean, std, min, max
└── nodes/
    └── <run_id>/                # MULTI_RUN  (a group of --rounds runs)
        ├── metrics/             # Stats across the rounds
        └── nodes/
            └── <round_n>/       # RUN        (one execution)
                ├── metrics/
                ├── regions/     # optional, monitored functions
                └── nodes/
                    └── <hostname>/          # NODE (one compute node)
                        ├── metrics/
                        └── nodes/
                            └── <device_group>/   # DEVICE_GROUP
                                └── nodes/
                                    └── <sensor_id>/   # SENSOR (leaf)
                                        ├── metrics/
                                        └── raw_data/   # the samples

Key conventions:

  • Node type is stored in each group’s type attribute (APP, MULTI_RUN, RUN, NODE, DEVICE_GROUP, SENSOR).

  • Metadata is stored as group attributes (all serialized to strings).

  • Metrics live under a metrics/ subgroup. A leaf Metric has a value attribute; an aggregated Stats has sum/mean/std/ min/max attributes. Both carry unit info in unit, mag (the magnitude/SI prefix) and dtype.

  • Raw samples live under raw_data/ on sensor nodes, as two datasets: timesteps and values (plus an optional alt_values — for energy sensors, values becomes derived power and alt_values keeps the original energy counter).

Quick look with the HDF5 tooling

If you just want to browse the structure, the h5py and hdf5 packages ship the h5ls / h5dump command line tools:

$ h5ls -r my_app.hdf5 | head
/                        Group
/my_app                  Group
/my_app/metrics          Group
/my_app/metrics/ENERGY   Group
/my_app/nodes            Group
...

Reading metrics with h5py

import h5py

with h5py.File("my_app.hdf5", "r") as f:
    app = f[list(f.keys())[0]]          # the single root (application) group

    # Application-level aggregated metrics (Stats):
    energy = app["metrics"]["ENERGY"]
    print("unit:", energy.attrs["unit"], "mag:", energy.attrs["mag"])
    print("total energy:", energy.attrs["sum"])
    print("mean per run:", energy.attrs["mean"])

    # Iterate every run and read its runtime/energy:
    for run_id, mr in app["nodes"].items():
        for round_n, run in mr["nodes"].items():
            m = run["metrics"]
            print(run_id, round_n,
                  "runtime", m["RUNTIME"].attrs["value"],
                  "energy", m["ENERGY"].attrs["value"])

Note

Values are stored in the magnitude given by the mag attribute (an SI prefix factor, e.g. 1000 for kilo). Multiply the stored value by mag to get the base-unit value.

Reading raw samples with h5py + pandas

Every sensor keeps its time series under raw_data/:

import h5py
import pandas as pd

def sensor_timeseries(path):
    rows = []
    with h5py.File(path, "r") as f:
        app = f[list(f.keys())[0]]
        for run_id, mr in app["nodes"].items():
            for round_n, run in mr["nodes"].items():
                for host, host_node in run["nodes"].items():
                    for grp, dg in host_node["nodes"].items():
                        for sensor_id, sensor in dg["nodes"].items():
                            if "raw_data" not in sensor:
                                continue
                            raw = sensor["raw_data"]
                            ts = raw["timesteps"][:]
                            vals = raw["values"][:]
                            for t, v in zip(ts, vals):
                                rows.append((run_id, round_n, host,
                                             sensor_id, t, v))
    return pd.DataFrame(
        rows,
        columns=["run_id", "round", "host", "sensor", "timestep", "value"],
    )

df = sensor_timeseries("my_app.hdf5")
print(df.head())

Exporting from the command line

If you would rather not touch h5py at all, use perun export to convert an existing output file into an easier-to-consume format. The input can be any of perun’s structured formats (hdf5, json, pickle) and the output one of csv, json, bench, text (and back to hdf5/pickle):

# Raw per-sample table for the last run, as CSV:
$ perun export my_app.hdf5 csv

# Pick a specific run id:
$ perun export -i 2024-05-20T12:00:00 my_app.hdf5 csv

# Human-readable summary table:
$ perun export my_app.hdf5 text

The csv export produces one row per sample with the columns run id, hostname, device_group, sensor, unit, magnitude, timestep, value — ready to load with pandas.read_csv() and plot.

import pandas as pd

df = pd.read_csv("my_app_<run_id>.csv")
# Average power draw per sensor over time, for example:
df.groupby("sensor")["value"].mean()

Loading back into perun

If you want the rich Python objects (metrics with units, regions, etc.) rather than flat tables, you can re-import the tree:

from pathlib import Path
from perun.io.io import importFrom, IOFormat

data = importFrom(Path("my_app.hdf5"), IOFormat.HDF5)
print(data.id, data.metrics.keys())

See Data for a description of the DataNode structure returned here.