perun.data_model.data

Storage Module.

Attributes

log

Classes

NodeType	DataNode type enum.
MetricType	Metric Type enum.
AggregateType	Types of data aggregation.
Metric	Struct with resulting metrics and the metadata.
Stats	Collects statistics based on multiple metrics of the same type.
RawData	Contains timesteps and recorded values from sensors, including information on the values.
LocalRegions	Stores local region data while an application is being monitored.
Region	Stores region data from all MPI ranks.
DataNode	Recursive data structure that contains all the information of a monitored application.

Module Contents

perun.data_model.data.log

class perun.data_model.data.NodeType(*args, **kwds)

Bases: enum.Enum

DataNode type enum.

APP

MULTI_RUN

RUN

NODE

DEVICE_GROUP

SENSOR

class perun.data_model.data.MetricType

Bases: str, enum.Enum

Metric Type enum.

RUNTIME = 'runtime'

POWER = 'power'

CPU_POWER = 'cpu_power'

GPU_POWER = 'gpu_power'

DRAM_POWER = 'dram_power'

OTHER_POWER = 'other_power'

CPU_UTIL = 'cpu_util'

GPU_UTIL = 'gpu_util'

OTHER_UTIL = 'other_util'

DRAM_MEM = 'dram_mem'

GPU_MEM = 'gpu_mem'

NET_READ = 'net_read'

NET_WRITE = 'net_write'

DISK_READ = 'disk_read'

DISK_WRITE = 'disk_write'

ENERGY = 'energy'

CPU_ENERGY = 'cpu_energy'

GPU_ENERGY = 'gpu_energy'

DRAM_ENERGY = 'dram_energy'

OTHER_ENERGY = 'other_energy'

OTHER_MEM = 'other_memory'

CPU_CLOCK = 'cpu_clock'

GPU_CLOCK = 'gpu_clock'

N_RUNS = 'n_runs'

MONEY = 'money'

CO2 = 'co2'

fromString(value: str) → MetricType

Create MetricType from string.

Parameters:

value (str) – MetricType value.

Returns:

MetricType object.

Return type:

MetricType

class perun.data_model.data.AggregateType

Bases: str, enum.Enum

Types of data aggregation.

SUM = 'sum'

MEAN = 'mean'

MAX = 'max'

MIN = 'min'

class perun.data_model.data.Metric

Struct with resulting metrics and the metadata.

type: MetricType

value: perun.data_model.measurement_type.Number

metric_md: perun.data_model.measurement_type.MetricMetaData

agg: AggregateType

classmethod fromDict(metricDict: Dict) → Metric

Create RawData object from a dictionary.

copy() → Metric

Create copy metric object.

Returns:

Copy of object.

Return type:

_type_

class perun.data_model.data.Stats

Collects statistics based on multiple metrics of the same type.

type: MetricType

metric_md: perun.data_model.measurement_type.MetricMetaData

sum: perun.data_model.measurement_type.Number

mean: perun.data_model.measurement_type.Number

std: perun.data_model.measurement_type.Number

max: perun.data_model.measurement_type.Number

min: perun.data_model.measurement_type.Number

classmethod fromMetrics(metrics: List[Metric]) → Stats

Create stats object from list of metrics with the same type.

Parameters:

metrics (List[Metric]) – List of metrics with the same type.

Returns:

Stats object.

Return type:

Stats

Raises:

Exception – If metrics are not from the same type.

property value: perun.data_model.measurement_type.Number

Value property (mean).

For compatibility with Metric dataclass.

Returns:

Return the mean value of the stats object.

Return type:

Number

classmethod fromDict(statsDict: Dict) → Stats

Stats constructor from a dictionory.

class perun.data_model.data.RawData

Contains timesteps and recorded values from sensors, including information on the values.

timesteps: numpy.ndarray[Any, numpy.dtype[numpy.floating]]

values: numpy.ndarray[Any, numpy.dtype[numpy.integer | numpy.floating]]

t_md: perun.data_model.measurement_type.MetricMetaData

v_md: perun.data_model.measurement_type.MetricMetaData

alt_values: numpy.ndarray | None = None

alt_v_md: perun.data_model.measurement_type.MetricMetaData | None = None

classmethod fromDict(rawDataDict: Dict) → RawData

Create RawData object from a dictionary.

Parameters:

rawDataDict (Dict) – Dictionary with same keys as RawData object.

Returns:

RawData object.

Return type:

_type_

class perun.data_model.data.LocalRegions

Stores local region data while an application is being monitored.

addEvent(region_name: str) → None

Mark a new event for the named region.

Parameters:

region_name (str) – Region to mark the event from.

isEmpty() → bool

Check if there are any regions marked.

Returns:

True if there are no regions marked.

Return type:

bool

class perun.data_model.data.Region

Stores region data from all MPI ranks.

For each marked region (decorated function), an numpy array with timestamps indicating function starts and ends.

id: str = ''

raw_data: Dict[int, numpy.ndarray[Any, numpy.dtype[numpy.floating]]]

runs_per_rank: Stats | None = None

metrics: Dict[MetricType, Stats]

processed: bool = False

toDict() → Dict[str, Any]

Convert regions to a python dictionary.

Returns:

Dictionary with region data.

Return type:

Dict[str, Dict[int, np.ndarray]]

classmethod fromDict(regionDictionary: Dict[str, Any]) → Region

Create Regions object from a dictionary.

Parameters:

regionDictionary (Dict[str, Dict[int, Any]]) – Region dictionary.

Returns:

Regions object.

Return type:

Regions

class perun.data_model.data.DataNode(id: str, type: NodeType, metadata: Dict = {}, nodes: Dict[str, Any] | None = None, metrics: Dict[MetricType, Metric | Stats] | None = None, deviceType: perun.data_model.sensor.DeviceType | None = None, raw_data: RawData | None = None, regions: Dict[str, Region] | None = None, processed: bool = False)

Recursive data structure that contains all the information of a monitored application.

id

type

metadata: Dict[str, Any]

nodes: Dict[str, Any]

metrics: Dict[MetricType, Metric | Stats]

deviceType: perun.data_model.sensor.DeviceType | None = None

raw_data: RawData | None = None

regions: Dict[str, Region] | None = None

processed = False

addRegionData(localRegions: List[LocalRegions], start_time: int) → None

Add region information to to data node.

Parameters:

• localRegions (List[LocalRegions]) – Gathered local regions from all MPI ranks

• start_time (int) – ‘Official’ start time of the run.

toDict(include_raw_data: bool = True) → Dict

Transform object to dictionary.

classmethod fromDict(resultsDict: Dict) → DataNode

Create dataNode from python dictionary.

Parameters:

resultsDict (Dict) – Dictionary with data node attributes.

Returns:

DataNode object.

Return type:

_type_