simplebench.reporters.reporter package🔗

Reporter base package in the reporters package.

class simplebench.reporters.reporter.Reporter(
config: ReporterConfig,
)[source]🔗

Bases: ABC, _ReporterArgparseMixin, _ReporterOrchestrationMixin, _ReporterPrioritizationMixin, _ReporterTargetMixin, ReporterProtocol

Base class for Reporter classes.

A Reporter is responsible for generating reports based on benchmark results from a Session and Case. Reporters can produce reports in various formats and output them to different targets.

All Reporter subclasses must implement the methods defined in this interface. Reporters should handle their own output, whether to console, file system, HTTP endpoint, display device, via a callback or other output.

The Reporter interface ensures that all reporters provide a consistent set of functionalities, making it easier to manage and utilize different

reporting options within the SimpleBench framework.

Initialize the Reporter instance.

Parameters:

config (ReporterConfig) – The configuration object for the reporter.

Raises:

SimpleBenchTypeError – If the provided config is not a ReporterConfig instance.

add_choice(choice: None) → None[source]🔗

Add a Choice to the reporter’s choices.

Parameters:

choice (Choice) – The Choice instance to add.

Raises:
property choices: Choices🔗

The Choices for the reporter.

The Choices instance contains one or more Choice instances, each representing a specific combination of sections, targets, and formats, command line flags, and descriptions.

This property allows access to the reporter’s choices for generating reports and customizing report output and available options.

Returns:

The Choices instance for the reporter.

Return type:

Choices

property config: ReporterConfig🔗

The configuration object for the reporter.

property default_targets: frozenset[Target]🔗

The default set of Targets for the reporter.

property description: str🔗

A brief description of the reporter.

property file_append: bool🔗

Whether output files should be appended to.

property file_suffix: str🔗

The file suffix for reporter output files.

property file_unique: bool🔗

Whether output files should have unique names.

static find_options_by_type(
options: Iterable[ReporterOptions] | None,
cls: type[T],
) → T | None[source]🔗

Retrieve an instance of type cls (if present) from a collection of ReporterOptions.

This is used to extract reporter specific options from an iterable container of generic ReporterOptions such as those associated with a Choice or Case.

For example, a CSVReporter may define a CSVReporterOptions class that extends ReporterOptions and use this method to extract the CSVReporterOptions instance from the options iterable:

options = Reporter.find_options_by_type(case.options, CSVReporterOptions)

Parameters:
Returns:

The instance of the class cls if found, otherwise None.

Return type:

T | None

get_all_stats_values(
results: list[Results],
section: Section,
) → list[float][source]🔗

Gathers all primary statistical values for a given section across multiple results.

It collects mean, median, minimum, maximum, 5th percentile, and 95th percentile, from each Results instance for the specified section.

This method is useful in determining appropriate scaling factors or units for reporting by analyzing the range of values across all results.

Adjusted standard deviation is not included in this collection because it can be NaN for results with insufficient data points, or orders of magnitude different from the other statistics, which can skew scaling calculations.

Parameters:

  • results (list[Results]) – A list of Results instances to gather statistics from.

  • section (Section) – The section to gather statistics for.

Returns:

A list of all gathered statistical values.

Return type:

list[float]

get_base_unit_for_section(
section: Section,
) → str[source]🔗

Return the base unit for the specified section.

Parameters:

section (Section) – The section to get the base unit for.

Returns:

The base unit for the section.

Return type:

str

classmethod get_default_options() → ReporterOptions[source]🔗

Get the default options for the reporter.

Returns the default options set via set_default_options() if set, otherwise returns the built-in hardcoded default options from get_hardcoded_default_options().

Returns:

The default options.

Return type:

ReporterOptions

classmethod get_hardcoded_default_options() → Any[source]🔗

Get the built-in hardcoded default options for the reporter.

This abstract method must be implemented by all Reporter subclasses. It defines the base class default options for a reporter and must be available in all Reporter subclasses.

It returns the hardcoded default ReporterOptions sub-class instance specific to the reporter.

Sub-classes must implement the following class variables:

_OPTIONS_TYPE: ClassVar[type[MyOptions]] = MyOptions _OPTIONS_KWARGS: ClassVar[dict[str, Any]] = {…}

or the method will raise an exception.

Returns:

The built-in hardcoded default ReporterOptions instance.

Raises:

SimpleBenchNotImplementedError – If required class variables are not implemented or are of incorrect types.

property name: str🔗

The unique identifying name of the reporter.

property options_type: type[ReporterOptions]🔗

The specific ReporterOptions subclass associated with this reporter.

abstract render(
*,
case: Case,
section: Section,
options: ReporterOptions,
) → str | bytes | Text | Table[source]🔗

Render the report for a specific case and section.

This abstract method must be implemented by all Reporter subclasses. It is responsible for generating the actual report content for a given case and section, based on the provided options.

The output can be a string, bytes, or a Rich object (Text or Table).

Parameters:
Returns:

The rendered report content.

Return type:

str | bytes | Text | Table

Raises:

NotImplementedError – If the method is not implemented in a subclass.

report(
*,
log_metadata: ReportLogMetadata,
args: Namespace,
case: Case,
choice: Choice,
path: Path | None = None,
session: Session | None = None,
callback: ReporterCallback | None = None,
) → None[source]🔗

Generate a report based on the benchmark results.

This method performs validation and then calls the subclass’s run_report() method.

Parameters:

  • log_metadata (ReportLogMetadata) – The metadata for the report log.

  • args (Namespace) – The parsed command-line arguments.

  • case (Case) – The Case instance containing benchmark results.

  • choice (Choice) – The Choice instance specifying the report configuration.

  • path (Path | None, optional) – The path to the directory where the report can be saved if needed. Leave as None if not saving to the filesystem. Defaults to None.

  • session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.

  • callback (ReporterCallback | None, optional) – A callback function for additional processing of the report. Defaults to None.

run_report(
*,
args: Namespace,
log_metadata: ReportLogMetadata,
case: Case,
choice: Choice,
path: Path | None = None,
session: Session | None = None,
callback: ReporterCallback | None = None,
) → None[source]🔗

Orchestration hook for report generation.

This method is the primary customization point for controlling how a report is generated. It is called by the public report() method after all inputs have been validated.

The default implementation calls render_by_section(), which is suitable for most reporters. Subclasses can override this method to provide alternative orchestration, such as calling render_by_case() for reports that are generated once per case.

Note

This is also the correct place to implement custom logic for non-standard targets like CUSTOM.

Parameters:

  • args (Namespace) – The parsed command-line arguments.

  • log_metadata (:class:`~simplebench.reporters.log.report_log_metadata.Report) – The metadata for the report log.

  • case (Case) – The Case instance representing the benchmarked code.

  • choice (Choice) – The Choice instance specifying the report configuration.

  • path (Path | None, optional) – The path to the directory where report files will be saved. Defaults to None.

  • session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.

  • callback (ReporterCallback | None, optional) – A callback function for additional processing. Defaults to None.

classmethod set_default_options(
options: ReporterOptions | None = None,
) → None[source]🔗

Set the default options for the reporter.

Parameters:

options (ReporterOptions or None, optional) – The options to set as the default, defaults to None

property subdir: str🔗

The subdirectory where report files will be saved.

supported_formats() → frozenset[Format][source]🔗

The set of supported Format for the reporter.

This is the set of Format that the reporter can output in.

Defined Choice can only include Format that are declared in this set.

supported_sections() → frozenset[Section][source]🔗

The set of supported Section for the reporter.

This is the set of Section that the reporter can include in its reports.

Defined Choice can only include Section that are declared in this set.

supported_targets() → frozenset[Target][source]🔗

The set of supported Target for the reporter.

This is the set of Target that the reporter can output to.

Defined Choice can only include Target that are declared in this set.

class simplebench.reporters.reporter.ReporterConfig(
*,
name: str,
description: str,
sections: frozenset[Section],
targets: frozenset[Target],
default_targets: frozenset[Target],
formats: frozenset[Format],
choices: ChoicesConf,
file_suffix: str,
file_unique: bool,
file_append: bool,
subdir: str,
)[source]🔗

Bases: object

Immutable, attribute-based base configuration for a Reporter.

This frozen dataclass serves as the foundation for all specific reporter configuration classes (e.g., RichTableConfig). It defines the common data structure and centralizes the validation of all parameters required by reporters.

The sections, targets, and formats parameters act as master lists, constraining the values that can be used within the choices and default_targets parameters.

As a frozen dataclass, instances of this class are immutable; their state cannot be changed after creation. Validation and normalization of inputs are performed automatically in the __post_init__ method.

name🔗

The unique name for the reporter (e.g., ‘rich-table’).

Type:

str

description🔗

A short description of what the reporter does.

Type:

str

sections🔗

The master set of sections this reporter can handle.

Type:

frozenset[Section]

targets🔗

The master set of targets this reporter can output to.

Type:

frozenset[Target]

default_targets🔗

The default subset of targets to use.

Type:

frozenset[Target]

formats🔗

The master set of formats this reporter can produce.

Type:

frozenset[Format]

choices🔗

Defines the reporter’s command-line interface flags.

Type:

ChoicesConf

file_suffix🔗

The file extension for filesystem targets, without the leading dot.

Type:

str

file_unique🔗

If True, generate a unique filename for each output.

Type:

bool

file_append🔗

If True, append to the output file if it already exists.

Type:

bool

subdir🔗

The subdirectory for saved files; '' means the root results directory.

Type:

str

choices: ChoicesConf🔗

A ChoicesConf object defining the reporter’s command-line interface flags. These flags allow end-users to precisely control report generation by specifying which sections to include, what output format to use, and where to send the report (targets). They can also control other reporter-specific options.

default_targets: frozenset[Target]🔗

The default subset of targets to use if not specified. Must be a subset of the main targets set.

description: str🔗

A short description of what the reporter does. Cannot be empty or blank.

file_append: bool🔗

If True, append to the output file if it already exists. Mutually exclusive with file_unique; exactly one must be True.

file_suffix: str🔗

The file extension to use for filesystem targets (e.g., ‘txt’), without the leading dot.

The suffix must consist only of alphanumeric characters and be 10 characters or less in length.

file_unique: bool🔗

If True, generate a unique filename for each output. Mutually exclusive with file_append; exactly one must be True.

formats: frozenset[Format]🔗

The master set of Format enums this reporter can produce. This constrains the formats that can be used by any ChoiceConf in the choices list.

name: str🔗

The unique name for the reporter (e.g., ‘rich-table’). Cannot be empty or blank.

sections: frozenset[Section]🔗

The master set of Section enums this reporter can handle. This constrains the sections that can be used by any ChoiceConf in the choices list.

subdir: str🔗

The subdirectory within the results directory to save files to. An empty string ('') specifies the root of the results directory (i.e., no subdirectory).

A subdirectory path should only contain valid directory name elements separated by forward slashes ('/'). Each element must consist only of alphanumeric, underscore, or dash characters, cannot be longer than 64 characters and cannot start or end with a dash or underscore.

Empty elements (i.e., consecutive slashes) are not allowed. Paths cannot start or end with a slash.

The total length of the subdirectory path must not exceed 255 characters.

targets: frozenset[Target]🔗

The master set of Target enums this reporter can output to. This constrains the targets that can be used by default_targets and any ChoiceConf in the choices list.

class simplebench.reporters.reporter.ReporterOptions[source]🔗

Bases: object

Marker base class for reporter related options.

This class serves as a base for all reporter-specific options classes used within the simplebench framework. It provides a common base type for all reporter options.

These options classes can be used to encapsulate configuration settings specific to different reporter implementations and are typically used in Choice() and Case() objects to customize reporter behavior.

Subpackages🔗

Submodules🔗