simplebench.reporters.reporter.reporter module🔗

Reporter base class.

This module defines the Reporter abstract base class, which serves as the foundation for all reporter implementations in the SimpleBench benchmarking framework.

It handles common functionality such as validating input arguments, configuring argparse CLI arguments, managing default options, sending reports to various targets, and orchestrating report generation.

To create a new reporter, a developer must subclass Reporter and implement the abstract render() method. For most reporters, the default run_report() implementation, which renders a report for each section, is sufficient.

A Reporter is responsible for generating reports based on benchmark results from a Session and Case. Reporters can produce reports in various formats and output them to different targets.

class simplebench.reporters.reporter.reporter.Reporter( config: ReporterConfig, )[source]🔗

Bases: ABC, _ReporterArgparseMixin, _ReporterOrchestrationMixin, _ReporterPrioritizationMixin, _ReporterTargetMixin, ReporterProtocol

Base class for Reporter classes.

A Reporter is responsible for generating reports based on benchmark results from a Session and Case. Reporters can produce reports in various formats and output them to different targets.

All Reporter subclasses must implement the methods defined in this interface. Reporters should handle their own output, whether to console, file system, HTTP endpoint, display device, via a callback or other output.

The Reporter interface ensures that all reporters provide a consistent set of functionalities, making it easier to manage and utilize different

reporting options within the SimpleBench framework.

Initialize the Reporter instance.

Parameters:: config (ReporterConfig) – The configuration object for the reporter.
Raises:: SimpleBenchTypeError – If the provided config is not a ReporterConfig instance.

add_choice(choice: None) → None[source]🔗

Add a Choice to the reporter’s choices.

Parameters:

choice (Choice) – The Choice instance to add.

Raises:

SimpleBenchTypeError – If the provided choice is not a Choice instance.
SimpleBenchValueError – If the choice’s sections, targets, or formats are not supported by the reporter.

property choices: Choices🔗

The Choices for the reporter.

The Choices instance contains one or more Choice instances, each representing a specific combination of sections, targets, and formats, command line flags, and descriptions.

This property allows access to the reporter’s choices for generating reports and customizing report output and available options.

Returns:: The Choices instance for the reporter.
Return type:: Choices

property config: ReporterConfig🔗: The configuration object for the reporter.

property default_targets: frozenset[Target]🔗: The default set of Targets for the reporter.

property description: str🔗: A brief description of the reporter.

property file_append: bool🔗: Whether output files should be appended to.

property file_suffix: str🔗: The file suffix for reporter output files.

property file_unique: bool🔗: Whether output files should have unique names.

static find_options_by_type( options: Iterable[ReporterOptions] | None, cls: type[T], ) → T | None[source]🔗

Retrieve an instance of type cls (if present) from a collection of ReporterOptions.

This is used to extract reporter specific options from an iterable container of generic ReporterOptions such as those associated with a Choice or Case.

For example, a CSVReporter may define a CSVReporterOptions class that extends ReporterOptions and use this method to extract the CSVReporterOptions instance from the options iterable:

options = Reporter.find_options_by_type(case.options, CSVReporterOptions)

Parameters:

options (Iterable[ReporterOptions]) – An iterable of ReporterOptions instances.
cls (type[T]) – The specific subclass type of ReporterOptions to find.

Returns:

The instance of the class cls if found, otherwise None.

Return type:

T | None

get_all_stats_values( results: list[Results], section: Section, ) → list[float][source]🔗

Gathers all primary statistical values for a given section across multiple results.

It collects mean, median, minimum, maximum, 5th percentile, and 95th percentile, from each Results instance for the specified section.

This method is useful in determining appropriate scaling factors or units for reporting by analyzing the range of values across all results.

Adjusted standard deviation is not included in this collection because it can be NaN for results with insufficient data points, or orders of magnitude different from the other statistics, which can skew scaling calculations.

Parameters:

results (list[Results]) – A list of Results instances to gather statistics from.
section (Section) – The section to gather statistics for.

Returns:

A list of all gathered statistical values.

Return type:

list[float]

get_base_unit_for_section( section: Section, ) → str[source]🔗

Return the base unit for the specified section.

Parameters:: section (Section) – The section to get the base unit for.
Returns:: The base unit for the section.
Return type:: str

classmethod get_default_options() → ReporterOptions[source]🔗

Get the default options for the reporter.

Returns the default options set via set_default_options() if set, otherwise returns the built-in hardcoded default options from get_hardcoded_default_options().

Returns:: The default options.
Return type:: ReporterOptions

classmethod get_hardcoded_default_options() → Any[source]🔗

Get the built-in hardcoded default options for the reporter.

This abstract method must be implemented by all Reporter subclasses. It defines the base class default options for a reporter and must be available in all Reporter subclasses.

It returns the hardcoded default ReporterOptions sub-class instance specific to the reporter.

Sub-classes must implement the following class variables:

_OPTIONS_TYPE: ClassVar[type[MyOptions]] = MyOptions _OPTIONS_KWARGS: ClassVar[dict[str, Any]] = {…}

or the method will raise an exception.

Returns:: The built-in hardcoded default ReporterOptions instance.
Raises:: SimpleBenchNotImplementedError – If required class variables are not implemented or are of incorrect types.

property name: str🔗: The unique identifying name of the reporter.

property options_type: type[ReporterOptions]🔗: The specific ReporterOptions subclass associated with this reporter.

abstract render( *, case: Case, section: Section, options: ReporterOptions, ) → str | bytes | Text | Table[source]🔗

Render the report for a specific case and section.

This abstract method must be implemented by all Reporter subclasses. It is responsible for generating the actual report content for a given case and section, based on the provided options.

The output can be a string, bytes, or a Rich object (Text or Table).

Parameters:

case (Case) – The Case instance containing the benchmark results.
section (Section) – The specific Section of the results to render.
options (ReporterOptions) – The reporter-specific ReporterOptions for rendering.

Returns:

The rendered report content.

Return type:

str | bytes | Text | Table

Raises:

NotImplementedError – If the method is not implemented in a subclass.

report( *, log_metadata: ReportLogMetadata, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Generate a report based on the benchmark results.

This method performs validation and then calls the subclass’s run_report() method.

Parameters:

log_metadata (ReportLogMetadata) – The metadata for the report log.
args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance containing benchmark results.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where the report can be saved if needed. Leave as None if not saving to the filesystem. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing of the report. Defaults to None.

run_report( *, args: Namespace, log_metadata: ReportLogMetadata, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Orchestration hook for report generation.

This method is the primary customization point for controlling how a report is generated. It is called by the public report() method after all inputs have been validated.

The default implementation calls render_by_section(), which is suitable for most reporters. Subclasses can override this method to provide alternative orchestration, such as calling render_by_case() for reports that are generated once per case.

Note

This is also the correct place to implement custom logic for non-standard targets like CUSTOM.

Parameters:

args (Namespace) – The parsed command-line arguments.
log_metadata (:class:`~simplebench.reporters.log.report_log_metadata.Report) – The metadata for the report log.
case (Case) – The Case instance representing the benchmarked code.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where report files will be saved. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing. Defaults to None.

classmethod set_default_options( options: ReporterOptions | None = None, ) → None[source]🔗

Set the default options for the reporter.

Parameters:: options (ReporterOptions or None, optional) – The options to set as the default, defaults to None

property subdir: str🔗: The subdirectory where report files will be saved.

supported_formats() → frozenset[Format][source]🔗

The set of supported Format for the reporter.

This is the set of Format that the reporter can output in.

Defined Choice can only include Format that are declared in this set.

supported_sections() → frozenset[Section][source]🔗

The set of supported Section for the reporter.

This is the set of Section that the reporter can include in its reports.

Defined Choice can only include Section that are declared in this set.

supported_targets() → frozenset[Target][source]🔗

The set of supported Target for the reporter.

This is the set of Target that the reporter can output to.

Defined Choice can only include Target that are declared in this set.

simplebench.reporters.reporter.reporter.deferred_core_imports() → None[source]🔗

Deferred import of core types to avoid circular imports during initialization.

This imports Choice when needed at runtime, preventing circular import issues during module load time while still allowing its use in creating Choice instances from ChoiceConf instances.