simplebench.reporters.reporter.protocols module🔗

Protocols for the Reporter class and its mixins.

class simplebench.reporters.reporter.protocols.ReporterProtocol(*args, **kwargs)[source]🔗

Bases: Protocol

Protocol for the Reporter class and its mixins.

add_boolean_flags_to_argparse( parser: ArgumentParser, choice: Choice, ) → None[source]🔗

Add boolean flags for a Choice to an ArgumentParser.

This method adds a --<choice.flag> flag to the parser, which acts as a boolean switch to enable the choice.

Parameters:

parser (ArgumentParser) – The ArgumentParser to add the flags to.
choice (Choice) – The Choice to add the flags for.

add_choice( choice: Choice, ) → None[source]🔗

Add a Choice to the reporter’s choices.

Parameters:

choice (Choice) – The Choice instance to add.

Raises:

SimpleBenchTypeError – If the provided choice is not a Choice instance.
SimpleBenchValueError – If the choice’s sections, targets, or formats are not supported by the reporter.

add_flags_to_argparse( parser: ArgumentParser, ) → None[source]🔗

Add command-line flags for the reporter’s choices to an ArgumentParser.

This method iterates through the reporter’s choices and adds the appropriate command-line flags for each choice to the provided ArgumentParser.

Parameters:: parser (ArgumentParser) – The ArgumentParser to add the flags to.

add_list_of_targets_flags_to_argparse( parser: ArgumentParser, choice: Choice, ) → None[source]🔗

Add a list of target flags for a Choice to an ArgumentParser.

This method adds a --<choice.flag>-targets flag to the parser, allowing users to specify a comma-separated list of output targets for the choice.

Parameters:

parser (ArgumentParser) – The ArgumentParser to add the flags to.
choice (Choice) – The Choice to add the flags for.

property choices: Choices🔗

The Choices for the reporter.

The Choices instance contains one or more Choice instances, each representing a specific combination of sections, targets, and formats, command line flags, and descriptions.

This property allows access to the reporter’s choices for generating reports and customizing report output and available options.

Returns:: The Choices instance for the reporter.
Return type:: Choices

property config: ReporterConfig🔗: The configuration object for the reporter.

property default_targets: frozenset[Target]🔗: The default set of Targets for the reporter.

property description: str🔗: A brief description of the reporter.

dispatch_to_targets( *, output: str | bytes | Text | Table, log_metadata: ReportLogMetadata, filename_base: str, args: Namespace, choice: Choice, case: Case, section: Section, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Deliver the rendered output to the specified targets.

This method orchestrates sending the generated report content to the various output targets (console, filesystem, callback) based on the selected options.

Parameters:

output (str | bytes | Text | Table) – The rendered report content.
log_metadata (ReportLogMetadata) – The metadata for the report log.
filename_base (str) – The base name for the output file.
args (Namespace) – The parsed command-line arguments.
choice (Choice) – The Choice instance for the report.
case (Case) – The Case instance for the report.
section (Section) – The Section of the report.
path (Path | None, optional) – The output path for filesystem targets. Defaults to None.
session (Session | None, optional) – The Session instance for the report. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for callback targets. Defaults to None.

property file_append: bool🔗: Whether output files should be appended to.

property file_suffix: str🔗: The file suffix for reporter output files.

property file_unique: bool🔗: Whether output files should have unique names.

static find_options_by_type( options: Iterable[ReporterOptions] | None, cls: type[T], ) → T | None[source]🔗

Retrieve an instance of type cls (if present) from a collection of ReporterOptions.

This is used to extract reporter specific options from an iterable container of generic ReporterOptions such as those associated with a Choice or Case.

For example, a CSVReporter may define a CSVReporterOptions class that extends ReporterOptions and use this method to extract the CSVReporterOptions instance from the options iterable:

options = Reporter.find_options_by_type(case.options, CSVReporterOptions)

Parameters:

options (Iterable[ReporterOptions]) – An iterable of ReporterOptions instances.
cls (type[T]) – The specific subclass type of ReporterOptions to find.

Returns:

The instance of the class cls if found, otherwise None.

Return type:

T | None

get_all_stats_values( results: list, section: Section, ) → list[float][source]🔗

Gathers all primary statistical values for a given section across multiple results.

It collects mean, median, minimum, maximum, 5th percentile, and 95th percentile, from each Results instance for the specified section.

This method is useful in determining appropriate scaling factors or units for reporting by analyzing the range of values across all results.

Adjusted standard deviation is not included in this collection because it can be NaN for results with insufficient data points, or orders of magnitude different from the other statistics, which can skew scaling calculations.

Parameters:

results (list[Results]) – A list of Results instances to gather statistics from.
section (Section) – The section to gather statistics for.

Returns:

A list of all gathered statistical values.

Return type:

list[float]

get_base_unit_for_section( section: Section, ) → str[source]🔗

Return the base unit for the specified section.

Parameters:: section (Section) – The section to get the base unit for.
Returns:: The base unit for the section.
Return type:: str

classmethod get_default_options() → ReporterOptions[source]🔗

Get the default options for the reporter.

Returns the default options set via set_default_options() if set, otherwise returns the built-in hardcoded default options from get_hardcoded_default_options().

Returns:: The default options.
Return type:: ReporterOptions

classmethod get_hardcoded_default_options() → ReporterOptions[source]🔗

Get the built-in hardcoded default options for the reporter.

This abstract method must be implemented by all Reporter subclasses. It defines the base class default options for a reporter and must be available in all Reporter subclasses.

It returns the hardcoded default ReporterOptions sub-class instance specific to the reporter.

Sub-classes must implement the following class variables:

_OPTIONS_TYPE: ClassVar[type[MyOptions]] = MyOptions _OPTIONS_KWARGS: ClassVar[dict[str, Any]] = {…}

or the method will raise an exception.

Returns:: The built-in hardcoded default ReporterOptions instance.
Raises:: SimpleBenchNotImplementedError – If required class variables are not implemented or are of incorrect types.

get_prioritized_default_targets( choice: Choice, ) → frozenset[Target][source]🔗

Get the prioritized default targets from the choice or reporter defaults.

Parameters:: choice (Choice) – The choice to get the prioritized default targets for.
Returns:: The prioritized default targets.
Return type:: frozenset[Target]

get_prioritized_file_append( choice: Choice, ) → bool[source]🔗

Get the prioritized file append flag from the choice or reporter.

Parameters:: choice (Choice) – The choice to get the prioritized file append flag for.
Returns:: The prioritized file append flag.
Return type:: bool

get_prioritized_file_suffix( choice: Choice, ) → str[source]🔗

Get the prioritized file suffix from the choice or reporter.

Parameters:: choice (Choice) – The choice to get the prioritized file suffix for.
Returns:: The prioritized file suffix.
Return type:: str

get_prioritized_file_unique( choice: Choice, ) → bool[source]🔗

Get the prioritized file unique flag from the choice or reporter.

Parameters:: choice (Choice) – The choice to get the prioritized file unique flag for.
Returns:: The prioritized file unique flag.
Return type:: bool

get_prioritized_options( case: Case, choice: Choice, ) → ReporterOptions[source]🔗

Get the prioritized ReporterOptions for the given case and choice.

Parameters:

case (Case) – The case to get the prioritized options for.
choice (Choice) – The choice to get the prioritized options for.

Returns:

The prioritized options.

Return type:

ReporterOptions

get_prioritized_subdir( choice: Choice, ) → str[source]🔗

Get the prioritized subdirectory from the choice or reporter defaults.

Parameters:: choice (Choice) – The choice to get the prioritized subdirectory for.
Returns:: The prioritized subdirectory.
Return type:: str

property name: str🔗: The unique identifying name of the reporter.

render( *, case: Case, section: Section, options: ReporterOptions, ) → str | bytes | Text | Table[source]🔗

Render the report for a specific case and section.

This abstract method must be implemented by all Reporter subclasses. It is responsible for generating the actual report content for a given case and section, based on the provided options.

The output can be a string, bytes, or a Rich object (Text or Table).

Parameters:

case (Case) – The Case instance containing the benchmark results.
section (Section) – The specific Section of the results to render.
options (ReporterOptions) – The reporter-specific ReporterOptions for rendering.

Returns:

The rendered report content.

Return type:

str | bytes | Text | Table

Raises:

NotImplementedError – If the method is not implemented in a subclass.

render_by_case( *, renderer: ReportRenderer, log_metadata: ReportLogMetadata, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Render a single report for the entire case.

This method is suitable for reporters that generate a single, consolidated output for all sections of the benchmark results.

Parameters:

renderer (ReportRenderer) – A callable that takes a case, section (which will be None), and options, and returns the rendered output.
log_metadata (ReportLogMetadata) – The metadata for the report log.
args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance for the report.
choice (Choice) – The Choice instance for the report.
path (Path | None, optional) – The output path for filesystem targets. Defaults to None.
session (Session | None, optional) – The Session instance for the report. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for callback targets. Defaults to None.

render_by_section( *, renderer: ReportRenderer, log_metadata: ReportLogMetadata, args: Namespace, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Render a report by iterating through each section specified in the choice.

This method is suitable for reporters that generate a separate output for each section of the benchmark results (e.g., OPS, TIMING, MEMORY).

Parameters:

renderer (ReportRenderer) – A callable that takes a case, section, and options, and returns the rendered output.
log_metadata (ReportLogMetadata) – The metadata for the report log.
args (Namespace) – The parsed command-line arguments.
case (Case) – The Case instance for the report.
choice (Choice) – The Choice instance for the report.
path (Path | None, optional) – The output path for filesystem targets. Defaults to None.
session (Session | None, optional) – The Session instance for the report. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for callback targets. Defaults to None.

report( *, args: Namespace, log_metadata: ReportLogMetadata, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Generate a report based on the benchmark results.

This method performs validation and then calls the subclass’s run_report() method.

Parameters:

args (Namespace) – The parsed command-line arguments.
log_metadata (ReportLogMetadata) – The metadata for the report log.
case (Case) – The Case instance containing benchmark results.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where the report can be saved if needed. Leave as None if not saving to the filesystem. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing of the report. Defaults to None.

rich_text_to_plain_text( rich_text: Text | Table, ) → str[source]🔗

Convert a Rich Text or Table object to plain text.

Parameters:: rich_text (Text | Table) – The Rich object to convert.
Returns:: The plain text representation of the Rich object.
Return type:: str

run_report( *, args: Namespace, log_metadata: ReportLogMetadata, case: Case, choice: Choice, path: Path | None = None, session: Session | None = None, callback: ReporterCallback | None = None, ) → None[source]🔗

Orchestration hook for report generation.

This method is the primary customization point for controlling how a report is generated. It is called by the public report() method after all inputs have been validated.

The default implementation calls render_by_section(), which is suitable for most reporters. Subclasses can override this method to provide alternative orchestration, such as calling render_by_case() for reports that are generated once per case.

Note

This is also the correct place to implement custom logic for non-standard targets like CUSTOM.

Parameters:

args (Namespace) – The parsed command-line arguments.
log_metadata (ReportLogMetadata) – The metadata for the report log.
case (Case) – The Case instance representing the benchmarked code.
choice (Choice) – The Choice instance specifying the report configuration.
path (Path | None, optional) – The path to the directory where report files will be saved. Defaults to None.
session (Session | None, optional) – The Session instance containing benchmark results. Defaults to None.
callback (ReporterCallback | None, optional) – A callback function for additional processing. Defaults to None.

select_targets_from_args( *, args: Namespace, choice: Choice, default_targets: Iterable[Target], ) → set[Target][source]🔗

Select output targets based on command-line arguments and Choice configuration.

This method determines the final set of output targets for a given choice by prioritizing command-line arguments over the choice’s default targets.

Parameters:

args (Namespace) – The parsed command-line arguments.
choice (Choice) – The Choice to select targets for.
default_targets (Iterable[Target]) – The default targets to use if no command-line arguments are provided.

Returns:

A set of the selected output targets.

Return type:

set[Target]

classmethod set_default_options( options: ReporterOptions | None = None, ) → None[source]🔗

Set the default options for the reporter.

Parameters:: options (ReporterOptions or None, optional) – The options to set as the default, defaults to None

property subdir: str🔗: The subdirectory where report files will be saved.

supported_formats() → frozenset[Format][source]🔗

The set of supported Format for the reporter.

This is the set of Format that the reporter can output in.

Defined Choice can only include Format that are declared in this set.

supported_sections() → frozenset[Section][source]🔗

The set of supported Section for the reporter.

This is the set of Section that the reporter can include in its reports.

Defined Choice can only include Section that are declared in this set.

supported_targets() → frozenset[Target][source]🔗

The set of supported Target for the reporter.

This is the set of Target that the reporter can output to.

Defined Choice can only include Target that are declared in this set.

target_callback( callback: ReporterCallback | None, case: Case, section: Section, output_format: Format, output: str | bytes | Text | Table, ) → None[source]🔗

Send report data to a callback function.

Parameters:

callback (ReporterCallback | None) – The callback function to call.
case (Case) – The Case instance for the report.
section (Section) – The Section of the report.
output_format (Format) – The Format of the output.
output (str | bytes | Text | Table) – The report content to send.

target_console( session: Session | None, output: str | bytes | Text | Table, ) → None[source]🔗

Output report data to the console.

This method uses the console associated with the Session to print the report output.

Parameters:

session (Session | None) – The Session instance containing the console.
output (str | bytes | Text | Table) – The content to print to the console.

target_filesystem( *, log_metadata: ReportLogMetadata, path: Path | None, subdir: str, filename: str, output: str | bytes | Text | Table, unique: bool, append: bool, ) → None[source]🔗

Write report data to the filesystem.

This method handles the logic for writing report output to a file, including creating directories, handling unique filenames, and appending to existing files.

Parameters:

log_metadata (ReportLogMetadata) – The metadata for the report log.
path (Path | None) – The base output path.
subdir (str) – The subdirectory within the base path to write to.
filename (str) – The name of the file to write.
output (str | bytes | Text | Table) – The content to write to the file.
unique (bool) – Whether to generate a unique filename if the file already exists.
append (bool) – Whether to append to the file if it already exists.