simplebench.reporters.json.reporter.reporter module🔗

Reporter for benchmark results using JSON files.

class simplebench.reporters.json.reporter.reporter.JSONReporter(
config: JSONConfig | None = None,
)[source]🔗

Bases: Reporter

Class for outputting benchmark results to JSON files.

It supports reporting statistics for various sections, either separately or together, to the filesystem, via a callback function, or to the console in JSON format.

The JSON files are tagged with metadata comments including the case title, description, and units for clarity.

Defined command-line flags:

  • --json: {filesystem, console, callback} (default=filesystem) Outputs statistical results to JSON.

  • --json-data: {filesystem, console, callback} (default=filesystem) Outputs results to JSON with full data.

Example usage:

program.py --json               # Outputs results to JSON files in the filesystem (default).
program.py --json filesystem    # Outputs results to JSON files in the filesystem.
program.py --json console       # Outputs results to the console in JSON format.
program.py --json callback      # Outputs results via a callback function in JSON format.
program.py --json filesystem console  # Outputs results to both JSON files and the console.
Variables:

  • name (str) – The unique identifying name of the reporter.

  • description (str) – A brief description of the reporter.

  • choices (Choices) – A collection of Choices instances defining the reporter instance, CLI flags, Choice name, supported Section objects, supported output Target objects, and supported output Format objects for the reporter.

  • targets (set[Target]) – The supported output targets for the reporter.

  • formats (set[Format]) – The supported output formats for the reporter.

Initialize the JSONReporter.

Note

The exception documentation below refers to validation of subclass configuration class variables _OPTIONS_TYPE and _OPTIONS_KWARGS. These must be correctly defined in any subclass of JSONReporter to ensure proper functionality.

Parameters:

config (JSONConfig | None) – An optional configuration object to override default reporter settings. If not provided, default settings will be used.

Raises:
render(
*,
case: Case,
section: Section,
options: ReporterOptions,
) → str[source]🔗

Convert the Case data for all sections to a JSON string.

Machine info is included in the JSON output under the ‘metadata’ key.

Parameters:

  • case – The Case instance holding the benchmarked code statistics.

  • section – The Section to render (ignored, all sections are included).

  • options – The JSONOptions instance specifying rendering options or None if not provided. (JSONOptions is a subclass of ReporterOptions.)

Returns:

The JSON string representation of the Case data.

run_report(
*,
args: Namespace,
log_metadata: ReportLogMetadata,
case: Case,
choice: Choice,
path: Path | None = None,
session: Session | None = None,
callback: ReporterCallback | None = None,
) → None[source]🔗

Output the benchmark results to a file as tagged JSON if available.

This method is called by the base class’s report() method after validation. The base class handles validation of the arguments, so subclasses can assume the arguments are valid without a large amount of boilerplate code. The base class also handles lazy loading of the reporter classes, so subclasses can assume any required imports are available.

The run_report() method’s main responsibilities are to select the appropriate output method (render_by_case() in this case) based on the provided arguments and to pass the actual rendering method to be used (the render() method in this case). The rendering method must conform with the ReportRenderer protocol.

Parameters:

  • args – The parsed command-line arguments.

  • log_metadata – The ReportLogMetadata instance containing metadata about the report being generated.

  • case – The Case instance representing the benchmarked code.

  • choice – The Choice instance specifying the report configuration.

  • path – The path to the directory where the JSON file(s) will be saved.

  • reports_log_path – The path to the reports log file.

  • session – The Session instance containing benchmark results.

  • callback – A callback function for additional processing of the report. The function should accept two arguments: the Case instance and the JSON data as a string. Leave as None if no callback is needed.