simplebench package🔗

Simple benchmarking framework.

class simplebench.CSVOptions(
fields: Sequence[CSVField] | None = None,
variation_cols_last: bool = False,
)[source]🔗

Bases: ReporterOptions

Class for holding CSV reporter specific options in a Choice.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice instance.

Parameters:

  • fields –

    A tuple of CSV fields to include in the output. If none is specifically set, a predefined set of fields is used. The fields appear in the order specified in the sequence.

    If specified, all fields must be from the CSVField enum.

    The default fields, in order, are:

  • variation_cols_last – Whether to place the variation columns (if any) at the end of the rows. Defaults to False - which places the variation columns at the start of the rows.

Raises:

Initialize CSVOptions instance.

property fields: tuple[CSVField, ...]🔗

Return the fields, in order, to include in the CSV table when rendering.

Returns:

A tuple of CSVField enums representing the default fields.

property variation_cols_last: bool🔗

Return whether variation columns are placed at the end of the rows.

Returns:

True if variation columns are placed at the end, False if at the start.

class simplebench.Case(
*,
benchmark_id: str | None = None,
git_info: GitInfo | None = None,
action: ActionRunner,
group: str = 'default',
title: str | None = None,
description: str | None = None,
iterations: int = 20,
warmup_iterations: int = 10,
rounds: int | None = None,
timer: Callable[[], int] | None = None,
min_time: float = 5.0,
max_time: float = 20.0,
timeout: float | int | None = None,
variation_cols: dict[str, str] | None = None,
kwargs_variations: dict[str, list[Any]] | None = None,
runner: type[SimpleRunner] | None = None,
callback: ReporterCallback | None = None,
options: Iterable[ReporterOptions] | None = None,
)[source]🔗

Bases: object

A benchmark case defines the specific benchmark to be run, including the action to be performed, the parameters for the benchmark, and any variations of those parameters as well as the reporting group and title for the benchmark.

It also defines the number of iterations, warmup iterations, rounds, minimum and maximum time for the benchmark, the benchmark runner to use, and any callbacks to be invoked to process the results of the benchmark for reporting purposes.

The min_time, max_time, iterations, and warmup_iterations parameters control how the benchmark is executed and measured and interact with each other as follows when using the default SimpleRunner: - The benchmark will perform warmup_iterations iterations before starting the timing

and measurement phase. This is done to allow for any setup or caching effects to stabilize. This is separate from the main benchmark iterations and does not count towards the iterations count or the min_time/max_time limits.

  • The benchmark will run for at least min_time wall clock seconds, but will stop on

    completing the first iteration that ends after max_time seconds during the timing phase.

  • If the benchmark completes iterations iterations after min_time but before

    reaching max_time, it will stop.

This means that the benchmark will run for at least min_time seconds and for at least one iteration during the timing phase. If min_time is reached before iterations is completed, the benchmark will continue running until either iterations or max_time is completed (whichever happens first).

rounds specifies the number of times the action will be executed per iteration to get a better average. Each iteration will run the specified number of rounds after setup and before teardown. The timing for the iteration will be the average time taken for the rounds in that iteration.

This helps to reduce the impact of variability in execution time for a single run of the action for very fast actions. This suppresses the overhead of the loop and timer quantization in Python during the actual timing benchmark/measurement phase. Internally, the action is called rounds times in an unrolled loop for each iteration, and the average time per call is used for the iteration timing.

This removes the overhead of the loop and timer quantization in Python during the actual timing benchmark/measurement phase by aggregating multiple calls to the action within a single iteration without the overhead of looping constructs. This allows for more accurate timing of very fast actions by reducing the relative impact of loop overhead and timer resolution limitations.

The trade-off is that total number of action calls is now iterations * rounds, and the reported time per action call is an average over the rounds in each iteration. This can dramatically improve the accuracy of timing measurements for very fast actions, at the cost of increased total execution time for the benchmark due to the additional calls to the action.

The unrolled loop means that setup and teardown functions (if any) are called only once per iteration, not once per round. All rounds in the same iteration share the same setup/teardown context.

It is recommended to leave rounds to its default of None. A setting of None enables auto-calibration of rounds based on the expected execution time of the action and the precision and overhead of the timer function. If you do use it, you may want to run dual benchmarks with rounds=1 and rounds >> 1 to see how much the reported variability and other metrics change.

The Case class is designed to be immutable after creation. Once a Case instance is created, its properties cannot be directly changed. This immutability ensures that benchmark cases remain consistent throughout their lifecycle.

The results of the benchmark runs are stored in the results property, which is a list of Results objects. Each Results object corresponds to a specific combination of keyword argument variations.

Minimal Example🔗
  from simplebench import (
      Case, SimpleRunner, Results, main)


  def my_benchmark_action(_bench: SimpleRunner,
                          **kwargs) -> Results:
      # Perform benchmark action here
      def benchmark_operation():
          sum(range(1000))  # Example operation to benchmark

      return _bench.run(benchmark_operation)


  if __name__ == '__main__':
      cases_list: list[Case] = [
          Case(action=my_benchmark_action)
      ]
      main(cases_list)

The only REQUIRED parameter is action.

Parameters:

  • benchmark_id –

    An optional unique identifier for the benchmark case.

    If None, a transient ID is assigned. This is meant to provide a stable identifier for the benchmark case across multiple runs for tracking purposes. If not provided, an attempt will be made to generate a stable ID based on the the action function name, signature, and group. If that is not possible, a transient ID based on the instance’s id() will be used. If a transient ID is used, it will differ between runs and cannot be used to correlate results across multiple runs.

    Benchmark ids must be unique within a benchmarking session and stable across runs or they cannot be used for tracking benchmark results over time.

  • git_info –

    An optional GitInfo instance representing the state of the Git repository.

    If not provided, the GitInfo will be automatically retrieved from the current context of the caller if the code is part of a Git repository.

  • action –

    The function to perform the benchmark.

    This function must accept a bench instance of type SimpleRunner and arbitrary keyword arguments (’**kwargs’). See the ActionRunner protocol for the exact signature required. It must return a Results object.

  • group –

    The benchmark reporting group to which the benchmark case belongs.

    Benchmarks with the same group can be selected for execution without running other benchmarks. If not specified, the default group ‘default’ is used.

  • title –

    The title of the benchmark case.

    If None, the name of the action function will be used. Cannot be blank.

  • description –

    A brief description of the benchmark case.

    If None, the docstring of the action function will be used, or ‘(no description)’ if no docstring is available. Cannot be blank.

  • iterations – The minimum number of iterations to run for the benchmark.

  • warmup_iterations – The number of warmup iterations to run before the benchmark.

  • rounds –

    The number of rounds to run for the benchmark.

    Rounds are multiple runs of calls to the action within an iteration to mitigate timer quantization, loop overhead, and other measurement effects for very fast actions. Setup and teardown functions are called only once per iteration (all rounds in the same iteration share the same setup/teardown context).

    If None, rounds will be auto-calibrated based on the precision and overhead of the timer function and the expected execution time of the action. If the action is very fast (e.g., under 10 microseconds), rounds will be set to a higher value to improve measurement accuracy with the goal of reducing timer quantization errors. If the action is slower, rounds will be set lower values.

    If specified, it must be a positive integer.

  • timer –

    The timer function to use for the benchmark. If None, the default timer from the Session() (if set) or from simplebench.defaults.DEFAULT_TIMER (perf_counter_ns) is used by benchmark runners that require a timer.

    The timer function should be a callable that returns a float or int representing the current time.

  • min_time – The minimum time for the benchmark to run in seconds. Its reference depends on the timer used, but by default it is wall-clock time.

  • max_time – The maximum time for the benchmark run in seconds. Its reference depends on the timer used, but by default it is wall-clock time.

  • timeout –

    How long to wait before timing out a benchmark run (in seconds). It is measured as wall-clock time.

    If None, it waits the full duration of max_time plus the default timeout grace period (10.0 seconds). It must be a positive float or int that is greater than max_time if provided. This is a safety mechanism to prevent runaway benchmarks.

    If the timeout is reached during a run, a SimpleBenchTimeoutError` will be raised, and the benchmark case’s state will be set to TIMED_OUT.

  • variation_cols –

    kwargs to be used for cols to denote kwarg variations.

    Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels.

    If None, an empty dict is used.

  • kwargs_variations –

    A map of keyword argument names to a list of possible values for that argument.

    Default is {}. When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. The action function will be called with a bench parameter that is an instance of the runner and the keyword arguments for the current variation. If None, an empty dict is used.

  • runner –

    A custom runner class for the benchmark.

    Any custom runner classes must be a subclass of SimpleRunner and must have a method named run that accepts the same parameters as SimpleRunner.run and returns a Results object. The action function will be called with a bench parameter that is an instance of the custom runner. It may also accept additional parameters to the run method as needed. If additional parameters are needed for the custom runner, they will need to be passed to the run method as keyword arguments. No support is provided for passing additional parameters to a custom runner from the @benchmark decorator.

  • callback –

    A callback function for additional processing of the report.

    The function should must four arguments: the Case instance, the Section, the Format, and the generated report data.

    • case (Case): The Case instance processed for the report.

    • section (Section): The Section of the report.

    • output_format (Format): The Format of the report.

    • output (Any): The generated report data. Note that the actual type of this data will

      depend on the Format specified for the report and the type generated by the reporter for that Format

    Omit if no callback is needed by a reporter.

  • options –

    A list of additional options for the benchmark case.

    Each option is an instance of ReporterOption or a subclass of ReporterOption. Reporter options can be used to customize the output of the benchmark reports for specific reporters. Reporters are responsible for extracting applicable ReporterOptions from the list of options themselves. If None, an empty list is used.

Raises:
property action: ActionRunner🔗

The function to perform the benchmark.

The function must accept a bench parameter of type SimpleRunner and arbitrary keyword arguments (’**kwargs’) and return a Results object.

Example:

def my_benchmark_action(*, bench: SimpleRunner, **kwargs) -> Results:
    def setup_function(size: int) -> None:
        # Setup code goes here
        pass

    def teardown_function(size: int) -> None:
        # Teardown code goes here
        pass

    def action_function(size: int) -> None:
        # The code to benchmark goes here
        lst = list(range(size))

    # Perform the benchmark using the provided SimpleRunner instance
    results: Results = bench.run(
        n=kwargs.get('size', 1),
        setup=setup_function, teardown=teardown_function,
        action=action_function, **kwargs)
    return results
as_dict(
full_data: bool = False,
) → dict[str, Any][source]🔗

Returns the benchmark case and results as a JSON serializable dict.

Only the results statistics are included by default. To include full results data, set full_data to True.

Parameters:

full_data – Whether to include full results data. Defaults to False.

Returns:

A JSON serializable dict representation of the benchmark case and results.

Return type:

dict[str, Any]

property benchmark_id: str🔗

A unique identifier for the benchmark case.

It is meant to provide a stable identifier for the benchmark case across multiple runs for tracking purposes.

If not provided, an attempt will be made to generate a stable ID based on the the action function name, signature, and group. If that is not possible, a transient ID based on the instance’s id() will be used.

If a transient ID is used, it will differ between runs and cannot be used to correlate results across multiple runs.

Benchmark ids must be unique within a benchmarking session.

Passed ids are stripped of leading and trailing whitespace and validated to be non-blank.

property callback: ReporterCallback | None🔗

A callback function for additional processing of a report.

A callback function to be called with the benchmark results in a reporter. This function should accept four arguments: the Case instance, the Section, the ReporterOption, and the output object. Leave as None if no callback is needed. (default: None)

property description: str🔗

A brief description of the benchmark case.

If not specified, defaults to the docstring of the action function or ‘(no description)’ if no docstring is available.

Cannot be blank.

property expanded_kwargs_variations: list[dict[str, Any]]🔗

All combinations of keyword arguments from the specified kwargs_variations.

A mapping of keyword argument names to their variations.

Each key is a keyword argument name, and the value is a list of possible values.

When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. For example, if kwargs_variations is

kwargs_variations argument example🔗
  ...
  kwargs_variations = {
          'size': [10, 100],
          'mode': ['fast', 'accurate']
      },
  ...

The benchmark will be run 4 times with the following combinations of keyword arguments:

Keyword (**kwargs) Argument Combinations🔗
1  {size=10, mode='fast'}
2  {size=10, mode='accurate'}
3  {size=100, mode='fast'}
4  {size=100, mode='accurate'}

The action function will be called with these keyword arguments accordingly and must accept them.

Returns:

A list of dictionaries, each representing a unique combination of keyword arguments.

Return type:

list[dict[str, Any]]

property git_info: GitInfo | None🔗

Git information for the benchmark case.

This is a read-only attribute that provides git information such as the current commit hash, branch name, and repository URL. If the benchmark is not in a file managed by a git repository, a None value is returned.

Returns:

A GitInfo object containing git information, or None if not in a git repository.

Return type:

GitInfo | None

property group: str🔗

The benchmark reporting group to which the benchmark case belongs for selection and reporting purposes.

Cannot be blank. It is used to categorize and filter benchmark cases.

property iterations: int🔗

The number of iterations to run for the benchmark.

property kwargs_variations: dict[str, list[Any]]🔗

Variations of keyword arguments for the benchmark.

Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels.

When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. For example, if kwargs_variations is

kwargs_variations argument example🔗
  ...
  kwargs_variations = {
          'size': [10, 100],
          'mode': ['fast', 'accurate']
      },
  ...

The benchmark will be run 4 times with the following combinations of keyword arguments:

Keyword (**kwargs) Argument Combinations🔗
1  {size=10, mode='fast'}
2  {size=10, mode='accurate'}
3  {size=100, mode='fast'}
4  {size=100, mode='accurate'}

The action function will be called with these keyword arguments accordingly and must accept them.

property max_time: float🔗

The maximum time for the benchmark in seconds.

property min_time: float🔗

The minimum time for the benchmark in seconds.

property options: list[ReporterOptions]🔗

A list of additional options for the benchmark case.

property results: list[Results]🔗

The benchmark list of Results for the case.

This is a read-only attribute. To add results, use the run method.

Returns:

A list of Results objects for each variation run of the benchmark case.

Return type:

list[Results]

property rounds: int | None🔗

The number of rounds to run for the benchmark for each iteration.

Rounds are multiple runs of the entire benchmark to get a better average for an iteration. Each iteration will run the specified number of rounds after setup and before teardown. (default: 1)

run(
session: Session | None = None,
) → None[source]🔗

Run the benchmark tests.

This method will execute the benchmark for each combination of keyword arguments and collect the results. After running the benchmarks, the results will be stored in the self.results attribute.

If passed, the session’s tasks will be used to display progress, control verbosity, and pass CLI arguments to the benchmark runner.

Parameters:

session – The session to use for the benchmark case.

Raises:

  • SimpleBenchTimeoutError – If a timeout occurs during the benchmark action.

  • SimpleBenchBenchmarkError – If an error occurs during the benchmark action.

property runner: type[SimpleRunner] | None🔗

A custom runner class for the benchmark.

If None, the default SimpleRunner is used. (default: None)

A custom runner class must be a subclass of SimpleRunner and must have a method named run that accepts the same parameters as SimpleRunner.run and returns a Results object. The action function will be called with a bench parameter that is an instance of the custom runner.

It may also accept additional parameters to the run method as needed. If additional parameters are required, they must be specified in the action function signature.

property timeout: float🔗

The timeout for the benchmark in seconds.

property timer: Callable[[], int] | None🔗

The timer function to use for the benchmark.

If None, the default timer from the Session() (if set) or from simplebench.defaults.DEFAULT_TIMER is used by benchmark runners.

The timer function should be a callable that returns an int representing the current time.

property title: str🔗

The name of the benchmark case.

If not specified, defaults to the name of the action function. Cannot be blank.

static validate_action_signature(
action: ActionRunner,
kwargs_variations: dict[str, Any],
) → ActionRunner[source]🔗

Validate that action has correct signature.

An action function must accept one of the two following formats for its parameters:

Two Parameters

  • _bench: SimpleRunner

  • **kwargs: Arbitrary keyword arguments

Explicit Parameters

  • _bench: SimpleRunner

  • any number of explicit parameters

This is equivalent to the ActionRunner protocol.

Parameters:

  • action – The action function to validate.

  • kwargs_variations – The kwargs variations for the case.

Returns:

The validated action function.

Raises:

SimpleBenchTypeError – If the action is not callable or has an invalid signature or if the kwargs_variations is not a dictionary with valid keys or if the action signature does not match the kwargs_variations keys.

static validate_kwargs_variations(
value: dict[str, list[Any]] | None,
) → dict[str, list[Any]][source]🔗

Validate the kwargs_variations dictionary.

Validates that the kwargs_variations is a dictionary where each key is a string that is a valid Python identifier, and each value is a non-empty list.

A shallow copy of the validated dictionary and the lists is performed before returning to prevent external modification.

Parameters:

value (dict[str, list[Any]] | None) – The kwargs_variations dictionary to validate. Defaults to {} if None.

Returns:

A shallow copy of the validated kwargs_variations dictionary or {} if not provided. The keys are strings that are valid Python identifiers, and the values are non-empty lists. The lists may contain any type of values.

Return type:

dict[str, list[Any]]

Raises:

  • SimpleBenchTypeError – If the kwargs_variations is not a dictionary or if any key is not a string that is a valid Python identifier.

  • SimpleBenchValueError – If any value is not a list or is an empty list.

static validate_options(
value: Iterable[ReporterOptions] | None,
) → list[ReporterOptions][source]🔗

Validate the options list.

Parameters:

value – The options iterable to validate or None.

Returns:

A shallow copy of the validated options as a list or an empty list if not provided.

Return type:

list[ReporterOptions]

Raises:

SimpleBenchTypeError – If options is not a list or if any entry is not a ReporterOption.

static validate_runner(
value: type[SimpleRunner] | None,
) → type[SimpleRunner] | None[source]🔗

Validate the runner class.

Parameters:

value (Optional[type[SimpleRunner]]) – The runner class to validate.

Returns:

The validated runner class or None.

Return type:

Optional[type[SimpleRunner]]

Raises:

SimpleBenchTypeError – If the runner is not a subclass of SimpleRunner or None.

static validate_time_range(
min_time: float,
max_time: float,
) → None[source]🔗

Validate that min_time < max_time for the case.

Parameters:

  • min_time (float) – The minimum time.

  • max_time (float) – The maximum time.

Raises:

SimpleBenchValueError – The min_time is greater than max_time.

static validate_variation_cols(
variation_cols: dict[str, str] | None,
kwargs_variations: dict[str, list[Any]],
) → dict[str, str][source]🔗

Validate the variation_cols dictionary.

Parameters:

  • variation_cols (dict[str, str] | None) – The variation_cols dictionary to validate or None.

  • kwargs_variations (dict[str, list[Any]]) – The kwargs_variations dictionary to validate against.

Returns:

A shallow copy of the validated variation_cols dictionary or {} if not provided. Each key is a keyword argument name from kwargs_variations, and each value is a non-blank string to be used as the column label for that argument in reports.

Return type:

dict[str, str]

Raises:

  • SimpleBenchTypeError – If the variation_cols is not a dictionary or if any key or value is not a string.

  • SimpleBenchValueError – If any key is not found in kwargs_variations or if any value is a blank string.

property variation_cols: dict[str, str]🔗

Keyword arguments to be used for columns to denote kwarg variations.

Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels.

Note that all keys in variation_cols must be present in kwargs_variations and updating it may require changes to both variation_cols and kwargs_variations_cols.

Updating variation_cols does not automatically update kwargs_variations, and vice versa.

Returns:

A dictionary mapping keyword argument names to column labels.

Return type:

dict[str, str]

property warmup_iterations: int🔗

The number of warmup iterations to run before the benchmark.

class simplebench.ImageType(value)[source]🔗

Bases: str, Enum

Enumeration of image types for graph output.

PNG = 'png'🔗

PNG (Portable Network Graphics) image format.

SVG = 'svg'🔗

SVG (Scalable Vector Graphics) image format.

class simplebench.JSONOptions(*, full_data: bool = False)[source]🔗

Bases: ReporterOptions

Class for holding JSON reporter specific options in a Choice or Case.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice or Case instance.

Variables:

full_data (bool) – Whether to include full data in the JSON output.

Initialize JSONChoiceOptions with default targets and subdirectory.

Parameters:

full_data (bool) – Whether to include full data in the JSON output. Defaults to False.

property full_data: bool🔗

Return whether to include full data in the JSON output.

Returns:

Whether to include full data in the JSON output.

Return type:

bool

class simplebench.Results(
*,
group: str,
title: str,
description: str,
n: int | float,
rounds: int,
total_elapsed: float,
iterations: Sequence[Iteration],
variation_cols: dict[str, str] | None = None,
variation_marks: dict[str, Any] | None = None,
interval_unit: str = 'ns',
interval_scale: float = 1e-09,
ops_per_interval_unit: str = 'ns',
ops_per_interval_scale: float = 1e-09,
memory_unit: str = 'bytes',
memory_scale: float = 1.0,
ops_per_second: OperationsPerInterval | None = None,
per_round_timings: OperationTimings | None = None,
memory: MemoryUsage | None = None,
peak_memory: PeakMemoryUsage | None = None,
extra_info: dict[str, Any] | None = None,
)[source]🔗

Bases: object

Container for the results of a single benchmark test.

The Results class holds all relevant information about a benchmark test’s execution and its outcomes. It is immutable after creation to ensure data integrity.

Variables:

  • group (str) – The reporting group to which the benchmark case belongs. (read only)

  • title (str) – The name of the benchmark case. (read only)

  • description (str) – A brief description of the benchmark case. (read only)

  • n (int | float) – The n weighting the benchmark assigned to the iteration for purposes of Big O analysis. (read only)

  • rounds (int) – The number of rounds in the benchmark case. (read only)

  • variation_marks (MappingProxyType[str, Any]) – A dictionary of variation marks used to identify the benchmark variation. (read only)

  • variation_cols (MappingProxyType[str, str]) – The columns to use for labelling kwarg variations in the benchmark. (read only)

  • interval_unit (str) – The unit of measurement for the interval (e.g. “ns”). (read only)

  • interval_scale (float) – The scale factor for the interval (e.g. 1e-9 for nanoseconds). (read only)

  • ops_per_interval_unit (str) – The unit of measurement for operations per interval (e.g. “ops/s”). (read only)

  • ops_per_interval_scale (float) – The scale factor for operations per interval (e.g. 1.0 for ops/s). (read only)

  • memory_unit (str) – The unit of measurement for memory usage (e.g. “bytes”). (read only)

  • memory_scale (float) – The scale factor for memory usage (e.g. 1.0 for bytes). (read only)

  • iterations (tuple[Iteration, ...]) – A tuple of Iteration objects representing each iteration of the benchmark. (read only)

  • ops_per_second (OperationsPerInterval) – Statistics for operations per interval. (read only)

  • per_round_timings (OperationTimings) – Statistics for per-round timings. (read only)

  • memory (MemoryUsage) – Statistics for memory usage. (read only)

  • peak_memory (PeakMemoryUsage) – Statistics for peak memory usage. (read only)

  • total_elapsed (float) – The total elapsed time for the benchmark. (read only)

  • extra_info (MappingProxyType[str, Any]) – Additional information about the benchmark run. This is a read-only property that returns a mapping proxy to prevent external mutation. (read only)

Initialize a Results object.

Parameters:

  • group (str) – The reporting group to which the benchmark case belongs.

  • title (str) – The name of the benchmark case.

  • description (str) – A brief description of the benchmark case.

  • n (int | float) – The n weighting assigned to the iteration for purposes of Big O analysis.

  • rounds (int) – The number of rounds in the benchmark case.

  • total_elapsed (float) – The total elapsed time for the benchmark.

  • iterations (list[Iteration]) – The list of Iteration objects representing each iteration of the benchmark.

  • variation_cols (dict[str, str], optional) – The columns to use for labelling kwarg variations in the benchmark. Defaults to None, which results in an empty dictionary.

  • variation_marks (dict[str, Any], optional) – A dictionary of variation marks used to identify the benchmark variation. Defaults to None, which results in an empty dictionary.

  • interval_unit (str, optional) – The unit of measurement for the interval (e.g. “ns”). Defaults to “ns”.

  • interval_scale (float, optional) – The scale factor for the interval (e.g. 1e-9 for nanoseconds). Defaults to 1e-9.

  • ops_per_interval_unit (str, optional) – The unit of measurement for operations per interval (e.g. “ops/s”). Defaults to “ops/s”.

  • ops_per_interval_scale (float, optional) – The scale factor for operations per interval (e.g. 1.0 for ops/s). Defaults to 1.0.

  • memory_unit (str, optional) – The unit of measurement for memory usage (e.g. “bytes”). Defaults to “bytes”.

  • memory_scale (float, optional) – The scale factor for memory usage (e.g. 1.0 for bytes). Defaults to 1.0.

  • ops_per_second (Optional[OperationsPerInterval], optional) – The operations per second for the benchmark. Defaults to a new OperationsPerInterval object initialized from the benchmark’s iterations.

  • per_round_timings (Optional[OperationTimings], optional) – The per-round timings for the benchmark. Defaults to a new OperationTimings object initialized from the benchmark’s iterations.

  • memory (Optional[MemoryUsage], optional) – The memory usage for the benchmark. Defaults to a new MemoryUsage object initialized from the benchmark’s iterations.

  • peak_memory (Optional[PeakMemoryUsage], optional) – The peak memory usage for the benchmark. Defaults to a new PeakMemoryUsage object initialized from the benchmark’s iterations.

  • extra_info (Optional[dict[str, Any]], optional) – Any extra information to include in the benchmark results. Defaults to {}.

Raises:
as_dict(
full_data: bool = False,
) → dict[str, Any][source]🔗

Returns the benchmark results and statistics as a JSON-serializable dictionary.

property description: str🔗

A brief description of the benchmark case.

property extra_info: dict[str, Any]🔗

Additional information about the benchmark run.

property group: str🔗

The reporting group to which the benchmark case belongs.

property interval_scale: float🔗

The scale factor for the interval (e.g. 1e-9 for nanoseconds).

property interval_unit: str🔗

The unit of measurement for the interval (e.g. “ns”).

property iterations: tuple[Iteration, ...]🔗

The tuple of Iteration objects representing each iteration of the benchmark.

property memory: MemoryUsage🔗

Statistics for memory usage.

property memory_scale: float🔗

The scale factor for memory usage (e.g. 1.0 for bytes).

property memory_unit: str🔗

The unit of measurement for memory usage (e.g. “bytes”).

property n: float🔗

The O() complexity analysis size/weighting.

property ops_per_interval_scale: float🔗

The scale factor for operations per interval (e.g. 1.0 for ops/s).

property ops_per_interval_unit: str🔗

The unit of measurement for operations per interval (e.g. “ops/s”).

property ops_per_second: OperationsPerInterval🔗

Statistics for operations per interval.

property peak_memory: PeakMemoryUsage🔗

Statistics for peak memory usage.

property per_round_timings: OperationTimings🔗

Statistics for per-round timings.

results_section(
section: Section,
) → Stats[source]🔗

Returns the requested section of the benchmark results.

Parameters:

section (Section) – The section of the results to return. Must be Section.OPS or Section.TIMING.

Returns:

The requested section of the benchmark results.

Return type:

Stats

property rounds: int🔗

The number of rounds the benchmark ran per iteration.

property title: str🔗

The name of the benchmark case.

property total_elapsed: float🔗

The total elapsed time for the benchmark.

property variation_cols: mappingproxy[str, str]🔗

The columns to use for labelling kwarg variations in the benchmark.

property variation_marks: mappingproxy[str, Any]🔗

A dictionary of variation marks used to identify the benchmark variation.

class simplebench.RichTableOptions(
virtual_width: int | None = None,
fields: Sequence[RichTableField] | None = None,
variation_cols_last: bool = False,
)[source]🔗

Bases: ReporterOptions

Class for holding Rich table reporter specific options in a Choice.

This class provides additional configuration options specific to the JSON reporter. It is accessed via the options attribute of a Choice instance.

Parameters:

  • virtual_width –

    The width of the Rich table output in characters when rendered to the filesystem or via callback. Must be between 80 and 1000 characters or None. If None, no width constraint is applied.

    The virtual width is used to determine how the table should be formatted when rendered to non-console outputs, such as files or callbacks. This allows for better control over the appearance of the table in different contexts.

  • fields –

    A tuple of Rich Table fields to include in the output. If none is specifically set, a predefined set of fields is used. The fields appear in the order specified in the sequence.

    If specified, all fields must be from the RichTableField enum.

    The default fields, in order, are:

  • variation_cols_last – Whether to place the variation columns (if any) at the end of the rows. Defaults to False - which places the variation columns at the start of the rows.

Raises:

Initialize RichTableOptions instance.

property fields: tuple[RichTableField, ...]🔗

Return the fields, in order, to include in the Rich table when rendering.

Returns:

A tuple of RichTableField enums representing the default fields.

property variation_cols_last: bool🔗

Return whether variation columns are placed at the end of the rows.

Returns:

True if variation columns are placed at the end, False if at the start.

property virtual_width: int | None🔗

Return the virtual width of the Rich table output.

The virtual width is used when rendered to the filesystem or via callback.

Returns:

The virtual width of the Rich table output in characters.

class simplebench.ScatterPlotOptions(
width: int | None = None,
height: int | None = None,
dpi: int | None = None,
y_starts_at_zero: bool | None = None,
x_labels_rotation: float | None = None,
style: Style | None = None,
theme: Theme | None = None,
image_type: ImageType | None = None,
)[source]🔗

Bases: MatPlotLibOptions

Scatter Plot options.

Defaults are inherited from MatPlotLibOptions:

  • width: int = 1500

  • height: int = 750

  • dpi: int = 150

  • y_starts_at_zero: bool = True

  • x_labels_rotation: float = 45.0

  • style: Style = Style.DARK_BACKGROUND

  • theme: Theme = Theme.Default

  • image_type: ImageType = ImageType.SVG

Create a MatPlotLibOptions instance.

To ensure accuracy in graph rendering, the default width, height, and DPI should be set with the width and height being exact multiples of the DPI. This is because the underlying rendering engine uses DPI to determine the pixel dimensions of the output graph from the specified width and height in inches.

If the width and height are not exact multiples of the DPI, it may lead to unexpected scaling or distortion of the graph when rendered, as the rendering engine may round the pixel dimensions to the nearest whole number.

The default values provided here are chosen to balance quality and performance for most common use cases. However, users can customize these values based on their specific requirements for graph resolution and size.

The use of None for (or, equivalently the omission of) any argument indicates that the default value defined in the next lower precedence level should be used.

Parameters:

  • width (int | None) – The width in pixels of a MatPlotLib output when rendered. It should be an exact multiple of the dpi and can only be between 500 and 4000 pixels. If None, the default MatPlotLibOptions default width will be used.

  • height (int | None) – The height in pixels of a MatPlotLib output when rendered. It should be an exact multiple of the dpi and can only be between 500 and 4000 pixels. If None, the default MatPlotLibOptions default height will be used.

  • dpi (int | None) – The default DPI (dots per inch) for a MatPlotLib output. It must be between 50 and 400.

  • y_starts_at_zero (bool | None) – Whether the Y-axis should start at zero.

  • x_labels_rotation (float | None) – The rotation angle in degrees for X-axis labels.

  • style (Style | None) – The default style for the MatPlotLib graphs.

  • theme (Theme | None) – The theme to use for the MatPlotLib graphs.

  • image_type (ImageType | None) – The output format for the graph files.

Raises:
class simplebench.Session(
*,
cases: Sequence[Case] | None = None,
verbosity: Verbosity = Verbosity.NORMAL,
default_runner: type[SimpleRunner] | None = None,
args_parser: ArgumentParser | None = None,
show_progress: bool = False,
output_path: Path | None = None,
console: Console | None = None,
timer: Callable[[], int] | None = None,
)[source]🔗

Bases: object

Container for session related information while running benchmarks.

The session is responsible for managing benchmark cases, command line arguments, progress display, and report generation.

This makes it the primary orchestrator for running benchmarks and generating reports.

Container and orchestrator for session related information while running benchmarks.

Parameters:

  • cases – A Sequence of benchmark cases for the session. If None, an empty list will be created. Defaults to None.

  • verbosity – The verbosity level for console output. Defaults to Verbosity.NORMAL.

  • default_runner – The default runner class to use for Cases that do not specify a runner. If None, the default SimpleRunner is used. Defaults to None.

  • args_parser – The ArgumentParser instance for the session. If None, a new ArgumentParser will be automatically created. Defaults to None.

  • show_progress – Whether to show progress bars during execution. Defaults to False.

  • output_path – The output path for reports. Defaults to None.

  • console – A Rich Console instance for displaying output. If None, a new Console will be automatically created. Defaults to None.

  • timer – A callable that returns the current time for timing benchmarks. If None, a default timer simplebench.defaults.DEFAULT_TIMER (perf_counter_ns) will be used. Defaults to None.

Raises:

SimpleBenchTypeError – If the arguments are of the wrong type.

add_case(
case: Case,
) → None[source]🔗

Add a Case to the Cases for this session.

Parameters:

case – benchmark case to add to the Session

Raises:

SimpleBenchTypeError – If the value is not a Case instance.

add_reporter_flags() → None[source]🔗

Add the command line flags for all registered reporters to the session’s ArgumentParser.

Any conflicts in flag names with already declared ArgumentParser flags will have to be handled by the reporters themselves.

It has its own method so that a user can customize the ArgumentParser before or after adding the reporter flags as needed.

It also allows the user to unregister reporters before adding the reporter flags if they want to omit specific built-in reporters entirely.

It is called internally by parse_args() if it has not already been called and does not re-add the reporter flags if called again.

Raises:

SimpleBenchArgumentError – If there is a conflict or other error in reporter flag names.

property args: Namespace | None🔗

The command line arguments for the session. This will be None until the parse_args() method has been called.

property args_parser: ArgumentParser🔗

The ArgumentParser instance for the session.

property cases: tuple[Case]🔗

Tuple of Cases for this session.

property console: Console🔗

The Rich Console instance for displaying output.

property default_runner: type[SimpleRunner] | None🔗

The session scoped default runner class to use for Cases that do not specify a runner.

extend_cases(
cases: Sequence[Case],
) → None[source]🔗

Extend the Cases for this session.

Parameters:

cases (Sequence[Case]) – Sequence of Cases to add to the Session

Raises:

SimpleBenchTypeError – If the value is not a Sequence of Cases.

property output_path: Path | None🔗

The output path for reports.

parse_args(
args: Sequence[str] | None = None,
) → None[source]🔗

Parse the command line arguments using the session’s ArgumentParser.

This method parses the command line arguments and stores them in the session’s args property. By default, it parses the arguments from sys.argv. If args is provided, it will parse the arguments from the provided sequence of strings instead.

This can be used to customize the command line arguments for testing or other purposes.

It automatically calls add_reporter_flags() to ensure that reporter flags are added to the ArgumentParser if it has not already been called.

If you wish to customize the ArgumentParser before or after adding the reporter flags, you can do so by calling add_reporter_flags() before calling this method.

Parameters:

args (Sequence[str], optional) – A list of command line arguments to parse. If None, the arguments will be taken from sys.argv. Defaults to None.

Raises:

SimpleBenchTypeError – If the args_parser is not set.

property progress: Progress🔗

The Rich Progress instance for displaying progress bars.

report() → None[source]🔗

Generate reports for all benchmark cases in the session.

report_keys() → list[str][source]🔗

Get a list of report keys for all reports to be generated in this session.

This filters the report choices based on the command line arguments that were set and parsed when the session was created and returns a list of report keys for the reports that should be generated.

The report keys correspond to the command line flags/args defined in the Choices of the registered reporters.

Note

This method requires that the session’s args property has been set by calling parse_args() prior to invoking this method.

Otherwise it will return an empty list.

Returns:

A list of report keys for all reports to be generated in this session.

property reporter_manager: ReporterManager🔗

Returns the ReporterManager instance for managing reporters.

The reporter manager handles the registration and discovery of all reporters available to the session. This allows you to customize reporting behavior by adding your own reporters or removing default ones.

Returns:

The ReporterManager instance for managing reporters.

Return type:

ReporterManager

run() → None[source]🔗

Run all benchmark cases in the session.

This method iterates over all Case instances in the session’s cases and invokes their run() method to execute the benchmarks.

If the parse_args() method has not been called prior to invoking this method, it will be called authomatically with no arguments to parse from sys.argv.

If you wish to customize argument parsing or run the session entirely programmatically without command line args, you should call parse_args() before calling this method.

Raises:

  • SimpleBenchTimeoutError – If a benchmark case times out during execution.

  • SimpleBenchBenchmarkError – If an error occurs during the execution of a benchmark.

property show_progress: bool🔗

Whether to show progress bars during execution.

property tasks: RichProgressTasks🔗

The RichProgressTasks instance for managing progress tasks.

property timer: Callable[[], int]🔗

The timer function used for benchmarking.

property verbosity: Verbosity🔗

The Verbosity level for this session.

class simplebench.Style(value)[source]🔗

Bases: str, Enum

Enumeration of graph styles.

Note

The styles correspond to those available in Matplotlib 3.10.6

BMH = 'bmh'🔗

Bayesian Methods for Hackers style for graphs.

See Bayesian Methods for Hackers

CLASSIC = 'classic'🔗

Light background style for graphs.

Classic matplotlib plotting style

DARK_BACKGROUND = 'dark_background'🔗

Dark background style for graphs.

Set black background default line colors to white.

FIVETHIRTYEIGHT = 'fivethirtyeight'🔗

FiveThirtyEight style for graphs.

Replicated styles from FiveThirtyEight.com

See dataorigami.net

GGPLOT = 'ggplot'🔗

ggplot style for graphs.

Replicates the style of R’s ggplot library.

See everyhue.me

GRAYSCALE = 'grayscale'🔗

Grayscale style for graphs.

Set all colors to grayscale

Note

strings of float values are interpreted by matplotlib as gray values.

PETTROF10 = 'petroff10'🔗

Petroff10 style for graphs.

Color cycle survey palette from Petroff (2021):

See arxiv.org and github.com

SEABORN_V0_8 = 'seaborn-v0_8'🔗

Base Seaborn style for graphs.

SEABORN_V0_8_BRIGHT = 'seaborn-v0_8-bright'🔗

Seaborn bright style for graphs.

SEABORN_V0_8_COLORBLIND = 'seaborn-v0_8-colorblind'🔗

Seaborn colorblind style for graphs.

SEABORN_V0_8_DARK = 'seaborn-v0_8-dark'🔗

Seaborn dark style for graphs.

SEABORN_V0_8_DARKGRID = 'seaborn-v0_8-darkgrid'🔗

Seaborn darkgrid style for graphs.

SEABORN_V0_8_DARK_PALETTE = 'seaborn-v0_8-dark-palette'🔗

Seaborn dark palette style for graphs.

SEABORN_V0_8_DEEP = 'seaborn-v0_8-deep'🔗

Seaborn deep style for graphs.

SEABORN_V0_8_MUTED = 'seaborn-v0_8-muted'🔗

Seaborn muted style for graphs.

SEABORN_V0_8_NOTEBOOK = 'seaborn-v0_8-notebook'🔗

Seaborn notebook style for graphs.

SEABORN_V0_8_PAPER = 'seaborn-v0_8-paper'🔗

Seaborn paper style for graphs.

SEABORN_V0_8_PASTEL = 'seaborn-v0_8-pastel'🔗

Seaborn pastel style for graphs.

SEABORN_V0_8_POSTER = 'seaborn-v0_8-poster'🔗

Seaborn poster style for graphs.

SEABORN_V0_8_TALK = 'seaborn-v0_8-talk'🔗

Seaborn talk style for graphs.

SEABORN_V0_8_TICKS = 'seaborn-v0_8-ticks'🔗

Seaborn ticks style for graphs.

SEABORN_V0_8_WHITE = 'seaborn-v0_8-white'🔗

Seaborn white style for graphs.

SEABORN_V0_8_WHITEGRID = 'seaborn-v0_8-whitegrid'🔗

Seaborn whitegrid style for graphs.

SOLARIZE_LIGHT2 = 'Solarize_Light2'🔗

Solarized light style for graphs.

Solarized color palette taken from ethanschoonover.com

TABLEAU_COLORBLIND10 = 'tableau-colorblind10'🔗

Tableau colorblind10 style for graphs.

class simplebench.Theme(
rcparams: dict[str, Any] | None = None,
)[source]🔗

Bases: RcParams

An immutable MatPlotLib base theme class for the graphs.

This is a subclass of matplotlib.RcParams that represents a theme for Matplotlib graphs. It can be used to define custom styles for Matplotlib graphs generated by SimpleBench.

See Customizing Matplotlib with style sheets and rcParams for more information on customizing Matplotlib themes.

Initialize a Theme instance.

Parameters:

rcparams (dict[str, Any] | None) – The rcParams to use for the theme. If None, the default Matplotlib rcParams will be used.

replace(
rcparams: dict[str, Any],
) → Theme[source]🔗

Replace some parameters in the theme and return a new Theme instance without modifying the current instance.

new_theme = old_theme.replace({'axes.grid': False, 'figure.dpi': 200})
Parameters:

rcparams (dict[str, Any]) – The new rcParams to use for the theme.

Returns:

A new Theme instance with the updated rcParams.

Return type:

Theme

class simplebench.Verbosity(value)[source]🔗

Bases: int, Enum

Verbosity level enums for console output.

Defined levels are:

  • QUIET: Only requested output, errors, warnings and critical messages are shown.

  • NORMAL: Normal messages are shown, including status displays during runs.

  • VERBOSE: All messages are shown and status displays during runs.

  • DEBUG: All messages are shown, including debug messages and status displays during runs.

DEBUG = 5🔗

All messages are shown, including debug messages and status displays during runs.

This is incompatible with quiet.

NORMAL = 1🔗

Normal messages are shown, including status displays during runs.

This is the default verbosity level and is incompatible with quiet.

QUIET = 0🔗

Only requested output, errors, warnings and critical messages are shown. Status displays are not shown during runs.

This is incompatible with all other output levels.

VERBOSE = 2🔗

All messages are shown and status displays during runs.

This is incompatible with quiet.

simplebench.benchmark(
group: str | Callable[[...], Any] = 'default',
/,
*,
title: str | None = None,
benchmark_id: str | None = None,
description: str | None = None,
iterations: int = 20,
warmup_iterations: int = 10,
rounds: int | None = None,
timer: Callable[[], int] | None = None,
min_time: float = 5.0,
max_time: float = 20.0,
timeout: float | None = None,
variation_cols: dict[str, str] | None = None,
kwargs_variations: dict[str, list[Any]] | None = None,
options: list[ReporterOptions] | None = None,
n: int | float = 1,
use_field_for_n: str | None = None,
) → Callable[[Callable[[P], R]], Callable[[P], R]][source]🔗

A decorator to register a function as a benchmark case.

This module uses a global registry to store benchmark cases created via the @benchmark decorator. This enables a streamlined workflow where users simply decorate functions and call main().

Note

Importing a module that uses @benchmark will register its cases globally. For testing, use clear_registered_cases() to reset state between tests.

This simplifies creating a Case by wrapping the decorated function. The decorated function should contain the code to be benchmarked.

It is important to note that the decorated function will be called within the context of a SimpleRunner.run() call, which means it should not handle its own timing or iterations.

The args provided to the decorator are used to create a Case instance, which is then added to a global registry. The original function is returned unmodified, allowing it to be called directly if needed.

The arguments to the decorator are largely the same as those for Case, with the exception of action, which is replaced by the decorated function.

n is included to allow n-weighting the complexity of the benchmark case when using runners that support it.

A minimal example:

from simplebench import benchmark, main


@benchmark
def addition_benchmark():
    '''A simple addition benchmark.'''
    sum(range(1000))


if __name__ == '__main__':
    extra_args = None if len(sys.argv) > 1 : ['--progress', '--rich-table.console']
main(extra_args=extra_args)

You should read the documentation for Case for full details on the parameters and their meanings.

Parameters:

  • group –

    The benchmark reporting group to which the benchmark case belongs.

    Used to categorize and filter benchmark cases for selection and reporting. Cannot be blank (a string composed only of whitespace). The group parameter is positional-only. All other parameters must be passed as keyword arguments. When the decorator is used without parameters, the group defaults to ‘default’. This has special handling to allow the decorator to be used easily without any parameters.

  • title – The title of the benchmark case. Uses the function name if None. Cannot be blank.

  • benchmark_id – An optional identifier for the benchmark case. If None, a benchmark ID is generated based on the function name and module.

  • description – A description for the case. Uses the function’s docstring if None or ‘(no description)’ if there is no docstring. Cannot be blank.

  • iterations – The minimum number of iterations to run for the benchmark.

  • warmup_iterations – The number of warmup iterations to run before the benchmark.

  • rounds –

    The number of rounds to run for the benchmark.

    Rounds are multiple runs of calls to the action within an iteration to mitigate timer quantization, loop overhead, and other measurement effects for very fast actions. Setup and teardown functions are called only once per iteration (all rounds in the same iteration share the same setup/teardown context).

    If None, rounds will be auto-calibrated based on the precision and overhead of the timer function and the expected execution time of the action. If the action is very fast (e.g., under 10 microseconds), rounds will be set to a higher value to improve measurement accuracy with the goal of reducing timer quantization errors. If the action is slower, rounds will be set to a lower value.

    If specified, it must be a positive integer.

  • timer – A callable that returns the current time. If None, the default timer is used. The timer function should return a float or int representing the current time.

  • min_time – The minimum time in seconds to run the benchmark. Must be a positive number. Its reference depends on the timer used, but by default it is wall-clock time.

  • max_time – The maximum time in seconds to run the benchmark. Must be a positive number greater than min_time. Its reference depends on the timer used, but by default it is wall-clock time.

  • timeout – The maximum time in seconds to allow the benchmark to run before timing out. If None, a default timeout of max_time + 10.0 is used. Must be a positive number. Time for timeout is always referenced to wall-clock time.

  • variation_cols – kwargs to be used for cols to denote kwarg variations. Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels. If None, an empty dict is used.

  • kwargs_variations – A mapping of keyword argument key names to a list of possible values for that argument. Default is {}. When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. The action function will be called with a bench parameter that is an instance of the runner and the keyword arguments for the current variation. If None, an empty dict is used.

  • options – A list of additional options for the benchmark case. Each option is an instance of ReporterOptions or a subclass of ReporterOptions. Reporter options can be used to customize the output of the benchmark reports for specific reporters. Reporters are responsible for extracting applicable ReporterOptionss from the list of options themselves.

  • n – The ‘n’ weighting of the benchmark case. Must be a positive integer or float.

  • use_field_for_n – If provided, use the value of this field from kwargs_variations to set ‘n’ dynamically for each variation.

  • timer – The timer function to use for the benchmark. If None, the default timer is used. The timer function should be a callable that returns a float or int representing the current time.

Returns:

A decorator that registers the function for benchmarking and returns it unmodified.

Return type:

Callable[[Callable[P, R]], Callable[P, R]]

Raises:
simplebench.main(
benchmark_cases: Sequence[Case] | None = None,
*,
argv: list[str] | None = None,
extra_args: list[str] | None = None,
no_exit: bool = False,
) → ExitCode[source]🔗

Main entry point for running benchmarks via a command-line interface.

This function is responsible for setting up the command-line interface, parsing arguments, and executing the benchmark cases.

@benchmark decorated benchmark cases are automatically included and added to the list of benchmark cases passed to this function.

Call this function from a script or the command line to execute benchmarks.

If passed a list of Case instances via the benchmark_cases argument, those cases will be benchmarked in addition to any cases registered via the @benchmark decorator.

If argv is provided, it will be used as the list of command-line arguments instead of sys.argv.

If extra_args is provided, those arguments will be appended to the command-line arguments before parsing.

If the no_exit argument is set to True, the function will return the exit code instead of calling sys.exit(). This is useful for testing or embedding the CLI functionality in other applications.

Parameters:

  • benchmark_cases – A Sequence of SimpleBench.Case instances to be benchmarked.

  • argv – A list of command-line arguments to parse. If None, defaults to sys.argv.

  • extra_args – Additional command-line arguments to include.

  • no_exit – If True, the function will not call sys.exit() and will return the exit code instead.

Returns:

  • ExitCode.SUCCESS (0) on success

  • ExitCode.RUNTIME_ERROR (1) runtime errors during execution.

  • ExitCode.CLI_ARGUMENTS_ERROR (2) for errors during CLI argument processing

  • ExitCode.KEYBOARD_INTERRUPT (3) if interrupted by keyboard interrupt

  • ExitCode.BENCHMARK_TIMED_OUT (4) if a timeout occurs during execution of a benchmark.

  • ExitCode.BENCHMARK_ERROR (5) if an error occurs during execution of a benchmark.

Raises:

SimpleBenchTypeError – If the extra_args argument is not None or a list of strings or if argv is not None or a list of strings.

simplebench.register_reporter(
cls: type[Reporter],
) → type[Reporter][source]🔗

Class decorator to register a Reporter subclass.

This decorator can be applied to any subclass of Reporter to register it with the system.

Subpackages🔗

Submodules🔗