simplebench.decorators module

Decorators for simplifying benchmark case creation.

simplebench.decorators.benchmark(

group: str | Callable[[...], Any] = 'default',

/,

*,

title: str | None = None,

benchmark_id: str | None = None,

description: str | None = None,

iterations: int = 20,

warmup_iterations: int = 10,

rounds: int | None = None,

timer: Callable[[], int] | None = None,

min_time: float = 5.0,

max_time: float = 20.0,

timeout: float | None = None,

variation_cols: dict[str, str] | None = None,

kwargs_variations: dict[str, list[Any]] | None = None,

options: list[ReporterOptions] | None = None,

n: int | float = 1,

use_field_for_n: str | None = None,

) → Callable[[Callable[[P], R]], Callable[[P], R]]

A decorator to register a function as a benchmark case.

This module uses a global registry to store benchmark cases created via the @benchmark decorator. This enables a streamlined workflow where users simply decorate functions and call main().

Note

Importing a module that uses @benchmark will register its cases globally. For testing, use clear_registered_cases() to reset state between tests.

This simplifies creating a Case by wrapping the decorated function. The decorated function should contain the code to be benchmarked.

It is important to note that the decorated function will be called within the context of a SimpleRunner.run() call, which means it should not handle its own timing or iterations.

The args provided to the decorator are used to create a Case instance, which is then added to a global registry. The original function is returned unmodified, allowing it to be called directly if needed.

The arguments to the decorator are largely the same as those for Case, with the exception of action, which is replaced by the decorated function.

n is included to allow n-weighting the complexity of the benchmark case when using runners that support it.

A minimal example:

from simplebench import benchmark, main


@benchmark
def addition_benchmark():
    '''A simple addition benchmark.'''
    sum(range(1000))


if __name__ == '__main__':
    extra_args = None if len(sys.argv) > 1 : ['--progress', '--rich-table.console']
main(extra_args=extra_args)

You should read the documentation for Case for full details on the parameters and their meanings.

Parameters:

• group –

  The benchmark reporting group to which the benchmark case belongs.

  Used to categorize and filter benchmark cases for selection and reporting. Cannot be blank (a string composed only of whitespace). The group parameter is positional-only. All other parameters must be passed as keyword arguments. When the decorator is used without parameters, the group defaults to ‘default’. This has special handling to allow the decorator to be used easily without any parameters.

• title – The title of the benchmark case. Uses the function name if None. Cannot be blank.

• benchmark_id – An optional identifier for the benchmark case. If None, a benchmark ID is generated based on the function name and module.

• description – A description for the case. Uses the function’s docstring if None or ‘(no description)’ if there is no docstring. Cannot be blank.

• iterations – The minimum number of iterations to run for the benchmark.

• warmup_iterations – The number of warmup iterations to run before the benchmark.

• rounds –

  The number of rounds to run for the benchmark.

  Rounds are multiple runs of calls to the action within an iteration to mitigate timer quantization, loop overhead, and other measurement effects for very fast actions. Setup and teardown functions are called only once per iteration (all rounds in the same iteration share the same setup/teardown context).

  If None, rounds will be auto-calibrated based on the precision and overhead of the timer function and the expected execution time of the action. If the action is very fast (e.g., under 10 microseconds), rounds will be set to a higher value to improve measurement accuracy with the goal of reducing timer quantization errors. If the action is slower, rounds will be set to a lower value.

  If specified, it must be a positive integer.

• timer – A callable that returns the current time. If None, the default timer is used. The timer function should return a float or int representing the current time.

• min_time – The minimum time in seconds to run the benchmark. Must be a positive number. Its reference depends on the timer used, but by default it is wall-clock time.

• max_time – The maximum time in seconds to run the benchmark. Must be a positive number greater than min_time. Its reference depends on the timer used, but by default it is wall-clock time.

• timeout – The maximum time in seconds to allow the benchmark to run before timing out. If None, a default timeout of max_time + 10.0 is used. Must be a positive number. Time for timeout is always referenced to wall-clock time.

• variation_cols – kwargs to be used for cols to denote kwarg variations. Each key is a keyword argument name, and the value is the column label to use for that argument. Only keywords that are also in kwargs_variations can be used here. These fields will be added to the output of reporters that support them as columns of data with the specified labels. If None, an empty dict is used.

• kwargs_variations – A mapping of keyword argument key names to a list of possible values for that argument. Default is {}. When tests are run, the benchmark will be executed for each combination of the specified keyword argument variations. The action function will be called with a bench parameter that is an instance of the runner and the keyword arguments for the current variation. If None, an empty dict is used.

• options – A list of additional options for the benchmark case. Each option is an instance of ReporterOptions or a subclass of ReporterOptions. Reporter options can be used to customize the output of the benchmark reports for specific reporters. Reporters are responsible for extracting applicable ReporterOptionss from the list of options themselves.

• n – The ‘n’ weighting of the benchmark case. Must be a positive integer or float.

• use_field_for_n – If provided, use the value of this field from kwargs_variations to set ‘n’ dynamically for each variation.

• timer – The timer function to use for the benchmark. If None, the default timer is used. The timer function should be a callable that returns a float or int representing the current time.

Returns:

A decorator that registers the function for benchmarking and returns it unmodified.

Return type:

Callable[[Callable[P, R]], Callable[P, R]]

Raises:

• SimpleBenchTypeError – If any argument is of an incorrect type.

• SimpleBenchValueError – If any argument has an invalid value.

simplebench.decorators.clear_registered_cases() → None

Clear all benchmark cases registered via the @benchmark decorator.

This can be useful in testing scenarios to reset the state.

simplebench.decorators.get_registered_cases() → list[Case]

Retrieve all benchmark cases registered via the @benchmark decorator.

Returns:

A list of Case objects.

Return type:

list[Case]

simplebench.decorators.validate_timer(

timer: Callable[[], int] | None,

param_name: str,

type_error_tag: _DecoratorsErrorTag,

return_type_error_tag: _DecoratorsErrorTag,

) → Callable[[], int] | None

Validate the timer parameter for the benchmark decorator.

Parameters:

• timer – The timer function to validate.

• param_name – The name of the parameter (for error messages).

• type_error_tag – The error tag to use for type errors.

• return_type_error_tag – The error tag to use for return type errors.

Returns:

The validated timer function or None.

Raises:

• SimpleBenchTypeError – If the timer is not callable.

• SimpleBenchTypeError – If the timer does not return a float or int.