Parameterized Benchmark🔗
This tutorial demonstrates how to create a parameterized benchmark using SimpleBench, run it, and generate a report.
Parameterized benchmarks allow you to run the same benchmark function multiple times with different input parameters.
The key components of a parameterized benchmark are:
Benchmark Function: A function decorated with @simplebench.benchmark that contains the code to be benchmarked. The function’s signature should explicitly define the parameters that will be varied. SimpleBench will automatically pass the values from the kwargs_variations to these parameters.
Parameters: Defined as arguments for the @simplebench.benchmark decorator, which specifies different values to be used as inputs to the benchmark function.
kwargs_variationsA dictionary where the keys are parameter names that match the arguments of the benchmarked function and values are lists of possible values for those parameters.
The benchmark will be executed for every combination of the provided parameter values.
Example
kwargs_variations={ 'arg1': [1, 2], 'arg2': ['a', 'b'] }
will result in the following combinations being tested:
{'arg1': 1, 'arg2': 'a'} {'arg1': 1, 'arg2': 'b'} {'arg1': 2, 'arg2': 'a'} {'arg1': 2, 'arg2': 'b'}
variation_colsA dictionary where keys are parameter names and values are display names for reporting. Fields not included here will usually not display in a report. Fields specified here must be a subset of the keys in
kwargs_variations.Example
variation_cols={ 'arg1': 'Argument 1', 'arg2': 'Argument 2' }
This will result in the report displaying columns/fields labeled ‘Argument 1’ and ‘Argument 2’ corresponding to the values of ‘arg1’ and ‘arg2’ used in each benchmark run.
use_field_for_nA string specifying a parameter to use as the ‘N’ field in reports, which is often used to indicate input size for complexity analysis. This parameter should be one of the keys in
kwargs_variations. It is optional; if not specified, the ‘N’ field will default to the value ‘1.0’.This is useful when you want to analyze how the performance of the benchmarked function scales with different input sizes or configurations. It is not required that the field be defined in
variation_colsto be used as the ‘N’ field.
Minimal Example🔗
The minimal code required to create and run a parameterized benchmark using SimpleBench is creating a script that defines a function to be benchmarked with @simplebench.benchmark, specifying parameters using the @simplebench.benchmark decorator, and that calls simplebench.main()
1"""A basic parameterized benchmark."""
2import simplebench
3
4
5@simplebench.benchmark(kwargs_variations={'size': [1, 5, 10, 50, 100, 500, 1000]},
6 variation_cols={'size': 'Input Size'},
7 use_field_for_n='size')
8def addition_benchmark(size: int) -> None:
9 """A simple addition benchmark of Python's built-in sum function."""
10 sum(range(size))
11
12
13if __name__ == "__main__":
14 simplebench.main()
Save this code to a file, for example minimal_parameterized_benchmark.py, and then run it from
your terminal:
python minimal_parameterized_benchmark.py --rich-table.ops --progress
This will produce output similar to the following:
addition_benchmark
operations per second
A simple addition benchmark of Python's built-in sum function.
┏━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ Elapsed ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃
┃ Input Size ┃ N ┃ Iterations ┃ Rounds ┃ Seconds ┃ mean kOps/s ┃ median kOps/s ┃ min kOps/s ┃ max kOps/s ┃ 5th kOps/s ┃ 95th kOps/s ┃ std dev kOps/s ┃ rsd% ┃
┡━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 1 │ 1.0 │ 1010264 │ 1 │ 0.22 │ 4710.00 │ 4810.00 │ 46.50 │ 12000.00 │ 4000.00 │ 5990.00 │ 483.00 │ 10.30% │
│ 5 │ 5.0 │ 792381 │ 1 │ 0.18 │ 4500.00 │ 4780.00 │ 40.10 │ 12000.00 │ 4000.00 │ 4810.00 │ 426.00 │ 9.48% │
│ 10 │ 10.0 │ 626015 │ 1 │ 0.15 │ 4170.00 │ 4000.00 │ 26.40 │ 8000.00 │ 4000.00 │ 4810.00 │ 388.00 │ 9.30% │
│ 50 │ 50.0 │ 481444 │ 1 │ 0.19 │ 2600.00 │ 2670.00 │ 42.20 │ 4000.00 │ 2400.00 │ 2670.00 │ 195.00 │ 7.51% │
│ 100 │ 100.0 │ 358702 │ 1 │ 0.20 │ 1840.00 │ 1850.00 │ 48.10 │ 2400.00 │ 1720.00 │ 2000.00 │ 116.00 │ 6.28% │
│ 500 │ 500.0 │ 38035 │ 1 │ 0.11 │ 340.00 │ 343.00 │ 47.70 │ 358.00 │ 333.00 │ 348.00 │ 20.90 │ 6.15% │
│ 1000 │ 1000.0 │ 13805 │ 1 │ 0.09 │ 147.00 │ 149.00 │ 11.50 │ 152.00 │ 144.00 │ 150.00 │ 8.38 │ 5.69% │
└────────────┴────────┴────────────┴────────┴─────────┴─────────────┴───────────────┴────────────┴────────────┴────────────┴─────────────┴────────────────┴────────┘
A detailed explanation of the output format and displayed statistics shown in this example report can be found in the Rich Table Report section of the documentation.
Note
The command shown above includes the --progress option to
display a progress bar while the benchmark is running. This option is not
required to run the benchmark, and can be omitted if you do not wish to see
the progress bar. It is not included in the expected output file since it
produces dynamic output that changes as the benchmark runs and is removed
from the terminal once the benchmark completes.
Multi-dimensional Parameters🔗
You can define multiple parameters for a benchmark function by defining multiple parameters for the kwargs_variations.
For example, to benchmark a function with two parameters, size and mode, you can define the benchmark as follows:
1"""A multidimensional parameterized benchmark."""
2import simplebench
3
4
5@simplebench.benchmark(
6 kwargs_variations={
7 'size': [10, 100, 1000],
8 'mode': ['fast', 'slow']
9 },
10 variation_cols={
11 'size': 'Input Size',
12 'mode': 'Mode'
13 },
14 use_field_for_n='size')
15def my_benchmark(size: int, mode: str) -> None:
16 """A benchmark for summing a range of numbers that varies by size and mode."""
17 match mode:
18 case 'fast':
19 sum(range(size))
20 case 'slow':
21 total = 0
22 for i in range(size):
23 total += i
24
25
26if __name__ == "__main__":
27 simplebench.main()
Save this code to a file, for example
multidimensional_parameterized_benchmark.py,
and then run it from your terminal:
python multidimensional_parameterized_benchmark.py --rich-table.ops --progress
This will run the benchmark for all combinations of size and mode, resulting in six different benchmark runs (3 sizes x 2 modes). The report will include columns for both ‘Input Size’ and ‘Mode’, allowing you to analyze the performance across these different dimensions.
And you will see output similar to the following:
my_benchmark
operations per second
A benchmark for summing a range of numbers that varies by size and mode.
┏━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ ┃ Elapsed ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃ ┃
┃ Input Size ┃ Mode ┃ N ┃ Iterations ┃ Rounds ┃ Seconds ┃ mean kOps/s ┃ median kOps/s ┃ min kOps/s ┃ max kOps/s ┃ 5th kOps/s ┃ 95th kOps/s ┃ std dev kOps/s ┃ rsd% ┃
┡━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 10 │ fast │ 10.0 │ 927749 │ 1 │ 0.25 │ 3800.00 │ 4000.00 │ 28.70 │ 8000.00 │ 3420.00 │ 4000.00 │ 392.00 │ 10.30% │
│ 10 │ slow │ 10.0 │ 728337 │ 1 │ 0.32 │ 2330.00 │ 2400.00 │ 35.00 │ 3440.00 │ 2180.00 │ 2670.00 │ 161.00 │ 6.92% │
│ 100 │ fast │ 100.0 │ 537489 │ 1 │ 0.31 │ 1740.00 │ 1720.00 │ 24.70 │ 2400.00 │ 1600.00 │ 1850.00 │ 116.00 │ 6.65% │
│ 100 │ slow │ 100.0 │ 139185 │ 1 │ 0.33 │ 425.00 │ 428.00 │ 26.20 │ 471.00 │ 414.00 │ 444.00 │ 25.50 │ 5.99% │
│ 1000 │ fast │ 1000.0 │ 22429 │ 1 │ 0.16 │ 140.00 │ 141.00 │ 27.60 │ 149.00 │ 138.00 │ 146.00 │ 8.45 │ 6.03% │
│ 1000 │ slow │ 1000.0 │ 7963 │ 1 │ 0.20 │ 40.50 │ 40.80 │ 9.17 │ 43.50 │ 39.10 │ 42.10 │ 2.20 │ 5.44% │
└────────────┴──────┴────────┴────────────┴────────┴─────────┴─────────────┴───────────────┴────────────┴────────────┴────────────┴─────────────┴────────────────┴────────┘
You can customize the parameter values and names as needed for your specific benchmarking scenarios.