Basic Benchmark

This tutorial demonstrates how to create a simple benchmark using SimpleBench, run it, and generate a report.

The minimal code required to create and run a benchmark using SimpleBench is creating a script that defines a function to be benchmarked with @simplebench.benchmark and that calls simplebench.main() and then running it.

When you run the code below, SimpleBench automatically runs the benchmark function addition_benchmark, collecting performance and memory statistics, and generating a report. By specifying command_line_options you can generate output in multiple formats, including rich text tables (like the one below), graphs, CSV files, and JSON reports and select which statistics to report.

It runs using defaults setting for number of iterations, warmup runs, etc. You can customize these settings via parameters set on the @simplebench.benchmark decorator.

A basic benchmark example

"""A basic benchmark."""
import simplebench


@simplebench.benchmark
def addition_benchmark():
    """A simple addition benchmark of Python's built-in sum function."""
    sum(range(1000))


if __name__ == "__main__":
    simplebench.main()

Save this code to a file, for example basic_benchmark.py, and then run it from your terminal:

Generate a rich table report of operations-per-second by running the benchmark

 python basic_benchmark.py --rich-table.ops --progress

This will produce output similar to the following:

Expected Output from Running the Basic Benchmark

                                                                  addition_benchmark                                                                   
                                                                 operations per second                                                                 
                                                                                                                                                       
                                            A simple addition benchmark of Python's built-in sum function.                                             
┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃        ┃            ┃        ┃ Elapsed ┃             ┃               ┃            ┃            ┃            ┃             ┃                ┃        ┃
┃   N    ┃ Iterations ┃ Rounds ┃ Seconds ┃ mean kOps/s ┃ median kOps/s ┃ min kOps/s ┃ max kOps/s ┃ 5th kOps/s ┃ 95th kOps/s ┃ std dev kOps/s ┃  rsd%  ┃
┡━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│    1.0 │    41789   │      1 │  0.30   │    141.00   │     140.00    │     22.80  │    153.00  │    138.00  │    151.00   │        9.28    │  6.59% │
└────────┴────────────┴────────┴─────────┴─────────────┴───────────────┴────────────┴────────────┴────────────┴─────────────┴────────────────┴────────┘

A detailed explanation of the output format and displayed statistics shown in this example report can be found in the Rich Table Report section of the documentation.

Note

The command shown above includes the --progress option to display a progress bar while the benchmark is running. This option is not required to run the benchmark, and can be omitted if you do not wish to see the progress bar. It is not included in the expected output file since it produces dynamic output that changes as the benchmark runs and is removed from the terminal once the benchmark completes.