CSV Report🔗
CSV (Comma-Separated Values) reports are designed for machine readability and are ideal for importing benchmark results into spreadsheet applications like Microsoft Excel or Google Sheets, or for analysis with data processing tools like Pandas.
To generate a CSV report, you can use an option like –csv.ops when running
your benchmarks. By default, this will save a CSV file to the output directory
(which defaults to .benchmarks).
Here is an example of how to run a benchmark script and generate a CSV report:
python my_benchmark_script.py --csv.ops
This command executes the benchmarks in my_benchmark_script.py and saves a CSV file containing the operations-per-second results. A basic CSV output file will look something like this:
# title: addition_benchmark
# description: A simple addition benchmark of Python's built-in sum function.
# unit: Ops/s
N,Iterations,Rounds,Elapsed Seconds,mean (kOps/s),median (kOps/s),min (kOps/s),max (kOps/s),5th (kOps/s),95th (kOps/s),std dev (kOps/s),rsd (%)
1,42801,1,0.291715587,147.0,148.0,22.0,153.0,142.0,151.0,6.77,4.6
Which corresponds to the following table:
N |
Iterations |
Rounds |
Elapsed Seconds |
mean (kOps/s) |
median (kOps/s) |
min (kOps/s) |
max (kOps/s) |
5th (kOps/s) |
95th (kOps/s) |
std dev (kOps/s) |
rsd (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|
1 |
42801 |
1 |
0.291715587 |
147.0 |
148.0 |
22.0 |
153.0 |
142.0 |
151.0 |
6.77 |
4.6 |
Note
Interpreting Outliers in Benchmark Results
In the sample output, you may notice that the min (Ops/s) value is an
extreme outlier. This is a realistic reflection of real-world benchmarking,
where system events like garbage collection can cause individual iterations
to be significantly slower. This is why SimpleBench provides a full suite of
statistics like the median (which is resistant to outliers) and RSD%
(which quantifies inconsistency) to help you get a complete and honest
picture of your code’s performance.
Note
To avoid “false precision”, statistical results are output with three significant digits. Due to the inherent variability of performance measurement, any further digits are typically meaningless statistical noise.
This is not an issue with SimpleBench itself, but rather a fundamental aspect of benchmarking and performance measurement in the real world.
Report Variations and Destinations🔗
SimpleBench provides several variations of the CSV report:
--csv: Generates a CSV file with all result types (ops, timing, and memory).--csv.ops: Generates a CSV file only for operations-per-second results.--csv.timing: Generates a CSV file only for timing results.--csv.memory: Generates a CSV file only for memory usage results.
By default, CSV reports are saved to the filesystem. You can send a report to
other destinations, such as the console, by appending the destination name. For
example, to print the CSV content directly to the terminal:
python my_benchmark_script.py --csv.ops console
The generated files are named based on the benchmarked function name and report type. To prevent collisions between identical benchmark names, a numeric prefix is added to ensure uniqueness.
Examples
001_addition_benchmark-memory_usage.csv
001_addition_benchmark-peak_memory_usage.csv
001_addition_benchmark-operations_per_second.csv
001_addition_benchmark-timing.csv
Advanced Features🔗
Parameterized Benchmarks🔗
When running parameterized benchmarks, the CSV report includes additional columns for each parameter variation. This makes it easy to sort, filter, and analyze how different input parameters affect performance.
For more information on creating parameterized benchmarks, see the ../advanced_usage documentation.
Custom Complexity Weightings🔗
The N column in the CSV report represents the complexity weighting of the
benchmark. This is particularly useful for analyzing the scalability of your
code with different input sizes.
For more details on how to use this feature, see the ../advanced_usage documentation.