criterion performance measurements
overview
want to understand this report?
matrix creation
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.866481371195224e-8 | 2.1831380632614558e-8 | 3.144140619688592e-8 |
| Standard deviation | 4.600702509657373e-9 | 1.736604219344019e-8 | 3.534634966219958e-8 |
Outlying measurements have severe (0.996491354289827%) effect on estimated standard deviation.
random matrix generation
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.734809426045651e-5 | 1.812080259863427e-5 | 1.918838892478034e-5 |
| Standard deviation | 2.1005234052357934e-6 | 2.920671997830287e-6 | 4.7850917506945866e-6 |
Outlying measurements have severe (0.9372871956355214%) effect on estimated standard deviation.
matrix multiplication
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8203006735459527e-8 | 2.298968868790755e-8 | 3.916349012308795e-8 |
| Standard deviation | 3.666725536607145e-9 | 2.7113828259338455e-8 | 5.722524861138808e-8 |
Outlying measurements have severe (0.9965371355528275%) effect on estimated standard deviation.
matrix slicing
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.1907637944481158e-6 | 3.8871514548393666e-6 | 4.638962271508763e-6 |
| Standard deviation | 2.0254907786096945e-6 | 2.312340535801347e-6 | 2.6576695832513467e-6 |
Outlying measurements have severe (0.9944484762627788%) effect on estimated standard deviation.
scalar multiplication
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0404491224998863e-6 | 2.385795326591255e-6 | 3.024534686624357e-6 |
| Standard deviation | 1.1626418168986267e-6 | 1.5724652095504906e-6 | 2.237829853537773e-6 |
Outlying measurements have severe (0.9947163134045706%) effect on estimated standard deviation.
matrix-vector addition
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3308123793751467e-8 | 2.7743751772912486e-8 | 3.781876280571033e-8 |
| Standard deviation | 1.330829564564107e-8 | 2.4290421933155525e-8 | 4.0813475260769964e-8 |
Outlying measurements have severe (0.9964574921353061%) effect on estimated standard deviation.
transpose
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3179678119775805e-8 | 3.246544164202608e-8 | 6.037914996175194e-8 |
| Standard deviation | 1.7696994262846732e-8 | 4.779452864504426e-8 | 9.153720318328687e-8 |
Outlying measurements have severe (0.9965150182960154%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.