Benchmarks¶
Package defaults against upstream defaults, same inputs, same metrics, numbers generated by the scripts in benchmarks/ and committed with the command that produced them. Nothing in this page is typed by hand. Run python benchmarks/run_all.py to regenerate benchmarks/results.md; the raw per-trial numbers are in benchmarks/results/*.json.
What is compared and why it is fair:
- Registration: the moving cloud is a rotated (35°), partial (70 % overlap) and noisy (0.5 % of the diagonal) copy of the target, a different noise draw per trial. Open3D's ICP is called the way its tutorial calls it (identity start, point-to-point, threshold 0.02); Open3D's global registration uses the tutorial's
voxel_size = 0.05; pardesline usestarget.align(moving)with the catalog defaults. The rotation error is measured against the known transform. - Reconstruction: a noisy partial scan with a hole; Open3D's
create_from_point_cloud_poisson(depth=8)versuspoints.to_mesh()(depth 9, density-trimmed); Chamfer distance to the original mesh and the fraction of reconstructed surface farther than 2 mm from it (the balloon over the hole).
The upstream calls are the documented, tutorial-level calls, which is what a practitioner copies first. An expert can of course tune Open3D to the same result: the point of the package is that the tuning is written down and applied by default.
pardesline 0.2.0, Python 3.11.14, Windows AMD64, CPU only.
Rigid point-cloud registration on the Stanford bunny (35 deg, 70 % overlap, 0.5 % noise)¶
python benchmarks/registration_bunny.py --trials 5
| Contender | Rotation error, median (deg) | Rotation error, worst (deg) | NN RMSE, median (m) | Time, median (s) |
|---|---|---|---|---|
open3d_icp_tutorial |
32.50 | 32.98 | 0.0071 | 0.2 |
open3d_ransac_tutorial |
134.50 | 173.21 | 0.0079 | 0.2 |
pardesline |
0.11 | 0.24 | 0.0014 | 5.2 |
Poisson surface reconstruction on a partial, noisy bunny scan¶
python benchmarks/reconstruction_bunny.py --trials 3
| Contender | Chamfer, median (mm) | Surface farther than 2 mm from truth, median | Time, median (s) |
|---|---|---|---|
open3d_default |
0.80 | 3.4 % | 1.7 |
pardesline |
1.14 | 0.8 % | 6.9 |
Reading the numbers¶
- Registration: the tutorial ICP call stays near its 32° start (a local optimiser from a wrong pose); the tutorial RANSAC call fails worse because its fixed
voxel_size = 0.05is a third of the bunny, so there are almost no features to match. The package's defaults derive the voxel from the bounding box and land at 0.11° median, 0.24° worst over five noise draws, at the cost of 5 s instead of 0.2 s. - Reconstruction: this one is a trade-off, not a win on every metric. The density-trimmed surface has four times less area far from the truth (0.8 % vs 3.4 %: the balloon over the hole is gone) but a higher Chamfer distance (1.14 mm vs 0.80 mm), because the symmetric Chamfer also measures truth-to-reconstruction, and a trimmed hole is "missing surface" where the untrimmed one offers a wrong but nearby guess. Which is better depends on what you do next: for metrology and printing, no invented surface; for rendering, the closed guess may look better. Both settings are one argument away (
density_quantile).
Limits¶
- The bunny is a public stand-in for industrial scans; the characteristics (partial overlap, noise, unknown pose) are representative, the material is not.
- RANSAC is randomised: expect a few hundredths of a degree of spread between runs on the registration median, more on the worst case.
- No benchmark yet for volume registration or segmentation defaults: the RIRE and MSD examples show single runs with their numbers, and the benchmarks tracking issue lists what comes next.