connectomics.metrics

Evaluation metrics for PyTorch Connectomics.

This package provides comprehensive evaluation metrics: - metrics_seg.py: Segmentation metrics (Adapted Rand, VOI, instance matching) - metrics_skel.py: Skeleton-based metrics for curvilinear structures - nerl.py: NERL scoring helpers backed by em_erl - oracle.py: oracle-merge transform (attainable false-merge-free ceiling) - tube.py: GT-free completeness and geometric analysis for tube-like structures

Note: PyTorch Lightning handles training monitoring and logging.

Import patterns:

from connectomics.metrics import AdaptedRandError, VariationOfInformation from connectomics.metrics import evaluate_image_pair from connectomics.evaluation import evaluate_directory from connectomics.metrics.segmentation_numpy import adapted_rand, instance_matching

class connectomics.metrics.AdaptedRandError(return_all_stats=False, dist_sync_on_step=False)

Torchmetrics-style wrapper around the numpy-based adapted Rand implementation.

This wrapper lets us accumulate scores during Lightning test_step without manual numpy<->torch conversions in the training loop.

Parameters

• return_all_stats (bool) – If True, also compute and return precision and recall

• dist_sync_on_step (bool) – Whether to sync across distributed processes on each step

Initialize internal Module state, shared by both nn.Module and ScriptModule.

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

Return type

Tensor

update(preds, target)

Override this method to update the state variables of your metric class.

Parameters

• preds (Tensor) –

• target (Tensor) –

Return type

None

class connectomics.metrics.InstanceAccuracy(thresh=0.5, criterion='iou', dist_sync_on_step=False)

Torchmetrics-style wrapper around instance_matching for instance-level accuracy.

Instance accuracy measures the fraction of correctly detected instances:

accuracy = TP / (TP + FP + FN)

Where: - TP (True Positives): Number of GT instances correctly matched to predictions - FP (False Positives): Number of predicted instances not matched to GT - FN (False Negatives): Number of GT instances not matched to predictions

Matching is based on IoU threshold (default 0.5).

Higher values are better (1.0 = perfect detection).

This wrapper lets us accumulate scores during Lightning test_step without manual numpy<->torch conversions in the training loop.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters

• thresh (float) –

• criterion (str) –

• dist_sync_on_step (bool) –

compute()

Return instance-level accuracy: TP / (TP + FP + FN).

Return type

Tensor

update(preds, target)

Override this method to update the state variables of your metric class.

Parameters

• preds (Tensor) –

• target (Tensor) –

Return type

None

class connectomics.metrics.InstanceAccuracySimple(thresh=0.5, criterion='iou', dist_sync_on_step=False)

Torchmetrics-style wrapper for relaxed instance-level accuracy (NO Hungarian matching).

WARNING: This is a RELAXED metric for debugging/analysis only, NOT for benchmark ranking. Unlike InstanceAccuracy, this does NOT use optimal bipartite matching.

Simple counting approach:

• Count all (GT, Pred) pairs with IoU >= threshold as TP

• fp = n_pred - tp

• fn = n_true - tp

• accuracy = tp / (tp + fp + fn)

This metric is useful for: - Quick debugging and sanity checks - Understanding raw overlap statistics - Comparing with strict Hungarian-based metrics

Higher values are better (1.0 = perfect detection).

This wrapper lets us accumulate scores during Lightning test_step without manual numpy<->torch conversions in the training loop.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters

• thresh (float) –

• criterion (str) –

• dist_sync_on_step (bool) –

compute()

Return relaxed instance-level accuracy: TP / (TP + FP + FN).

Return type

Tensor

compute_f1()

Return instance-level F1: 2*TP / (2*TP + FP + FN).

Return type

Tensor

compute_precision()

Return instance-level precision: TP / (TP + FP).

Return type

Tensor

compute_recall()

Return instance-level recall: TP / (TP + FN).

Return type

Tensor

update(preds, target)

Override this method to update the state variables of your metric class.

Parameters

• preds (Tensor) –

• target (Tensor) –

Return type

None

class connectomics.metrics.NerlGraphOptions(skeleton_id_attribute='id', skeleton_position_attribute='index_position', skeleton_edge_length_attribute='edge_length', skeleton_position_order='xyz', prediction_position_order=None)

Options for converting NetworkX skeleton pickles into ERL graphs.

Parameters

• skeleton_id_attribute (str) –

• skeleton_position_attribute (str) –

• skeleton_edge_length_attribute (str) –

• skeleton_position_order (str) –

• prediction_position_order (str | None) –

class connectomics.metrics.NerlScoreResult(nerl, pred_erl, gt_erl, num_skeletons, per_gt_erl, graph, voi_split=nan, voi_merge=nan, voi_total=nan)

Detailed NERL score output for stage-specific adapters.

Parameters

• nerl (float) –

• pred_erl (float) –

• gt_erl (float) –

• num_skeletons (int) –

• per_gt_erl (ndarray) –

• graph (Any) –

• voi_split (float) –

• voi_merge (float) –

• voi_total (float) –

class connectomics.metrics.TubeAnalysis(volume_shape, config, total_label_count, total_foreground_voxels, tubes)

Detailed and aggregate output from analyze_tubes().

Parameters

• volume_shape (tuple[int, int, int]) –

• config (TubeAnalysisConfig) –

• total_label_count (int) –

• total_foreground_voxels (int) –

• tubes (tuple[connectomics.metrics.tube.TubeRecord, ...]) –

largest_incomplete(limit=8)

Return the largest decent tubes whose two ends do not reach a border.

Parameters

limit (int) –

Return type

tuple[connectomics.metrics.tube.TubeRecord, …]

property summary: connectomics.metrics.tube.TubeAnalysisSummary

Compute aggregate statistics over the analyzed tubes.

class connectomics.metrics.TubeAnalysisConfig(substantial_min_z_slices=21, substantial_min_voxels=10000, long_span_fraction=0.25, decent_min_voxels=20000, border_margin=2, border_patch_min_voxels=10, multi_component_min_voxels=50, multi_component_slice_step=3, parallel_min_slices=15, parallel_fraction_threshold=0.3, disconnected_component_min_voxels=1000, bump_min_slices=40, bump_relative_excess=0.2, bump_absolute_excess=200, bump_max_slices=30, bump_median_window=31)

Thresholds for analyze_tubes().

A label enters the detailed analysis when it is either long or large. border_margin is the maximum voxel distance from a volume face, so a margin of two includes coordinates 0, 1, and 2. Parallel-strand fractions are estimated from every multi_component_slice_step slice; parallel_min_slices remains expressed in original z slices.

Parameters

• substantial_min_z_slices (int) –

• substantial_min_voxels (int) –

• long_span_fraction (float) –

• decent_min_voxels (int) –

• border_margin (int) –

• border_patch_min_voxels (int) –

• multi_component_min_voxels (int) –

• multi_component_slice_step (int) –

• parallel_min_slices (int) –

• parallel_fraction_threshold (float) –

• disconnected_component_min_voxels (int) –

• bump_min_slices (int) –

• bump_relative_excess (float) –

• bump_absolute_excess (int) –

• bump_max_slices (int) –

• bump_median_window (int) –

class connectomics.metrics.TubeAnalysisSummary(total_label_count, total_foreground_voxels, substantial_count, substantial_voxels, long_enough_count, decent_count, decent_voxels, complete_count, complete_voxels, valid_count, valid_voxels, parallel_count, disconnected_count, bumped_count, complete_fraction, complete_volume_fraction, valid_fraction, valid_volume_fraction, face_count_histogram, border_end_histogram)

Aggregate count- and volume-weighted tube statistics.

Parameters

• total_label_count (int) –

• total_foreground_voxels (int) –

• substantial_count (int) –

• substantial_voxels (int) –

• long_enough_count (int) –

• decent_count (int) –

• decent_voxels (int) –

• complete_count (int) –

• complete_voxels (int) –

• valid_count (int) –

• valid_voxels (int) –

• parallel_count (int) –

• disconnected_count (int) –

• bumped_count (int) –

• complete_fraction (float) –

• complete_volume_fraction (float) –

• valid_fraction (float) –

• valid_volume_fraction (float) –

• face_count_histogram (tuple[int, ...]) –

• border_end_histogram (tuple[int, ...]) –

class connectomics.metrics.TubeRecord(label, voxel_count, z_min, z_max, z_span, z_slice_count, z_occupancy_fraction, is_long_enough, is_decent, face_contacts, border_end_count, border_patch_count, median_cross_section_area, max_cross_section_area, bump_count, multi_component_sample_count, evaluated_sample_count, multi_component_fraction, is_parallel, component_count_3d, significant_component_count_3d, is_disconnected)

GT-free measurements for one substantial predicted instance.

Parameters

• label (int) –

• voxel_count (int) –

• z_min (int) –

• z_max (int) –

• z_span (int) –

• z_slice_count (int) –

• z_occupancy_fraction (float) –

• is_long_enough (bool) –

• is_decent (bool) –

• face_contacts (tuple[str, ...]) –

• border_end_count (int) –

• border_patch_count (int) –

• median_cross_section_area (float) –

• max_cross_section_area (int) –

• bump_count (int) –

• multi_component_sample_count (int) –

• evaluated_sample_count (int) –

• multi_component_fraction (float) –

• is_parallel (bool) –

• component_count_3d (int) –

• significant_component_count_3d (int) –

• is_disconnected (bool) –

property face_count: int

Number of distinct relaxed volume faces touched.

property is_complete: bool

Whether both z-directed tube ends reach a volume face.

property is_single_tube: bool

Whether no persistent parallel strand or 3D disconnection is found.

property is_valid_tube: bool

Whether the instance is both complete and geometrically single.

class connectomics.metrics.VariationOfInformation(dist_sync_on_step=False)

Torchmetrics-style wrapper around the numpy-based VOI implementation.

VOI (Variation of Information) measures the information-theoretic distance between two clusterings. It decomposes into: - VOI Split (H(X|Y)): Over-segmentation error (false splits) - VOI Merge (H(Y|X)): Under-segmentation error (false merges)

Lower values are better (0 = perfect match).

This wrapper lets us accumulate scores during Lightning test_step without manual numpy<->torch conversions in the training loop.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters

dist_sync_on_step (bool) –

compute()

Return total VOI (split + merge).

Return type

Tensor

compute_merge()

Return VOI merge (under-segmentation error).

Return type

Tensor

compute_split()

Return VOI split (over-segmentation error).

Return type

Tensor

update(preds, target)

Override this method to update the state variables of your metric class.

Parameters

• preds (Tensor) –

• target (Tensor) –

Return type

None

connectomics.metrics.adapted_rand(seg, gt, all_stats=False)

Compute Adapted Rand error as defined by the SNEMI3D contest [1]

Formula is given as 1 - the maximal F-score of the Rand index (excluding the zero component of the original labels). Adapted from the SNEMI3D MATLAB script, hence the strange style.

segnp.ndarray

the segmentation to score, where each value is the label at that point

gtnp.ndarray, same shape as seg

the groundtruth to score against, where each value is a label

all_statsboolean, optional

whether to also return precision and recall as a 3-tuple with rand_error

arefloat

The adapted Rand error; equal to $1 -

rac{2pr}{p + r}$,

where $p$ and $r$ are the precision and recall described below.

precfloat, optional

The adapted Rand precision. (Only returned when all_stats is True.)

recfloat, optional

The adapted Rand recall. (Only returned when all_stats is True.)

[1]: http://brainiac2.mit.edu/SNEMI3D/evaluation

connectomics.metrics.analyze_tubes(seg, config=None, stats=None)

Analyze a 3D instance segmentation without ground truth.

The detailed set includes every label meeting either the configured z-slice or voxel-count threshold. Completeness is reported separately for the stricter “decent” subset that is both large and spans a configured fraction of the volume in z.

Parameters

• seg (ndarray) – Three-dimensional non-negative integer instance segmentation.

• config (connectomics.metrics.tube.TubeAnalysisConfig | None) – Analysis thresholds. Defaults preserve the LiCONN research operating points.

• stats (Any) – Optional cached seg_stats output or (bounds, sizes).

Returns

Structured per-label and aggregate GT-free analysis.

Return type

TubeAnalysis

connectomics.metrics.completeness_report(seg, verbose_top=8, stats=None)

Print and return the complete/decent tube counts.

This preserves the original public completeness entry point while using the richer canonical tube analysis.

Parameters

• seg (ndarray) –

• verbose_top (int) –

• stats (Any) –

Return type

tuple[int, int]

connectomics.metrics.compute_nerl_score(segmentation, skeleton_value, *, skeleton_mask_value=None, resolution=None, merge_threshold=1, chunk_num=1, num_workers=1, graph_options=None)

Return (nerl, pred_erl, gt_erl) for one segmentation/skeleton pair.

Parameters

• segmentation (ndarray) –

• skeleton_value (Any) –

• skeleton_mask_value (Any) –

• resolution (Any) –

• merge_threshold (int) –

• chunk_num (int) –

• num_workers (int) –

• graph_options (connectomics.metrics.nerl.NerlGraphOptions | None) –

Return type

tuple[float, float, float]

connectomics.metrics.compute_nerl_score_details(segmentation, skeleton_value, *, skeleton_mask_value=None, resolution=None, merge_threshold=1, chunk_num=1, num_workers=1, graph_options=None)

Compute detailed NERL output for one segmentation/skeleton pair.

Parameters

• segmentation (ndarray) –

• skeleton_value (Any) –

• skeleton_mask_value (Any) –

• resolution (Any) –

• merge_threshold (int) –

• chunk_num (int) –

• num_workers (int) –

• graph_options (connectomics.metrics.nerl.NerlGraphOptions | None) –

Return type

NerlScoreResult

connectomics.metrics.evaluate_image_pair(pred, gt, threshold=128, dilation_size=5)

Evaluate single prediction-ground truth pair.

Parameters

• pred (ndarray) – Prediction mask (0-255 range)

• gt (ndarray) – Ground truth mask (0-255 range)

• threshold (int) – Threshold for binarizing prediction. Default: 128

• dilation_size (int) – Dilation size for skeleton matching. Default: 5

Returns

Tuple of (iou, correctness, completeness, quality) metrics

• Returns (1.0, 1.0, 1.0, 1.0) if GT is empty

• All values in range [0.0, 1.0]

Return type

Tuple[float, float, float, float]

connectomics.metrics.format_tube_analysis(analysis, top_incomplete=8)

Format a concise human-readable tube analysis report.

Parameters

• analysis (TubeAnalysis) –

• top_incomplete (int) –

Return type

str

connectomics.metrics.instance_matching(y_true, y_pred, thresh=0.5, criterion='iou', report_matches=False)

Calculate detection/instance segmentation metrics between ground truth and predictions.

Currently, the following metrics are implemented:

‘fp’, ‘tp’, ‘fn’, ‘precision’, ‘recall’, ‘accuracy’, ‘f1’, ‘criterion’, ‘thresh’, ‘n_true’, ‘n_pred’, ‘mean_true_score’, ‘mean_matched_score’, ‘panoptic_quality’

Corresponding objects of y_true and y_pred are counted as true positives (tp), false positives (fp), and false negatives (fn) when their intersection over union (IoU) >= thresh (for criterion=’iou’, which can be changed)

• mean_matched_score is the mean IoUs of matched true positives

• mean_true_score is the mean IoUs of matched true positives but normalized by the total number of GT objects

• panoptic_quality defined as in Eq. 1 of Kirillov et al. “Panoptic Segmentation”, CVPR 2019

Parameters

• y_true (ndarray) – ground truth label image (integer valued)

• y_pred (ndarray) – predicted label image (integer valued)

• thresh (float) – threshold for matching criterion (default 0.5)

• criterion (string) – matching criterion (default IoU)

• report_matches (bool) – if True, additionally calculate matched_pairs and matched_scores (returns gt-pred pairs even when scores are below ‘thresh’)

Return type

Matching object with different metrics as attributes

Examples

>>> y_true = np.zeros((100,100), np.uint16)
>>> y_true[10:20,10:20] = 1
>>> y_pred = np.roll(y_true,5,axis = 0)

>>> stats = instance_matching(y_true, y_pred)
>>> print(stats)
Matching(criterion='iou', thresh=0.5, fp=1, tp=0, fn=1, precision=0,
         recall=0, accuracy=0, f1=0, n_true=1, n_pred=1,
         mean_true_score=0.0, mean_matched_score=0.0, panoptic_quality=0.0)

connectomics.metrics.instance_matching_simple(y_true, y_pred, thresh=0.5, criterion='iou')

Calculate relaxed instance segmentation metrics without Hungarian matching.

WARNING: This is a RELAXED metric for debugging/analysis only, NOT for benchmark ranking. Unlike instance_matching(), this does NOT use optimal bipartite matching (Hungarian algorithm). Instead, it simply counts all (GT, Pred) pairs with IoU >= threshold as true positives.

This metric is useful for: - Quick debugging and sanity checks - Understanding raw overlap statistics - Comparing with strict Hungarian-based metrics

Metrics computed:

‘tp’, ‘fp’, ‘fn’, ‘precision’, ‘recall’, ‘accuracy’, ‘f1’, ‘criterion’, ‘thresh’, ‘n_true’, ‘n_pred’

Parameters

• y_true (ndarray) – ground truth label image (integer valued)

• y_pred (ndarray) – predicted label image (integer valued)

• thresh (float) – threshold for matching criterion (default 0.5)

• criterion (string) – matching criterion (default ‘iou’)

Return type

Dictionary with metrics (tp, fp, fn, precision, recall, accuracy, f1, etc.)

Examples

>>> y_true = np.zeros((100,100), np.uint16)
>>> y_true[10:20,10:20] = 1
>>> y_pred = np.roll(y_true, 5, axis=0)
>>> stats = instance_matching_simple(y_true, y_pred)
>>> print(f"Accuracy: {stats['accuracy']:.3f}")

connectomics.metrics.oracle_merge_segmentation(seg, gt)

Relabel each predicted fragment to its majority-overlap ground-truth ID.

Parameters

• seg (ndarray) –

• gt (ndarray) –

Return type

ndarray