EF — ISIC 2017 UNet2D: held-out test score for the selected E4 checkpoint

Scope. This notebook runs the already-selected E4 checkpoint once over the held-out 600-image test split and reports its per-image mean Dice / IoU (ISIC-2017 official averaging), then places the result in the context of the official ISIC 2017 challenge leaderboard.

  • Checkpoint input: seed 108, epoch192 (classical + attention_gate, lr = 3e-4). Set SELECTED_SEED to score a different one.

  • Threshold: Found on the validation set in the E4 threshold-sweep notebook (E4_isic2017_unet2d_threshold_sweep_analysis.ipynb) and pasted here as TAU_FROM_E4.

Metric. Per-image mean Dice/IoU via SkiNet.Utils.analysis.test_scoring — the same scoring core calibrate_threshold.py uses to batch-score all checkpoints.

1. Inputs — selected checkpoint + threshold

[1]:

import sys
from pathlib import Path

import torch

PROJECT_ROOT = Path('/teamspace/studios/this_studio/repos/SkiNet')
sys.path.insert(0, str(PROJECT_ROOT))

from SkiNet.ML.configs.load_config_from_yaml import load_config_from_yaml
from SkiNet.ML.dataloaders.create_dataloaders import create_segmentation_dataloaders
from SkiNet.Utils.analysis.test_scoring import build_ckpt_map, load_uncompiled, collect_probs, score_at_thresholds

# -- inputs (decided upstream, not here) ---------------------------------------
SELECTED_SEED = 108            # production checkpoint, chosen in E4 (epoch192)
TAU_FROM_E4   = None           # paste the val-found threshold from E4; None -> score only 0.5
DATA_ROOT     = "/teamspace/lightning_storage/isic2017/ISIC2017DATA_256/"          # e.g. '/teamspace/lightning_storage/isic2017/ISIC2017DATA_256/'; None -> use config

CONFIG_PATH = PROJECT_ROOT / 'main_config.yaml'
DB1 = PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep.db'        # seeds 100-105
DB2 = PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep-part2.db'  # seeds 106-109
CKPT_GLOB = str(PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep*' /
                '1' / '*' / 'artifacts' / 'checkpoints' / 'best' / '*.ckpt')

ckpt_map  = build_ckpt_map(DB1, DB2, glob_pattern=CKPT_GLOB, project_root=PROJECT_ROOT)
ckpt_path = ckpt_map[SELECTED_SEED]

thresholds = [0.5] + ([float(TAU_FROM_E4)] if TAU_FROM_E4 is not None else [])
print(f"Selected checkpoint : seed {SELECTED_SEED} -> {ckpt_path.relative_to(PROJECT_ROOT)}")
print(f"Thresholds to score : {thresholds}"
      + ("" if TAU_FROM_E4 is not None else "   (set TAU_FROM_E4 to add the val-found threshold)"))

/home/zeus/miniconda3/envs/cloudspace/lib/python3.12/site-packages/azureml/dataprep/api/_loggerfactory.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

Selected checkpoint : seed 108 -> mlruns/E4-isic2017-unet2d-thres-sweep-part2/1/a8cb781b18f64549a0d36ca9e096cee5/artifacts/checkpoints/best/epoch192.ckpt
Thresholds to score : [0.5]   (set TAU_FROM_E4 to add the val-found threshold)

2. Run test inference once

Build the official test split, load the selected checkpoint uncompiled, and collect per-image sigmoid probabilities + masks in a single pass. Needs the ISIC-2017 data available (set DATA_ROOT if not on the configured path) and ideally a GPU.

[2]:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

cfg = load_config_from_yaml(CONFIG_PATH)
if DATA_ROOT:
    cfg.dataconfig.local_data_root = DATA_ROOT
cfg.trainconfig.test_on_val_split = False

dl = create_segmentation_dataloaders(cfg)
assert dl.test is not None, "No test dataloader — check predefined_split / test_on_val_split."

model = load_uncompiled(cfg, ckpt_path)
test_probs, test_masks = collect_probs(model, dl.test, device)
print(f"Inference done on {test_probs.shape[0]} test images (device={device}).")

EarlyStopping monitor is set to default MetricsKey.VAL_MEAN_DICE_PER_IMAGE — make sure your LightningModule logs this exact key.
/home/zeus/miniconda3/envs/cloudspace/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1118: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
  super().__init__(loader)

Inference done on 600 test images (device=cpu).

3. Held-out test score (per-image mean Dice, ISIC-official)

[3]:

scores = score_at_thresholds(test_probs, test_masks, thresholds, n_boot=1000, seed=SELECTED_SEED)
scores['role'] = ['deployable default' if t == 0.5 else 'val-found τ (from E4)' for t in scores['threshold']]
scores['dice_95ci'] = [f"[{lo:.4f}, {hi:.4f}]" for lo, hi in zip(scores['dice_lo'], scores['dice_hi'])]

display(scores[['threshold', 'role', 'dice', 'iou', 'dice_95ci']].style.hide(axis='index').format(
    {'threshold': '{:.2f}', 'dice': '{:.4f}', 'iou': '{:.4f}'}))

base = scores.loc[scores['threshold'] == 0.5].iloc[0]
print(f"\nSelected checkpoint : seed {SELECTED_SEED}, {ckpt_path.name}")
print(f"Held-out test split : {test_probs.shape[0]} images (official ISIC-2017), per-image mean Dice")
print(f"Headline @0.5       : test Dice = {base['dice']:.4f}  CI {base['dice_95ci']}  | test IoU = {base['iou']:.4f}")
if TAU_FROM_E4 is not None:
    tau = scores.loc[scores['threshold'] != 0.5].iloc[0]
    print(f"At E4 τ={tau['threshold']:.2f}    : test Dice = {tau['dice']:.4f}  CI {tau['dice_95ci']}  | test IoU = {tau['iou']:.4f}")
threshold role dice iou dice_95ci
0.50 deployable default 0.8356 0.7494 [0.8208, 0.8494] 

Selected checkpoint : seed 108, epoch192.ckpt
Held-out test split : 600 images (official ISIC-2017), per-image mean Dice
Headline @0.5       : test Dice = 0.8356  CI [0.8208, 0.8494]  | test IoU = 0.7494

Reading the table. Both rows score the same checkpoint on the same test images; they differ only in the threshold:

Threshold

Where it comes from

Role

0.5

fixed default, no tuning

the deployable, ISIC-comparable headline

τ (from E4)

found on the 150 val images in the E4 notebook

the alternative if E4 decided to threshold-tune

This notebook reports both numbers and stops — whether to ship 0.5 or τ is the E4 notebook’s decision, made on validation. No threshold is fitted on the test set here.

4. Conclusion & ISIC 2017 Leaderboard Comparison

Headline result

Metric

Score

95 % bootstrap CI

Dice @ 0.5

0.8356

[0.8208, 0.8494]

IoU @ 0.5

0.7494

—

Checkpoint: seed 108, epoch192 · architecture: UNet2D + attention gate · 600 held-out test images (official ISIC-2017 split).

ISIC 2017 Task 1 leaderboard (ranked by Jaccard / IoU)

Source: https://challenge.isic-archive.com/leaderboards/2017/

Rank

Team

IoU (Jaccard)

1

Mt. Sinai

0.765

2

NLP LOGIX / WISEEYEAI

0.762

3

USYD-BMIT (MResNet-Seg)

0.760

4

USYD-BMIT (ResNet + extra data)

0.758

5

RECOD Titans

0.754

6

Jer

0.752

7

NedMos — Tarbiat Modares University

0.749

~ 8

SkiNet UNet2D (this work, @0.5)

0.7494

8

INESC TEC Porto / Tecnalia

0.735

9

CV Institute, Shenzhen University

0.718

10

GAMMA Group

0.715

Interpretation. An IoU of 0.7494 places SkiNet just outside the top-7, only 0.0004 behind rank 7 and 0.016 behind the 2017 winner. This is a competitive result for a clean UNet2D baseline — no ensemble, no test-time augmentation, no extra training data.

Two caveats for fair interpretation:

  • The leaderboard competitors were scored in 2017 on the same held-out split, so the comparison is metric-equivalent.

  • TAU_FROM_E4 = None here — the val-found threshold from the E4 sweep (if < 0.5) could push IoU slightly higher.

5. Discussion: factors contributing to competitive performance

The IoU of 0.7494 is competitive with the 2017 challenge top-10 despite using a single checkpoint with a fixed threshold. Three factors explain this.

Post-competition architecture. Attention gates in encoder–decoder networks were not published until 2018 (Oktay et al., Attention U-Net: Learning Where to Look for the Pancreas, MIDL 2018) and were therefore unavailable to the 2017 entrants. The attention mechanism selectively suppresses irrelevant background activations at skip connections, which is directly beneficial for lesion segmentation where the foreground occupies a small, irregular region.

Modern training methodology. The 2017 leaderboard teams relied primarily on SGD with hand-tuned momentum and learning-rate schedules. SkiNet benefits from improvements that have become standard in the intervening years: the Adam family of optimisers with principled learning-rate selection (lr = 3×10⁻⁴), Optuna-driven hyperparameter search across seeds 100–109, and PyTorch Lightning’s training loop which reduces implementation variance. The production checkpoint (seed 108, epoch 192) was selected on the validation split after this automated sweep — an advantage unavailable to competition participants operating under a submission deadline.

Seed and checkpoint selection. The reported score reflects the best checkpoint out of a ten-seed sweep evaluated on held-out validation data, not a single training run. This is methodologically sound (no test-set information is used for selection) but does confer a selection advantage relative to challenge submissions, which were typically one or a small number of manual attempts.

Scope of the comparison. The metric is directly comparable — per-image mean Jaccard on the same 600-image official test split — but the conditions are not equivalent. SkiNet was developed with access to eight years of published advances. The appropriate interpretation is that a well-tuned, attention-augmented UNet2D trained with modern tooling is competitive with 2017 state of the art, not that it surpasses those teams under equal experimental conditions.

Remaining gap to first place. The 0.016 IoU gap to the Mt. Sinai winner (0.765) is unlikely to be closed by further single-model tuning alone. The most probable routes are: multi-model ensembling, test-time augmentation, a stronger convolutional backbone (e.g. ResNet or EfficientNet encoder), or a transformer-based architecture. These are left as future work.