{ "cells": [ { "cell_type": "markdown", "id": "67cdf9ad", "metadata": {}, "source": [ "# EF — ISIC 2017 UNet2D: held-out test score for the selected E4 checkpoint\n", "\n", "**Scope.** This notebook **runs the already-selected E4 checkpoint once over\n", "the held-out 600-image test split and reports its per-image mean Dice / IoU** (ISIC-2017 official\n", "averaging), then places the result in the context of the official ISIC 2017 challenge leaderboard.\n", "\n", "- **Checkpoint input:** seed 108, `epoch192` (`classical + attention_gate`, `lr = 3e-4`). Set `SELECTED_SEED`\n", " to score a different one.\n", "- **Threshold:** Found on the *validation* set in the E4 threshold-sweep\n", " notebook (`E4_isic2017_unet2d_threshold_sweep_analysis.ipynb`) and pasted here as `TAU_FROM_E4`.\n", "\n", "**Metric.** Per-image mean Dice/IoU via `SkiNet.Utils.analysis.test_scoring` — the same scoring core\n", "`calibrate_threshold.py` uses to batch-score all checkpoints." ] }, { "cell_type": "markdown", "id": "fbeb7cde", "metadata": {}, "source": [ "## 1. Inputs — selected checkpoint + threshold" ] }, { "cell_type": "code", "execution_count": 1, "id": "76d728e6", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/zeus/miniconda3/envs/cloudspace/lib/python3.12/site-packages/azureml/dataprep/api/_loggerfactory.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n", " import pkg_resources\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Selected checkpoint : seed 108 -> mlruns/E4-isic2017-unet2d-thres-sweep-part2/1/a8cb781b18f64549a0d36ca9e096cee5/artifacts/checkpoints/best/epoch192.ckpt\n", "Thresholds to score : [0.5] (set TAU_FROM_E4 to add the val-found threshold)\n" ] } ], "source": [ "import sys\n", "from pathlib import Path\n", "\n", "import torch\n", "\n", "PROJECT_ROOT = Path('/teamspace/studios/this_studio/repos/SkiNet')\n", "sys.path.insert(0, str(PROJECT_ROOT))\n", "\n", "from SkiNet.ML.configs.load_config_from_yaml import load_config_from_yaml\n", "from SkiNet.ML.dataloaders.create_dataloaders import create_segmentation_dataloaders\n", "from SkiNet.Utils.analysis.test_scoring import build_ckpt_map, load_uncompiled, collect_probs, score_at_thresholds\n", "\n", "# -- inputs (decided upstream, not here) ---------------------------------------\n", "SELECTED_SEED = 108 # production checkpoint, chosen in E4 (epoch192)\n", "TAU_FROM_E4 = None # paste the val-found threshold from E4; None -> score only 0.5\n", "DATA_ROOT = \"/teamspace/lightning_storage/isic2017/ISIC2017DATA_256/\" # e.g. '/teamspace/lightning_storage/isic2017/ISIC2017DATA_256/'; None -> use config\n", "\n", "CONFIG_PATH = PROJECT_ROOT / 'main_config.yaml'\n", "DB1 = PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep.db' # seeds 100-105\n", "DB2 = PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep-part2.db' # seeds 106-109\n", "CKPT_GLOB = str(PROJECT_ROOT / 'mlruns' / 'E4-isic2017-unet2d-thres-sweep*' /\n", " '1' / '*' / 'artifacts' / 'checkpoints' / 'best' / '*.ckpt')\n", "\n", "ckpt_map = build_ckpt_map(DB1, DB2, glob_pattern=CKPT_GLOB, project_root=PROJECT_ROOT)\n", "ckpt_path = ckpt_map[SELECTED_SEED]\n", "\n", "thresholds = [0.5] + ([float(TAU_FROM_E4)] if TAU_FROM_E4 is not None else [])\n", "print(f\"Selected checkpoint : seed {SELECTED_SEED} -> {ckpt_path.relative_to(PROJECT_ROOT)}\")\n", "print(f\"Thresholds to score : {thresholds}\"\n", " + (\"\" if TAU_FROM_E4 is not None else \" (set TAU_FROM_E4 to add the val-found threshold)\"))" ] }, { "cell_type": "markdown", "id": "a6030508", "metadata": {}, "source": [ "## 2. Run test inference once\n", "\n", "Build the official test split, load the selected checkpoint uncompiled, and collect per-image\n", "sigmoid probabilities + masks in a single pass. Needs the ISIC-2017 data available (set `DATA_ROOT`\n", "if not on the configured path) and ideally a GPU." ] }, { "cell_type": "code", "execution_count": 2, "id": "576a3f9b", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "EarlyStopping monitor is set to default MetricsKey.VAL_MEAN_DICE_PER_IMAGE — make sure your LightningModule logs this exact key.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/zeus/miniconda3/envs/cloudspace/lib/python3.12/site-packages/torch/utils/data/dataloader.py:1118: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Inference done on 600 test images (device=cpu).\n" ] } ], "source": [ "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", "\n", "cfg = load_config_from_yaml(CONFIG_PATH)\n", "if DATA_ROOT:\n", " cfg.dataconfig.local_data_root = DATA_ROOT\n", "cfg.trainconfig.test_on_val_split = False\n", "\n", "dl = create_segmentation_dataloaders(cfg)\n", "assert dl.test is not None, \"No test dataloader — check predefined_split / test_on_val_split.\"\n", "\n", "model = load_uncompiled(cfg, ckpt_path)\n", "test_probs, test_masks = collect_probs(model, dl.test, device)\n", "print(f\"Inference done on {test_probs.shape[0]} test images (device={device}).\")" ] }, { "cell_type": "markdown", "id": "a190b492", "metadata": {}, "source": [ "## 3. Held-out test score (per-image mean Dice, ISIC-official)" ] }, { "cell_type": "code", "execution_count": 3, "id": "21d5b133", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
thresholdrolediceioudice_95ci
0.50deployable default0.83560.7494[0.8208, 0.8494]
\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Selected checkpoint : seed 108, epoch192.ckpt\n", "Held-out test split : 600 images (official ISIC-2017), per-image mean Dice\n", "Headline @0.5 : test Dice = 0.8356 CI [0.8208, 0.8494] | test IoU = 0.7494\n" ] } ], "source": [ "scores = score_at_thresholds(test_probs, test_masks, thresholds, n_boot=1000, seed=SELECTED_SEED)\n", "scores['role'] = ['deployable default' if t == 0.5 else 'val-found τ (from E4)' for t in scores['threshold']]\n", "scores['dice_95ci'] = [f\"[{lo:.4f}, {hi:.4f}]\" for lo, hi in zip(scores['dice_lo'], scores['dice_hi'])]\n", "\n", "display(scores[['threshold', 'role', 'dice', 'iou', 'dice_95ci']].style.hide(axis='index').format(\n", " {'threshold': '{:.2f}', 'dice': '{:.4f}', 'iou': '{:.4f}'}))\n", "\n", "base = scores.loc[scores['threshold'] == 0.5].iloc[0]\n", "print(f\"\\nSelected checkpoint : seed {SELECTED_SEED}, {ckpt_path.name}\")\n", "print(f\"Held-out test split : {test_probs.shape[0]} images (official ISIC-2017), per-image mean Dice\")\n", "print(f\"Headline @0.5 : test Dice = {base['dice']:.4f} CI {base['dice_95ci']} | test IoU = {base['iou']:.4f}\")\n", "if TAU_FROM_E4 is not None:\n", " tau = scores.loc[scores['threshold'] != 0.5].iloc[0]\n", " print(f\"At E4 τ={tau['threshold']:.2f} : test Dice = {tau['dice']:.4f} CI {tau['dice_95ci']} | test IoU = {tau['iou']:.4f}\")" ] }, { "cell_type": "markdown", "id": "8d80d8e5", "metadata": {}, "source": [ "**Reading the table.** Both rows score the *same* checkpoint on the *same* test images; they differ\n", "only in the threshold:\n", "\n", "| Threshold | Where it comes from | Role |\n", "|---|---|---|\n", "| **0.5** | fixed default, no tuning | the deployable, ISIC-comparable headline |\n", "| **τ (from E4)** | found on the 150 val images in the E4 notebook | the alternative *if* E4 decided to threshold-tune |\n", "\n", "This notebook reports both numbers and stops — whether to ship `0.5` or `τ` is the **E4 notebook's**\n", "decision, made on validation. No threshold is fitted on the test set here." ] }, { "cell_type": "markdown", "id": "5d8240ee", "metadata": {}, "source": [ "## 4. Conclusion & ISIC 2017 Leaderboard Comparison\n", "\n", "### Headline result\n", "\n", "| Metric | Score | 95 % bootstrap CI |\n", "|---|---|---|\n", "| **Dice @ 0.5** | **0.8356** | [0.8208, 0.8494] |\n", "| **IoU @ 0.5** | **0.7494** | — |\n", "\n", "Checkpoint: seed 108, `epoch192` · architecture: UNet2D + attention gate · 600 held-out test images (official ISIC-2017 split).\n", "\n", "---\n", "\n", "### ISIC 2017 Task 1 leaderboard (ranked by Jaccard / IoU)\n", "\n", "Source: https://challenge.isic-archive.com/leaderboards/2017/\n", "\n", "| Rank | Team | IoU (Jaccard) |\n", "|------|------|:---:|\n", "| 1 | Mt. Sinai | 0.765 |\n", "| 2 | NLP LOGIX / WISEEYEAI | 0.762 |\n", "| 3 | USYD-BMIT (MResNet-Seg) | 0.760 |\n", "| 4 | USYD-BMIT (ResNet + extra data) | 0.758 |\n", "| 5 | RECOD Titans | 0.754 |\n", "| 6 | Jer | 0.752 |\n", "| 7 | NedMos — Tarbiat Modares University | 0.749 |\n", "| **~ 8** | **SkiNet UNet2D (this work, @0.5)** | **0.7494** |\n", "| 8 | INESC TEC Porto / Tecnalia | 0.735 |\n", "| 9 | CV Institute, Shenzhen University | 0.718 |\n", "| 10 | GAMMA Group | 0.715 |\n", "\n", "**Interpretation.** An IoU of **0.7494 places SkiNet just outside the top-7**, only 0.0004 behind rank 7\n", "and 0.016 behind the 2017 winner. This is a competitive result for a clean UNet2D baseline —\n", "no ensemble, no test-time augmentation, no extra training data.\n", "\n", "Two caveats for fair interpretation:\n", "- The leaderboard competitors were scored in 2017 on the same held-out split, so the comparison is\n", " metric-equivalent.\n", "- `TAU_FROM_E4 = None` here — the val-found threshold from the E4 sweep (if < 0.5) could push IoU\n", " slightly higher." ] }, { "cell_type": "markdown", "id": "4878a0e8", "metadata": {}, "source": [ "## 5. Discussion: factors contributing to competitive performance\n", "\n", "The IoU of 0.7494 is competitive with the 2017 challenge top-10 despite using a single checkpoint\n", "with a fixed threshold. Three factors explain this.\n", "\n", "**Post-competition architecture.** Attention gates in encoder–decoder networks were not published\n", "until 2018 (Oktay et al., *Attention U-Net: Learning Where to Look for the Pancreas*, MIDL 2018)\n", "and were therefore unavailable to the 2017 entrants. The attention mechanism selectively suppresses\n", "irrelevant background activations at skip connections, which is directly beneficial for lesion\n", "segmentation where the foreground occupies a small, irregular region.\n", "\n", "**Modern training methodology.** The 2017 leaderboard teams relied primarily on SGD with\n", "hand-tuned momentum and learning-rate schedules. SkiNet benefits from improvements that have\n", "become standard in the intervening years: the Adam family of optimisers with principled\n", "learning-rate selection (lr = 3×10⁻⁴), Optuna-driven hyperparameter search across seeds\n", "100–109, and PyTorch Lightning's training loop which reduces implementation variance. The\n", "production checkpoint (seed 108, epoch 192) was selected on the validation split after this\n", "automated sweep — an advantage unavailable to competition participants operating under a\n", "submission deadline.\n", "\n", "**Seed and checkpoint selection.** The reported score reflects the best checkpoint out of a\n", "ten-seed sweep evaluated on held-out validation data, not a single training run. This is\n", "methodologically sound (no test-set information is used for selection) but does confer a\n", "selection advantage relative to challenge submissions, which were typically one or a small\n", "number of manual attempts.\n", "\n", "**Scope of the comparison.** The metric is directly comparable — per-image mean Jaccard on the\n", "same 600-image official test split — but the *conditions* are not equivalent. SkiNet was\n", "developed with access to eight years of published advances. The appropriate interpretation is\n", "that a well-tuned, attention-augmented UNet2D trained with modern tooling is competitive with\n", "2017 state of the art, not that it surpasses those teams under equal experimental conditions.\n", "\n", "**Remaining gap to first place.** The 0.016 IoU gap to the Mt. Sinai winner (0.765) is unlikely\n", "to be closed by further single-model tuning alone. The most probable routes are: multi-model\n", "ensembling, test-time augmentation, a stronger convolutional backbone (e.g. ResNet or\n", "EfficientNet encoder), or a transformer-based architecture. These are left as future work." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 5 }