# Data

## ISIC 2017 dataset

ISIC 2017 is the primary dataset used to train, validate, and test the UNet2D segmentation model in SkiNet. The project uses the official predefined splits and does not re-shuffle them.

**Citation:** Codella N, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza S, Kalloo A, Liopyris K, Mishra N, Kittler H, Halpern A. "Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)". [arXiv: 1710.05006](https://doi.org/10.48550/arXiv.1710.05006)

### Class distribution

Three mutually exclusive classes, encoded as binary columns in the metadata CSV:

| Class | `melanoma` | `seborrheic_keratosis` | Training count |
|---|---|---|---|
| Melanoma | 1 | 0 | 374 |
| Seborrheic keratosis | 0 | 1 | 254 |
| Nevus | 0 | 0 | 1372 |

Validation: 150 images. Test: 600 images. Class proportions are consistent across splits.

### Expected directory structure

As generated by the ISIC 2017 challenge download:

```
<local_data_root>/
├── ISIC-2017_Training_Data/
│   └── ISIC-2017_Training_Data/        # archive extracts a same-named inner folder
│       ├── ISIC_0000000.jpg
│       └── ...
├── ISIC-2017_Training_Part1_GroundTruth/
│   └── ISIC-2017_Training_Part1_GroundTruth/
│       ├── ISIC_0000000_segmentation.png
│       └── ...
├── ISIC-2017_Training_Part3_GroundTruth.csv
├── ISIC-2017_Validation_Data/
├── ISIC-2017_Validation_Part1_GroundTruth/
├── ISIC-2017_Validation_Part3_GroundTruth.csv
├── ISIC-2017_Test_v2_Data/
├── ISIC-2017_Test_v2_Part1_GroundTruth/
└── ISIC-2017_Test_v2_Part3_GroundTruth.csv
```

Each `*_Data/` and `*_Part1_GroundTruth/` archive extracts to a same-named inner
folder, so image and mask files sit one level deeper than the top-level split
directories. This is why the glob patterns below contain an extra `*/` level.

### File naming conventions

- **Images:** `ISIC_XXXXXXX.jpg` (7-digit zero-padded ID), inside the same-named inner folder under `*_Data/`.
- **Masks:** `ISIC_XXXXXXX_segmentation.png`, binary PNG, inside the same-named inner folder under `*_Part1_GroundTruth/`.
- **Diagnosis CSVs:** One per split, named `ISIC-2017_{Split}_Part3_GroundTruth.csv`, containing `image_id`, `melanoma`, and `seborrheic_keratosis` columns.

The `sampleid` is the filename stem with `_segmentation` stripped (e.g. `ISIC_0000000`).

### Preprocessing notes

- Images are JPEG RGB; masks are binary PNG (values 0 or 255). No format conversion is required.
- The metadata CSV generator reads file paths via glob patterns (`ISIC-2017_*_Data/*/*.jpg` for images,
  `ISIC-2017_*_Part1_GroundTruth/*/*_segmentation.png` for masks) and determines split membership from
  the parent directory name (`Training`/`Validation`/`Test`).
- Pre-computed per-channel normalization statistics for the ISIC 2017 training split:
  `mean = [0.699, 0.556, 0.512]`, `std = [0.158, 0.156, 0.171]`. Paste these under
  `TRANSFORM_CONFIG` as `normalization_mean` / `normalization_std` with `normalization_mode: "standard"`.
  To recompute, run [compute_dataset_stats.py](https://github.com/pkliui/SkiNet/blob/dev/SkiNet/ML/transformations/compute_dataset_stats.py) — it reads the dataset from the YAML config, uses a single-pass pixel-weighted Welford algorithm over raw training images, and prints `TRANSFORM_CONFIG`-ready values:
  ```bash
  python -m SkiNet.ML.transformations.compute_dataset_stats --config main_config.yaml
  ```

### Download: from ISIC website
Download the three splits (Training Data, Validation Data, Test Data) and the corresponding Part 1 and Part 3
ground truth archives from [https://challenge.isic-archive.com/data/](https://challenge.isic-archive.com/data/).
Extract all archives into the same `<local_data_root>`.

### Download: from a prepared Kaggle dataset on Lightning Studio

```bash
pip install --quiet kaggle
```

- Data is downloaded to Lightning Storage:

```bash
ISIC_OUT_DIR="${ISIC_OUT_DIR:-/teamspace/lightning_storage/isic2017/ISIC2017DATA_256}"
kaggle datasets download -d johnchfr/isic-2017 -p $ISIC_OUT_DIR --unzip
```

The ISIC directory is then bind-mounted inside the Docker container at `CONTAINER_MOUNT_PATH="${CONTAINER_MOUNT_PATH:-/mnt/data}"`. When running inside the Docker container via
startup scripts, set `local_data_root: "/mnt/data/"`.

**WARNING:** Download the data on the **host** (into Lightning Storage), then bind-mount that directory at `/mnt/data` — this is what the startup scripts do. If you instead launch a container manually and run `kaggle datasets download -p /mnt/data` *inside* it without mounting `/mnt/data`, the data goes to the container's ephemeral layer and is lost when the container is removed. See [development.md](development.md#lightning-studio) for the explicit `--mount …,dst=/mnt/data` flag.

### Generate metadata CSV

Run once after download. Writes `isic2017_metadata.csv` into `<local_data_root>`:

```bash
python -m SkiNet.ML.datasets.preprocessing.metadata_csv_factory \
  --dataset-key-str ISIC2017 \
  --local-data-root "PATH_TO_LOCAL_DATA_ROOT_OR_BIND_MOUNT"
```

#### Metadata CSV columns

| Column | Type | Description |
|---|---|---|
| `sampleid` | str | Unique sample ID, e.g. `ISIC_0000000`. Derived from the filename stem with `_segmentation` stripped; join key for merging path rows with diagnosis labels. |
| `datapath` | str | Path to the image or mask file |
| `datatype` | str | `"image"` or `"mask"` |
| `predefined_split` | str | `"train"`, `"val"`, or `"test"` (from directory name); falls back to `"unknown"` if no split keyword matches the parent directory |
| `melanoma` | float | `1.0` if melanoma, `0.0` otherwise |
| `seborrheic_keratosis` | float | `1.0` if seborrheic keratosis, `0.0` otherwise |

**Row granularity — one row per file, not per sample.** Each sample contributes two rows: an
`image` row and a `mask` row (distinguished by `datatype`), sharing the same `sampleid`. The
diagnosis labels (`melanoma`, `seborrheic_keratosis`) are joined onto **both** rows, so filtering or
counting on the raw CSV without first restricting to `datatype == "image"` will double every count.

For a complete ISIC 2017 download, the generated `isic2017_metadata.csv` has:

| | image rows | mask rows | total |
|---|---|---|---|
| train | 2000 | 2000 | 4000 |
| val | 150 | 150 | 300 |
| test | 600 | 600 | 1200 |
| **total** | **2750** | **2750** | **5500** |

A row count that is short of 5500 indicates a partial download or an unmatched image/mask pair.

The merge is performed in `ISIC2017BaseCSVBuilder.create_merged_isic2017_metadata()` ([isic2017_csv_builder.py](https://github.com/pkliui/SkiNet/blob/dev/SkiNet/ML/datasets/preprocessing/isic2017_csv_builder.py)) via `DataFrame.merge(..., on="sampleid", how="left")`. The path-derived metadata (images + masks rows) is the left frame; the diagnosis CSVs are concatenated across splits and their `image_id` column is renamed to `sampleid` before the join.

### Config

- For Docker container runs:

```yaml
GENERAL_CONFIG:
  experiment_type: "segmentation"
  model: "UNET2D_MODEL"
  dataset: "ISIC2017_DATASET"
DATA_CONFIG:
  local_data_root: "/mnt/data/"
  azure_data: False
  split_random_seed: 100
```

- For runs on e.g. Kaggle or any other machine, specify the path to the data root on that machine:
```yaml
GENERAL_CONFIG:
  experiment_type: "segmentation"
  model: "UNET2D_MODEL"
  dataset: "ISIC2017_DATASET"
DATA_CONFIG:
  local_data_root: "/kaggle/working/isic2017_data_256/"
  azure_data: False
  split_random_seed: 100
```


> **Note:** ISIC 2017 is large enough that `cache_in_ram: true` may exhaust available RAM on a Lightning Studio
> instance. Set `cache_in_ram: false` in `TRAIN_CONFIG` when RAM is limited.

---

## Metadata lazy loading

`BaseDataConfig` exposes dataset metadata as a pandas DataFrame through the `metadata` property.
The mechanism is **lazy** (no I/O at construction time), **validated** (required columns checked on
first load), and **environment-aware** (handles local paths and Azure Blob Storage transparently).

### Lazy loading

The first call to `.metadata` triggers a CSV read and column validation. Subsequent calls return the
in-memory cache immediately:

```python
cfg = ISIC2017DatasetConfig(local_data_root="/data/isic2017")

df = cfg.metadata   # reads CSV, validates columns, caches
df = cfg.metadata   # returns cache — no disk I/O
```

The cache lives in the private attribute `_metadata` (`PrivateAttr`, default `None`). It is excluded
from Pydantic validation and serialization.

### Deepcopy behaviour

`BaseDataConfig.__deepcopy__` resets `_metadata` to `None` in every copy. Config fields (paths, split
parameters) are deep-copied normally; only the transient cache is cleared.

During Optuna sweeps `deepcopy(main_config)` is called once per trial:

```python
from copy import deepcopy

_ = main_config.dataconfig.metadata          # loads once on template config

trial_cfg = deepcopy(main_config)
assert trial_cfg.dataconfig._metadata is None  # no DataFrame duplication

df = trial_cfg.dataconfig.metadata           # lazy-loads fresh copy per trial
```

### Required subclass configuration

```python
class MyDatasetConfig(BaseDataConfig):
    METADATA_CSV_NAME: ClassVar[str] = "metadata.csv"
    REQUIRED_COLUMNS: ClassVar[frozenset[str]] = frozenset({"sampleid", "datapath", "datatype"})
    DATASET_KEY: ClassVar[Optional[DatasetKey]] = DatasetKey.MY_DATASET
```

A `ValueError` is raised on first `.metadata` access if required columns are absent or all empty.

> **Note:** `REQUIRED_COLUMNS` validates only the join/path columns (`sampleid`, `datapath`,
> `datatype`). The diagnosis columns (`melanoma`, `seborrheic_keratosis`) are **not** in this set,
> so missing or empty diagnosis labels pass validation but will break stratified splitting, which
> relies on `split_stratify_column`.