Datasets and dataloaders

This document describes the dataset system used in SkiNet. See the API reference for full parameter documentation.

Quickstart

  • At present, only segmentation datasets are supported.

  • The recommended way to create datasets is via create_segmentation_datasets_from_config, which handles splitting, transform construction, and dataset instantiation in one call.

  • The resulting DatasetSplit container provides train, val, and test datasets that can be passed to PyTorch dataloaders directly:

from SkiNet.ML.configs.load_config_from_yaml import load_config_from_yaml
from SkiNet.ML.datasets.dataset_factory import create_segmentation_datasets_from_config
from torch.utils.data import DataLoader

cfg = load_config_from_yaml(cfg_path)
datasets = create_segmentation_datasets_from_config(cfg)

train_loader = DataLoader(datasets.train, batch_size=8, shuffle=True,  num_workers=4)
val_loader   = DataLoader(datasets.val,   batch_size=8, shuffle=False, num_workers=4)

from SkiNet.ML.dataloaders.create_dataloaders import create_dataloaders_from_datasets
from SkiNet.ML.datasets.dataset_factory import DatasetSplit, create_segmentation_datasets_from_config
from SkiNet.ML.datasets.segmentation_dataset import SegmentationDataset

segm_datasets: DatasetSplit[SegmentationDataset] = create_segmentation_datasets_from_config(main_config)
loaders = create_dataloaders_from_datasets(segm_datasets, main_config.trainconfig)
print("Train dataloader: ", loaders.train)
print("Val dataloader: ", loaders.val)
print("Test dataloader: ", loaders.test)

Under the hood this creates RepeatDataLoader instances that avoid spawning new worker processes each epoch — see Note about modification to PyTorch’s default dataloader.

Dataset Class

SegmentationDataset (ISIC 2017)

SkiNet.ML.datasets.segmentation_dataset.SegmentationDataset implements a PyTorch Dataset for semantic segmentation. Each sample is an image–mask pair. It is the only dataset the factory builds (registered in _DATASET_FACTORIES for ExperimentType.SEGMENTATION).

Constructor signature:

SegmentationDataset(
    data_root: Path,
    dataframe: pd.DataFrame,
    transform: SampleTransformAdapter,
    mode: MLWorkflowState,
    cache_in_ram: bool = True,
)

Each item returned by __getitem__ is a dict:

item  = dataset[0]
image = item["image"]   # transformed image tensor
mask  = item["mask"]    # transformed mask tensor
specs = item["specs"]   # sample metadata dict

cache_in_ram=True pre-loads all images into a RAM dict at construction time using a ThreadPoolExecutor (up to 8 workers). Workers then only perform augmentation, never disk reads. Set cache_in_ram=False for ISIC 2017 or other large datasets when RAM is limited.

Dataset Splits

Splitting into train/val/test subsets is handled by split_segmentation_metadata, which returns a DataFrameSplits object. The factory calls this internally; call it directly if you need the raw DataFrames:

from SkiNet.Utils.data.split_data import split_segmentation_metadata

main_config  = load_config_from_yaml(cfg_path)
split_config = main_config.dataconfig.get_split_config()
metadata_df  = main_config.dataconfig.metadata

splits   = split_segmentation_metadata(df=metadata_df, split_config=split_config)
train_df = splits.train
val_df   = splits.val
test_df  = splits.test

For ISIC 2017, get_split_config() returns a SplitConfig that uses the predefined_split column instead of a random split, respecting the official challenge splits. Set predefined_split_column: null in DATA_CONFIG to fall back to random splitting.

Modifications to the default Dataset class

The following describes modifications to PyTorch’s default Dataset class to prevent the “Memory-on-copy” problem documented in https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662

Background and Motivation

Jupyter notebook: SkiNet/ML/datasets/experiments/MemoryUsage_Dataset.ipynb

When num_workers > 0, memory usage increases with each epoch for some users due to copy-on-write behaviour in forked Python processes. From the GitHub thread:

“If your DataLoaders iterate across a list of filenames, the references to that list add up over time, occupying memory. … The simplest workaround is to replace native Python objects (dicts, lists) with array objects that only have one refcount (pandas, numpy, pyarrow).”

SkiNet stores file paths in np.array(..., dtype=np.bytes_) rather than Python lists.

Augmentation of data

Augmentation uses Albumentations. For training, the process has four stages:

  1. Cropping to a pre-defined size

  2. Spatial transformations (applied to both image and mask)

  3. Photometric transformations (image only)

  4. Normalization and conversion to tensor

Transformation pipelines

Default pipelines for cropping, spatial, and photometric transforms are defined in SkiNet/ML/transformations/transform_pipelines.py. By default, all transformations except normalization are disabled and must be enabled in the YAML config.

Affine rotation uses nearest-neighbor interpolation (Albumentations defaults to cv2.INTER_NEAREST for mask targets), preserving pixel values at 0 or 255 for uint8 masks.

Normalization mode is controlled by normalization_mode in TRANSFORM_CONFIG:

Mode

Behaviour

When to use

"image_per_channel" (default)

Per-sample per-channel mean/std on the augmented image

General purpose; no dataset stats needed

"standard"

Fixed global mean/std from normalization_mean / normalization_std

When dataset statistics are pre-computed

Example for ISIC 2017 pre-computed statistics:

TRANSFORM_CONFIG:
  normalization_mode: "standard"
  normalization_mean: [0.699, 0.556, 0.512]
  normalization_std:  [0.158, 0.156, 0.171]

normalization_mean and normalization_std are required when normalization_mode: "standard" — a ValueError is raised at pipeline construction time if they are absent.

Configuration for transformation

TRANSFORM_CONFIG:
  seed_value: 100
  crop:
    crop_apply: False
    crop_type: "random_resized_crop"
    size: [256, 256]
    scale: [0.8, 1.0]
  spatial_augmentation:
    square_symmetry_apply: True
    square_symmetry_p: 0.5
    affine_apply: True
    affine_scale: [0.8, 1.0]
    affine_translate_percent: {"x": [-0.05, 0.05], "y": [-0.05, 0.05]}
    affine_rotate: [-20, 20]
    perspective_apply: True
    perspective_scale: [0.05, 0.1]
    perspective_p: 0.2
    elastic_apply: True
    elastic_alpha: 120
    elastic_sigma: 10
    elastic_p: 0.1
  photometric_augmentation:
    color_jitter_apply: True
    color_jitter_brightness: 0.2
    color_jitter_contrast: 0.2
    color_jitter_p: 0.3
    gaussian_blur_apply: True
    gaussian_blur_sigma_limit: [0.5, 2.0]
    gaussian_blur_p: 0.2
    gaussian_noise_apply: True
    gaussian_noise_std_range: [0.01, 0.05]
    gaussian_noise_p: 0.2

The crop size must be divisible by stride ^ number_of_downsampling_layers (e.g., 16 for 4 downsampling layers with stride 2). This is validated at config-load time by SkiNet.ML.configs.experiment_config.ExperimentConfig.

Seeding and reproducibility

  • seed_value controls the augmentation sequence.

  • Using the same seed with different num_workers settings produces different augmentation sequences. Reproducibility requires the same seed AND the same num_workers.

  • Each worker gets a unique seed: base_seed + torch.initial_seed().

Train, val, and test pipelines

Training augmentations are strictly separate from validation and test preprocessing. get_transform_from_config builds mode-specific pipelines:

  • train: stochastic spatial and photometric augmentations

  • val/test: deterministic steps only (crop, pad, normalize)

from SkiNet.ML.configs.load_config_from_yaml import load_config_from_yaml
from SkiNet.ML.transformations.transform_data import get_transform_from_config, TransformsContainer

cfg = load_config_from_yaml(cfg_path)
transform: TransformsContainer = get_transform_from_config(cfg)

print("train pipeline: ", transform.train)
print("val pipeline: ", transform.val)
print("test pipeline: ", transform.test)

Visual inspection

from SkiNet.ML.transformations.plot_transformed_data import visualize_augmented_data

dataset = SegmentationDataset(data_root=..., dataframe=..., transform=transform.train, mode=...)
visualize_augmented_data(dataset, samples=20)

Saves individual images and a grid of augmented samples to a specified directory.