Data Configs

pydantic model SkiNet.ML.configs.data_configs.base_data_config.BaseDataConfig[source]

Bases: BaseModel

Base class to configure dataset from metadata

Parameters:

  • azure_data – Whether data and metadata reside in Azure Blob Storage. If True, metadata is read directly from a mounted Azure Blob Storage, assuming the appropriate Managed Identity permissions and configurations are in place. If False, metadata and files are read from a local_data_root.

  • azure_blob_mount_point – The mount point for the Azure Blob Storage (if using Azure). This is not a path to data and CSV file, but the root mount point for the blob storage. The actual path to the dataset and metadata CSV within the blob storage is determined by the DATASET_KEY and AZURE_SETTINGS_YAML config.

  • local_data_root – The root path to the local data (if not using Azure). This is the directory where all data and the metadata CSV file resides.

Class attributes (must be set in subclasses):

  • METADATA_CSV_NAME — filename of the metadata CSV as defined in project paths.

  • REQUIRED_COLUMNS — frozenset of column names that must exist in the CSV.

  • DATASET_KEY — one of the DatasetKey values; must match the key in the YAML config.

Example usage (local CSV):

cfg = MyDatasetConfig(local_data_root=”some/local/path/to/data”, azure_data=False)

Example usage (Azure CSV):

cfg = MyDatasetConfig(azure_blob_mount_point=”mnt/data”, azure_data=True)

# in both cases, metadata dataframe is available through df = cfg.metadata

Note

  • For Azure, the value of the dataset key (DATASET_KEY.value) must match the key in the YAML config file under PATH_ON_DATASTORE.

  • The CSV file must be present in the specified location (local or Azure) and must contain the required columns as per REQUIRED_COLUMNS.

Fields:
field azure_blob_mount_point: str | None = None

The mount point for the Azure Blob Storage. Required if azure_data is True. Ignored if azure_data is False.

field azure_data: bool = False

Indicates if the data is stored in Azure Blob Storage.If True, requires azure_blob_mount_point to be provided by user.If False, local_data_root must be provided by user.

field local_data_root: str | None = None

The root path to data and metadata locally. The path should point to a directory that contains folders with samples of data uniquely identifiable by their ID. Only used when no azure_data argument is set.

field predefined_split_column: str | None = None

When set, the dataset factory uses this column’s values (‘train’/’val’/’test’) to assign each row to its split instead of performing a random split. split_train_size / split_val_size / split_test_size are ignored in this case.

field split_random_seed: int = 42

Random seed of the train/val/test splits.

Constraints:

  • ge = 0

field split_stratify_column: str | None = None

Column name in the metadata CSV to use forstratified splitting into train/val/test splits. Should be a column in the metadata CSV that has categorical labels for stratification.

field split_test_size: float = 0.2

Proportion of the dataset in the test split.

Constraints:

  • ge = 0.0

  • le = 1.0

field split_train_size: float = 0.6

Proportion of the dataset in the train split.

Constraints:

  • ge = 0.0

  • le = 1.0

field split_val_size: float = 0.2

Proportion of the dataset in the validation split.

Constraints:

  • ge = 0.0

  • le = 1.0

pydantic model SkiNet.ML.configs.data_configs.ph2dataset_config.ph2dataset_config.PH2DatasetConfig[source]

Bases: BaseDataConfig

Configuration for the PH2 dataset.

Attributes:

REQUIRED_COLUMNS (ClassVar[Set[str]]): Set of required columns in the metadata CSV. DATASET_KEY (ClassVar[DatasetKey]): Key for the Azure dataset. METADATA_CSV_NAME (ClassVar[str]): Name of the metadata CSV file as defined in project paths.

Example usage for a local dataset: ` cfg = PH2DatasetConfig(local_data_root="local_path/PH2Data", azure_data=False) df = cfg.metadata `

Example usage for an Azure dataset: ` cfg = PH2DatasetConfig(azure_blob_mount_point="mnt/data", azure_data=True) df = cfg.metadata `

Fields:
field kind: Literal['ph2'] = 'ph2'

Dataset kind identifier for config selection and validation.

field split_stratify_column: PH2StratificationOptions | None = PH2StratificationOptions.PH2_CLINICAL_DIAGNOSIS

Column name in the metadata CSV to use for stratified splitting into train/val/test splits. Should be a column that exists in the metadata CSV and contains categorical labels for stratification. Set to None to disable stratification. For PH2, we use PH2StratificationOptions for stratification.

pydantic model SkiNet.ML.configs.data_configs.isic2017dataset_config.isic2017dataset_config.ISIC2017DatasetConfig[source]

Bases: BaseDataConfig

Configuration for the ISIC 2017 dataset.

Attributes:

REQUIRED_COLUMNS (ClassVar[frozenset[str]]): Set of required columns in the metadata CSV. DATASET_KEY (ClassVar[DatasetKey]): Key for the Azure dataset. METADATA_CSV_NAME (ClassVar[str]): Name of the metadata CSV file as defined in project paths.

The metadata CSV is expected to have been generated by ISIC2017LocalCSVBuilder or ISIC2017AzureCSVBuilder and must contain at minimum the sampleid, datapath, datatype, melanoma, and seborrheic_keratosis columns, plus a predefined_split column that records the original ISIC 2017 challenge split for each sample.

Example usage for a local dataset: ` cfg = ISIC2017DatasetConfig(local_data_root="local_path/ISIC2017Data", azure_data=False) df = cfg.metadata `

Example usage for an Azure dataset: ` cfg = ISIC2017DatasetConfig(azure_blob_mount_point="mnt/data", azure_data=True) df = cfg.metadata `

Fields:
field kind: Literal['isic2017'] = 'isic2017'

Dataset kind identifier for config selection and validation.

field predefined_split_column: str | None = 'predefined_split'

Use the official ISIC 2017 challenge splits recorded in this column. Set to None to fall back to a random split via split_train_size/val/test.

field split_stratify_column: ISIC2017StratificationOptions | None = ISIC2017StratificationOptions.ISIC2017_MELANOMA

Column name in the metadata CSV to use for stratified splitting into train/val/test splits. Should be a column that exists in the metadata CSV and contains categorical labels for stratification. Set to None to disable stratification. For ISIC 2017, we use ISIC2017StratificationOptions for stratification.