title: API — datasets, manifests and sources description: pardesline.data: load, scan, Dataset, Sample, Manifest, Layout, License, the source schemes.

Data¶

pardesline.data ¶

pl.load: one call for a file, a folder, a bundled dataset or a hub repo.

pl.load("scan.ply") -> Object pl.load("scan.bin", layout={...}) -> PointCloud read through its layout pl.load("rire_brain") -> Dataset (bundled manifest, files fetched on access) pl.load("sample://popi_lung_4dct") -> same pl.load("hf://org/name") -> Dataset from the bundled manifest of that repo, or a draft with questions (like a private folder) pl.load("hf://org/name/sub/folder") -> same, listing only that folder of the repo pl.load("s3://openneuro.org/ds000228") -> a public bucket prefix, same flow pl.load("tcia://Pancreas-CT") -> a public TCIA collection (DICOM series), same flow pl.load("my_dataset.manifest.json") -> Dataset from your own manifest pl.load("/data/study_A") -> Dataset after pl.scan; raises NeedsConfirmation until licence/units/labels are answered

load ¶

load(ref: str | PathLike, *, manifest: str | PathLike | None = None, assume: bool = False, interactive: bool = False, save_draft: bool = True, layout: Layout | dict[str, Any] | None = None, kind: str | None = None, **answers: Any) -> Object | Dataset

Open a file as an :class:Object or a dataset as a :class:Dataset.

A raw array file (.npy, .npz, .bin) needs layout= (dtype, shape, columns / key) and takes kind= ("points", "volume", "labels"); with a layout and no kind, a (N,k) shape is points and a (a,b,c) shape is a grid.

For a folder without a confirmed manifest - or a Hub repo without a bundled one - pl.scan drafts a manifest. Then: strict (default) raises :class:NeedsConfirmation listing the open questions and saves the draft; assume=True loads with unknowns left unknown (and warns); interactive=True asks in the terminal; keyword answers (license=, units=, frame=, labels=, pairing=, types=, layout=) fill the draft directly.

scan_hf ¶

scan_hf(ref: str, *, max_files: int = 20000) -> Manifest

Draft a manifest for a Hub repo (or one of its folders) from its file listing: pl.scan("hf://org/name/sub/folder"). Needs [hf].

scan_uri ¶

scan_uri(text: str, *, max_files: int = 20000) -> Manifest

Draft a manifest for an s3:// prefix or a tcia:// collection from its listing.

scan_folder ¶

scan_folder(folder: str | Path, *, name: str | None = None, read_headers: bool = True) -> Manifest

Draft a manifest for folder. The result carries questions for every field that needs a human; pl.load(folder, ...) consumes it.

datasets ¶

datasets(*, tag: str | None = None, scheme: str | None = None) -> list[Manifest]

The manifests bundled with the package (pl.load(name) works on each). tag="medical" or scheme="hf" filter the list.

parse_hf_ref ¶

parse_hf_ref(text: str) -> tuple[str, str | None]

hf://org/name/sub/folder -> ("org/name", "sub/folder").

cache_dir ¶

cache_dir() -> Path

The root folder where datasets are cached.

PARDESLINE_CACHE or :func:set_cache_dir override the default ~/.cache/pardesline. The folder is created on first use.

Examples:

>>> import pardesline as pl
>>> pl.cache_dir().name
'pardesline'

set_cache_dir ¶

set_cache_dir(path: str | PathLike | None) -> Path

Point the cache somewhere else (None restores the default).

pardesline.data.dataset.Dataset ¶

Dataset(manifest: Manifest, source: Source, *, samples: list[SampleSpec] | None = None)

A manifest bound to its source. Indexing is lazy: nothing is fetched until a role of a sample is accessed.

filter ¶

filter(**fields: str) -> Dataset

Samples whose fields match, e.g. ds.filter(phase="00").

split ¶

split(by: str | None = None, fractions: tuple[float, ...] = (0.8, 0.2), seed: int = 0) -> tuple[Dataset, ...]

Leakage-free split: all samples sharing the same by field (default the manifest's pairing key, else the sample id) land in the same part.

fetch_all ¶

fetch_all() -> int

Download every file now. Returns the number of files.

export ¶

export(folder: str | Path, *, overwrite: bool = False) -> Manifest

Write every sample of this (sub)dataset as standard files in folder - raw arrays become PLY or MetaImage through their layout, DICOM series become one .mha - each with its .pardesline.json sidecar (frame, units, labels, licence, provenance), plus a local pardesline.manifest.json so pl.load(folder) reopens the copy without questions. Returns that manifest.

pardesline.data.dataset.Sample ¶

Sample(dataset: Dataset, spec: SampleSpec)

One unit of the dataset (a patient, a scan, a frame). Roles are attributes: sample.ct, sample.mask. Files are fetched on access.

path ¶

path(role: str) -> Path

Fetch (if needed) and return the local path of a role. A folder entry (DICOM series) fetches every file it holds; a MetaImage header brings its .raw / .zraw sibling along.

pardesline.data.manifest.Manifest ¶

Bases: _Model

pardesline.data.manifest.Role ¶

Bases: _Model

needs_layout ¶

needs_layout(relpath: str) -> bool

True when relpath is a raw array and this role has no layout.

pardesline.data.manifest.Layout ¶

Bases: _Model

How to read a raw array file (.npy, .npz, .bin) as data.

Points: shape="(N,4)" with columns=["x","y","z","reflectance"]. Grids: shape="(L,W,H)" (voxel values) with optional spacing and origin in the role's units. dtype is required for headerless .bin / .raw files; key picks an array in an .npz.

dims ¶

dims() -> list[int | None]

"(N,4)" -> [None, 4]; letters mean 'whatever the file holds'.

pardesline.data.manifest.LicenseSpec ¶

Bases: _Model

pardesline.data.manifest.SourceSpec ¶

Bases: _Model

Where the files come from.

local: root folder (absolute, or relative to the manifest file).
https: files with checksums, optional mirrors.
hf: repo_id on the Hugging Face Hub; paths restricts the listing to sub-folders (a 100k-file repo is never listed whole); max_files caps what one listing may return.
open3d: a class of open3d.data.
archive: one remote url (or several urls that form one dataset) of zip or uncompressed tar read with HTTP Range requests: the index costs a few KB, a member costs its own size, the archive is never downloaded whole when the server supports ranges. prefix restricts the listing to members under it.
s3: a public bucket listed with ListObjectsV2 (bucket, endpoint, prefix); relative paths are the keys under prefix.
tcia: a public collection of The Cancer Imaging Archive through the NBIA REST API (collection, optional modalities); one entry per DICOM series as <PatientID>/<Modality>/<SeriesInstanceUID>/.

key ¶

key() -> str

What identifies this source for pl.load lookups: the Hub repo, the bucket and prefix, the TCIA collection, the archive url(s).

pardesline.data.sources ¶

Sources: where a manifest's files come from and how one is fetched.

Every source answers two questions: fetch(relpath) -> local Path (download, verify, cache) and list() -> relpaths or None when listing is possible.

open_source ¶

open_source(spec: SourceSpec, manifest: Manifest, *, manifest_dir: Path | None = None) -> Source

Instantiate the source that a manifest's source block describes.

Parameters:

Name	Type	Description	Default
`spec`	`SourceSpec`	the `SourceSpec` (scheme `local`, `https`, `hf`, `open3d`, `archive`, `s3`, `tcia` or `urllist`).	required
`manifest`	`Manifest`	the manifest, used for the dataset name and explicit samples.	required
`manifest_dir`	`Path \| None`	folder the manifest was loaded from, for relative local roots.	`None`

Returns:

Type	Description
`Source`	An object with `list()` and `fetch(relpath)` (and `fetch_folder` for
`Source`	DICOM series), caching under the dataset's cache folder.

Raises:

Type	Description
`ManifestError`	when the scheme is unknown or its required fields are missing.
`MissingDependency`	when the scheme needs an extra (`huggingface_hub`, Open3D).