dandi.files
¶
New in version 0.36.0.
dandi.files
defines functionality for working with local files & directories
(as opposed to remote resources on a DANDI Archive server) that are of interest
to DANDI. The classes for such files & directories all inherit from
DandiFile
, which has two immediate subclasses: DandisetMetadataFile
, for
representing dandiset.yaml
files, and LocalAsset
, for representing
files that can be uploaded as assets to DANDI Archive.
- class dandi.files.BIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]¶
Bases:
LocalFileAsset
New in version 0.46.0.
Base class for non-
dataset_description.json
assets in BIDS datasets- property bids_dataset_description: BIDSDatasetDescriptionAsset¶
The
dataset_description.json
file for the containing dataset
- bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]¶
A weak reference to the
dataset_description.json
file for the containing dataset.Users are advised to use
bids_dataset_description
to access thedataset_description.json
file instead.
- property bids_path: str¶
/
-separated path to the asset from the root of the BIDS dataset
- property bids_root: Path¶
The directory on the filesystem in which the BIDS dataset is located
- class dandi.files.BIDSDatasetDescriptionAsset(filepath: Path, dandiset_path: Path | None, path: str, dataset_files: list[BIDSAsset] = <factory>, _dataset_errors: list[ValidationResult] | None = None, _asset_errors: dict[str, list[ValidationResult]] | None = None, _asset_metadata: dict[str, BareAsset] | None = None, _bids_version: str | None = None)[source]¶
Bases:
LocalFileAsset
New in version 0.46.0.
The
dataset_description.json
file for a BIDS dataset, used to perform operations on the dataset as a whole- property bids_root: Path¶
The directory on the filesystem in which the BIDS dataset is located
- class dandi.files.DandiFile(filepath: Path, dandiset_path: Path | None)[source]¶
Bases:
ABC
Abstract base class for local files & directories of interest to DANDI
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) CommonModel [source]¶
Return the Dandi metadata for the file
- abstract get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Attempt to validate the file and return a list of errors encountered
- property modified: datetime¶
The time at which the file was last modified
- property size: int¶
The size of the file
- class dandi.files.DandisetMetadataFile(filepath: Path, dandiset_path: Path | None)[source]¶
Bases:
DandiFile
Representation of a
dandiset.yaml
file- as_readable() LocalReadableFile [source]¶
New in version 0.50.0.
Returns a
Readable
instance wrapping the local file
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- class dandi.files.GenericAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalFileAsset
Representation of a generic regular file, one that is not of any known type
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.GenericBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]¶
Bases:
BIDSAsset
,GenericAsset
New in version 0.46.0.
An asset in a BIDS dataset that is not an NWB file, a Zarr directory, or a
dataset_description.json
file. Note that, unlike the non-BIDS classes, this includes video files.- bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]¶
A weak reference to the
dataset_description.json
file for the containing dataset.Users are advised to use
bids_dataset_description
to access thedataset_description.json
file instead.
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Attempt to validate the file and return a list of errors encountered
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.LocalAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
DandiFile
Representation of a file or directory that can be uploaded to a DANDI Archive as an asset of a Dandiset
- abstract get_digest() Digest [source]¶
Calculate a DANDI etag digest for the asset using the appropriate algorithm for its type
- abstract get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Attempt to validate the file and return a list of errors encountered
- abstract iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) Iterator[dict] [source]¶
Upload the asset with the given metadata to the given Dandiset, returning a generator of status
dict
s.- Parameters:
dandiset (RemoteDandiset) – the Dandiset to which the asset will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s
path
attribute if no such field is already present.jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset
- Returns:
A generator of
dict
s containing at least a"status"
key. Upon successful upload, the lastdict
will have a status of"done"
and an"asset"
key containing the resultingRemoteAsset
.
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) RemoteAsset [source]¶
Upload the file as an asset with the given metadata to the given Dandiset and return the resulting asset. Blocks until the upload is complete.
- Parameters:
dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s
path
attribute if no such field is already present.jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset
- Return type:
- class dandi.files.LocalDirectoryAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalAsset
,Generic
[P
]Representation of a directory that can be uploaded to a DANDI Archive as a single asset of a Dandiset. It is generic in
P
, bound todandi.misctypes.BasePath
.- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- abstract property filetree: P¶
The path object for the root of the hierarchy of files within the directory
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- property size: int¶
The total size of the files in the directory
- class dandi.files.LocalFileAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalAsset
Representation of a regular file that can be uploaded to a DANDI Archive as an asset of a Dandiset
- as_readable() LocalReadableFile [source]¶
New in version 0.50.0.
Returns a
Readable
instance wrapping the local file
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) Iterator[dict] [source]¶
Upload the file as an asset with the given metadata to the given Dandiset, returning a generator of status
dict
s.- Parameters:
dandiset (RemoteDandiset) – the Dandiset to which the file will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s
path
attribute if no such field is already present.jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset
- Returns:
A generator of
dict
s containing at least a"status"
key. Upon successful upload, the lastdict
will have a status of"done"
and an"asset"
key containing the resultingRemoteAsset
.
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.LocalZarrEntry(parts: tuple[str, ...], filepath: Path, zarr_basepath: Path)[source]¶
Bases:
BasePath
A file or directory within a
ZarrAsset
- filepath: Path¶
The path to the actual file or directory on disk
- get_digest() Digest [source]¶
Calculate the DANDI etag digest for the entry. If the entry is a directory, the algorithm will be the Dandi Zarr checksum algorithm; if it is a file, it will be MD5.
- iterdir() Iterator[LocalZarrEntry] [source]¶
Returns a generator of the paths under the instance, which must be a directory
- property modified: datetime¶
The time at which the entry was last modified
- property parent: LocalZarrEntry¶
The parent path of the object. The parent of the root of a path hierarchy is itself.
- property size: int¶
The size of the entry. For a directory, this is the total size of all entries within it.
- zarr_basepath: Path¶
The path to the root of the Zarr file tree
- class dandi.files.NWBAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalFileAsset
Representation of a local NWB file
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Validate NWB asset
If
schema_version
was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.NWBBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]¶
-
New in version 0.46.0.
An NWB file in a BIDS dataset
- bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset]¶
A weak reference to the
dataset_description.json
file for the containing dataset.Users are advised to use
bids_dataset_description
to access thedataset_description.json
file instead.
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Validate NWB asset
If
schema_version
was provided, we only validate basic metadata, and completely skip validation using nwbinspector.inspect_nwbfile
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.VideoAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalFileAsset
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.ZarrAsset(filepath: Path, dandiset_path: Path | None, path: str)[source]¶
Bases:
LocalDirectoryAsset
[LocalZarrEntry
]Representation of a local Zarr directory
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- property filetree: LocalZarrEntry¶
The
LocalZarrEntry
for the root of the hierarchy of files within the Zarr asset
- get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Attempt to validate the file and return a list of errors encountered
- iter_upload(dandiset: RemoteDandiset, metadata: dict[str, Any], jobs: int | None = None, replacing: RemoteAsset | None = None) Iterator[dict] [source]¶
Upload the Zarr directory as an asset with the given metadata to the given Dandiset, returning a generator of status
dict
s.- Parameters:
dandiset (RemoteDandiset) – the Dandiset to which the Zarr will be uploaded
metadata (dict) – Metadata for the uploaded asset. The “path” field will be set to the value of the instance’s
path
attribute if no such field is already present.jobs (int) – Number of threads to use for uploading; defaults to 5
replacing (RemoteAsset) – If set, replace the given asset, which must have the same path as the new asset; if the old asset is a Zarr, the Zarr will be updated & reused for the new asset
- Returns:
A generator of
dict
s containing at least a"status"
key. Upon successful upload, the lastdict
will have a status of"done"
and an"asset"
key containing the resultingRemoteAsset
.
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.ZarrBIDSAsset(filepath: Path, dandiset_path: Path | None, path: str, bids_dataset_description_ref: weakref.ref[BIDSDatasetDescriptionAsset])[source]¶
-
New in version 0.46.0.
A Zarr directory in a BIDS dataset
- dandiset_path: Path | None¶
The path to the root of the Dandiset, if there is one
- filepath: Path¶
The path to the actual file or directory on disk
- get_metadata(digest: Digest | None = None, ignore_errors: bool = True) BareAsset [source]¶
Return the Dandi metadata for the asset
- get_validation_errors(schema_version: str | None = None, devel_debug: bool = False) list[ValidationResult] [source]¶
Attempt to validate the file and return a list of errors encountered
- path: str¶
The forward-slash-separated path to the asset within its local Dandiset (i.e., relative to the Dandiset’s root)
- class dandi.files.ZarrStat(size: int, digest: Digest, files: list[LocalZarrEntry])[source]¶
Bases:
object
Details about a Zarr asset
- files: list[LocalZarrEntry]¶
A list of all files in the asset in unspecified order
- size: int¶
The total size of the asset
- dandi.files.dandi_file(filepath: str | Path, dandiset_path: str | Path | None = None, bids_dataset_description: BIDSDatasetDescriptionAsset | None = None) DandiFile [source]¶
Return a
DandiFile
instance of the appropriate type for the file atfilepath
inside the Dandiset rooted atdandiset_path
. Ifdandiset_path
is not set, it will default tofilepath
’s parent directory.If
bids_dataset_description
is set, the file will be assumed to lie within the BIDS dataset with the givendataset_description.json
file at its root, resulting in aBIDSAsset
.If
filepath
is a directory, it must be of a type represented by aLocalDirectoryAsset
subclass; otherwise, anUnknownAssetError
exception will be raised.A regular file named
dandiset.yaml
will only be represented by aDandisetMetadataFile
instance if it is at the root of the Dandiset.A regular file that is not of a known type will be represented by a
GenericAsset
instance.
- dandi.files.find_bids_dataset_description(dirpath: str | Path, dandiset_path: str | Path | None = None) BIDSDatasetDescriptionAsset | None [source]¶
New in version 0.46.0.
Look for the topmost
dataset_description.json
file in the directorydirpath
and each of its parents, stopping when adandiset.yaml
file is found ordandiset_path
is reached.
- dandi.files.find_dandi_files(*paths: str | Path, dandiset_path: str | Path | None = None, allow_all: bool = False, include_metadata: bool = False) Iterator[DandiFile] [source]¶
Yield all DANDI files at or under the paths in
paths
(which may be either files or directories). Files & directories whose names start with a period are ignored. Directories are only included in the return value if they are of a type represented by aLocalDirectoryAsset
subclass, in which case they are not recursed into.- Parameters:
dandiset_path – The path to the root of the Dandiset in which the paths are located. All paths in
paths
must be equal to or subpaths ofdandiset_path
. IfNone
, then the Dandiset path for each asset found is implicitly set to the parent directory.allow_all – If true, unrecognized assets and the Dandiset’s
dandiset.yaml
file are returned asGenericAsset
andDandisetMetadataFile
instances, respectively. If false, they are not returned at all.include_metadata – If true, the Dandiset’s
dandiset.yaml
file is returned as aDandisetMetadataFile
instance. If false, it is not returned at all (unlessallow_all
is true).