Michael Hanke
f86c6b75c1
All checks were successful
Deploy on webserver / Build site with hugo and deploy on success (push) Successful in 6m47s
45 lines
2.4 KiB
Markdown
45 lines
2.4 KiB
Markdown
---
|
|
title: Archive DICOMs
|
|
---
|
|
|
|
As the "true" raw data, DICOMs are rarely (re)accessed and hardly ever need to change.
|
|
However, they need to be stored somewhere.
|
|
Tracking DICOMs in DataLad datasets allows dependency tracking for conversion to NIfTI.
|
|
However, it's good to keep DataLad optional (allow DataLad and non-DataLad access).
|
|
|
|
## Historical precedent: ICF at FZJ
|
|
|
|
The following solution has been proposed for the Imaging Core Facility at FZJ:
|
|
|
|
- DICOMs are packed into tar files (tarballs) [^1]
|
|
- the tarballs are placed on a web server (intranet only), organized
|
|
by project (HTTP Basic Authentication for access management)
|
|
- DataLad datasets record availability via
|
|
[archivist](https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.annexremotes.archivist.html)
|
|
and
|
|
[uncurl](https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.annexremotes.uncurl.html)
|
|
special remotes, which translates to:
|
|
- a file is available from a tarball (archivist special remote)
|
|
- a tarball is available from a given URL, pointing to the web server (uncurl special remote)[^2].
|
|
- Only the Git repository (no annex) is stored by the consuming institute;
|
|
the ICF web server is the permanent DICOM storage.
|
|
|
|
The system has been documented in https://inm-icf-utilities.readthedocs.io/en/latest/
|
|
and the tarball & dataset generation tools implementation is in https://github.com/psychoinformatics-de/inm-icf-utilities.
|
|
|
|
[^1]: timestamps are normalized to ensure re-packing the same data does not change tarball checksums
|
|
[^2]: uncurl is chosen because it allows re-writing patterns with just configuration, e.g., should the base URL change
|
|
|
|
## TRR reimplementation
|
|
|
|
One of the TRR sites indicated intent to use a Forgejo instance for DICOM storage.
|
|
A particular challenge for the underlying system was [inode](https://en.wikipedia.org/wiki/Inode) limitation.
|
|
For this reason, an adaptation of the ICF system has been proposed:
|
|
|
|
- a dataset is generated upfront, and DICOM tarball is stored with the dataset in Forgejo (as annex)
|
|
- we keep using the archivist remote (file in tarball) to avoid using up thousands of inodes for individual files
|
|
(Git can pack its repository into several files, so we only add one more for the tarball).
|
|
|
|
A proof of principle for dataset generation (using re-written ICF
|
|
code) has been proposed in https://hub.trr379.de/q02/dicom-utilities. See the README for more
|
|
detailed explanations (and commit messages for even more detail).
|