q04/data.trr379.de
q04/data.trr379.de
3
0
Fork 0
Code Issues 4 Pull requests Projects Releases Packages Wiki Activity Actions

Catalog chokes on dataset-id containing forward-slash #3

New issue
Closed
opened 2025-11-17 22:08:48 +00:00 by jsheunis · 3 comments
jsheunis commented 2025-11-17 22:08:48 +00:00
Member
Copy link

BTW, uv is awesome, and this buildpage script.

Got some "impossible" results locally when building the catalog:

>> uv run buildpage.py ctlg/

Using CPython 3.13.3 interpreter at: /usr/local/opt/python@3.13/bin/python3.13
Creating virtual environment at: .venv
      Built jsmin==3.0.1
Installed 80 packages in 501ms

catalog_create(ok): ctlg [Catalog successfully created at: ctlg]
catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=698cec3c-e424-47e0-9da4-7a75e5ed6078, dataset_version=any.]
catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=698cec3c-e424-47e0-9da4-7a75e5ed6078, dataset_version=any)]
catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=https://trr379.de/datasets/q01, dataset_version=any.]
catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=https://trr379.de/datasets/q01, dataset_version=any)]
catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=main, dataset_version=latest.]
catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=main, dataset_version=latest)]
catalog_set(ok): ctlg [Home page successfully set to: dataset_id=main, dataset_version=latest.]

Then, looking at the metadata in the catalog after the build:

>> tree metadata

metadata
├── 698cec3c-e424-47e0-9da4-7a75e5ed6078
│   └── any
│       └── e53
│           └── 5ef26804b8ac63fe4b5339b01f430.json
├── https:
│   └── trr379.de
│       └── datasets
│           └── q01
│               └── any
│                   └── b36
│                       └── d1556aa16c634344c03bb39a386f7.json
├── main
│   └── latest
│       └── 9ad
│           └── e37008f9ec0e693ebb2791e00a1b7.json
└── super.json

So some step in the metadata creation process takes the dataset id https://trr379.de/datasets/q01 and then incorrectly creates a hierarchy of directories matching that "path". This needs to be fixed.

BTW, `uv` is awesome, and this buildpage script. Got some "impossible" results locally when building the catalog: ``` >> uv run buildpage.py ctlg/ Using CPython 3.13.3 interpreter at: /usr/local/opt/python@3.13/bin/python3.13 Creating virtual environment at: .venv Built jsmin==3.0.1 Installed 80 packages in 501ms catalog_create(ok): ctlg [Catalog successfully created at: ctlg] catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=698cec3c-e424-47e0-9da4-7a75e5ed6078, dataset_version=any.] catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=698cec3c-e424-47e0-9da4-7a75e5ed6078, dataset_version=any)] catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=https://trr379.de/datasets/q01, dataset_version=any.] catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=https://trr379.de/datasets/q01, dataset_version=any)] catalog_remove(impossible): ctlg [No dataset record found in catalog for: dataset_id=main, dataset_version=latest.] catalog_add(ok): ctlg [Metadata record successfully added to catalog (dataset: dataset_id=main, dataset_version=latest)] catalog_set(ok): ctlg [Home page successfully set to: dataset_id=main, dataset_version=latest.] ``` Then, looking at the metadata in the catalog after the build: ``` >> tree metadata metadata ├── 698cec3c-e424-47e0-9da4-7a75e5ed6078 │   └── any │   └── e53 │   └── 5ef26804b8ac63fe4b5339b01f430.json ├── https: │   └── trr379.de │   └── datasets │   └── q01 │   └── any │   └── b36 │   └── d1556aa16c634344c03bb39a386f7.json ├── main │   └── latest │   └── 9ad │   └── e37008f9ec0e693ebb2791e00a1b7.json └── super.json ``` So some step in the metadata creation process takes the dataset id `https://trr379.de/datasets/q01` and then incorrectly creates a hierarchy of directories matching that "path". This needs to be fixed.
jsheunis commented 2025-11-17 22:13:20 +00:00
Author
Member
Copy link

Well, on inspecting the catalog, it seems like some functionality at least works as it should. The dataset id is URL encoded when accessing it in the served catalog, and this leads to the correct information being rendered. There might still be other functionality that doesn't work (like the catalog_remove calls), but it needs to be explored further to figure out the implications

Well, on inspecting the catalog, it seems like some functionality at least works as it should. The dataset id is URL encoded when accessing it in the served catalog, and this leads to the correct information being rendered. There might still be other functionality that doesn't work (like the `catalog_remove` calls), but it needs to be explored further to figure out the implications
msz commented 2025-11-18 11:39:11 +00:00
Collaborator
Copy link

Thanks for the observation. In the pool we have the PID to work with first and foremost, we are not guaranteed to have any DataLad dataset distributions (dataset IDs). The catalog pages worked all right for me, so I didn't investigate, but in the end this is a "lazy" choice for determining the dataset ID: if the PID is https://pid.datalad.org/datasets/<uuid>... then take the UUID from there, otherwise use the PID as-is. This probably should be urlencoded or (even better) there should be another special treatment for the trr379.de/datasets namespace.

Thanks for the observation. In the pool we have the PID to work with first and foremost, we are not guaranteed to have any DataLad dataset distributions (dataset IDs). The catalog pages worked all right for me, so I didn't investigate, but in the end this is a "lazy" choice for determining the dataset ID: if the PID is `https://pid.datalad.org/datasets/<uuid>...` then take the UUID from there, otherwise use the PID as-is. This probably should be urlencoded or (even better) there should be another special treatment for the `trr379.de/datasets` namespace.
msz commented 2025-11-21 14:21:56 +00:00
Collaborator
Copy link

For PIDs like https://trr379.de/datasets/q01, I'm thinking of either:

  • removing http://, replacing . -> - & / -> _: produces trr379-de_datasets_qo1
  • using UUIDv5(nsURL, "https://trr379.de/datasets/q01"): produces 83b98d3e-9a39-5b4f-adea-a3bee7276640
  • a two-step UUID5(UUID5(nsURL, "https://trr379.de"), "datasets/q01"): produces d657b52b-a6e2-5424-9c21-ffe23c3b482a

Personally I am not a fan of squeezing everything into UUID format, considering that the dataset ID can be any string, but it would make the dataset IDs more uniform 🤔

For PIDs like `https://trr379.de/datasets/q01`, I'm thinking of either: - removing `http://`, replacing `. -> -` & `/ -> _`: produces `trr379-de_datasets_qo1` - using `UUIDv5(nsURL, "https://trr379.de/datasets/q01")`: produces `83b98d3e-9a39-5b4f-adea-a3bee7276640` - a two-step `UUID5(UUID5(nsURL, "https://trr379.de"), "datasets/q01")`: produces `d657b52b-a6e2-5424-9c21-ffe23c3b482a` Personally I am not a fan of squeezing everything into UUID format, considering that the dataset ID can be any string, but it would make the dataset IDs more uniform 🤔
msz closed this issue 2025-11-21 16:15:49 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
q04/data.trr379.de#3
No description provided.