Philip Durbin 🚀 (Aug 18 2025 at 16:02): Philip Durbin 🚀 (Aug 18 2025 at 16:02):test_native_upload.py is failing with a no-op PR: https://github.com/gdcc/python-dvuploader/pull/37
Why? :thinking:
Philip Durbin 🚀 (Aug 18 2025 at 16:09):I wonder if it's related to this:
allow for tag replacement #11359
Jan Range (Aug 18 2025 at 18:41):I am also puzzled, it is working fine locally with the latest 6.7 release, but it fails in the CI.
Philip Durbin 🚀 (Aug 18 2025 at 18:42):I'm trying to run it locally. Need poetry set up first: #python > poetry not working for python-dvuploader
Jan Range (Aug 18 2025 at 18:42):It seems like either the directoryLabel is not passed correctly or something is failing on the DV-side
Jan Range (Aug 18 2025 at 18:42):You can use pip install -e . as well
Philip Durbin 🚀 (Aug 18 2025 at 18:49):From within a venv, right? That is, I'll run python -m venv venv first, right?
Jan Range (Aug 18 2025 at 18:57):Yes, you can use the venv in that case
Philip Durbin 🚀 (Aug 18 2025 at 20:56):Hmm, I ranpip install -e . but I'm getting this:
% python -m pytest -v
/Users/PDurbin/github/gdcc/python-dvuploader/venv/bin/python: No module named pytest
So I ran pip install pytest. But now I'm getting this:
% python -m pytest -v
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --cov=dvuploader
inifile: /Users/PDurbin/github/gdcc/python-dvuploader/pyproject.toml
rootdir: /Users/PDurbin/github/gdcc/python-dvuploader
Sorry, since you are not using poetry you need to install the test-dependencies first:
pip install pytest-cov pytest-asyncio pytest-httpx
Will add this to the Readme to prevent confusion :slight_smile:
Philip Durbin 🚀 (Aug 20 2025 at 13:11):Thanks, I can run python -m pytest -v now.
Philip Durbin 🚀 (Aug 20 2025 at 13:21):I can replicate the error:
===================================== short test summary info =====================================
FAILED tests/integration/test_native_upload.py::TestNativeUpload::test_native_upload_by_handler - AssertionError: Directory label does not match for file {"description": "", "label": "file.txt", "restricted": false, "version": 1, "datasetVersionId": 4, "categories": ["DATA"], "dataFile": {"id": 16, "persistentId": "", "filename": "file.txt", "contentType": "text/plain", "friendlyType": "Plain Text", "filesize": 13, "description": "", "categories": ["DATA"], "storageIdentifier": "local://198c79fc9ca-f998aa3fe2fa", "rootDataFileId": -1, "md5": "65a8e27d8879283831b664bd8b7f0ad4", "checksum": {"type": "MD5", "value": "65a8e27d8879283831b664bd8b7f0ad4"}, "tabularData": false, "creationDate": "2025-08-20", "lastUpdateTime": "2025-08-20T13:16:35Z", "fileAccessRequest": false}}
assert '' == 'subdir'
- subdir
====================== 1 failed, 33 passed, 13 warnings in 107.59s (0:01:47) ======================
I created an issue: https://github.com/gdcc/python-dvuploader/issues/39
Philip Durbin 🚀 (Aug 20 2025 at 13:32):@Jan Range what if you delete your local "unstable" image and re-download it? Can you reproduce the problem then?
Jan Range (Aug 20 2025 at 13:56):Good idea! Will try it after our meeting :slight_smile:
Jan Range (Aug 21 2025 at 12:12):I was able to reproduce the error :slight_smile: Will give it a deeper dive asap
Philip Durbin 🚀 (Aug 21 2025 at 13:02):Nice! It's only a matter of time now!
Philip Durbin 🚀 (Aug 21 2025 at 13:03):My money is on something that changed on the backend. :grimacing:
But we'll see, I guess!
Philip Durbin 🚀 (Sep 18 2025 at 20:04):Philip Durbin 🚀 said:
I'm trying to run it locally. Need poetry set up first: #python > poetry not working for python-dvuploader
This part is resolved.
Philip Durbin 🚀 (Sep 18 2025 at 20:04):export $(grep "API_TOKEN" "dv/bootstrap.exposed.env") doesn't work for me. From https://github.com/gdcc/python-dvuploader#running-tests-locally
Philip Durbin 🚀 (Sep 18 2025 at 20:05):Nothing there:
pdurbin@beamish python-dvuploader % file dv/bootstrap.exposed.env
dv/bootstrap.exposed.env: directory
pdurbin@beamish python-dvuploader % ls -al dv/bootstrap.exposed.env
total 0
drwxr-xr-x 2 pdurbin staff 64 Sep 18 16:01 .
drwxr-xr-x 4 pdurbin staff 128 Sep 18 16:01 ..
pdurbin@beamish python-dvuploader %
Am I missing something? :thinking:
Jan Range (Sep 18 2025 at 20:08):The Docker compose should actually expose the .env from which you can grab the token. Did this change?
Philip Durbin 🚀 (Sep 18 2025 at 20:08):buh, not sure
Philip Durbin 🚀 (Sep 18 2025 at 20:08):Is there supposed to be a .env file in that directory?
Jan Range (Sep 18 2025 at 20:10):I think this is the way the GitHub Action makes it possible to grab the token. In the compose within dvuploader the corresponding directory is exposed:
volumes:
- ${PWD}/dv/data:/dv
- ${PWD}:/secrets
Maybe it has changed and is not exposed anymore. If so, the Readme should be changed to manually add the token to the env.
Philip Durbin 🚀 (Sep 18 2025 at 20:11):That's what I did last time. I manually grabbed the API token.
Jan Range (Sep 18 2025 at 20:11):I think this is more stable and straight forward
Philip Durbin 🚀 (Sep 18 2025 at 20:13):I created an issue: https://github.com/gdcc/python-dvuploader/issues/41
Philip Durbin 🚀 (Sep 18 2025 at 20:16):Here's where Oliver Bertuch added saving the targetEnvFile: https://github.com/IQSS/dataverse/pull/9935
Philip Durbin 🚀 (Sep 18 2025 at 20:18):Right, here's more of the config:
volumes:
- ${PWD}/dv/bootstrap.exposed.env:/.env
command:
- sh
- -c
- "bootstrap.sh -e /.env dev"
File /.env not found, is a directory or not writeable
I think that explains it :point_of_information:
Philip Durbin 🚀 (Sep 18 2025 at 20:20):Not sure what the fix is though :thinking:
Philip Durbin 🚀 (Sep 18 2025 at 20:23):% python3 -m pytest -v
/opt/homebrew/opt/python@3.13/bin/python3.13: No module named pytest
Should poetry have installed pytest for me? :thinking:
Jan Range (Sep 18 2025 at 20:24):Did you run the installation with poetry install --with test?
Philip Durbin 🚀 (Sep 18 2025 at 20:24):yep!
Jan Range (Sep 18 2025 at 20:26):Okay, I guess the terminal python is not using the poetry env. In the CI I am using poetry run pytest, which should use the correct environment.
Philip Durbin 🚀 (Sep 18 2025 at 20:27):yes! working! do you want a PR?
Jan Range (Sep 18 2025 at 20:27):That would be awesome! 
Philip Durbin 🚀 (Sep 18 2025 at 20:28):here you go: https://github.com/gdcc/python-dvuploader/pull/42
Jan Range (Sep 18 2025 at 20:29):Thanks :slight_smile:
Philip Durbin 🚀 (Sep 18 2025 at 20:30):I can reproduce the failure: fail.txt
Philip Durbin 🚀 (Sep 18 2025 at 20:30):I think this is old news, though. You were able to reproduce it too, if memory serves.
Jan Range (Sep 18 2025 at 20:30):That is good news
Philip Durbin 🚀 (Sep 18 2025 at 20:31):And so was I, a while ago: https://github.com/gdcc/python-dvuploader/issues/39
I just wasn't using poetry back then.
Jan Range (Sep 18 2025 at 20:34):I was looking into this too today, but I am still puzzled what the source problem is. FYI: in the non-direct upload, files are zipped and uploaded for efficiency. When inspecting the Zip that is sent, it correctly contains the directory structure. Even so, when I upload the archive using python-dvuploader, the directory labels are fine.
I think that the last step, which updates the metadata of each file could cause the problem. Will comment that part out and see if it overwrites the directory labels in some way.
Jan Range (Sep 18 2025 at 20:36):I have also cross-checked with DVCLI, which also supports uploading entire directories as zip, but there everything is fine.
Philip Durbin 🚀 (Sep 18 2025 at 20:37):Ok, cool. We suspect something changed upstream, right? In Dataverse itself, I mean.
Jan Range (Sep 18 2025 at 20:38):It could be, havent changed that part of the code. Maybe sending None as directory label removes it, which may not happened in previous versions.
Philip Durbin 🚀 (Sep 18 2025 at 20:39):Let's see, you're using DATAVERSE_IMAGE=docker.io/gdcc/dataverse:unstable
Philip Durbin 🚀 (Sep 18 2025 at 20:39):Let's take a look at 6.8 (unreleased) at https://preview.guides.gdcc.io/en/develop/api/changelog.html
Philip Durbin 🚀 (Sep 18 2025 at 20:40): Jan Range (Sep 18 2025 at 20:40):Okay nice, that explains it!
Jan Range (Sep 18 2025 at 20:41):Will try it now
Philip Durbin 🚀 (Sep 18 2025 at 20:41):I noticed that while writing the release notes the other day (#community > Release 6.8 Timeline)
Philip Durbin 🚀 (Sep 18 2025 at 20:42):This is the 6.8 PR by the way: File Metadata Update - Empty values clear fields #11439
Jan Range (Sep 18 2025 at 20:43):Fixed it!
Philip Durbin 🚀 (Sep 18 2025 at 20:44):Nice!
Jan Range (Sep 18 2025 at 20:45):The File class directory label defaults to empty literals, causing the metadata update to clear out the directory label all the time. I have now added an explicit check to only include the directory label when it is non-empty.
Jan Range (Sep 18 2025 at 20:45):
Philip Durbin 🚀 (Sep 18 2025 at 20:45):Perfect. I suppose the PR should close this issue: https://github.com/gdcc/python-dvuploader/issues/39
Jan Range (Sep 18 2025 at 20:45):So awesome, thanks for the help!
Jan Range (Sep 18 2025 at 20:45):I will open a PR
Philip Durbin 🚀 (Sep 18 2025 at 20:45):Oh sure :smile:
Philip Durbin 🚀 (Sep 18 2025 at 20:46):Once we fix this we can get back to the contributors, who first let us know about it. :sweat_smile:
Philip Durbin 🚀 (Sep 18 2025 at 20:46):Poor things. Failing tests weren't their fault! :upside_down:
Jan Range (Sep 18 2025 at 20:46):True that, all those red x's are nerve wracking :smile:
Philip Durbin 🚀 (Sep 18 2025 at 20:47):I guess this is the main one to follow up on, from bnavigator: https://github.com/gdcc/python-dvuploader/pull/34
Philip Durbin 🚀 (Sep 18 2025 at 20:47):But I'm getting ahead of myself :smile:
Jan Range (Sep 18 2025 at 20:48):I will mention his PR in mine and notify him to merge the main branch into his PR, once the fix has been merged
Jan Range (Sep 18 2025 at 21:04):Done https://github.com/gdcc/python-dvuploader/pull/43
Philip Durbin 🚀 (Oct 10 2025 at 19:52):Great work. You've even merged his PR already. We can probably resolve this topic. :tada:
Last updated: Nov 01 2025 at 14:11 UTC