Philip Durbin 🚀 (Mar 28 2024 at 20:10): Philip Durbin 🚀 (Mar 28 2024 at 20:10):I'm getting excited about the Distribits conference next week! If you check the schedule, @Oliver Bertuch @Jan Range and I are giving a talk entitled "Distributed Metadata and Data with Dataverse".
Philip Durbin 🚀 (Mar 28 2024 at 20:10):I invited others in the community to register. We'll see. :grinning:
Philip Durbin 🚀 (Mar 28 2024 at 20:11):@Asbjørn Skødt as it turns out, my layover in in Copenhagen. :grinning:
Philip Durbin 🚀 (Mar 28 2024 at 20:45):I'm bringing three sizes of Dataverse stickers.
Asbjørn Skødt (Mar 30 2024 at 09:39):Hey Philip. :grinning: You are very welcome to contact if you have enough time in Copenhagen to meetup.
Philip Durbin 🚀 (Mar 30 2024 at 12:08):Ha, thanks! It's a tight layover, less than an hour, but hopefully I'll be back, maybe even with the family.
Philip Durbin 🚀 (Apr 02 2024 at 19:40):Heh. Bumped into Yarik and Chris at the airport.
Juan Pablo Tosca Villanueva (Apr 02 2024 at 19:56):Where are the pictures? :rolling_on_the_floor_laughing:
Philip Durbin 🚀 (Apr 02 2024 at 20:22):I thought about it
Philip Durbin 🚀 (Apr 02 2024 at 20:55):They were telling me I should do what they're doing, spend time in the layover city. They're going to spend half a day exploring Amsterdam.
Philip Durbin 🚀 (Apr 02 2024 at 21:01):We also chatted about https://aws.amazon.com/opendata/open-data-sponsorship-program/
Philip Durbin 🚀 (Apr 02 2024 at 21:01):At least I think that's the right program. Free egress from S3.
Philip Durbin 🚀 (Apr 03 2024 at 08:05):Just landed in Düsseldorf
f2eea90f-cbe9-46a2-998d-3802f7d52e1f.jpg
Oliver Bertuch (Apr 03 2024 at 08:32):Welcome to timezone CEST
Oliver Bertuch (Apr 03 2024 at 08:33):I hope you got some sleep earlier :sleeping:
Philip Durbin 🚀 (Apr 03 2024 at 08:58):A little bit. Happy they had a room ready for me at Hotel Favor
Philip Durbin 🚀 (Apr 03 2024 at 08:59):I have a view of the Rhine Tower
bb3a77cf-292b-4e84-a732-d5842cd3325c.jpg
Philip Durbin 🚀 (Apr 03 2024 at 09:57):When are you guys arriving? Do you want to have dinner?
Jan Range (Apr 03 2024 at 09:58):I will arrive later, at 10 PM, unfortunately. If you are still at the bar though, I am happy to join :grinning_face_with_smiling_eyes:
Philip Durbin 🚀 (Apr 03 2024 at 10:20):If I'm up, I'll try to find you :big_smile:
Jan Range (Apr 03 2024 at 10:20):Awesome! How's the hotel? Booked it too :smile:
Philip Durbin 🚀 (Apr 03 2024 at 10:50):The hotel is good. Surprisingly, the mini bar is included. The gym is no longer across the street. It's an 8 minute walk. Looking forward to an easy breakfast
Juan Pablo Tosca Villanueva (Apr 03 2024 at 12:51):finally some pictures!
Jan Range (Apr 03 2024 at 13:52):Philip Durbin schrieb:
The hotel is good. Surprisingly, the mini bar is included. The gym is no longer across the street. It's an 8 minute walk. Looking forward to an easy breakfast
Hehe the minibar is good news :grinning_face_with_smiling_eyes:
Philip Durbin 🚀 (Apr 03 2024 at 13:53):Oh and free umbrellas to borrow. It's raining.
Jan Range (Apr 03 2024 at 13:53):Yea, german weather is not nice at the moment :frown:
Philip Durbin 🚀 (Apr 03 2024 at 13:54):Warmer than Boston!
Jan Range (Apr 03 2024 at 13:54):Surely some Kölsch will help to cope with it
Jan Range (Apr 03 2024 at 13:54):Oh okay, didn't expect it to be warmer
Philip Durbin 🚀 (Apr 03 2024 at 13:54):I saw lots of people standing at tables under umbrellas drinking. :grinning:
Jan Range (Apr 03 2024 at 13:55):The stereotype holds true :grinning:
Juan Pablo Tosca Villanueva (Apr 03 2024 at 14:00):Philip Durbin said:
Warmer than Boston!
AFF6DD3F-3F11-43DB-8B2E-0F8DC25F6F7F.jpg
Juan Pablo Tosca Villanueva (Apr 03 2024 at 14:00)::upside_down:
Philip Durbin 🚀 (Apr 03 2024 at 16:02):Phew! Ok, I think I'm done futzing with my slides. Time for dinner. Will practice later.
Oliver Bertuch (Apr 03 2024 at 16:49):Jan Range said:
Surely some Kölsch will help to cope with it
Don't order Kölsch in Düsseldorf!!!
Oliver Bertuch (Apr 03 2024 at 16:50):They might throw stones or empty mugs
Jan Range (Apr 03 2024 at 16:50):True, I forgot! Alt is the right one :grinning:
Philip Durbin 🚀 (Apr 03 2024 at 19:22):It's fun watching but Leverkusen is killing us
IMG_20240403_212103673.jpg
Philip Durbin 🚀 (Apr 03 2024 at 20:09):Don't worry, it was all Alt
Philip Durbin 🚀 (Apr 04 2024 at 07:03):Livestream of Distribits day 1: https://youtube.com/live/BwRy3z_hQ70?feature=share
Philip Durbin 🚀 (Apr 04 2024 at 07:04):55049c93-f9f7-408e-a3d0-b2a3069a5188.jpg
Philip Durbin 🚀 (Apr 04 2024 at 07:52):This tool sounded interesting: https://pypi.org/project/tinuous/
Philip Durbin 🚀 (Apr 04 2024 at 07:55):git-annex organelles
a85d6fc7-ec62-4f92-a3e4-1d72d1ca20ca.jpg
Oliver Bertuch (Apr 04 2024 at 08:05):Is Dataverse a good candidate to become a DataLad / Git-Annex Proxy? Question popping up in my head from the git-annex talk by Joey. (Or should Dataverse be able to talk to such proxies to ingest data / access data on tapes / ...?)
Jan Range (Apr 04 2024 at 08:11):I think the latter case would be very useful
Philip Durbin 🚀 (Apr 04 2024 at 08:13):Well, that proxy thing is just an idea for now, but sure, let's pick Joey's brain while we're here. :grinning:
Philip Durbin 🚀 (Apr 04 2024 at 08:16):413a45ec-1077-4b6f-846f-3045d4364eae.jpg
Oliver Bertuch (Apr 04 2024 at 08:24):Dataverse hosting a git repository using datalad-annex. Fancy! (Though slow - but probably good enough for a publication that keeps history / provenance)
Philip Durbin 🚀 (Apr 04 2024 at 08:25):Yep! Here's a pic:
47355de7-a58a-4f14-b5d2-0b1a5769226a.jpg
Philip Durbin 🚀 (Apr 04 2024 at 08:26):I will admit I was having trouble with it yesterday: https://github.com/datalad/datalad-dataverse/issues/302
Philip Durbin 🚀 (Apr 04 2024 at 08:28):03cc47fe-8fa3-40ee-9538-a115d97c1842.jpg
Philip Durbin 🚀 (Apr 04 2024 at 08:30):Schema for a generic data distribution record - https://concepts.datalad.org/s/distribution/unreleased/
Philip Durbin 🚀 (Apr 04 2024 at 08:31):73a2e853-a0ec-4914-81b6-8168a2730d96.jpg
Philip Durbin 🚀 (Apr 04 2024 at 08:34):196a9708-08a8-4f49-b485-89e94b13875d.jpg
Oliver Bertuch (Apr 04 2024 at 08:35):Sounds like Dataverse has some work cut out here: we might need to enable better per-file metadata...
Oliver Bertuch (Apr 04 2024 at 08:36):(Which also has been a request independent from DataLad since at least 2020, where I first heard it at Tromso)
Philip Durbin 🚀 (Apr 04 2024 at 08:46):Yeah. We have auxiliary files, at least!
Philip Durbin 🚀 (Apr 04 2024 at 09:50):Mmm, pull requests for datasets. :yum:
Philip Durbin 🚀 (Apr 04 2024 at 09:54):What would that look like, I wonder. I'll have to ask.
Philip Durbin 🚀 (Apr 04 2024 at 09:58):Huh, I hadn't heard of Fast Data Transfer - http://monalisa.cern.ch/FDT/ - but the speaker just said rsync is faster. :thinking:
Philip Durbin 🚀 (Apr 04 2024 at 12:16):a5d0d056-e2c6-4b58-9ceb-a0d76f2d5f65.jpg
Oliver Bertuch (Apr 04 2024 at 12:17):Wondering if the idea behind https://neurobagel.org/ might be implemented with Dataverse repositories as well...
Oliver Bertuch (Apr 04 2024 at 12:17):We do have a limited set of rather harmonized metadata, but obviously it gets much harder for file metadata
Oliver Bertuch (Apr 04 2024 at 12:18):As said above, maybe one of the next big goals for Dataverse?
Philip Durbin 🚀 (Apr 04 2024 at 12:25):Cool, they seem to be doing joins across variables?
IMG_20240404_141316361.jpg
Philip Durbin 🚀 (Apr 04 2024 at 12:29): Philip Durbin 🚀 (Apr 04 2024 at 12:31):@Oliver Bertuch let's compare notes and maybe corner the speaker :big_smile:
Oliver Bertuch (Apr 04 2024 at 12:31):@Jan Range pyDataverse module to add a https://docs.pyfilesystem.org for Dataverse?
Oliver Bertuch (Apr 04 2024 at 12:32):(As seen in the OneData presentation)
Jan Range (Apr 04 2024 at 12:33):Isnt this similar to what Stefano (Compute on Data) developed?
Oliver Bertuch (Apr 04 2024 at 12:33):I don't know :shrug:
Jan Range (Apr 04 2024 at 12:34):I think so, because he was also basically fetching data files as if these are present on the filesystem.
Jan Range (Apr 04 2024 at 12:35):Would be ace having something like this in PyDdataverse - Kind of an DvDownloader
Oliver Bertuch (Apr 04 2024 at 12:44):I would also :heart: a Nextcloud external storage plugin for Dataverse, so we could have this Windows Sync+Share feeling provided by the Nextcloud client without developing our own client...
Oliver Bertuch (Apr 04 2024 at 12:44):(As they seem to do for OneData)
Oliver Bertuch (Apr 04 2024 at 12:45):We have a lot of researchers that are very much accustomed to using SMB shares - the Sync+Share experience is probably the closest you can get to that.
Philip Durbin 🚀 (Apr 04 2024 at 12:53):Yes, would be nice
Oliver Bertuch (Apr 04 2024 at 12:53):https://github.com/libis/rdm-integration/issues/4
Philip Durbin 🚀 (Apr 04 2024 at 12:55):Could OneData's (beta) S3 storage driver be used with Dataverse? As a way to have distributed data, maybe?
Philip Durbin 🚀 (Apr 04 2024 at 12:56):Or is OneData simply a competitor from the Dataverse perspective?
Philip Durbin 🚀 (Apr 04 2024 at 12:57): Oliver Bertuch (Apr 04 2024 at 12:58):And I also opened https://github.com/gdcc/pyDataverse/issues/178
Oliver Bertuch (Apr 04 2024 at 13:03):And I just opened an issue for our hackathon idea @Jan Range @Philip Durbin https://github.com/distribits/distribits-2024-hackathon/issues/3
Philip Durbin 🚀 (Apr 04 2024 at 13:34):Ha! Great!
Philip Durbin 🚀 (Apr 04 2024 at 13:35):And I do hope to hack on trying to add OneData as an S3 provider. The speaker seems game
Oliver Bertuch (Apr 04 2024 at 13:36):Or as a Globus-like thing?
Oliver Bertuch (Apr 04 2024 at 13:36):He said that files stored have a unique identifier, so very similar to Globus
Philip Durbin 🚀 (Apr 04 2024 at 13:46):Group pic
1000007932_20240404154448~2.jpg
Philip Durbin 🚀 (Apr 04 2024 at 13:48):Yeah, the main thing for me is that he isn't offended if we try to use Onedata "just" as a storage backend. :sweat_smile:
Juan Pablo Tosca Villanueva (Apr 04 2024 at 13:48):If I have the conversion right, your talk is at 9:35 EST guys?
Juan Pablo Tosca Villanueva (Apr 04 2024 at 13:50):Ok I think I am right lol
image.png
Philip Durbin 🚀 (Apr 04 2024 at 13:51):Our talk is tomorrow
Juan Pablo Tosca Villanueva (Apr 04 2024 at 13:53):Yeah I was just checking the time :rolling_on_the_floor_laughing: sorry about the confusion
Oliver Bertuch (Apr 04 2024 at 13:55):https://time.is/compare/1535_5_Apr_2024_in_D%C3%BCsseldorf/Boston
Philip Durbin 🚀 (Apr 04 2024 at 14:10):Fancy!
bc335f8f-9b7b-4bba-8477-da1ebce8d5d4.jpg
Oliver Bertuch (Apr 04 2024 at 14:38):You should see the new cluster :sweat_smile:
Philip Durbin 🚀 (Apr 04 2024 at 15:27):Having drinks at the same place we had lunch if you'd like to join
Jan Range (Apr 04 2024 at 16:25):Joining soon - Currently in slide flow :grinning_face_with_smiling_eyes:
Oliver Bertuch (Apr 04 2024 at 17:03):Julia and I would love some company... The main table was already full :sweat_smile:
Jan Range (Apr 04 2024 at 17:10):On my way :raised_hands:
Philip Durbin 🚀 (Apr 04 2024 at 18:12):
Philip Durbin 🚀 (Apr 05 2024 at 07:56):The xz vulnerability just mentioned: https://arstechnica.com/security/2024/03/backdoor-found-in-widely-used-linux-utility-breaks-encrypted-ssh-connections/
Philip Durbin 🚀 (Apr 05 2024 at 08:09):Mention of https://tom.preston-werner.com/2009/05/19/the-git-parable.html
Philip Durbin 🚀 (Apr 05 2024 at 09:02):2efc5eb7-cdb9-4e76-aee0-56ff133d6fbf.jpg
Philip Durbin 🚀 (Apr 05 2024 at 09:42):
Philip Durbin 🚀 (Apr 05 2024 at 11:56):
Philip Durbin 🚀 (Apr 05 2024 at 13:07):4d06f2d2-1633-4961-b068-42abb70491ed.jpg
Juan Pablo Tosca Villanueva (Apr 05 2024 at 14:20):I am finishing watching your talk! Congratulations @Philip Durbin @Oliver Bertuch && @Jan Range
Philip Durbin 🚀 (Apr 05 2024 at 14:27):Yeah? It was ok? :sweat_smile:
Philip Durbin 🚀 (Apr 05 2024 at 14:46):The inspiration for the name DataLad
17123283444132688976197007845241.jpg
Juan Pablo Tosca Villanueva (Apr 05 2024 at 14:53):Philip Durbin said:
Yeah? It was ok? :sweat_smile:
10/10 would watch again :smile:
Juan Pablo Tosca Villanueva (Apr 05 2024 at 14:54):Philip Durbin said:
The inspiration for the name DataLad
17123283444132688976197007845241.jpg
How does it always comes down to the Simpsons :rolling_on_the_floor_laughing:
Philip Durbin 🚀 (Apr 05 2024 at 15:20):Please like and subscribe! :crazy:
Philip Durbin 🚀 (Apr 05 2024 at 17:17):
Oliver Bertuch (Apr 05 2024 at 21:11):Juan Pablo Tosca Villanueva said:
Philip Durbin said:
Yeah? It was ok? :sweat_smile:
10/10 would watch again :smile:
Thx JP! :blush:
Jan Range (Apr 06 2024 at 07:59):@Oliver Bertuch @Philip Durbin happy hacking! :raised_hands: Which ideas did you go for?
Oliver Bertuch (Apr 06 2024 at 08:05):We're going for the datalad-dataverse integration and talking to Lukasz about OneData
Oliver Bertuch (Apr 06 2024 at 08:06):I'm hacking on the action in parallel
Jan Range (Apr 06 2024 at 08:06):Sounds great! Looking forward
Jan Range (Apr 06 2024 at 08:06):I was looking into PyFilesystem on the train yesterday and will implement a proof of concept. Looks straightforward!
Oliver Bertuch (Apr 06 2024 at 08:08):Awesome!!!
Philip Durbin 🚀 (Apr 06 2024 at 08:08):@Jan Range are you suffering from FOMO? :crazy:
Jan Range (Apr 06 2024 at 08:09):Yeeees :sob:
Philip Durbin 🚀 (Apr 06 2024 at 08:31):I'm half done putting our talk on DataverseTV. Here's the link to the timestamp at least: https://www.youtube.com/watch?v=L1MKaUgg1xs&t=24405s
Philip Durbin 🚀 (Apr 06 2024 at 08:32):@Jan Range are you good at combining PDFs? If so, maybe you could combine the two slide decks together? If not, please send along a PDF of yours and I'll try to figure it out. I'd like to add it to https://dataverse.org/presentations and then link to it from DataverseTV.
Jan Range (Apr 06 2024 at 08:32):Yes, will do :-)
Jan Range (Apr 06 2024 at 08:35):There you go: Dataverse_for_distributed_data_Distribits_DurbinBertuchRange.pdf
Oliver Bertuch (Apr 06 2024 at 09:16):Hacking on the Dataverse Action: @Jan Range instead of mangling a "s3_enabled" option, maybe it would be easier to have a flavor thing already? The idea would be to make people go for a directory with it that has a compose file and a bootstrap.sh...
Jan Range (Apr 06 2024 at 09:21):Alright, that sounds cool! If I understood correctly, we would provide an interface for other users to ingest their own compose flavor?
Oliver Bertuch (Apr 06 2024 at 09:25):Obsolutely!
Jan Range (Apr 06 2024 at 09:35):Really nice! Shall we host a couple of examples people could just grab and use plus get inspired?
Oliver Bertuch (Apr 06 2024 at 09:36):I will use this for the S3 flavor for now... :wink:
Oliver Bertuch (Apr 06 2024 at 09:36):That's at least 1 example
Philip Durbin 🚀 (Apr 06 2024 at 10:42):3fe468ee-c9e8-4ffa-956e-b8e103184ac1.png
Philip Durbin 🚀 (Apr 06 2024 at 10:42):b350cc0e-b6b4-4d75-b633-3c8302754d0d.jpg
Philip Durbin 🚀 (Apr 06 2024 at 10:42):Beautiful day!
Jan Range (Apr 06 2024 at 11:11):Is this what you thought about? Listing files and downloading on demand works well!
Jan Range (Apr 06 2024 at 11:14):@Philip Durbin I see you've got the hang out of Alt! :grinning_face_with_smiling_eyes:
Jan Range (Apr 06 2024 at 11:25):You can also list particular subdirs
image.png
Jan Range (Apr 06 2024 at 11:28):In terms of editing and uploading data, I was thinking of a context manager:
Jan Range (Apr 06 2024 at 11:31):I would use the DVUploader in the backend to perform the upload and handle S3/native routes
Oliver Bertuch (Apr 06 2024 at 11:42):That looks pretty promising already!
Oliver Bertuch (Apr 06 2024 at 11:43):Seems it was easy to get started!
Oliver Bertuch (Apr 06 2024 at 11:44):IMHO all the more reason to make S3 direct up/down handling part of pyDataverse
Jan Range (Apr 06 2024 at 11:44):True, completely agree now!
Jan Range (Apr 06 2024 at 11:45):Any other things you think would be nice to have there?
Oliver Bertuch (Apr 06 2024 at 11:51):How about making it possible to retrieve a single file from a ZIP on Dataverse?
Oliver Bertuch (Apr 06 2024 at 11:51):Dataverse knows about ranged requests
Oliver Bertuch (Apr 06 2024 at 11:51):So it might be possible to use a pyfilesystem for ZIPs here
Jan Range (Apr 06 2024 at 14:12):Sounds good! Is there a way to retrieve infos about the content of a zip beforehand? Could add it to the listdir-method
Philip Durbin 🚀 (Apr 06 2024 at 14:12):@Oliver Bertuch https://guides.dataverse.org/en/6.2/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled
Oliver Bertuch (Apr 06 2024 at 14:39):I'm not sure how the ZIP file previewer does it, but it seems possible @Jan Range
Oliver Bertuch (Apr 06 2024 at 16:13):https://github.com/nedbat/scriv
Philip Durbin 🚀 (Apr 06 2024 at 17:34):Ha! nedbat is a friend of mine!
Philip Durbin 🚀 (Apr 06 2024 at 20:33):
Philip Durbin 🚀 (Apr 07 2024 at 18:56):I just listened again to the question at https://www.youtube.com/watch?v=L1MKaUgg1xs&t=27125s by Stephan Heunis about using the API for requesting and granting access to files. It looks like the dataset he's talking about it at https://dataverse.nl/dataset.xhtml?persistentId=doi:10.34894/R1TNL8 which is running Dataverse 6.0 so the APIs he needs should be available:
Philip Durbin 🚀 (Apr 07 2024 at 18:59):Now, like Jan pointed out in his answer, these endpoints may not be available via pyDataverse.
Philip Durbin 🚀 (Apr 07 2024 at 19:01):Actually, I see both request_access and grant_file_access at https://pydataverse.readthedocs.io/en/latest/reference.html
Philip Durbin 🚀 (Apr 07 2024 at 19:01):Let me see if I can get in touch with Stephan.
Oliver Bertuch (Apr 07 2024 at 19:02):He should be on Matrix :smile_cat:
Philip Durbin 🚀 (Apr 07 2024 at 19:03):Ja. I just DM'ed him.
Philip Durbin 🚀 (Apr 08 2024 at 11:11):I just created this issue: Add DataLad to list of integrations #10468
Philip Durbin 🚀 (Apr 08 2024 at 19:44):I was able to see the eclipse from the plane! :tada:
Philip Durbin 🚀 (Apr 09 2024 at 14:37):Nice, here's just our talk: https://www.youtube.com/watch?v=jSzwAIqjq-o
I'll update DataverseTV
Philip Durbin 🚀 (Apr 09 2024 at 14:37):We should also post our slides but we should combine into one PDF, I think
Philip Durbin 🚀 (Apr 09 2024 at 16:31):Oh, Jan already posted it above. I sent it off to be added to https://dataverse.org/presentations
Philip Durbin 🚀 (Apr 09 2024 at 17:39):I just posted a mini trip report at https://groups.google.com/g/dataverse-community/c/huhI8TyE8a0/m/7xldNlW8AQAJ
Philip Durbin 🚀 (Apr 09 2024 at 17:46):Just posted: https://dataverse.org/presentations/distributed-metadata-and-data-dataverse
Philip Durbin 🚀 (Jan 09 2025 at 20:19):@Oliver Bertuch @Jan Range and others, this just came in:
"@room We have just reserved the venue for a distribits 2025. It will take place in Düsseldorf again. This time Oct 23-25. More information and an official announcement will follow soon. We are hoping to see you again!"
Is anyone thinking about going? Should we start a new topic for Distribits 2025?
Jan Range (Jan 15 2025 at 13:41):Sounds great! Can't tell if I will be available, but if there's time I am happy to join again :smile:
Philip Durbin 🚀 (Jan 15 2025 at 15:42):Personally, I'd be happy to visit a new city but whatever, Düsseldorf was great.
Last updated: Nov 01 2025 at 14:11 UTC