Brian C. Keegan (Jun 20 2026 at 03:41): Brian C. Keegan (Jun 20 2026 at 03:41):Hi folks, newcomer to the community (and actually wrapping up at Harvard as visiting scientist!).
I'll be returning to Boulder in August and rebooting my lab group. I am interested in exploring the intersection of agentic AI, public interest data liberation, and digital archives.
One tool I'm prototyping built a CI pipeline for a daily scrape of my local police department's "Flock Transparency Portal." I'd like to archive and DOI-ify it while updating it at a regular interval. https://github.com/brianckeegan/boulderpolitics/tree/main/Part%2006%20-%20Flock%20ALPR%20contract
What advice, standards, docs, or examples would you recommend about archiving/DOI-ing a dataset that updates on a monthly/quarterly/annual frequency?
Brian C. Keegan (Jun 20 2026 at 03:43):Feedback on these prototype skills also welcome:
https://github.com/CUPIDS-Lab/data-project-skill
https://github.com/CUPIDS-Lab/data-liberation-skill
Philip Durbin 🚀 (Jun 20 2026 at 15:00):Hi! Welcome! For now, I'll just say that I'm reminded of this issue, which you might find interesting:
Create a configurable, scheduled process for the BARI researchers to process and deposit ("stream") Analyze Boston data into their dataverse automatically. #4530
Johannes D (Jun 22 2026 at 07:22):We also have some customers that update their data each night, which results on a lot of datasets versions. There is potential for quite some performance improvement but in general it does work as expected.
Philip Durbin 🚀 (Jun 22 2026 at 13:37):@Brian C. Keegan my colleague @Leo Andreev pointed to some concrete examples in a reply to "Institutional datasets with monthly updates — looking for examples & best practice" at https://groups.google.com/g/dataverse-community/c/iSvAnq_H4sE/m/w6aOcjmnAQAJ
Philip Durbin 🚀 (Jun 22 2026 at 13:44):He points to https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/VH6GVH which now has 430 versions. This is quite a lot for Dataverse (it takes a while for the "Versions" tab to load) so you might want to be careful about the frequency of updates.
Oliver Bertuch (Jun 22 2026 at 14:24):It might be interesting to take a look at what's going on under the hood there by extending the performance testing framework I contributed. I bet there are a few low hanging fruits to improve this.
Last updated: Aug 18 2026 at 08:27 UTC