Do you think its feasible to embed a plotting routine for the given arrays within an HDF5 file? I think that even with a simple visualization this would be very handy to inspect an HDF5 file. Similar to what H5Web does.
I like to say that anything is possible with code. :happy:
However, I'm a bit concerned about the effort involved.
Generally speaking, we allow external tools to do dataviz. That was one of our the architectural changes when we did the big rewrite of DVN 3 into Dataverse 4.
So here I am fretting that perhaps you want this in the core? In Java? Or were you thinking about an external tool making plots from arrays? If it's the latter, sure, especially if it's contributed!
That said, perhaps it would be faster to get H5Web working? :thinking:
Alright, that makes sense. At first I was thinking about core, but you are correct, an external tool is way better and easier to achieve. Since H5Web is a React Library we could build a simple page for that.
Good news! H5Web is working right out of the box. We can use their example to simply host an instance and supply the files via an URL. They are currently hosting a demo instance to which you can send data, but I am not sure if they are happy when the complete Dataverse community is causing traffic on their server :-D
Hmm! @Jan Range can you please DM me a link? You're right, we should be careful not to overload it. :sweat_smile:
Here is an example taken from Harvard Dataverse, where we just need to supply the direct URL to the file. That aligns pretty nice with the external tools workflow. Next step would be to write a JSON manifest.
https://h5web.panosc.eu/h5wasm?url=https://dataverse.harvard.edu/api/access/datafile/6938813
Awesome! Thanks! I must say, though, the first impression is a little confusing... "No visualization available for this entity."
Screen-Shot-2023-03-28-at-10.52.36-AM.png
But, once you click under "DATA" you start seeing good stuff!
Screen-Shot-2023-03-28-at-10.54.00-AM.png
@Ana Trisovic check this out!! ^^
Yea, I was also confused at first :-D
I would suggest we take their code and containerize it so that other Dataverse installations can host it themselves to use it for file previews. What do you think?
Oh, I thought we would add it to https://github.com/gdcc/dataverse-previewers instead. Is that a possibility?
Generally speaking each previewer wraps a javascript library that does something, like show a preview of a PDF.
I dont know if this is possible since this is a React application. Or is it possible to host it there too?
Not sure. I see two paths, possibly not mutually exclusive:
If this doesn't make sense, lemme know. I'll keep typing or we can do a quick call. :happy:
I can build it to HTML, CSS and JS - That should work at the file previewers repository
Great!
Switched to Vite by the way. Definitely superior to create-react-app!
Heh. Nice.
Had to fight through some bugs, but I finally managed to provide H5Web as static site! Here you can find an example hosted on my server.:
You may replace url and name to try try other files.
Amazing! Great work, @Jan Range !
I don't know what "D" is but it works!
Screenshot-2023-04-13-at-5.34.08-PM.png
Ah, D is wind direction.
Thanks a lot! Happy to see it is also working for you :-)
I can also change the design of the web-app. The current color scheme looks kinda meh :laughing: Maybe adapt it to the Dataverse scheme or something more "friendly"?
Another quick styling :-)
Also add a Dataverse logo?
Sorry for spamming here, but is it possible to embed the external tool within the "Preview" tab?
Ha! Yes, we can definitely embed the tool within the Preview tab. We just make sure "types" includes "preview": https://guides.dataverse.org/en/5.13/api/external-tools.html#terminology
As for the design, logos, and etc., I was thinking that it might be nice to add HD5Web to the standard "previewers" repo at https://github.com/gdcc/dataverse-previewers
If it helps, here where I added the NcML previewer: https://github.com/gdcc/dataverse-previewers/pull/18
It's actually a good example of "types":["preview"].
If it makes more sense to have a separate, standalone repo for HD5Web, that's fine too. I hope we aren't forking! :sweat_smile: Typically, we wrap a nice, existing tool (PDF preview, for example) written in Javascript in our "previewers" framework. That said, we don't have to follow this pattern if it's limiting.
Perfect! I am really excited to see it working on an installation ![]()
I think we can push it to Dataverse Previewers although the built files are everything but readable. Along this I would add the React source code from which it was built. This way we can can modify it easily.
Afaik the previewers are hosted on a GitHub page?
Yes, exactly. GItHub Pages. For example: https://gdcc.github.io/dataverse-previewers/previewers/betatest/NcmlPreview.html?fileid=6987711&siteUrl=https://dataverse.harvard.edu&datasetid=6987552&datasetversion=1.0&locale=en
Alright, will submit a PR once back home.
Bit off topic, but which app do you use to obtain these nice window screenshots? They look great!
Tested it on Github.io, works like a charm. Going to open a PR at GDCC previewers.
Opened a PR:
@Jan Range incredible! Thank you!
We absolutely should consider maintainability which means the source should go somewhere. Let touch base with Jim Myers, who created and has contributed the most to the previewers repo, if he thinks it's a good fit for HD5Web.
I'm just using the normal "take a screenshot" built into Mac.
This is really amazing @Jan Range :smiley:
@Jan Range I just added it as an external tool on my dev1 server but here's how the query parameters are passed:
So the question is, can we change hdf5.js a bit to handle fileid=1038 and siteUrl=https://dev1.dataverse.org as separate parameters?
By the way, since we have two preview tools for NetCDF files now, you can see how the UI lets you toggle between them: https://dev1.dataverse.org/file.xhtml?fileId=1038&version=2.0
@Philip Durbin thanks for testing it! Of course, I can change the query parameters to match the default ones. The name parameter is also not that important as it is just a display of the file name within the viewer. The ID will also work. I am currently not at home, will implement it once back and push the update to the PR.
Regarding the source, what do you think about an extra directory which hosts React code and an action that deploys it automatically to the betatest directory? This would open up to other previewers for files. Was thinking about Plotly for the visualisation of tabular data.
Hmm, for the source maybe you could add it to a separate branch and link to it from the PR, just to show how much there is and your idea of where it might go.
Alright, sounds like a plan :ok:
@Jan Range I went ahead and put HD5Web on the agenda for today's meeting: https://docs.google.com/document/d/1cZJr0JsASzOTxl70RynWtMjIa5dssCCpKPhyV-Bvx0c/edit?usp=sharing
I've managed to reproduce the error and it seems like, when the Previewers HTML filename is not index it fails to load it. I try to adjust it on the React side, my guess its something within the JS files that expect the index.html and isnt working if any other name is given.
Tried it with a different name, coming from React and still not possible using serve -s .. It Only works, when its called index.html - I may be missing something here. Do you know how to fix this?
I happen to make it work with a different name locally!
Seems like there was some expectations from react-router
Going to push the new build. @Philip Durbin can you try it out once its up?
Absolutely!
Its up now ![]()
Used this query for testing:
http://localhost:3000/HDF5Preview?siteUrl=https://dataverse.harvard.edu/&fileid=6938813
Deployed but no preview at https://dev1.dataverse.org/file.xhtml?fileId=1038&version=2.0
Do these query parameters look right? https://dev1.dataverse.org/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html?fileid=1038&siteUrl=https://dev1.dataverse.org&datasetid=1036&datasetversion=2.0&locale=en
I'm happy to change the manifest to line up with what you're expecting.
https://dev1.dataverse.org/js/hdf5.js is a 404
Oh thats a pity :sad:
I changed it from /js to ./js
Console says that apparently it is not able to find the corresponding css and js
That's better!
Screenshot-2023-04-25-at-13-54-25-ICOADS_R3.0.0_1662-10.nc-Root.png
Niiice! How did you make it work?
Ah alright, you changed the path to the js within the HTML?
<script type="module" crossorigin [-src="/js/hdf5.js"></script>-]{+src="./js/hdf5.js"></script>+}
<link type="text/css" rel="stylesheet" [-href="/css/preview.css"-]{+href="./css/preview.css"+} />
<link rel="stylesheet" [-href="/css/hdf5.css"-]{+href="./css/hdf5.css"+} />
yeah
Sometimes its the simple stuff ... :laughter_tears:
This change should then work?
Yes, exactly.
Cool! Will push it now
That saved my day :laughing:
Heh. The last thing is to update the markdown file with curl commands.
Alright, currently editing it
Awesome.
Oh, and we still need to decide where to put the source code, right?
Yes, I will create another PR that proposes a new dir and action to support React apps built with vite
One question, we were talking about it to be a previewer and an explorer - Is explorer then the right addition to types
Ok. I'm also asking Jim about it at https://iqss.slack.com/archives/CVB2SMDFX/p1682446081208379?thread_ts=1682445613.157419&cid=CVB2SMDFX and @mentioned you. Might be faster. :happy:
Saw it on Slack :smile:
So great its working now! Been biting my teeth out on this last mile :laughing:
This is how I added the tool, twice because there are two content types. You may need fewer query parameters. And you should probably use the regular toolUrl pattern instead of my local URL: tool.txt
@Philip Durbin glad I have seen the comment to the NcML previewer. The PR at the previewers needs to be updated with the most recent source.
@Jan Range sorry, probably doing too many things at once but what do you mean? :sweat_smile:
The issue with ncml.js not using the API Token upon fetching. Just saw the comment at your dataverse PR that something is not working.
Haven't synced the H5Web build with the source code yet. There the API-Token header is included.
If you're saying your previewer works with drafts, great! Someday I'll go fix mine. :happy:
Good news! #9480 is in the sprint!
At minimum this will involve adding the new H5Web previewer to the list in the Dataverse Guides.
I'm also happy to test it more, help figure out where to put the source code, etc.
Did you know H5Web works with Nexus files? https://github.com/IQSS/dataverse/issues/7947#issuecomment-1549809295
Just saw the comment on GH - Thats awesome!!
I also just gave a demo of H5Web at the community call. The map of the US was right side up this time!
Nice! Can you send me the link to the file that has the map? Would love to see it :-)
Sure it's bi_1985.nc (I forget where we got this). Currently at https://dev1.dataverse.org/file.xhtml?fileId=1037&version=2.0
I'm pretty sure it came from https://www.northwestknowledge.net/metdata/data/bi_1985.nc
That northwest site is linked from the bottom of our design doc: https://docs.google.com/document/d/1Ax_sMdgx5ROkIBA7-IC4_hySvgXkk6O8qTZLIvWWnqE/edit?usp=sharing
Wow this is beautiful! You can even go through the time dimension :heart_eyes:
This would be a perfect demo file for our slides
orly
How do you go through the time dimension?
You can use the slider on the left. Very nice!
I see. I guess I'm on day 161 of 1985. Now I'll go back to the future.
I just made a PR into your PR (and left some comments): https://github.com/gdcc/dataverse-previewers/pull/30#issuecomment-1553653523
Overall, this is looking good! :rocket:
I just made this PR: add H5Web to guides #9600
Including a screenshot of the map, right side up :happy:
Screen-Shot-2023-05-19-at-9.50.41-AM.png
@Jan Range for the screenshot I manually removed the tool title and the blue box.
The PR is up :raised_hands:
https://github.com/gdcc/dataverse-previewers/pull/35
Looks great! I'm a little confused though. Can I go ahead and close the older PR now and say that the newer one is replacing it?
Yes, I left it open though for the case if Jim might not accept the addition of the PR. But we can re-open it anyway if thats the case.
Ok, I went ahead and closed it. Yes, we can re-open it if necessary.
I did click "request review" from Jim.
@Jan Range did you get a chance to read Jim's feedback? https://github.com/gdcc/dataverse-previewers/pull/35#issuecomment-1593671142
One part of the feedback is pretty straightforward. I think he wants you to move your stuff into the "betatest" directory.
Hi @Philip Durbin :wave: Yes, I saw it and will implement the changes today. Indeed, they look straightforward :-)
No rush, but thanks!
Updated the PR - Now puts everything in betatest as requested by Jim
@Philip Durbin is it possible to test it at your dev instance?
Yes! I got hit with some paperwork but soon! Thanks!
@Jan Range I need to look closer but https://dev1.dataverse.org/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html?fileid=1066&siteUrl=https://dev1.dataverse.org&datasetid=1065&datasetversion=1.0&locale=en isn't working.
It might be a relative vs absolute path thing.
It's trying to pull down https://dev1.dataverse.org/js/hdf5.js which probably isn't right.
@Philip Durbin thanks for testing! Yes, the js and css paths need to be relative in the resulting html. Going to check if this can be taken care of in the build phase.
Thanks so much!
Fixed it :smile: Vite is so nice! Was able to include this in the build file and now it should work
Closer! I pulled but now I get a different error:
Screenshot-2023-06-22-at-9.41.57-AM.png
Same URL as above if you'd like to try it.
Thanks! Do you use a private dataset or a public one?
Public. Here's the file: https://dev1.dataverse.org/file.xhtml?fileId=1066&version=1.0
Seems like there is some authorization error. I tested your link, but exchanged siteURL and fileID which point to Harvard's Dataverse. Here everything works fine, will check where the diffs are and hopefully find a solution.
Ah, maybe we saw this before? Or maybe it's just deja vu. :sweat_smile:
Haha yea I have the feeling we've been there already
I can download with wget https://dev1.dataverse.org/api/access/datafile/1066
No auth problems.
Maybe the issue is that I pass an empty "X-Dataverse-key" header since the URL does not include apiToken. Could that be the source?
Huh. Maybe?!?
Because that is the novelty compared to the previous builds we've tested
Want me to copy https://github.com/JR-1991.keys into my dev1 server so you can log in?
Okay, if you wait fro long enough H5Web returns an error message, that the given file is not a valid HDF5 - Thats quite odd!
Weird. This one is super small but it's still spinning: https://dev1.dataverse.org/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html?fileid=1038&siteUrl=https://dev1.dataverse.org&datasetid=1036&datasetversion=2.0&locale=en
I get the same error with H5Web's instance
https://h5web.panosc.eu/h5wasm
So maybe it doesn't matter how big the file is... maybe it'll eventually just give that error about not being a valid file.
Hey! It DOES work for a draft file!
Here, I'll paste the URL and then reset my API token: https://dev1.dataverse.org/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html?fileid=1073&siteUrl=https://dev1.dataverse.org&apiToken=6c626b0a-7f7f-4f47-816c-34fc44a3c6a9&datasetid=1072&datasetversion=:draft&locale=en
Philip Durbin schrieb:
Hey! It DOES work for a draft file!
Awesome!!!
So yeah, maybe try not sending the token if there is no token??
Your idea above :big_smile:
Alright, I'll fix that and push an update
The changes are now online, but it is still not working. This might be a problem related to H5Web itself. If we are not able to resolve it, I will open an issue there
Online? I haven't pulled yet. Should I?
Ah okay, thought it would be updated automatically. The built files are updated in the PR though
Cool, let me go pull. I'm hosting it locally:
"toolUrl":"/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html",
Thats so odd since they claim nc files are working
https://dev1.dataverse.org/dataexplore/dataverse-previewers/previewers/betatest/HDF5Preview.html?fileid=1038&siteUrl=https://dev1.dataverse.org&datasetid=1036&datasetversion=2.0&locale=en looks good! That's the small file, only 120 KB or so
Does it work for you? I am stuck in the loading page
Screenshot-2023-06-22-at-10.37.30-AM.png
Works!
Niiice!
Ah now it works on my side too :raised_hands:
Screenshot-2023-06-22-at-10.40.59-AM.png
I just moved https://github.com/IQSS/dataverse/pull/9600 to "in review"... just a doc change to advertise the new previewer.
Philip Durbin has marked this topic as resolved.
Last updated: Nov 01 2025 at 14:11 UTC