enable previewers · containers · Zulip Chat Archive

Stream: containers

Topic: enable previewers

Philip Durbin 🚀 (Apr 18 2024 at 16:10):

As promised in today's meeting, I tried using the previewers hosted on GitHub Pages. Didn't work. CORS error. I opened an issue: enable previewers in containers #10506

Oliver Bertuch (Apr 19 2024 at 09:15):

The container serving the files needs to set the CORS header. For local development, * should be fine for Access-Control-Allow-Origin

Philip Durbin 🚀 (Apr 22 2024 at 10:44):

Which container should we use? Caddy? nginx? Traefik?

Oliver Bertuch (Apr 22 2024 at 10:47):

Maybe busybox?

Oliver Bertuch (Apr 22 2024 at 10:47):

https://lipanski.com/posts/smallest-docker-image-static-website

Philip Durbin 🚀 (Apr 22 2024 at 10:48):

Sure. We could always switch later.

Oliver Bertuch (Apr 22 2024 at 10:48):

Image size matters :see_no_evil:

Oliver Bertuch (Apr 22 2024 at 10:51):

If we want more configurability and support to manipulate headers, we might look at Caddy or Nginx.

Philip Durbin 🚀 (Apr 22 2024 at 10:52):

Well, we already use Caddy for the dev compose.

And for the demo compose we talked about using Traefik. I just started a topic at #containers > add Traefik

Guido Schmutz (Nov 15 2024 at 16:42):

Hi @Philip Durbin 🐉 , as mentioned in the Github issue I have implemented a solution for providing the previewers as a docker container. I wasn't aware of this thread and started implementing it inside the dataverse-previewers project. You can find it in my fork here: https://github.com/gschmutz/dataverse-previewers in the docker folder.

Docker image is using Nginx and I'm invoking the localinstall.sh script using env vars for the parameters.

Philip Durbin 🚀 (Nov 15 2024 at 16:44):

Awesome! I'm looking at the "compare" at https://github.com/gdcc/dataverse-previewers/compare/develop...gschmutz:dataverse-previewers:develop

Guido Schmutz (Nov 15 2024 at 16:44):

but I'm open to move it somewhere else, could copy the previewers from the release

Philip Durbin 🚀 (Nov 15 2024 at 16:46):

What's cool is that you seem to also be addressing this issue:

Consider Traefik or similar for HTTPS etc. (milestone E) #10359

Guido Schmutz (Nov 15 2024 at 16:47):

I also have a solution to register the previewers on a running Dataverse instance using docker. It's currently in it's own project (https://github.com/TrivadisPF/docker-dataverse-previewer), but I was planning to bring it together

Guido Schmutz (Nov 15 2024 at 16:49):

I also had configbaker in mind, but first wanted to just do a PoC, in docker compose it looks like that:

  dataverse-setup-previewers:
    container_name: dataverse-setup-previewers
    hostname: dataverse-setup-previewers
    image: trivadis/dataverse-deploy-previewers:latest
    init: true
    environment:
      - API_TOKEN=${API_TOKEN}
      - DATAVERSE_URL=http://${PUBLIC_IP}:28394
      - TIMEOUT=10m
      - PREVIEWER_PROVIDER_URL=http://${PUBLIC_IP}:28395
      - INCLUDE_PREVIEWERS=text,html,pdf,csv,comma-separated-values,tsv,tab-separated-values,jpeg,png,gif,markdown,x-markdown
      - EXCLUDE_PREVIEWERS=
      - REMOVE_EXISTING=True
    command:
      - deploy
    restart: no

Philip Durbin 🚀 (Nov 15 2024 at 16:49):

I don't know if this helps, but Dataverse developers like me use this docker-compose file everyday: https://github.com/IQSS/dataverse/blob/develop/docker-compose-dev.yml

We also have a demo/eval compose file here: https://github.com/IQSS/dataverse/blob/develop/docker/compose/demo/compose.yml

I'm interested in adding previewers to both of these somehow. :thinking:

Philip Durbin 🚀 (Nov 15 2024 at 16:52):

That is, I'd love a pull request against these files. :grinning:

Philip Durbin 🚀 (Nov 15 2024 at 16:54):

Developers use mvn -Pct clean package docker:run as explained at https://guides.dataverse.org/en/6.4/developers/dev-environment.html#quickstart

For the demo/eval use case, we have a separate quickstart at https://guides.dataverse.org/en/6.4/container/running/demo.html#quickstart

Philip Durbin 🚀 (Nov 15 2024 at 16:54):

Do you have any interest in trying either of these? :sweat_smile:

Guido Schmutz (Nov 15 2024 at 16:55):

just want to ask :-) do you want me to do a PR using the solution I have now and we then discuss how to move it forward?

Philip Durbin 🚀 (Nov 15 2024 at 16:55):

I'm not sure, to be honest. It's another Docker thing to maintain. :sweat_smile:

Philip Durbin 🚀 (Nov 15 2024 at 16:56):

But if we can get some of your ideas into a PR against one for both files above, it would be a big win.

Philip Durbin 🚀 (Nov 15 2024 at 16:57):

I'm happy to do a zoom call, by the way, if that's easier.

Guido Schmutz (Nov 15 2024 at 16:57):

sorry, I meant changing the two files but using the two docker images currently hosted on the trivadis docker hub? Just the docker solution I have now and then we can discuss if we want to consolidate it somewhere else

Guido Schmutz (Nov 15 2024 at 16:58):

Yes let's do a Zoom call, easier to discuss things than here ... but I'm just having dinner now, would you also be available in about 1 hour?

Philip Durbin 🚀 (Nov 15 2024 at 16:59):

I think I understand... use the trivadis images... maybe!

Yes, lunch time here. Let's check back in 1 or 1.5 hours, please. Thanks!

Philip Durbin 🚀 (Nov 15 2024 at 18:15):

@Guido Schmutz ready for a zoom link? :grinning:

Philip Durbin 🚀 (Nov 15 2024 at 18:25):

@Guido Schmutz https://harvard.zoom.us/my/pdurbin?pwd=em1WNUZGbnY2YjhxNEdSbjJJMXNSUT09

Philip Durbin 🚀 (Nov 15 2024 at 18:25):

Sorry, I didn't notice the :thumbs_up:

Philip Durbin 🚀 (Nov 15 2024 at 20:10):

Thanks for chatting and thanks for the future pull request!

Philip Durbin 🚀 (Nov 15 2024 at 20:12):

I'm going to throw some links in here so I don't lose them:

https://github.com/TrivadisPF/docker-dataverse-previewer
https://hub.docker.com/r/trivadis/dataverse-previewers-provider

https://github.com/TrivadisPF/docker-dataverse-previewer
https://hub.docker.com/r/trivadis/dataverse-deploy-previewers

Guido Schmutz (Nov 15 2024 at 22:04):

thanks as well, I have just submitted the PR on the compose.yml. Took me a bit longer to make it work. Had to change the DATAVERSE_SITEURL and set it to http://${MACHINE_IP:-localhost}:8080" on the dataverse service

Philip Durbin 🚀 (Nov 15 2024 at 22:14):

/me looks at Add two additional containers to provide Previewers locally #11025

Philip Durbin 🚀 (Nov 15 2024 at 22:15):

@Guido Schmutz thanks! :heart:

Philip Durbin 🚀 (Nov 15 2024 at 22:24):

@Guido Schmutz merged! Thanks! I noticed your branch was "develop" so I fast-tracked it. (See #10261 for problems we have with both branches being called "develop".) We can add docs in another pull request.

Philip Durbin 🚀 (Nov 15 2024 at 22:25):

Screenshot 2024-11-15 at 5.19.12 PM.png

Philip Durbin 🚀 (Nov 15 2024 at 22:25):

Screenshot 2024-11-15 at 5.19.32 PM.png

Philip Durbin 🚀 (Nov 15 2024 at 22:25):

Works great!

Philip Durbin 🚀 (Nov 15 2024 at 22:26):

Also, I added it to the agenda for our next meeting. (See also #containers > weekly meeting .)

Philip Durbin 🚀 (Nov 15 2024 at 22:26):

Thanks again! Have a good weekend!

Guido Schmutz (Nov 17 2024 at 12:53):

@Philip Durbin 🐉 great hat it worked on your machine and sorry for not using a branch, I realized it but then forgot to change it before I pushed it. Yes I'm aware that docs are needed as well, can do that in the coming wee.

Philip Durbin 🚀 (Nov 17 2024 at 14:52):

Cool, cool, I'm happy to help with docs as well. Thanks again!

Philip Durbin 🚀 (Jan 16 2025 at 17:31):

@Oliver Bertuch @Don Sizemore @Kris Dekeyser (KU Leuven) et al. I just got this approved and moved into "sprint ready": enable previewers in containers #10506

(As discussed during today's container meeting.)

Philip Durbin 🚀 (Jan 16 2025 at 17:32):

Pretty high up too. Number 4 right now: https://github.com/orgs/IQSS/projects/34/views/28 :tada:

Philip Durbin 🚀 (Jan 23 2025 at 15:43):

@Guido Schmutz hi! We just talked all about your previewers. :big_smile:

We recorded it, if you're interested: https://harvard.zoom.us/rec/share/mB8vTbBo_KAENu0QNyov7YXlEYe8-tDxRKSIJzFKZtOdsyKTw_dI73WTjmIX4y7l.9xgvoFYY4tGqWETB

Notes at https://docs.google.com/document/d/1ma1G2QiN8OPqVEb5dB5cdG7x4qRXgA8DS9bQlmgx0sA/edit?usp=sharing

We plan to pick up the conversation next week if you'd like to join! Please see #containers > weekly meeting

Philip Durbin 🚀 (Jan 23 2025 at 16:21):

The new PR is away! add file previewers to dev env, more docs #11181

Philip Durbin 🚀 (Jan 23 2025 at 16:24):

It's in "ready for review". Please feel free to leave a review, all.

Philip Durbin 🚀 (Jan 30 2025 at 19:27):

@Kris Dekeyser (KU Leuven) and others who attended today's meeting (notes, recording), thanks for the feedback! I just added more docs about which previewers are available: https://github.com/IQSS/dataverse/pull/11181/commits/9fa64ed863ca706a4110b9df4d3a464d275d9b1a

Philip Durbin 🚀 (Jan 30 2025 at 19:27):

Please take a look and let me know what you think.

Philip Durbin 🚀 (Jan 30 2025 at 19:28):

Also, I'm realizing I didn't bring this up during the meeting:

"Should we consider removing this MACHINE_IP stuff from this compose file? We already removed it from the dev version."

From the same PR. Any thoughts on this?

Kris Dekeyser (KU Leuven) (Jan 30 2025 at 19:48):

I added some comments to the Github issue. But overall I like the change you made. Thank you.

Philip Durbin 🚀 (Jan 30 2025 at 19:55):

Thanks! I replied!

Philip Durbin 🚀 (Feb 06 2025 at 15:48):

@Guido Schmutz Hi! Today @Kris Dekeyser (KU Leuven) and I noticed 404s for files like http://localhost:9000/previewers/v1.4/fonts/glyphicons-halflings-regular.ttf

Screenshot 2025-02-06 at 10.46.14 AM.png

Philip Durbin 🚀 (Feb 06 2025 at 15:50):

Any thoughts on this? Could it be an issue with https://hub.docker.com/r/trivadis/dataverse-previewers-provider ? Thanks! :heart:

Benedikt (Feb 06 2025 at 16:37):

Philip Durbin ☃️ said:

Any thoughts on this? Could it be an issue with https://hub.docker.com/r/trivadis/dataverse-previewers-provider ? Thanks! :heart:

Problemet eksisterer allerede...
https://github.com/gdcc/dataverse-previewers/issues/86

That comes from the localinstall.sh and he uses it in the Docker images:
https://github.com/gschmutz/dataverse-previewers/tree/develop

Philip Durbin 🚀 (Feb 06 2025 at 16:41):

Interesting! Thanks.

Kris Dekeyser (KU Leuven) (Feb 06 2025 at 20:19):

Hmm. The localinstall.sh in the docker image is the latest and includes the latest fix. I don't think the github issue is the problem.

Your screenshot indicates that the problem is that it tries to download the glyph font from the local host. Why would it do that?

Looking at the localinstall.sh closer, I think the problem is with the concept of the localinstall.sh itself. It downloads all the remote js and css files. And I believe that is what is killing it.

One of the downloaded CSS files is the minified version of the bootstrap style sheet. That file contains:

@font-face {
  font-family: "Glyphicons Halflings";
  src: url("../fonts/glyphicons-halflings-regular.eot");
  src: url("../fonts/glyphicons-halflings-regular.eot?#iefix") format("embedded-opentype"), url("../fonts/glyphicons-halflings-regular.woff2") format("woff2"), url("../fonts/glyphicons-halflings-regular.woff") format("woff"), url("../fonts/glyphicons-halflings-regular.ttf") format("truetype"), url("../fonts/glyphicons-halflings-regular.svg#glyphicons_halflingsregular") format("svg");
}

Therefore copying that file locally will cause the referenced fonts to also be searched locally.

An easy fix would be to nuke the whole idea of copying the files locally.

The proper way to make a local install work is by installing the NPM package, but that is another story.

Philip Durbin 🚀 (Feb 06 2025 at 20:25):

Right, for example, it's trying to download http://localhost:9000/previewers/v1.4/fonts/glyphicons-halflings-regular.ttf

Philip Durbin 🚀 (Feb 06 2025 at 20:28):

Ah, interesting, Jim created the localinstall script but it was based on script by @Benedikt according to the commit message. And the PR.

Philip Durbin 🚀 (Feb 06 2025 at 20:29):

Anyway, we should probably pull Jim in at some point.

Benedikt (Feb 07 2025 at 07:59):

That's true, NPM as you mean, @Kris Dekeyser (KU Leuven) , would be best. However, it would somewhat overturn the whole concept of the previewer. It was just an idea of mine to run everything completely locally without an internet connection.
I didn't pay attention to the internally linked data.

I will make an adjustment there now.

Thanks...