Stream: containers

Topic: curl DNS broken

view this post on Zulip Jutta Schnabel (Dec 21 2023 at 12:09):

Hi! Sorry if I am wrong here (Newbie) and feel free to push me to the right chat channel, but this seemed the most fitting:

I am trying to test dataverse with the docker installation on a test server. On starting the containers I get the Payara standard screen displayed, after entering the admin interface at :4848 and clicking "reload" I get The Dataverse Project displayed, but with a 404-page not found, and no menues etc. I am using v6.0 of dataverse and put the endpoint to 8081 from 8080 due to that port not being available on my server.

Any suggestions where to start debugging?

view this post on Zulip Philip Durbin 🚀 (Dec 22 2023 at 13:00):

Hmm, it should be possible to run on a port other than 8080 but I haven't tested it recently. Would it be possible to try again on 8080? I would suggest deleting the docker-dev-volumes directory first for a clean start (that's where the database, etc. is kept).

view this post on Zulip Jutta Schnabel (Dec 27 2023 at 07:34):

Thanks for creating the new chat! Sorry, it took me a while to realize. I tried launching the containers on 8080 now (after deleting dev-volumes), but got the same result - Payara page on startup, and 404 Dataverse page after relaunch in the admin interface. Am I too impatient?

view this post on Zulip Philip Durbin 🚀 (Jan 04 2024 at 11:42):

@Jutta Schnabel hi! It's my first day back after a long break. Did you get it working? Are you interested in joining our weekly meeting? (Please see #containers > weekly meeting .)

view this post on Zulip Jutta Schnabel (Jan 04 2024 at 13:14):

Hi and thanks for the hints. Unfortunately, moving to 8080 and deleting the dev-volumes did not solve the problem, the behaviour is as observed before, so I went back to the other port. Thanks for the hint to the meeting, I would drop in and listen in to your activities, perhaps that helps me getting a better idea :)

view this post on Zulip Philip Durbin 🚀 (Jan 04 2024 at 13:25):

Great! We'd love for you to join!

view this post on Zulip Jutta Schnabel (Jan 04 2024 at 15:26):

dev_dataverse           |   Context path from ServletContext:  differs from path from bundle: /|#]
dev_dataverse           |
dev_dataverse           | [#|2024-01-04T15:25:31.315+0000|INFO|Payara 6.2023.8||_ThreadID=81;_ThreadName=http-thread-pool::http-listener-1(4);_TimeMillis=1704381931315;_LevelValue=800;|
dev_dataverse           |   Loaded [1] org.ocpsoft.rewrite.spi.GlobalParameterProvider [org.ocpsoft.rewrite.instance.WildcardParameterProvider<0>]|#]
dev_dataverse           |
dev_bootstrap           | curl: (6) Could not resolve host: dataverse
dev_bootstrap exited with code 6
``

view this post on Zulip Jutta Schnabel (Jan 04 2024 at 15:27):

Docker version: Docker version 24.0.7

view this post on Zulip Philip Durbin 🚀 (Jan 04 2024 at 15:33):

Thanks. As I mentioned, it works in Ubuntu when spun up by GitHub Actions. For example: https://github.com/gdcc/api-test-runner/actions/runs/7185003205/job/19567333935

I'm not sure what the difference is. :thinking:

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 06:57):

Morning! @Jutta Schnabel have you figured out what the problem is? I am a lot closer (timezone-wise) to you than @Philip Durbin, happy to help.

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:42):

Hi! Thanks @Oliver Bertuch for the offer! I just tried the installation now locally on my machine but interestingly I ended up with exactly the same error. So it might actually not be the server setup which is the problem.

I used the current repo version, Docker 24.0.7 on ubuntu 22.04 (the server runs 20.04)
image.png

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:43):

Hi @Jutta Schnabel ! At least it looks like you can access Dataverse now and not just see the Payara default domain page!

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:43):

When opening localhost, the docker output is the following:
image.png

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:44):

Oh yeah that one is not to worry about - this is normal during the first bootstrap. Could you share the output of conigbaker?

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:44):

I see the default 404, but this was also the case before when doing a relaunch from the admin interface on the server

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:45):

How can I get that output?

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:45):

But you see a 404 in the layout of Dataverse - not just Payara. Which means the app at least deployed succesfully

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:46):

Jutta Schnabel said:

How can I get that output?

Depending on how you started your container, use docker logs. You can find out the container name using docker container ls -a, because it will probably be stopped already.

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:50):

Interestingly I have three instances of that container, but only the first one has log content. In a repetitive fashon it contains

2024-01-08T11:30:40Z ERR Expectation failed error="failed to establish an http connection, caused by: Get \"http://dataverse:8080/api/info/version\": dial tcp 172.21.0.10:8080: connect: connection refused" address=http://dataverse:8080/api/info/version
2024-01-08T11:30:48Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-01-08T11:30:51Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-01-08T11:30:59Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-01-08T11:30:59Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200
2024-01-08T11:31:07Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-01-08T11:31:10Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-01-08T11:31:18Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-01-08T11:31:21Z ERR Expectation failed error="timed out while making an http call, caused by: Get \"http://dataverse:8080/api/info/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" timeout=3s
2024-01-08T11:31:29Z INF [HTTP] Checking the http://dataverse:8080/api/info/version ...
2024-01-08T11:31:29Z ERR Expectation failed error="the status code doesn't expect" actual=404 expect=200

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:52):

Yeah, that is expected - it waits in the background for Dataverse to deploy, so it can reach out and do it's magic to bootstrap the instance (add a superuser, add metadata blocks and create root collection)

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:52):

What I don't see is that it actually comes around to doing sth

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:52):

No timeout, too...

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:53):

I assume you have at least >8 GB of RAM in your box, aye?

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:55):

Not in a box, but total RAM is 8GB

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:57):

That should be enough.

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 11:58):

@Jutta Schnabel Do you want to do a quick Zoom chat and let me take a look at live logs?

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 11:59):

yes, thanks: https://fau.zoom-x.de/j/65435375286

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:15):

OK the problem with no bootstrapping happening is REALLY tricky

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:15):

We dug into this and turns out: curl doesn't like the way DNS was set up in her environment

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 13:15):

Thanks a lot for the great help! Whatever the problem was is yours to report ;)

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:16):

I did a bootstrap manually now by circumventing the DNS lookup (provided the Dataverse URL as command argument)

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:17):

When I find some time (but happy for someone else to pick up the task), I would like to have some vanilla Ubuntu 22.04 installed on a VM, get Docker on top and see if we can reproduce the problem

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 13:17):

Is it ok if I take the containers down and restart with endpoint 8081? It clashes with one of the other services now ...

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:18):

We exec'ed into configbaker and nslookup, ping, etc all were fine, just curl didn't want to cooperate. We saw it reaching out to the DNS server. Maybe a tcpdump would have helped, too, to analyze what goes over the wire.

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:18):

Jutta Schnabel said:

Is it ok if I take the containers down and restart with endpoint 8081? It clashes with one of the other services now ...

Yes, that should be fine

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:19):

I just renamed the topic so we can dig more into it later

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:20):

Please create a new topic for more questions that might arise, thx @Jutta Schnabel

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 13:20):

Will do, and thanks again for the deep dive!

view this post on Zulip Oliver Bertuch (Jan 08 2024 at 13:21):

We were not in Texas but still deep into the heart of configbaker... :see_no_evil:

view this post on Zulip Jutta Schnabel (Jan 08 2024 at 13:28):

btw., the restart now again only displayed the payara screen, but relaunch gave the working application

view this post on Zulip Philip Durbin 🚀 (Jan 08 2024 at 15:18):

Would it be premature to create an issue to track this problem? We'd like to reproduce it first?

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 14:57):

It looks like this is related to this PR, that has been merged recently. https://github.com/c-ares/c-ares/pull/685. See this issue: https://github.com/c-ares/c-ares/issues/683

cURL on Alpine uses c-ares for DNS resolution as of alpine 3.19. As we are using alpine:3 as our base image, we are on 3.19. The fixed version of c-ares has been released, but the last build of curl dates back to 12.2023 (https://pkgs.alpinelinux.org/packages?name=curl&branch=v3.19)

So probably two options: we tag our base with 3.18 or we wait for a new release. I wouldn't go for the "edge" package.

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:07):

Interesting. So downgrade to Alpine 3.18 and leave a comment saying "don't upgrade to 3.19 or beyond until the version of curl is built with a version of c-ares that has a fix for https://github.com/c-ares/c-ares/issues/683 "

Something like that? I can't easily tell which version of curl has the fix.

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:09):

3.19 still is on c-ares 1.24. See https://pkgs.alpinelinux.org/package/v3.19/main/x86_64/c-ares

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:10):

Edge has c-ares 1.26.0, containing the fix https://pkgs.alpinelinux.org/package/edge/main/x86/c-ares

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:11):

Curl in Alpine 3.18 does not have the c-ares dependency, see https://pkgs.alpinelinux.org/package/v3.18/main/x86_64/libcurl

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:16):

https://github.com/IQSS/dataverse/issues/10413

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:19):

Ok, bottom line for the issue you opened is to downgrade to Alpine 3.18. Sounds good.

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:20):

I'm way ahead

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:20):

Do we know which version of curl has the fix?

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:20):

PR in 3 ...

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:20):

ha

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:22):

https://github.com/IQSS/dataverse/pull/10414

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:26):

just left a review, thanks!

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:35):

I just saw that they did a backport for 3.19... See https://git.alpinelinux.org/aports/commit/?id=6edd0d6d61c7f8b8a37e84333b37aa02e6507bfb

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:36):

Suggestions: if 3.18 works, lets see if we can make @Deirdre Kirmis look how old her local image pull is. (docker image ls "gdcc/*")

view this post on Zulip Philip Durbin 🚀 (Mar 22 2024 at 15:39):

That commit is two months from what I can tell.

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:41):

Yeah, it's from 2024-01-23

view this post on Zulip Deirdre Kirmis (Mar 22 2024 at 15:44):

yes ^

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:46):

Yes? As in "my image is that old"?

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 15:48):

At least I can tell that the problems @Jutta Schnabel experienced were from before that backport happened

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 16:26):

From the debugging I did with @Deirdre Kirmis at #containers > demo tutorial issues I learned that the "fix" they made in c-ares does not really fix it. Will report there.

view this post on Zulip Oliver Bertuch (Mar 22 2024 at 16:35):

https://github.com/c-ares/c-ares/issues/683#issuecomment-2015471931

Last updated: Oct 30 2025 at 05:14 UTC