Docker Swarm deployment of Dataverse · containers

Stream: containers

Topic: Docker Swarm deployment of Dataverse

Tamas Gal (Jul 02 2024 at 08:57):

Does anyone have deployed a Dataverse swarm stack and if so, is there any documentation on that? I am currently trying to set up a stack and fighting myself through Zulip threads, docs and GitHub/DockerHub repositories ;) I thought I should ask here before diving too deep...

Philip Durbin 🚀 (Jul 02 2024 at 12:03):

Dumb question: does Docker Swarm have Kubernetes under the hood these days?

Tamas Gal (Jul 02 2024 at 14:57):

No, that's two different things. Docker Swarm is more like an extended version of Docker Compose, which scales horizontally :)

Philip Durbin 🚀 (Jul 02 2024 at 15:11):

Oh, I see. Well, I obviously don't know much about Docker Swarm, but I'm happy to put it on the agenda for our next meeting ( https://ct.gdcc.io ), which you are welcome to join, to see if anyone has any ideas.

Tamas Gal (Jul 02 2024 at 15:44):

Alright, thanks! I am currently trying to figure out how the Docker Compose configuration of Dataverse works. The Swarm translation is quite straightforward but the devil is in the details...

Philip Durbin 🚀 (Jul 02 2024 at 15:55):

Good luck! I did go ahead and add Docker Swarm to the agenda: https://docs.google.com/document/d/1mdziHDJTIZGgI1ks8HFNTkfEfQqfsIRmNh4ufF516pk/edit?usp=sharing

Tamas Gal (Jul 03 2024 at 19:37):

Many thanks :)
Btw. I am a bit confused about the env-variables. There is DATAVERSE_DB_HOST (and user/password/name etc.) and also POSTGRES_SERVER... etc. Why are there two databases?

Philip Durbin 🚀 (Jul 03 2024 at 19:47):

There is only one database. You're using container's right. You probably want to use DATAVERSE_DB_HOST like we do here: https://github.com/IQSS/dataverse/blob/v6.3/docker/compose/demo/compose.yml#L13

Where are you seeing POSTGRES_SERVER?

Tamas Gal (Jul 03 2024 at 19:48):

That was in the sample.env :) OK, I'll continue, thanks ;) Yes, containers!

Tamas Gal (Jul 03 2024 at 20:06):

I just looked at the compose file... that uses different services. There is no minio for example and the dataverse image is not coronawhy/dataverse:... but gdcc/dataverse.
I am a bit confused now

Tamas Gal (Jul 03 2024 at 20:07):

Now I see it's from two weeks ago, so I guess that's the state of the art?

Philip Durbin 🚀 (Jul 03 2024 at 20:21):

Wait, are you following https://guides.dataverse.org/en/6.3/container/ or dataverse-docker which is community led (please see https://guides.dataverse.org/en/6.3/developers/containers.html )?

Philip Durbin 🚀 (Jul 03 2024 at 20:22):

The situation is definitely confusing and we have an issue to address it: Document competing containerization efforts and how to choose #10522

Philip Durbin 🚀 (Jul 03 2024 at 20:27):

@Slava Tykhonov just told me he's moving dataverse-docker ("archive in a box") to the gdcc images, if that helps.

Tamas Gal (Jul 03 2024 at 20:28):

ok thanks, i'll try my luck... definitely a tough piece to get working on docker stack compared to the other services i deployed in the past :D

Philip Durbin 🚀 (Jul 03 2024 at 20:28):

The gdcc images are maintained by core contributors like me and @Oliver Bertuch

Tamas Gal (Jul 03 2024 at 20:28):

but i see that the number of volumes and file mappings are reduced in the latest compose, which makes things a bit easier

Philip Durbin 🚀 (Jul 03 2024 at 20:28):

They are somewhat new, the gdcc images. Slava filled a gap with his coronawhy images until the gdcc images were ready.

Philip Durbin 🚀 (Jul 03 2024 at 20:29):

Yeah, the demo compose is intentionally smaller than the dev compose in the root of the repo.

Philip Durbin 🚀 (Jul 03 2024 at 20:30):

We have all that stuff like minio in the dev compose for testing.

Tamas Gal (Jul 03 2024 at 20:30):

i see

Philip Durbin 🚀 (Jul 03 2024 at 20:31):

And sorry, I keep moving your messages around. I can put them back under your original topic if you want: #containers > Docker Swarm deployment of Dataverse

You're still trying to use Docker Swarm? Or are you trying Docker Compose?

Tamas Gal (Jul 03 2024 at 20:58):

No worries, you can move stuff as it fits, I am not so handy with Zulip!

Tamas Gal (Jul 03 2024 at 20:58):

Swarm. I think I now got all the services running on different nodes with shared filesystems

Tamas Gal (Jul 03 2024 at 20:59):

Screenshot-2024-07-03-at-22.57.26.png

Tamas Gal (Jul 03 2024 at 20:59):

looks promising... now trying to get the load balancer with SSL termination to talk to Dataverse :)

Philip Durbin 🚀 (Jul 03 2024 at 20:59):

Ah, great. Yeah, hopefully having fewer services to deal with helps. :sweat_smile:

Tamas Gal (Jul 03 2024 at 21:00):

It's a bit less confusing at least ;)

Philip Durbin 🚀 (Jul 03 2024 at 21:00):

I'll put the messages back under the original topic.

Tamas Gal (Jul 03 2024 at 21:00):

Btw. is an Nginx or so recommended or can I directly use the HTTP connection to dataverse?

Philip Durbin 🚀 (Jul 03 2024 at 21:01):

Well, Dataverse runs on port 8080.

Oliver Bertuch (Jul 03 2024 at 21:01):

Evening. Lots of chatter here, great. I always would strongly recommend putting Dataverse behind some kind of reverse proxy

Philip Durbin 🚀 (Jul 03 2024 at 21:01):

You can configure Payara to serve on 80 and 443 easily enough but adding an SSL cert to Payara is a nightmare.

Philip Durbin 🚀 (Jul 03 2024 at 21:02):

In production we (Harvard) front with Apache but you could use nginx or whatever.

Tamas Gal (Jul 03 2024 at 21:02):

SSL certi is not a concern, I have an SSL layer termination in front of each service

Oliver Bertuch (Jul 03 2024 at 21:03):

Then you already have a reverse proxy / load balancer in place and are probably good to go

Tamas Gal (Jul 03 2024 at 21:03):

I was just wondering if the built-in webserver is "enough"

Tamas Gal (Jul 03 2024 at 21:03):

alright thanks :)

Oliver Bertuch (Jul 03 2024 at 21:03):

If you can make that ingress handler take care of routing as well, that's even better

Tamas Gal (Jul 03 2024 at 21:03):

Yep I do

Oliver Bertuch (Jul 03 2024 at 21:04):

Great. Sound like you're good to got then.

Tamas Gal (Jul 03 2024 at 21:04):

I'll report back and share the stack configuration too

Tamas Gal (Jul 03 2024 at 21:04):

there are some differences between Swarm and Compose

Oliver Bertuch (Jul 03 2024 at 21:05):

Keep in mind that when you want to use the ip groups feature you need to make your LB/RP send headers with the original addresses and apply some config to make Dataverse accept those

Tamas Gal (Jul 03 2024 at 21:05):

yes that will be probably tricky. is that documented somewhere?

Oliver Bertuch (Jul 03 2024 at 21:05):

It should be IIRC at the IP groups page

Tamas Gal (Jul 03 2024 at 21:05):

OK thanks

Oliver Bertuch (Jul 03 2024 at 21:08):

Hmm seems like it's not on that page. Let me dig around the guides

Oliver Bertuch (Jul 03 2024 at 21:10):

Ah it's in this section: https://guides.dataverse.org/en/latest/installation/config.html#blocking-api-endpoints

Oliver Bertuch (Jul 03 2024 at 21:11):

It's this config option: https://guides.dataverse.org/en/latest/installation/config.html#dataverse-useripaddresssourceheader

Oliver Bertuch (Jul 03 2024 at 21:12):

I'm afraid it's one if those leftover options we did not yet enable to set using MPCOnFIG. We're still working on that. But that's not the case with the coronawhy images, too

Oliver Bertuch (Jul 03 2024 at 21:14):

If you're interested in supporting us to get Dataverse containerization production ready, feel free to join the working group meetings. Deets on https://ct.gdcc.io. Anything helps, all feedback is appreciated, coordinated PRs are very welcome.

Tamas Gal (Jul 03 2024 at 21:16):

Thanks Oliver, I'll check the docs and let's see how far I get. If I can free up some time, I will also help of course!

Tamas Gal (Jul 03 2024 at 21:17):

Still fighting with the services, they crash after 3 minutes

Oliver Bertuch (Jul 03 2024 at 21:17):

Which ones?

Oliver Bertuch (Jul 03 2024 at 21:17):

It would be very weird if Solr or Postgres crash after 3 minutes

Tamas Gal (Jul 03 2024 at 21:17):

dataverse. Probably some communication problem with the network

Tamas Gal (Jul 03 2024 at 21:18):

[Entrypoint] running /opt/payara/scripts/init_3_wait_dataverse_db_host.sh
Operation timed out

Oliver Bertuch (Jul 03 2024 at 21:18):

Ah!

Tamas Gal (Jul 03 2024 at 21:18):

i'll check if the ports and internal DNS is correctly working

Oliver Bertuch (Jul 03 2024 at 21:18):

Yes, that sound like you need to check the networking part

Oliver Bertuch (Jul 03 2024 at 21:19):

That script is a poor man's workaround for our compose file not being able to work with healthchecks yet

Tamas Gal (Jul 03 2024 at 21:19):

whatever works ;)

Tamas Gal (Jul 03 2024 at 21:20):

forgot to attach the backend network to dataverse, that should be it...

Oliver Bertuch (Jul 03 2024 at 21:20):

(deleted)

Oliver Bertuch (Jul 03 2024 at 21:20):

Oh OK

Tamas Gal (Jul 03 2024 at 21:21):

only the one which is shared with the load balancer was attached to the dataverse service

Tamas Gal (Jul 03 2024 at 21:22):

ok, i see some success ;) Boot Command create-system-properties returned with result SUCCESS :

Tamas Gal (Jul 03 2024 at 21:22):

Screenshot-2024-07-03-at-23.22.18.png

Oliver Bertuch (Jul 03 2024 at 21:22):

You need to watch out for sth along the lines of "application dataverse deployed at /"

Oliver Bertuch (Jul 03 2024 at 21:23):

Depending on your resources this is usually done after 60 to 120 secs

Oliver Bertuch (Jul 03 2024 at 21:23):

Did you add the bootstrapping thing in your swarm setup?

Tamas Gal (Jul 03 2024 at 21:23):

yep

Tamas Gal (Jul 03 2024 at 21:23):

looks good :) running

Oliver Bertuch (Jul 03 2024 at 21:23):

Great. Once you see that one succeeding you're good to go

Tamas Gal (Jul 03 2024 at 21:24):

so you mean it's still configuring?

Oliver Bertuch (Jul 03 2024 at 21:24):

Yes. The Payara page means the appserver is responding

Oliver Bertuch (Jul 03 2024 at 21:25):

Deploying a large webapp like Dataverse taken some time

Oliver Bertuch (Jul 03 2024 at 21:25):

This is not Django :grinning_face_with_smiling_eyes:

Tamas Gal (Jul 03 2024 at 21:25):

i hope so ;)

Oliver Bertuch (Jul 03 2024 at 21:26):

Usually Java webapps are made to run a lot longer than you take time for starting them... :melting_face:

Tamas Gal (Jul 03 2024 at 21:26):

i see a bunch of warnings and one error (regarding log4j, which seems to fall back to simple logging)

Tamas Gal (Jul 03 2024 at 21:26):

Screenshot-2024-07-03-at-23.26.33.png

Oliver Bertuch (Jul 03 2024 at 21:26):

Most of these errors are expected. Good friends, we don't want to miss them

Tamas Gal (Jul 03 2024 at 21:26):

ok, that looks working

Oliver Bertuch (Jul 03 2024 at 21:26):

Yes!

Oliver Bertuch (Jul 03 2024 at 21:27):

The real test now is creating a dataset and uploading a file

Oliver Bertuch (Jul 03 2024 at 21:27):

That's usually a good smoketest

Oliver Bertuch (Jul 03 2024 at 21:27):

Because it also means you configured your storage successfully

Tamas Gal (Jul 03 2024 at 21:27):

yep :) i'll try

Tamas Gal (Jul 03 2024 at 21:28):

i have not configured the admin port yet though

Oliver Bertuch (Jul 03 2024 at 21:28):

Don't hate us for the messy configuration stuff. We're actively working on improving that. It's a lot... :innocent:

Oliver Bertuch (Jul 03 2024 at 21:29):

Even in classic installations the amount of stuff to configure can be overwhelming

Oliver Bertuch (Jul 03 2024 at 21:29):

The admin port of what? Payara?

Tamas Gal (Jul 03 2024 at 21:30):

Screenshot-2024-07-03-at-23.30.26.png

Oliver Bertuch (Jul 03 2024 at 21:30):

That looks promising!

Tamas Gal (Jul 03 2024 at 21:31):

yep, i see the data on the shared filesystem

Oliver Bertuch (Jul 03 2024 at 21:31):

Phew!

Tamas Gal (Jul 03 2024 at 21:31):

i thought the admin port for Payara is needed for further configuration

Tamas Gal (Jul 03 2024 at 21:31):

sorry, i am a noob with dataverse ;) but I have my right hand for that

Oliver Bertuch (Jul 03 2024 at 21:31):

Nope, you usually don't need to touch that

Oliver Bertuch (Jul 03 2024 at 21:32):

You can use it, but it's annoying and it won't help you with database options

Tamas Gal (Jul 03 2024 at 21:32):

alright, thanks so far! i'll clean up the swarm configuration the next days and them publish them in case someone is interested

Tamas Gal (Jul 03 2024 at 21:33):

and i'll check the ip header forwarding. i guess currently everyone has the same IP address (the one from the first layer docker instance of the load balancer ;)

Oliver Bertuch (Jul 03 2024 at 21:34):

If you want, feel free to create an issue at IQSS/dataverse and open a PR to include this in the guides. We probably should talk about a good place for it, but it would nice to include it. Give people options

Oliver Bertuch (Jul 03 2024 at 21:35):

Personally I wouldn't go for Swarm and opt for K8s, but it's a valid option if you're OK with the vendor thing.

Tamas Gal (Jul 03 2024 at 21:36):

at some point I might convert all our IT services to K8s but currently we live with that ;) I like it to be honest, it's easy to maintain and I have tons of ansible playbooks to automate things like redeployment, backups and details like certificate management

Oliver Bertuch (Jul 03 2024 at 21:37):

Whatever works for you folks!

Oliver Bertuch (Jul 03 2024 at 21:37):

That's the nice thing about containers. So many options

Tamas Gal (Jul 03 2024 at 21:37):

i thought about going all in with K8s but my hope was that some of the students help out. Swarm is quite easy to get started with if you already know Docker :D