✔ Issues with sending mails · containers

Stream: containers

Topic: ✔ Issues with sending mails

Tamas Gal (Sep 20 2024 at 14:08):

A quick question: is there any problem known regarding sending mails vie the maildev service? I can't get it to work. Mails are simply stored as files on the containers volume but not sent out. I think that the auto-relay feature is off (by default) and enabling it via an environment variable does not work since the set value (like true) is then taken as the receiver mail address. I opened an issue here https://github.com/maildev/maildev/issues/501 but no answer yet.
Maybe I am doing something wrong ;)
The mails are here

root@ecap-s051:/raid/km3net-dataverse/mail# ls -al
total 48
drwxr-xr-x  2 tgal tgal 4096 Sep 20 16:01 .
drwxr-xr-x 10 root root 4096 Jul  3 22:56 ..
-rw-r--r--  1 tgal tgal  799 Sep 13 15:11 6CCFTjjF.eml
-rw-r--r--  1 tgal tgal  738 Sep 13 10:23 8nszk2Cu.eml
-rw-r--r--  1 tgal tgal  797 Sep 20 16:01 AdhVA5rZ.eml
-rw-r--r--  1 tgal tgal  752 Sep 13 10:46 cnpXNCO8.eml

and look OK-ish (apart that the link to Dataverse is shown as localhost:8080)

Philip Durbin 🚀 (Sep 20 2024 at 15:33):

Do you mean you don't see emails at http://localhost:1080 like this?

maildev.png

Tamas Gal (Sep 20 2024 at 16:45):

I did not expose that port. I thought it would send the mals automatically ;) I assume that if the mails are stored locally, they are not sent yet.

Tamas Gal (Sep 20 2024 at 16:48):

to be clear: I meant that nothing is arriving at the user. The instance has a valid URL with proper SSL certificate. I don’t know why it still puts “localhost” in the mail body but the mail header itself looks fine with the correct noreply email address etc. They are simply not arriving in anyone’s inbox. I can only see those mails as files in the mail folder of the maildev container.

Oliver Bertuch (Sep 20 2024 at 16:58):

Please don't use maildev as an MTA. You should look into either use some MTA of your institution or, if you need to mangle the mail before it's sent to a user (e.g. for SMIME signing), into deploying Postfix or other small-scale MTAs and use it as a relay.

Oliver Bertuch (Sep 20 2024 at 16:59):

Maildev really is meant to be used within a development scenario, where you don't want mails to be sent to arbitrary places. We use maildev also in integration tests to verify the sending process, etc.

Oliver Bertuch (Sep 20 2024 at 17:00):

In 6.2 the mail subsystem configuration of Dataverse has been reworked and should be pretty easy to use with any MTA out there.

Oliver Bertuch (Sep 20 2024 at 17:01):

Also keep in mind that for real usage, you should make sure you use servers as your final MTAs when leaving your organization that are added to the DNS based SPF records. Otherwise alot of receiving mailservers of your users will start rejecting/discarding the mails you're sending as Spam.

Oliver Bertuch (Sep 20 2024 at 17:08):

(I'm not even starting to talk about DANE and DKIM. SPF is the most crucial one these days.)

Philip Durbin 🚀 (Sep 20 2024 at 17:37):

Hmm, you're making me think that when we declare our containers ready for production, in the compose file we should swap out Maildev for a real SMTP server. :thinking:

Tamas Gal (Sep 20 2024 at 20:16):

To be honest, i was already wondering :sweat_smile:

alright, i will amend ;)

Philip Durbin 🚀 (Sep 20 2024 at 20:26):

Maildev is very handy in dev! I used it this week! :grinning:

Oliver Bertuch (Sep 20 2024 at 20:41):

Actually we probably should leave any MTA out or just have a commented out Postfix example. People need to make up their minds about this stuff for production.

Tamas Gal (Sep 21 2024 at 07:44):

I agree @Oliver Bertuch !

Tamas Gal (Sep 21 2024 at 07:46):

Now that I use the GMail SMTP server directly, I get a

jakarta.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. For more information, go to
535 5.7.8  https://support.google.com/mail/?p=BadCredentials 5b1f17b1804b1-42e754a69a7sm69421145e9.34 - gsmtp

which is strange since the credentials are definitely correct.
I am using these environment variables

      DATAVERSE_MAIL_MTA_HOST: "smtp.gmail.com"
      DATAVERSE_MAIL_MTA_PORT: "465"
      DATAVERSE_MAIL_MTA_SSL_ENABLE: "true"
      DATAVERSE_MAIL_MTA_AUTH: "true"
      DATAVERSE_MAIL_MTA_USER: "foo@gmail.com"
      DATAVERSE_MAIL_MTA_PASSWORD: "..."

According to the docs, the settings should be named correctly. :confused:

Oliver Bertuch (Sep 21 2024 at 07:53):

Did you activate SMTP in your Gmail Account?

Oliver Bertuch (Sep 21 2024 at 07:54):

Did you use an application password? (In case you're using 2FA)

Oliver Bertuch (Sep 21 2024 at 07:56):

Can you use the same credentials and connection details from some other app, like an email program?

Tamas Gal (Sep 21 2024 at 08:04):

Ah, that's a new Google Account and I just realised that they restricted the authentication methods
I now need to activate 2FA and create an App password

Oliver Bertuch (Sep 21 2024 at 08:04):

Please note that you might also need to enable SMTP access!

Tamas Gal (Sep 21 2024 at 08:04):

yep, now going through all the hassle ;)

Tamas Gal (Sep 21 2024 at 08:05):

thanks for the hint

Tamas Gal (Sep 21 2024 at 08:05):

this also means I will need to update a few other services with Gmail SMTP, since end of september they drop the basic auth method

Oliver Bertuch (Sep 21 2024 at 08:06):

You could try using Sasl Oauth2

Oliver Bertuch (Sep 21 2024 at 08:06):

It should be possibly to configure this since v6.2

Tamas Gal (Sep 21 2024 at 08:06):

aah that's also an option

Tamas Gal (Sep 21 2024 at 08:06):

very nice

Oliver Bertuch (Sep 21 2024 at 08:06):

It would also be helpful to document your findings in the guide

Oliver Bertuch (Sep 21 2024 at 08:06):

So others can benefit, too

Tamas Gal (Sep 21 2024 at 08:08):

yes, definitely

Oliver Bertuch (Sep 21 2024 at 08:08):

Keep in mind that there is a debug switch in the mail settings. It will make the logs more verbose in case you need to do some diggin'

Tamas Gal (Sep 21 2024 at 08:08):

Oliver Bertuch (Sep 21 2024 at 08:08):

Happy hacking!

Tamas Gal (Sep 21 2024 at 08:08):

currently the only annoying thing is that i need to restart the full service, which takes about 10 minutes to start up

Tamas Gal (Sep 21 2024 at 08:08):

thanks ;)

Oliver Bertuch (Sep 21 2024 at 08:09):

10 minutes????

Tamas Gal (Sep 21 2024 at 08:09):

yes roughly

Oliver Bertuch (Sep 21 2024 at 08:09):

:exploding_head:

Tamas Gal (Sep 21 2024 at 08:09):

I see the Payara page for a very long time until the actual Dataverse appears

Oliver Bertuch (Sep 21 2024 at 08:09):

Man you need some beef on that machine!

Tamas Gal (Sep 21 2024 at 08:09):

oh i have

Oliver Bertuch (Sep 21 2024 at 08:09):

It usually takes about 2 minutes

Tamas Gal (Sep 21 2024 at 08:09):

128 corese, 512GB RAM

Oliver Bertuch (Sep 21 2024 at 08:09):

10 is very odd!

Tamas Gal (Sep 21 2024 at 08:09):

currently not restricted at all

Oliver Bertuch (Sep 21 2024 at 08:10):

Can you say from the timestamps in the logs where it slows down?

Oliver Bertuch (Sep 21 2024 at 08:10):

There might be some timeout for sth

Oliver Bertuch (Sep 21 2024 at 08:11):

Usually I redeploy on my laptop in under 2 minutes, full cycle

Tamas Gal (Sep 21 2024 at 08:11):

ah wait, i am an idiot, i have restricted it to 2GB RAM and 1 CPU :laughing:

Tamas Gal (Sep 21 2024 at 08:11):

Screenshot-2024-09-21-at-10.11.37.png

Oliver Bertuch (Sep 21 2024 at 08:11):

That's the default for the container, yes

Tamas Gal (Sep 21 2024 at 08:12):

i totally missed that...

Tamas Gal (Sep 21 2024 at 08:12):

ok so it will utilise all the CPUs if i give more, or is there a maximum?

Oliver Bertuch (Sep 21 2024 at 08:13):

If you don't give any restrictions, it will take and run with what's there

Tamas Gal (Sep 21 2024 at 08:13):

perfect

Oliver Bertuch (Sep 21 2024 at 08:13):

Beware that you really should limit the Ram

Oliver Bertuch (Sep 21 2024 at 08:13):

Otherwise it will allocate a very large heap, 70% of what the container has available

Oliver Bertuch (Sep 21 2024 at 08:13):

You won't want that to happen :sweat_smile:

Oliver Bertuch (Sep 21 2024 at 08:15):

A reasonable size to getting started would be 4g or 8g. Probably @Don Sizemore ears are burning now

Tamas Gal (Sep 21 2024 at 08:15):

yes don't worry :D

Tamas Gal (Sep 21 2024 at 08:15):

i started with 8GB and 4CPUs, initialises much faster

Tamas Gal (Sep 21 2024 at 08:15):

SMTP works too

Tamas Gal (Sep 21 2024 at 08:15):

one last thing: the URL is still localhost:8080

Tamas Gal (Sep 21 2024 at 08:16):

Screenshot-2024-09-21-at-10.14.57.png

Tamas Gal (Sep 21 2024 at 08:16):

i could not figure out yet where to change that ;)

Tamas Gal (Sep 21 2024 at 08:17):

i have a commented out _CT_DATAVERSE_SITEURL env var

Tamas Gal (Sep 21 2024 at 08:17):

but don't remember anymore what it was commented out :sweat_smile:

Oliver Bertuch (Sep 21 2024 at 08:17):

If you're running 6.3 with Payara 6.2024.6 you don't need the _ct thing anymore

Tamas Gal (Sep 21 2024 at 08:17):

https://guides.dataverse.org/en/latest/installation/config.html

Tamas Gal (Sep 21 2024 at 08:17):

ok so i use that with DATAVERSE_SITEURL as env var,

Oliver Bertuch (Sep 21 2024 at 08:17):

You can just use Dataverse_siteurl and dataverse_fqdn now

Tamas Gal (Sep 21 2024 at 08:17):

Oliver Bertuch (Sep 21 2024 at 08:18):

All uppercase please. Too hard to type on mobile

Tamas Gal (Sep 21 2024 at 08:18):

yesyes :) thanks, just confirming

Notification Bot (Sep 21 2024 at 08:19):

Tamas Gal has marked this topic as resolved.

Oliver Bertuch (Sep 21 2024 at 08:22):

One last thing: if you would like to see a reload admin api endpoint to trigger re-init of subsystems like mail etc, please create an issue! I've been thinking about this for some time. Not sure when it would get implemented, but it certainly would help. Might become relevant for SPA work which makes it more likely to be implemented quicker

Tamas Gal (Sep 21 2024 at 08:24):

Yes that might be something but not sure if it's worth the hassle. I mean, it might save one or two hours of deployment work but that's usually a one-time thing ;)

Tamas Gal (Sep 21 2024 at 08:26):

everything works now :) deployed in Docker Swarm successfully

Tamas Gal (Sep 21 2024 at 08:28):

sorry for being off-topic but the bootstrap service is still trying every few seconds, I guess i can simply remove it now? i understood that it's only needed for the very first deployment

Tamas Gal (Sep 21 2024 at 08:28):

Screenshot-2024-09-21-at-10.28.44.png

Oliver Bertuch (Sep 21 2024 at 08:29):

It might be nice to put down your experiences with Swarm. It's a nice compromise between compose and k8s

Tamas Gal (Sep 21 2024 at 08:29):

i will check out how to contribute to your docs in the next days

Oliver Bertuch (Sep 21 2024 at 08:29):

The bootstrap will detect it doesn't need to do anything and quit. If it bothers you, drop it.

Tamas Gal (Sep 21 2024 at 08:30):

  bootstrap:
    image: gdcc/configbaker:alpha
    environment:
      - TIMEOUT=3m
    command:
      - bootstrap.sh
      - dev

Tamas Gal (Sep 21 2024 at 08:30):

that's the service right now. not sure if dev is correct in production

Oliver Bertuch (Sep 21 2024 at 08:30):

No it usually isn't 1:laughing:

Tamas Gal (Sep 21 2024 at 08:30):

ok ;) so remove dev and let it cycle? ;)

Oliver Bertuch (Sep 21 2024 at 08:30):

You really shouldn't :grimacing:

Oliver Bertuch (Sep 21 2024 at 08:31):

No, you need to provide a different profile

Oliver Bertuch (Sep 21 2024 at 08:31):

IIRC the demo guides page had something about this documented

Tamas Gal (Sep 21 2024 at 08:31):

ok i will check

Oliver Bertuch (Sep 21 2024 at 08:32):

We are officially not yet production ready with all of this. Still a long way to go to make it easily accessible and polished enough

Oliver Bertuch (Sep 21 2024 at 08:32):

You're breaking new ground :wink:

Tamas Gal (Sep 21 2024 at 08:32):

ahah ok ;)

Tamas Gal (Sep 21 2024 at 08:37):

i only found this about bootstrap: https://guides.dataverse.org/en/latest/container/running/demo.html# and it lists dev and demo
i am a bit confused but I think demois now the correct choice? or is bootstrapping not needed at all since settings are already settled? maybe I just don't understand if "boostrapping" is related to the very first deployment (setting up databases, directories etc.) or the actual start of the docker service stack...

Tamas Gal (Sep 21 2024 at 08:38):

especially this sentence:

One of the main differences between the “dev” persona and our new “demo” persona is that we are now running the setup-all script without the --insecure flag. This makes our installation more secure, though it does block “admin” APIs that are useful for configuration.