Stream: troubleshooting

Topic: ✔ Dataverse crashes on Postgres restart

view this post on Zulip Santiago Florez (Jul 03 2024 at 22:10):

Hello everyone, today I was informed that my Dataverse was down. This image shows the error message when I tried to access Dataverse

imagen_2024-07-03_165236903.png

After a while, Dataverse was up. I took a look at the logs while Postgres was restarting. This is the server.log

I think it is about connection pool. Someone know What can I do? Or What I need to check?

view this post on Zulip Henning Timm (Jul 08 2024 at 11:53):

I have the same issue after updating postgres packages in my (Debian) system. So far I have restarted the server to get it running again. So I would be interested in this as well ;)

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 12:32):

Have you tried some of these? https://guides.dataverse.org/en/latest/installation/config.html#advanced-database-settings There are also links to more docs by Payara.

view this post on Zulip Philip Durbin 🚀 (Jul 08 2024 at 13:34):

In general, yes, I've found that if PostgreSQL goes down I need to restart Payara to re-establish the connection.

I do like the sound of "allow server to reconnect in case of failure" in the link from Oliver above.

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 13:36):

Yes, that's the one I'd try first

view this post on Zulip Santiago Florez (Jul 08 2024 at 19:46):

@Oliver Bertuch and @Philip Durbin, thanks for the response. I will check the documentation that Oliver has shared. If I get it, I will let you know.

view this post on Zulip Philip Durbin 🚀 (Jul 08 2024 at 19:49):

Good luck!

view this post on Zulip Santiago Florez (Jul 08 2024 at 21:08):

I have configured the next property <system-property name="dataverse.db.is-connection-validation-required" value="true"></system-property>, in my domain.xml file, I was restarting Payara and got the next error:

RAR5114 : Error allocating connection : [Error in allocating a connection. Cause: Cannot invoke "String.toLowerCase(java.util.Locale)" because the return value of "com.sun.gjc.common.DataSourceSpec.getDetail(int)" is null]]]

[2024-07-08T15:53:18.454-0500] [Payara 6.2023.8] [GRAVE] [] [javax.enterprise.system.tools.deployment.common] [tid: _ThreadID=32 _ThreadName=RunLevelControllerThread-1720471969883] [timeMillis: 1720471998454] [levelValue: 1000] [[
  Exception while invoking class org.glassfish.ejb.startup.EjbApplication start method
jakarta.ejb.EJBException: jakarta.ejb.CreateException: Initialization failed for Singleton StartupFlywayMigrator
    at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:745)
    at com.sun.ejb.containers.AbstractSingletonContainer.instantiateSingletonInstance(AbstractSingletonContainer.java:477)
    at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:219)
    at org.glassfish.ejb.startup.SingletonLifeCycleManager.initializeSingleton(SingletonLifeCycleManager.java:180)
    at org.glassfish.ejb.startup.SingletonLifeCycleManager.doStartup(SingletonLifeCycleManager.java:159)
    at org.glassfish.ejb.startup.EjbApplication.start(EjbApplication.java:171)
    at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123)
    at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:292)
    at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:361)
    at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:633)
    at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:346)
    at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:303)
    at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:351)
    at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
    at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:281)
    at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
    at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
    at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1213)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1144)
    at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:762)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: jakarta.ejb.CreateException: Initialization failed for Singleton StartupFlywayMigrator
    at com.sun.ejb.containers.AbstractSingletonContainer.createSingletonEJB(AbstractSingletonContainer.java:598)
    at com.sun.ejb.containers.AbstractSingletonContainer$SingletonContextFactory.create(AbstractSingletonContainer.java:743)
    ... 24 more
Caused by: org.flywaydb.core.internal.exception.FlywaySqlException: Unable to obtain connection from database: Error in allocating a connection. Cause: Cannot invoke "String.toLowerCase(java.util.Locale)" because the return value of "com.sun.gjc.common.DataSourceSpec.getDetail(int)" is null

This is my domain.xml file configuration.
image.png

view this post on Zulip Santiago Florez (Jul 08 2024 at 21:09):

Is that the way to set the property, or should I set it as a JVM option?

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 21:11):

You will need to define the method how to validate, too, I think. There is no default selection.

view this post on Zulip Santiago Florez (Jul 08 2024 at 21:13):

@Oliver Bertuch do you mean this dataverse.db.connection-validation-method property?

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 21:27):

Can you try with custom validation and set the class to org.glassfish.api.jdbc.validation.PostgresConnectionValidation?

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 21:27):

I'm curious if that works...

view this post on Zulip Santiago Florez (Jul 08 2024 at 22:00):

@Oliver Bertuch I have configured my domain.xml file as follows:
image.png

First, I restarted Payara, and it loaded correctly, and then I restarted Postgresql, to validate that everything is ok, and it works. So, thank you very much for helping me.

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 22:01):

Well it's not what I asked, but glad it seems to work for you!

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 22:01):

Please let us know if your problems reappear

view this post on Zulip Oliver Bertuch (Jul 08 2024 at 22:02):

In case it's smooth sailin' from now on, we might include these things in the docs as useful tips

view this post on Zulip Philip Durbin 🚀 (Jul 09 2024 at 12:42):

Docs, docs, docs.

view this post on Zulip Notification Bot (Jul 11 2024 at 06:42):

Oliver Bertuch has marked this topic as resolved.

Last updated: Oct 30 2025 at 06:21 UTC