Oliver Bertuch (Aug 14 2023 at 13:15): Oliver Bertuch (Aug 14 2023 at 13:15):A note: we could use the REST API to change logging levels. The configbaker image could carry scripts with well known / often used parts of Dataverse and accept list of stuff where to enable / disable logs.
Johannes D (Aug 17 2023 at 13:10):I'd rather like to have means to configure the log levels via a documented ENV variable...
Oliver Bertuch (Aug 17 2023 at 13:11):How should such an env var look like?
Oliver Bertuch (Aug 17 2023 at 13:13):And would you be OK with rebooting the container or should it be possible to alter logging while running?
Johannes D (Aug 17 2023 at 13:16):I would love spring like syntax...https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#appendix.application-properties
Oliver Bertuch (Aug 17 2023 at 13:20):Could you sketch an example?
Johannes D (Aug 17 2023 at 13:22):Similar to this [1]. However, I'm not sure if that works with current configuration slf4j-jdk11. It does work with log4j and logback...
Philip Durbin 🚀 (Aug 17 2023 at 13:25):I think @Don Sizemore does something like that with a properties file. Not sure.
Oliver Bertuch (Aug 17 2023 at 13:25):Yeah, there is domain1/config/logging.properties.
Philip Durbin 🚀 (Aug 17 2023 at 13:27):I have it on the agenda: https://ct.gdcc.io
Oliver Bertuch (Aug 17 2023 at 13:28):3 minutes! :clock:
Johannes D (Aug 17 2023 at 13:29):Initially, I would be sufficient to configure the input for as-admin set-log-level via env vars. This requires a restart but already eases the container debugging..
Oliver Bertuch (Aug 17 2023 at 13:36):@Johannes D have your say during the meeting! We're on now :smile:
Don Sizemore (Aug 17 2023 at 13:47):Johannes D said:
Initially, I would be sufficient to configure the input for
as-admin set-log-levelvia env vars. This requires a restart but already eases the container debugging..
to my experience, Glassfish and (now) Payara pick up changes to logging.properties on-the-fly, unless you're averse to cat and/or awk/sed?
Oliver Bertuch (Aug 17 2023 at 13:49):O rly? On the fly?
Johannes D (Aug 17 2023 at 13:51):If thats the case, we could just update the init_2_configure.sh script to echo an ENV var to the file, right?
Oliver Bertuch (Aug 17 2023 at 13:59):We probably put this in the base image, not the app image script. This is useful for more than just the app image.
Oliver Bertuch (Aug 17 2023 at 13:59):One thing we need to take care of: we must not "just write in the file". This will write into the overlayfs, which is a baaaaad idea
Oliver Bertuch (Aug 17 2023 at 14:37):So here's a thought. We could make a script sit in the background within the container, watch for some logging config to appear somewhere (on K8s that would be a mounted configmap key or similar, autoupdating on change), merge it with the original and replace it. I already put the infrastructure in place to do such things in a safe way with the init system in place. @Johannes D @Don Sizemore what do you think?
Johannes D (Aug 18 2023 at 06:36):@Oliver Bertuch Sound good!
Philip Durbin 🚀 (Aug 31 2023 at 09:53):Yesterday, I made PR #9863 that changes log levels (as a workaround for #9806). I used asadmin, like normal.
Philip Durbin 🚀 (Sep 07 2023 at 10:49):Should we have an issue or at least a note to track what we want to change? Use the REST API to change logging levels? Is that it?
Oliver Bertuch (Sep 07 2023 at 10:58):Issues sounds good :-)
Philip Durbin 🚀 (Sep 07 2023 at 11:01):Would you like to do the honors? :big_smile:
Don Sizemore (Sep 08 2023 at 11:45):Oliver Bertuch said:
So here's a thought. We could make a script sit in the background within the container, watch for some logging config to appear somewhere (on K8s that would be a mounted configmap key or similar, autoupdating on change), merge it with the original and replace it. I already put the infrastructure in place to do such things in a safe way with the init system in place. Johannes D Don Sizemore what do you think?
sounds great to me. in my experience Payara picks up changes to logging.properties on the fly, so your changes could be as granular (numerous) as needed.
Philip Durbin 🚀 (Apr 11 2024 at 13:53):We're talking about this again. :grinning: See https://docs.google.com/document/d/1toiFxJKGvrYMy9KrW9GZ4U6FLGhMaCipJ9ggtvZFwgc/edit?usp=sharing
Philip Durbin 🚀 (Apr 11 2024 at 15:42):Here's the recording: https://harvard.zoom.us/rec/share/LtMO_BjkgEHvqdgjnCv9V0ko6_I3L-VZ-wZapO6JiZf9aFRSs7tkDP182plACuus.fnTfrsTArCtjKFng
Oliver Bertuch (Jul 10 2024 at 18:02):I'm investigating adding a JUL to SLF4J bridge to make use of Logback for the server logs. Logback supports hot-reloading logging config files as well as MDC etc.
Oliver Bertuch (Jul 10 2024 at 18:02):It would be neat if we make Dataverse use SLF4J instead of JUL, too.
Oliver Bertuch (Jul 10 2024 at 18:03):That way we could make use enabling debugging stream in production more selectively, e.g. for a single user.
Oliver Bertuch (Jul 10 2024 at 18:04):(Reason for me looking into this: I want to disable warnings about missing property files for our custom metadata block where we just don't care...)
Philip Durbin 🚀 (Jul 10 2024 at 18:49):But JUL is the standard. Shouldn't we use the standard? Fewer dependencies.
Oliver Bertuch (Jul 10 2024 at 19:17):A standard with flaws...
Oliver Bertuch (Jul 10 2024 at 19:18):We need SLF4J anyway. Best practice for any library, plugin etc is to use a logging facade. Usually that is SLF4J
Oliver Bertuch (Jul 11 2024 at 07:44):If we're talking standards and less deps: one should use System.Logger then :wink:
Oliver Bertuch (Jul 11 2024 at 07:44):Yesterday I really dug deep into this. OSGi and everything.
Oliver Bertuch (Jul 11 2024 at 07:45):It looks like we cannot use the SLF4JBridge handler without getting the application involved.
Oliver Bertuch (Jul 11 2024 at 07:47):The logging is initialized very early in the boot process from glassfish/modules/logging.jar, which is an OSGi module. Looking into that one reveals that it has a MANIFEST that declares a certain class path and it seems impossible to make some other lib appear on that path from the outside. (At least I don't know how to do that.)
Oliver Bertuch (Jul 11 2024 at 07:48):I even went ahead and forked https://github.com/goodees/goodees/blob/master/payara-logback/README.adoc into a more modern version, but no luck to inject it.
Oliver Bertuch (Jul 11 2024 at 07:50):It seems we're stuck with JUL... Glassfish completely rewrote their logging subsystem to use "GlassFish Java Util Logging Extension", see also https://www.youtube.com/watch?v=j2gACdwWyNk
Oliver Bertuch (Jul 11 2024 at 08:13):What we can do though is to enable switch between log formats, e.g. from ULF to JSON. JSON logs are much better to handle when using fluentd etc to ship the logs to analysis. We need to set java.util.logging.ConsoleHandler.formatter using env var support. Defaulting to com.sun.enterprise.server.logging.UniformLogFormatter, obviously...
Oliver Bertuch (Jul 11 2024 at 09:59):Notes to self:
Philip Durbin 🚀 (Jul 12 2024 at 14:53):From that video: "For GlassFish 7 we completely changed the logging system". Interesting. Does that apply to us, though? Since we're on Payara?
Oliver Bertuch (Jul 12 2024 at 23:27):I might have been doing it wrong with how I added the packages... Maybe I need to retry with what is written here: https://docs.payara.fish/community/docs/Technical%20Documentation/Application%20Development/Class%20Loaders.html#extension-libraries
Oliver Bertuch (Jul 12 2024 at 23:28):Also, I did add a modules.java - maybe I sealed the package by accident, so it could not load any classes from it.
Oliver Bertuch (May 08 2025 at 20:35):I've been looking into Observability as of late. A part of OpenTelemetry infrastructure like Sentry, Loki etc is sending these tools logs.
Oliver Bertuch (May 08 2025 at 20:35):It makes a lot of sense to enable such things in containers, as all of these tools are often around in a container production environment, especially when running on K8s
Philip Durbin 🚀 (May 08 2025 at 20:36):Nice. I think this topic was originally about changing log levels on the fly. I still don't know how to do this. 
Oliver Bertuch (May 08 2025 at 20:36):So now I'm wondering if I should use the JUl-to-SFL4J bridge to re-route all messages from JUL to Logback
Oliver Bertuch (May 08 2025 at 20:37):With Logback, you can use JMX to change the logging levels on the fly without app restarts
Philip Durbin 🚀 (May 08 2025 at 20:37):orly
Oliver Bertuch (May 08 2025 at 20:37):With Logback, you can use a config file which can be reloaded dynamically
Oliver Bertuch (May 08 2025 at 20:37):So maybe that gives us leverage to support all of these cases!
Oliver Bertuch (May 08 2025 at 20:38):So I'd say we could include https://github.com/jiaqi/jmxterm in the image / configbaker or maybe even add REST Admin API endpoints that change the levels using JMX
Oliver Bertuch (May 08 2025 at 20:39):We definitely need to make use of a proper logging library for the non-standard logs of Dataverse... :grimacing:
Oliver Bertuch (May 08 2025 at 20:40):Otherwise it's hard to get them out. Or at least make sure the path where they are written is configurable, so one can use a sidecar container to ship them.
Oliver Bertuch (May 08 2025 at 20:40):Writing these logs to the overlayfs is a baaaaad idea
Philip Durbin 🚀 (May 08 2025 at 20:45):Sure, should we take this to #dev? Or start with a GitHub issue to log properly?
Oliver Bertuch (May 08 2025 at 20:46):Let's just keep this here for a moment. This is mostly a brain dump, so I can circle back later.
Oliver Bertuch (May 08 2025 at 20:54):@Philip Durbin ☀️ Do you think we could get a PR into Dataverse that changes all loggers from JUL to SLF4J?
Oliver Bertuch (May 08 2025 at 20:54):That would spare us some of the hassle (not all, as Payara uses JUL internally)
Philip Durbin 🚀 (May 08 2025 at 20:55):JUL is the standard but it's no good?
Oliver Bertuch (May 08 2025 at 20:55):Technically JUL is an implementation.
Oliver Bertuch (May 08 2025 at 20:55):It's a part of the JDK since 1.4
Oliver Bertuch (May 08 2025 at 20:56):These days, if you want it in a Jakarta like way, the industry standard is SLF4J.
Oliver Bertuch (May 08 2025 at 20:56):That's the facade and there are plenty of implementations around to send it to some concrete log system
Philip Durbin 🚀 (May 08 2025 at 21:03):Are we doing it right in https://github.com/IQSS/dataverse-hub or https://github.com/IQSS/dataverse-marketplace ? /cc @Juan Pablo Tosca Villanueva
Oliver Bertuch (May 08 2025 at 22:04):Note to self: keep https://github.com/goodees/goodees/tree/master/payara-logback in mind and OpenTelemetry log shipping.
Oliver Bertuch (May 19 2025 at 18:19):So I've pogged and prodded on this a bit today, trying to use Claude 3.7 to mix together an HK2 model to load SLF4J bridge plus Logback. So far, no success. Might give this tomorrow another shot. If this doesn't work out, I'm wondering if I should just create a patched Payara version that ships with the necessary libs added to the core.
Oliver Bertuch (May 19 2025 at 18:20):@pdurbin is that overkill, creating such a patch? Not sure if the upstream would accept it... they don't seem very interested in this.
Oliver Bertuch (May 19 2025 at 18:22):We could of course create an issue for this and vote vote vote vote. :grinning_face_with_smiling_eyes:
Philip Durbin 🚀 (May 19 2025 at 18:23):whatever you do, it would be nice to get it into upstream
Oliver Bertuch (May 19 2025 at 21:41):PROGRESS!
Oliver Bertuch (May 19 2025 at 21:41):[#|2025-05-19T21:34:59.403+0000|SEVERE|Payara 6.2025.3|jakarta.enterprise.logging|_ThreadID=34;_ThreadName=RunLevelControllerThread-1747690499218;_TimeMillis=1747690499403;_LevelValue=1000;_MessageID=NCLS-LOGGING-00002;|
Could not apply the logging configuration changes.
MultiException stack 1 of 2
com.sun.enterprise.module.ResolveError: Failed to start OSGiModuleImpl:: Bundle = [io.gdcc.payara-logback-hk2 [396]], State = [NEW]
[...]
Caused by: org.osgi.framework.BundleException: Unable to resolve io.gdcc.payara-logback-hk2 [396](R 396.0): missing requirement [io.gdcc.payara-logback-hk2 [396](R 396.0)] osgi.wiring.package; (&(osgi.wiring.package=org.slf4j)(version>=2.0.0)(!(version>=3.0.0))) Unresolved requirements: [[io.gdcc.payara-logback-hk2 [396](R 396.0)] osgi.wiring.package; (&(osgi.wiring.package=org.slf4j)(version>=2.0.0)(!(version>=3.0.0)))]
This stuff is hard.
Oliver Bertuch (May 19 2025 at 21:42):YESSSS
Oliver Bertuch (May 19 2025 at 21:42):#!## LogManagerService.postConstruct : rootFolder=/opt/payara/appserver/glassfish
#!## LogManagerService.postConstruct : templateDir=/opt/payara/appserver/glassfish/lib/templates
#!## LogManagerService.postConstruct : src=/opt/payara/appserver/glassfish/lib/templates/logging.properties
#!## LogManagerService.postConstruct : dest=/opt/payara/appserver/glassfish/domains/domain1/config/logging.properties
SLF4J Bridge bundle is starting...
Installing SLF4J Bridge Handler
Go go go! 
Oliver Bertuch (Aug 01 2025 at 11:18):Note to self: apparently the class loading didn't properly work out. Maybe try again with an Extension Library and add to bootdelegation? https://docs.payara.fish/community/docs/6.2025.7/Technical%20Documentation/Application%20Development/Class%20Loaders.html#installing-libraries
Last updated: Oct 30 2025 at 05:14 UTC