Oliver Bertuch (May 10 2023 at 14:43): Oliver Bertuch (May 10 2023 at 14:43):It seems like docker:watch is not an option for now - it complains it doesn't know a thing about assemblies added to the image and bails out. Looks like a bug due to our very special config with compose
Philip Durbin π (May 10 2023 at 14:44):bummer
Oliver Bertuch (May 10 2023 at 14:47):Haha easy, it's simply not possible. Either go for all or die. Maybe with compose
Oliver Bertuch (May 10 2023 at 14:59):Ha works with compose! Tricky tricky tricky
Oliver Bertuch (May 10 2023 at 15:09):OK I pushed some notes about that as well
Oliver Bertuch (May 10 2023 at 15:13):https://dataverse-guide--9584.org.readthedocs.build/en/9584/container/dev-usage.html#re-deploying
Philip Durbin π (May 10 2023 at 15:15):Looks good. You've documented what I'm doing, a full clean and build each time. Thanks.
Oliver Bertuch (May 10 2023 at 15:16):It's the hard way :smiley:
Oliver Bertuch (May 10 2023 at 15:16):Hopefully we can have something else some other time
Philip Durbin π (May 10 2023 at 15:17):The only way, currently. Oh well. At least I'm not missing out on anything. :sweat_smile: #fomo
Oliver Bertuch (May 10 2023 at 15:17):Let me try something with mounting as a volume
Oliver Bertuch (May 10 2023 at 15:40):Ok that seems to have no effect - the files don't get picked up. Looks like this is not working in their layout...
Oliver Bertuch (May 10 2023 at 15:41):But I do see a hot redeploy thing using their plugins for IDEs... Maybe that's a thing? Let's give it a shot!
Oliver Bertuch (May 10 2023 at 16:07):Ha that works astonishingly well!
Oliver Bertuch (May 10 2023 at 16:07):Just one problem: I need to ship a WAR file. We build that as "dataverse-5.13.war". That conflicts with our deployment name (dataverse)
Oliver Bertuch (May 10 2023 at 16:08):If I rename the war, all goes well
Oliver Bertuch (May 10 2023 at 16:08):And it actually is pretty fast
Oliver Bertuch (May 10 2023 at 16:08):Takes about 10 seconds on my machine to redeploy
Oliver Bertuch (May 10 2023 at 16:09):I'm going to open a new topic for this and move messages...
Oliver Bertuch (May 10 2023 at 16:10):I followed this guide (as I am on IntelliJ) https://blog.payara.fish/payara-server-basics-adding-payara-server-to-intellij-idea
Oliver Bertuch (May 10 2023 at 16:11):But I see plenty of other plugins for Eclipse, Netbeans, VSCode in the docs: https://docs.payara.fish/community/docs/Technical%20Documentation/Ecosystem/IDE%20Integration/IntelliJ%20Plugin/Payara%20Server.html#deploying-application-payara-server-from-intellij
Philip Durbin π (May 10 2023 at 16:35):Do you think it'll work with the free (community edition) version of Intellij?
Oliver Bertuch (May 10 2023 at 17:06):No it won't. Explicitly says Ultimate only
Oliver Bertuch (May 10 2023 at 17:07):But there's Eclipse, Netbeans, Vscode so plenty of free options
Philip Durbin π (May 10 2023 at 18:42):I took a quick look at the docs. @Oliver Bertuch maybe you can give me a demo of your 10 second redeploy sometime. We could even show it during tomorrow's container meeting if you like.
Oliver Bertuch (May 10 2023 at 18:44):I'm free to do whatever I want now - just tucked in the kids
Philip Durbin π (May 10 2023 at 18:48):Nice. I added it to the agenda. Hope you don't mind: https://docs.google.com/document/d/1eQVm88dP2rgM9DKn4ivoWBx6MOK6aXfkLhsZN-Y3fsc/edit?usp=sharing
Oliver Bertuch (May 10 2023 at 18:52):Meaning you don't want a demo now?
Philip Durbin π (May 10 2023 at 19:50):I'd have to find a room. Plus if we do it tomorrow, it'll be recorded. Let me know. I can find a room if you want to demo it now.
Oliver Bertuch (May 10 2023 at 19:59):LOL - I tried with VSCode. That did not work _at all_. Great. "Cannot read properties of undefined (reading '0')" :rolling_eyes: And the only thing kind of similar I could find was this https://github.com/payara/ecosystem-support/issues/33
Oliver Bertuch (May 10 2023 at 22:14):OK 10 seconds was a very subjective impression. It's ~30 seconds (excluding compile/package time):
[2023-05-11 12:13:11,007] Artifact dataverse:war: Artifact is being deployed, please wait...
Connected to server
[2023-05-11 12:13:39,266] Artifact dataverse:war: Artifact is deployed successfully
[2023-05-11 12:13:39,266] Artifact dataverse:war: Deploy took 28,259 milliseconds
[2023-05-11 12:13:40,708] Artifact dataverse:war: Artifact is deployed successfully
64 seconds as of this comment: https://github.com/IQSS/dataverse/issues/5593#issuecomment-786190813
Oliver Bertuch (May 10 2023 at 22:36):Yeah, starting all the DB and index containers does take some time.
Philip Durbin π (May 10 2023 at 22:37):I'm looking forward to the demo.
Philip Durbin π (May 10 2023 at 22:39):How long would it take for https://github.com/IQSS/dataverse-people ?
Philip Durbin π (May 10 2023 at 22:41):It's basically the hello world app described in #5736. (I already left a comment there about it.)
Philip Durbin π (May 11 2023 at 02:58):Does ENABLE_RELOAD work? From https://guides.dataverse.org/en/5.13/container/base-image.html#tunables
Philip Durbin π (May 11 2023 at 02:58):I tried to use it here: https://github.com/IQSS/dataverse-people/blob/6bd128e678cd1a9a580da7ffebf0e80cf616cfba/dataverse-people/wad-docker.sh#L13
Oliver Bertuch (May 11 2023 at 08:36):Philip Durbin said:
Does ENABLE_RELOAD work? From https://guides.dataverse.org/en/5.13/container/base-image.html#tunables
I looked into this again and noticed that I actually might have forgotten to enable a certain setting. I will need to adapt the base image for that!
Oliver Bertuch (May 11 2023 at 08:37):I probably should go ahead and create a new issue and PR for faster redeploy...
Philip Durbin π (May 11 2023 at 09:41):Should I create an issue about ENABLE_RELOAD?
Oliver Bertuch (May 11 2023 at 11:04):Well not if we can't reproduce it. I have a fix here but I'm searching for a test
Oliver Bertuch (May 11 2023 at 11:04):(The fix is basically to re-enable also the second setting...)
Oliver Bertuch (May 11 2023 at 11:30):So mounting the files and touching ".reload" does not seem to do the trick...
Oliver Bertuch (May 11 2023 at 11:30):Even with a fixed setting and JSF set to Development status
Philip Durbin π (May 11 2023 at 11:34):You can use that wad docker commit above to test ^^
Philip Durbin π (May 11 2023 at 11:34):The war file is being updated in the deployments dir
Philip Durbin π (May 11 2023 at 11:34):but it isn't being deployed
Oliver Bertuch (May 11 2023 at 11:48):I might have a way to configure that... The autodeploy thing is usually under the domain root
Oliver Bertuch (May 11 2023 at 11:48):Just found out the setting to change that location
Oliver Bertuch (May 11 2023 at 11:51):And we might also need to change the path to the app dir - that is by default also under the domain dir
Oliver Bertuch (May 11 2023 at 11:51):Otherwise using .reload won't do anything
Oliver Bertuch (May 11 2023 at 11:54):Ah DANG we can't do that - there are internal apps like the EJB timer etc in there
Oliver Bertuch (May 11 2023 at 11:54):Dunno if a symlink will be followed, but we can try :-D
Philip Durbin π (May 11 2023 at 11:56):Do you think redeploy will work with an official Payara image? (I can't run these on my M2 Mac: https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/helping.20Payara.20with.20M1.2FM2.20builds )
Philip Durbin π (May 11 2023 at 11:59):(I tried with Open Liberty first but got a bit lost.)
Philip Durbin π (May 11 2023 at 12:00):(because Open Liberty is what's used in the blog post and video I was following: https://blog.sebastian-daschner.com/entries/reloading-javaee-apps-wad )
Oliver Bertuch (May 11 2023 at 12:00):I disbelieve that - I'm using the same scripting logic, so why would that be any different?
Philip Durbin π (May 11 2023 at 12:01):Ok, maybe I'll try to get it working with Open Liberty, then.
Philip Durbin π (May 11 2023 at 12:01):The video is pretty compelling: https://www.youtube.com/watch?v=V_haSihHHV8
Philip Durbin π (May 11 2023 at 12:02):And it's IDE-independent, which I like.
Oliver Bertuch (May 11 2023 at 12:03):You did see
image.png
Oliver Bertuch (May 11 2023 at 12:03):That directory is the important bit...
Philip Durbin π (May 11 2023 at 12:04):Yes, I saw that. Again, I can't try the official Payara image on my M2.
Oliver Bertuch (May 11 2023 at 12:04):Ok but it's the same thing.
Oliver Bertuch (May 11 2023 at 12:04):Both images are the same server
Oliver Bertuch (May 11 2023 at 12:04):Same structure
Oliver Bertuch (May 11 2023 at 12:05):I did copy most of the init scripting anyway
Philip Durbin π (May 11 2023 at 12:05):Are you going to try my commit or not? :happy:
Oliver Bertuch (May 11 2023 at 12:05):Commit?
Oliver Bertuch (May 11 2023 at 12:05):Link link link
Oliver Bertuch (May 11 2023 at 12:05):Ah I see
Philip Durbin π (May 11 2023 at 12:05): Oliver Bertuch (May 11 2023 at 12:06):Philip Durbin said:
I tried to use it here: https://github.com/IQSS/dataverse-people/blob/6bd128e678cd1a9a580da7ffebf0e80cf616cfba/dataverse-people/wad-docker.sh#L13
Philip Durbin π (May 11 2023 at 12:06):That's your reproducer. Nice and small. :happy:
Oliver Bertuch (May 11 2023 at 12:07):I won't try it yet :-D We need to reconfigure the app server first to pickup modules with autodeploy from the other directory first
Philip Durbin π (May 11 2023 at 12:07):I had to rename the "backend" directory to "dataverse-people" to make wad.jar happy.
Oliver Bertuch (May 11 2023 at 12:07):Otherwise you can try very very very long :wink:
Oliver Bertuch (May 11 2023 at 12:08):Oh you could try something. Mount to /opt/payara/appserver/glassfish/domains/domain1/autodeploy for me
Philip Durbin π (May 11 2023 at 12:08):I see. Are you saying in our base image Payara will do deployments from two directories? The official one and the one we added?
Oliver Bertuch (May 11 2023 at 12:08):Yes and no
Oliver Bertuch (May 11 2023 at 12:09):What we do is manually craft a deployment and send that Payaras way. It basically is a "deploy" command
Oliver Bertuch (May 11 2023 at 12:09):That can source any dir / file whatever
Oliver Bertuch (May 11 2023 at 12:09):But autodeploy has a scanner that looks into a certain directory (the one above)
Oliver Bertuch (May 11 2023 at 12:10):Ah dang if you try this you need to reenable autodeploy as well... Hmm you can do that once the server runs via asadmin I believe.
Oliver Bertuch (May 11 2023 at 12:11):Or go via https://localhost:4848 and enable it from there. If it needs a restart, it should be possible to do that from there too
Oliver Bertuch (May 11 2023 at 12:12):
Philip Durbin π (May 11 2023 at 12:12):This is the diff you want in my wad docker script?
-DEPLOY_DIR=/opt/payara/deployments
+DEPLOY_DIR=/opt/payara/appserver/glassfish/domains/domain1/autodeploy
Yes
Philip Durbin π (May 11 2023 at 12:16):Doesn't work. I can see the war file changing...
payara@4ef27fd1982b:~$ ls -l /opt/payara/appserver/glassfish/domains/domain1/autodeploy
total 4
-rw-r--r-- 1 payara payara 3456 May 11 12:13 dataverse-people.war
payara@4ef27fd1982b:~$ ls -l /opt/payara/appserver/glassfish/domains/domain1/autodeploy
total 4
-rw-r--r-- 1 payara payara 3460 May 11 12:15 dataverse-people.war
... but the output from curl doesn't change at all.
Philip Durbin π (May 11 2023 at 12:17):Nothing in server.log. No mention of attempted deployments.
Philip Durbin π (May 11 2023 at 12:17):You can try it yourself if you don't believe me. :happy:
Oliver Bertuch (May 11 2023 at 12:21):Did you re-enable the autodeploy scanner?
Oliver Bertuch (May 11 2023 at 12:34):GNARF. No use. Even with changing the directories it won't reload / autodeploy. I'm not sure if this works with exploded WARs
Oliver Bertuch (May 11 2023 at 12:35):Should I push you my base image to GHCR and you try with your WAD? You're using WARs @Philip Durbin
Oliver Bertuch (May 11 2023 at 12:49):I just did: https://github.com/orgs/gdcc/packages/container/base/92458934?tag=reload
Philip Durbin π (May 11 2023 at 12:57):$ docker pull ghcr.io/gdcc/base:reload
Error response from daemon: Head "https://ghcr.io/v2/gdcc/base/manifests/reload": unauthorized
Oh. Might be private. Sec
Oliver Bertuch (May 11 2023 at 12:59):Try again, just made it public
Philip Durbin π (May 11 2023 at 13:04):I think I'm missing a step. The scanner or something.
Philip Durbin π (May 11 2023 at 13:05):This is what I've done so far: https://github.com/IQSS/dataverse-people/commit/74dd7e4
Philip Durbin π (May 11 2023 at 13:05):Please let me know what I'm missing.
Philip Durbin π (May 11 2023 at 13:06):By the way, I've played with fast redeploys with Quarkus. It's a game changer. I would love to boost our productivity with a faster feedback loop.
Philip Durbin π (Aug 09 2023 at 13:29):sekmiller and I are both having trouble with https://guides.dataverse.org/en/5.14/container/dev-usage.html#re-deploying
Philip Durbin π (Aug 09 2023 at 13:29):$ docker compose -f docker-compose-dev.yml create dev_dataverse
[+] Running 4/4
β Ώ Container dev_postgres Created 0.0s
β Ώ Container dev_solr_initializer Recreated 0.1s
β Ώ Container dev_solr Recreated 0.1s
β Ώ Container dev_dataverse Recreated 1.0s
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ae0ae4848ac1 moby/buildkit:buildx-stable-1 "buildkitd" 2 weeks ago Up 2 weeks buildx_buildkit_maven0
$ curl http://localhost:8080/api/info/version
curl: (7) Failed to connect to localhost port 8080 after 7 ms: Connection refused
I've never used these instructions until he said they weren't working this morning. I'm pretty sure he's right.
Philip Durbin π (Aug 09 2023 at 13:30):Should I create an issue?
Oliver Bertuch (Aug 09 2023 at 13:32):Did you start the containers with compose before the recreation?
Philip Durbin π (Aug 09 2023 at 13:36):No.
Oliver Bertuch (Aug 09 2023 at 13:37):Quote from the docs:
In case of using Docker Compose and starting the containers in the background, you can use a workaround to only restart the application container
Oliver Bertuch (Aug 09 2023 at 13:37):Maybe that sentence needs to be more precise in wording
Philip Durbin π (Aug 09 2023 at 13:38):Yeah, maybe I'll create a PR.
Oliver Bertuch (Aug 09 2023 at 13:45):The explanation is easy - compose need to track the state of the containers, otherwise it can't maintain 'em. :wink:
Philip Durbin π (Aug 09 2023 at 14:17):The foreground/background stuff might be a bit confusing.
Philip Durbin π (Aug 09 2023 at 16:11):I can't get the docker compose -f docker-compose-dev.yml create dev_dataverse thing to work at all. I'll just propose a commit to remove it for now.
Philip Durbin π (Aug 09 2023 at 16:19):Actually, instead of removing it, I just flagged it as a bit sus.
Philip Durbin π (Aug 09 2023 at 16:19):@Oliver Bertuch how does this look to you? https://github.com/IQSS/dataverse/compare/9616-container-dev...9616-redeploy
Philip Durbin π (Aug 09 2023 at 17:53):I sent a link to sekmiller as well.
Philip Durbin π (Aug 09 2023 at 21:02):Whoops. New link because that PR got merged: https://github.com/IQSS/dataverse/compare/8305-payara6-ee10-v3...9616-redeploy
Philip Durbin π (Aug 11 2023 at 09:39):At standup yesterday we agreed it's fine for me to push this commit into the Payara 6 branch. I will, when I get a minute. Or I could remove the whole "Redeploying Just Dataverse with Docker Compose" section (rather than saying we're not sure it works) because I can't get it working. @Oliver Bertuch what do you think? Merge as is? Remove that section? Try to get it working?
Oliver Bertuch (Aug 11 2023 at 10:14):If you feel it is more trouble than feature, remove it if you want...
Philip Durbin π (Aug 11 2023 at 10:16):Well, it must work for you, or you wouldn't have written it up. I wonder if I'm missing something. Do you use that method all the time?
Oliver Bertuch (Aug 11 2023 at 10:17):Nope, as I'm not using docker-compose
Oliver Bertuch (Aug 11 2023 at 10:17):To much to type :wink:
Oliver Bertuch (Aug 11 2023 at 10:17):But I tried it before and it did work (otherwise I would not have written it)
Philip Durbin π (Aug 11 2023 at 10:25):Would you want to try it again? And help me get it working?
Oliver Bertuch (Aug 11 2023 at 10:27):Can try
Oliver Bertuch (Aug 11 2023 at 10:28):Can we focus on topic #9769 first? I want to get that fixed ASAP because it's breaking people's workflows
Philip Durbin π (Aug 11 2023 at 10:29):Absolutely!
Philip Durbin π (Aug 11 2023 at 15:19):Related topic: #containers > re-deploying (dev usage)
Oliver Bertuch (Aug 14 2023 at 12:56):I might have found a better way to do this...
Oliver Bertuch (Aug 14 2023 at 12:56):It's hacky.
Oliver Bertuch (Aug 14 2023 at 12:57):So I can't make "dynamic reloads" work. It simply doesn't pick up the touches on the .reload file.
Oliver Bertuch (Aug 14 2023 at 12:57):As a consequence, we can't use the builtin feature...
Oliver Bertuch (Aug 14 2023 at 12:57):But...
Oliver Bertuch (Aug 14 2023 at 12:58):I did some experimentation with "reload" from the Admin Console and it seems to work.
Oliver Bertuch (Aug 14 2023 at 12:59):The interesting part however is that internally, this "reload" seems to be a disable/enable command
Oliver Bertuch (Aug 14 2023 at 13:00):It's loosing an app's state. But maybe that's OK?
Oliver Bertuch (Aug 14 2023 at 13:00):I did experiment with asadmin redeploy --keepstate as well
Oliver Bertuch (Aug 14 2023 at 13:00):That does work, although one has to ignore all the SQL exception. We really should make those go away BTW
Oliver Bertuch (Aug 14 2023 at 13:01):That said, I'm now wondering how we can make Payara do these things for us
Oliver Bertuch (Aug 14 2023 at 13:01):It certainly is quicker to do these reloads than stopping and starting the containers
Oliver Bertuch (Aug 14 2023 at 13:02):As we need to bring all the peaces together, one requirement for the workflow would be to execute the package command in some way. Otherwise we don't have all the apps stuff in one place
Oliver Bertuch (Aug 14 2023 at 13:03):But probably that is reasonable - redeploying after compiling seems not so reasonable to me
Oliver Bertuch (Aug 14 2023 at 13:04):BTW simply editing an XHTML file will not be enough to make it reload. And getting these in the right place is also tricky, needs packaging...
Oliver Bertuch (Aug 14 2023 at 13:05):(because we have a bunch of single files at the top location - would be easier if we had all of this in subfolders. But as JSF is going to go away anyway... well... maybe that's fine.)
Oliver Bertuch (Aug 14 2023 at 13:05):We can mount the exploded package into the container. Easy
Oliver Bertuch (Aug 14 2023 at 13:06):What's left is the trigger telling Payara to disable/enable or undeploy/deploy
Oliver Bertuch (Aug 14 2023 at 13:07):Here's a thought about that: we could use another container for this. It will sit on the same network. And it will talk to Payara via it's admin API. https://docs.payara.fish/community/docs/Technical%20Documentation/Payara%20Server%20Documentation/Management%20and%20Monitoring%20REST%20API/Overview.html
Oliver Bertuch (Aug 14 2023 at 13:07):Using the REST API we don't need Java in that container, it's simple curl/wget commands.
Oliver Bertuch (Aug 14 2023 at 13:08):We could write a script for configbaker... :smile:
Philip Durbin π (Aug 14 2023 at 13:09):Thanks for hacking on this! Please let me know if you'd like me to try any commands.
Oliver Bertuch (Aug 14 2023 at 13:10):Do you think that workflow for a dev "package the app" is reasonable?
Oliver Bertuch (Aug 14 2023 at 13:10):Or should the DX be something else?
Philip Durbin π (Aug 14 2023 at 13:14):The developer joy DX of Quarkus would be ideal but I'm willing to try anything. :big_smile:
Oliver Bertuch (Aug 14 2023 at 13:16):Technically we could write a scripts that runs on your host maschine, monitors the compiled files and triggers packaging for you
Oliver Bertuch (Aug 14 2023 at 13:17):It would still need to reach out to Payara to reload... So this REST API part would be the same
Philip Durbin π (Aug 14 2023 at 13:26):Sort of like https://github.com/AdamBien/wad ?
Oliver Bertuch (Aug 14 2023 at 14:19):Yes, sort of
Philip Durbin π (Aug 14 2023 at 14:25):I used wad with https://github.com/IQSS/dataverse-people and it worked fine.
Philip Durbin π (Aug 14 2023 at 14:25):We could use that tiny repo as a test bed, if you want.
Philip Durbin π (Aug 14 2023 at 14:26):If we can't come up with a good DX for a tiny repo, what hope do we have for Dataverse?
Oliver Bertuch (Aug 14 2023 at 20:05):Adding a ref here: https://stackoverflow.com/questions/31035669/glassfish-application-reload-with-asadmin
Philip Durbin π (Aug 15 2023 at 14:21):I don't think we need a whole topic for this because JSF is going away, but I just got a report that a downside of Docker is that changing an xhtml file take a whole redeployment cycle. Previously, when you have Netbeans configured to redeploy, a change to an xhtml file was instantaneous.
Oliver Bertuch (Aug 15 2023 at 15:51):Ehm... Told you so? :see_no_evil:
Philip Durbin π (Aug 15 2023 at 15:52):It's not news to me. I'm just passing along feedback from people who have tried Docker and are using the classic method instead.
Oliver Bertuch (Aug 15 2023 at 15:54):Yeah, thx for the feedback, appreciate it. Can only say: workin' on it...
Philip Durbin π (Sep 07 2023 at 11:02):Should we put this on the agenda for today's meeting? Any news? Anything to discuss! 3m to deploy according to #containers > running using compose ! :grimacing:
Philip Durbin π (Sep 07 2023 at 11:04):#9769 has been closed. Should we discuss redeploying here or in #containers > faster redeploy ? Are they separate topics? Should we close one of them?
Oliver Bertuch (Sep 07 2023 at 11:06):No news yet - sorry, my attention is requested at other places ATM
Oliver Bertuch (Sep 07 2023 at 11:06):Let's move discussion over there. Or can we merge?
Philip Durbin π (Sep 07 2023 at 11:15):Or a fresh topic even :big_smile:
Oliver Bertuch (Sep 25 2023 at 20:05):@Philip Durbin saw your comment on https://github.com/IQSS/dataverse/issues/9590#issuecomment-1734223276
Oliver Bertuch (Sep 25 2023 at 20:06):When you are using Netbeans...
Oliver Bertuch (Sep 25 2023 at 20:06):Is Netbeans starting the app server for you?
Oliver Bertuch (Sep 25 2023 at 20:06):Or is it just deploying to a running webserver?
Oliver Bertuch (Sep 25 2023 at 20:07):And if you change that XHTML, can you tell if it is redeploying the whole WAR / exploded WAR?
Philip Durbin π (Sep 25 2023 at 20:07):Yes, Netbeans is starting Payara. I'm happy to give you a quick demo if you like.
Oliver Bertuch (Sep 25 2023 at 20:08):Sure!
Philip Durbin π (Sep 25 2023 at 20:08):https://harvard.zoom.us/my/pdurbin?pwd=em1WNUZGbnY2YjhxNEdSbjJJMXNSUT09
Ellen K (Sep 25 2023 at 20:16):@Philip Durbin and @Oliver Bertuch , just curious, were you able to get the redeploy any faster?
Oliver Bertuch (Sep 25 2023 at 20:44):Still hacking on it... :smile_cat:
Philip Durbin π (Sep 25 2023 at 20:47):Nothing yet. Took some notes here: https://github.com/IQSS/dataverse/issues/9590#issuecomment-1734223276
Oliver Bertuch (Sep 26 2023 at 13:46):So I played with this some more and at least the JSF part is actually easy to pull of.
Oliver Bertuch (Sep 26 2023 at 13:46):The key to success is making JSF work in dev mode and have a refresh time of 0
Philip Durbin π (Sep 26 2023 at 13:46):Oh, right, right, right. I forgot that bit.
Philip Durbin π (Sep 26 2023 at 13:47):@Ellen K and I were just talking about the redeploy over coffee.
Oliver Bertuch (Sep 26 2023 at 13:47):I did a lot of more reading on this as well to look into hot redeploys for Java code
Oliver Bertuch (Sep 26 2023 at 13:48):It looks like this might not (yet) be supported with IntelliJ, but should be doable with Netbeans, Eclipse and VSCode
Philip Durbin π (Sep 26 2023 at 13:48):/me looks at... As a developer, I want changes I make to the front end to be visible when I deploy from Netbeans #6953
Oliver Bertuch (Sep 26 2023 at 13:49):And by the way: the way Netbeans copies on save is a special hack by them. I did not find any such functionality with the other IDEs
Philip Durbin π (Sep 26 2023 at 13:49):Well isn't that special
Oliver Bertuch (Sep 26 2023 at 13:50):The other IDEs require you to do a build for this... :wink:
Philip Durbin π (Sep 26 2023 at 13:52):Here we define a refresh period of 1: https://guides.dataverse.org/en/6.0/developers/classic-dev-env.html#configure-your-development-environment-for-gui-edits
Philip Durbin π (Sep 26 2023 at 13:52):I imagine any low number is fine.
Philip Durbin π (Sep 26 2023 at 14:02):Still does't work. Am I don't this wrong?
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..ec67243e42 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -14,6 +14,8 @@ services:
- DATAVERSE_DB_USER=${DATAVERSE_DB_USER}
- ENABLE_JDWP=1
- DATAVERSE_FEATURE_API_BEARER_AUTH=1
+ - ENABLE_RELOAD=1
+ - javax.faces.FACELETS_REFRESH_PERIOD=0
ports:
- "8080:8080" # HTTP (Dataverse Application)
- "4848:4848" # HTTP (Payara Admin Console)
@@ -27,6 +29,7 @@ services:
volumes:
- ./docker-dev-volumes/app/data:/dv
- ./docker-dev-volumes/app/secrets:/secrets
+ - ./target/dataverse-6.0:/opt/payara/deployments/dataverse
tmpfs:
- /dumps:mode=770,size=2052M,uid=1000,gid=1000
- /tmp:mode=770,size=2052M,uid=1000,gid=1000
Oh wait, it should be dataverse.jsf.refresh-period but formatted in caps and such, I think.
Oliver Bertuch (Sep 26 2023 at 14:05):Yes. I'm about to make a PR about faster reloads. Personally, I would like to include this in the ENABLE_RELOAD switch, as it would make things easier then setting lots of things.
Philip Durbin π (Sep 26 2023 at 14:05):"Environment variables follow the key, replacing any dot, colon, dash, etc. into an underscore β_β and all uppercase letters. Example: dataverse.db.host -> DATAVERSE_DB_HOST"
Oliver Bertuch (Sep 26 2023 at 14:05):We could also have a simple Maven property to enable this...
Philip Durbin π (Sep 26 2023 at 14:06):Speaking of ENABLE_RELOAD=1, do I need it?
Oliver Bertuch (Sep 26 2023 at 14:06):Probably not right now
Oliver Bertuch (Sep 26 2023 at 14:06):I'll just want to make it more convenient that way
Philip Durbin π (Sep 26 2023 at 14:10):It works! This is all I need:
diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index ab44dbc180..2b9a69f9ba 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -14,6 +14,7 @@ services:
- DATAVERSE_DB_USER=${DATAVERSE_DB_USER}
- ENABLE_JDWP=1
- DATAVERSE_FEATURE_API_BEARER_AUTH=1
+ - DATAVERSE_JSF_REFRESH_PERIOD=1
ports:
- "8080:8080" # HTTP (Dataverse Application)
- "4848:4848" # HTTP (Payara Admin Console)
@@ -27,6 +28,7 @@ services:
volumes:
- ./docker-dev-volumes/app/data:/dv
- ./docker-dev-volumes/app/secrets:/secrets
+ - ./target/dataverse-6.0:/opt/payara/deployments/dataverse
tmpfs:
- /dumps:mode=770,size=2052M,uid=1000,gid=1000
- /tmp:mode=770,size=2052M,uid=1000,gid=1000
Muhahaha!
Philip Durbin π (Sep 26 2023 at 14:11):Of course we need to factor out that 6.0 in there.
Oliver Bertuch (Sep 26 2023 at 14:11):Actually I do have an idea for that. Let me pick your brain about this.
Oliver Bertuch (Sep 26 2023 at 14:12):We can make the WAR plugin (that creates the exploded and the WAR) skip the version number
Oliver Bertuch (Sep 26 2023 at 14:12):Would that make sense? Would it be disruptive to someone else?
Philip Durbin π (Sep 26 2023 at 14:13):Well, my first question is if we have a variable we can reuse for the number, the 6.0. It's in the pom already.
Oliver Bertuch (Sep 26 2023 at 14:13):(Obviously we could just change the config in the ct profile, but then it might get harder to use with IDE rebuilds, as you would need to configure the profile)
Oliver Bertuch (Sep 26 2023 at 14:14):Not if we insist on enabling people to run this just from using docker compose
Oliver Bertuch (Sep 26 2023 at 14:14):It's would be no problem if everybody uses mvn to start the containers
Oliver Bertuch (Sep 26 2023 at 14:14):(Because then we can use variable substitution))
Oliver Bertuch (Sep 26 2023 at 14:15):We can of course add an env var in there, but that would need to be adapted with every version increase. Meh
Philip Durbin π (Sep 26 2023 at 14:16):For the next PR, I say we:
DATAVERSE_JSF_REFRESH_PERIOD=1 to the compose file (can't hurt)./target/dataverse-6.0:/opt/payara/deployments/dataverse as a volume to the compose file and explain that 6.0 is a placeholder Philip Durbin π (Sep 26 2023 at 14:17):instant value
Oliver Bertuch (Sep 26 2023 at 14:18):I would like to make it a little more complete
Oliver Bertuch (Sep 26 2023 at 14:18):Just a tiny bit
Oliver Bertuch (Sep 26 2023 at 14:18):But in general, I agree
Philip Durbin π (Sep 26 2023 at 14:21):@Ellen K it works!
Philip Durbin π (Sep 26 2023 at 14:21):@Oliver Bertuch so are you making a PR or should I?
Oliver Bertuch (Sep 26 2023 at 14:21):On it
Oliver Bertuch (Sep 26 2023 at 14:53):https://github.com/IQSS/dataverse/pull/9959
BalΓ‘zs Pataki (Oct 28 2023 at 13:25):I have a new Dataverse 6.0 docker installation as per https://guides.dataverse.org/en/latest/container/dev-usage.html and when I try to access the Payara console at http://localhost:4848 it redirects to https://localhost:4848 with a warning about the self signed cert provided by this endpoint.
Is there a reason for serving the Payara console via https in development mode?
I actually wanted to try IntelliJ's Payara remote run config, which may enable deploying a new Dataverse version from IntelliJ directly, thus making development easier, but because of the invalid cert it cannot connect at localhost:4848.
Philip Durbin π (Oct 30 2023 at 13:50):I'm VERY interested in improving the DX (dev experience) for Dataverse. This remote run stuff falls under this issue, for sure: Faster turn-around times for devs using containers #9590
I might be out of luck with the community edition. It seems like need IntelliJ IDEA Ultimate to install Payara Platform Tools:
Screenshot-2023-10-30-at-9.46.54-AM.png
I was trying to follow https://blog.jetbrains.com/idea/2021/04/fast-and-efficient-development-with-payara-server-intellij-idea-ultimate/
Anyway, what would be the best way to unblock you? To install a valid cert?
BalΓ‘zs Pataki (Oct 30 2023 at 15:14):I have the ultimate edition and there I have these options:
When I use remote, I get these options:
Well, the best option would be no SSL at all. If I click "Use SSL connection" it will try to connect to port 61008, without this check it tries to connect to http 4848 which is actually a https port now.
Philip Durbin π (Oct 30 2023 at 15:24):I see. From the docker compose perspective, we're simply mapping ports one to one like this:
ports:
- "8080:8080" # HTTP (Dataverse Application)
- "4848:4848" # HTTP (Payara Admin Console)
- "9009:9009" # JDWP
- "8686:8686" # JMX
I wonder if there's a non-SSL port we can map.
BalΓ‘zs Pataki (Oct 30 2023 at 15:27):But why is 4848 serving actually HTTPS? I couldn't find anything about it in the docker configs. Is it a new Payar 6 "feature" maybe?
Can you, please, confirm that when you open http://localhost:4848 you also get the warning about the cert (in Chrome, for example)?
Philip Durbin π (Oct 30 2023 at 15:32):Not sure. I found this (perhaps) related issue: https://github.com/payara/ecosystem-support/issues/59
Philip Durbin π (Oct 30 2023 at 15:38):Hmm, maybe this is what's doing it:
modules/container-base/src/main/docker/Dockerfile
141: ${ASADMIN} enable-secure-admin
@BalΓ‘zs Pataki do you want to try rebuilding the base image without that?
BalΓ‘zs Pataki (Oct 30 2023 at 16:07):Sure, I give it a try!
BalΓ‘zs Pataki (Oct 31 2023 at 13:23):I tried commenting that out and also using
${ASADMIN} disable-secure-admin
But it did nothing. Then I tried it disabling via the admin console UI:
Configurations > default-config > Protocols > sec-admin-listener > Use the Secure Administration page to enable/disable secure administration at https://localhost:4848/common/appServer/securityAdmin.jsf
This worked! Now I can access the admin console at http://localhost:4848, however when try to log in it gives me:
Configuration Error
Secure Admin must be enabled to access the DAS remotely.
So, it seems to be a Catch-22 for me: I can disable secure admin via the admin console, but then I cannot log in via HTTP anymore.
BalΓ‘zs Pataki (Oct 31 2023 at 15:25):Ok, I got a little bit further by adding an nginx to the docker-compose, which proxies HTTP 4849 to the backend HTTPS 4848. This way I can access from the Intellij run config Payara admin at http://localhost:4849. However, during redeploy I get:
dev_dataverse> java.lang.Exception: Virtual server server already has a web module dataverse loaded at / therefore web module dataverse-6.0 cannot be loaded at this context path on this virtual server
Now, I undeployed the dataverse app by hand via the admin console, and now I can redeploy the app from IntelliJ or hot reload classes with minimal changes.
The redeploy is ~20 sec.
Unfortunately the Update classes and resources is not available, which allows hot reloading XHTML files in normal local Payara usage:
Philip Durbin π (Oct 31 2023 at 15:27):Huh, it looks available in the screenshot
BalΓ‘zs Pataki (Oct 31 2023 at 15:30):The first one with the 2 items is what is availble with remote Payara run config (ie. Payara in Docker), the later is when IntelliJ can run asadmin locally for a locally running Payara.
Philip Durbin π (Oct 31 2023 at 15:32):Oh, I see. Thanks.
Philip Durbin π (Oct 31 2023 at 15:32):Well, you've made a lot of progress. We can see the limitations, at least.
Philip Durbin π (Oct 31 2023 at 15:33):But can we overcome them?
BalΓ‘zs Pataki (Oct 31 2023 at 15:46):I'm not really sure, but probably not.
As far as I understand, in case of remote deployment IntelliJ has 2 API-s available:
Unfortunately, I don't know what is the magic behind Update classes and resources, but it doesn't seem to be related to remote java debugging via JDWP. Is it maybe a Payara feature with a magic classloader? I don't know.
Philip Durbin π (Oct 31 2023 at 17:22):Hmm. Have you heard of JRebel? https://www.jrebel.com . I wonder how they do it.
BalΓ‘zs Pataki (Nov 01 2023 at 06:44):Yes, I heard, they do it for a lot of $$$ :-) The closest free thing is http://hotswapagent.org/ I tried it a couple of years ago, works quite nicely, allows reloading classes with deeper changes (new methods, fields added, etc), but I am not sure how it handles xhtml files, which are not compiled at IDE time.
BalΓ‘zs Pataki (Nov 01 2023 at 07:00):There's also this, but I haven't tried it: https://manik.imixs.org/
BalΓ‘zs Pataki (Nov 01 2023 at 07:09):Manik also seem to work with a local Payara installation https://manik.imixs.org/userguide.html#payara
I wonder whether it is possible to make Payara in docker use an exploded war available in a mapped volume and pick up changes there.
Philip Durbin π (Nov 01 2023 at 12:15):As a Netbeans user, I'm no longer worried about xhtml files. Recently we made this pull request to address that: allow fast xhtml edits in Docker #9961
Philip Durbin π (Nov 01 2023 at 12:18):Manik looks interesting! For the PR I just mentioned we have this under "volumes" in our docker compose: ./target/dataverse-6.0:/opt/payara/deployments/dataverse. Maybe it'll work with Manik?!? Not sure.
BalΓ‘zs Pataki (Nov 01 2023 at 14:01):Philip Durbin said:
As a Netbeans user, I'm no longer worried about xhtml files. Recently we made this pull request to address that: allow fast xhtml edits in Docker #9961
The main things seems to be DATAVERSE_JSF_REFRESH_PERIOD=1 here. How does it relate to or specific to Netbeans?
Once the war is loaded from a mapped volume I feel a lot of options open up regarding managing reloads, because this would be very simiilar to a local Payara install. But it needs experimenting, sure.
Philip Durbin π (Nov 01 2023 at 14:20):It's not specific to Netbeans. It tells the app server how often to look for changes to XHTML files. In production it's basically set to "never". In dev, we set it to every second, which is plenty fast enough.
Philip Durbin π (Nov 01 2023 at 14:36):Which should we try first, HotswapAgent or Manik?
Philip Durbin π (Nov 01 2023 at 14:38):I see Payara mentioned here: https://manik.imixs.org/userguide.html#payara
Philip Durbin π (Nov 01 2023 at 14:52):I feel like we should rename this thread. It's becoming more about iterating on code quickly, right?
Philip Durbin π (Nov 01 2023 at 14:58):I could even move these messages to #containers > faster redeploy ... Any objection, @BalΓ‘zs Pataki ?
BalΓ‘zs Pataki (Nov 01 2023 at 15:00):Sure, you can move it/rename it.
I think HotswapAgent could be the first target. This would augment the current class reload capabilities via JDWP, which would be a big win.
Notification Bot (Nov 01 2023 at 15:00):30 messages were moved here from #containers > Payara admin console redirects to https by Philip Durbin.
Philip Durbin π (Nov 01 2023 at 15:13):Yeah? HotswapAgent? Ok. No objection. They do seem to support Java 17.
BalΓ‘zs Pataki (Nov 02 2023 at 15:40):Submitted PR for the nginx workaround: https://github.com/IQSS/dataverse/pull/10088
Philip Durbin π (Nov 02 2023 at 16:03):Aw. You removed "hack" from the name. Otherwise, looks perfect! Thanks! :heart:
Philip Durbin π (Nov 02 2023 at 18:16):@Oliver Bertuch heads up that @Juan Pablo Tosca Villanueva was struggling with the re-deploy instructions this morning and I told him he's welcome to go ahead and make a PR to remove them. He just did: #10089
Philip Durbin π (Nov 02 2023 at 18:17):Or rather, drastically shorten them: https://dataverse-guide--10089.org.readthedocs.build/en/10089/container/dev-usage.html#re-deploying
Philip Durbin π (Nov 02 2023 at 18:21):Merged.
Philip Durbin π (Nov 04 2023 at 22:33):I just made this PR: Containers: set autodeploy-enabled=true for ENABLE_RELOAD #10100
Philip Durbin π (Nov 04 2023 at 22:34):@Oliver Bertuch no rush but I'm curious what you think.
Philip Durbin π (Nov 05 2023 at 01:56):In the toy "people" app, I'm starting to document how to redeploy to Payara running in Docker, starting with the most obvious ways: https://github.com/IQSS/dataverse-people/commit/afb59e8
Philip Durbin π (Nov 05 2023 at 01:56):I'm hoping to apply anything I learn through this exercise to Dataverse, of course.
Oliver Bertuch (Nov 05 2023 at 09:55):Philip Durbin said:
Oliver Bertuch no rush but I'm curious what you think.
Nice! Just one hint... have you seen #9959? :hug::yum:
Oliver Bertuch (Nov 05 2023 at 09:57):Actually Manik which @BalΓ‘zs Pataki mentioned could be one of the missing pieces of the puzzle here: something watching for changes and repackaging the application.
Oliver Bertuch (Nov 05 2023 at 10:01):(I experimented a bit with changing the autodeploy dir in the domain config, but had no luck, didn't work. Maybe we can try again)
Philip Durbin π (Nov 05 2023 at 12:31):Ah, so you also discovered we don't have autodeploy-enabled=true. Do you care if I merge my pull request?
Oliver Bertuch (Nov 05 2023 at 13:07):Dunno. Wouldn't it make sense to figure out some more stuff first to get this going? As I said, the autodeploy dir is different, so this will need some more tuning and experimentation
Philip Durbin π (Nov 05 2023 at 13:38):To me it's a harmless change that improves the base image. Without it, if you put a war file in the autodeploy directory, nothing happens.
Philip Durbin π (Nov 05 2023 at 14:26):Sadly, the autodeploy directory method only works the first time you copy a war file into it: https://github.com/IQSS/dataverse-people/commit/6e290c5
Philip Durbin π (Nov 06 2023 at 16:36):@BalΓ‘zs Pataki so! I'm on your branch, this PR: Add gdcc/nginx:unstable image and dev_nginx container for IntelliJ Payara plugin workaround #10088
Philip Durbin π (Nov 06 2023 at 16:37):It says "connected to server":
Screenshot-2023-11-06-at-11.36.52-AM.png
Philip Durbin π (Nov 06 2023 at 16:37):Do you have time for a Zoom? :sweat_smile:
Philip Durbin π (Nov 06 2023 at 18:08):It works! @BalΓ‘zs Pataki is my hero!! 
Oliver Bertuch (Nov 06 2023 at 19:23):I hope no one minds about improvements?
Oliver Bertuch (Nov 06 2023 at 19:24):I'd say let's make 4848 HTTP by default and do not create another image... Instead just use vanilla nginx.
Oliver Bertuch (Nov 06 2023 at 19:25):But yes, I love where this is going!
Philip Durbin π (Nov 06 2023 at 19:29):Merged! I added some docs: https://dataverse-guide--10102.org.readthedocs.build/en/10102/container/dev-usage.html#intellij-idea-ultimate-and-payara-platform-tools
Philip Durbin π (Nov 06 2023 at 19:46):@Oliver Bertuch yes, please feel free to refactor away!
Philip Durbin π (Nov 06 2023 at 23:46):I just got it working in Netbeans too! :tada:
Philip Durbin π (Nov 07 2023 at 01:10):Hmm, having trouble reproducing what I did in Netbeans. :thinking:
Philip Durbin π (Nov 07 2023 at 01:13):Here are some screenshots on what I was doing, at least:
apply-code-changes.png
compile.png
run.png
server-common.png
server-java.png
Philip Durbin π (Nov 07 2023 at 01:42):Actually, it stopped working in IntelliJ too. :disappointed:
Philip Durbin π (Nov 07 2023 at 01:44):And when I start over and try to undeploy the war file that's initially deployed in our container, I'm not getting this error:
dev_dataverse> [#|2023-11-07T01:41:56.980+0000|INFO|Payara 6.2023.8||_ThreadID=228;_ThreadName=admin-thread-pool::admin-listener(7);_TimeMillis=1699321316980;_LevelValue=800;|
dev_dataverse> RewriteFilter shutting down...|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:56.980+0000|INFO|Payara 6.2023.8||_ThreadID=228;_ThreadName=admin-thread-pool::admin-listener(7);_TimeMillis=1699321316980;_LevelValue=800;|
dev_dataverse> RewriteFilter deactivated.|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:57.145+0000|INFO|Payara 6.2023.8|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=228;_ThreadName=admin-thread-pool::admin-listener(7);_TimeMillis=1699321317145;_LevelValue=800;|
dev_dataverse> No timers to be deleted for id: 111366718806163456|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:57.146+0000|WARNING|Payara 6.2023.8|jakarta.enterprise.web.util|_ThreadID=228;_ThreadName=admin-thread-pool::admin-listener(7);_TimeMillis=1699321317146;_LevelValue=900;|
dev_dataverse> The web application [] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:57.154+0000|SEVERE|Payara 6.2023.8|com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl|_ThreadID=124;_ThreadName=Executor-Service-9;_TimeMillis=1699321317154;_LevelValue=1000;|
dev_dataverse> Could not instantiate service: fish.payara.jbatch.persistence.rdbms.LazyBootPersistenceManager due to exception:java.lang.ClassNotFoundException: fish.payara.jbatch.persistence.rdbms.LazyBootPersistenceManager not found by fish.payara.server.internal.batch.payara-jbatch [372]|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:57.154+0000|SEVERE|Payara 6.2023.8|com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl|_ThreadID=124;_ThreadName=Executor-Service-9;_TimeMillis=1699321317154;_LevelValue=1000;|
dev_dataverse> Could not instantiate service: com.ibm.jbatch.container.impl.BatchKernelImpl due to exception:java.lang.reflect.InvocationTargetException|#]
dev_dataverse>
dev_dataverse> [#|2023-11-07T01:41:57.154+0000|WARNING|Payara 6.2023.8|javax.enterprise.system.core|_ThreadID=124;_ThreadName=Executor-Service-9;_TimeMillis=1699321317154;_LevelValue=900;_MessageID=NCLS-CORE-00069;|
dev_dataverse> Exception while dispatching an event
dev_dataverse> java.util.ServiceConfigurationError: jakarta.batch.operations.JobOperator: Provider com.ibm.jbatch.container.api.impl.JobOperatorImpl could not be instantiated
dev_dataverse> at java.base/java.util.ServiceLoader.fail(Unknown Source)
dev_dataverse> at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source)
dev_dataverse> at java.base/java.util.ServiceLoader$ProviderImpl.get(Unknown Source)
dev_dataverse> at java.base/java.util.ServiceLoader$3.next(Unknown Source)
dev_dataverse> at jakarta.batch.runtime.BatchRuntime.getJobOperator(BatchRuntime.java:48)
dev_dataverse> at org.glassfish.batch.spi.impl.BatchRuntimeHelper.event(BatchRuntimeHelper.java:216)
dev_dataverse> at org.glassfish.kernel.event.EventsImpl$1.run(EventsImpl.java:123)
dev_dataverse> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
dev_dataverse> at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
dev_dataverse> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
dev_dataverse> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
dev_dataverse> at java.base/java.lang.Thread.run(Unknown Source)
dev_dataverse> Caused by: java.lang.RuntimeException: Could not instantiate service com.ibm.jbatch.container.impl.BatchKernelImpl
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadServiceHelper(ServicesManagerImpl.java:424)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader.getService(ServicesManagerImpl.java:403)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader.access$300(ServicesManagerImpl.java:388)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl.getService(ServicesManagerImpl.java:304)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl.getBatchKernelService(ServicesManagerImpl.java:348)
dev_dataverse> at com.ibm.jbatch.container.api.impl.JobOperatorImplDelegate.<init>(JobOperatorImplDelegate.java:74)
dev_dataverse> at com.ibm.jbatch.container.api.impl.JobOperatorImpl.<init>(JobOperatorImpl.java:73)
dev_dataverse> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
dev_dataverse> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
dev_dataverse> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
dev_dataverse> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
dev_dataverse> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
dev_dataverse> ... 11 more
dev_dataverse> Caused by: java.lang.reflect.InvocationTargetException
dev_dataverse> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
dev_dataverse> at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
dev_dataverse> at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
dev_dataverse> at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
dev_dataverse> at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadService(ServicesManagerImpl.java:447)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadServiceHelper(ServicesManagerImpl.java:420)
dev_dataverse> ... 22 more
dev_dataverse> Caused by: java.lang.RuntimeException: Could not instantiate service fish.payara.jbatch.persistence.rdbms.LazyBootPersistenceManager
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadServiceHelper(ServicesManagerImpl.java:424)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader.getService(ServicesManagerImpl.java:403)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader.access$300(ServicesManagerImpl.java:388)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl.getService(ServicesManagerImpl.java:304)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl.getPersistenceManagerService(ServicesManagerImpl.java:333)
dev_dataverse> at com.ibm.jbatch.container.impl.BatchKernelImpl.<init>(BatchKernelImpl.java:87)
dev_dataverse> ... 29 more
dev_dataverse> Caused by: java.lang.ClassNotFoundException: fish.payara.jbatch.persistence.rdbms.LazyBootPersistenceManager not found by fish.payara.server.internal.batch.payara-jbatch [372]
dev_dataverse> at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1591)
dev_dataverse> at org.apache.felix.framework.BundleWiringImpl.access$300(BundleWiringImpl.java:79)
dev_dataverse> at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1976)
dev_dataverse> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadService(ServicesManagerImpl.java:441)
dev_dataverse> at com.ibm.jbatch.container.servicesmanager.ServicesManagerImpl$ServiceLoader._loadServiceHelper(ServicesManagerImpl.java:420)
dev_dataverse> ... 34 more
dev_dataverse> |#]
dev_dataverse>
This is on my laptop at home. It was working. Both IntelliJ and Netbeans. But now neither one works. :thinking:
Philip Durbin π (Nov 07 2023 at 01:53)::shrug:
Philip Durbin π (Nov 07 2023 at 13:54):I rebooted my laptop and now fast redeploy from both IntelliJ and Netbeans are working. :tada:
Philip Durbin π (Nov 07 2023 at 13:54):Mind you, I'm not sure rebooting was necessary but I was getting frustrated. :sweat_smile:
Philip Durbin π (Nov 07 2023 at 16:43):I recorded a video of fast redeploy with Netbeans: https://harvard.zoom.us/rec/share/BCEJnBDUFCiRxkN0r54FWNku0GWdKJysRubQoYg2NCm6sYqsNPyq0_0kwzsT-8yZ.f1v-LGnfP9pPnJwv
Philip Durbin π (Nov 07 2023 at 16:43):Once YouTube is done processing it, it should also appear here: https://www.youtube.com/watch?v=yo3aKOg96f0
Philip Durbin π (Nov 07 2023 at 16:45):One take! :grinning:
Oliver Bertuch (Nov 07 2023 at 16:47):I was working on improving the deets this morning to make the setup easier and have less new dependencies
Oliver Bertuch (Nov 07 2023 at 16:49):We have #9590, should we write down some stuff we have in mind?
Oliver Bertuch (Nov 07 2023 at 16:50):@Philip Durbin this works for XHTML edits, too, right?
Philip Durbin π (Nov 07 2023 at 16:55):Sure. Or we could do it live in the container meeting on Thursday. Or type here for now.
Philip Durbin π (Nov 07 2023 at 16:57):When I do "apply code changes" on xhtml, Netbeans says "No class to reload".
Oliver Bertuch (Nov 07 2023 at 17:00):So XHTML isn't propagated after a change using the Payara Tools?
Philip Durbin π (Nov 07 2023 at 17:01):This is builtin Netbeans functionality I'm using.
Philip Durbin π (Nov 07 2023 at 17:02): Oliver Bertuch (Nov 07 2023 at 17:04):That's a bummer! As the files are hotreload via a WAR file being thrown an the server, I don't see how this builtin thing actually would help with this. Can you do me a favor and try this?
Philip Durbin π (Nov 07 2023 at 17:05):Do you want to do a zoom?
Oliver Bertuch (Nov 07 2023 at 17:08):https://fz-juelich-de.zoom.us/j/66663859869?pwd=ZjZaRjFSK3QzV3RPK293N3dhUWlDZz09
Oliver Bertuch (Nov 07 2023 at 17:08):Need to run soon
Philip Durbin π (Nov 07 2023 at 18:45):I just let people know about my Netbeans video: https://groups.google.com/g/dataverse-community/c/zNBDzSMF2Q0/m/Z-xS6fA2BgAJ
Philip Durbin π (Nov 08 2023 at 02:31):I just posted about the video here: https://forum.payara.fish/t/fast-redeployment/721
Philip Durbin π (Nov 08 2023 at 02:32):And here (Netbeans mailing list): https://lists.apache.org/thread/y7x918bm30mdzyr88xfhvwlnqm3x51gs
Oliver Bertuch (Nov 08 2023 at 15:57):In the little time I could spend on https://github.com/IQSS/dataverse/pull/9959 today, I'm hacking away on making it more polished
Oliver Bertuch (Nov 08 2023 at 15:58):Here's a sneak peak at the docs I'm writing, hoping you might add steps for Netbeans later. Maybe @Juan Pablo Tosca Villanueva can add steps for VSCode, too.
grafik.png
Philip Durbin π (Nov 08 2023 at 15:59):Looks good but @Juan Pablo Tosca Villanueva is gonna want more emojis in the docs. :big_smile:
Oliver Bertuch (Nov 08 2023 at 18:55):The docs are up: https://dataverse-guide--9959.org.readthedocs.build/en/9959/container/dev-usage.html#ide-triggered-re-deployments
Philip Durbin π (Nov 08 2023 at 19:53):"limitations in what the JVM HotswapAgent can do for you" ... huh, so this is built into the JVM, I guess
Oliver Bertuch (Nov 08 2023 at 20:06):Absolutely. There is a reason why people spend money on JRebel
Oliver Bertuch (Nov 08 2023 at 20:07):And the Hotswap thing is also only activated when going via JDWP. That's the reason why you need to run the debug mode in the IDE
Oliver Bertuch (Nov 08 2023 at 20:08):@Philip Durbin what do you think about the docs? Is this more graphical thing useful?
Philip Durbin π (Nov 08 2023 at 20:29):I haven't finished reading them but sure, screenshots are nice.
Philip Durbin π (Nov 08 2023 at 20:29):They do get out of date sometimes.
Oliver Bertuch (Nov 08 2023 at 20:33):Yeah, they are out of date in the Payara docs, too
Oliver Bertuch (Nov 08 2023 at 20:33):But it seems like the UI doesn't change as much, so maybe we are fine for some time...
Oliver Bertuch (Nov 08 2023 at 20:34):Complainers will be held to create pull requests :stuck_out_tongue:
Philip Durbin π (Nov 08 2023 at 20:53):heh, sounds fine
Philip Durbin π (Nov 09 2023 at 17:03):Great demo by @Oliver Bertuch ! Here's the recording: https://harvard.zoom.us/rec/share/HqZQYeYdmB_yXyzGF4TqbGG7ta3nThvtqnuCtulDJQj5YMmcjk3PUvPVLJaLRykJ.4bUUH8_fenHuLmWa
BalΓ‘zs Pataki (Nov 10 2023 at 06:06):@Oliver Bertuch really cool updates, documentation and demo!
Oliver Bertuch (Nov 11 2023 at 07:58):Thanks for getting us there in the first place!
Philip Durbin π (Nov 14 2023 at 14:07):@Oliver Bertuch I just tried your https://github.com/IQSS/dataverse/pull/9959 but I'm getting an unexpected error: dev_solr> Caused by: org.apache.solr.common.SolrException: Error loading solr config from /var/solr/data/collection1/conf/solrconfig.xml
I don't see any reason for this, based on the changes in the PR. Very strange.
Philip Durbin π (Nov 14 2023 at 14:14):Actually, never mind. It's fine now.
Philip Durbin π (Nov 14 2023 at 14:15):I am having trouble getting -Dapp.deploy.skip to work though. Maybe I'll try the env var.
Philip Durbin π (Nov 14 2023 at 14:17):Changing .env didn't seem to work.
Philip Durbin π (Nov 14 2023 at 14:28):Neither did changing skip from 0 to 1 in the docker compose file.
Philip Durbin π (Nov 14 2023 at 14:28):I have a demo in 30 minutes. I'm switching back to develop. :sweat_smile:
Oliver Bertuch (Nov 14 2023 at 18:13):My money :money: is on "you forgot to build the base image" :see_no_evil:
Philip Durbin π (Nov 14 2023 at 18:37):Hmm, we should put that on a t-shirt.
Philip Durbin π (Nov 14 2023 at 18:38):Anyway, here's the demo I gave this morning: https://harvard.zoom.us/rec/share/AsB1xRUM3bsNmDrqTVa9ZM91FaHM5F1wdVAdoBAbbM5bXLllDUKfBfAbkLb4G7Ej.-Jyz6CjRt18R3Hfq
Philip Durbin π (Nov 14 2023 at 18:38):I mentioned this thread, so maybe some people will join the discussion.
Philip Durbin π (Nov 29 2023 at 21:03):@Eryk Kulikowski are you using VS Code? Have you tried getting this fast redeploy stuff working?
Eryk Kulikowski (Nov 30 2023 at 13:14):Philip Durbin said:
Eryk Kulikowski are you using VS Code? Have you tried getting this fast redeploy stuff working?
I have tried, but I could not get it working. The plugin for VS Code connected to the server, but it was giving errors. I did manage to undeploy the application, but did not have any success on deploying it.
Philip Durbin π (Nov 30 2023 at 13:47):@Eryk Kulikowski thanks. It sounds like @Juan Pablo Tosca Villanueva is having the same trouble. You should both feel free to add it to the agenda if you like. Meeting in 45 minutes. See #containers > weekly meeting
Philip Durbin π (Nov 30 2023 at 14:13):@Oliver Bertuch I'm running this:
mvn -Pct clean package docker:run -Dapp.deploy.skip
But it keeps checking http://dataverse:8080/api/info/version over and over. Is this the expected behavior?
Philip Durbin π (Nov 30 2023 at 14:14):I did get this, which is good:
dev_dataverse> Skipping deployment of /opt/payara/deployments/dataverse as requested.
Oliver Bertuch (Nov 30 2023 at 14:16):But it keeps checking http://dataverse:8080/api/info/version over and over. Is this the expected behavior?
Yes, that was intentional. You might want to deploy a fresh install and then bootstrapping gracefully waits for you to do the deployment. If you think we should not do that, we can alter the behaviour.
Oliver Bertuch (Nov 30 2023 at 14:17):Also: we can alter the Maven property name, too. Maybe it would be a nice idea to make it more inline with the other skipping things like -DskipTests etc
Eryk Kulikowski (Nov 30 2023 at 15:05):The Payara plugin in VS Code worked for me, but only with manual deploy. First, I manually build the war file with maven (mvn compile war:war), it only compliles changed files so it is fast. Then I right-click on the war file and choose "Run on Payara Server". This step takes some time, but it is still faster than rebuilding and replacing the docker image. This method works fine with our custom-build docker image, such that my other settings work just fine.
Eryk Kulikowski (Nov 30 2023 at 15:35):I measured the redeployment time, it is roughly 30 seconds. It is a large improvement for me.
Philip Durbin π (Nov 30 2023 at 15:42):Great! We just created an issue do add some docs: add Visual Studio Code to guides #10156
Philip Durbin π (Nov 30 2023 at 15:43):@Juan Pablo Tosca Villanueva like I said, you are very welcome to edit the description of that issue with whatever you like, links to resources, etc.
Philip Durbin π (Nov 30 2023 at 15:43):@Eryk Kulikowski you're very welcome to add comments and/or we can keep discussing here. Or maybe a dedicated topic if that's better.
Juan Pablo Tosca Villanueva (Dec 01 2023 at 14:09):Hi @Eryk Kulikowski ! I added some notes on #10156 with the setup and the workflow that I been using. It would be great that if you have any sugestions or find anything better you could share it with us :smile:
Eryk Kulikowski (Dec 01 2023 at 14:40):@Juan Pablo Tosca Villanueva I use exactly the same flow now. Since you mentioned the recommended plugin for docker (which I find very good and useful), you may also mention the "Extension Pack for Java" (mainly for java language support and maven) and XML plugins. I also find the "Git History" and "Git Graph" plugins useful. Finally, a good hint is to bump the memory setting for the Java language plugin up to 4 GB, I use the settings from here: https://github.com/redhat-developer/vscode-java/issues/2232
{
"java.configuration.updateBuildConfiguration": "interactive",
"java.compile.nullAnalysis.mode": "automatic",
"java.jdt.ls.vmargs": "-XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xmx4G -Xms100m -Xlog:disable"
}
Yeah, I been hit by the memory limit a couple of times already. :laughter_tears:
Oliver Bertuch (Jan 17 2024 at 07:51):Should we put https://github.com/IQSS/dataverse/pull/9959 in non-draft and let it get merged? @BalΓ‘zs Pataki was just asking about the nice screenshoted guide :see_no_evil:
Oliver Bertuch (Jan 17 2024 at 07:51):We can add more guides with nice pics later
Oliver Bertuch (Jan 17 2024 at 08:18):Here's what @BalΓ‘zs Pataki and I talked about in another place:
BalΓ‘zs Pataki: Hi Oliver, as I remember you created a very nice screenshot illustrated guide to running/developing Dataverse in docker. I thought I would find it at https://guides.dataverse.org/en/latest/container/index.html, but I couldn't. Could you please help me locate where it is available?
Oliver Bertuch: I think it hasn't been merged yet
Oliver Bertuch: This is the PR: https://github.com/IQSS/dataverse/pull/9959
Oliver Bertuch: And the guide you're looking for is here: https://dataverse-guide--9959.org.readthedocs.build/en/9959/container/dev-usage.html#ide-triggered-re-deployments
BalΓ‘zs Pataki: Oh, i see, thanks! I actually wanted to see if you figured out how one can edit .xhtml in Intellij and have it reloaded in the container without redeploying. I don't know if it is at all feasible all.
Oliver Bertuch: We found that Netbeans has a special trick to copy the files into the target directory. Maybe there is a similar thing for IntelliJ?
BalΓ‘zs Pataki: What is the "target" dir? It must be something mapped into the container, right?
Oliver Bertuch: /target is where Maven puts the build artifacts. And yes, we mounted that directory into the container. (Not /target, but the subpath where the exploded WAR lives and is updated)
BalΓ‘zs Pataki: Ok, thought about something similar. Is this mapping already part of the current docker:run mechanism? In IntelliJ one can add an action to run after deployment and it can be any script, so we could copy files anywhere.
Oliver Bertuch: It is not mapped by default, no. The PR linked above would introduce that though.
BalΓ‘zs Pataki: Great, I will check it out!
Oliver Bertuch: Please note the different mechanisms working under the hood here. If you want to on-the-fly edit XHTML, this is not going to work together with working on hot-reloading code. That is because hot-redeploy needs to ship a WAR file and then used JDWP to replace methods etc. The on-the-fly XHTML edit is using the exploded WAR mounted into the container, but that is not going to allow using the hot-redeploy stuff. The easiest solution to have both and both proper: buy a JRebel license...
BalΓ‘zs Pataki: Another idea: in IntelliJ one can run an action once a file is saved. At this point I could copy the changed (only .xhtml) file to the mapped directory. Would this also interfere with the hot-reloading mechanism? When fully redeploying the war, the changed .xhtml file would simply overwrite the separately copied version, but both would be identical at the end.
Oliver Bertuch: Yes, redeploying the WAR would replace it. But it will require waiting for the redeploy to be done (~30 secs). People were asking about instant reload - change, save, switch to browser, hit F5 and done
BalΓ‘zs Pataki: Yes, sure, that's what I want as well ... without JRebel.
Oliver Bertuch: Oh wait - redeploying the WAR using the hot redeploy method is not going to replace the copied file! If you use the exploded WAR to hot-edit the XHTML, that deployment needs to be unloaded and will be replaced with a WAR file based deployment. So these are completely independent sources. When using the hotredeploy method, editing and replacing the XHTML requires a redeployment of the WAR file, thus building and shipping will cost you time. But maybe that's fine for what you're trying to do. As long as you do not need to change XHMTML and Java code on the fly at the same time, this is good enough...?
BalΓ‘zs Pataki: Yes, I will do some experiments. Can you just help me with one thing? What is the payara setting we need to pick up changes to .xhtml files immediately?
Oliver Bertuch (Jan 17 2024 at 08:20):If you are going to use what I did in PR #9959, you only need to specify 'ENABLE_RELOAD=1' in your container env in some way. That is already done within the compose file of that PR.
Oliver Bertuch (Jan 17 2024 at 08:21):See https://github.com/IQSS/dataverse/pull/9959/files#diff-0f68198e0abf41811cffbc8d15bfb11e7b1d4491769f1362597a7e43126e8082 for all the things done to enable this reloading :wink:
Oliver Bertuch (Jan 17 2024 at 08:21):The missing part is to make IntelliJ copy the saved file into the right spot
Oliver Bertuch (Jan 17 2024 at 08:28):There might be an easy way to do this just yet... https://plugins.jetbrains.com/plugin/21538-save-actions-tool
Oliver Bertuch (Jan 17 2024 at 08:28):That plugin can be fed actions you define as "quick lists"
Oliver Bertuch (Jan 17 2024 at 08:29):And it can be triggered not only on every save (which might not be what you want), but using other key combos
Oliver Bertuch (Jan 17 2024 at 08:29):The configuration is per project
BalΓ‘zs Pataki (Jan 17 2024 at 12:41):Here's my solution for IntelliJ for quick webstuff update with hot class reload working as before.
Assuming that 'dataverse.jsf.refresh-period=1' is already set:
volumes:
...
- ./docker-dev-volumes/glassfish/applications:/opt/payara/appserver/glassfish/domains/domain1/applications
war files deployed from IntelliJ will be placed here
Added the script cpwebapp.sh, which helps copying files under src/main/webapp to ./docker-dev-volumes/glassfish/applications
Added a 'File watcher' configuration like this:
Screenshot-2024-01-17-at-13.36.58.png
Now when I edit an .xhtml file (or .js or anything in webapp) and save it it is automatically copied to the appropriate ./docker-dev-volumes/glassfish/applications directory and the change is available in the browser after page reload.
BalΓ‘zs Pataki (Jan 17 2024 at 12:46):The watcher setting can even be exported/imported, so it could be added by others easily without typing out the watcher config:
BalΓ‘zs Pataki (Jan 17 2024 at 13:00):It is also possible to create an "External tool" setting, which calls the same script as requested.
It can be invoked from a menu:
Or you can also assign a shortcut key to it.
The "Actions on save" like plugins in the IntelliJ marketplace are not really usable and are not actually necessary.
Philip Durbin π (Jan 17 2024 at 14:59):Sure, let's make it non-draft, resolve merge conflicts, etc.
Philip Durbin π (Jan 17 2024 at 21:23):I just put #9959 into the "in review" column on our project board: https://github.com/orgs/IQSS/projects/34
Philip Durbin π (Jan 17 2024 at 21:23):That is to say, I had it prioritized during our sprint kickoff today.
BalΓ‘zs Pataki (Jan 18 2024 at 07:42):@Philip Durbin Should I create a PR with my solution above? So that it could be handled together with #9959. Or maybe Oliver could add it to his PR if that is suitable.
Philip Durbin π (Jan 18 2024 at 12:23):Hmm. @BalΓ‘zs Pataki would it make sense for you to make a PR into @Oliver Bertuch 's PR?
Philip Durbin π (Jan 18 2024 at 12:41):@BalΓ‘zs Pataki I just invited you to join a readonly team so I can co-assign the issue to you. You should see the invite at https://github.com/IQSS
BalΓ‘zs Pataki (Jan 18 2024 at 12:43):Thanks, accepted invitation. I don't know if PR-s on PR-s is a thing or not, but I can try. :smile:
Philip Durbin π (Jan 18 2024 at 12:44):I do it all the time. :grinning:
BalΓ‘zs Pataki (Jan 18 2024 at 12:44):All right then!
Philip Durbin π (Jan 18 2024 at 12:45):I guess it's a PR, not an issue, but I co-assigned you to it: #9959
Philip Durbin π (Jan 18 2024 at 12:45):I wasn't following very closely what you wrote above, what you want to add, but I'm sure it's good stuff.
BalΓ‘zs Pataki (Jan 18 2024 at 12:46):I solves in Intellij what you already have in Netbeans, ie. fast xhtml/web stuff editing and instant reload in browser.
Philip Durbin π (Jan 18 2024 at 12:48):Ah, great.
Philip Durbin π (Jan 18 2024 at 13:07):I put this topic on the agenda for today's meeting: https://docs.google.com/document/d/1L2GPXGLpcrzMRjLLCFTXTu1NgK9dGX-tXVR_cn4elh0/edit?usp=sharing
BalΓ‘zs Pataki (Jan 19 2024 at 11:45):Submitted a PR with what I demoed yesterday: https://github.com/IQSS/dataverse/pull/10247
Philip Durbin π (Jan 22 2024 at 15:04):Thanks! I just added a review... wondering if we should comment out the volume mapping for now.
Philip Durbin π (Jan 22 2024 at 16:18):I just checked at standup and we'll pull this PR into the current sprint.
Philip Durbin π (Jan 22 2024 at 16:20):I assigned it to myself. @Oliver Bertuch are you interested in looking as well?
BalΓ‘zs Pataki (Jan 22 2024 at 16:21):Philip Durbin said:
Thanks! I just added a review... wondering if we should comment out the volume mapping for now.
OK, answered in the PR.
Philip Durbin π (Jan 24 2024 at 12:10):@BalΓ‘zs Pataki I see your :thumbs_up: on my suggestion on #10247? Do you want to go ahead and click "accept suggestion" to add the commit?
Philip Durbin π (Jan 24 2024 at 12:10):@Oliver Bertuch since you use IntelliJ are you interested in testing the scripts in #10247?
Philip Durbin π (Jan 25 2024 at 15:52):@BalΓ‘zs Pataki I just merged your PR: Intellij IDEA webapp hotreload #10247 Thanks!
BalΓ‘zs Pataki (Jan 25 2024 at 16:16):Great, thanks!
BalΓ‘zs Pataki (Feb 05 2024 at 13:40):TIL that docker compose has a "watch" function
https://docs.docker.com/compose/file-watch/
https://www.docker.com/blog/announcing-docker-compose-watch-ga-release/
One mode of its operation is that it can watch a given directory and sync any changes from that directory into the container.
Has anyone tried this feature yet? I can imagine that this could provide a generic solution for updating changed xhtml files in payara in docker regardless of the IDE.
Philip Durbin π (Feb 05 2024 at 14:18):Huh, cool. No, I haven't even heard of it.
Philip Durbin π (Feb 13 2024 at 17:08):I'm back on PR #9959, resolving merge conflicts and trying IntelliJ but I'm getting "Error: Debug: there are no runners for Payara Server: Payara Remote". I'm not sure what I've done wrong. Here's the error in context:
Screenshot-2024-02-13-at-12.07.18-PM.png
Philip Durbin π (Feb 13 2024 at 17:25):I gave up on IntelliJ. Netbeans works fine (-Dapp.deploy.skip is a huge time saver!). I guess I'll work on adding Netbeans tabs to the docs. I'll push what I've got first, the resolved merge conflicts.
Philip Durbin π (Feb 14 2024 at 04:17):@Oliver Bertuch since you made PR #9959 and I've hacked on it considerably, you might want to take a look. I just put it in "ready for review". Overall, works great! Thank you! :heart:
Philip Durbin π (Feb 14 2024 at 04:17):Of course others are welcome to review it as well. @BalΓ‘zs Pataki and @Juan Pablo Tosca Villanueva etc. You can preview the docs here: https://dataverse-guide--9959.org.readthedocs.build/en/9959/container/dev-usage.html#ide-triggered-redeployments
BalΓ‘zs Pataki (Feb 14 2024 at 13:26):Are you sure you tried the "Payara Remote" run config. It should look like this:
Yours look like the "Payara Local" run config.
Philip Durbin π (Feb 14 2024 at 13:34):Huh. Thanks, I'll take a look.
Oliver Bertuch (Feb 22 2024 at 07:29):I greatly extended the PR #9959 with more docs, more features and more... :smile_cat:
Oliver Bertuch (Feb 22 2024 at 07:30):@BalΓ‘zs Pataki I also adapted and incorporated your findings with the IntelliJ autocopy!
Oliver Bertuch (Feb 22 2024 at 07:30):This works pretty much perfect!
Oliver Bertuch (Feb 22 2024 at 07:31):Have a preview at the docs here: https://dataverse-guide--9959.org.readthedocs.build/en/9959/container/dev-usage.html#redeploying
BalΓ‘zs Pataki (Feb 22 2024 at 07:56):Cool, can't wait to try it!
You write in the documentation:
This imitates the Netbeans builtin function to copy changes to files under src/main/webapp into a destination folder.
It is different in the way that it will copy the files into the running container deployment without using a bind mount.
In my original solution I actually needed a bind volume. Do you really copy now right inside the container?
Oliver Bertuch (Feb 22 2024 at 07:57):Yes. Much simpler setup, no need to adapt any path.
BalΓ‘zs Pataki (Feb 22 2024 at 07:57):That's great!
Oliver Bertuch (Feb 22 2024 at 07:58):docker cp to the rescue
Oliver Bertuch (Feb 22 2024 at 07:58):Also removed the Perl dependency :wink:
BalΓ‘zs Pataki (Feb 22 2024 at 08:02):I mentioned couple of days ago that docker compose has a "watch" function
https://docs.docker.com/compose/file-watch/
https://www.docker.com/blog/announcing-docker-compose-watch-ga-release/
Haven't tried it yet, but this may provide an even simpler and generic solution.
Oliver Bertuch (Feb 22 2024 at 08:09):Sure, feel free to try out stuff. Much appreciated!
Oliver Bertuch (Feb 22 2024 at 08:11):@Philip Durbin question for you when you return to desk today: should we move the "dev usage" page into the "running dv in d/backend dev" page?
Oliver Bertuch (Feb 22 2024 at 08:13):Should it even live within the Dev Guide, now that the standard dev env is dockerized?
Philip Durbin π (Feb 23 2024 at 20:26):As discussed in the container meeting yesterday, let's think more about moving the "dev usage" page in the future.
Philip Durbin π (Feb 23 2024 at 20:27):More importantly, I just merged #9959! Thanks, all! Enjoy the faster redeploy!
Aamir Muhammad (Mar 19 2024 at 11:26):Hi Everyone,
I was trying to run dataverse locally. first I run this command to run on foreground mvn -Pct clean package docker:run. All the container started and application was running on 8080.
The next step I was trying the faster redeploy. I have intellij ultimate. I installed Payara Platform Tools. and following the steps mentioned on this link https://dataverse-guide--9959.org.readthedocs.build/en/9959/container/dev-usage.html#redeploying
while adding dataverse:war I cannot see any artifact. I hope someone can help me with this.
Philip Durbin π (Mar 19 2024 at 11:27):Hmm. @BalΓ‘zs Pataki @Oliver Bertuch you use IntelliJ more than I do. Any tips?
Philip Durbin π (Mar 19 2024 at 11:28):Actually, @Aamir Muhammad can you please look at the preview version at https://preview.guides.gdcc.io/en/develop/container/dev-usage.html#redeploying ?
Philip Durbin π (Mar 19 2024 at 11:28):There are many more screenshots.
Philip Durbin π (Mar 19 2024 at 11:28):(preview of how the guides will look for Dataverse 6.2)
Aamir Muhammad (Mar 19 2024 at 11:29):The same I am following with screen shot. here is my configuration
Oliver Bertuch (Mar 19 2024 at 11:49):I'm not sure what's going on here or how to reproduce... When you click the "+" sign, you can't select dataverse:war? (See screenshot at https://preview.guides.gdcc.io/en/develop/_images/intellij-payara-config-server.png)
Oliver Bertuch (Mar 19 2024 at 11:51):Also as you are using Maven to start the containers, don't forget to use -Dapp.skipDeploy!
Aamir Muhammad (Mar 19 2024 at 13:46):sorry I am new to most of these things. but this time I tried mvn -Pct clean package docker:run -Dapp.skipDeploy and was not able to run the application.
than I try to start it again using the old command mvn -Pct clean package docker:run and the application didn't start. I am also deleting the target and docker-dev-volumes folder everytime.
Anyhow than I try this command mvn -Pct package && docker compose -f docker-compose-dev.yml up -dthe application is started and can see the docker containers. but still when I click the "+" sign, I am not able to see. instead it gives me some options to select from which I am not able to take screenshot. but I am selecting Build artifacts than I am getting this page which is empty
dataverse-artifact-1.png
Oliver Bertuch (Mar 19 2024 at 13:48):Sorry, I have no idea what's going on on your machine. :shrug:
Philip Durbin π (Mar 19 2024 at 13:53):I would say let's talk about it more at our weekly meeting ( https://ct.gdcc.io ) but this week we have a special guest so we'll probably be quite busy. Also, I'm not sure if that time works for you, @Aamir Muhammad
Aamir Muhammad (Mar 19 2024 at 14:04):@Philip Durbin sure I will be happy to join. Thanks for the invitation.
Philip Durbin π (Mar 19 2024 at 14:14):You're very welcome but again, it'll be a busy meeting. I expect we'll spend the whole time talking about Kubernetes and OpenShift with our guest from Red Hat.
Juan Pablo Tosca Villanueva (Mar 23 2024 at 19:47):I was going trough the new guides for Intellij and when I try the IntelliJ approach with the volumes initialized on the command line postgress exits with the following message:
2024-03-23 19:47:09.946 UTC [1] DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 13.14 (Debian 13.14-1.pgdg120+2).
dev_postgres exited with code 1
:thinking:
Oliver Bertuch (Mar 23 2024 at 19:57):You probably need to delete the volumes and do a fresh deployment.
Oliver Bertuch (Mar 23 2024 at 19:57):Not sure when you tested with Postgres 16, but I had that before as well.
Juan Pablo Tosca Villanueva (Mar 23 2024 at 19:59):That definitely works, if I delete the volumes it just initializes them and works fine, but I would love to be able to have it working on both ways. :thinking:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 19:59):Not sure if I am doing something special to use 16
Oliver Bertuch (Mar 23 2024 at 19:59):Not sure either :innocent:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:00):First time using Intellij so I just followed the guide as best as I could
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:00):Probably need to update the part of the guide now that the port is 4949 by default :rolling_on_the_floor_laughing:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:01):Oh wait, it is 4848 because it is the "local" instance of payara right?
Oliver Bertuch (Mar 23 2024 at 20:02):No no no that is all ancient history
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:02):So it should be 4949?
Oliver Bertuch (Mar 23 2024 at 20:02):On 4848 is a little proxy that makes SSL go away so the Payara Tools work
Oliver Bertuch (Mar 23 2024 at 20:03):So please stick to the guide and use 4848 :blush:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:04):Question regarding the guides
Oliver Bertuch (Mar 23 2024 at 20:04):The Postgres thing should be independent from IntelliJ. I saw this on CLI, too
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:05):On the step 4 it says "NOTE: You might need to change the Docker Compose executable in your IDE settings to docker if you have no docker-compose bin (File > Settings > Build > Docker > Tools)."
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:05):I am on MacOS
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:05):And the menu doesn't show settings under File
Oliver Bertuch (Mar 23 2024 at 20:06):Simply try docker compose on the CLI. If you can execute docker-compose you're fine.
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:06):Actually to find the configurations had to go trough Run > Edit Configurations
Oliver Bertuch (Mar 23 2024 at 20:07):If it's docker compose only, you need to change the setting
Oliver Bertuch (Mar 23 2024 at 20:08):It should be a global setting, too. Just go to settings and use the search bar to filter for Docker.
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:08):Oh but I was just never able to find that option so I was wondering if this was a difference because the version of the OS or the IDE or I was just looking at the wrong place, like "File > Settings > Build > Docker > Tools" just doesn't exist for me
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:09):I was able to run it and get the configuration with the hot swap
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:09):but I was wondering if this is something we need to update or change or I was just looking at the wrong side
Oliver Bertuch (Mar 23 2024 at 20:10):Maybe. I'm on Linux, that might be having a different menu layout
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:10):
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:11):I think this is the equivalent on Mac
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:11):
Oliver Bertuch (Mar 23 2024 at 20:11):No, that's run configs
Oliver Bertuch (Mar 23 2024 at 20:11):These are the same on every OS
Oliver Bertuch (Mar 23 2024 at 20:12):Is there a menu under the IntelliJ entry in the menu?
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:12):Yes!
Oliver Bertuch (Mar 23 2024 at 20:12):I bet a dollar there's a settings menu entry
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:12):
Oliver Bertuch (Mar 23 2024 at 20:13):There we go :grinning_face_with_smiling_eyes:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:14):Should I add a quick edit there to add this? :thinking:
Oliver Bertuch (Mar 23 2024 at 20:14):Sure. Not sure what the best way for that would be
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:16):NOTE: You might need to change the Docker Compose executable in your IDE settings to docker if you have no docker-compose bin File > Settings > Build > Docker > Tools or IntelliJ IDEA > Settings > Build > Docker > Tools (if you are on MacOS)
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:16):How about that
Oliver Bertuch (Mar 23 2024 at 20:18):Hmm... maybe it should be splitted ? The path is the same once you're in the settings
Oliver Bertuch (Mar 23 2024 at 20:18):The only difference is how to get there
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:19):How about
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:21):NOTE: You might need to change the Docker Compose executable in your IDE settings to docker if you have no docker-compose bin, Start from the File menu if you are on Linux or IntelliJ IDEA on Mac and then go to Settings > Build > Docker > Tools.
Oliver Bertuch (Mar 23 2024 at 20:21):We could add Windows, too. I suppose it will be under File, too.
Oliver Bertuch (Mar 23 2024 at 20:21):Aside from that I like it
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:22):I have to test that on windows :rolling_on_the_floor_laughing:
Oliver Bertuch (Mar 23 2024 at 20:23):Oh so you're no longer our Windows and VsCode man then?
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:23):I use Windows for personal use :smile:
Oliver Bertuch (Mar 23 2024 at 20:23):Haha same here :laughing:
Oliver Bertuch (Mar 23 2024 at 20:23):Gaming and content creation
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:24):I wanted to see if following the intellij and NetBeans guide could shed some light and help me find my way to make this work on VS Code lol
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:24):What do you play @Oliver Bertuch ?
Oliver Bertuch (Mar 23 2024 at 20:24):I see! Splendid idea!
Oliver Bertuch (Mar 23 2024 at 20:25):Juan Pablo Tosca Villanueva said:
What do you play Oliver Bertuch ?
Let's move that to a non public channel or at least a different topic in #community :grinning_face_with_smiling_eyes:
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:26):Me always getting off-topic :rolling_on_the_floor_laughing: classic
Juan Pablo Tosca Villanueva (Mar 23 2024 at 20:31):Ok I just checked, it is under File on windows
Juan Pablo Tosca Villanueva (Mar 24 2024 at 00:02):I spent a lot of time with vs-code today and no luck :(
Oliver Bertuch (Mar 24 2024 at 00:36):That's a real bummer!
Juan Pablo Tosca Villanueva (Mar 25 2024 at 00:08):Oliver Bertuch said:
Not sure when you tested with Postgres 16, but I had that before as well.
This is on the pom.xml for the ct profile so I wonder why IJ takes the 13 :thinking:
<postgresql.server.version>16</postgresql.server.version>
Ok so I found the problem and a possible solution, you guys will let me know (Or @Oliver Bertuch since he is probably the one that uses IJ the most :rolling_on_the_floor_laughing: ) It seems to me that since this configuration utilizes directly the compose file and the PG version variable comes from the pom.xml which is processed by the mvn docker pluggin. The solution I found was this:
Juan Pablo Tosca Villanueva (Mar 25 2024 at 18:11):
Juan Pablo Tosca Villanueva (Mar 25 2024 at 18:13):Adding POSTGRES_VERSION=16 to the env variables makes and PG is running on 16, this works for me but since I am not the author of the guide I am not sure 100% if I missed something or if we should add this to the guide
Juan Pablo Tosca Villanueva (Mar 25 2024 at 18:14):(I am writing this as I triple check the guide that I didn't missed this somewhere) :laughing:
Oliver Bertuch (Mar 25 2024 at 18:32):No probably we overlooked sth somewhere else.
Juan Pablo Tosca Villanueva (Mar 25 2024 at 18:58):Ok I sent a PR, thanks :smile:
Philip Durbin π (Jul 26 2024 at 14:11):Philip Durbin said:
I am having trouble getting
-Dapp.deploy.skipto work though. Maybe I'll try the env var.
I'm using mvn -Pct docker:run -Dapp.skipDeploy and it works fine. I'm not sure if it's because it was renamed or what, but it works great. A real time saver.
Oliver Bertuch (Jul 26 2024 at 14:23):Great! Should we resolve this topic? There are documentation bits for this :smile: https://guides.dataverse.org/en/latest/container/base-image.html#tunables has SKIP_DEPLOY and https://guides.dataverse.org/en/latest/container/dev-usage.html#ide-trigger-code-deploy mentions the Maven property :innocent:
Philip Durbin π (Jul 26 2024 at 14:46):Sure, resolving. Everyone should feel free to open fresh topics.
Notification Bot (Jul 26 2024 at 14:46):Philip Durbin has marked this topic as resolved.
Last updated: Oct 30 2025 at 05:14 UTC