testing S3 · containers · Zulip Chat Archive

It should be possible to supply all the necessary S3 config via MPCONFIG, right? That way I can continue my streak of doing all development with containers. :happy: It's been over a month! :tada:

I'd rather not switch back to Payara running directly on my Mac if I can avoid it!

Oliver Bertuch (Apr 26 2023 at 14:28):

Yes it is possible but only dirty, because the files part is not yet supporting MPCONFIG. So you need to use the "dataverse_files_xxx_option=..." env var hack. Mind the casing.

Philip Durbin 🚀 (Apr 26 2023 at 14:30):

Interesting. Do you have a list of env vars handy I can start with? If not, I'll look at the guides.

Oliver Bertuch (Apr 26 2023 at 14:32):

Philip Durbin 🚀 (Apr 26 2023 at 14:33):

Oliver Bertuch (Apr 26 2023 at 14:34):

Philip Durbin 🚀 (Apr 26 2023 at 14:35):

Oliver Bertuch (Apr 26 2023 at 14:36):

Sure. Don't put too much effort into it - should go away anyway with making the config of storage via MPCONFIG available. Someone needs to code that...

Philip Durbin 🚀 (Apr 26 2023 at 14:36):

Philip Durbin 🚀 (Apr 26 2023 at 14:37):

Am I right that we don't plan to create a PR until milestone G? "MPCONFIG for remaining dozen JVM settings (Size: 80)"

Oliver Bertuch (Apr 26 2023 at 14:38):

Philip Durbin 🚀 (Apr 26 2023 at 14:38):

Oh, wait. Milestone D. "Configurability: Make storage configuration use MPCONFIG (Size: 33)"

Oliver Bertuch (Apr 26 2023 at 14:38):

So all I have for now is a local branch and a stash of changes on my laptop for the work on the file storage thing

Philip Durbin 🚀 (Apr 26 2023 at 14:39):

Oliver Bertuch (Apr 26 2023 at 14:39):

There was some discussion a while ago during some tech hour about if it would be OK to require a list of storage names which IIRC was eventually put on hold with "hmm yeah dunno maybe fine lets see"

Oliver Bertuch (Apr 26 2023 at 14:40):

Well there is 7000, but maybe we can close that and create issues for the remaining parts

Philip Durbin 🚀 (Apr 26 2023 at 14:40):

Oliver Bertuch (Apr 26 2023 at 14:40):

Philip Durbin 🚀 (Apr 26 2023 at 14:40):

Oliver Bertuch (Apr 26 2023 at 14:41):

Oliver Bertuch (Apr 26 2023 at 14:42):

Glad that it seems to be appreciated now among the devs... That lookup thing seems to be intriguing... :grinning_face_with_smiling_eyes:

Philip Durbin 🚀 (Apr 26 2023 at 14:43):

Oliver Bertuch (Apr 26 2023 at 14:44):

Hrhr... I remember we talked about this stuff long time ago and one of my arguments was "you want this for containers" :yum:

Oliver Bertuch (Apr 26 2023 at 14:45):

Oliver Bertuch (Apr 26 2023 at 14:46):

IIRC this is part of another milestone, would it make sense to take a look into how spin up the storage container, too?

Philip Durbin 🚀 (Apr 26 2023 at 14:46):

I'm going to use real S3. It's possible to configure it with the hacks above, right?

Oliver Bertuch (Apr 26 2023 at 14:46):

Oliver Bertuch (Apr 26 2023 at 14:47):

Oh wait: one thing... What I already had a chance to bring in for MPCONFIG and S3 is the keys thing

Oliver Bertuch (Apr 26 2023 at 14:47):

So you don't need to think about how to deal with the AWS profile and config files in ~/.aws

Oliver Bertuch (Apr 26 2023 at 14:47):

That is documented in the main section, it's been sitting there for a while and made @Don Sizemore happy before

Philip Durbin 🚀 (May 01 2023 at 19:07):

I started editing my .env file today but got so confused I gave up and just ran Payara locally on my Mac. These are the (surely incorrect) values I was editing when I gave up:

DATAVERSE_FILES_STORAGE__DRIVER__ID=s3
DATAVERSE_FILES_LOCAL_TYPE=s3
DATAVERSE_FILES_LOCAL_LABEL=S3
DATAVERSE.FILES.<ID>.ACCESS-KEY=
DATAVERSE.FILES.<ID>.SECRET-KEY=

I'm happy to try again but I'd need some hand holding. Or more time (than I want to spend today) studying the code. Or I can just wait for MPCONFIG. :sweat_smile:

Oliver Bertuch (May 02 2023 at 07:16):

Oliver Bertuch (May 02 2023 at 07:18):

Philip Durbin 🚀 (May 02 2023 at 11:03):

Oliver Bertuch (May 02 2023 at 11:29):

Both Secret and Access Key are already using MPCONFIG, so you may use either way

Philip Durbin 🚀 (May 17 2023 at 12:14):

Oliver Bertuch (May 17 2023 at 12:15):

Philip Durbin 🚀 (May 17 2023 at 12:19):

Philip Durbin 🚀 (May 19 2023 at 15:23):

Philip Durbin 🚀 (May 19 2023 at 15:24):

diff --git a/docker-compose-dev.yml b/docker-compose-dev.yml
index 30c55661a2..298a20b6ff 100644
--- a/docker-compose-dev.yml
+++ b/docker-compose-dev.yml
@@ -13,6 +13,14 @@ services:
       - DATAVERSE_DB_PASSWORD=secret
       - DATAVERSE_DB_USER=${DATAVERSE_DB_USER}
       - DATAVERSE_FEATURE_API_BEARER_AUTH=1
+      - dataverse_files_storage__driver__id=s3
+      - dataverse_files_s3_type=s3
+      - dataverse_files_s3_label=S3
+      - dataverse_files_s3_custom__endpoint__url=s3.us-east-2.amazonaws.com
+      - dataverse_files_s3_custom__endpoint__region=us-east-2
+      - dataverse_files_s3_bucket__name=pdurbin
+      - dataverse_files_s3_access__key=REDACTED
+      - dataverse_files_s3_secret__key=REDACTED
     ports:
       - "8080:8080" # HTTP (Dataverse Application)
       - "4848:4848" # HTTP (Payara Admin Console)

Philip Durbin 🚀 (May 19 2023 at 15:25):

As you can see earlier in this thread, I was using UPPER CASE before, like any good sysadmin. But it seems like one must use lower case.

Oliver Bertuch (May 19 2023 at 19:09):

Yeah like I said (did I?) before in this thread, this is a limitation because we have a hack in place for now until the storage subsystem is MPCONFIGified

Oliver Bertuch (May 19 2023 at 19:09):

Philip Durbin 🚀 (May 19 2023 at 19:34):

Oliver Bertuch (May 19 2023 at 19:35):

Philip Durbin 🚀 (May 19 2023 at 19:39):

Philip Durbin 🚀 (Aug 11 2023 at 15:18):

I just thought I'd point out that this issue made it to the board. Currently it's in "sprint ready": Add S3 tests to the regular integration test suite #6783

Philip Durbin 🚀 (Sep 07 2023 at 11:02):

Any discussion or prep for this S3 testing work? We could discuss it today or a future ct meeting if people want to.

Philip Durbin 🚀 (Sep 19 2023 at 19:48):

Oliver Bertuch (Sep 19 2023 at 19:52):

Oliver Bertuch (Sep 19 2023 at 19:53):

Philip Durbin 🚀 (Sep 19 2023 at 19:56):

Philip Durbin 🚀 (Sep 19 2023 at 19:59):

Philip Durbin 🚀 (Oct 04 2023 at 18:26):

Should I pick up this issue? Add S3 tests to the regular integration test suite #6783

Philip Durbin 🚀 (Oct 04 2023 at 18:27):

@Oliver Bertuch are we in a better spot now that #9273 has been merged? Do you have ideas on how to proceed with S3 testing?

Oliver Bertuch (Oct 04 2023 at 18:27):

Oliver Bertuch (Oct 04 2023 at 18:28):

Philip Durbin 🚀 (Oct 04 2023 at 18:30):

Oliver Bertuch (Oct 04 2023 at 18:30):

Oliver Bertuch (Oct 04 2023 at 18:31):

Philip Durbin 🚀 (Oct 04 2023 at 18:31):

Oliver Bertuch (Oct 04 2023 at 18:32):

The way I see it, we should definitely take a look at adding integration test using Testcontainers

Oliver Bertuch (Oct 04 2023 at 18:32):

There are basically 2 ways to do this: go for LocalStack or use some other S3 compatible thing

Oliver Bertuch (Oct 04 2023 at 18:33):

Oliver Bertuch (Oct 04 2023 at 18:34):

As we are using the official AWS Java SDK, it might be worth going for LocalStack

Philip Durbin 🚀 (Oct 04 2023 at 18:34):

Oliver Bertuch (Oct 04 2023 at 18:37):

You will have to do some ugly hacks with configuration... Sorry, but most dataverse.files S3 options ain't MPCONFIG enabled yet, so no nice simple wrapper with @JvmSetting

Philip Durbin 🚀 (Oct 04 2023 at 18:37):

Oliver Bertuch (Oct 04 2023 at 18:37):

Philip Durbin 🚀 (Oct 04 2023 at 18:38):

Oliver Bertuch (Oct 04 2023 at 18:38):

Oliver Bertuch (Oct 04 2023 at 18:39):

Oliver Bertuch (Oct 04 2023 at 18:40):

Philip Durbin 🚀 (Oct 04 2023 at 18:42):

@Container
static KeycloakContainer keycloakContainer = new KeycloakContainer("quay.io/keycloak/keycloak:22.0")

Philip Durbin 🚀 (Oct 04 2023 at 19:17):

Oliver Bertuch (Oct 10 2023 at 11:25):

Oliver Bertuch (Oct 10 2023 at 11:26):

Would it be in scope to refactor the dataverse.files.<id> thing for this? It would make writing these tests much easier...

Philip Durbin 🚀 (Oct 10 2023 at 11:47):

Philip Durbin 🚀 (Oct 10 2023 at 11:48):

I don't know what to say about the refactoring. I'm buried in HR paperwork again and we have a visitor all week (which I'm looking forward to). I haven't been able to work on S3 testing at all. :disappointed:

Oliver Bertuch (Oct 10 2023 at 11:54):

It's your first day of the week, so no worries. And you did cleanse out loads of stuff over the weekend! So much dust came off!

Oliver Bertuch (Oct 10 2023 at 12:24):

In case it helps: there's also https://github.com/gaul/s3proxy which might be interesting for testing (transient memory storage), but might also add support to drop more storage backends and rely on a different component to create a bridge between Dataverse -> S3 -> sth else

Oliver Bertuch (Oct 10 2023 at 12:27):

If you are looking for sth. more lightweight not involving testcontainers, maybe https://github.com/findify/s3mock would be worth a look

Philip Durbin 🚀 (Oct 16 2023 at 13:30):

#6783 starts by talking about S3AccessIT, how we should add it to our Jenkins runs. But if we switch to S3 storage we won't be testing the filesystem anymore.

Philip Durbin 🚀 (Oct 16 2023 at 13:30):

Oliver Bertuch (Oct 16 2023 at 15:02):

Oliver Bertuch (Oct 16 2023 at 15:03):

Philip Durbin 🚀 (Oct 16 2023 at 15:18):

Ha. Well, I thought I'd start with what we requested in the issue. I don't want to lose sight of that.

Philip Durbin 🚀 (Oct 16 2023 at 15:19):

Oliver Bertuch (Oct 16 2023 at 15:25):

Oliver Bertuch (Oct 16 2023 at 15:26):

Wouldn't that safe the trouble of jumping through the E2E hoops with config and all?

Philip Durbin 🚀 (Oct 16 2023 at 15:27):

Philip Durbin 🚀 (Oct 16 2023 at 15:29):

Oliver Bertuch (Oct 16 2023 at 18:46):

To me this test looks like it's simply testing an upload and a deletion, making sure along the way the prefix is correct

Oliver Bertuch (Oct 16 2023 at 18:46):

Philip Durbin 🚀 (Oct 16 2023 at 19:30):

You saying that using Testcontainers we could create a dataset? Let me go look at the test you added.

Philip Durbin 🚀 (Oct 16 2023 at 19:41):

Oliver Bertuch (Oct 16 2023 at 19:44):

Philip Durbin 🚀 (Oct 16 2023 at 19:46):

Oliver Bertuch (Oct 16 2023 at 20:35):

Philip Durbin 🚀 (Oct 16 2023 at 20:37):

Philip Durbin 🚀 (Oct 17 2023 at 18:48):

I'm sending "hello" into S3AccessIO and pulling it out again, making sure it's still "hello". Code coverage of that class has jumped from 0% to 7.88%!

Philip Durbin 🚀 (Oct 17 2023 at 20:34):

Oh, wait. I'm not sure code coverage is being reported properly in Netbeans. :thinking:

Philip Durbin 🚀 (Oct 17 2023 at 20:37):

Philip Durbin 🚀 (Oct 17 2023 at 20:38):

Ok, 67% when I look at

target/site/jacoco-integration-test-coverage-report/edu.harvard.iq.dataverse.authorization.providers.oauth2.oidc/OIDCAuthProvider.html

. Phew. :sweat_smile:

Philip Durbin 🚀 (Oct 17 2023 at 21:16):

I wonder if there's a way to skip MOST tests except the one I'm working on. Something like this: mvn verify -DskipTests=true -Dtest=S3AccessIOIT.

Philip Durbin 🚀 (Oct 17 2023 at 21:19):

Philip Durbin 🚀 (Oct 17 2023 at 21:42):

Oliver Bertuch (Oct 17 2023 at 22:08):

Philip Durbin 🚀 (Oct 18 2023 at 19:57):

Thanks, I'll circle back to excluding Keycloak when testing S3 but right now I'm trying to configure multiple stores. I must be doing it wrong.

Philip Durbin 🚀 (Oct 18 2023 at 19:58):

- dataverse_files_storage__driver__id=file-1
- dataverse_files_file-1_type=file
- dataverse_files_file-1_label=Filesystem
- dataverse_files_file-1_directory=${STORAGE_DIR}/store
- dataverse_files_s3-3reals3_type=s3
- dataverse_files_s3-3reals3_label=Real S3
- dataverse_files_s3-3reals3_custom__endpoint__url=s3.us-east-2.amazonaws.com
- dataverse_files_s3-3reals3_custom__endpoint__region=us-east-2
- dataverse_files_s3-3reals3_bucket__name=pdurbin
- dataverse_files_s3-3reals3_upload__redirect=true
- dataverse_files_s3-3reals3_access__key=REDACTED
- dataverse_files_s3-3reals3_secret__key=REDACTED

Philip Durbin 🚀 (Oct 18 2023 at 19:59):

Philip Durbin 🚀 (Oct 18 2023 at 20:01):

Philip Durbin 🚀 (Oct 18 2023 at 20:02):

{
    "status": "OK",
    "data": {
        "Filesystem": "file-1"
    }
}

Philip Durbin 🚀 (Oct 18 2023 at 20:03):

Philip Durbin 🚀 (Oct 18 2023 at 20:06):

I've only ever configured a single store in containerized Dataverse. I'm not sure if it's supported.

Eryk Kulikowski (Oct 19 2023 at 08:17):

I am testing with two stores: one s3 and one file store. I use JVM_ARGS in my docker-compose like this:

    environment:
      JVM_ARGS:
        -Ddataverse.timerServer=true
        -Xmx24g
        -Xms4g
        -XX:+HeapDumpOnOutOfMemoryError
        -XX:MaxMetaspaceSize=2g
        -XX:MetaspaceSize=256m
        -XX:+UseG1GC
        -XX:+UseStringDeduplication
        -XX:+DisableExplicitGC
        -Ddataverse.files.s3.upload-out-of-band=true
        -Ddataverse.api.allow-incomplete-metadata=true
        -Ddataverse.ui.allow-review-for-incomplete=false
        -Ddataverse.ui.show-validity-filter=true
        -Ddataverse.files.directory=/data
        -Ddataverse.files.uploads=/uploads
        -Ddataverse.files.s3.type=s3
        -Ddataverse.files.s3.label=s3
        -Ddataverse.files.s3.bucket-name=dataverse-local
        -Ddataverse.files.s3.download-redirect=true
        -Ddataverse.files.s3.upload-redirect=true
        -Ddataverse.files.s3.path-style-access=true
        -Ddataverse.files.s3.custom-endpoint-url=https://rdmo.icts.kuleuven.be
        -Ddataverse.files.s3.custom-endpoint-region=us-east-1
        -Ddataverse.files.s3.disable-tagging=true
        -Ddataverse.files.file.type=file
        -Ddataverse.files.file.label=file
        -Ddataverse.files.file.directory=/data

Notice the space on the end of each line (otherwise it will not work).
This is the output from listing the storage:

{
   "status": "OK",
   "data":    {
      "s3": "s3",
      "file": "file"
   }
}

Eryk Kulikowski (Oct 19 2023 at 08:22):

It looks like Zulip strips the spaces, but there is space at the end of each line, before the end-of-line character, it functions as a separator for the jvm args.

Eryk Kulikowski (Oct 19 2023 at 08:27):

    volumes:
      - /path/to/aws/config/.aws:/opt/payara/.aws
      -...

Eryk Kulikowski (Oct 19 2023 at 08:51):

  dataverse:
    image: ${DATAVERSE_IMAGE_TAG}
    hostname: dataverse
    user: payara
    networks:
      - app_net
      - db_net
    ports:
      - "7005:4848"
      - "7008:8080"
    environment:
      TZ: "Europe/Brussels"
      DATAVERSE_DB_HOST: db
      DATAVERSE_DB_PORT: 5432
      DATAVERSE_DB_USER: ${DB_USER}
      DATAVERSE_DB_NAME: ${DB_NAME}
      ENABLE_JDWP: 0
      DATAVERSE_MAIL_HOST: ${MAIL_SERVICE_HOST}
      DATAVERSE_MAIL_FROM: ${MAIL_SENDER}
      dataverse_fqdn: ${FQDN}
      dataverse_siteUrl: ${DATAVERSE_URL}
      dataverse_files_storage__driver__id: ${DRIVER_ID}
      db_SystemEmail: ${MAIL_SENDER_NAME}
      CUSTOM_INSTALL: "/opt/payara/custominstall"
      API_DEBUG: "false"
      LANG_DIR: "/languages"
      STORAGE_DIR: "/data"
      SECRETS_DIR: "/run/secrets"
      DUMPS_DIR: "/dumps"
      UPLOAD_DIR: "/uploads"
      MP_CONFIG_PROFILE: ${RDM_STAGE}
      JVM_ARGS:
        -Ddataverse.timerServer=true
        -Xmx24g
        -Xms4g
        -XX:+HeapDumpOnOutOfMemoryError
        -XX:MaxMetaspaceSize=2g
        -XX:MetaspaceSize=256m
        -XX:+UseG1GC
        -XX:+UseStringDeduplication
        -XX:+DisableExplicitGC
        -Ddataverse.files.s3.upload-out-of-band=true
        -Ddataverse.api.allow-incomplete-metadata=true
        -Ddataverse.ui.allow-review-for-incomplete=false
        -Ddataverse.ui.show-validity-filter=true
        -Ddataverse.files.directory=/data
        -Ddataverse.files.uploads=/uploads
        -Ddataverse.files.s3.type=s3
        -Ddataverse.files.s3.label=s3
        -Ddataverse.files.s3.bucket-name=dataverse-local
        -Ddataverse.files.s3.download-redirect=true
        -Ddataverse.files.s3.upload-redirect=true
        -Ddataverse.files.s3.path-style-access=true
        -Ddataverse.files.s3.custom-endpoint-url=https://rdmo.icts.kuleuven.be
        -Ddataverse.files.s3.custom-endpoint-region=us-east-1
        -Ddataverse.files.s3.disable-tagging=true
        -Ddataverse.files.file.type=file
        -Ddataverse.files.file.label=file
        -Ddataverse.files.file.directory=/data
        -Ddataverse.solr.host=solr
        -Ddataverse.mail.cc-support-on-contact-email=rdr@kuleuven.be
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config/dataverse:/opt/payara/custominstall
      - ./config/dataverse/set_language.sh:/opt/payara/scripts/init.d/set_language.sh
      - ./config/dataverse/change-admin-password.sh:/opt/payara/scripts/init.d/change-admin-password.sh
      - ${DV_FILES}:/data
      - ${DV_DUMPS}:/dumps
      - ${DV_UPLOADS}:/uploads
      - ${DV_LANG_DIR}:/languages
      - ${DOCROOT}:/opt/payara/appserver/glassfish/domains/domain1/docroot
      - ${SECRETS}:/run/secrets
      - ${SECRETS}/microprofile:/secrets
      - ${DV_PATH}/aws:/opt/payara/.aws
    depends_on:
      - proxy
      - db
      - index
    restart: unless-stopped
    privileged: false

I use a standard image built with maven. I think that the interesting script is the one that changes the admin password (you see it mounted in volumes):

#!/usr/bin/env bash

# Check and load secrets
if [ ! -s "${SECRETS_DIR}/admin/password" ]; then
  echo "No admin password present. Failing."
  exit 126
fi
ADMIN_PASSWORD=$(cat "${SECRETS_DIR}/admin/password")

echo "AS_ADMIN_PASSWORD=admin" > /tmp/password-change-file.txt
echo "AS_ADMIN_NEWPASSWORD=${ADMIN_PASSWORD}" >> /tmp/password-change-file.txt
echo "AS_ADMIN_PASSWORD=${ADMIN_PASSWORD}" >> ${PASSWORD_FILE}
asadmin --user=${ADMIN_USER} --passwordfile=/tmp/password-change-file.txt change-admin-password --domain_name=${DOMAIN_NAME} || true
rm /tmp/password-change-file.txt

Philip Durbin 🚀 (Oct 19 2023 at 11:08):

@Eryk Kulikowski that actually works?!? JVM_ARGS?!? @Oliver Bertuch have you seen this?

Philip Durbin 🚀 (Oct 19 2023 at 11:08):

Eryk Kulikowski (Oct 19 2023 at 12:10):

It works, you can pass any JVM/microprofile settings like this, just remember to add a space ( " ") after each argument. Otherwise, it will be sticking together. You can check if it worked in your server log when using it:

[Entrypoint] running /opt/payara/scripts/startInForeground.sh in foreground
Executing Payara Server with the following command line:
/opt/java/openjdk/bin/java
-cp
/opt/payara/appserver/glassfish/domains/domain1/lib/ext/*:/opt/payara/appserver/glassfish/modules/glassfish.jar
-Ddataverse.timerServer=true
-Xmx24g
-Xms4g
-XX:+HeapDumpOnOutOfMemoryError
-XX:MaxMetaspaceSize=2g
-XX:MetaspaceSize=256m
-XX:+UseG1GC
-XX:+UseStringDeduplication
-XX:+DisableExplicitGC
-Ddataverse.files.s3.upload-out-of-band=true
-Ddataverse.api.allow-incomplete-metadata=true
-Ddataverse.ui.allow-review-for-incomplete=false
-Ddataverse.ui.show-validity-filter=true
-Ddataverse.files.directory=/data
-Ddataverse.files.uploads=/uploads
-Ddataverse.files.s3.type=s3
-Ddataverse.files.s3.label=s3
-Ddataverse.files.s3.bucket-name=dataverse-local
-Ddataverse.files.s3.download-redirect=true
-Ddataverse.files.s3.upload-redirect=true
-Ddataverse.files.s3.path-style-access=true
-Ddataverse.files.s3.custom-endpoint-url=https://rdmo.icts.kuleuven.be
-Ddataverse.files.s3.custom-endpoint-region=us-east-1
-Ddataverse.files.s3.disable-tagging=true
-Ddataverse.files.file.type=file
-Ddataverse.files.file.label=file
-Ddataverse.files.file.directory=/data
-Ddataverse.solr.host=solr
-Ddataverse.mail.cc-support-on-contact-email=rdr@kuleuven.be
-Ddataverse.lang.directory=/languages

-XX:+UnlockDiagnosticVMOptions
...
-Dfelix.fileinstall.bundles.startTransient=true

-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
...
-Djava.library.path=/opt/payara/appserver/glassfish/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
com.sun.enterprise.glassfish.bootstrap.ASMain
/opt/payara/config/pre-boot-commands.asadmin
-upgrade
false
-read-stdin
true
-postbootcommandfile
/opt/payara/config/post-boot-commands.asadmin
-domainname
domain1
-domaindir
/opt/payara/appserver/glassfish/domains/domain1
-asadmin-args
--host,,,localhost,,,--port,,,4848,,,--user,,,admin,,,--passwordfile,,,/opt/payara/passwordFile,,,--secure=false,,,--terse=false,,,--extraterse=false,,,--echo=false,,,--interactive=false,,,--autoname=false,,,start-domain,,,--verbose=false,,,--watchdog=false,,,--debug=false,,,--domaindir,,,/opt/payara/appserver/glassfish/domains,,,domain1
-instancename
server
-type
DAS
-verbose
false
-asadmin-classpath
/opt/payara/appserver/glassfish/lib/client/appserver-cli.jar
-debug
false
-asadmin-classname
com.sun.enterprise.admin.cli.AdminMain
-watchdog
false

Launching Payara Server on Felix platform
Oct 18, 2023 2:02:14 PM com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner createBundleProvisioner
INFO: Create bundle provisioner class = class com.sun.enterprise.glassfish.bootstrap.osgi.BundleProvisioner.
Registered com.sun.enterprise.glassfish.bootstrap.osgi.EmbeddedOSGiGlassFishRuntime@7c581736 in service registry.
Reading in commandments from /opt/payara/config/pre-boot-commands.asadmin

Eryk Kulikowski (Oct 19 2023 at 12:17):

Eryk Kulikowski (Oct 19 2023 at 12:21):

Philip Durbin 🚀 (Oct 19 2023 at 12:29):

@Eryk Kulikowski what are you doing in an hour? Would you like to join the Containerization Working Group meeting? :grinning: https://ct.gdcc.io

Oliver Bertuch (Oct 19 2023 at 12:32):

Oliver Bertuch (Oct 19 2023 at 12:33):

Oliver Bertuch (Oct 19 2023 at 12:38):

@Eryk Kulikowski I'm glad to see you are basing your work on the WG outputs! Glad it's being reused. Looks like we made it customizable enough.

Oliver Bertuch (Oct 19 2023 at 12:41):

FWIW @Eryk Kulikowski, using Xmx, Xms etc in a container context is not considered good practice. Did you see the env vars for memory management in the tunables table? http://preview.guides.gdcc.io/en/develop/container/base-image.html#tunables You can easily asign the 24g of RAM defining the container limits in the compose definition, by default 70% of that will be allocated as heap.

Eryk Kulikowski (Oct 19 2023 at 12:50):

Don Sizemore (Oct 19 2023 at 14:56):

Philip Durbin 🚀 (Oct 19 2023 at 16:12):

@Don Sizemore hmm, would it be a permanent break on testing S3 proper? Or more that if we someday want to test both S3 and LocalStack that we'd need to do further refactoring on the Ansible side?

Philip Durbin 🚀 (Oct 19 2023 at 16:12):

@Deirdre Kirmis if your ears were burning a couple hours ago, we were trying to remember what flavor of S3 you use. Something on Dell, I think.

Deirdre Kirmis (Oct 19 2023 at 16:26):

we use aws s3 .. we tried for about a year to use Dell Isilon S3 for one of our stores but never got it working =)

Philip Durbin 🚀 (Oct 19 2023 at 16:27):

Oliver Bertuch (Oct 19 2023 at 16:28):

Deirdre Kirmis (Oct 19 2023 at 16:28):

i think there is actually a fix to dataverse now (it was a bug in apache on the isilon server) .. but we decided to go all in on aws

Deirdre Kirmis (Oct 19 2023 at 16:28):

i did try fronting it with minio .. i need to look back at my notes to see how that went .. i think i ran into some problems there too

Deirdre Kirmis (Oct 19 2023 at 16:29):

but i also don't exactly always know what i'm doing :grinning_face_with_smiling_eyes:

Oliver Bertuch (Oct 19 2023 at 16:30):

Deirdre Kirmis (Oct 19 2023 at 16:30):

Don Sizemore (Oct 19 2023 at 17:06):

@Philip Durbin Ansible can create and configure the buckets in LocalStack, but if for some/any reason Dataverse wanted to, it could? I don't think you want this to be the case, but just checking.

Philip Durbin 🚀 (Oct 19 2023 at 17:57):

@Don Sizemore I think I'm a little slow today but I think we're on the same page. All good. :grinning:

Philip Durbin 🚀 (Oct 19 2023 at 19:09):

After today's meeting I was all excited to try the magic trick with store names that don't have dashes in them.

Philip Durbin 🚀 (Oct 19 2023 at 19:09):

- dataverse_files_storage__driver__id=file1
- dataverse_files_file1_type=file
- dataverse_files_file1_label=Filesystem
- dataverse_files_file1_directory=${STORAGE_DIR}/store
- dataverse_files_s3c_type=s3
- dataverse_files_s3c_label=Real S3
- dataverse_files_s3c_custom__endpoint__url=s3.us-east-2.amazonaws.com
- dataverse_files_s3c_custom__endpoint__region=us-east-2
- dataverse_files_s3c_bucket__name=pdurbin
- dataverse_files_s3c_upload__redirect=true
- dataverse_files_s3c_access__key=REDACTED
- dataverse_files_s3c_secret__key=REDACTED

Philip Durbin 🚀 (Oct 19 2023 at 19:10):

{
    "status": "OK",
    "data": {
        "Filesystem": "file1"
    }
}

Philip Durbin 🚀 (Oct 19 2023 at 19:10):

Philip Durbin 🚀 (Oct 19 2023 at 19:38):

[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.43.4:run (default-cli) on project dataverse: Execution default-cli of goal io.fabric8:docker-maven-plugin:0.43.4:run failed: while scanning a simple key
[ERROR]  in 'reader', line 23, column 9:
[ERROR]             -Ddataverse.files.storage-driver ...
[ERROR]             ^
[ERROR] could not find expected ':'
[ERROR]  in 'reader', line 24, column 9:
[ERROR]             -Ddataverse.files.file1.type=file
[ERROR]             ^

Philip Durbin 🚀 (Oct 19 2023 at 19:39):

Philip Durbin 🚀 (Oct 19 2023 at 20:26):

Philip Durbin 🚀 (Oct 19 2023 at 20:33):

Philip Durbin 🚀 (Oct 19 2023 at 20:35):

Philip Durbin 🚀 (Oct 19 2023 at 21:00):

{
    "status": "OK",
    "data": {
        "RealS3": "s3c",
        "Local": "local",
        "Filesystem": "file1"
    }
}

Philip Durbin 🚀 (Oct 19 2023 at 21:00):

Philip Durbin 🚀 (Oct 19 2023 at 21:06):

Philip Durbin 🚀 (Oct 23 2023 at 20:04):

Just capturing some commands and output that @Don Sizemore helped me figure out to prove to myself that we can upload a test file from the Dataverse/Payara container to the Localstack container:

payara@dataverse:~$ echo -e '[default]\nregion = us-east-2' > ~/.aws/config
payara@dataverse:~$ cat ~/.aws/config
[default]
region = us-east-2
payara@dataverse:~$ echo -e '[default]\naws_access_key_id = default\naws_secret_access_key = default' > ~/.aws/credentials
payara@dataverse:~$ cat ~/.aws/credentials
[default]
aws_access_key_id = default
aws_secret_access_key = default
payara@dataverse:~$
payara@dataverse:~$ aws --endpoint-url=http://localstack:4566 s3 ls
2023-10-23 18:02:38 mybucket
payara@dataverse:~$
payara@dataverse:~$ echo foo > test.txt
payara@dataverse:~$ cat test.txt
foo
payara@dataverse:~$ aws --endpoint-url=http://localstack:4566 s3 cp test.txt s3://mybucket/
upload: ./test.txt to s3://mybucket/test.txt
payara@dataverse:~$ aws --endpoint-url=http://localstack:4566 s3 ls s3://mybucket/
2023-10-23 18:13:23          4 test.txt
payara@dataverse:~$

Philip Durbin 🚀 (Oct 23 2023 at 20:06):

But we still can't upload files from Dataverse itself to LocalStack. Here's the config I'm using:

-Ddataverse.files.localstack1.type=s3
-Ddataverse.files.localstack1.label=LocalStack
-Ddataverse.files.localstack1.custom-endpoint-url=http://localstack:4566
-Ddataverse.files.localstack1.custom-endpoint-region=us-east-2
-Ddataverse.files.localstack1.bucket-name=mybucket
-Ddataverse.files.localstack1.upload-redirect=false
-Ddataverse.files.localstack1.access-key=default
-Ddataverse.files.localstack1.secret-key=default

Philip Durbin 🚀 (Oct 23 2023 at 20:06):

Philip Durbin 🚀 (Oct 23 2023 at 21:20):

-Ddataverse.files.minio1.path-style-access=true

Philip Durbin 🚀 (Oct 23 2023 at 21:20):

Philip Durbin 🚀 (Oct 24 2023 at 18:28):

@Don Sizemore please remind me, do you have a preference for LocalStack or MinIO? I'm certainly happy to see if the trick above works with LocalStack.

Don Sizemore (Oct 24 2023 at 18:59):

I really don't. LocalStack sounds better for testing overall; MinIO sounds better for sites like ASU who use Dell MinIO. Also because I have to keep correcting myself when I type Minion.

Philip Durbin 🚀 (Oct 24 2023 at 19:03):

Philip Durbin 🚀 (Oct 24 2023 at 21:16):

Philip Durbin 🚀 (Oct 25 2023 at 14:06):

LocalStack is finickier about bucket creation. I've only had luck creating the bucket with awslocal s3 mb s3://mybucket (rather than Java).

Philip Durbin 🚀 (Oct 25 2023 at 15:18):

Philip Durbin 🚀 (Oct 25 2023 at 15:21):

Guido Schmutz (Oct 27 2023 at 08:41):

@Philip Durbin sorry a bit late, as you have figured it out with the S3 properties.

I have used the following way to specify the properties (just as environment variables on the container):

  dataverse:
    image: gdcc/dataverse:unstable
    container_name: dataverse
    hostname: dataverse
    user: payara
    labels:
      com.platys.name: dataverse
      com.platys.webui.title: Dataverse UI
      com.platys.webui.url: http://dataplatform:28294
      com.platys.restapi.title: Dataverse UI
      com.platys.restapi.url: http://dataplatform:28294/api/info/metrics/dataverses
    ports:
      - 28294:8080
      - 4848:4848
      - 9009:9009   # JDWP
      - 8686:8686   # JMX
    environment:
      - DATAVERSE_FQDN=${PUBLIC_IP}
      - _CT_DATAVERSE_SITEURL=http://${PUBLIC_IP}:28294
      - DATAVERSE_DB_HOST=dataverse-postgresql
      - DATAVERSE_DB_USER=dataverse
      - DATAVERSE_DB_PASSWORD=abc123!
      - DATAVERSE_MAIL_HOST=mailpit
      - DATAVERSE_MAIL_FROM=dataverse@localhost
      - _CT_DATAVERSE_SOLR_HOST=dataverse-solr
      - DATAVERSE_SOLR_PORT=8983
      - ENABLE_JMX=0
      - ENABLE_RELOAD=0
      - ENABLE_JDWP=1
      - dataverse_files_s3_type=s3
      - dataverse_files_s3_label="Object Storage"
      - dataverse_files_s3_bucket__name="dataverse-bucket"
      - dataverse_files_s3_custom__endpoint__url=http://${EADP_IP}:9000
      - dataverse_files_s3_custom__endpoint__region='us-east-1'
      - dataverse_files_s3_path__style__access=True
      - dataverse_files_s3_access__key=${PLATYS_AWS_ACCESS_KEY:?PLATYS_AWS_ACCESS_KEY must be set either in .env or as an environment variable}
      - dataverse_files_s3_secret__key=${PLATYS_AWS_SECRET_ACCESS_KEY:?PLATYS_AWS_SECRET_ACCESS_KEY must be set either in .env or as an environment variable}

Philip Durbin 🚀 (Oct 27 2023 at 11:32):

@Guido Schmutz right, I'm able to configure a single file store like that, with environment variables, but when I tried to configure multiple stores, it didn't work. If you get a chance to try this, please let me know! Otherwise, I'll wait for a fix for #9998.

Guido Schmutz (Oct 27 2023 at 12:57):

ok, I see, you configure multiple stores, challenge accepted ;-) ... took me a while to figure it out, but now it works

  dataverse:
    image: gdcc/dataverse:unstable
    container_name: dataverse
    hostname: dataverse
    user: payara
    labels:
      com.platys.name: dataverse
      com.platys.webui.title: Dataverse UI
      com.platys.webui.url: http://dataplatform:28294
      com.platys.restapi.title: Dataverse UI
      com.platys.restapi.url: http://dataplatform:28294/api/info/metrics/dataverses
    ports:
      - 28294:8080
      - 4848:4848
      - 9009:9009   # JDWP
      - 8686:8686   # JMX
    environment:
      - DATAVERSE_FQDN=${PUBLIC_IP}
      - _CT_DATAVERSE_SITEURL=http://${PUBLIC_IP}:28294
      - DATAVERSE_DB_HOST=dataverse-postgresql
      - DATAVERSE_DB_USER=dataverse
      - DATAVERSE_DB_PASSWORD=abc123!
      - DATAVERSE_MAIL_HOST=mailpit
      - DATAVERSE_MAIL_FROM=dataverse@localhost
      - _CT_DATAVERSE_SOLR_HOST=dataverse-solr
      - DATAVERSE_SOLR_PORT=8983
      - ENABLE_JMX=0
      - ENABLE_RELOAD=0
      - ENABLE_JDWP=1
      - dataverse_files_storage__driver__id=s3
      - dataverse_files_file1_type=file
      - dataverse_files_file1_label="Local Filesystem"
      - dataverse_files_file1_directory=${STORAGE_DIR}/store
      - dataverse_files_s3_type=s3
      - dataverse_files_s3_label="Object Storage"
      - dataverse_files_s3_bucket__name="dataverse-bucket"
      - dataverse_files_s3_custom__endpoint__url=http://${EADP_IP}:9000
      - dataverse_files_s3_custom__endpoint__region='us-east-1'
      - dataverse_files_s3_path__style__access=True
      - dataverse_files_s3_access__key=${PLATYS_AWS_ACCESS_KEY:?PLATYS_AWS_ACCESS_KEY must be set either in .env or as an environment variable}
      - dataverse_files_s3_secret__key=${PLATYS_AWS_SECRET_ACCESS_KEY:?PLATYS_AWS_SECRET_ACCESS_KEY must be set either in .env or as an environment variable}
    volumes:
      - ./data-transfer:/data-transfer
    restart: unless-stopped

$ curl -s -X GET -H 'X-Dataverse-key: REDACTED' 'http://192.168.116.137:28294/api/admin/dataverse/storageDrivers' | jq
{
  "status": "OK",
  "data": {
    "Object Storage": "s3",
    "Local Filesystem": "file1"
  }
}

Guess it was the space in the label (- dataverse_files_s3c_label=Real S3) in your case, which caused an error.

Philip Durbin 🚀 (Oct 27 2023 at 12:59):

Ah, right. I caught that later, the space problem, in the context of the JVM_ARGS. I'll try it again. Thanks!

Philip Durbin 🚀 (Nov 07 2023 at 12:53):

Philip Durbin 🚀 (Nov 07 2023 at 12:54):

So my thought is that once we merge #10044, we'll copy the config I added over to the "develop" and "manual" jobs. (And eventually the "alpha" job, once we make a release.)

Philip Durbin 🚀 (Nov 08 2023 at 15:27):

Philip Durbin 🚀 (Nov 08 2023 at 15:28):

Don Sizemore (Dec 05 2023 at 15:51):

I can tell Dataverse-Ansible not to template ~dataverse/.aws/credentials (and config) though it gets severely ticked when I rename it. Is Dataverse preferring global AWS creds to datastore-defined credentials expected, and do we want to open an issue to change this order?

Philip Durbin 🚀 (Dec 05 2023 at 15:51):

Don Sizemore (Dec 05 2023 at 15:52):

Philip Durbin 🚀 (Dec 05 2023 at 15:52):

Don Sizemore (Dec 05 2023 at 15:53):

I mean, I _can_ nuke that file just for testing runs, but that won't address any installation with multiple datastores who expect / need a global credential file as well.

Philip Durbin 🚀 (Dec 05 2023 at 15:53):

Don Sizemore (Dec 05 2023 at 15:55):

ah. I remember now. Jim's PR corrects credentials preference when RBAC is involved. Won't affect conf file vs. jvm-option precedence.

Philip Durbin 🚀 (Dec 05 2023 at 15:56):

Meanwhile, following the tweaks I just made to the tests (changing the access and secret keys to match Jenkins), the first manual run on the API test runner failed but I'm hoping it's transient. I kicked off a second run.

Philip Durbin 🚀 (Dec 05 2023 at 17:22):

Error: S3AccessIT.setUp:66 » SdkClient Unable to execute HTTP request: Connect to s3.localhost.localstack.cloud:4566 [s3.localhost.localstack.cloud/127.0.0.1] failed: Connection refused

Don Sizemore (Dec 05 2023 at 17:50):

I left my instance up from this morning if you want to test anything further. If not, I'll kill it to save y'all $$$

Don Sizemore (Dec 05 2023 at 18:38):

sounds like we want Jim's PR merged and then everything should start to work (at least WRT S3 creds). Ima kill this morning's EC2 instance.

Philip Durbin 🚀 (Dec 05 2023 at 18:54):

Don Sizemore (Dec 05 2023 at 20:00):

I'd say we merge Jim's PR, then merge that into your S3 testing branch, let me test manually once more, and we'll see if things run cleanly? (they should)

Juan Pablo Tosca Villanueva (Dec 05 2023 at 20:00):

Philip Durbin 🚀 (Dec 05 2023 at 20:10):

Sounds good, and meanwhile, I'll try to figure out while I'm getting that "s3.localhost.localstack.cloud:4566... Connection refused" error, which JP points out is the same he saw when not on VPN. Makes no sense to me. :sweat_smile:

Philip Durbin 🚀 (Dec 06 2023 at 15:52):

Philip Durbin 🚀 (Dec 06 2023 at 15:53):

Philip Durbin 🚀 (Dec 06 2023 at 19:15):

Philip Durbin 🚀 (Dec 06 2023 at 19:20):

Philip Durbin 🚀 (Dec 06 2023 at 19:21):

Philip Durbin 🚀 (Dec 06 2023 at 20:41):

I had confused myself with the api-test-runner. For now, until we merge, I set up a dedicated S3 job. Once we merge I'll put that extra config in the "develop" job. And once we release, into the "alpha" job.

Philip Durbin 🚀 (Dec 06 2023 at 20:42):

The config basically spins up LocalStack and MinIO and tells Dataverse to use them.

Philip Durbin 🚀 (Dec 06 2023 at 20:42):

Philip Durbin 🚀 (Dec 06 2023 at 21:03):

Philip Durbin 🚀 (Dec 06 2023 at 21:04):

Juan Pablo Tosca Villanueva (Dec 06 2023 at 21:08):

Juan Pablo Tosca Villanueva (Dec 06 2023 at 21:09):

Speaking about Doker, I broke their website this morning and I had to share their error page

Juan Pablo Tosca Villanueva (Dec 06 2023 at 21:09):

Philip Durbin 🚀 (Dec 06 2023 at 21:11):

Philip Durbin 🚀 (Dec 07 2023 at 14:16):

Philip Durbin 🚀 (Dec 07 2023 at 14:17):

Could be better but I'm happy it's higher than what we had with only unit tests, 7%.

Stream: containers

Topic: testing S3

Philip Durbin 🚀 (Apr 26 2023 at 14:16):

Oliver Bertuch (Apr 26 2023 at 14:28):

Philip Durbin 🚀 (Apr 26 2023 at 14:30):

Oliver Bertuch (Apr 26 2023 at 14:32):

Philip Durbin 🚀 (Apr 26 2023 at 14:33):

Oliver Bertuch (Apr 26 2023 at 14:34):

Philip Durbin 🚀 (Apr 26 2023 at 14:35):

Oliver Bertuch (Apr 26 2023 at 14:36):

Philip Durbin 🚀 (Apr 26 2023 at 14:36):

Philip Durbin 🚀 (Apr 26 2023 at 14:37):

Oliver Bertuch (Apr 26 2023 at 14:38):

Philip Durbin 🚀 (Apr 26 2023 at 14:38):

Oliver Bertuch (Apr 26 2023 at 14:38):

Oliver Bertuch (Apr 26 2023 at 14:38):

Philip Durbin 🚀 (Apr 26 2023 at 14:39):

Oliver Bertuch (Apr 26 2023 at 14:39):

Oliver Bertuch (Apr 26 2023 at 14:40):

Philip Durbin 🚀 (Apr 26 2023 at 14:40):

Oliver Bertuch (Apr 26 2023 at 14:40):

Philip Durbin 🚀 (Apr 26 2023 at 14:40):

Oliver Bertuch (Apr 26 2023 at 14:41):

Oliver Bertuch (Apr 26 2023 at 14:42):

Philip Durbin 🚀 (Apr 26 2023 at 14:43):

Oliver Bertuch (Apr 26 2023 at 14:44):

Oliver Bertuch (Apr 26 2023 at 14:45):

Oliver Bertuch (Apr 26 2023 at 14:45):

Oliver Bertuch (Apr 26 2023 at 14:45):

Oliver Bertuch (Apr 26 2023 at 14:46):

Philip Durbin 🚀 (Apr 26 2023 at 14:46):

Oliver Bertuch (Apr 26 2023 at 14:46):

Oliver Bertuch (Apr 26 2023 at 14:47):

Oliver Bertuch (Apr 26 2023 at 14:47):

Oliver Bertuch (Apr 26 2023 at 14:47):

Philip Durbin 🚀 (May 01 2023 at 19:07):

Oliver Bertuch (May 02 2023 at 07:16):

Oliver Bertuch (May 02 2023 at 07:18):

Philip Durbin 🚀 (May 02 2023 at 11:03):

Philip Durbin 🚀 (May 02 2023 at 11:03):

Philip Durbin 🚀 (May 02 2023 at 11:03):

Oliver Bertuch (May 02 2023 at 11:29):

Philip Durbin 🚀 (May 17 2023 at 12:14):

Oliver Bertuch (May 17 2023 at 12:15):

Philip Durbin 🚀 (May 17 2023 at 12:19):

Philip Durbin 🚀 (May 19 2023 at 15:23):

Philip Durbin 🚀 (May 19 2023 at 15:24):

Philip Durbin 🚀 (May 19 2023 at 15:25):

Oliver Bertuch (May 19 2023 at 19:09):

Oliver Bertuch (May 19 2023 at 19:09):

Philip Durbin 🚀 (May 19 2023 at 19:34):

Philip Durbin 🚀 (May 19 2023 at 19:34):

Oliver Bertuch (May 19 2023 at 19:35):

Philip Durbin 🚀 (May 19 2023 at 19:39):

Philip Durbin 🚀 (Aug 11 2023 at 15:18):

Philip Durbin 🚀 (Sep 07 2023 at 11:02):

Philip Durbin 🚀 (Sep 19 2023 at 19:48):

Oliver Bertuch (Sep 19 2023 at 19:52):

Oliver Bertuch (Sep 19 2023 at 19:53):

Philip Durbin 🚀 (Sep 19 2023 at 19:56):

Philip Durbin 🚀 (Sep 19 2023 at 19:59):

Philip Durbin 🚀 (Oct 04 2023 at 18:26):

Philip Durbin 🚀 (Oct 04 2023 at 18:27):

Oliver Bertuch (Oct 04 2023 at 18:27):

Oliver Bertuch (Oct 04 2023 at 18:28):

Philip Durbin 🚀 (Oct 04 2023 at 18:30):

Oliver Bertuch (Oct 04 2023 at 18:30):

Oliver Bertuch (Oct 04 2023 at 18:31):

Philip Durbin 🚀 (Oct 04 2023 at 18:31):

Philip Durbin 🚀 (Oct 04 2023 at 18:31):

Oliver Bertuch (Oct 04 2023 at 18:32):

Oliver Bertuch (Oct 04 2023 at 18:32):

Oliver Bertuch (Oct 04 2023 at 18:33):

Oliver Bertuch (Oct 04 2023 at 18:33):

Oliver Bertuch (Oct 04 2023 at 18:34):

Philip Durbin 🚀 (Oct 04 2023 at 18:34):

Philip Durbin 🚀 (Oct 04 2023 at 18:34):

Oliver Bertuch (Oct 04 2023 at 18:37):

Philip Durbin 🚀 (Oct 04 2023 at 18:37):

Oliver Bertuch (Oct 04 2023 at 18:37):