OpenShift · containers · Zulip Chat Archive

Currently customizing the base image to be more OpenShift security policy friendly

Philip Durbin 🚀 (Feb 08 2024 at 14:26):

Oliver Bertuch (Feb 09 2024 at 17:35):

Philip Durbin 🚀 (Feb 09 2024 at 17:37):

Oliver Bertuch (Feb 09 2024 at 18:01):

Oliver Bertuch (Feb 09 2024 at 18:02):

At least I can start Dataverse and get to a successful deploy and bootstrap :grinning_face_with_smiling_eyes:

Philip Durbin 🚀 (Feb 09 2024 at 18:06):

Ah, you even link to CRC. Great. Doing a million things but I'll try to get it installed after lunch. This looks great! Thanks, @Oliver Bertuch !

Oliver Bertuch (Feb 09 2024 at 18:09):

Oliver Bertuch (Feb 09 2024 at 18:10):

Philip Durbin 🚀 (Feb 09 2024 at 19:19):

Oliver Bertuch (Feb 09 2024 at 19:19):

Oliver Bertuch (Feb 09 2024 at 19:22):

Philip Durbin 🚀 (Feb 10 2024 at 16:19):

pdurbin@air dataverse % mvn -Pct k8s:resource k8s:deploy
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/maven-metadata.xml (21 kB at 108 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-metadata.xml (14 kB at 75 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.879 s
[INFO] Finished at: 2024-02-10T11:18:08-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] No plugin found for prefix 'k8s' in the current project and in the plugin groups [org.apache.maven.plugins, org.codehaus.mojo] available from the repositories [local (/Users/pdurbin/.m2/repository), central (https://repo.maven.apache.org/maven2)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/NoPluginFoundForPrefixException

Oliver Bertuch (Feb 10 2024 at 16:21):

Philip Durbin 🚀 (Feb 10 2024 at 23:26):

pdurbin@air container-k8s % eval $(crc oc-env)
pdurbin@air container-k8s % oc get pods -w
NAME                        READY   STATUS    RESTARTS   AGE
dataverse-894d594c7-xdr2x   0/2     Pending   0          45s
postgres-6c788b5588-2gvn4   1/1     Running   0          45s

Philip Durbin 🚀 (Feb 10 2024 at 23:30):

I had to do just crc start though because crc start -c 8 -d 64 -m 16384 (and lower amounts of memory and cpu) keeps failing on my laptop at home. :disappointed:

Philip Durbin 🚀 (Feb 10 2024 at 23:33):

Philip Durbin 🚀 (Feb 10 2024 at 23:52):

I did crc stop and crc delete and tried again with more memory crc start -m 12288 (12 GB instead of 9). I have 16 GB on this laptop.

Philip Durbin 🚀 (Feb 10 2024 at 23:52):

pdurbin@air container-k8s % oc get pods -w
NAME                        READY   STATUS              RESTARTS   AGE
dataverse-894d594c7-xm8tw   0/2     ContainerCreating   0          6s
postgres-6c788b5588-8vh2s   0/1     ContainerCreating   0          6s
postgres-6c788b5588-8vh2s   0/1     Running             0          14s
postgres-6c788b5588-8vh2s   1/1     Running             0          21s
dataverse-894d594c7-xm8tw   1/2     Running             0          39s
dataverse-894d594c7-xm8tw   0/2     Error               0          3m39s
dataverse-894d594c7-xm8tw   1/2     Running             1 (2s ago)   3m40s

Philip Durbin 🚀 (Feb 10 2024 at 23:53):

Caused by: edu.harvard.iq.dataverse.settings.ConfigCheckService$ConfigurationError: Not all configuration checks passed successfully. See logs above.

Philip Durbin 🚀 (Feb 10 2024 at 23:56):

Directory /dv/docroot for docroot space (see dataverse.files.docroot) exists, but is not writeable|#]

Philip Durbin 🚀 (Feb 12 2024 at 18:16):

At work, where I have double the RAM (32 GB), crc start -c 8 -d 64 -m 16384 from the PR works, but I get the same "directory not writeable" error:

Philip Durbin 🚀 (Feb 15 2024 at 15:52):

Philip Durbin 🚀 (Feb 15 2024 at 15:53):

One thing that was bothering me was that while we did see the ERROR StatusLogger message on my machine too, we determined it was non-fatal, but then we didn't go on to see what the next fatal error is.

Philip Durbin 🚀 (Feb 15 2024 at 15:54):

Philip Durbin 🚀 (Feb 15 2024 at 15:56):

One thing I'm confused about is how to access Dataverse from the host, from my Mac. Under "details" I can see a Pod IP of 10.217.0.63 and a Host IP of 192.168.126.11 but they both timeout when I try to reach them.

Philip Durbin 🚀 (Feb 15 2024 at 15:59):

Oh, I should point out that this is with the suggestion from Danni that I showed in the meeting, a change to modules/container-k8s/src/main/jkube/dataverse-deployment.yaml:

   replicas: 1
   template:
     spec:
+      securityContext:
+        runAsUser: 1000
+        fsGroup: 1000
       containers:
         - name: dataverse
           image: ghcr.io/gdcc/dataverse:openshift-poc

Philip Durbin 🚀 (Feb 15 2024 at 19:31):

$ kubectl exec --stdin --tty dataverse-5f6bb9f895-lrfvb -- bash
Defaulted container "dataverse" out of: dataverse, bootstrap
payara@dataverse-5f6bb9f895-lrfvb:~$ curl -s localhost:8080/api/dataverses/:root | jq .
{
  "status": "OK",
  "data": {
    "id": 1,
    "alias": "root",
    "name": "Root",
    "dataverseContacts": [
      {
        "displayOrder": 0,
        "contactEmail": "root@mailinator.com"
      }
    ],
    "permissionRoot": true,
    "description": "The root dataverse.",
    "dataverseType": "UNCATEGORIZED",
    "creationDate": "2024-02-15T14:29:34Z"
  }
}

Philip Durbin 🚀 (Feb 15 2024 at 19:33):

I'm exec'ing into the container because I'm not sure how to access these API endpoints from the host, my Mac. :thinking:

Philip Durbin 🚀 (Feb 15 2024 at 19:35):

Philip Durbin 🚀 (Feb 15 2024 at 19:46):

$ oc port-forward pods/dataverse-5f6bb9f895-lrfvb 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Philip Durbin 🚀 (Feb 15 2024 at 19:46):

Philip Durbin 🚀 (Feb 15 2024 at 19:47):

Like Oliver said during the meeting, there's Solr work to do but this is great progress!

Philip Durbin 🚀 (Feb 15 2024 at 21:11):

Philip Durbin 🚀 (Feb 16 2024 at 12:14):

@Oliver Bertuch should we go ahead and add the commit above about securityContext/runAsUser to #10314? Maybe with a TODO to remove it some day?

Philip Durbin 🚀 (Feb 23 2024 at 20:57):

Ok, I just merged #10314 BUT before I did, I narrowed the scope, as discussed with Oliver earlier today. We need the various fixes to get our images running under OpenShift. However, I removed the other stuff we don't need yet, the config files, etc.

Philip Durbin 🚀 (Feb 23 2024 at 20:58):

That's why I changed the title of the PR to "OpenShift prep work". I think this better reflects what I merged.

Philip Durbin 🚀 (Feb 23 2024 at 20:59):

Philip Durbin 🚀 (Feb 23 2024 at 21:11):

Gursahib (Feb 28 2025 at 13:44):

I successfully followed the demo at Dataverse Guide and was able to run it on my local WSL Docker for testing.

Now, I am attempting to replicate this setup on OpenShift within my organization. However, I am encountering issues with Solr not functioning as expected. I am currently using the GDCC image of Dataverse along with Solr 9, but I am uncertain if this configuration is compatible.

Additionally, I have come across the IQSS images for Dataverse and Solr. Would these be a better alternative?

Are there any specific guides or images available for deploying Dataverse on OpenShift? I would appreciate any guidance you can provide.

Philip Durbin 🚀 (Feb 28 2025 at 14:13):

Hi! The IQSS images on Docker Hub are old. You shouldn't use them. I'm glad the GDCC images are working for you, the demo version at least.

Philip Durbin 🚀 (Feb 28 2025 at 14:14):

We have a topic going at #containers > OpenShift but it's been a while since anyone has posted anything.

Philip Durbin 🚀 (Feb 28 2025 at 14:15):

Philip Durbin 🚀 (Feb 28 2025 at 14:16):

Gursahib (Feb 28 2025 at 14:32):

I am encountering an issue where the dvinitializer job requires root privileges, causing it to fail. However, even without it, I was able to deploy Dataverse to the UI level and log in as an admin.

In this deployment, I can create datasets and upload files, but Solr is unable to perform searches or create a Dataverse. I suspect that certain Solr actions require root privileges.

chown: /var/solr/data: Operation not permitted
chown: /var/solr/data: Operation not permitted
chown: /var/solr/logs/solr_gc.log: Operation not permitted
chown: /var/solr/logs/solr.log: Operation not permitted
chown: /var/solr/logs/solr_slow_requests.log: Operation not permitted
chown: /var/solr/logs/2025_02_25.request.log: Operation not permitted
chown: /var/solr/logs/2025_02_26.request.log: Operation not permitted
chown: /var/solr/logs/2025_02_27.request.log: Operation not permitted
chown: /var/solr/logs/2025_02_28.request.log: Operation not permitted
chown: /var/solr/logs: Operation not permitted
chown: /var/solr/logs: Operation not permitted
chown: /var/solr/log4j2.xml: Operation not permitted
chown: /var/solr: Operation not permitted
chown: /var/solr: Operation not permitted

Would you happen to have any Solr images that do not require root privileges? Any guidance on resolving this issue would be greatly appreciated.

Philip Durbin 🚀 (Feb 28 2025 at 14:43):

Huh. I'm surprised Solr requires root. @Oliver Bertuch you've been messing around with Kubernetes lately. Any insight on this?

Philip Durbin 🚀 (Feb 28 2025 at 14:48):

Philip Durbin 🚀 (Feb 28 2025 at 14:51):

Oh, wait. I think @zbenta is (or was?) using OpenStack, not OpenShift, based on these slides: https://osf.io/fjw9t

Philip Durbin 🚀 (Feb 28 2025 at 14:53):

Oliver Bertuch (Feb 28 2025 at 14:55):

Philip Durbin 🚀 (Feb 28 2025 at 14:55):

Gursahib (Feb 28 2025 at 14:56):

Alright, Thank you Philip.
it seems the dv_initializer running script fix-fs-perm.sh in image gdcc/configbacker:alpha needs root.

8a7400b16cbd:/scripts# cat fix-fs-perms.sh
#!/bin/bash

# [INFO]: Fix folder permissions using 'chown' to be writeable by containers not running as root.

set -euo pipefail

if [[ "$(id -un)" != "root" ]]; then
  echo "This script must be run as user root (not $(id -un)), otherwise no fix is possible."
fi

DEF_DV_PATH="/dv"
DEF_SOLR_PATH="/var/solr"
DEF_DV_UID="1000"
DEF_SOLR_UID="8983"

function usage() {
  echo "Usage: $(basename "$0") (dv|solr|[1-9][0-9]{3,4}) [PATH [PATH [...]]]"
  echo ""
  echo "You may omit a path when using 'dv' or 'solr' as first argument:"
  echo "  - 'dv' will default to user $DEF_DV_UID and $DEF_DV_PATH"
  echo "  - 'solr' will default to user $DEF_SOLR_UID and $DEF_SOLR_PATH"
  exit 1
}

# Get a target name or id
TARGET=${1:-help}
# Get the rest of the arguments as paths to apply the fix to
PATHS=( "${@:2}" )

ID=0
case "$TARGET" in
  dv)
    ID="$DEF_DV_UID"
    # If there is no path, add the default for our app image
    if [[ ${#PATHS[@]} -eq 0 ]]; then
      PATHS=( "$DEF_DV_PATH" )
    fi
    ;;
  solr)
    ID="$DEF_SOLR_UID"
    # In case there is no path, add the default path for Solr images
    if [[ ${#PATHS[@]} -eq 0 ]]; then
      PATHS=( "$DEF_SOLR_PATH" )
    fi
    ;;
  # If there is a digit in the argument, check if this is a valid UID (>= 1000, ...)
  *[[:digit:]]* )
    echo "$TARGET" | grep -q "^[1-9][0-9]\{3,4\}$" || usage
    ID="$TARGET"
    ;;
  *)
    usage
    ;;
esac

# Check that we actually have at least 1 path
if [[ ${#PATHS[@]} -eq 0 ]]; then
  usage
fi

# Do what we came for
chown -R "$ID:$ID" "${PATHS[@]}"

Gursahib (Feb 28 2025 at 14:58):

Oliver Bertuch (Feb 28 2025 at 14:58):

I suppose you will be running into problems because of copying the Docker Compose stuff 1:1 onto K8s. Depending on your OpenShift environment, you probably are enforced to use a Pod Security Admission of "restricted" or "baseline". (K8s docs, OS docs)

Oliver Bertuch (Feb 28 2025 at 14:59):

You should not use these scripts in init containers on K8s, they are meant for Docker Compose usage.

Philip Durbin 🚀 (Feb 28 2025 at 15:00):

@Gursahib that fix perms script was written by @Oliver Bertuch so please listen to any advice he has! :big_smile:

Oliver Bertuch (Feb 28 2025 at 15:00):

Oliver Bertuch (Feb 28 2025 at 15:01):

And are you (as you should if someone is already impossing baseline or restricted on you) setting all the necessary security bits for your Deployment's .spec.template.spec.securityContext and .spec.template.containers[].securityContext ?

Philip Durbin 🚀 (Feb 28 2025 at 15:01):

Another thought is that we had a call with a Red Hat engineer almost a year ago. He's really into Solr so I assume he has it working well on OpenShift. You can see the notes from the call, watch the recording, and check out his https://github.com/smartabyar-smartvillage/smartvillage-operator project, which uses Solr. Maybe we can even pull him in and ask questions.

Gursahib (Feb 28 2025 at 15:04):

First of all i would like to clear, I am just a Technical designer exploring this for next solution :sweat_smile:
and this openshift stuff is new to me.

Oliver Bertuch (Feb 28 2025 at 15:05):

Oliver Bertuch (Feb 28 2025 at 15:06):

Well for exploration and evaluation I strongly suggest going for Docker Compose first. In the foreseeable future I will be working on better K8s support via Helmcharts and/or Operators for Dataverse, but there are other problems to address first.

Oliver Bertuch (Feb 28 2025 at 15:09):

Gursahib (Feb 28 2025 at 15:11):

Well I have already explored dataverse on Docker and of course it works flawlessly there. Now i am tasked to replicate the same in our openshift, its a big learning experience.
I believe I am using neither as I am defining my service myself.
here is the way I create yaml:

apiVersion: apps/v1
kind: StatefulSet
# kind: Deployment
metadata:
  name: solr
  labels:
    app: solr
spec:
  serviceName: solr
  replicas: 1
  selector:
    matchLabels:
      app: solr
  template:
    metadata:
      labels:
        app: solr
    spec:
      securityContext:
        fsGroup: 1000810000
      containers:
      - name: solr
        image: solr:9.4.1
        ports:
        - containerPort: 8983
        command: ["solr-precreate", "collection1", "/template"]
        volumeMounts:
          - name: solr-data
            mountPath: /var/solr
          - name: solr-template
            mountPath: /template
        securityContext:
          runAsUser: 1000810000
          runAsGroup: 1000810000
          allowPrivilegeEscalation: false
      volumes:
        - name: solr-data
          persistentVolumeClaim:
            claimName: solr-data-pvc
        - name: solr-template
          persistentVolumeClaim:
            claimName: solr-template-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: solr
  labels:
    app: solr
spec:
  type: ClusterIP
  ports:
    - name: solr
      port: 8983
      targetPort: 8983
  selector:
    app: solr

Oliver Bertuch (Feb 28 2025 at 15:20):

Have you tried to verify in a running container if the fsGroups are properly applied to the volumes and the user is in that group?

Gursahib (Feb 28 2025 at 15:22):

Oliver Bertuch (Feb 28 2025 at 15:24):

Oliver Bertuch (Feb 28 2025 at 15:25):

(I don't see the configbaker image anymore here as init container, as it should be :-) )

Oliver Bertuch (Feb 28 2025 at 15:26):

BTW - how do you propagate the /template PV? Solr will be having a hard time if there is no template in that folder...

Gursahib (Feb 28 2025 at 15:34):

command: ["sh", "-c", "fix-fs-perms.sh solr && cp -a /template/* /solr-template"]

apiVersion: batch/v1
kind: Job
metadata:
  name: solr-initializer
spec:
  template:
    metadata:
      name: solr-initializer
    spec:
      securityContext:
        fsGroup: 1000810000
      containers:
      - name: solr-initializer
        image: gdcc/configbaker:alpha
        # command: ["sh", "-c", "mkdir -p /var/solr/data && chown -R 8983:8983 /var/solr && fix-fs-perms.sh solr && cp -a /template/* /solr-template"]
        command: ["sh", "-c", "fix-fs-perms.sh solr && cp -a /template/* /solr-template"]
        volumeMounts:
          - name: solr-data
            mountPath: /var/solr
          - name: solr-template
            mountPath: /solr-template
        securityContext:
          runAsUser: 1000810000
          runAsGroup: 1000810000
          allowPrivilegeEscalation: false
      restartPolicy: Never
      volumes:
        - name: solr-data
          persistentVolumeClaim:
            claimName: solr-data-pvc
        - name: solr-template
          persistentVolumeClaim:
            claimName: solr-template-pvc

Oliver Bertuch (Feb 28 2025 at 15:34):

Gursahib (Feb 28 2025 at 15:35):

Oliver Bertuch (Feb 28 2025 at 15:36):

B) As you are on K8s, you don't need to fix the permissions. This is taken care of by K8s for you, using the fsGroup trick that will ensure the volume is ready to go for you. So you won't need the first part of the shell command.

Oliver Bertuch (Feb 28 2025 at 15:37):

Oliver Bertuch (Feb 28 2025 at 15:40):

D) I also suggest moving the runAsUser and runAsGroup bits to .spec.template.spec.securityContext, so you only need to set it once for all containers in the pod.

Gursahib (Feb 28 2025 at 15:41):

Thank you @Oliver Bertuch for your insights. As I mentioned earlier, this is a new area (we are shifting to DevOps) for both myself and most of our team, so we are continuously learning these concepts over time.
I appreciate your suggestions and will incorporate them accordingly.

Oliver Bertuch (Feb 28 2025 at 15:41):

Also, it is a common (but not good/best) practice to set fsGroup to sth. else than runAsGroup. But that's just a minor thing, you can probably ignore it.

Philip Durbin 🚀 (Feb 28 2025 at 15:48):

@Gursahib once you have this working, would you want to add a page about OpenShift to the Container Guide? I'm happy to help.

Oliver Bertuch (Feb 28 2025 at 15:48):

@Gursahib I also VERY STRONGLY suggest to set limits for CPU, memory, and storage, drop the capabilities, set liveness and readiness probes, try to use read-only root fs, set a pull image policy.

Also, you seem to be running in default namespace, which is a bad idea for production but might be OK for testing.

Also, the most common labels are missing, you are using very old standards here...

Gursahib (Feb 28 2025 at 15:50):

Sure @Philip Durbin ☃️ , Once I have this running would share what I learned and can help in documenting it as well.

Gursahib (Feb 28 2025 at 15:52):

The environment i am working in is kind of a playground for testing so nothing to worry about here

Gursahib (Mar 24 2025 at 13:46):

Hey guys I am back and good news, after following Oliver's guidance the Dataverse is running on openshift environment. Last thing left is the previewers not working properly. I have tried everything but can't seem to find any solutions. The preview register is working fine. I am not sure what I am missing, if this works then I think we can have a complete documentation of deploying in Openshift.

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/init.sh
Using Provider Url=http://previewers-provider:9000 for versions v1.4,betatest
Downloading local copies of remote JavaScript libraries:
./localinstall.sh: line 22: replace_js.sh: Permission denied
./localinstall.sh: line 23: urls_js.txt: Permission denied
./localinstall.sh: line 24: replace_js.sh: No such file or directory
cat: urls_js.txt: No such file or directory
Downloading local copies of remote CSS files:
./localinstall.sh: line 28: replace_css.sh: Permission denied
./localinstall.sh: line 29: urls_css.txt: Permission denied
./localinstall.sh: line 30: replace_css.sh: No such file or directory
cat: urls_css.txt: No such file or directory
./localinstall.sh: line 39: ../urls_js.txt: No such file or directory
./localinstall.sh: line 49: ../urls_css.txt: No such file or directory
Changing example curl commands to use local URLs
sed: couldn't open temporary file ./sedl9f3ws: Permission denied
Done changing example curl commands to use local URLs
Cleaning Up...
rm: cannot remove 'urls_js.txt': No such file or directory
rm: cannot remove 'urls_css.txt': No such file or directory
rm: cannot remove 'replace_js.sh': No such file or directory
rm: cannot remove 'replace_css.sh': No such file or directory
Done
cp: cannot create regular file 'previewers/v1.4/5.2curlcommands.md': Permission denied
cp: cannot create regular file 'previewers/v1.4/6.1curlcommands.md': Permission denied
cp: cannot create regular file 'previewers/v1.4/CONTRIBUTING.md': Permission denied
cp: cannot create regular file 'previewers/v1.4/README.md': Permission denied
cp: cannot create regular file 'previewers/v1.4/pre5.2curlcommands.md': Permission denied
Downloading local copies of remote JavaScript libraries:
./localinstall.sh: line 22: replace_js.sh: Permission denied
./localinstall.sh: line 23: urls_js.txt: Permission denied
./localinstall.sh: line 24: replace_js.sh: No such file or directory
cat: urls_js.txt: No such file or directory
Downloading local copies of remote CSS files:
./localinstall.sh: line 28: replace_css.sh: Permission denied
./localinstall.sh: line 29: urls_css.txt: Permission denied
./localinstall.sh: line 30: replace_css.sh: No such file or directory
cat: urls_css.txt: No such file or directory
./localinstall.sh: line 39: ../urls_js.txt: No such file or directory
./localinstall.sh: line 49: ../urls_css.txt: No such file or directory
Changing example curl commands to use local URLs
sed: couldn't open temporary file ./sedPdDuuk: Permission denied
Done changing example curl commands to use local URLs
Cleaning Up...
rm: cannot remove 'urls_js.txt': No such file or directory
rm: cannot remove 'urls_css.txt': No such file or directory
rm: cannot remove 'replace_js.sh': No such file or directory
Done
rm: cannot remove 'replace_css.sh': No such file or directory
cp: cannot create regular file 'previewers/betatest/5.2curlcommands.md': Permission denied
cp: cannot create regular file 'previewers/betatest/6.1curlcommands.md': Permission denied
cp: cannot create regular file 'previewers/betatest/CONTRIBUTING.md': Permission denied
cp: cannot create regular file 'previewers/betatest/README.md': Permission denied
cp: cannot create regular file 'previewers/betatest/pre5.2curlcommands.md': Permission denied
/docker-entrypoint.sh: Configuration complete; ready for start up
2025/03/24 13:38:43 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2
2025/03/24 13:38:43 [notice] 1#1: using the "epoll" event method
2025/03/24 13:38:43 [notice] 1#1: nginx/1.27.2
2025/03/24 13:38:43 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2025/03/24 13:38:43 [notice] 1#1: OS: Linux 4.18.0-372.131.1.el8_6.x86_64
2025/03/24 13:38:43 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2025/03/24 13:38:43 [notice] 1#1: start worker processes

# kustomize/base/register-previewers.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: previewers-provider
  labels:
    app: previewers-provider
spec:
  replicas: 1
  selector:
    matchLabels:
      app: previewers-provider
  template:
    metadata:
      labels:
        app: previewers-provider
    spec:
      securityContext:
        fsGroup: 1000810000
      containers:
      - name: previewers-provider
        image: dockerhub/trivadis/dataverse-previewers-provider:latest
        securityContext:
          runAsUser: 1000810000
          runAsGroup: 1000810000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: false  # Allow writing to mounted dirs
        ports:
          - containerPort: 9000
        env:
          - name: NGINX_HTTP_PORT
            value: "9000"
          - name: PREVIEWERS_PROVIDER_URL
            value: "http://previewers-provider:9000"
          - name: VERSIONS
            value: "v1.4,betatest"
        volumeMounts:
          - name: nginx-config
            mountPath: /etc/nginx/conf.d/default.conf
            subPath: default.conf
          - name: previewers-storage
            mountPath: /usr/share/nginx/html/previewers
          - name: cache-volume
            mountPath: /var/cache/nginx
          - name: run-volume
            mountPath: /var/run
      volumes:
        - name: nginx-config
          configMap:
            name: nginx-config  # Using subPath for correct mount
        - name: previewers-storage
          emptyDir: {}  # Allowing writing
        - name: cache-volume
          emptyDir: {}
        - name: run-volume
          emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: previewers-provider
  labels:
    app: previewers-provider
spec:
  type: ClusterIP
  ports:
    - port: 9000
      targetPort: 9000
  selector:
    app: previewers-provider

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  default.conf: |
    server {
      listen 9000;
      server_name localhost;

      # NGINX configuration to handle the requests
      location /previewers/ {
        root /usr/share/nginx/html;
        try_files $uri $uri/ =404;
      }
    }

Philip Durbin 🚀 (Mar 24 2025 at 13:56):

@Gursahib do you want to add something about the Container Guide about this? :big_smile:

Philip Durbin 🚀 (Mar 24 2025 at 13:57):

As for the previewers, I'm not sure how to test this. Maybe using OpenShift Local? :thinking:

Philip Durbin 🚀 (Mar 24 2025 at 13:59):

Gursahib (Mar 24 2025 at 14:02):

Sure @Philip Durbin ☀️ I can contribute to the whole section on Openshift, but right now I want to deliver this to researchers I am working with. And I want it to be working 100%. I promise you I'll add a full featured document of OpenShift installation.

Philip Durbin 🚀 (Mar 24 2025 at 14:03):

Gursahib (Mar 24 2025 at 14:03):

I have heard of both systems but I am pretty sure these local deployments will work without a doubt, as there will be less security constraints. Still I would give it a try if nothing works.

Philip Durbin 🚀 (Mar 24 2025 at 14:04):

@Gursahib as a starting point, could you please add an issue to https://github.com/IQSS/dataverse/issues about the trouble you're having with previewers in OpenShift?

Oliver Bertuch (Mar 24 2025 at 14:13):

@Gursahib these Trivadis Previewer Containers are not supported by the community. I would neither call them production ready nor well documented. Also, I wouldn't endorse using them at this point.

That said, judging from looking at the localinstall.sh script inside the container (the sources are nowhere to be found), the scripts wants to write to /app, which isn't backed by a volume in your Deployment.

Gursahib (Mar 24 2025 at 14:17):

Gursahib (Mar 24 2025 at 14:18):

Philip Durbin 🚀 (Mar 24 2025 at 14:30):

Right, we did recently add those Trivadis images to the "demo or eval" use case in #11181.

Philip Durbin 🚀 (Mar 24 2025 at 14:31):

We've talked about building our own images but for now the Trivadis images are what we have.

Philip Durbin 🚀 (Mar 24 2025 at 14:33):

Oliver Bertuch (Mar 24 2025 at 14:49):

Thanks @Philip Durbin ☀️ for the link! I searched under the Trivadis repos (there is a preview register container) and tried searching, but did come up empty handed.

Oliver Bertuch (Mar 24 2025 at 14:50):

Indeed, the Trivadis Preview containers are currently the only thing that's out there @Gursahib, so it's probably fine that they are not production ready for your use case.

Gursahib (Mar 24 2025 at 14:57):

Its Alright, I will see what I can do. If it doesnt work then that's how it is :upside_down:.
Thank you for the help @Oliver Bertuch and @Philip Durbin ☀️
I'll see when i can contribute for OpenShift deployment. Probably from next week.

Philip Durbin 🚀 (Mar 24 2025 at 14:58):

Gursahib (Mar 24 2025 at 14:59):

btw I added a volume for /app as well, and as you mentioned indeed seems to be empty:

docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/init.sh
Using Provider Url=http://previewers-provider:9000 for versions v1.4,betatest
/docker-entrypoint.d/init.sh: line 15: ./localinstall.sh: No such file or directory
cp: cannot stat '*.md': No such file or directory
/docker-entrypoint.d/init.sh: line 19: ./localinstall.sh: No such file or directory
cp: cannot stat '*.md': No such file or directory
cp: cannot stat './previewers': No such file or directory

Oliver Bertuch (Mar 24 2025 at 15:00):

Oliver Bertuch (Mar 24 2025 at 15:01):

Let me take a look if we can trick the script into cooperating or if it needs to be extended

Oliver Bertuch (Mar 24 2025 at 15:04):

Urgh this is hard to do... The entrypoint changes the folder to the one where the previewers are located within the container.

Oliver Bertuch (Mar 24 2025 at 15:04):

Oliver Bertuch (Mar 24 2025 at 15:05):

That's not gonna be easy to work around. You'd have to make a fork and create your own image based on what's there...

Oliver Bertuch (Mar 24 2025 at 15:06):

At some point I will look into this, but don't have the bandwidth at the moment.

Gursahib (Mar 24 2025 at 15:08):

Gursahib (Mar 24 2025 at 15:09):

Don't worry its not breaking the Dataverse, its just one feature which need some polishing.
Anyways thank you for your help @Oliver Bertuch

Oliver Bertuch (Mar 24 2025 at 15:09):

Yeah well there's so many things that still require tweaking for a smooth container / K8s experience...

Oliver Bertuch (Mar 24 2025 at 15:10):

Oliver Bertuch (Mar 24 2025 at 15:11):

I assume to start working on our migration from Dataverse 4.20 on K8s to the latest and want to get some stuff done in a fork and upstream the changes later. Maybe some of these pain points align :smile_cat:

Philip Durbin 🚀 (Mar 24 2025 at 15:12):

Oliver Bertuch (Mar 24 2025 at 15:13):

Philip Durbin 🚀 (Mar 24 2025 at 15:14):

Gursahib (Mar 24 2025 at 15:25):

Haha, sure. I had to do lot of changes in the components along with Dataverse, but here are few Major ones:

The classic example for point 2 is smtp on port 25, and due to its failure, as port 25 is not allowed, there was connection request timeout period on whenever an Admin mail is created. And during this time out Dataverse freezes in operations like, creating new dataverse, dataset; publishing the dataset/ dataverse. Basically anything which needed to send an email notification.

I had used 2525 for port but default script uses 25, so I had to add a variable for that to.
These are few pain points I can think of and the major one is previewer functionality

Oliver Bertuch (Mar 24 2025 at 15:27):

Ad 1) I've been using init containers for that reason and it's also why the base image contains wait4x

Gursahib (Mar 24 2025 at 15:28):

Oliver Bertuch (Mar 24 2025 at 15:29):

Philip Durbin 🚀 (Mar 24 2025 at 15:31):

Oliver Bertuch (Mar 24 2025 at 15:32):

Philip Durbin 🚀 (Mar 24 2025 at 15:32):

Oliver Bertuch (Mar 24 2025 at 15:33):

I don't. But yes please let's rope in a discussion around IRC into this thread... The good ol' days :smiling_devil: :see_no_evil: :crazy:

Gursahib (Mar 24 2025 at 15:34):

Alright guys its getting late on my side of the world, I'll come back with news soon either good or bad idk. Enjoy your day.

Stream: containers

Topic: OpenShift

Philip Durbin 🚀 (Dec 18 2023 at 15:21):

Philip Durbin 🚀 (Feb 05 2024 at 14:40):

Oliver Bertuch (Feb 08 2024 at 14:19):

Oliver Bertuch (Feb 08 2024 at 14:19):

Philip Durbin 🚀 (Feb 08 2024 at 14:26):

Oliver Bertuch (Feb 09 2024 at 17:35):

Philip Durbin 🚀 (Feb 09 2024 at 17:37):

Oliver Bertuch (Feb 09 2024 at 18:01):

Oliver Bertuch (Feb 09 2024 at 18:01):

Oliver Bertuch (Feb 09 2024 at 18:02):

Oliver Bertuch (Feb 09 2024 at 18:02):

Philip Durbin 🚀 (Feb 09 2024 at 18:06):

Oliver Bertuch (Feb 09 2024 at 18:09):

Oliver Bertuch (Feb 09 2024 at 18:10):

Oliver Bertuch (Feb 09 2024 at 18:10):

Philip Durbin 🚀 (Feb 09 2024 at 19:19):

Oliver Bertuch (Feb 09 2024 at 19:19):

Oliver Bertuch (Feb 09 2024 at 19:19):

Oliver Bertuch (Feb 09 2024 at 19:22):

Philip Durbin 🚀 (Feb 10 2024 at 16:19):

Oliver Bertuch (Feb 10 2024 at 16:21):

Philip Durbin 🚀 (Feb 10 2024 at 23:26):

Philip Durbin 🚀 (Feb 10 2024 at 23:26):

Philip Durbin 🚀 (Feb 10 2024 at 23:30):

Philip Durbin 🚀 (Feb 10 2024 at 23:33):

Philip Durbin 🚀 (Feb 10 2024 at 23:52):

Philip Durbin 🚀 (Feb 10 2024 at 23:52):

Philip Durbin 🚀 (Feb 10 2024 at 23:53):

Philip Durbin 🚀 (Feb 10 2024 at 23:56):

Philip Durbin 🚀 (Feb 12 2024 at 18:16):

Philip Durbin 🚀 (Feb 15 2024 at 15:52):

Philip Durbin 🚀 (Feb 15 2024 at 15:53):

Philip Durbin 🚀 (Feb 15 2024 at 15:54):

Philip Durbin 🚀 (Feb 15 2024 at 15:56):

Philip Durbin 🚀 (Feb 15 2024 at 15:59):

Philip Durbin 🚀 (Feb 15 2024 at 19:31):

Philip Durbin 🚀 (Feb 15 2024 at 19:33):

Philip Durbin 🚀 (Feb 15 2024 at 19:35):

Philip Durbin 🚀 (Feb 15 2024 at 19:46):

Philip Durbin 🚀 (Feb 15 2024 at 19:46):

Philip Durbin 🚀 (Feb 15 2024 at 19:47):

Philip Durbin 🚀 (Feb 15 2024 at 21:11):

Philip Durbin 🚀 (Feb 16 2024 at 12:14):

Philip Durbin 🚀 (Feb 23 2024 at 20:57):

Philip Durbin 🚀 (Feb 23 2024 at 20:58):

Philip Durbin 🚀 (Feb 23 2024 at 20:59):

Philip Durbin 🚀 (Feb 23 2024 at 21:11):

Gursahib (Feb 28 2025 at 13:44):

Philip Durbin 🚀 (Feb 28 2025 at 14:13):

Philip Durbin 🚀 (Feb 28 2025 at 14:14):

Philip Durbin 🚀 (Feb 28 2025 at 14:15):

Philip Durbin 🚀 (Feb 28 2025 at 14:16):

Gursahib (Feb 28 2025 at 14:32):

Philip Durbin 🚀 (Feb 28 2025 at 14:43):

Philip Durbin 🚀 (Feb 28 2025 at 14:48):

Philip Durbin 🚀 (Feb 28 2025 at 14:51):

Philip Durbin 🚀 (Feb 28 2025 at 14:53):

Oliver Bertuch (Feb 28 2025 at 14:55):

Philip Durbin 🚀 (Feb 28 2025 at 14:55):

Gursahib (Feb 28 2025 at 14:56):

Gursahib (Feb 28 2025 at 14:58):

Oliver Bertuch (Feb 28 2025 at 14:58):

Oliver Bertuch (Feb 28 2025 at 14:59):

Philip Durbin 🚀 (Feb 28 2025 at 15:00):

Oliver Bertuch (Feb 28 2025 at 15:00):

Oliver Bertuch (Feb 28 2025 at 15:01):

Philip Durbin 🚀 (Feb 28 2025 at 15:01):

Gursahib (Feb 28 2025 at 15:04):

Oliver Bertuch (Feb 28 2025 at 15:05):

Oliver Bertuch (Feb 28 2025 at 15:06):

Oliver Bertuch (Feb 28 2025 at 15:09):

Gursahib (Feb 28 2025 at 15:11):

Oliver Bertuch (Feb 28 2025 at 15:20):

Gursahib (Feb 28 2025 at 15:22):

Oliver Bertuch (Feb 28 2025 at 15:24):

Oliver Bertuch (Feb 28 2025 at 15:25):

Oliver Bertuch (Feb 28 2025 at 15:26):

Gursahib (Feb 28 2025 at 15:34):