redhat-openshift-ai-notebooks-as-sudo-user

README.md

This Gist outlines the process of adding a sudo user to a Red Hat OpenAI (RHOAI) Workbench container image. Whether building the image from scratch using a Containerfile or utilizing a pre-built image, the steps provided here ensure the inclusion of a sudo user for enhanced permissions within the containerized environment.

1. Build the Container image using Containerfile file. The important part is. You can also use pre-built image which is published here quay.io/bpandey/rhoai-workbench:sudo-jupyteruser-v3
2. Create notebook using notebook.yaml file. Make sure to replace namespace, container image, OCP domain name. You can also create a workbench from RHOAI ui and patch following lines later.

          securityContext:
            allowPrivilegeEscalation: true
            runAsGroup: 1001
            runAsUser: 1001

One issue I've encountered with the patch is the following error message. The root cause of this error is that the /opt/app-root/src/ directory is persistent. When you launch the workbench from the RHOAI UI for the first time, it creates this directory with a random user. After applying the patch, we attempt to run the workbench as user 1001, which causes permission conflicts. To address this, we could implement an initialization script to resolve the permission issue automatically. Alternatively, we could create the workbench using the notebook.yaml file from the outset, eliminating this issue altogether

[I 2024-05-06 18:27:50.042 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[E 2024-05-06 18:27:50.042 ServerApp] Failed to write server-info to /opt/app-root/src/.local/share/jupyter/runtime/jpserver-13.json: PermissionError(13, 'Permission denied')
Traceback (most recent call last):
 File "/opt/app-root/bin/jupyter-lab", line 8, in <module>
   sys.exit(main())
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/extension/application.py", line 617, in launch_instance
   serverapp.start()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2952, in start
   self.start_app()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2856, in start_app
   self.write_browser_open_files()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2723, in write_browser_open_files
   self.write_browser_open_file()
 File "/opt/app-root/lib64/python3.9/site-packages/jupyter_server/serverapp.py", line 2746, in write_browser_open_file
   with open(self.browser_open_file, "w", encoding="utf-8") as f:
PermissionError: [Errno 13] Permission denied: '/opt/app-root/src/.local/share/jupyter/runtime/jpserver-13-open.html'
[INFO 2024-05-06 18:27:52.152 TabnineDownloader]: Finish download Tabnine Binary to /opt/app-root/lib/python3.9/site-packages/jupyterlab_tabnine/binaries/4.159.0/x86_64-unknown-linux-musl

3. Add scc profile

oc adm policy add-scc-to-user privileged -z sudo-jupyteruser

Security Disclaimer:

Adding a sudo user introduces security vulnerabilities by granting elevated privileges within the containerized environment. Exercise caution and consider the security implications before implementing this solution.

Additionally, creating the workbench from the RHOAI UI introduces complexity due to potential permission conflicts and error logs as mentioned previously.

Blog Reference

Containerfile

FROM registry.access.redhat.com/ubi8/python-39:1-176.1712880517

LABEL name="workbench-images:cuda-jupyter-pytorch-c9s-py39_2024b_20240418" \
    summary="cuda-jupyter-pytorch workbench image with Python py39 based on c9s" \
    description="cuda-jupyter-pytorch workbench image with Python py39 based on c9s" \
    io.k8s.description="cuda-jupyter-pytorch workbench image  with Python py39 based on c9s for ODH or RHODS" \
    io.k8s.display-name="cuda-jupyter-pytorch workbench image  with Python py39 based on c9s" \
    authoritative-source-url="https://github.com/opendatahub-contrib/workbench-images" \
    io.openshift.build.commit.ref="2024b" \
    io.openshift.build.source-location="https://github.com/opendatahub-contrib/workbench-images" \
    io.openshift.build.image="https://quay.io/opendatahub-contrib/workbench-images:cuda-jupyter-pytorch-c9s-py39_2024b_20240418"

##########################
# Deploy Python packages #
##########################

USER 1001

WORKDIR /opt/app-root/bin/

# Copy packages list
COPY --chown=1001:0 requirements.txt ./

# Install packages and cleanup
# (all commands are chained to minimize layer size)
RUN echo "Installing softwares and packages" && \
    # Install Python packages \
    pip install --no-cache-dir -r requirements.txt && \
    # Fix permissions to support pip in Openshift environments \
    chmod -R g+w /opt/app-root/lib/python3.9/site-packages && \
    fix-permissions /opt/app-root -P

WORKDIR /opt/app-root/src/

##########################

###########################
# Deploy OS Packages      #
###########################

USER 0

WORKDIR /opt/app-root/bin/

# COPY --chown=1001:0 os-ide/os-packages.txt ./os-ide/os-packages.txt

# RUN yum install -y $(cat os-ide/os-packages.txt) && \
#     rm -f os-ide/os-packages.txt && \
#     yum -y clean all --enablerepo='*' && \
#     rm -rf /var/cache/dnf && \
#     find /var/log -type f -name "*.log" -exec rm -f {} \;

###########################
# Deploy CUDA Packages    #
###########################

# Add the CUDA repository and GPG key
RUN dnf install -y dnf-plugins-core \
    && dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo \
    && rpm --import https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/7fa2af80.pub

# Install the CUDA toolkit (adjust version as needed)
RUN dnf clean all \
    && dnf -y install cuda-toolkit-11-2 \
    && dnf clean all

# Set PATH so it includes CUDA's bin directory
ENV PATH /usr/local/cuda-11.2/bin:${PATH}

# Set LD_LIBRARY_PATH to include CUDA's lib64 directory
ENV LD_LIBRARY_PATH /usr/local/cuda-11.2/lib64:${LD_LIBRARY_PATH}

###########################

##############################
# Deploy Jupyterlab packages #
##############################
USER 0

RUN dnf install -y jq

USER 1001

WORKDIR /opt/app-root/bin

# Copy packages list
COPY --chown=1001:0 requirements-jupyter.txt ./

# Copy notebook launcher and utils
COPY --chown=1001:0 utils utils/
COPY --chown=1001:0 start-notebook.sh ./

# Streamlit extension installation
COPY --chown=1001:0 streamlit-launcher.sh ./
COPY --chown=1001:0 streamlit-menu/dist/jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl ./

# Copy Elyra setup to utils so that it's sourced at startup
COPY --chown=1001:0 setup-elyra.sh ./utils/

# Install packages and cleanup
# (all commands are chained to minimize layer size)
RUN echo "Installing softwares and packages" && \
    # Install Python packages \
    pip install --no-cache-dir -r requirements-jupyter.txt && \
    pip install --no-cache-dir ./jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl && \
    rm -f ./jupyterlab_streamlit_menu-0.1.0-py3-none-any.whl && \
    # setup path for runtime configuration \
    mkdir /opt/app-root/runtimes && \
    # switch to Data Science Pipeline \
    cp utils/pipeline-flow.svg /opt/app-root/lib/python3.9/site-packages/elyra/static/icons/kubeflow.svg && \
    sed -i "s/Kubeflow Pipelines/Data Science/g" /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/runtime_type.py && \
    sed -i "s/Kubeflow Pipelines/Data Science Pipelines/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
    sed -i "s/kubeflow-service/data-science-pipeline-service/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
    sed -i "s/\"default\": \"Argo\",/\"default\": \"Tekton\",/g" /opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/kfp.json && \
    # Workaround for passing ssl_sa_cert and to ensure that Elyra redirects to a correct pipeline run URL \
    patch /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/kfp/kfp_authentication.py -i utils/kfp_authentication.patch && \
    patch /opt/app-root/lib/python3.9/site-packages/elyra/pipeline/kfp/processor_kfp.py -i utils/processor_kfp.patch && \
    # switch to Data Science Pipeline in component catalog \
    DIR_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/local-directory-catalog.json" && \
    FILE_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/local-file-catalog.json" && \
    URL_COMPONENT="/opt/app-root/lib/python3.9/site-packages/elyra/metadata/schemas/url-catalog.json" && \
    tmp=$(mktemp) && \
    jq '.properties.metadata.properties.runtime_type = input' $DIR_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $DIR_COMPONENT && \
    jq '.properties.metadata.properties.runtime_type = input' $FILE_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $FILE_COMPONENT && \
    jq '.properties.metadata.properties.runtime_type = input' $URL_COMPONENT utils/component_runtime.json > "$tmp" && mv "$tmp" $URL_COMPONENT && \
    sed -i "s/metadata.metadata.runtime_type/\"DATA_SCIENCE_PIPELINES\"/g" /opt/app-root/share/jupyter/labextensions/@elyra/pipeline-editor-extension/static/lib_index_js.*.js && \
    # Remove Elyra logo from JupyterLab because this is not a pure Elyra image \
    sed -i "s/widget\.id === \x27jp-MainLogo\x27/widget\.id === \x27jp-MainLogo\x27 \&\& false/" /opt/app-root/share/jupyter/labextensions/@elyra/theme-extension/static/lib_index_js.*.js && \
    # Replace Notebook's launcher, "(ipykernel)" with Python's version 3.x.y \
    sed -i -e "s/Python.*/$(python --version | cut -d '.' -f-2)\",/" /opt/app-root/share/jupyter/kernels/python3/kernel.json && \
    # Remove default Elyra runtime-images \
    rm /opt/app-root/share/jupyter/metadata/runtime-images/*.json && \
    # Fix permissions to support pip in Openshift environments \
    chmod -R g+w /opt/app-root/lib/python3.9/site-packages && \
    fix-permissions /opt/app-root -P

# Copy Elyra runtime-images definitions and set the version
COPY --chown=1001:0 runtime-images/ /opt/app-root/share/jupyter/metadata/runtime-images/
RUN sed -i "s/RELEASE/2024b/" /opt/app-root/share/jupyter/metadata/runtime-images/*.json 

# Jupyter Server config to allow hidden files/folders in explorer. Ref: https://jupyterlab.readthedocs.io/en/latest/user/files.html#displaying-hidden-files
# Jupyter Lab config to hide disabled exporters (WebPDF, Qtpdf, Qtpng)
COPY --chown=1001:0 etc/ /opt/app-root/etc/jupyter/

USER 0

RUN echo "sudoers: files" >> /etc/nsswitch.conf
RUN dnf install -y sudo
RUN echo "default	ALL=(ALL)	NOPASSWD:ALL" >> /etc/sudoers

USER 1001

WORKDIR /opt/app-root/src

ENTRYPOINT ["start-notebook.sh"]

notebook-initconainer-to-fix-permission.yaml

apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
  annotations:
    notebooks.opendatahub.io/inject-oauth: 'true'
    notebooks.opendatahub.io/last-image-selection: 'custom-sudo-jupyteruser:sudo-jupyteruser'
    notebooks.opendatahub.io/last-size-selection: Small
    notebooks.opendatahub.io/oauth-logout-url: >-
      https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
    opendatahub.io/accelerator-name: ''
    opendatahub.io/image-display-name: sudo-jupyteruser
    opendatahub.io/username: cloud-user
    openshift.io/description: ''
    openshift.io/display-name: sudo-jupyteruser
  labels:
    app: sudo-jupyteruser
    opendatahub.io/dashboard: 'true'
    opendatahub.io/odh-managed: 'true'
    opendatahub.io/user: cloud-2duser
  name: sudo-jupyteruser
  namespace: modelserver
spec:
  template:
    spec:
      affinity: {}
       initContainers:
        - name: fix-permissions
          image: registry.access.redhat.com/ubi8/python-39:1-176.1712880517
          command: ["sh", "-c", "chown -R default:1001 /opt/app-root/src/"]
          securityContext:
            runAsUser: 0
            allowPrivilegeEscalation: false
          volumeMounts:
            - mountPath: /opt/app-root/src
              name: sudo-jupyteruser
      containers:
        - env:
            - name: NOTEBOOK_ARGS
              value: |-
                --ServerApp.port=8888
                                  --ServerApp.token=''
                                  --ServerApp.password=''
                                  --ServerApp.base_url=/notebook/modelserver/sudo-jupyteruser
                                  --ServerApp.quit_button=False
                                  --ServerApp.tornado_settings={"user":"cloud-2duser","hub_host":"https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run","hub_prefix":"/projects/modelserver"}
            - name: JUPYTER_IMAGE
              value: >-
                image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
          image: >-
            image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /notebook/modelserver/sudo-jupyteruser/api
              port: notebook-port
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          name: sudo-jupyteruser
          ports:
            - containerPort: 8888
              name: notebook-port
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /notebook/modelserver/sudo-jupyteruser/api
              port: notebook-port
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: '2'
              memory: 8Gi
            requests:
              cpu: '1'
              memory: 8Gi
          securityContext:
            allowPrivilegeEscalation: true
            runAsGroup: 1001
            runAsUser: 1001
          volumeMounts:
            - mountPath: /opt/app-root/src
              name: sudo-jupyteruser
            - mountPath: /dev/shm
              name: shm
          workingDir: /opt/app-root/src
        - args:
            - '--provider=openshift'
            - '--https-address=:8443'
            - '--http-address='
            - '--openshift-service-account=sudo-jupyteruser'
            - '--cookie-secret-file=/etc/oauth/config/cookie_secret'
            - '--cookie-expire=24h0m0s'
            - '--tls-cert=/etc/tls/private/tls.crt'
            - '--tls-key=/etc/tls/private/tls.key'
            - '--upstream=http://localhost:8888'
            - '--upstream-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
            - '--email-domain=*'
            - '--skip-provider-button'
            - >-
              --openshift-sar={"verb":"get","resource":"notebooks","resourceAPIGroup":"kubeflow.org","resourceName":"sudo-jupyteruser","namespace":"$(NAMESPACE)"}
            - >-
              --logout-url=https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: >-
            registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /oauth/healthz
              port: oauth-proxy
              scheme: HTTPS
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          name: oauth-proxy
          ports:
            - containerPort: 8443
              name: oauth-proxy
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /oauth/healthz
              port: oauth-proxy
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 100m
              memory: 64Mi
            requests:
              cpu: 100m
              memory: 64Mi
          volumeMounts:
            - mountPath: /etc/oauth/config
              name: oauth-config
            - mountPath: /etc/tls/private
              name: tls-certificates
      enableServiceLinks: false
      serviceAccountName: sudo-jupyteruser
      volumes:
        - name: sudo-jupyteruser
          persistentVolumeClaim:
            claimName: sudo-jupyteruser
        - emptyDir:
            medium: Memory
          name: shm
        - name: oauth-config
          secret:
            defaultMode: 420
            secretName: sudo-jupyteruser-oauth-config
        - name: tls-certificates
          secret:
            defaultMode: 420
            secretName: sudo-jupyteruser-tls

notebook.yaml

apiVersion: kubeflow.org/v1
kind: Notebook
metadata:
  annotations:
    notebooks.opendatahub.io/inject-oauth: 'true'
    notebooks.opendatahub.io/last-image-selection: 'custom-sudo-jupyteruser:sudo-jupyteruser'
    notebooks.opendatahub.io/last-size-selection: Small
    notebooks.opendatahub.io/oauth-logout-url: >-
      https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
    opendatahub.io/accelerator-name: ''
    opendatahub.io/image-display-name: sudo-jupyteruser
    opendatahub.io/username: cloud-user
    openshift.io/description: ''
    openshift.io/display-name: sudo-jupyteruser
  labels:
    app: sudo-jupyteruser
    opendatahub.io/dashboard: 'true'
    opendatahub.io/odh-managed: 'true'
    opendatahub.io/user: cloud-2duser
  name: sudo-jupyteruser
  namespace: modelserver
spec:
  template:
    spec:
      affinity: {}
      containers:
        - env:
            - name: NOTEBOOK_ARGS
              value: |-
                --ServerApp.port=8888
                                  --ServerApp.token=''
                                  --ServerApp.password=''
                                  --ServerApp.base_url=/notebook/modelserver/sudo-jupyteruser
                                  --ServerApp.quit_button=False
                                  --ServerApp.tornado_settings={"user":"cloud-2duser","hub_host":"https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run","hub_prefix":"/projects/modelserver"}
            - name: JUPYTER_IMAGE
              value: >-
                image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
          image: >-
            image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/custom-sudo-default-user-v3:sudo-jupyteruser-v3
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /notebook/modelserver/sudo-jupyteruser/api
              port: notebook-port
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          name: sudo-jupyteruser
          ports:
            - containerPort: 8888
              name: notebook-port
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /notebook/modelserver/sudo-jupyteruser/api
              port: notebook-port
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: '2'
              memory: 8Gi
            requests:
              cpu: '1'
              memory: 8Gi
          securityContext:
            allowPrivilegeEscalation: true
            runAsGroup: 1001
            runAsUser: 1001
          volumeMounts:
            - mountPath: /opt/app-root/src
              name: sudo-jupyteruser
            - mountPath: /dev/shm
              name: shm
          workingDir: /opt/app-root/src
        - args:
            - '--provider=openshift'
            - '--https-address=:8443'
            - '--http-address='
            - '--openshift-service-account=sudo-jupyteruser'
            - '--cookie-secret-file=/etc/oauth/config/cookie_secret'
            - '--cookie-expire=24h0m0s'
            - '--tls-cert=/etc/tls/private/tls.crt'
            - '--tls-key=/etc/tls/private/tls.key'
            - '--upstream=http://localhost:8888'
            - '--upstream-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
            - '--email-domain=*'
            - '--skip-provider-button'
            - >-
              --openshift-sar={"verb":"get","resource":"notebooks","resourceAPIGroup":"kubeflow.org","resourceName":"sudo-jupyteruser","namespace":"$(NAMESPACE)"}
            - >-
              --logout-url=https://rhods-dashboard-redhat-ods-applications.apps.cloud1c.example.denver.ocp.run/projects/modelserver?notebookLogout=sudo-jupyteruser
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: >-
            registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /oauth/healthz
              port: oauth-proxy
              scheme: HTTPS
            initialDelaySeconds: 30
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          name: oauth-proxy
          ports:
            - containerPort: 8443
              name: oauth-proxy
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /oauth/healthz
              port: oauth-proxy
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: 100m
              memory: 64Mi
            requests:
              cpu: 100m
              memory: 64Mi
          volumeMounts:
            - mountPath: /etc/oauth/config
              name: oauth-config
            - mountPath: /etc/tls/private
              name: tls-certificates
      enableServiceLinks: false
      serviceAccountName: sudo-jupyteruser
      volumes:
        - name: sudo-jupyteruser
          persistentVolumeClaim:
            claimName: sudo-jupyteruser
        - emptyDir:
            medium: Memory
          name: shm
        - name: oauth-config
          secret:
            defaultMode: 420
            secretName: sudo-jupyteruser-oauth-config
        - name: tls-certificates
          secret:
            defaultMode: 420
            secretName: sudo-jupyteruser-tls