Kubernetes Operator :: Cluster Config :: Mount Issue

Enonic version: current
OS: MAC and WIN

I am preparing my production environment and test to create a xp-cluster in my Test-Environment (K3-Cluster on QNAP).

I successfully was able to deploy a single node configuration.
With the cluster-configuration I get the following error on all worker and master-pods

Unable to attach or mount volumes: unmounted volumes=[blobstore snapshots export], unattached volumes=[kube-api-access-dhp98 config extra-config deploy index blobstore snapshots export]: timed out waiting for the condition

I have no error on the persistent volumes and persistern volume claims


I have NFS-Service on the Host and also the nfs-server-nfs-server-provisioner running.

Any ideas what I am missing ?

Hi! Could you share your deployment configuration, please?

@vbr :here we go :slight_smile:

# Create a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: osde-ns
  annotations:
    # Delete this namespace it the deployment is deleted
    enonic.cloud/remove.with.xp7deployment: osde-deploy
---
# Create deployment in the namespace
apiVersion: enonic.cloud/v1
kind: Xp7Deployment
metadata:
  name: osde-deploy
  namespace: osde-ns
spec:
  enabled: true
  xpVersion: 7.12.2

  # Preinstall snapshotter on all nodes
  nodesPreinstalledApps:
    - name: snapshotter
      url: https://repo.enonic.com/public/com/enonic/app/snapshotter/3.0.2/snapshotter-3.0.2.jar

  # Create volumes shared by all nodes in this deployment
  nodesSharedDisks:
    - name: blobstore
      size: 1Gi

    - name: snapshots
      size: 1Gi

    - name: export # Dumps and other data
      size: 1Gi

  # Create nodes
  nodeGroups:
    # 3 master nodes
    - name: master
      replicas: 3

      data: false
      master: true

      resources:
        cpu: "0.5"
        memory: 1Gi

        # Volumes private to the node
        disks:
          - name: deploy  # Apps installed in the deploy folder
            size: 1Gi
          - name: index   # Node ES index
            size: 1Gi

    # 2 data nodes
    - name: worker
      replicas: 2

      data: true
      master: false

      resources:
        cpu: "1"
        memory: 1Gi

        # Volumes private to the node
        disks:
          - name: deploy  # Apps installed in the deploy folder
            size: 1Gi
          - name: index   # Node ES index
            size: 1Gi
---
# Install content studio
apiVersion: enonic.cloud/v1
kind: Xp7App
metadata:
  name: contentstudio
  namespace: osde-ns
spec:
  url: https://repo.enonic.com/public/com/enonic/app/contentstudio/4.5.1/contentstudio-4.5.2.jar
  sha512: e5662edb8757ceb6f085d1a8d85abf965c0f45a98acbc767ef31f4e8d860fc88cc995aa7cfcb1167f356c3e261129524f3c38de93cdc08baca95e51943a99365
# Add your own custom config
#apiVersion: enonic.cloud/v1
#kind: Xp7Config
#metadata:
#  name: my-config
#  namespace: osde-ns
#spec:
#  nodeGroup: all
#  file: com.my-app.cfg
#  data: |
#    my = config
---
# Expose XP site on frontend nodes through an ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-domain-com-site
  namespace: osde-ns
  annotations:
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.source: /
    enonic.cloud/xp7.vhost.mapping.my-mapping-site.target: /site/default/master/homepage
spec:
  rules:
    - host: lie-nas-2.m27.local
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: worker
                port:
                  number: 8080
---
# Expose XP admin on admin nodes through an ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: lie-nas-2-m27-admin
  namespace: osde-ns
  annotations:
    # Enable sticy sessions with nginx
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "stickyXpAdmin"
    nginx.ingress.kubernetes.io/session-cookie-expires: "129600" # 36 hours
    nginx.ingress.kubernetes.io/session-cookie-max-age: "129600" # 36 hours
    nginx.ingress.kubernetes.io/session-cookie-change-on-failure: "true"

    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.source: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.target: /admin
    enonic.cloud/xp7.vhost.mapping.my-mapping-admin.idproviders: system
spec:
  rules:
    - host: lie-nas-2.m27.local
      http:
        paths:
          - path: /admin
            pathType: ImplementationSpecific
            backend:
              service:
                name: worker
                port:
                  number: 8080

The config is correct. What about the events, anything that looks like a lead there?

@vbr which Events do you mean ?

I mean events from the created namespace, you can get it by a command like:
kubectl get events -n osde-ns --sort-by=‘.metadata.creationTimestamp’

Unforntunately that does not tell us more :frowning:

23s         Warning   FailedMount   pod/master-2   Unable to attach or mount volumes: unmounted volumes=[export blobstore snapshots], unattached volumes=[export kube-api-access-b9wzn config extra-config deploy index blobstore snapshots]: timed out waiting for the condition

Hmm, I was able to run your config both on local minikube and Google Cloud. But I know about some issues with pvc on Azure Cloud. Could you check how your cluster works with the requested volumes and volume claims types?

You can play with storage types by this config:
(sharedDisks) → operator.charts.values.storage.shared.storageClassName=nfs
(node volumes) → operator.charts.values.storage.default.storageClassName=standard

After a long break - I continued working on my project. :slight_smile:

I am still not able to successfully test cluster-configuration on my local environment and in GKE.

This is what I tested so far:

  1. K3 Cluster on QNAP-Container Station does not support NFS

  2. MacBook (M1 Chip) with Minikube and Docker for VM- I tried the most current K8-Version (v.1.27.3) and also v1.19.11 as requested in the docs. Everything is created (PV, Pods, Services) - In the logs of the worker-pods I see some errors - and I am not able to access Admin-Pages using minikube-tunnel

  3. Windows PC
    3.1 minikube and VirtualBox → everything is created but very instable. Pods are crashing and timeouts when using kubectl
    3.2 minikube and docker → runs better , but also pods are crashing (here the result after 1 hour)

PS D:\Developement\enonic\xp-operator> kubectl get po -n osde-ns
NAME       READY   STATUS             RESTARTS   AGE
master-0   2/2     Running            16         62m
master-1   1/2     CrashLoopBackOff   14         62m
master-2   2/2     Running            14         62m
worker-0   2/2     Running            4          62m
worker-1   2/2     Running            4          62m

and here are all events in namespase osde-ns

LAST SEEN   TYPE      REASON                   OBJECT                                  MESSAGE
69m         Warning   FailedToUpdateEndpoint   endpoints/all                           Failed to update endpoint osde-ns/all: Operation cannot be fulfilled on endpoints "all": the object has been modified; please apply your changes to the latest version and try again
76m         Normal    ExternalProvisioning     persistentvolumeclaim/blobstore         waiting for a volume to be created, either by external provisioner "cluster.local/nfs-server-nfs-server-provisioner" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/blobstore         External provisioner is provisioning volume for claim "osde-ns/blobstore"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/blobstore         Successfully provisioned volume pvc-9f7001d0-4b89-4979-9366-e6f9b7ec61d6
68m         Warning   FailedToUpdateEndpoint   endpoints/cluster-discovery             Failed to update endpoint osde-ns/cluster-discovery: Operation cannot be fulfilled on endpoints "cluster-discovery": the object has been modified; please apply your changes to the latest version and try again
76m         Normal    ExternalProvisioning     persistentvolumeclaim/deploy-master-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/deploy-master-0   External provisioner is provisioning volume for claim "osde-ns/deploy-master-0"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/deploy-master-0   Successfully provisioned volume pvc-1f437cf1-642e-4fa7-91c4-8d64de2eed44
76m         Normal    ExternalProvisioning     persistentvolumeclaim/deploy-master-1   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/deploy-master-1   External provisioner is provisioning volume for claim "osde-ns/deploy-master-1"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/deploy-master-1   Successfully provisioned volume pvc-126b33ee-5b0b-4cce-b7ba-e0c4fdbf8d6d
76m         Normal    ExternalProvisioning     persistentvolumeclaim/deploy-master-2   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/deploy-master-2   External provisioner is provisioning volume for claim "osde-ns/deploy-master-2"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/deploy-master-2   Successfully provisioned volume pvc-98294e8f-0909-481a-afab-5894921b7dd3
76m         Normal    Provisioning             persistentvolumeclaim/deploy-worker-0   External provisioner is provisioning volume for claim "osde-ns/deploy-worker-0"
76m         Normal    ExternalProvisioning     persistentvolumeclaim/deploy-worker-0   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/deploy-worker-0   Successfully provisioned volume pvc-5447b5df-dae4-472a-81e0-6250032cc404
76m         Normal    ExternalProvisioning     persistentvolumeclaim/deploy-worker-1   waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/deploy-worker-1   External provisioner is provisioning volume for claim "osde-ns/deploy-worker-1"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/deploy-worker-1   Successfully provisioned volume pvc-d577c4ee-3959-46ba-8fff-3941e7795b0c
76m         Normal    ExternalProvisioning     persistentvolumeclaim/export            waiting for a volume to be created, either by external provisioner "cluster.local/nfs-server-nfs-server-provisioner" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/export            External provisioner is provisioning volume for claim "osde-ns/export"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/export            Successfully provisioned volume pvc-41219e4c-8cd6-486a-b60f-8b97e215aa3b
76m         Normal    ConfigModified           configmap/extra-config                  ConfigMap modified
76m         Normal    ExternalProvisioning     persistentvolumeclaim/index-master-0    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/index-master-0    External provisioner is provisioning volume for claim "osde-ns/index-master-0"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/index-master-0    Successfully provisioned volume pvc-a59288b2-1dd5-4559-9fc2-614e987c81f3
76m         Normal    ExternalProvisioning     persistentvolumeclaim/index-master-1    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/index-master-1    External provisioner is provisioning volume for claim "osde-ns/index-master-1"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/index-master-1    Successfully provisioned volume pvc-3eb2b40e-ec1c-4067-8d4b-312c503ba454
76m         Normal    ExternalProvisioning     persistentvolumeclaim/index-master-2    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/index-master-2    External provisioner is provisioning volume for claim "osde-ns/index-master-2"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/index-master-2    Successfully provisioned volume pvc-4b878029-674d-4746-a7a6-a170c982fbe4
76m         Normal    Provisioning             persistentvolumeclaim/index-worker-0    External provisioner is provisioning volume for claim "osde-ns/index-worker-0"
76m         Normal    ExternalProvisioning     persistentvolumeclaim/index-worker-0    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/index-worker-0    Successfully provisioned volume pvc-c0e3b880-fdc4-4f14-ac4a-790f41d738fe
76m         Normal    ExternalProvisioning     persistentvolumeclaim/index-worker-1    waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
76m         Normal    Provisioning             persistentvolumeclaim/index-worker-1    External provisioner is provisioning volume for claim "osde-ns/index-worker-1"
76m         Normal    ProvisioningSucceeded    persistentvolumeclaim/index-worker-1    Successfully provisioned volume pvc-c95dcd69-6fec-4cfe-ac73-5f9bbf6da867
76m         Normal    Sync                     ingress/localhost-admin                 Scheduled for sync
71m         Normal    Sync                     ingress/localhost-admin                 Scheduled for sync
25m         Normal    Sync                     ingress/localhost-admin                 Scheduled for sync
76m         Normal    Sync                     ingress/localhost-page                  Scheduled for sync
71m         Normal    Sync                     ingress/localhost-page                  Scheduled for sync
25m         Normal    Sync                     ingress/localhost-page                  Scheduled for sync
76m         Warning   FailedScheduling         pod/master-0                            0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
76m         Normal    Scheduled                pod/master-0                            Successfully assigned osde-ns/master-0 to minikube
76m         Normal    Pulling                  pod/master-0                            Pulling image "busybox"
76m         Normal    Pulled                   pod/master-0                            Successfully pulled image "busybox" in 3.237902499s
76m         Normal    Created                  pod/master-0                            Created container configure-sysctl
76m         Normal    Started                  pod/master-0                            Started container configure-sysctl
76m         Normal    Pulling                  pod/master-0                            Pulling image "enonic/xp:7.11.1-ubuntu"
75m         Normal    Pulled                   pod/master-0                            Successfully pulled image "enonic/xp:7.11.1-ubuntu" in 49.529760922s
75m         Normal    Created                  pod/master-0                            Created container exp
75m         Normal    Started                  pod/master-0                            Started container exp
75m         Normal    Pulled                   pod/master-0                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
75m         Normal    Created                  pod/master-0                            Created container events
75m         Normal    Started                  pod/master-0                            Started container events
71m         Normal    SandboxChanged           pod/master-0                            Pod sandbox changed, it will be killed and re-created.
71m         Normal    Pulled                   pod/master-0                            Container image "busybox" already present on machine
71m         Normal    Created                  pod/master-0                            Created container configure-sysctl
71m         Normal    Started                  pod/master-0                            Started container configure-sysctl
64m         Normal    Pulled                   pod/master-0                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
64m         Normal    Created                  pod/master-0                            Created container exp
64m         Normal    Started                  pod/master-0                            Started container exp
71m         Normal    Pulled                   pod/master-0                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
71m         Normal    Created                  pod/master-0                            Created container events
71m         Normal    Started                  pod/master-0                            Started container events
36m         Warning   BackOff                  pod/master-0                            Back-off restarting failed container
46m         Warning   NodeNotReady             pod/master-0                            Node is not ready
26m         Normal    SandboxChanged           pod/master-0                            Pod sandbox changed, it will be killed and re-created.
26m         Normal    Pulled                   pod/master-0                            Container image "busybox" already present on machine
26m         Normal    Created                  pod/master-0                            Created container configure-sysctl
26m         Normal    Started                  pod/master-0                            Started container configure-sysctl
18m         Normal    Pulled                   pod/master-0                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
18m         Normal    Created                  pod/master-0                            Created container exp
18m         Normal    Started                  pod/master-0                            Started container exp
26m         Normal    Pulled                   pod/master-0                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
26m         Normal    Created                  pod/master-0                            Created container events
26m         Normal    Started                  pod/master-0                            Started container events
22m         Warning   NodeNotReady             pod/master-0                            Node is not ready
98s         Warning   BackOff                  pod/master-0                            Back-off restarting failed container
71m         Normal    ConfigReload             pod/master-0                            Pod master-0 reloaded ConfigMap master
26m         Normal    ConfigReload             pod/master-0                            Pod master-0 reloaded ConfigMap master
75m         Normal    ConfigReload             pod/master-0                            Pod master-0 reloaded ConfigMap master
76m         Warning   FailedScheduling         pod/master-1                            0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
76m         Normal    Scheduled                pod/master-1                            Successfully assigned osde-ns/master-1 to minikube
76m         Normal    Pulled                   pod/master-1                            Container image "busybox" already present on machine
76m         Normal    Created                  pod/master-1                            Created container configure-sysctl
76m         Normal    Started                  pod/master-1                            Started container configure-sysctl
76m         Normal    Pulling                  pod/master-1                            Pulling image "enonic/xp:7.11.1-ubuntu"
75m         Normal    Pulled                   pod/master-1                            Successfully pulled image "enonic/xp:7.11.1-ubuntu" in 47.393845407s
75m         Normal    Created                  pod/master-1                            Created container exp
75m         Normal    Started                  pod/master-1                            Started container exp
75m         Normal    Pulled                   pod/master-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
75m         Normal    Created                  pod/master-1                            Created container events
75m         Normal    Started                  pod/master-1                            Started container events
71m         Normal    SandboxChanged           pod/master-1                            Pod sandbox changed, it will be killed and re-created.
71m         Normal    Pulled                   pod/master-1                            Container image "busybox" already present on machine
71m         Normal    Created                  pod/master-1                            Created container configure-sysctl
71m         Normal    Started                  pod/master-1                            Started container configure-sysctl
64m         Normal    Pulled                   pod/master-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
64m         Normal    Created                  pod/master-1                            Created container exp
64m         Normal    Started                  pod/master-1                            Started container exp
71m         Normal    Pulled                   pod/master-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
71m         Normal    Created                  pod/master-1                            Created container events
71m         Normal    Started                  pod/master-1                            Started container events
69m         Warning   NodeNotReady             pod/master-1                            Node is not ready
35m         Warning   BackOff                  pod/master-1                            Back-off restarting failed container
26m         Normal    SandboxChanged           pod/master-1                            Pod sandbox changed, it will be killed and re-created.
26m         Normal    Pulled                   pod/master-1                            Container image "busybox" already present on machine
26m         Normal    Created                  pod/master-1                            Created container configure-sysctl
26m         Normal    Started                  pod/master-1                            Started container configure-sysctl
17m         Normal    Pulled                   pod/master-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
17m         Normal    Created                  pod/master-1                            Created container exp
17m         Normal    Started                  pod/master-1                            Started container exp
26m         Normal    Pulled                   pod/master-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
26m         Normal    Created                  pod/master-1                            Created container events
26m         Normal    Started                  pod/master-1                            Started container events
22m         Warning   NodeNotReady             pod/master-1                            Node is not ready
6m29s       Warning   BackOff                  pod/master-1                            Back-off restarting failed container
71m         Normal    ConfigReload             pod/master-1                            Pod master-1 reloaded ConfigMap master
26m         Normal    ConfigReload             pod/master-1                            Pod master-1 reloaded ConfigMap master
75m         Normal    ConfigReload             pod/master-1                            Pod master-1 reloaded ConfigMap master
76m         Warning   FailedScheduling         pod/master-2                            0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
76m         Normal    Scheduled                pod/master-2                            Successfully assigned osde-ns/master-2 to minikube
76m         Normal    Pulled                   pod/master-2                            Container image "busybox" already present on machine
76m         Normal    Created                  pod/master-2                            Created container configure-sysctl
76m         Normal    Started                  pod/master-2                            Started container configure-sysctl
76m         Normal    Pulling                  pod/master-2                            Pulling image "enonic/xp:7.11.1-ubuntu"
75m         Normal    Pulled                   pod/master-2                            Successfully pulled image "enonic/xp:7.11.1-ubuntu" in 50.890399246s
75m         Normal    Created                  pod/master-2                            Created container exp
75m         Normal    Started                  pod/master-2                            Started container exp
75m         Normal    Pulled                   pod/master-2                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
75m         Normal    Created                  pod/master-2                            Created container events
75m         Normal    Started                  pod/master-2                            Started container events
71m         Normal    SandboxChanged           pod/master-2                            Pod sandbox changed, it will be killed and re-created.
71m         Normal    Pulled                   pod/master-2                            Container image "busybox" already present on machine
71m         Normal    Created                  pod/master-2                            Created container configure-sysctl
71m         Normal    Started                  pod/master-2                            Started container configure-sysctl
35m         Normal    Pulled                   pod/master-2                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
63m         Normal    Created                  pod/master-2                            Created container exp
65m         Normal    Started                  pod/master-2                            Started container exp
71m         Normal    Pulled                   pod/master-2                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
71m         Normal    Created                  pod/master-2                            Created container events
71m         Normal    Started                  pod/master-2                            Started container events
46m         Warning   NodeNotReady             pod/master-2                            Node is not ready
36m         Warning   BackOff                  pod/master-2                            Back-off restarting failed container
26m         Normal    SandboxChanged           pod/master-2                            Pod sandbox changed, it will be killed and re-created.
26m         Normal    Pulled                   pod/master-2                            Container image "busybox" already present on machine
26m         Normal    Created                  pod/master-2                            Created container configure-sysctl
26m         Normal    Started                  pod/master-2                            Started container configure-sysctl
16m         Normal    Pulled                   pod/master-2                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
16m         Normal    Created                  pod/master-2                            Created container exp
16m         Normal    Started                  pod/master-2                            Started container exp
26m         Normal    Pulled                   pod/master-2                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
26m         Normal    Created                  pod/master-2                            Created container events
26m         Normal    Started                  pod/master-2                            Started container events
22m         Warning   NodeNotReady             pod/master-2                            Node is not ready
6m9s        Warning   BackOff                  pod/master-2                            Back-off restarting failed container
26m         Normal    ConfigReload             pod/master-2                            Pod master-2 reloaded ConfigMap master
71m         Normal    ConfigReload             pod/master-2                            Pod master-2 reloaded ConfigMap master
75m         Normal    ConfigReload             pod/master-2                            Pod master-2 reloaded ConfigMap master
76m         Normal    ConfigModified           configmap/master                        ConfigMap modified
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim deploy-master-0 Pod master-0 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim index-master-0 Pod master-0 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Pod master-0 in StatefulSet master successful
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim deploy-master-1 Pod master-1 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim index-master-1 Pod master-1 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Pod master-1 in StatefulSet master successful
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim deploy-master-2 Pod master-2 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Claim index-master-2 Pod master-2 in StatefulSet master success
76m         Normal    SuccessfulCreate         statefulset/master                      create Pod master-2 in StatefulSet master successful
69m         Warning   FailedToUpdateEndpoint   endpoints/master                        Failed to update endpoint osde-ns/master: Operation cannot be fulfilled on endpoints "master": the object has been modified; please apply your changes to the latest version and try again
76m         Normal    ConfigModified           configmap/master                        ConfigMap modified
71m         Normal    Created                  pod/worker-1                            Created container exp
71m         Normal    Started                  pod/worker-1                            Started container exp
71m         Normal    Pulled                   pod/worker-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
71m         Normal    Created                  pod/worker-1                            Created container events
71m         Normal    Started                  pod/worker-1                            Started container events
46m         Warning   NodeNotReady             pod/worker-1                            Node is not ready
26m         Normal    SandboxChanged           pod/worker-1                            Pod sandbox changed, it will be killed and re-created.
26m         Normal    Pulled                   pod/worker-1                            Container image "busybox" already present on machine
26m         Normal    Created                  pod/worker-1                            Created container configure-sysctl
26m         Normal    Started                  pod/worker-1                            Started container configure-sysctl
26m         Normal    Pulled                   pod/worker-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
26m         Normal    Created                  pod/worker-1                            Created container exp
26m         Normal    Started                  pod/worker-1                            Started container exp
26m         Normal    Pulled                   pod/worker-1                            Container image "enonic/xp:7.11.1-ubuntu" already present on machine
26m         Normal    Created                  pod/worker-1                            Created container events
26m         Normal    Started                  pod/worker-1                            Started container events
22m         Warning   NodeNotReady             pod/worker-1                            Node is not ready
71m         Normal    ConfigReload             pod/worker-1                            Pod worker-1 reloaded ConfigMap worker
26m         Normal    ConfigReload             pod/worker-1                            Pod worker-1 reloaded ConfigMap worker
76m         Normal    SuccessfulCreate         statefulset/worker                      create Claim deploy-worker-0 Pod worker-0 in StatefulSet worker success
76m         Normal    SuccessfulCreate         statefulset/worker                      create Claim index-worker-0 Pod worker-0 in StatefulSet worker success
76m         Normal    SuccessfulCreate         statefulset/worker                      create Pod worker-0 in StatefulSet worker successful
76m         Normal    SuccessfulCreate         statefulset/worker                      create Claim index-worker-1 Pod worker-1 in StatefulSet worker success
76m         Normal    SuccessfulCreate         statefulset/worker                      create Claim deploy-worker-1 Pod worker-1 in StatefulSet worker success
76m         Normal    SuccessfulCreate         statefulset/worker                      create Pod worker-1 in StatefulSet worker successful
76m         Normal    ConfigModified           configmap/worker                        ConfigMap modified
76m         Normal    ConfigModified           configmap/worker                        ConfigMap modified
  1. Google GKE
    I created a K8-Cluster using Autopilot and storage classes standard and standard-rwx
    Deployment stocks at creating PVC (Waiting for waiting for first consumer to be created before binding) and xp7Deployment is waiting for pods and I guess pods are waiting for PV (but they are not visible)
    @vbr You wrote that you were able to run my config on GKE. How did you configure you K8 there ?

Additional questions:
What else can I do, to get this working?
What is the latest K8-Version the XP-Operator was tested with ?

Hi!
I use an nfs-provisioner to run it in GKE and all others tests, you can read about it here. It should create a storage class for shared disks:

image

And then you can create PVC with this new class:

I’m not sure what is background for standard-rwx class in Autopilot mode, but as far as I remember standard mode supports read-write-many volumes only through Filestore CSI driver and I haven’t tested it. (And I assume that is what Autopilot uses as well).

So you can try to use nfs-provisioner and use custom storage class. Just a few reasons:

  1. Filestore HDD minimum size is 1TBi, with the provider you can use any disk size supported by GKE.
  2. Your pvc config doesn’t depend on the cloud provider.

Current k8 version is 1.26.5-gke.2100

Thank for your hints @vbr
Unforntunately this does not work in my case.

I tried to deply the nfs-provisiones with the following command.

helm upgrade --install \
    --namespace kube-system \
    --set persistence.enabled=true \
    --set persistence.size=1Gi \
    --set storageClass.name=nfs \
    --set storageClass.allowVolumeExpansion=true \
    --version 1.1.3 \
    nfs-server \
    stable/nfs-server-provisioner

and get this response

Release "nfs-server" does not exist. Installing it now.
WARNING: This chart is deprecated
Error: create: failed to create: secrets is forbidden: User "xxx" cannot create resource "secrets" in API group "" in the namespace "kube-system": GKE Warden authz [denied by managed-namespaces-limitation]: the namespace "kube-system" is managed and the request's verb "create" is denied

So I changed the namespace to default with the following response:

Release "nfs-server" does not exist. Installing it now.
WARNING: This chart is deprecated
W0731 21:05:13.689870   33041 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet default/nfs-server-nfs-server-provisioner: defaulted unspecified resources for containers [nfs-server-provisioner] (see http://g.co/gke/autopilot-defaults)
Error: admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-default-linux-capabilities]":["linux capability 'DAC_READ_SEARCH,SYS_RESOURCE' on container 'nfs-server-provisioner' not allowed; Autopilot only allows the capabilities: 'AUDIT_WRITE,CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,MKNOD,NET_BIND_SERVICE,NET_RAW,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT,SYS_PTRACE'."]}

Nevertheless the storage class is there now, but it does not work. PVC stucks in pending mode.
So I guess additional configuration is necessary to get this working.

Do you use GKE in Autopilot or Standard -Mode ?

Update: I managed to get the shared storage by creating an (NFS) Filestore in Google Cloud and deploying an NFS-Client-Provisioner - but I do not see the PVCs for the standard storage!!!

Update: I get the following error when deploying the xp7deployment

2023-08-01 11:34:14,504 ERROR io.fab.kub.cli.inf.imp.cac.SharedProcessor - enonic.cloud/v1/xp7deployments failed invoking com.enonic.kubernetes.operator.xp7deployment.OperatorXp7DeploymentHelm@f
b40d01 event handler: Failure executing: POST at: https://10.68.0.1:443/apis/apps/v1/namespaces/osde-ns/statefulsets. Message: admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-disallow-privilege]":["container configure-sysctl is privileged; not allowed in Autopilot"]}
Requested by user: 'system:serviceaccount:default:xp-operator', groups: 'system:serviceaccounts,system:serviceaccounts:default,system:authenticated'.. Received status: Status(apiVersion=v1, code
=400, details=null, kind=Status, message=admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.

so @vbr How did you get this running ?

Hi!
I use Standard Mode to run my environment. According to the error messages it looks like Autopilot doesn’t allow to create custom nfs-provisioner container. So I would say Filestore + CSI driver is the only way to get NFS storage class in Autopilot mode (without installing “stable/nfs-server-provisioner”). But I haven’t tried it, so it’s just my assumption :slight_smile:

I’ve tried to set up the Autopilot cluster and Filestore CSI driver enabled by default there and cannot be disabled. There is a list of storage classes:

So I can see both standard RWO and filestore’s RWX classes and it should be enough to specify it during xp-operator installation in values.xml file.

@vbr Unfortunately not, because as you can see in one of my recent posts I am able to mount NFS -Storage using this client-provisioner and a Google Cloud Filestore, which I created seperately.

The point now is that the standard storage is not mounted and this is because the security-rules of the autopilot do not allow to execute configure-sysctl what you can see in the log-snippet I attached in my last post.

So I guess Autopilot does not work with XP-Operator yet and I have to use Standard Mode!

@tsi Can you check/validate this with your DEVOPS-Experts ?

Hello!

  1. configure-sysctl is responsible for configuring Elasticsearch-related virtual memory settings. Unfortunately, it seems that Autopilot doesn’t allow to set it, causing xp-operator to be incompatible with GKE Autopilot at the moment. But it seems like we can work it around, so we’ll create an issue to address this.
  2. Alternatively configure-sysctl can be disabled by adding operator.charts.values.pods.sysctlInitContainer=false to the values.yaml file. Although this will allow XP to start and run, please note that it’s intended for testing purposes only, as it may lead to out-of-memory exceptions, especially when dealing with large datasets.

I’ve tested this setup on Autopilot + Filestore and managed to get it up and running successfully.

2 Likes