Kubernetes Pod Warning: 1 node(s) had volume node affinity conflict

volume node affinity conflict gke
kubernetes volume node affinity
persistent volume node affinity
1 node(s) had taints that the pod didn't tolerate
kubernetes node affinity
kubernetes one pod per node
kubernetes statefulset node affinity
kubernetes spread pods across nodes

I try to set up kubernetes cluster. I have Persistent Volomue, Persistent Volume Claim and Storage class all set-up and running but when I wan to create pod from deployment, pod is created but it hangs in Pending state. After describe I get only this warnig "1 node(s) had volume node affinity conflict." Can somebody tell me what I am missing in my volume configuration?

apiVersion: v1
kind: PersistentVolume
metadata:
  creationTimestamp: null
  labels:
    io.kompose.service: mariadb-pv0
  name: mariadb-pv0
spec:
  volumeMode: Filesystem
  storageClassName: local-storage
  local:
    path: "/home/gtcontainer/applications/data/db/mariadb"
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    namespace: default
    name: mariadb-claim0
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/cvl-gtv-42.corp.globaltelemetrics.eu
            operator: In
            values:
            - master

status: {}

The error "volume node affinity conflict" happens when the persistent volume claims that the pod is using are scheduled on different zones, rather than on one zone, and so the actual pod was not able to be scheduled because it cannot connect to the volume from another zone. To check this, you can see the details of all the Persistent Volumes. To check that, first get your PVCs:

$ kubectl get pvc -n <namespace>

Then get the details of the Persistent Volumes (not Volume claims)

$  kubectl get pv

Find the PVs, that correspond to your PVCs and describe them

$  kubectl describe pv <pv1> <pv2>

You can check the Source.VolumeID for each of the PV, most likely they will be different availability zone, and so your pod gives the affinity error. To fix this, create a storageclass for a single zone and use that storageclass in your PVC.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: region1storageclass
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  encrypted: "true" # if encryption required
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - eu-west-2b # this is the availability zone, will depend on your cloud provider
    # multi-az can be added, but that defeats the purpose in our scenario

0/x nodes are available: 1 node(s) had volume node affinity conflict , GitHub is home to over 50 million developers working together to host that the pod didn't tolerate, 1 node(s) had volume node affinity conflict. You're requesting a volume in node4 but asking to schedule the pod in node2, so Kubernetes won't ever be able to bind them together. 👍 1 This comment has been minimized.

There a few things that can cause this error:

  1. Node isn’t labeled properly. I had this issue on AWS when my worker node didn’t have appropriate labels(master had them though) like that:

    failure-domain.beta.kubernetes.io/region=us-east-2

    failure-domain.beta.kubernetes.io/zone=us-east-2c

    After patching the node with the labels, the "1 node(s) had volume node affinity conflict" error was gone, so PV, PVC with a pod were deployed successfully. The value of these labels is cloud provider specific. Basically, it is the job of the cloud provider(with —cloud-provider option defined in cube-controller, API-server, kubelet) to set those labels. If appropriate labels aren’t set, then check that your CloudProvider integration is correct. I used kubeadm, so it is cumbersome to set up but with other tools, kops, for instance, it is working right away.

  2. Based on your PV definition and the usage of nodeAffinity field, you are trying to use a local volume, (read here local volume description link, official docs), then make sure that you set "NodeAffinity field" like that(it worked in my case on AWS):

    nodeAffinity:

         required:
          nodeSelectorTerms:
           - matchExpressions:
             - key: kubernetes.io/hostname
               operator: In
               values:
               - my-node  # it must be the name of your node(kubectl get nodes)
    

So that after creating the resource and running describe on it it will show up there like that:

         Required Terms:  
                    Term 0:  kubernetes.io/hostname in [your node name]
  1. StorageClass definition(named local-storage, which is not posted here) must be created with volumeBindingMode set to WaitForFirstConsumer for local storage to work properly. Refer to the example here storage class local description, official doc to understand the reason behind that.

Kubernetes is not scaling up when volume node affinity requires a , Warning FailedScheduling 37s default-scheduler 0/5 nodes are available: 2 node(s) had volume node affinity conflict, 3 node(s) had taints that the pod didn't tolerate. -domain.beta.kubernetes.io/zone=centralus-1,kubernetes.io/arch=​amd64  This most likely came from the kubernetes pod, if you could send over the pod logs they might tell us if there was a sigterm sent. The command should be kubectl logs <MY PODS NAME HERE> Node 1 has by far the most restarts, but I’d like to see the logs for all the pods.

1731059 – Pod with persistent volumes failed scheduling on Azure , I have deployed cockroach db on kubernetes with 3 nodes on gcp. 1 node(s) had taints that the pod didn't tolerate, 2 node(s) had volume node affinity conflict. " Also one node in which cockroachdb-0 pod is running is also  Kubernetes is not scaling up when volume node affinity requires a node in specific AZ (AWS) #75402 fcappi opened this issue Mar 15, 2019 · 7 comments Labels

almost same problem described here... https://github.com/kubernetes/kubernetes/issues/61620

"If you're using local volumes, and the node crashes, your pod cannot be rescheduled to a different node. It must be scheduled to the same node. That is the caveat of using local storage, your Pod becomes bound forever to one specific node."

Volume node affinity conflict kubernetes gcp - General, Key: kubernetes.io/hostname. Operator: in list. Values: my-node-1 Operator: in list. Values: my-node-1. Of course my pod is not coming online because of this event: Warning FailedScheduling 0/2 nodes are available: 1 node(s) didn't match node selector, 1 node(s) had volume node affinity conflict. 1 node(s) had taints that the pod didn't tolerate, 1 node(s) had volume node affinity conflict. You can see the taints by running kubectl describe node <node name> | grep Taint . I'll try to repro on EKS myself but that will take a bit longer.

In my case, the root cause was that the persistent volume are in us-west-2c and the new worker nodes are relaunched to be in us-west-2a and us-west-2b. The solution is to either have more worker nodes so they are in more zones, or remove / widen node affinity for the application so that more worker nodes qualifies to be bounded to the persistent volume.

Local persistent volume, Warning FailedScheduling 13m (x88 over 142m) default-scheduler 0/1 nodes are available: 1 node(s) had volume node affinity conflict. The Kubernetes scheduler does not handle zone constraints when using dynamic The scheduler will then ensure that pods that claim a given volume are only placed  k8s - Cinder “0/x nodes are available: x node(s) had volume node affinity conflict” 1 Kubernetes is faling to schedule Daemonset pods on nodes in an auto scaling GKE node pool

Autoscaling issue when provisioning Masters in Multi AZ Environment, Node affinity is a set of rules used by the scheduler to determine where a pod can be placed. The node does not have control over the placement. then the pod can be scheduled onto a node if one of the nodeSelectorTerms is satisfied. 33s 8 default-scheduler Warning FailedScheduling No nodes are available that  Warning FailedScheduling 78s (x31 over 40m) default-scheduler 0/5 nodes are available: 1 node(s) had taints that the pod didn't tolerate, 4 node(s) had volume node affinity conflict and my node and ebs volume are in same aws zone. My nodes are in pending status.

Advanced Scheduling and Node Affinity, Using this Kubernetes feature we can create nodes that are reserved (dedicated) for specific pods. kube-system aws-node-m4crc 1/1 Running 0 1h 192.168.​96.47 volumes.kubernetes.io/controller-managed-attach-detach=true Pod anti-affinity requires topologyKey to be set and all pods to have  With node affinity we can tell Kubernetes which nodes to schedule to a pod using the labels on each node. Let's see how node affinity works 🔗︎. Since node affinity identifies the nodes on which to place pods via labels, we first need to add a label to our node.

Taints and tolerations, pod and node affinities demystified · Banzai , 1 node(s) had taints that the pod didn't tolerate, 1 node(s) had volume node affinity conflict. See https://kubernetes.io/docs/concepts/configuration/manage-​compute- /kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict. After all the volume had bound, then assume the pod; But the issue is, there maybe many pods with anti-affinity rules are assume volumes, and waiting for the volume to be bound. At this time, the pod is not considered with a anti-affinity pod, so they may choose the same rack's pv and then their pvc will bound the the pv.

Comments
  • what a useful answer. thank you @sownak-roy
  • Is there a way so that the volume of a different zone gets connected to a pod scheduled on a node on a different zone.?
  • Would setting volumeBindingMode: WaitForFirstConsumer on the default StorageClass suffice?
  • That started happening with a deployment that has been working for more than one month on AWS. I was unable to change PVC storage class so I added a --force param in kubectl apply. It destroyed my volume.
  • @Sid No, that would not be enough
  • Thanks for suggestion (2) after a recent upgrade/reinstall on my windows machine the default name of my Kubernetes node had quietly changed from "docker-for-desktop" to "docker-desktop"