Cloud Build Bazel Error: "kubectl toolchain was not properly configured so apply cannot be executed"

Related searches

I am trying to use rules_k8s for Bazel to deploy to my Kubernetes cluster.

Thus I have this cloudbuild.yaml file, which is executed by Google Cloud Build:

steps:
  - name: gcr.io/cloud-builders/bazel
    args: ['run', '//:kubernetes.apply']

(//:kubernetes is just a k8s_objects)


On my local machine running bazel run //:kubernetes.apply works fine, but although the Google Cloud Build succeeds, it logs those errors. So the configuration is not applied to my Kubernetes cluster:

Target //:kubernetes.apply up-to-date:
  bazel-bin/kubernetes.apply
INFO: Elapsed time: 29.863s, Critical Path: 0.14s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/kubernetes.apply
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 1 total action
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.

I also get a warning from the bazel cache:

DEBUG: /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/io_bazel_rules_k8s/toolchains/kubectl/kubectl_toolchain.bzl:28:9: No kubectl tool was found or built, executing run for rules_k8s targets might not work.

P.S.: I get the same errors when using //:kubernetes.create

My setup

Deployments

load("@io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")
k8s_object(
  name = "k8s_deployment",
  kind = "deployment",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.deployment.yaml",
  images = {
    "gcr.io/cents-ideas/ideas:latest": ":image"
  },
)

Services

k8s_object(
  name = "k8s_service",
  kind = "service",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.service.yaml",
)

Aggregations

load("@io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
k8s_objects(
    name = "k8s",
    objects = [
      ":k8s_deployment",
      ":k8s_service",
    ]
)

Final composition

k8s_objects(
    name = "kubernetes",
    objects = [
        "//services/ideas:k8s",
        # ...
    ]
)
Update

I've now tried to make my own docker image with Bazel and kubectl:

FROM gcr.io/cloud-builders/bazel

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

I pushed it to GCR and changed my cloudbuild.yaml to:

steps:
  - name: eu.gcr.io/cents-ideas/bazel-kubectl
    args: ["run", "//:kubernetes.apply"]

I firstly noticed, that the step took way longer than before. However at the end it throws an error:

$ /usr/local/bin/kubectl --kubeconfig= --cluster=gke_cents-ideas_europe-west3-a_cents-ideas --context= --user= apply -f -
error: cluster "gke_cents-ideas_europe-west3-a_cents-ideas" does not exist

Here is the full log.

As for the updated question, now you need to authenticate somehow to GKE inside the container.

First thing, I recommend installing gcloud tool to your container. Btw, as for the huge container size 1.2 GB, that's because cloud-builders/bazel is huge :)

Have a look at our example on slim bazel container version: https://github.com/aspect-development/bazel-k8s-example/blob/master/tools/Dockerfile.dazel

And here is Dockerfile for installing gcloud and kubectl, so you can grab needed parts from both files: https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile

The second thing is authenticating, after gcloud is installed it should be easy. Overall cloudbuild step should look similar to this:

- name: <link to your container>
  entrypoint: /bin/sh
  args:
  - -c
  - |
    gcloud container clusters get-credentials cents-ideas --zone europe-west3-a --project cents-ideas
    bazel run //:kubernetes.apply

bazel builder is broken � Issue #53 � GoogleCloudPlatform/cloud , When I run gcloud container builds submit --config=cloudbuild.yaml . with the bazel builder, it fails in Step #2 with this error (build label: 0.4.5):� I recently started using Cloud Build with Bazel. So I have a basic cloudbuild.yaml steps: - id: 'run unit tests' name: gcr.io/cloud-builders/bazel args: ['test

It complains that it can't find kubectl on your machine. If GKE is used, also the gcloud sdk needs to be installed.

And also check that authentication is configured for those tools: kubectl auth, GKE auth.

Cloud Build Bazel Error: "kubectl toolchain was not properly , As for the updated question, now you need to authenticate somehow to GKE inside the container. First thing, I recommend installing gcloud tool� Another related issue is that we cannot mount Persistent Volumes and therefore make use of local cache. Platforms like Bazel are heavily affected by this. GCB has a bad IAM permissions management. IAM permissions are a powerful tool provided by the Google Cloud Platform, unfortunately GCB does not make good use of them (yet).

The main issue is that kubectl doesn't come with the gcr.io/cloud-builders/bazel docker image. That is why it can't be found.

There are however, efforts to implement the kubectl toolchain into Bazel manually: https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl#kubectl-toolchain. But as those features are currently in experimental status, they often don't work.

More info can be found in this issue: https://github.com/bazelbuild/rules_k8s/issues/512

For now, your best bet will be to install kubectl in the container.

Cloud builders | Cloud Build Documentation, bazel, gcr.io/cloud-builders/bazel, bazel example � docker, gcr.io/cloud-builders/ docker, docker example � git, gcr.io/cloud-builders/git� This is happening with as few as a dozen simultaneous requests, with a single client on a single machine. I basically haven't been able to build bazel on my machine with the remote worker if I change jobs to a bit more than I have cores in my machine.

Build configuration overview, The following snippet shows build steps calling the bazel , gcloud to specify the behavior when there is an error in the substitution checks. If you don't have it installed, you'll see an error when running make create on OS X. You'll see scripts/create.sh automatically detects if you're on OS X, and runs Planter to help you execute your Bazel build commands in a linux container, so they'll run on the GKE linux container. Troubleshooting

Experimenting with Google Cloud Build and Bazel, Experimenting with Google Cloud Build and Bazel. I'm Christian Roggia, a Backend Engineer at Engel & V�lkers Technology. At our company choosing the best� Build steps are analogous to commands in a script and provide you with the flexibility of executing arbitrary instructions in your build. If you can package a build tool into a container, Cloud Build can execute it as part of your build. By default, Cloud Build executes all steps of a build serially on the same machine.

The Cloud Build developer community provides open-source builders that you can use to execute your tasks. Pre-built images are not available for these builders; to use these builders, download the source code from the cloud builders community GitHub repository and then build the image.

Comments
  • Sounds good! I will try it. But what would be the FROM in my Dockerfile? debian:9.11-slim or rather gcloud-slim?
  • And I will probably don't need all the gcloud components, or do I?
  • I think you can use gcloud-slim. The result will be just a litter bigger than if using debian-slim, but prob more convenient. Dockerfile of gcloud-slim is right there in the same repo: github.com/GoogleCloudPlatform/cloud-builders/blob/master/… And yes, you don't need all gcloud components, it's just an example.
  • I get an error: /bin/sh: 0: Can't open gcloud container clusters get-credentials cents-ideas --zone=europe-west3-a --project=cents-ideas bazel run //:kubernetes.apply with this Dockerfile: github.com/flolude/cents-ideas/blob/feature/…
  • Try the updated version (I forgot -c flag) and let's see how it goes. Actually I didn't check this setup.
  • I think there is a misunderstanding. It works fine on my local machine. But it does not work in the Google Cloud Build :)
  • Ahh, right. So, the gcr.io/cloud-builders/bazel container doesn't contain kubectl tool that's why it doesn't work. You may need to build your own docker image that contains both kubectl and bazel. There is also an attempt from rules_k8s authors to have kubectl as a toolchain if it's not found in the environment: github.com/bazelbuild/rules_k8s/tree/master/toolchains/…. But when I tried it didn't work for me; not surprising: they state toolchain features as experimental.
  • building a custom docker image is something i would like to avoid, which of those three methods didn't work for you? (github.com/bazelbuild/rules_k8s/tree/master/toolchains/…)
  • I've now build a custom docker image and it does not find the cluster (see updated question)
  • You should add the cluster as shown in this command `kubectl config current-context' which will print the correct cluster that you need.