1. The Kubernetes API

components of kubernetes

The core of Kubernetes' control plane is the API server. The API server exposes an HTTP API that lets end users, different parts of your cluster, and external components communicate with one another.

The Kubernetes API lets you query and manipulate the state of API objects in Kubernetes (for example: Pods, Namespaces, ConfigMaps, and Events).

Most operations can be performed through the kubectl command-line interface or other command-line tools, such as kubeadm, which in turn use the API. However, you can also access the API directly using REST calls.

1.1. API Groups and Versioning

To make it easier to eliminate fields or restructure resource representations, Kubernetes supports multiple API versions, each at a different API path, such as /api/v1 or /apis/batch/v1.

API groups make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the apiVersion field of a serialized object.

  • The core (also called legacy) group is found at REST path /api/v1.

    The core group is not specified as part of the apiVersion field, for example, apiVersion: v1.

  • The named groups are at REST path /apis/$GROUP_NAME/$VERSION and use apiVersion: $GROUP_NAME/$VERSION (for example, apiVersion: batch/v1).

You can find the full list of supported API groups in Kubernetes API reference.

API resources are distinguished by their API group, resource type, namespace (for namespaced resources), and name.

  • The API server handles the conversion between API versions transparently: all the different versions are actually representations of the same persisted data.

  • Kubernetes stores the serialized state of objects by writing them into etcd.

  • The API server may serve the same underlying data through multiple API versions.

The Kubernetes API can be extended in one of two ways:

  • Custom resources let you declaratively define how the API server should provide your chosen resource API.

  • You can also extend the Kubernetes API by implementing an aggregation layer.

2. Using kubectl Command Line Tools

The kubectl command line tool lets you control Kubernetes clusters. For configuration, kubectl looks for a file named config in the $HOME/.kube directory. You can specify other kubeconfig files by setting the KUBECONFIG environment variable or by setting the --kubeconfig flag.

By default kubectl will first determine if it is running within a pod, and thus in a cluster. It starts by checking for the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables and the existence of a service account token file at /var/run/secrets/kubernetes.io/serviceaccount/token. If all three are found in-cluster authentication is assumed.

$ kubectl exec -n default devnetools -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=devnetools
KUBERNETES_PORT=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443

$ kubectl cp -n default $(which kubectl) devnetools:tmp

$ kubectl exec -n default -it devnetools -- /tmp/kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list resource "services" in API group "" in the namespace "kube-
system"
command terminated with exit code 1

$ kubectl create clusterrolebinding default:default:view --clusterrole=view --serviceaccount=default:default
clusterrolebinding.rbac.authorization.k8s.io/default:default:view created

$ kubectl exec -n default -it devnetools -- /tmp/kubectl cluster-info
Kubernetes master is running at https://172.20.0.1:443
CoreDNS is running at https://172.20.0.1:443/api/v1/namespaces/kube-system/services/coredns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl delete clusterrolebindings.rbac.authorization.k8s.io default:default:view
clusterrolebinding.rbac.authorization.k8s.io "default:default:view" deleted

Many of the examples provide an introduction to using kubectl and complete documentation is found in the kubectl manual.

3. Using the REST API

Kubectl handles locating and authenticating to the apiserver. If you want to directly access the REST API with an http client like curl or wget, or a browser, there are several ways to locate and authenticate:

  • Run kubectl in proxy mode.

    • Recommended approach.

      • Uses stored apiserver location.

      • Verifies identity of apiserver using self-signed cert. No MITM possible.

      • Authenticates to apiserver.

      • In future, may do intelligent client-side load-balancing and failover.

  • Provide the location and credentials directly to the http client.

    • Alternate approach.

    • Works with some types of client code that are confused by using a proxy.

    • Need to import a root cert into your browser to protect against MITM.

3.1. Using kubectl proxy

The following command runs kubectl in a mode where it acts as a reverse proxy. It handles locating the apiserver and authenticating.

Run it like this:

$ kubectl proxy --port 8080 --address [::1]
Starting to serve on [::1]:8080

Open another terminal:

$ curl -i6  http://localhost:8080/api/
HTTP/1.1 200 OK
Cache-Control: no-cache, private
Content-Length: 186
Content-Type: application/json
Date: Thu, 11 Nov 2021 05:55:19 GMT

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "104.197.5.247:6443"
    }
  ]
}

See kubectl proxy for more details.

3.2. Without kubectl proxy

The following command uses service account token to access the API.

sa-token.sh
#!/bin/bash
set -e

server=$(kubectl config view -ojsonpath="{.clusters[*].cluster.server}")
prd-k8s@kube-admin:~/proxy$ cat securce-sa-token.sh
#!/bin/bash
set -e

server=$(kubectl config view -ojsonpath="{.clusters[*].cluster.server}")

token=$(kubectl \
    get secrets -n default \
    $(kubectl get sa -n default \
        default -ojsonpath="{.secrets[].name}") \
    -ojsonpath="{.data.token}" \
    | base64 -d)

# With `--insecure` flag, this leaves it subject to MITM attacks.
# curl --include --insecure $server/api/ -H "Authorization: Bearer $token"

curl --include \
     --cacert <(kubectl config view \
                --raw \
                -ojsonpath="{.clusters[].cluster.certificate-authority-data}" \
               | base64 -d) \
    $server/api/ -H "Authorization: Bearer $token"
$ bash sa-token.sh
HTTP/2 200
cache-control: no-cache, private
content-type: application/json
content-length: 184
date: Sun, 19 Dec 2021 12:32:26 GMT

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.24.128.43:5444"
    }
  ]
}

4. Programmatic access to the API

Kubernetes officially supports Go and Python client libraries.

  • To get the go client library, run the following command: go get k8s.io/client-go@kubernetes-<kubernetes-version-number>, see INSTALL.md for detailed installation instructions. See https://github.com/kubernetes/client-go to see which versions are supported.

  • Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., import "k8s.io/client-go/kubernetes" is correct.

The Go client can use the same kubeconfig file as the kubectl CLI does to locate and authenticate to the apiserver.

$ mkdir -p github.com/samples/gocli

$ cd github.com/samples/gocli/

$ go mod init github.com/samples/gocli
go: creating new go.mod: module github.com/samples/gocli

$ cat > main.go <<EOF
package main

import (
    "context"
    "flag"
    "fmt"
    "os"
    "path/filepath"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/homedir"
    "k8s.io/klog/v2"
)

func main() {
    var kubeconfig *string
    if home := homedir.HomeDir(); home != "" {
        kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
    } else {
        kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
    }
    flag.Parse()

    // try to create the in-cluster config
    config, err := rest.InClusterConfig()
    if err != nil {
        // use the current context in kubeconfig
        config, err = clientcmd.BuildConfigFromFlags("", *kubeconfig)
        if err != nil {
            klog.Error(err)
            os.Exit(1)
        }
    }

    // creates the clientset
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        klog.Error(err)
        os.Exit(1)
    }

    pods, err := clientset.CoreV1().Pods("").List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        klog.Error(err)
        os.Exit(1)
    }
    fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
}
EOF

$ go mod tidy
go: finding module for package k8s.io/client-go/kubernetes
go: downloading k8s.io/client-go v0.23.1
go: finding module for package k8s.io/client-go/rest
go: finding module for package k8s.io/client-go/tools/clientcmd
go: finding module for package k8s.io/client-go/util/homedir
go: finding module for package k8s.io/klog/v2
go: downloading k8s.io/klog/v2 v2.40.1
...

$ go build

$ ./gocli
There are 138 pods in the cluster

5. Accessing the API from a Pod

When accessing the API from a pod, locating and authenticating to the apiserver are somewhat different.

  • The recommended way to locate the apiserver within the pod is with the kubernetes.default.svc DNS name, which resolves to a Service IP which in turn will be routed to an apiserver.

  • The recommended way to authenticate to the apiserver is with a service account credential.

    • By kube-system, a pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

    • If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the apiserver.

    • Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

From within a pod the recommended ways to connect to API are:

  • Run kubectl proxy in a sidecar container in the pod, or as a background process within the container.

    This proxies the Kubernetes API to the localhost interface of the pod, so that other processes in any container of the pod can access it.

  • Use the Go client library, and create a client using the rest.InClusterConfig() and kubernetes.NewForConfig() functions.

    They handle locating and authenticating to the apiserver.

    package main
    
    import (
    	"k8s.io/client-go/kubernetes"
    	"k8s.io/client-go/rest"
    )
    
    func main() {
    	// creates the in-cluster config
    	config, err := rest.InClusterConfig()
    	if err != nil {
    		panic(err.Error())
    	}
    	// creates the clientset
    	clientset, err := kubernetes.NewForConfig(config)
    	if err != nil {
    		panic(err.Error())
    	}
    	_ = clientset
    }

In each case, the credentials of the pod are used to communicate securely with the apiserver.

6. Accessing services running on the cluster

In Kubernetes, the nodes, pods and services all have their own IPs. In many cases, the node IPs, pod IPs, and some service IPs on a cluster will not be routable, so they will not be reachable from a machine outside the cluster, such as your desktop machine.

6.1. Ways to connect

You have several options for connecting to nodes, pods and services from outside the cluster:

  • Access services through public IPs.

    • Use a service with type NodePort or LoadBalancer to make the service reachable outside the cluster.

    • Depending on your cluster environment, this may only expose the service to your corporate network, or it may expose it to the internet. Think about whether the service being exposed is secure. Does it do its own authentication?

    • Place pods behind services. To access one specific pod from a set of replicas, such as for debugging, place a unique label on the pod and create a new service which selects this label.

    • In most cases, it should not be necessary for application developer to directly access nodes via their nodeIPs.

  • Access services, nodes, or pods using the Proxy Verb.

    • Does apiserver authentication and authorization prior to accessing the remote service. Use this if the services are not secure enough to expose to the internet, or to gain access to ports on the node IP, or for debugging.

    • Proxies may cause problems for some web applications.

    • Only works for HTTP/HTTPS.

  • Access from a node or pod in the cluster.

    • Run a pod, and then connect to a shell in it using kubectl exec. Connect to other nodes, pods, and services from that shell.

    • Some clusters may allow you to ssh to a node in the cluster. From there you may be able to access cluster services. This is a non-standard method, and will work on some clusters but not others. Browsers and other tools may or may not be installed. Cluster DNS may not work.

6.2. Discovering builtin services

Typically, there are several services which are started on a cluster by kube-system. Get a list of these with the kubectl cluster-info command:

kubectl cluster-info

The output is similar to this:

Kubernetes control plane is running at https://104.197.5.247:6443
CoreDNS is running at https://104.197.5.247:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
#$ kubectl get svc -n kube-system kube-dns -oyaml
apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
...

This shows the proxy-verb URL for accessing each service.

To create proxy URLs that include service endpoints, suffixes, and parameters, you append to the service’s proxy URL:

http://api-server_address/api/v1/namespaces/namespace_name/services/service_name[:port_name]/proxy

If you haven’t specified a name for your port, you don’t have to specify port_name in the URL.

http://api-server_address/api/v1/namespaces/namespace_name/services/service_name[:port_num]/proxy

You can also use the port number in place of the port_name for both named and unnamed ports.

By default, the API server proxies to your service using http. To use https, prefix the service name with https:

http://api-server_address_/api/v1/namespaces/namespace_name/services/https:service_name:[port_name]/proxy

The supported formats for the name segment of the URL are:

  • <service_name> - proxies to the default or unnamed port using http

  • <service_name>:<port_name> - proxies to the specified port name or port number using http

  • https:<service_name>: - proxies to the default or unnamed port using https (note the trailing colon)

  • https:<service_name>:<port_name> - proxies to the specified port name or port number using https

Examples

$ kubectl create -n default deployment echo --image=k8s.gcr.io/echoserver:1.10
deployment.apps/echo created

$ kubectl expose -n default deployment echo --port 80 --target-port 8080
service/echo exposed

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

$ curl http://localhost:8001/api/v1/namespaces/default/services/echo/proxy/


Hostname: echo

Pod Information:
	-no pod information available-

Server values:
	server_version=nginx: 1.13.3 - lua: 10008

Request Information:
	client_address=172.25.0.1
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://localhost:8080/

Request Headers:
	accept=*/*
	accept-encoding=gzip
	host=localhost:8001
	user-agent=curl/7.74.0
	x-forwarded-for=127.0.0.1, 10.24.128.43
	x-forwarded-uri=/api/v1/namespaces/default/services/echo/proxy/

Request Body:
	-no body in request-