1. The Kubernetes API

The core of Kubernetes' control plane is the API server. The API server exposes an HTTP API that lets end users, different parts of your cluster, and external components communicate with one another. [1]

Components of Kubernetes

Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run) terminates at the API server. None of the other control plane components are designed to expose remote services. The API server is configured to listen for remote connections on a secure HTTPS port (typically 443) with one or more forms of client authentication enabled. One or more forms of authorization should be enabled, especially if anonymous requests or service account tokens are allowed. [2]

Most operations can be performed through the kubectl command-line interface or other command-line tools, such as kubeadm, which in turn use the API. However, you can also access the API directly using REST calls.

1.1. API Groups and Versioning

To make it easier to eliminate fields or restructure resource representations, Kubernetes supports multiple API versions, each at a different API path, such as /api/v1 or /apis/batch/v1. [1]

API groups make it easier to extend the Kubernetes API. The API group is specified in a REST path and in the apiVersion field of a serialized object.

  • The core (also called legacy) group is found at REST path /api/v1.

    The core group is not specified as part of the apiVersion field, for example, apiVersion: v1.

  • The named groups are at REST path /apis/$GROUP_NAME/$VERSION and use apiVersion: $GROUP_NAME/$VERSION (for example, apiVersion: batch/v1).

The Kubernetes API can be extended in one of two ways: [1]

  • Custom resources let you declaratively define how the API server should provide your chosen resource API.

  • You can also extend the Kubernetes API by implementing an aggregation layer.

1.2. OpenAPI

The Kubernetes API server serves an aggregated OpenAPI v2 spec via the /openapi/v2 endpoint, and supports publishing a description of its APIs as OpenAPI v3, with a discovery endpoint /openapi/v3 to see a list of all group/versions available. [1]

A list of all group versions supported by a cluster is published at the /api and /apis endpoints. Each group version also advertises the list of resources supported via /apis/<group>/<version> (for example: /apis/rbac.authorization.k8s.io/v1). These endpoints are used by kubectl to fetch the list of resources supported by a cluster.

$ kubectl api-resources # Print the supported API resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
configmaps                        cm           v1                                     true         ConfigMap
endpoints                         ep           v1                                     true         Endpoints
$ kubectl proxy # Run a proxy to the Kubernetes API server.
Starting to serve on 127.0.0.1:8001
$ curl -s localhost:8001/openapi/v2 | jq | head -n 10
{
  "swagger": "2.0",
  "info": {
    "title": "Kubernetes",
    "version": "v1.26.0"
  },
  "paths": {
    "/.well-known/openid-configuration/": {
      "get": {
        "description": "get service account issuer OpenID configuration, also known as the 'OIDC discovery doc'",

$ curl -s localhost:8001/openapi/v3 | jq | head -n 10
{
  "paths": {
    ".well-known/openid-configuration": {
      "serverRelativeURL": "/openapi/v3/.well-known/openid-configuration?hash=41054813FD81725211A3B09D3C9FA87F2B041E99B64B1C4A6FD0AF072CEB1622726E80278D5F762B445839FD7F625B56622D56B68963DB550DFCCE30BE2C11F1"
    },
    "api": {
      "serverRelativeURL": "/openapi/v3/api?hash=AB5AC9C5AB05D854B4B4489A3DE2E019BB9EA07DF8CB7E0B79F0B938FC7F7E300960DDE723878E9435E9B13AF07C0CF135A3ABCC6D2FB8FFE5F980CB3BA84F08"
    },
    "api/v1": {
      "serverRelativeURL": "/openapi/v3/api/v1?hash=FB484AAC3A02DD7CC60312AB967AEE53CBD692C1D22332A846C17D94CD603392374A93C6427AA5CA5CBBC1DA1B4AF6ED2A28F933C295C6A9F46F79FA87B64A78"

$ curl -s localhost:8001/apis/rbac.authorization.k8s.io/v1 | head -n 10
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "rbac.authorization.k8s.io/v1",
  "resources": [
    {
      "name": "clusterrolebindings",
      "singularName": "",
      "namespaced": false,
      "kind": "ClusterRoleBinding",

2. Using kubectl Command Line Tools

Kubernetes provides kubectl, a command line tool, for communicating with a Kubernetes cluster’s control plane, using the Kubernetes API.

By default kubectl will first determine if it is running within a pod, and thus in a cluster. It starts by checking for the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables and the existence of a service account token file at /var/run/secrets/kubernetes.io/serviceaccount/token. If all three are found in-cluster authentication is assumed. [3] [6]

$ kubectl create deployment devnetools --image docker.io/qqbuby/net-tools:2.0 -- sleep 24h
deployment.apps/devnetools created
$ kubectl exec devnetools-847d89666-28psk -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=devnetools-847d89666-28psk
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=tcp://10.96.0.1:443
HOME=/root

// copy `kubectl` cmd into container
$ kubectl cp $(which kubectl) devnetools-847d89666-28psk:tmp

// show the cluster info using `kubectl` in pod with in-cluster mode
$ kubectl exec devnetools-847d89666-28psk -- /tmp/kubectl cluster-info
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:default" cannot list resource "services" in API group "" in the namespace "kube-system"

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
command terminated with exit code 1

// bind the cluster view role to the service account
$ kubectl create clusterrolebinding default:default:view --clusterrole=view --serviceaccount=default:default
clusterrolebinding.rbac.authorization.k8s.io/default:default:view created
$ kubectl exec devnetools-847d89666-28psk -- /tmp/kubectl cluster-info
Kubernetes control plane is running at https://10.96.0.1:443
CoreDNS is running at https://10.96.0.1:443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl delete clusterrolebindings.rbac.authorization.k8s.io default:default:view
clusterrolebinding.rbac.authorization.k8s.io "default:default:view" deleted

2.1. Directly accessing the REST API

Kubectl handles locating and authenticating to the apiserver. If you want to directly access the REST API with an http client like curl or wget, or a browser, there are several ways to locate and authenticate: [4]

2.1.1. Run kubectl in proxy mode.

The following command runs kubectl in a mode where it acts as a reverse proxy. It handles locating the apiserver and authenticating.

$ kubectl proxy --port 8080 --address [::1]
Starting to serve on [::1]:8080

Open another terminal:

$ curl localhost:8080/version
{
  "major": "1",
  "minor": "26",
  "gitVersion": "v1.26.0",
  "gitCommit": "b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d",
  "gitTreeState": "clean",
  "buildDate": "2022-12-08T19:51:45Z",
  "goVersion": "go1.19.4",
  "compiler": "gc",
  "platform": "linux/amd64"
}

2.1.2. Provide the location and credentials directly to the http client.

Use kubectl apply and kubectl describe secret…​ to create a token for the default service account with grep/cut:

First, create the Secret, requesting a token for the default ServiceAccount:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: default-token
  annotations:
    kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token
EOF

Next, wait for the token controller to populate the Secret with a token:

while ! kubectl describe secret default-token | grep -E '^token' >/dev/null; do
  echo "waiting for token..." >&2
  sleep 1
done

Capture and use the generated token:

APISERVER=$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")
TOKEN=$(kubectl describe secret default-token | grep -E '^token' | cut -f2 -d':' | tr -d " ")

curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure

The output is similar to this:

{
  "major": "1",
  "minor": "26",
  "gitVersion": "v1.26.0",
  "gitCommit": "b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d",
  "gitTreeState": "clean",
  "buildDate": "2022-12-08T19:51:45Z",
  "goVersion": "go1.19.4",
  "compiler": "gc",
  "platform": "linux/amd64"
}

3. Programmatic access to the API

Kubernetes officially supports Go and Python client libraries. [4]

  • To get the go client library, run the following command: go get k8s.io/client-go@latest, see INSTALL.md for detailed installation instructions. See Compatibility matrix to see which versions are supported.

  • Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g., import "k8s.io/client-go/kubernetes" is correct.

The Go client can use the same kubeconfig file as the kubectl CLI does to locate and authenticate to the apiserver.

When accessing the API from a pod, locating and authenticating to the apiserver are somewhat different. [5]

  • Using Official Client Libraries

    The easiest and recommended way to use the Kubernetes API from a Pod is to use one of the official client libraries. For a Go client, use the official Go client library. The rest.InClusterConfig() function handles API host discovery and authentication automatically.

  • Directly accessing the REST API

    While running in a Pod, your container can create an HTTPS URL for the Kubernetes API server by fetching the KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT_HTTPS environment variables. The API server’s in-cluster address is also published to a Service named kubernetes in the default namespace so that pods may reference kubernetes.default.svc as a DNS name for the local API server.

    The recommended way to authenticate to the API server is with a service account credential. By default, a Pod is associated with a service account, and a credential (token) for that service account is placed into the filesystem tree of each container in that Pod, at /var/run/secrets/kubernetes.io/serviceaccount/token.

    If available, a certificate bundle is placed into the filesystem tree of each container at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, and should be used to verify the serving certificate of the API server.

    Finally, the default namespace to be used for namespaced API operations is placed in a file at /var/run/secrets/kubernetes.io/serviceaccount/namespace in each container.

# create a go module.
mkdir -p github.com/samples/gocli
cd github.com/samples/gocli/
go mod init github.com/samples/gocli
// main.go
package main

import (
	"context"
	"flag"
	"fmt"
	"os"
	"path/filepath"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
	"k8s.io/klog/v2"
)

func main() {
	var kubeconfig *string
	if home := homedir.HomeDir(); home != "" {
		kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
	} else {
		kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
	}
	flag.Parse()

	// try to create the in-cluster config
	config, err := rest.InClusterConfig()
	if err != nil {
		// use the current context in kubeconfig
		config, err = clientcmd.BuildConfigFromFlags("", *kubeconfig)
		if err != nil {
			klog.Error(err)
			os.Exit(1)
		}
	}

	// creates the clientset
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		klog.Error(err)
		os.Exit(1)
	}

	pods, err := clientset.CoreV1().Pods("").List(context.TODO(), metav1.ListOptions{})
	if err != nil {
		klog.Error(err)
		os.Exit(1)
	}
	fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
}
$ go mod tidy
go: finding module for package k8s.io/klog/v2
go: finding module for package k8s.io/client-go/rest
go: finding module for package k8s.io/apimachinery/pkg/apis/meta/v1
go: finding module for package k8s.io/client-go/tools/clientcmd
go: finding module for package k8s.io/client-go/util/homedir
go: finding module for package k8s.io/client-go/kubernetes
go: downloading k8s.io/klog v1.0.0
go: downloading k8s.io/klog/v2 v2.120.1
go: downloading k8s.io/apimachinery v0.29.2
go: downloading k8s.io/client-go v0.29.2
...
$ go build
$ ./gocli
There are 138 pods in the cluster

4. Accessing services running on the cluster

In Kubernetes, the nodes, pods and services all have their own IPs. In many cases, the node IPs, pod IPs, and some service IPs on a cluster will not be routable, so they will not be reachable from a machine outside the cluster, such as your desktop machine. [7]

$ kubectl get no -owide
NAME     STATUS   ROLES           AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                 CONTAINER-RUNTIME
node-0   Ready    control-plane   16d   v1.26.0    192.168.56.130   <none>        Debian GNU/Linux 12 (bookworm)   6.1.0-17-amd64                 containerd://1.6.28
node-2   Ready    <none>          2d    v1.26.13   192.168.56.132   <none>        CentOS Linux 7 (Core)            3.10.0-1160.108.1.el7.x86_64   containerd://1.6.28

$ kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                      AGE
ingress-nginx-controller             NodePort    10.98.74.33    <none>        10254:31196/TCP,80:30080/TCP,443:30443/TCP   47h
ingress-nginx-controller-admission   ClusterIP   10.100.192.1   <none>        443/TCP                                      47h
$ kubectl get po -n ingress-nginx -owide
NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
ingress-nginx-controller-5d84c5dd56-qdh4t   1/1     Running   0          46h   10.244.1.35   node-2   <none>           <none>

// access the services with `NodePort` and node IPs reachable outside the cluster
$ curl -iI 192.168.56.130:30080/healthz
HTTP/1.1 200 OK
Date: Tue, 20 Feb 2024 09:58:11 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive

// ssh to a node in the cluster, and access the services or pods directly
$ curl -iI 10.98.74.33/healthz
HTTP/1.1 200 OK
Date: Tue, 20 Feb 2024 09:50:08 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive

$ curl -iI 10.244.1.35/healthz
HTTP/1.1 200 OK
Date: Tue, 20 Feb 2024 09:50:22 GMT
Content-Type: text/html
Content-Length: 0
Connection: keep-alive

Typically, there are several services which are started on a cluster by kube-system.

$ kubectl cluster-info
Kubernetes control plane is running at https://cluster-endpoint:6443
CoreDNS is running at https://cluster-endpoint:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
#$ kubectl get svc -n kube-system kube-dns -oyaml
apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: kube-dns
  namespace: kube-system
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
...

To create proxy URLs that include service endpoints, suffixes, and parameters, you append to the service’s proxy URL:

http://kubernetes_master_address/api/v1/namespaces/namespace_name/services/https:service_name[:port_name]/proxy

If you haven’t specified a name for your port, you don’t have to specify port_name in the URL. You can also use the port number in place of the port_name for both named and unnamed ports.

By default, the API server proxies to your service using http. To use https, prefix the service name with https:

http:<kubernetes_master_address>/api/v1/namespaces/<namespace_name>/services/<service_name>/proxy

The supported formats for the <service_name> segment of the URL are:

  • <service_name> - proxies to the default or unnamed port using http

  • <service_name>:<port_name> - proxies to the specified port name or port number using http

  • https:<service_name>: - proxies to the default or unnamed port using https (note the trailing colon)

  • https:<service_name>:<port_name> - proxies to the specified port name or port number using https

Examples

$ kubectl create -n default deployment echo --image=k8s.gcr.io/echoserver:1.10
deployment.apps/echo created

$ kubectl expose -n default deployment echo --port 80 --target-port 8080
service/echo exposed

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

$ curl http://localhost:8001/api/v1/namespaces/default/services/echo/proxy/


Hostname: echo

Pod Information:
	-no pod information available-

Server values:
	server_version=nginx: 1.13.3 - lua: 10008

Request Information:
	client_address=172.25.0.1
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://localhost:8080/

Request Headers:
	accept=*/*
	accept-encoding=gzip
	host=localhost:8001
	user-agent=curl/7.74.0
	x-forwarded-for=127.0.0.1, 10.24.128.43
	x-forwarded-uri=/api/v1/namespaces/default/services/echo/proxy/

Request Body:
	-no body in request-

References