When pods communicate with the API server, they use a service account to authenticate.
The token for this is defined in a secret, which is mounted as a volume into the pod in question. The token is signed by the key specified by the API servers --service-account-key-file property

You can test this token by making a request to the API server with curl as follows:

curl -H  “Authorization: Bearer <token>"  https://<master ip addr>:6443/api/v1/namespaces/<namespace>/pods/<pod name>

You can derive the token from the relevant secret as follows:

kubectl describe secrets -n kube-system
Name:         default-token-jvrkf
Namespace:    kube-system
Labels:       
Annotations:  kubernetes.io/service-account.name=default
              kubernetes.io/service-account.uid=ba8add2d-12ab-11e8-9719-080027dd95a3

Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1066 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW
50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVy
bmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZX
JuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5l
dGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOT
cxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVs
dCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoC
sKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqG
xbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZ
pIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds
_ug

 

Here's an example curl request:

curl -k -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLWp2cmtmIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiYThhZGQyZC0xMmFiLTExZTgtOTcxOS0wODAwMjdkZDk1YTMiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.JYVNgnSNO-Gs4zuqjZHOQs1e4RG31c_trE5AcJXqlexM5On0SriWjb-qICS2o5_M0jNmSKED67lfAomoCsKOk7a4zlAdpkc4eVWxMXEaWemu6yxQAIcJjiaBgEpOm38FMQJKEsb1D3kdD0W0HD1MNcgY0LXxyXuWv7NToqGxbYn7QJqrEEFTM5qEx5IAAA7YeWOkXqQ35Vtg1Co1u-Ilw2Cr1GPtmCazCL0t8bpP4oi7qDr98lYmWNMo8KcPZpIBoS9Cg0bX3gONOE3Fga6Aa0e6VxLPy9VyXdezXvNtsmMtW-ryJ8fvplap2wj2JPFWCEO26DIXQSx8OX_h3ds_ug" https://192.168.99.100:8443/api/v1/namespaces/kube-system/pods/kube-dns-54cccfbdf8-xqgz6

You can see how the pod uses this token by running kubectl describe pods kube-dns-54cccfbdf8-xqgz6 -n kube-system and checking the Mounts section in the output.

Mounts:
/kube-dns-config from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jvrkf (ro)

It's worth noting that if you run kubectl get secrets -n kube-system -o yaml as opposed to kubectl describe secrets -n kube-system, the token will be base64 encoded so you'll need to pipe it through base64 --decode before you copy and paste it into your curl command

If you regenerate your cluster certificates for any reason, you'll need to regenerate these tokens. To do this just delete the secrets with kubectl, new secrets will be generated automatically. However (and this is really important and had me going for a while) these secrets will have a different name, therefore your pods will still be trying to mount the old secrets. To resolve this you also need to delete the pods (e.g. kubectl delete pod ) -n kube-system . If the pods are managed by deployments (and this is why you should use deployments by the way) they will automatically be regenerated and will mount the new secrets automatically.

Your best friend by far in all of this is the command journalctl -u kubelet -f. If you have a pod stuck in ContainerCreating and therefore are not able to look at that it's container logs, simply log on the node in question and run journalctl -u kubelet -f | grep .. priceless tool

NOTE: if you're using Calico networking, this new token will need to be set in /etc/cni/net.d/10-calico.conf which is in turn derived from the ConfigMap calico-config. If you edit the contents of this file directly, the changes will be overwritten by the contents of the calico-config ConfigMap at the next kubelet restart.