Configure Tanzu Kubernetes Grid to use OIDC with Okta

VMware Tanzu^TM Kubernetes Grid^TM (TKG) provides user authentication by using Dex and Gangway. Dex is an OpenID Connect (OIDC) provider, that enables authentication for Kubernetes clusters by connecting to external Identity Providers like Okta (OIDC). Gangway is a web application that enables the OIDC authentication flow, it is configured as a client of an upstream Identity Service that speaks OIDC. TKG running on Amazon EC2 supports only OIDC, so we are going to use Okta as the Identity provider and enable User Authentication on TKG clusters running on AWS EC2.

Before you begin

• I am using Mac to run the below commands, you may need to change some if you are using Windows especially.
• We are using TKG v1.1
• tkg cli v1.1 downloaded and configured
• You have TKG management cluster up and running on EC2 and have access to its config.
• Setup a developer account on Okta.
  • I am using Okta, but you may use any supported OIDC providers, however, the configurations on the OIDC provider may vary.
• Have your Okta developer account url. Ex: https://dev-000000.okta.com

Let’s dive in…

The shared services on TKG are provided as TKG Extensions Bundle.

Download the TKG Extensions Bundle by following the steps described in official documentation.

Unpack into the same directory where you are running tkg / kubectl commands. In my case i have unpacked under ~/workspace/tkg1.1

tar -xzf tkg-extensions-manifests-v1.1.0-vmware.1.tar.gz

Observe the tkg-extensions are unpacked under ~/workspace/tkg1.1. I will be executing all my commands relative to this path.

Enabling User Authentication for TKG Clusters involves multiple steps

• Deploying Dex on your Management Cluster running on EC2
• Create OIDC enabled TKG Workload Cluster
• Enable Gangway on OIDC enabled TKG Workload Cluster.
• Configure Workload Cluster as Client in Dex
• Access the cluster with Okta credentials.

Step 1: Deploying Dex on your Management Cluster on EC2

In this step we are going to deploy Dex to work with Okta as OpenID Provider. And once it is configured, dex will act as an Authentication provider for our TKG Workload Clusters.

Set your kubectl context to point to TKG management cluster.

Ex: kubectl config use-context tkg-mgmt-003a-admin@tkg-mgmt-003a

1. Create namespace

Create a namespace called tanzu-system-auth in your management cluster

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/01-namespace.yaml

2. Create a Service for Dex

Create the Dex Service, this will create a LoadBalancer type service

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/02-service.yaml

Get the hostname of the load balancer of the Dex Service

kubectl get svc dexsvc -n tanzu-system-auth -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Example LB hostname: a552c796e0846c2b83792e64ba1be4-834747576.us-east-1.elb.amazonaws.com

I would like to use a more readable name than this so I am going to use my own DNS zone in AWS to add an alias for this lb hostname. Though this is not mandatory, I would recommend to do it.

Add a DNS entry in Route53

In Route53, go to your host zone, click on CreateRecord Set.

• Enter the name as dex
• Select Type A
• Select Alias – Yes
• Paste the LB hostname in Alias Target, click Create

So, my dns name for dex service is dex.helium.k8scloud.cf

3. Create Self-Signed Certificate

Open tkg-extensions-v1.1.0/authentication/dex/aws/oidc/03-certs-selfsigned.yaml in text editor. Replace <DEX_SVC_LB_HOSTNAME> with the dex dns name.

Run the below command to create self-signed certificate.

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/03-certs-selfsigned.yaml

4. Create Dex Configuration (ConfigMap)

Open the Dex Configuration Map file in the text editor. tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml.

• Replace <DEX_SVC_LB_HOSTNAME> with dex dns name.
  • Ex: dex.helium.k8scloud.cf
• Replace <OIDC_IDP_URL> with okta developer account url.
  • Ex: https://dev-000000.okta.com

Run the below command to create the configmap.

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml

5. Create RBAC

Run the below command to Configure Role Based Access Control (RBAC).

This creates a Service Account, Cluster Role and Cluster Role Binding.

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/05-rbac.yaml

Create an Application in Okta

As we are using Okta as our OIDC provider, we need to create an application in Okta to integrate with dex. Login to Okta developer account with your admin credentials

Goto Applications and click on Add Application.

On the next screen, select Web application icon and click on Next

On the Settings screen, enter the below information and click ‘Done‘.

• Name of the application, ex: TKG_EC2_Dex
• Base URIs – Provide as your Dex URL, ex: https://dex.helium.k8scloud.cf
• Login redirect URIs – Provide the value from redirectURI of dex config map (see dex configmap) , ex: https://dex.helium.k8scloud.cf/callback
• Logout redirect URIs – Provide as your Dex URL, ex: https://dex.helium.k8scloud.cf
• Grant type allowed, select Client Credentials.

In the next screen, under Client Credentials, make note of Client ID and Client secret, which we will be using in next step.

6. Configure Secret

Back to dex configuration. A sample secret file has been provided in the bundle, run the below command to copy to 06-0-secret.yaml.

cp tkg-extensions-v1.1.0/authentication/dex/aws/oidc/06-0-secret.example tkg-extensions-v1.1.0/authentication/dex/aws/oidc/06-0-secret.yaml

Open the secret yaml file in text editor tkg-extensions-v1.1.0/authentication/dex/aws/oidc/06-0-secret.yaml

Run below command by replacing <OKTA_CLIENT_ID> with Client ID obtained in the above step

echo -n "<OKTA_Client_ID>" | base64

Replace the <CLIENT_ID> in secret yaml with the output of the above command

Run the below command by replacing <OKTA_CLIENT_SECRET> with Client secret obtained in the above step.

echo -n "<OKTA_CLIENT_SECRET>" | base64

Replace the <CLIENT_SECRET> in secret yaml with the output of the above command

Save the file and Run the below command to create the secret

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/06-0-secret.yaml

7. Create the Dex deployment

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/06-deployment.yaml

Validate if the dex pods are running by using

kubectl get pods -n tanzu-system-auth

8. Export Dex CA Certificate

Export the Dex CA Certificate to file, this will be used in the Gangway configuration step (Step 3).

kubectl get secret dex-cert-tls -n tanzu-system-auth -o 'go-template={{ index .data "ca.crt" }}' | base64 -D > dex-ca.crt

Note: If you are using Linux, replace base64 -D with base64 -d.

Step 2: Create OIDC enabled TKG Workload Cluster

In previous step, we have enabled Dex to provide Authentication service. Now we have to create the TKG Workload Clusters which uses Dex for cluster user Authentication.

1. Create new TKG Plan

In the TKG Extensions Bundle, we have a special plan which is using Dex as Authentication provider. Copy/Move the file cluster-template-oidc.yaml to the Cluster API provider for EC2. The tkg configuration directory is ~/.tkg in Linux or Mac.

mv tkg-extensions-v1.1.0/authentication/dex/aws/cluster-template-oidc.yaml ~/.tkg/providers/infrastructure-aws/v0.5.3/cluster-template-oidc.yaml

2. Export OIDC environment variables

Make sure that your kubectl context is set to the management cluster. Ex: kubectl config use-context tkg-mgmt-003a-admin@tkg-mgmt-003a

Set the below environment variables

Replace <DEX_SVC_LB_HOSTNAME> with dex dns name. Ex: export OIDC_ISSUER_URL=https://dex.helium.k8scloud.cf

export OIDC_ISSUER_URL=https://<DEX_SVC_LB_HOSTNAME>

export OIDC_USERNAME_CLAIM=email
export OIDC_GROUPS_CLAIM=groups
export DEX_CA=$(kubectl get secret dex-cert-tls -n tanzu-system-auth -o 'go-template={{ index .data "ca.crt" }}' | base64 -D | gzip | base64)

Note: If you are using Linux, replace the base64 -D with base64 -d in DEX_CA variable.

The above values will be passed as parameters to kube-apiserver while creating the cluster.

oidc-issuer-url: ${ OIDC_ISSUER_URL }
oidc-username-claim: ${ OIDC_USERNAME_CLAIM }
oidc-groups-claim: ${ OIDC_GROUPS_CLAIM }
oidc-ca-file: /etc/tkg/pki/dex-ca.crt

3. Create TKG Workload Cluster

Set your tkg cli context to your management cluster. Ex: tkg set management-cluster tkg-mgmt-003a

tkg set management-cluster <management-cluster>

Create the TKG workload Cluster using the new plan provided (oidc). I have used demo-oidc-cluster as the name for my new cluster.

tkg create cluster demo-oidc-cluster --plan=oidc

This will take some time and create the cluster. Once cluster is created, get the cluster credentials

tkg get credentials demo-oidc-cluster

Set the kubectl context to our new workload cluster

kubectl config use-context demo-oidc-cluster-admin@demo-oidc-cluster

Step 3: Enable Gangway on OIDC enabled TKG Workload Cluster

This step will enable Gangway on our workload cluster.

Make sure your kubectl context is set to workload cluster.

1. Install the cert-manager to workload cluster

kubectl apply -f tkg-extensions-v1.1.0/cert-manager/

2. Create a namespace

Create namespace in workload cluster with name tanzu-system-auth

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/vsphere/01-namespace.yaml

3. Create the Gangway Service

Run the below command to create Gangway Service of type LoadBalancer.

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/aws/02-service.yaml

Get the host name of the Gangway service load balancer

kubectl get svc gangwaysvc -n tanzu-system-auth -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Add a DNS entry in Route53 for this url. Ex: gangway.helium.k8scloud.cf. Refer to Add DNS entry in Route53 section above.

4. Create Gangway Configuration (ConfigMap)

Open the gangway configmap file in any text editor tkg-extensions-v1.1.0/authentication/gangway/aws/03-config.yaml. Modify the below

• Replace all instances of <WORLOAD_CLUSTER_NAME> with our cluster name. Ex: demo-oidc-cluster
• Replace <DEX_SVC_LB_HOSTNAME> with the dex dns name. Ex: dex.helium.k8scloud.cf
• Replace <GANGWAY_SVC_LB_HOSTNAME> with gangway dns name we created in above step.
• Replace <APISERVER_URL> with our workload Kubernetes Cluster endpoint. This is AWS load balancer name with name like ‘demo-oidc-cluster-*.elb.amazonaws.com’. You can get it easily by running kubectl config view and take just the hostname part from server property of our workload cluster.

Create ConfigMap

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/aws/03-config.yaml

5. Create Secret

A sample secret file is provided in bundle, copy the file to create a secret yaml 04-secret.yaml.

cp tkg-extensions-v1.1.0/authentication/gangway/aws/04-secret.example tkg-extensions-v1.1.0/authentication/gangway/aws/04-secret.yaml

Open the tkg-extensions-v1.1.0/authentication/gangway/aws/04-secret.yaml in text editor

Run the below commands at command line to create session key & client secrets using openssl.

Note: In the below command pbcopy copies the output of the command to clipboard.

openssl rand -base64 32 | pbcopy

In secret file in editor (04-secret.yaml), update the sesssionKey value by pasting.

Run the following commands to create client secret.

Note: Please run as given below, we are going to use clientSecret variable value generated in this step later in the procedure.

clientSecret=$(openssl rand -base64 32)
echo -n "$clientSecret" | base64

In the secret file in editor(04-secret.yaml), update the clientSecret property value with the output of above echo command.

Apply the secret to file to cluster

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/aws/04-secret.yaml

6. Create Self-Signed Certificate for Gangway Service

Open the file tkg-extensions-v1.1.0/authentication/gangway/aws/05-cert-selfsigned.yaml in text editor.

• Replace the <GANGWAY_SVC_LB_HOSTNAME> with the gangway dns name. Ex: gangway.helium.k8scloud.cf

Create the self-signed certificate

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/aws/05-cert-selfsigned.yaml

7. Create ConfigMap with Dex CA Cert

Create the ConfigMap with the Dex CA Certificate which we created in Step 1. Ensure the file dex-ca.crt exist in your working directory.

kubectl create cm dex-ca -n tanzu-system-auth --from-file=dex-ca.crt=dex-ca.crt

8. Create the gangway deployment

kubectl apply -f tkg-extensions-v1.1.0/authentication/gangway/aws/06-deployment.yaml

Ensure gangway is up & running

kubectl get pods -n tanzu-system-auth

Step 4: Configure Workload Cluster as Client in Dex

In this section, we are going to configure Dex with the new workload cluster as a Client. For this we have to modify the Dex ConfigMap we created earlier. We are going to update the same yaml file we used to create Dex ConfigMap and modify the contents and re-apply changes to Management Cluster.

1. Modify Dex Configuration File

Open the Dex ConfigMap yaml file in any text editor tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml

You will see the staticClients property as shown in the above picture. This is an array element, modify this as below

 staticClients:
 - id: <WORKLOAD_CLUSTER_NAME>
   redirectURIs:
   - 'https://<GANGWAY_SVC_LB_HOSTNAME>/callback'
   name: '<WORKLOAD_CLUSTER_NAME>'
   secret: <clientSecret>

• Replace <WORKLOAD_CLUSTER_NAME> with our workload cluster name. Ex: demo-oidc-cluster
• Replace <GANGWAY_SVC_LB_HOSTNAME> with gangway dns name. Ex: gangway.helium.k8scloud.cf
• Replace <clientSecret> with the output of the below command. (This variable we generated in Step 3 ).
  • echo $clientSecret

The Modified content may look like this

2. Apply changes to Dex ConfigMap

Set the kubectl context to your management cluster. Ex: kubectl config use-context tkg-mgmt-003a-admin@tkg-mgmt-003a

kubectl config use-context <mgmt-cluster-context>

Apply the modified configuration to Management Cluster.

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml

3. Delete Running Dex Pod

To reflect the changes we made to ConfigMap, delete the currently running Dex pod, so the dex deployment will start a new pod and which picks up new changes in configmap.

kubectl get pods --namespace tanzu-system-auth

The pods list may look like this, copy the pod name

Run the below command with the dex pod name (Replace it with your pod name).

kubectl delete pod dex-565599c4b6-w99wn -n tanzu-system-auth

Ensure a new pod came up

kubectl get pods -n tanzu-system-auth

With this step, we have configured Dex in management cluster, gangway in workload cluster and informed Dex about the client in our workload cluster.

Step 5: Access the cluster with Okta credentials

In this step we are going to validate our Dex, Gangway integration. We are going to use gangway url to access the cluster login page and use our developer okta credentials to login to get kubeconfig to access our workload cluster with our okta user.

1. Get the Kube config

Open the browser and access the gangway url. Ex: https://gangway.helium.k8scloud.cf

You might have to accept the browser warning as we are using self-signed certs.

Click on SIGN IN, this will redirect url to dex, then to okta login page. Use your okta credentials to login.

Note: I made some modifications to my okta profile to allow username as not email. But for you it may be email to start with.

Once the login and token generation is successful, you will see below screen.

Click on DOWNLOAD KUBECONFIG to download the kubeconf for the cluster.

2. Verify access to Cluster

As we are already using the current terminal window to access our workload cluster using admin credentials, we are going to validate the new credentials using another terminal.

Copy the downloaded kubeconf file to working directory.

Open a new terminal, and navigate to your workind directory, set KUBECONFIG=kubeconf. Run the below command to get the list of pods in the default namespace

kubectl get pods

You may get access error like below because the user doesn’t have permission to perform any actions like create, update, reconfigure, delete etc..

3. Provider Cluster permission to Okta user

The appropriate cluster roles and cluster role bindings must be defined. Refer to Kubernetes documentation for Using RBAC Authorization. In the below example I am using the default role cluster-admin, which gives full access to cluster.

Create the file cluster-user-rbac.yaml with the below content by changing the username to your Okta username.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: okta-user-access-binding
subjects:
  - kind: User
    name: [email protected]
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Go to your original terminal where the context is set to your workload cluster with admin user. Run the below to apply Cluster Role Binding

kubectl apply -f cluster-user-rbac.yaml

Now, run the below command to test the access

kubectl get pods -A

With this now you can access to workload cluster using the OIDC Authentication.

Summary

With this integration of Dex and Gangway to our TKG management and workload clusters, we are able to access our workload cluster with okta credentials and we can enable access to all required users in the organization. However, with the configurations we have used in this post it will only allow us to provide permissions to individual users, in the next post I will cover the steps to enable groups access.

Additional Info

With the default configuration of Dex above, the id token will expire in 5m, if you would like to increase the token expiry, modify the below

Open the tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml in text editor.

Change the idTokens value to desired duration, ex: 24h

expiry:
      signingKeys: "10m"
      idTokens: "5m"

Apply the changes to management cluster

kubectl apply -f tkg-extensions-v1.1.0/authentication/dex/aws/oidc/04-cm.yaml

Delete the running dex pod

kubectl get pods --namespace tanzu-system-auth
kubectl delete pod <dex pod name> -n tanzu-system-auth

References

• VMware Tanzu Kubernetes Grid Official documentation
• https://github.com/dexidp/dex
• https://github.com/heptiolabs/gangway
• https://developer.okta.com/docs/reference/api/oidc/