The OpsCorner

Issue:

After deploying k8s clusters I usually run through a yelb app deployment to test and validate the cluster fundamental operations. After deploying TKG and the yelb app in a new namespace I noticed the ReplicaSets were not bringing pods online.

kubectl get pods shows no pods and kubectl get rs shows the below.

After looking at the ReplicaSet using describe I found the following error.

kubectl describe rs redis-server-6d46df89bb | select -Last 5
---- ------ ---- ---- -------
Warning FailedCreate 14m (x16 over 17m) replicaset-controller Error creating: pods "redis-server-6d46df89bb-" is forbidden: unable to validate against any pod security policy: []

These yelb deployments require a privileged pod security policy to allow the containers to run . This is generally not a best practice however these deployments are usually deleted soon after validating and this is a great opportunity to look at using PodSecurityPolicies (PSPs) with VMware TKG to relax some of the security of the deployment. And they are not my apps 🙂

Solution:

Use the built-in TKG PodSecurityPolicies to relax security for the yelb app deployments.

What you will need:

• vSphere7 HA/DRS cluster deployed with Workload Domain and Supervisor Cluster
• TanzuKubernetesGrid (v1.16.8) deployed into a namespace created after the Supervisor Cluster is deployed
• Kubectl client

By default VMware TKG will come with two PodSecurityPolicies shown below.

You can list these by running..

kubectl get psp

You can look closer at the vmware-system-privileged policy by running.

kubectl describe psp vmware-system-privileged

You can always create new PSPs defining specific rules or you may use the default vmware-system-privileged PSP provided. In this example we will be creating a service account to run our containers and using clusterrolebindings to grant access to the privileged built-in PSP.

First things first we create a new namespace to house our work and deployment. We will be using the yelb app in this example as it will not work without relaxing the security so its a good use case.

This solution assumes you have deployed TKG in a namespace after you have configured your workload domain in vSphere7.

1. Create the yelb namespace

kubectl create ns yelb

2. Create the service acccount in the yelb namespace

kubectl create sa sa-priv-user -n yelb

Once the account is created we need a role to assign to the serviceaccount.

TKG comes with two cluster wide roles listed below by issuing the kubectl get clusterrole command.

3. We will need to bind the service account we just created to the role in the yelb namespace with the below yaml.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: psp:vmware-system-privileged
roleRef:
  kind: ClusterRole
  name: psp:vmware-system-privileged
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: sa-priv-user
  namespace: yelb

Now that we have our service account in the namespace and applied the ClusterRoleBinding, we can use the service account and privileged context to get the yelb pods app running. We need to make adjustments to the yelb deployments using the below information.

spec.template.spec.serviceaccount=sa-priv-user
spec.template.spec.containers.securitycontext.privileged=true

Like so in each deployment declaration.

4. Redeploy your yelb app with the serviceaccount and securitycontext specs.

The yaml files used in this example can be found here.

Couple of things…..

• Be mindful of the the namespace and your context.
• If you deployed the yelb app before making these changes delete the app completely and redeploy after the changes.

I hope this walk through of TKG vmware-system-privileged PSP was helpful!

Enjoy!