Troubleshooting Kubernetes Secrets and Identifying Reasons for Pod Startup Failure

Amazon Web Services (AWS) provides a tool called AWS Secrets Manager that makes it easier to securely store and manage sensitive data, such as database credentials and API keys, that are utilized in modern applications. By allowing developers to consolidate storage and automate rotation operations, it simplifies the handling of sensitive data.

Many people could run into problems when trying to integrate their service with a new API, particularly when trying to get into AWS Secret Manager in order to add the new secret as an environment variable. They might, however, run into problems finishing this assignment.

Let’s quickly go over how to add new secrets to the AWS Secrets Manager.

If you have already visited Secrets Manager, simply go to it from a recent visit or type it into the search bar.

Click Store New Secret

We need to integrate with a new API so we will choose  Other type of secret

Now, here’s the trick: if you select a key/value pair, it can be accurately accessed through code rather than being readily set as an environment variable, so we’ll choose the Plaintext option.

Paste your secret after removing the curly braces.

Lastly, assign a name to your secret.

This is the second part: since you have now contributed a new secret to aws secrets, we must use it inside your deployment file.

deployment/
|-- helm/
|-- templates/
|   |-- deployment.yaml
|   |-- service.yaml
|   |-- secrets.yaml
|   |-- secret-provider.yaml
|-- values.yaml
|-- Chart.yaml  
|-- .helmignore

Now, we must change secret-provider.yaml to reflect the recently added secret. We will also be adding a new secret to the ones we already have.

env:      
- name: THIRD_PARTY_CLIENT_ID
  value: {{.Values.ThirdPartyClientId}}
- name: CLIENT_AUTH_SECRET
  valueFrom:
    secretKeyRef:
      name: {{ include "deployed-system.fullname" . }}
      key: thirdParty_Api_Secret

Sadly, a pod was generated but the container had not begun, thus it’s time to test the new deployment and see if the auth has been restored or not.

Error from server (BadRequest): container "myapp-worker" in pod "myapp-worker-7957797764-2j9dd" is waiting to start: CreateContainerConfigError

The deployment won’t immediately update with the new secrets; you have to refresh the secrets.

The Secrets Store CSI Driver consists of two reconcilers:

1. The first reconciler generates the initial Kubernetes secret upon pod creation. It does not update the secret if it already exists. This prevents inconsistencies in values between pods if the SecretProviderClass is updated and new pods are created.
2. The rotation reconciler updates both the mount and the Kubernetes secrets with the latest values at the specified rotation interval. After rotation, if a pod is restarted, it will use the latest Kubernetes secret containing the newly added keys from the SecretProviderClass. However, if a pod is restarted before rotation, the first reconciler checks if the secret exists and does not update it with the new keys.

If you want to update both the pod and Kubernetes secret without using the rotation feature, you can refer to the instructions provided here: https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/master/docs/README.limitations.md#mounted-content-and-kubernetes-secret-not-updated-after-secret-is-updated-in-external-secrets-store.

You can simply remove the secrets mount and then launch a fresh deployment, which results in the recreation of the secrets. You have the option to manage the Kubernetes cluster using Lens.

You can resolve the issue by selecting the secrets and deleting them from the UI. After that, retrigger your pipeline, and everything will be fine.

Alternatively, you can achieve the same outcome with a few commands.

1- Delete the pod and let it recreate and remount the secret

kubectl delete pod my-pod

2- Roll the pod to force it to be recreated with the latest secret:

kubectl rollout restart deployment my-deployment

Alternatively, you can patch the pod by introducing a new dummy environment variable to force pod recreation.

Reference

https://blog.devops.dev/manage-secrets-on-aws-and-helm-as-environment-variables-f7ec998c58fc