Unleashing Terraform for Kubernetes secret management with IBM Cloud Kubernetes Service and Secrets Manager

In this blog post, we explore the practical implementation of utilizing Terraform on IBM Cloud to create and manage secrets by seamlessly integrating your IBM Cloud Kubernetes Service with IBM Cloud Secrets Manager.

Previously, this functionality to manage TLS and non-TLS certificates and secrets was primarily accessed through the CLI using the namespace ibmcloud ks ingress secret. This API enables users to create an “Ingress secret” resource by passing Secrets Manager secret CRNs to the API to establish a managed corresponding secret in their Kubernetes cluster. Notably, any updates made to the secrets within the Secrets Manager instance are automatically reflected within the associated Kubernetes cluster, ensuring synchronization between the two environments.

Architecture and behavior

The IBM Cloud Kubernetes Service reconciles the created Ingress secrets in the following way:

The user has an existing IBM Cloud Secrets Manager instance and IBM Cloud Kubernetes Service instance.

The user registers the Secrets Manager instance to ensure its secret CRNs will be synchronized between the Secrets Manager secret and corresponding Ingress secret(s).

The user then creates an IBM Cloud Kubernetes Ingress secret that can either be an Opaque or TLS secret with a Secrets Manager CRN (ID). This creates a backing resource in the cloud that correlates the secret CRN to the ClusterID/SecretName/SecretNamespace.

IBM Cloud Kubernetes Service fetches the Secrets Manager secret via the CRN.

IBM Cloud Kubernetes Service creates a Kubernetes secret in the cluster with the values of the CRN(s).

IBM Cloud Kubernetes Service ensures that the secrets values stay in sync with the corresponding Secrets Manager secret CRN.

Benefits

By utilizing the integration with IBM Cloud Kubernetes Service and IBM Cloud Secrets Manager, you can leverage the following benefits:

Seamlessly create and manage Secrets Manager secrets with built-in autorotation for enhanced security.

Effortlessly provision Kubernetes secrets using the secret CRN of any Secrets Manager instance you own, ensuring consistent and reliable secret management.

Automatically synchronize and persist your secrets within your Kubernetes cluster on a regular basis, eliminating the need for manual updates and reducing the risk of outdated secrets.

Easily track and monitor the expiration dates of your secrets directly from the IBM Cloud console, ensuring timely rotation and preventing potential security vulnerabilities.

Maintain control over access to your secrets by creating secret groups, allowing you to grant permissions only to approved users and enhancing the overall security of your applications.

Hands-on example

The below example shows an integration of IBM Cloud Kubernetes and IBM Cloud Secrets Manager via a Terraform script. To follow along in the full sample, go to this example. You will provision an IBM Cloud Secrets Manager instance, register it to an IBM Cloud Kubernetes Service, and create managed IBM Cloud Kubernetes Ingress secrets backed by Secrets Manager secrets.

Prerequisites

To follow this example, you will require the following:

Walking through the Terraform script

1. Create an IBM Cloud Secrets Manager instance

Create an IBM Cloud Secrets Manager instance and secret group to host your secrets. Learn more about Creating a Secrets Manager service instance:

resource “ibm_resource_instance” {
name = var.sm_instance_name
service = “secrets-manager”
plan = var.sm_instance_plan
location = var.sm_instance_region
timeouts {
create = “60m”
delete = “2h”
}

}

resource “ibm_sm_secret_group” {
instance_id = ibm_resource_instance.sm_instance.guid
region = ibm_resource_instance.sm_instance.location
name = var.sm_secret_group_name
description = var.sm_secret_group_description
}

2. Set up service-to-service authorization through IAM

See more about what configurations are needed to enable service-to-service communication:

resource “ibm_iam_authorization_policy” {
source_service_name = “containers-kubernetes”
target_service_name = “secrets-manager”
roles = [“Manager”] }

3. Register the Secrets Manager instance to the IBM Cloud Kubernetes Service cluster

When you register a Secrets Manager instance to your cluster as the default, all new Ingress subdomain certificates are stored in that instance:

resource “ibm_container_ingress_instance” {
cluster = var.cluster_name_or_id
secret_group_id = ibm_sm_secret_group.sm_secret_group.secret_group_id
instance_crn = ibm_resource_instance.sm_instance.id
is_default = true
}

4. Create secrets in Secrets Manager and enable automatic rotation

Create an arbitrary and username credential secret in Secrets Manager. Learn more about different secret types:

resource “ibm_sm_arbitrary_secret” {
instance_id = ibm_resource_instance.sm_instance.guid
region = ibm_resource_instance.sm_instance.location
endpoint_type = var.sm_endpoint_type
name = var.sm_arbitrary_secret_name
description = var.sm_arbitrary_secret_description
expiration_date = var.sm_arbitrary_secret_expiration_date
labels = var.sm_arbitrary_secret_labels
secret_group_id = ibm_sm_secret_group.sm_secret_group.secret_group_id
payload = var.sm_arbitrary_secret_payload
}

resource “ibm_sm_username_password_secret” {
instance_id = ibm_resource_instance.sm_instance.guid
region = ibm_resource_instance.sm_instance.location
endpoint_type = var.sm_endpoint_type
name = var.sm_username_password_secret_name
description = var.sm_username_password_secret_description
expiration_date = var.sm_username_password_secret_expiration_date
labels = var.sm_username_password_secret_labels
secret_group_id = ibm_sm_secret_group.sm_secret_group.secret_group_id
rotation {
auto_rotate = true
interval = 1
unit = “day”
}

username = var.sm_username_password_secret_username
password = var.sm_username_password_secret_password
}

5. In the cluster, create a persistent Opaque secret that is backed by the CRN of the secrets in Secrets Manager

Create an Ingress Opaque secret in the cluster. Now, anytime the secrets in Secrets Manager are updated, the corresponding Kubernetes Opaque secret will be updated once a day. The persistence field ensures that if a user inadvertently deletes the secret from the cluster, it will be recreated:

resource “ibm_container_ingress_secret_opaque” {
cluster = var.cluster_name_or_id
secret_name = var.opaque_secret_name
secret_namespace = var.opaque_secret_namespace
persistence = true
fields {
crn = ibm_sm_arbitrary_secret.sm_arbitrary_secret.crn
}
fields {
crn = ibm_sm_username_password_secret.sm_username_password_secret.crn
}
}

Creating the infrastructure

Now that you’ve gone through what each block of the Terraform script will be doing, let’s create the infrastructure.

Run terraform init in your directory.

Copy the main.tf and output.tf files from the example repo.

Create a .tfvars file and fill in the corresponding variables needed. You can learn more about what variables are needed in the variables.tf file.

Run terraform plan -var-file=<file_name>.

Create the resources with terraform apply -var-file=<file_name>.

Verifying created resources

Now that these resources are created, go into the IBM Cloud Dashboard to view the created resources under Resource list:

Navigate to the created IBM Cloud Secrets Manager instance and view the created secrets:

Navigate to the IBM Cloud Kubernetes Service, click on Ingress, then select the Secrets tab to view the Opaque secret:

Contact us

This sample serves as a starting point to showcase the benefits and functionality of integrating Terraform with IBM Cloud Kubernetes Service and IBM Cloud Secrets Manager. Feel free to expand and tailor this approach to fit your use case.

If you have questions, engage our team via Slack by registering here and join the discussion in the #general channel on our public IBM Cloud Kubernetes Service Slack.