TLDR:

Mount your credentials in kubernetes as files without having them in kubernetes secrets. Requires 1Password Connect installed and configured in your cluster. Github repo here.

Why 1Password for kubernetes secrets?

• If you are tied to a provider (aws,gcp,azure), then by using workload/pod identity you can certainly increase your security by not having secrets in the first place.

• If you are multi-cloud or not in one of those providers, your options end up being much less clear - hashicorp vault is probably the go-to, now there's an alternative!

• UI/UX the cloud secret providers are lacking, with 1password, a dev can use the nicer 1password gui and get the password into the cluster so much easier (even if for e.g. only the devops/sre team have access to the vaults for production)

Why secrets-store-csi-driver?

• By having your passwords in your cluster as secrets, you have to be very careful about what access you give to developers etc.

• By mapping those secrets to environment variables, any vulnerability exploited in your cluster to simply running env and the attacker can access destinations.

• This gets worse when you have such things as "ruby shell" and similar.

• A future enhancement to secrets-store-csi-driver hopefully will be to have (optional) expiring mounts. This means your application reads the config on startup and then the credentials disappear shortly after.

• If you MUST support k8s secrets (for example for third-party helm charts) or env vars, you can actually do that with secrets-store-csi-driver, I just prefer to really reduce the attack surface.

But there's external-secrets and the 1password-connect-sync operator?

• See the Why secrets-store-csi-driver? section above, yes, these are perfectly valid ways to get your credentials into your pods but IMO mounting the creds as files is the right way to go.

• Both external-secrets and secrets-store-csi-driver are solid parts of the Kubernetes ecosystem. I prefer the latter, so this is my take on making it better.

How did this take shape?

Quite simply actually. The documentation for writing drivers for the secrets-store-csi-driver is sparse - this is often a problem with deep tech stuff, so don't judge the project too harshly.

First, I took the GCP driver and ripped out the GCP stuff! Sincerest apologies to the people who wrote the Google code.

Then I used the OnePassword Connect Go SDK and wired in the code back to the places where I ripped out the Google code.

Then I had to make some decisions like how to reference the credentials. I went with the structure of the URLs in the browser so it is easy to copy/paste. Also, I wanted to make it easy to reference both an entire entry and a single field, so I provided a "full entry as json" type path and the optional further part of the url as the item.

I also wanted to get the code building and pushing to a public repository, doing this required me to remove the tests. I'm two steps removed from good Dev practices like TDD these days (step 1: DevOps/SRE, step 2: manager). This enabled me to get github actions working. Please, if you have ideas on how to test something worth testing, PRs welcome. I'm struggling to find things to test because this is wiring two fully-tested projects together, testing that either is working doesn't feel so useful, especially as you would need to mock out so much.

I then, of course, got my employer's permission to open source the project and set up GitHub actions and a quay.io image repository (most clear on "free for opensource" without a big application process).

How to deploy it

First you will need two API keys - one for 1Password Connect to connect to the 1Password API and one for secrets-store-csi-driver-1password-provider to talk to 1Password Connect with.

Then deploy 1Password Connect to your cluster by following the documentation provided by 1Password. You don't need to install the operator as that's the thing that this project kind of replaces!

Deploying the secrets-store-csi-driver is the next step - I'm not going to go step by step because it's quite a simple daemonset deployment that the devs of that project have documented.

You can deploy the secrets-store-csi-driver-1password-provider using kubectl apply the deploy/ folder or helm apply from the charts folder. The install of the provider is even simpler, as it communicates with secrets-store-csi-driver over a Unix Domain Socket (this is why it deploys as a DaemonSet). This is what you need the second API key for, set it either by editing the yaml if you prefer that route, or when using helm --set connect.token.

There's a whole discussion to be had around protecting the connect token. By installing it in kube-system, you will often already have setup some protections, but it's beyond the scope of this post.

Then to mount a secret you create a SecretProviderClass which defines the vault and item you want to mount, and reference it in your pod.

The provider is really neat and by using it this way, you can surprisingly easily get your application manifests safely in git.

Footnote: this hashnode.com blog post is part of the 1Password Hackathon