2024-12-28

Hashicorp Boundary and the Hoops of Pain

Hashicorp released Boundary as a Privileged Access Management (PAM) in late 2020, and I did not yet had a chance to implement it together with Vault.

This changed and over the course of several days I had to jump through a lot of hoops getting this running.

I intended this post to be a rant but decided to just extend the excellent discovery of Alan Longuet (links below; 2022-07 // Discovery of Boundary) with my findings. Why? Because ranting about “source available” software without willing/being able/state your reason to participate is just lame.

I used Boundary 0.18.2 and Terraform provider 1.2.0.

Deployment

Based on Alan’s helm charts this was OK to adapt to my environment.

Controller pitfalls

Initialising the controller needs a bunch of additional flags to prevent unmanaged configurations

boundary database init \ -skip-auth-method-creation -skip-initial-login-role-creation -skip-scopes-creation \ -skip-host-resources-creation -skip-target-creation \ -config /boundary/config.hcl
Upgrading the controller is more painful as the pre-upgrade job needs exclusive access to the database. You have to scale the deployment to zero first (separate SA and role needed).
Upgrades are not backward compatible on the database; 0.18.0 -> 0.18.2 -> 0.18.0 is not possible, test releases accordingly!

Worker pitfalls

Workers can NOT be scaled horizontally, you need separate deployments
Each worker needs its own HCL configuration file with its public_address and tags
Each worker needs to be reachable by clients on a given TCP port -> you have to use a layer 4 loadbalancer

Configuration via Terraform

The provider requires glibc; if running on Alpine, you have to add libc6-compat - #384 // [bug] terraform plan fails with error reading wrappers from “recovery_kms_hcl”
You have to use separate TF stacks for initial resources (KMS, databases) and controller configuration as the provider requires a valid and working configuration even if not used - #565 // feat - lazy initializing of provider configuration
Running terraform locally, the provider does not support AWS_PROFILE environment but needs full ENV settings - #62 // Boundary provider needs assume_role

use eval "$(aws configure export-credentials --format env)"
You have to import the global scope
The boundary_auth_method_password can not be set as the primary auth method in code (UI or CLI are able to); this is a problem later on - #116 // Unable to set Primary Auth Method resource

Operator Experience

After sorting out the initial Terraform setup, configuring the controller in code is OK. Not easy/trivial but OK.

Targets and default client ports

If there is a chance that 2 boundary_target would love to open the same client port (e.g. databases), do NOT define default_client_port. As it will break the UI and CLI - #4759 // Creating multiple target connections with the same client port hangs the boundary CLI

Audit trail

Using Vault as a credential library for databases, Boundary uses a token to authenticate against Vault.
From Vault’s perspective it is always the same entity authenticating (Boundary), and there is no information available about the authenticated user in Boundary. This means the Vault temporary credentials can NOT contain data about the authenticated human for audit purposes.

The workaround is to keep the Boundary Audit log containing the provided Vault credentials (in clear, there is a bug preventing to redact only the password)
in the Boundary audit log, the internal user id is shown and external data (from OIDC) are not logged. One has to dump the user base and match it with the audit log - #4158 // add email & username to audit logs records
To dump out the user base, you have to use a Boundary user as the command boundary users list -recursive is only available as a client command and not available in the terraform provider either.

This also requires to know the auth_method_id, see the password default authentication method issue above.

User Experience

the Desktop UI supports only 1 active controller session - #5210 // support multiple user profules in boundary desktop
the CLI is not very user-friendly, I created some wrappers to make the usage easier
Integration of database targets into IDEs or 3rd party tooling is not good, users have to update their connections (username, password, port). This is still a TODO for me, I’ll update the post.

Wrapping it up

Once set up, having jumped through the hoops, hit your head on the desk and gnawed your fingernails off - it is a solid piece of software and especially the Vault integration is great.

Quoting Alan from 2022 here:

Overall Boundary is a really good tool that enables fine-grain control around who can access internal resources, it has some features that I didn’t talk about which are really impressive (e.g. the ability to correlate targets and workers based on tags is really impressive and enhances the user experience). But its user/operator experience really drives its rating down

I got the feeling that the Boundary project lost attention from Hashicorp, which is sad as, to quote Alan again, “[…] achieving this level of quality is insane!”

An alternative would be Teleport but it lacks the native Vault integration. A biased comparision over here ;-)

Standing on the shoulders of giants