Wednesday, November 19, 2025

HashiCorp Vault Secrets Engines

HashiCorp Vault Secrets Engines - Complete Implementation Guide

HashiCorp Vault Secrets Engines - Complete Implementation Guide

🎯 Prerequisites: Vault Fundamentals → Authentication Methods → Policies & Access Control → Token Management
📚 What You'll Master:
Complete secrets engine implementation: static vs dynamic secrets, KV versioning, Transit encryption, Consul integration, response wrapping with production examples

Secrets Engines Overview

What Are Secrets Engines?

Secrets engines are the what of Vault - they define what Vault actually does. They are plugins incorporated into Vault that handle sensitive data in three fundamental ways:

🔑 Three Core Functions:
  • Store sensitive data - Manage access to externally-generated secrets
  • Generate sensitive data - Dynamically create credentials and manage their lifecycle
  • Encrypt data - Provide encryption services without storing the data itself

Secrets Engine Categories

Cloud Secrets Engines

  • AWS - Dynamic IAM credentials
  • Azure - Service principal management
  • GCP - Service account key generation

Database Secrets Engines

  • PostgreSQL - Dynamic database credentials
  • MySQL/MariaDB - User provisioning
  • MongoDB - Role-based access
  • Microsoft SQL Server - Dynamic login creation

Internal Vault Engines

  • Key-Value (KV) - Static secret storage
  • Identity - Client tracking and entity management
  • Transit - Encryption as a service
  • Cubbyhole - Per-token isolated storage

Identity Source Engines

  • Active Directory - AD credential rotation
  • OpenLDAP - LDAP credential management

Certificate Engines

  • PKI - Internal Certificate Authority
  • SSH - SSH certificate generation

HashiCorp Product Engines

  • Consul - ACL token generation
  • Nomad - Token management
📊 Engine Count: Vault includes approximately 22+ different secrets engines, each designed for specific use cases. Check the official documentation for the complete and current list.

Special Engines: Identity & Cubbyhole

Identity Secrets Engine

The Identity engine is a special internal engine that maintains client information within Vault.

✅ Identity Engine Characteristics:
  • Enabled by default - Active immediately when Vault starts
  • Cannot be disabled - Required for Vault operations
  • Single instance only - No multiple instances allowed
  • Tracks authenticated clients - Maintains entity records

Identity Engine Components:

  • Entities - Represent clients that have authenticated to Vault
  • Aliases - Map authentication method identities to entities
  • Groups - Organize entities for policy management

Cubbyhole Secrets Engine

The Cubbyhole engine provides per-token isolated storage - a private storage area for each service token.

🔒 Cubbyhole Unique Properties:
  • Enabled by default - Automatically available
  • Cannot be disabled or moved - Fixed infrastructure component
  • Per service token allocation - Each token gets its own cubby
  • Token-exclusive access - Only the owning token can access
  • Root token has NO access - True token isolation
🎯 Key Concept:
The Cubbyhole is one of the rare instances where the root token has no special access. Each token's cubbyhole is completely isolated and accessible only by that specific token.

Static vs Dynamic Secrets

Static Secrets

Static secrets store existing data securely within Vault.

Characteristics:

  • Store existing data - Data already generated externally
  • Manual lifecycle - You control when secrets are updated
  • No automatic expiration - Secrets persist until manually deleted
  • Version control - Track changes over time (KV v2)
  • Access control via policies - Define who can read/write

Primary Engine: Key-Value (KV) Secrets Engine

Dynamic Secrets

Dynamic secrets are generated on-demand by Vault and automatically managed throughout their lifecycle.

Characteristics:

  • Generated on demand - Created when requested
  • Lease-based lifetime - Every secret has an expiration
  • Automatic lifecycle management - Vault handles creation and revocation
  • Renewable (optional) - Extend lifetime within max TTL
  • Integration with external systems - Vault manages credentials in target systems

Common Dynamic Engines: Database, AWS, Azure, GCP, Consul, Nomad

Database Secrets Engine Use Case

🎯 Globomantics Scenario:
Requirement: Database administrators need to provide application and developer access to MySQL database with dynamically generated, short-lived credentials.

Solution:
  1. Enable database secrets engine with MySQL plugin
  2. Configure roles for applications (longer TTL) and developers (shorter TTL)
  3. Assign policies granting access to respective roles
  4. Applications and developers request credentials when needed
  5. Vault automatically revokes credentials when leases expire

Key-Value Engine Deep Dive

KV Engine Overview

The Key-Value engine stores static data as key-value pairs at specified paths.

Concept: Path = Key (storage location)
Storage: Key-value pairs stored at that path

KV Version Comparison

Feature Version 1 Version 2
Versioning ❌ No versioning - overwrites on update ✅ Maintains configurable version history
Metadata ❌ No metadata tracking ✅ Full metadata support
Performance ⚡ Faster - fewer storage calls ⚠️ Slightly slower - additional metadata
Delete Behavior ❌ Permanent deletion ✅ Soft delete - recoverable
Upgrade Path ✅ Can upgrade to v2 ❌ Cannot downgrade to v1
Default Version ✅ Default when enabling ⚠️ Must specify during creation

KV Engine Use Case

🎯 Globomantics Scenario:
Requirement: Application developer needs secure storage for API keys with versioning support.

Requirements:
  • Secure location for API key storage
  • Version tracking for API keys
  • Access to previous versions
  • Vault does not generate keys (developer-managed)
Solution:
  1. Enable KV engine version 2
  2. Create policy granting developers access to their path
  3. Developers store and retrieve versioned API keys

KV Version 2 Commands

Write Secret:
vault kv put secret/apikeys/app1 token=xyz123

Read Secret:
vault kv get secret/apikeys/app1
vault kv get -version=1 secret/apikeys/app1

List Keys:
vault kv list secret/apikeys

Delete (Soft):
vault kv delete -versions=1 secret/apikeys/app1

Undelete:
vault kv undelete -versions=1 secret/apikeys/app1

Destroy (Permanent):
vault kv destroy -versions=1 secret/apikeys/app1

Metadata Operations:
vault kv metadata get secret/apikeys/app1
vault kv metadata delete secret/apikeys/app1

Transit Engine - Encryption as a Service

Transit Engine Overview

🔐 Critical Concept:
The Transit engine provides Encryption as a Service - it does NOT store any data submitted to it. Transit only manages encryption keys and performs cryptographic operations.

Transit Engine Capabilities

  • Encrypt/Decrypt - Symmetric encryption operations
  • Sign/Verify - Cryptographic signatures
  • Hash - Generate cryptographic hashes
  • Random Bytes - High-quality random data generation
  • Key Management - Vault manages encryption keys
  • Key Rotation - Automatic key rotation support

Transit Engine Use Case

🎯 Globomantics Scenario:
Requirement: Application needs to encrypt data before writing to object storage.

Challenges:
  • Object storage encryption not trusted or unavailable
  • Application generates data dynamically
  • Vault should not store application data
  • Need centralized key management
Solution:
  1. Enable Transit secrets engine
  2. Create encryption key for application
  3. Grant application access via policy
  4. Application encrypts data before storage
  5. Application decrypts data after retrieval
  6. Vault manages keys, never sees data

Transit Engine Commands

Enable Transit Engine:
vault secrets enable transit

Create Encryption Key:
vault write -f transit/keys/my-key

Encrypt Data:
vault write transit/encrypt/my-key plaintext=$(base64 <<< "secret data")

Decrypt Data:
vault write transit/decrypt/my-key ciphertext="vault:v1:..."

Rotate Key:
vault write -f transit/keys/my-key/rotate

Enabling & Configuring Engines

Secrets Engine Lifecycle

  1. Enable - Mount plugin instance at specified path
  2. Configure - Set engine-specific settings
  3. Tune - Adjust common mount settings
  4. Use - Interact with the engine
  5. Move (optional) - Change mount path if needed
  6. Disable - Remove engine and all data

Core Management Paths

System Mount Path: /sys/mounts
Enable Engine: POST /sys/mounts/:path
Tune Engine: POST /sys/mounts/:path/tune
Move Engine: POST /sys/remount
Disable Engine: DELETE /sys/mounts/:path

CLI Commands

List Engines:

vault secrets list

Enable Engine:

# Default path (engine name)
vault secrets enable kv

# Custom path
vault secrets enable -path=globokv kv

# Specific version
vault secrets enable -path=globokv -version=2 kv

Tune Engine:

vault secrets tune -description="Globomantics KV" globokv
vault secrets tune -default-lease-ttl=30m globokv
vault secrets tune -max-lease-ttl=2h globokv

Move Engine:

vault secrets move globokv globokv-v2
⚠️ Move Impact:
  • Lease revocation - All existing leases are revoked
  • Policy updates required - Paths change, policies must be updated

Disable Engine:

vault secrets disable globokv
🚨 Warning: Disabling a secrets engine permanently deletes all data stored by that engine. This action cannot be undone!

Configuration vs Tuning

Tuning - Common settings for all engines:

  • Description
  • Default TTL
  • Max TTL
  • Force no-cache
  • Audit non-HMAC request/response keys

Configuration - Engine-specific settings:

  • External service connection details
  • Plugin-specific parameters
  • Role definitions
  • Access patterns

Consul Secrets Engine Integration

Consul Engine Overview

The Consul secrets engine dynamically generates Consul ACL tokens based on predefined roles.

Setup Process

Step 1: Enable Consul Engine

vault secrets enable consul

Step 2: Configure Consul Connection

vault write consul/config/access \
    address=127.0.0.1:8500 \
    token=${CONSUL_HTTP_TOKEN}

Step 3: Create Role

vault write consul/roles/web \
    policies="web" \
    ttl=1h \
    max_ttl=2h

Step 4: Generate Credentials

vault read consul/creds/web

Complete Consul Integration Example

# Start Consul agent
consul agent -dev -config-dir=./config

# Bootstrap ACL system
consul acl bootstrap
export CONSUL_HTTP_TOKEN=<SecretID>

# Create Consul policy
consul acl policy create -name web -rules @web-policy.hcl

# Configure Vault
vault write consul/config/access \
    address=127.0.0.1:8500 \
    token=${CONSUL_HTTP_TOKEN}

# Create Vault role
vault write consul/roles/web \
    name=web \
    policies=web \
    ttl=3600 \
    max_ttl=7200

# Generate credentials
vault read consul/creds/web

Response Wrapping & Secure Distribution

Response Wrapping Concept

Response wrapping enables secure secret distribution without the intermediary knowing the secret value.

Problem Statement

⚠️ Challenge:
Control server needs to distribute secrets to worker cluster members. If the control server retrieves secrets and forwards them, the control server knows all secret values. If compromised, all secrets are exposed.

Response Wrapping Solution

✅ Solution Flow:
  1. Control server requests secret with wrapping
  2. Vault creates cubbyhole with secret copy
  3. Vault generates single-use wrapping token
  4. Control server receives only the token (not secret)
  5. Control server distributes token to worker
  6. Worker unwraps token to retrieve secret
  7. Token and cubbyhole automatically destroyed after use

Response Wrapping Benefits

  • Intermediary protection - Distribution systems never see secrets
  • Single-use tokens - Cannot be replayed
  • Automatic cleanup - Cubbyhole destroyed after retrieval
  • Audit trail - Wrapping token usage tracked
  • Time-limited - Wrapping tokens expire

Response Wrapping Commands

Wrap KV Secret:

vault kv get -wrap-ttl=30m secret/apikeys/app1

Wrap Dynamic Credentials:

vault read -wrap-ttl=30m consul/creds/web

Unwrap Secret:

vault unwrap <wrapping_token>

Inspect Wrapping Token:

vault token lookup <wrapping_token>

Wrapping Token Properties

🔑 Token Characteristics:
  • TTL - Set via -wrap-ttl parameter
  • Non-renewable - Cannot extend lifetime
  • Single-use - Destroyed after unwrapping
  • Service token - Batch tokens don't support cubbyholes
  • No max TTL extension - Explicit max TTL equals creation TTL

Complete Hands-On Lab

Lab Environment Setup

Start Vault Dev Server:
vault server -dev

Configure Environment:
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='<root_token>'

Login:
vault login ${VAULT_TOKEN}

Lab 1: KV Engine Configuration

Enable KV v2 Engine:

vault secrets enable -path=globokv -version=2 kv

Add Description:

vault secrets tune -description="Globomantics KV v2" globokv

Configure KV Settings:

# Check current config
vault read globokv/config

# Set max versions
vault write globokv/config max_versions=5

# Verify changes
vault read globokv/config

Store Secrets:

# Write initial version
vault kv put globokv/apitokens/d101 token=version1
vault kv put globokv/apitokens/d102 token=version1
vault kv put globokv/apitokens/d103 token=version1

# List keys
vault kv list globokv/apitokens

# Read secret
vault kv get globokv/apitokens/d101

# Update to version 2
vault kv put globokv/apitokens/d101 token=version2

# Read specific version
vault kv get -version=1 globokv/apitokens/d101
vault kv get -version=2 globokv/apitokens/d101

Version Management:

# Delete version (soft delete)
vault kv delete -versions=1 globokv/apitokens/d101

# Check metadata
vault kv metadata get globokv/apitokens/d101

# Undelete version
vault kv undelete -versions=1 globokv/apitokens/d101

# Destroy permanently
vault kv destroy -versions=1 globokv/apitokens/d101

# Delete entire key
vault kv metadata delete globokv/apitokens/d101

Lab 2: Consul Secrets Engine

Setup Consul:

# Create data directory
mkdir -p data

# Start Consul agent
consul agent -dev -config-dir=./config

# Bootstrap ACL (new terminal)
consul acl bootstrap
export CONSUL_HTTP_TOKEN=<SecretID>

# Create policy
consul acl policy create -name web -rules @web-policy.hcl

Configure Vault Consul Engine:

# Enable engine
vault secrets enable consul

# Configure connection
vault write consul/config/access \
    address=127.0.0.1:8500 \
    token=${CONSUL_HTTP_TOKEN}

# Create role
vault write consul/roles/web \
    name=web \
    policies=web \
    ttl=3600 \
    max_ttl=7200

# Generate credentials
vault read consul/creds/web

# Verify on Consul
consul acl token list

Lab 3: API Access

Read Secret via API:

# Get latest version
curl -H "X-Vault-Token: ${VAULT_TOKEN}" \
    ${VAULT_ADDR}/v1/globokv/data/apitokens/d102 | jq

# Get specific version
curl -H "X-Vault-Token: ${VAULT_TOKEN}" \
    "${VAULT_ADDR}/v1/globokv/data/apitokens/d102?version=1" | jq

Lab 4: Response Wrapping

# Wrap KV secret
vault kv get -wrap-ttl=30m globokv/apitokens/d102
# Returns: wrapping_token, wrapping_accessor, creation_time, ttl

# Lookup wrapping token
vault token lookup <wrapping_token>

# Unwrap secret
vault unwrap <wrapping_token>

# Try to lookup again (will fail - single use)
vault token lookup <wrapping_token>

# Wrap dynamic credentials
vault read -wrap-ttl=30m consul/creds/web

# Unwrap credentials
vault unwrap <wrapping_token>

Production Best Practices

🎯 Deployment Guidelines:
  1. Engine Selection - Choose appropriate engine for use case (static vs dynamic)
  2. Version Planning - KV v2 for versioning needs, v1 for performance
  3. TTL Configuration - Set reasonable defaults and maximums
  4. Response Wrapping - Use for secret distribution in automation
  5. Least Privilege - Grant minimal required permissions
  6. Regular Audits - Monitor engine usage and access patterns
  7. Backup Strategy - Regular backups before destructive operations
  8. Testing - Validate in dev before production deployment

Key Takeaways

  • Three Functions - Secrets engines store, generate, or encrypt data
  • Special Engines - Identity and Cubbyhole are unique, default-enabled engines
  • Static vs Dynamic - Choose based on lifecycle management needs
  • KV Versioning - Version 2 provides history, v1 provides performance
  • Transit for Encryption - Provides crypto without storing data
  • Dynamic Lifecycle - Vault manages creation and revocation automatically
  • Response Wrapping - Secure distribution without intermediary knowledge
  • Engine Mobility - Can move engines but impacts leases and policies

What's Next?

With secrets engines mastered, the next critical topic is lease management - understanding how Vault controls the lifetime of dynamically generated secrets and tokens, renewal strategies, and revocation patterns for production environments.

🎯 Practice Challenge:
Build a complete secrets management system:
  1. Deploy multiple KV engines for different application tiers
  2. Configure Consul engine with role-based access patterns
  3. Implement Transit engine for application encryption
  4. Design response wrapping workflow for CI/CD pipeline
  5. Create comprehensive policies for each engine
  6. Build monitoring for secret access patterns

Ready for enterprise secrets management? You now have the foundation to implement comprehensive secrets management with Vault - from simple key-value storage to dynamic credential generation and encryption services. Time to put these concepts into production!

By at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)