[DEV-140033] retreive secrets based on dotenv files with sm:// paths

README.md

@@ -1,10 +1,15 @@
 # Snagsby [![Build Status](https://travis-ci.org/roverdotcom/snagsby.svg?branch=master)](https://travis-ci.org/roverdotcom/snagsby)
 
-Snagsby reads a JSON object from an S3 bucket and outputs the keys and values
-in a format that can be evaluated by a shell to set environment variables.
+Snagsby reads configuration and secrets from multiple sources and outputs them
+as environment variables in a format that can be evaluated by a shell.
 
-It's useful for reading configuration into environment variables from S3
-objects in Docker containers.
+**Supported sources:**
+- Local env files (`file://`) with dotenv format
+- AWS S3 JSON objects (`s3://`)
+- AWS Secrets Manager (`sm://`)
+
+It's useful for reading configuration and secrets into environment variables in
+Docker containers and other deployment scenarios.
 
 It can help with workflows like this one: https://blogs.aws.amazon.com/security/post/Tx2B3QUWAA7KOU/How-to-Manage-Secrets-for-Amazon-EC2-Container-Service-Based-Applications-by-Usi
 
@@ -55,10 +60,11 @@ export YES="1"
 You can supply sources in a comma delimited `SNAGSBY_SOURCE` environment variable:
 
 ```bash
-SNAGSBY_SOURCE="s3://my-bucket/secrets1.json, s3://my-bucket/secrets2.json" ./bin/snagsby
+SNAGSBY_SOURCE="file://base.snagsby,  s3://my-bucket/secrets.json" ./bin/snagsby
 
 # -e will fail on errors and exit 1
 ./bin/snagsby -e \
+  file://production.snagsby \
   s3://my-bucket/config.json \
   s3://my-bucket/config2.json
 ```
@@ -70,13 +76,103 @@ An example docker entrypoint may look like:
 
 set -e
 
+# Combine local config with remote secrets
 eval $(./bin/snagsby \
-  s3://my-bucket/config.json?region=us-west-2 \
-  s3://my-bucket/config-production.json?region-us-west-1)
+  file://config/base.snagsby \
+  file://config/production.snagsby \
+  s3://my-bucket/config.json?region=us-west-2)
 
 exec "$@"
 ```
 
+## Env File Format
+
+Snagsby supports reading environment variables from local files using the `file://` scheme with standard dotenv format.
+
+### Basic Usage
+
+```bash
+snagsby file://local.snagsby
+```
+
+### File Format
+
+Env files use the standard dotenv format (`KEY=VALUE`):
+
+```bash
+# Comments are supported
+DATABASE_URL=postgres://localhost:5432/mydb
+API_KEY=abc123
+DEBUG=true
+
+# Values can be quoted to preserve special characters
+MESSAGE="Hello # this is not a comment"
+PATH_WITH_SPACES="  /path/with/spaces  "
+
+# Empty values are allowed
+OPTIONAL_KEY=
+```
+
+### Secret References
+
+Values can reference AWS Secrets Manager using the `sm://` prefix:
+
+```bash
+# Direct values
+DATABASE_HOST=localhost
+DATABASE_PORT=5432
+
+# References to secrets in AWS Secrets Manager
+DATABASE_PASSWORD=sm://production/db/password
+API_SECRET=sm://production/api/secret
+```
+
+Snagsby will automatically fetch the secrets from AWS Secrets Manager and populate the environment variables with the actual values.
+
+### File Naming Conventions
+
+While Snagsby accepts any file extension, we recommend using extensions that clearly indicate the file contains **secret references**, not actual secrets:
+
+- `.snagsby` - Clear about the tool used to resolve the secrets
+- `.env.vault` - Suggests secrets/vault references
+- `.env.ref` - Short for "references"
+- `.envmap` - Conveys "mapping to secrets"
+
+It is recommended to **avoid using `.env`** for files with secret references, as it may give developers a false sense that the file is safe to commit with actual secrets or that the file will not be committed to the repository.
+
+### Multiple Sources
+
+You can combine multiple source types:
+
+```bash
+snagsby \
+  file://base.snagsby \
+  file://production.snagsby \
+  s3://my-bucket/config.json?region=us-west-2
+```
+
+### Validation and Key Handling
+
+**Env files require strict POSIX-compliant variable names:**
+
+Environment variable names in env files must follow shell naming conventions:
+- Start with a letter or underscore (`[a-zA-Z_]`)
+- Contain only letters, digits, and underscores (`[a-zA-Z0-9_]`)
+
+Invalid keys (e.g., `my-key` with dashes, `my.key` with dots, or `123key` starting with a digit) will be rejected with a clear error message.
+
+**Why strict validation?**
+
+Unlike other Snagsby resolvers (S3, Secrets Manager) that normalize arbitrary keys (e.g., converting `my-key` to `MY_KEY`), env files are meant to define actual shell environment variables. Since shells only accept POSIX-compliant names, we validate at parse time to catch errors early.
+
+**Key preservation:**
+
+Keys in env files are preserved exactly as written, including their case. For example:
+- `DATABASE_URL=...` stays as `DATABASE_URL` (not normalized to uppercase)
+- `lowercase_var=...` stays as `lowercase_var`
+
+This matches standard `.env` file behavior and ensures variables are set exactly as intended.
+
 ## AWS Configuration
 
 You can configure AWS any way the golang sdk supports:

pkg/connectors/secretsmanager.go

@@ -27,6 +27,9 @@ type SecretsManagerAPIClient interface {
 	GetSecretValueAPIClient
 }
 
+// SecretsManagerConnector provides methods for retrieving secrets from AWS Secrets Manager.
+// The struct fields are private to prevent direct instantiation outside this package.
+// Use NewSecretsManagerConnector to create instances.
 type SecretsManagerConnector struct {
 	secretsmanagerClient SecretsManagerAPIClient
 	source               *config.Source
@@ -40,6 +43,12 @@ func NewSecretsManagerConnector(source *config.Source) (*SecretsManagerConnector
 	return &SecretsManagerConnector{secretsmanagerClient: secretsManagerClient, source: source}, nil
 }
 
+// NewSecretsManagerConnectorWithClient creates a new SecretsManagerConnector with a custom API client.
+// This is primarily used for testing to inject a mock client. Production code should use NewSecretsManagerConnector instead.
+func NewSecretsManagerConnectorWithClient(client SecretsManagerAPIClient, source *config.Source) *SecretsManagerConnector {
+	return &SecretsManagerConnector{secretsmanagerClient: client, source: source}
+}
+
 func (sm *SecretsManagerConnector) getConcurrencyOrDefault(keyLength int) int {
 	// Pull concurrency settings
 	getConcurrency, hasSetting := os.LookupEnv("SNAGSBY_SM_CONCURRENCY")
@@ -72,7 +81,7 @@ func (sm *SecretsManagerConnector) fetchSecretValue(secretName string) (string,
 
 	getSecret, err := sm.secretsmanagerClient.GetSecretValue(ctx, input)
 	if err != nil {
-		return "", err
+		return "", fmt.Errorf("fetching secret %q: %w", secretName, err)
 	}
 
 	if getSecret.SecretString == nil {

pkg/connectors/secretsmanager_test.go

@@ -6,6 +6,7 @@ import (
 	"net/url"
 	"os"
 	"strconv"
+	"strings"
 	"sync/atomic"
 	"testing"
 	"time"
@@ -160,19 +161,19 @@ func TestGetSecretWithVersionID(t *testing.T) {
 // TestGetSecretErrors tests error handling in GetSecret
 func TestGetSecretErrors(t *testing.T) {
 	tests := []struct {
-		name          string
-		mockBehavior  mockAWSSecretsManagerBehavior
-		expectedError string
+		name               string
+		mockBehavior       mockAWSSecretsManagerBehavior
+		expectedErrorSubstring string
 	}{
 		{
-			name:          "access denied error",
-			mockBehavior:  errorBehavior(errors.New("access denied")),
-			expectedError: "access denied",
+			name:               "access denied error",
+			mockBehavior:       errorBehavior(errors.New("access denied")),
+			expectedErrorSubstring: "access denied",
 		},
 		{
-			name:          "secret not found error",
-			mockBehavior:  errorBehavior(errors.New("secret not found")),
-			expectedError: "secret not found",
+			name:               "secret not found error",
+			mockBehavior:       errorBehavior(errors.New("secret not found")),
+			expectedErrorSubstring: "secret not found",
 		},
 	}
 
@@ -187,8 +188,8 @@ func TestGetSecretErrors(t *testing.T) {
 
 			if err == nil {
 				t.Error("Expected error but got none")
-			} else if err.Error() != tt.expectedError {
-				t.Errorf("Expected error '%s', got '%s'", tt.expectedError, err.Error())
+			} else if !strings.Contains(err.Error(), tt.expectedErrorSubstring) {
+				t.Errorf("Expected error containing '%s', got '%s'", tt.expectedErrorSubstring, err.Error())
 			}
 		})
 	}

pkg/connectors/testing/mocks.go

@@ -1,5 +1,16 @@
 package testing
 
+import (
+	"context"
+	"fmt"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
+	"github.com/aws/aws-sdk-go-v2/service/secretsmanager/types"
+	"github.com/roverdotcom/snagsby/pkg/config"
+	"github.com/roverdotcom/snagsby/pkg/connectors"
+)
+
 // MockSecretsConnector is a reusable mock for any connector that implements secret retrieval methods.
 // It can be used by all resolvers (manifest, envfile, secretsmanager, etc.) that need to mock secret operations.
 //
@@ -45,3 +56,74 @@ func (m *MockSecretsConnector) ListSecrets(prefix string) ([]string, error) {
 	}
 	return []string{}, nil
 }
+
+// MockSecretsManagerAPIClient is a mock implementation of the AWS Secrets Manager API client.
+// This allows testing the full integration of Resolver + Connector with a mocked AWS SDK client.
+//
+// Usage example:
+//
+//	mock := &testing.MockSecretsManagerAPIClient{
+//		Secrets: map[string]string{
+//			"path/to/secret": "secret-value",
+//		},
+//	}
+type MockSecretsManagerAPIClient struct {
+	// Secrets maps secret names to their string values
+	Secrets map[string]string
+
+	// GetSecretValueFunc allows custom behavior for GetSecretValue
+	GetSecretValueFunc func(ctx context.Context, params *secretsmanager.GetSecretValueInput, optFns ...func(*secretsmanager.Options)) (*secretsmanager.GetSecretValueOutput, error)
+
+	// ListSecretsFunc allows custom behavior for ListSecrets
+	ListSecretsFunc func(ctx context.Context, params *secretsmanager.ListSecretsInput, optFns ...func(*secretsmanager.Options)) (*secretsmanager.ListSecretsOutput, error)
+}
+
+// GetSecretValue implements the GetSecretValueAPIClient interface.
+func (m *MockSecretsManagerAPIClient) GetSecretValue(ctx context.Context, params *secretsmanager.GetSecretValueInput, optFns ...func(*secretsmanager.Options)) (*secretsmanager.GetSecretValueOutput, error) {
+	if m.GetSecretValueFunc != nil {
+		return m.GetSecretValueFunc(ctx, params, optFns...)
+	}
+
+	if m.Secrets == nil {
+		return nil, fmt.Errorf("ResourceNotFoundException: secret not found")
+	}
+
+	secretName := aws.ToString(params.SecretId)
+	value, exists := m.Secrets[secretName]
+	if !exists {
+		return nil, fmt.Errorf("ResourceNotFoundException: secret %s not found", secretName)
+	}
+
+	return &secretsmanager.GetSecretValueOutput{
+		SecretString: aws.String(value),
+	}, nil
+}
+
+// ListSecrets implements the ListSecretsAPIClient interface.
+func (m *MockSecretsManagerAPIClient) ListSecrets(ctx context.Context, params *secretsmanager.ListSecretsInput, optFns ...func(*secretsmanager.Options)) (*secretsmanager.ListSecretsOutput, error) {
+	if m.ListSecretsFunc != nil {
+		return m.ListSecretsFunc(ctx, params, optFns...)
+	}
+
+	return &secretsmanager.ListSecretsOutput{
+		SecretList: []types.SecretListEntry{},
+	}, nil
+}
+
+// NewSecretsManagerConnectorWithFakeSecrets creates a SecretsManagerConnector with fake secrets for testing.
+// This is a convenience function that encapsulates the mock setup so tests don't need to know about the mock client implementation.
+//
+// Usage example:
+//
+//	connector := testing.NewSecretsManagerConnectorWithFakeSecrets(
+//		map[string]string{
+//			"path/to/secret": "secret-value",
+//		},
+//		source,
+//	)
+func NewSecretsManagerConnectorWithFakeSecrets(secrets map[string]string, source *config.Source) *connectors.SecretsManagerConnector {
+	mockClient := &MockSecretsManagerAPIClient{
+		Secrets: secrets,
+	}
+	return connectors.NewSecretsManagerConnectorWithClient(mockClient, source)
+}