tm_back/pkg/config/README.md

Environment-Based Configuration System

This package provides a configuration system for the Tender Management backend that loads configuration from both OS environment variables and .env files, with OS environment variables taking priority over .env file values. It maintains the same structure as the previous YAML-based system while using environment variables for all configuration values.

Priority System

The configuration loader follows this priority order:

1. OS Environment Variables (highest priority) - System-level environment variables
2. .env File Values (lower priority) - File-based configuration for development

This means that if you set SERVER_PORT=8080 in your .env file but have SERVER_PORT=9000 as an OS environment variable, the system will use 9000.

Architecture

Base Configuration (BaseConfig)

The configuration system provides common configuration structs that are used across all commands:

• ServerConfig: HTTP server configuration (host, port, timeouts)
• DatabaseConfig: MongoDB connection configuration
• CacheConfig: Redis connection configuration
• LoggingConfig: Logging system configuration
• RateLimitConfig: Rate limiting configuration

Command-Specific Configuration

Each command can extend the base configuration by embedding the common configs and adding their own specific configuration fields.

Usage

1. Create Command-Specific Config

Create a config file for your command (e.g., cmd/yourcommand/config.go):

package main

import (
    "time"
    "tm/pkg/config"
)

// Config holds configuration for your command
type Config struct {
    config.ServerConfig
    config.DatabaseConfig
    config.CacheConfig
    config.LoggingConfig
    YourSpecific YourSpecificConfig
}

type YourSpecificConfig struct {
    SomeField    string        `env:"SOME_FIELD"`
    SomeTimeout  time.Duration `env:"SOME_TIMEOUT"`
    SomeNumber   int           `env:"SOME_NUMBER"`
}

2. Load Configuration

In your command's main.go or bootstrap file:

func initConfig() Config {
    conf, err := config.LoadConfig(".", &Config{})
    if err != nil {
        panic(fmt.Sprintf("Failed to load config: %v", err))
    }
    
    return *conf
}

3. Configuration Sources

Option A: Using .env File Only (Development)

Create a .env file in your command directory:

# Server Configuration
SERVER_HOST=localhost
SERVER_PORT=8080
SERVER_TIMEOUT=30s
SERVER_READ_TIMEOUT=10s
SERVER_WRITE_TIMEOUT=10s

# Database Configuration
MONGODB_URI=mongodb://localhost:27017
MONGODB_NAME=your_database
MONGODB_TIMEOUT=10s
MONGODB_MAX_POOL_SIZE=100

# Cache Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
REDIS_POOL_SIZE=10

# Logging Configuration
LOG_LEVEL=info
LOG_FORMAT=json
LOG_OUTPUT=stdout
LOG_FILE_PATH=/var/log/tm/your-command.log
LOG_FILE_MAX_SIZE=100
LOG_FILE_MAX_BACKUPS=5
LOG_FILE_MAX_AGE=30
LOG_FILE_COMPRESS=true

# Your command-specific configuration
SOME_FIELD=value
SOME_TIMEOUT=30s
SOME_NUMBER=42

Option B: Using OS Environment Variables (Production)

# Set environment variables directly
export SERVER_HOST=0.0.0.0
export SERVER_PORT=8080
export MONGODB_URI=mongodb://mongo:27017
export MONGODB_NAME=production_db
export LOG_LEVEL=error

# Run your application
./your-command

Option C: Mixed Approach (Recommended)

1. Create a .env file with development defaults
2. Override specific values with OS environment variables for production:

# .env file contains development defaults
# Override critical values with OS env vars
export MONGODB_URI=mongodb://production-server:27017
export MONGODB_NAME=production_db
export LOG_LEVEL=error

# Run application - will use .env defaults except for overridden values
./your-command

Priority Example

Given this .env file:

SERVER_HOST=localhost
SERVER_PORT=8080
LOG_LEVEL=debug

And these OS environment variables:

export SERVER_PORT=9000
export LOG_LEVEL=error

The final configuration will be:

• SERVER_HOST=localhost (from .env file, no OS override)
• SERVER_PORT=9000 (OS environment variable overrides .env)
• LOG_LEVEL=error (OS environment variable overrides .env)

Advanced Features

Checking Configuration Source

You can check whether a configuration value came from OS environment or .env file:

import "tm/pkg/config"

func main() {
    value, isFromOS := config.GetEnvWithPriority("SERVER_PORT")
    if isFromOS {
        fmt.Printf("SERVER_PORT=%s (from OS environment)\n", value)
    } else {
        fmt.Printf("SERVER_PORT=%s (from .env file)\n", value)
    }
}

Graceful .env File Handling

The system gracefully handles missing .env files:

• If .env file exists: loads values as defaults
• If .env file is missing: continues with OS environment variables only
• No errors are thrown for missing .env files

Examples

Web Command Config

type Config struct {
    config.ServerConfig
    config.DatabaseConfig
    config.CacheConfig
    config.LoggingConfig
    config.RateLimitConfig
    UserAuth     AuthConfig
    CustomerAuth AuthConfig
    Assets       AssetsConfig
}

type AuthConfig struct {
    JWT JWTConfig
}

type JWTConfig struct {
    AccessSecret     string `env:"USER_AUTH_ACCESS_SECRET"`
    RefreshSecret    string `env:"USER_AUTH_REFRESH_SECRET"`
    AccessExpiresIn  int    `env:"USER_AUTH_ACCESS_EXPIRES_IN"`
    RefreshExpiresIn int    `env:"USER_AUTH_REFRESH_EXPIRES_IN"`
}

type AssetsConfig struct {
    FlagsPath string `env:"ASSETS_FLAGS_PATH"`
}

Scraper Command Config

type Config struct {
    config.DatabaseConfig
    config.LoggingConfig
    TED ScraperConfig
}

type ScraperConfig struct {
    BaseURL          string        `env:"TED_BASE_URL"`
    Timeout          time.Duration `env:"TED_TIMEOUT"`
    MaxRetries       int           `env:"TED_MAX_RETRIES"`
    RetryDelay       time.Duration `env:"TED_RETRY_DELAY"`
    UserAgent        string        `env:"TED_USER_AGENT"`
    MaxConcurrency   int           `env:"TED_MAX_CONCURRENCY"`
    DownloadDir      string        `env:"TED_DOWNLOAD_DIR"`
    CleanupAfter     time.Duration `env:"TED_CLEANUP_AFTER"`
    ScrapingInterval time.Duration `env:"TED_SCRAPING_INTERVAL"`
}

Features

Type Safety

The configuration loader uses Go generics to provide type-safe configuration loading. The function signature ensures that you get back exactly the type you expect.

Environment Variable Priority

The configuration system loads values with explicit priority:

1. OS environment variables (system-level, highest priority)
2. .env file values (file-based, development defaults)

Reflection-Based Parsing

The system uses reflection to automatically parse environment variables into struct fields based on the env tags.

Supported Types

The configuration system supports the following Go types:

• string
• int, int8, int16, int32, int64
• bool
• float32, float64
• time.Duration (parsed from strings like "30s", "1h", "5m")

Best Practices

1. Use descriptive environment variable names: Choose clear, descriptive names for your environment variables (e.g., MONGODB_URI instead of DB_URI).

2. Use consistent naming conventions: Use uppercase with underscores for environment variable names (e.g., SERVER_HOST, LOG_LEVEL).

3. Group related variables: Use prefixes to group related configuration (e.g., MONGODB_*, REDIS_*, LOG_*).

4. Document your configuration: Add comments in your .env files explaining what each variable does.

5. Use appropriate types: Use time.Duration for durations, proper numeric types for numbers, etc.

6. Provide sensible defaults in .env: Use .env files to provide development defaults and override with OS environment variables for production.

7. Leverage the priority system: Use .env for development defaults and OS environment variables for production overrides.

8. Keep secrets in OS environment variables: Never put sensitive values like API keys or passwords in .env files; use OS environment variables instead.

Migration from YAML System

If you're migrating from the old YAML-based configuration system:

1. Remove mapstructure tags from your config structs
2. Add env tags to your config struct fields
3. Create .env files instead of config.yaml files
4. Update your LoadConfig calls to use the new system
5. Update function signatures to use the new config types

Example migration:

// Old way (YAML)
type Config struct {
    Server config.ServerConfig `mapstructure:"server"`
    Database config.DatabaseConfig `mapstructure:"database"`
}

// New way (.env)
type Config struct {
    Server config.ServerConfig
    Database config.DatabaseConfig
}

// Old way (YAML loading)
conf, err := config.LoadConfig(".", &Config{})

// New way (.env loading) - same API!
conf, err := config.LoadConfig(".", &Config{})

Environment Variable Naming Convention

The system follows these naming conventions:

• Server: SERVER_* (e.g., SERVER_HOST, SERVER_PORT)
• Database: MONGODB_* (e.g., MONGODB_URI, MONGODB_NAME)
• Cache: REDIS_* (e.g., REDIS_HOST, REDIS_PORT)
• Logging: LOG_* (e.g., LOG_LEVEL, LOG_FORMAT)
• Rate Limiting: RATE_LIMIT_* (e.g., RATE_LIMIT_REQUESTS_PER_MINUTE)

Error Handling

The configuration system provides detailed error messages when:

• The .env file exists but cannot be loaded
• Environment variables cannot be parsed into the expected types
• Required fields are missing (handled by your application logic)

Note: Missing .env files do not cause errors - the system will continue with OS environment variables only.

Security Considerations

1. Never commit .env files with secrets: Add .env to your .gitignore file and never include sensitive values
2. Use OS environment variables for production secrets: Set sensitive values like API keys and database passwords as OS environment variables
3. Use environment-specific files: Create different .env files for different environments (.env.local, .env.development, etc.)
4. Validate sensitive values: Always validate sensitive configuration values like API keys and database URIs
5. Use secure defaults: Provide secure default values for security-related configuration
6. Leverage priority system for security: Keep development defaults in .env and override with secure OS environment variables in production

Deployment Strategies

Development

# Use .env file for all configuration
./your-command

Staging

# Use .env for defaults, override key values
export MONGODB_URI=mongodb://staging-db:27017
export LOG_LEVEL=warn
./your-command

Production

# Override all critical values with OS environment variables
export MONGODB_URI=mongodb://prod-cluster:27017
export MONGODB_NAME=production
export LOG_LEVEL=error
export SERVER_HOST=0.0.0.0
./your-command

Docker

# Set production values in Dockerfile or docker-compose
ENV MONGODB_URI=mongodb://mongo:27017
ENV LOG_LEVEL=info
ENV SERVER_HOST=0.0.0.0