mazyar/tm_back
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor Configuration System to Use Environment Variables

Browse Source 
- Removed YAML configuration files for the scraper and web commands, transitioning to an environment-based configuration system.
- Updated configuration structs to utilize `env` tags for loading values from `.env` files, enhancing security and flexibility.
- Implemented reflection-based parsing for environment variables, ensuring type-safe configuration loading.
- Revised documentation to reflect the new configuration approach, including usage examples and best practices for environment variable naming conventions.
This commit is contained in:
n.nakhostin
2025-09-09 18:19:42 +03:30
parent 44d0f57082
commit 57c29dc58f
6 changed files with 348 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/config/README.md
+146 -108
View File
@@ -1,21 +1,22 @@
# Modular Configuration System
# Environment-Based Configuration System
This package provides a modular configuration system for the Tender Management backend. It allows each command (web, scraper, worker, etc.) to have its own specific configuration while sharing common base configurations.
This package provides a configuration system for the Tender Management backend that loads configuration exclusively from `.env` files. It maintains the same structure as the previous YAML-based system while using environment variables for all configuration values.
## Architecture
### Base Configuration (`BaseConfig`)
The `BaseConfig` struct contains common configuration fields that are used across all commands:
The configuration system provides common configuration structs that are used across all commands:
- **Server**: HTTP server configuration (host, port, timeouts)
- **Database**: MongoDB connection configuration
- **Cache**: Redis connection configuration
- **Logging**: Logging system configuration
- **ServerConfig**: HTTP server configuration (host, port, timeouts)
- **DatabaseConfig**: MongoDB connection configuration
- **CacheConfig**: Redis connection configuration
- **LoggingConfig**: Logging system configuration
- **RateLimitConfig**: Rate limiting configuration
### Command-Specific Configuration
Each command can extend the `BaseConfig` by embedding it and adding its own specific configuration fields.
Each command can extend the base configuration by embedding the common configs and adding their own specific configuration fields.
## Usage
@@ -33,14 +34,17 @@ import (
// Config holds configuration for your command
type Config struct {
config.BaseConfig
YourSpecific YourSpecificConfig `mapstructure:"your_specific"`
config.ServerConfig
config.DatabaseConfig
config.CacheConfig
config.LoggingConfig
YourSpecific YourSpecificConfig
}
type YourSpecificConfig struct {
SomeField string `mapstructure:"some_field"`
SomeTimeout time.Duration `mapstructure:"some_timeout"`
SomeNumber int `mapstructure:"some_number"`
SomeField string `env:"SOME_FIELD"`
SomeTimeout time.Duration `env:"SOME_TIMEOUT"`
SomeNumber int `env:"SOME_NUMBER"`
}
```
@@ -59,50 +63,45 @@ func initConfig() Config {
}
```
### 3. Create YAML Configuration File
### 3. Create .env Configuration File
Create a `config.yaml` file in your command directory:
Create a `.env` file in your command directory:
```yaml
# Base configuration (common to all commands)
server:
host: "localhost"
port: 8080
timeout: 30s
read_timeout: 10s
write_timeout: 10s
```env
# Server Configuration
SERVER_HOST=localhost
SERVER_PORT=8080
SERVER_TIMEOUT=30s
SERVER_READ_TIMEOUT=10s
SERVER_WRITE_TIMEOUT=10s
database:
mongodb:
uri: "mongodb://localhost:27017"
name: "your_database"
timeout: 10s
max_pool_size: 100
# Database Configuration
MONGODB_URI=mongodb://localhost:27017
MONGODB_NAME=your_database
MONGODB_TIMEOUT=10s
MONGODB_MAX_POOL_SIZE=100
cache:
redis:
host: "localhost"
port: 6379
password: ""
db: 0
pool_size: 10
# Cache Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
REDIS_POOL_SIZE=10
logging:
level: "info"
format: "json"
output: "stdout"
file:
path: "/var/log/tm/your-command.log"
max_size: 100
max_backups: 5
max_age: 30
compress: true
# Logging Configuration
LOG_LEVEL=info
LOG_FORMAT=json
LOG_OUTPUT=stdout
LOG_FILE_PATH=/var/log/tm/your-command.log
LOG_FILE_MAX_SIZE=100
LOG_FILE_MAX_BACKUPS=5
LOG_FILE_MAX_AGE=30
LOG_FILE_COMPRESS=true
# Your command-specific configuration
your_specific:
some_field: "value"
some_timeout: 30s
some_number: 42
SOME_FIELD=value
SOME_TIMEOUT=30s
SOME_NUMBER=42
```
## Examples
@@ -111,12 +110,29 @@ your_specific:
```go
type Config struct {
config.BaseConfig
Queue QueueConfig `mapstructure:"queue"`
UserAuthorization AuthConfig `mapstructure:"user_authorization"`
CustomerAuthorization AuthConfig `mapstructure:"customer_authorization"`
AI AIConfig `mapstructure:"ai"`
RateLimit RateLimitConfig `mapstructure:"rate_limiting"`
config.ServerConfig
config.DatabaseConfig
config.CacheConfig
config.LoggingConfig
config.RateLimitConfig
UserAuth AuthConfig
CustomerAuth AuthConfig
Assets AssetsConfig
}
type AuthConfig struct {
JWT JWTConfig
}
type JWTConfig struct {
AccessSecret string `env:"USER_AUTH_ACCESS_SECRET"`
RefreshSecret string `env:"USER_AUTH_REFRESH_SECRET"`
AccessExpiresIn int `env:"USER_AUTH_ACCESS_EXPIRES_IN"`
RefreshExpiresIn int `env:"USER_AUTH_REFRESH_EXPIRES_IN"`
}
type AssetsConfig struct {
FlagsPath string `env:"ASSETS_FLAGS_PATH"`
}
```
@@ -124,28 +140,21 @@ type Config struct {
```go
type Config struct {
config.BaseConfig
TED TEDConfig `mapstructure:"ted"`
config.DatabaseConfig
config.LoggingConfig
TED ScraperConfig
}
type TEDConfig struct {
BaseURL string `mapstructure:"base_url"`
MaxRetries int `mapstructure:"max_retries"`
MaxConcurrency int `mapstructure:"max_concurrency"`
DownloadDir string `mapstructure:"download_dir"`
CleanupAfter time.Duration `mapstructure:"cleanup_after"`
}
```
### Worker Command Config
```go
type Config struct {
config.BaseConfig
Queue QueueConfig `mapstructure:"queue"`
AI AIConfig `mapstructure:"ai"`
Worker WorkerConfig `mapstructure:"worker"`
Scraping ScrapingConfig `mapstructure:"scraping"`
type ScraperConfig struct {
BaseURL string `env:"TED_BASE_URL"`
Timeout time.Duration `env:"TED_TIMEOUT"`
MaxRetries int `env:"TED_MAX_RETRIES"`
RetryDelay time.Duration `env:"TED_RETRY_DELAY"`
UserAgent string `env:"TED_USER_AGENT"`
MaxConcurrency int `env:"TED_MAX_CONCURRENCY"`
DownloadDir string `env:"TED_DOWNLOAD_DIR"`
CleanupAfter time.Duration `env:"TED_CLEANUP_AFTER"`
ScrapingInterval time.Duration `env:"TED_SCRAPING_INTERVAL"`
}
```
@@ -157,58 +166,87 @@ The configuration loader uses Go generics to provide type-safe configuration loa
### Environment Variable Support
The configuration system automatically supports environment variable overrides via Viper's `AutomaticEnv()` feature.
The configuration system loads all values from `.env` files using the `github.com/subosito/gotenv` library.
### YAML Support
### Reflection-Based Parsing
Configuration files use YAML format for easy readability and maintenance.
The system uses reflection to automatically parse environment variables into struct fields based on the `env` tags.
### Validation
### Supported Types
Each command can implement its own validation logic after loading the configuration.
The configuration system supports the following Go types:
- `string`
- `int`, `int8`, `int16`, `int32`, `int64`
- `bool`
- `float32`, `float64`
- `time.Duration` (parsed from strings like "30s", "1h", "5m")
## Best Practices
1. **Keep BaseConfig minimal**: Only add fields to `BaseConfig` that are truly common across all commands.
1. **Use descriptive environment variable names**: Choose clear, descriptive names for your environment variables (e.g., `MONGODB_URI` instead of `DB_URI`).
2. **Use descriptive names**: Choose clear, descriptive names for your configuration structs and fields.
2. **Use consistent naming conventions**: Use uppercase with underscores for environment variable names (e.g., `SERVER_HOST`, `LOG_LEVEL`).
3. **Document your config**: Add comments to your configuration structs explaining what each field does.
3. **Group related variables**: Use prefixes to group related configuration (e.g., `MONGODB_*`, `REDIS_*`, `LOG_*`).
4. **Use appropriate types**: Use `time.Duration` for durations, proper numeric types for numbers, etc.
4. **Document your configuration**: Add comments in your `.env` files explaining what each variable does.
5. **Provide defaults**: Consider providing sensible defaults in your YAML files.
5. **Use appropriate types**: Use `time.Duration` for durations, proper numeric types for numbers, etc.
6. **Environment variables**: Use environment variables for sensitive data like API keys and passwords.
6. **Provide sensible defaults**: Consider providing sensible defaults in your code when environment variables are not set.
## Migration from Old System
## Migration from YAML System
If you're migrating from the old `infra.Config` system:
If you're migrating from the old YAML-based configuration system:
1. Remove imports of `tm/infra`
2. Add imports of `tm/pkg/config`
3. Change your config struct to embed `config.BaseConfig`
4. Update your `LoadConfig` call to use the new generic function
1. Remove `mapstructure` tags from your config structs
2. Add `env` tags to your config struct fields
3. Create `.env` files instead of `config.yaml` files
4. Update your `LoadConfig` calls to use the new system
5. Update function signatures to use the new config types
Example migration:
```go
// Old way
func initConfig() infra.Config {
config, err := infra.LoadConfig(".")
if err != nil {
panic(fmt.Sprintf("Failed to load config: %v", err))
}
return *config
// Old way (YAML)
type Config struct {
Server config.ServerConfig `mapstructure:"server"`
Database config.DatabaseConfig `mapstructure:"database"`
}
// New way
func initConfig() Config {
conf, err := config.LoadConfig(".", &Config{})
if err != nil {
panic(fmt.Sprintf("Failed to load config: %v", err))
}
return *conf
// New way (.env)
type Config struct {
Server config.ServerConfig
Database config.DatabaseConfig
}
```
// Old way (YAML loading)
conf, err := config.LoadConfig(".", &Config{})
// New way (.env loading)
conf, err := config.LoadConfig(".", &Config{})
```
## Environment Variable Naming Convention
The system follows these naming conventions:
- **Server**: `SERVER_*` (e.g., `SERVER_HOST`, `SERVER_PORT`)
- **Database**: `MONGODB_*` (e.g., `MONGODB_URI`, `MONGODB_NAME`)
- **Cache**: `REDIS_*` (e.g., `REDIS_HOST`, `REDIS_PORT`)
- **Logging**: `LOG_*` (e.g., `LOG_LEVEL`, `LOG_FORMAT`)
- **Rate Limiting**: `RATE_LIMIT_*` (e.g., `RATE_LIMIT_REQUESTS_PER_MINUTE`)
## Error Handling
The configuration system provides detailed error messages when:
- The `.env` file cannot be loaded
- Environment variables cannot be parsed into the expected types
- Required fields are missing (handled by your application logic)
## Security Considerations
1. **Never commit `.env` files**: Add `.env` to your `.gitignore` file
2. **Use environment-specific files**: Create different `.env` files for different environments (`.env.local`, `.env.production`, etc.)
3. **Validate sensitive values**: Always validate sensitive configuration values like API keys and database URIs
4. **Use secure defaults**: Provide secure default values for security-related configuration