Refactor Configuration System to Use Environment Variables
- Removed YAML configuration files for the scraper and web commands, transitioning to an environment-based configuration system. - Updated configuration structs to utilize `env` tags for loading values from `.env` files, enhancing security and flexibility. - Implemented reflection-based parsing for environment variables, ensuring type-safe configuration loading. - Revised documentation to reflect the new configuration approach, including usage examples and best practices for environment variable naming conventions.
This commit is contained in:
@@ -1,28 +0,0 @@
|
||||
database:
|
||||
mongodb:
|
||||
uri: "mongodb://localhost:27017"
|
||||
name: "tender_management"
|
||||
timeout: 10s
|
||||
max_pool_size: 50
|
||||
|
||||
logging:
|
||||
level: "info"
|
||||
format: "json"
|
||||
output: "file" # stdout, stderr, file
|
||||
file:
|
||||
path: "./logs/scraper.log"
|
||||
max_size: 100 # Max size in MB before rotation
|
||||
max_backups: 5 # Number of backup files to keep
|
||||
max_age: 30 # Max age in days to keep log files
|
||||
compress: true # Compress rotated files
|
||||
|
||||
ted:
|
||||
base_url: "https://ted.europa.eu/packages/daily"
|
||||
timeout: 300s # Increased to 5 minutes for large files
|
||||
max_retries: 5
|
||||
retry_delay: 10s
|
||||
user_agent: "TenderManagement-TED-Scraper/1.0"
|
||||
max_concurrency: 2 # Reduced to avoid overwhelming the server
|
||||
download_dir: "./tmp/ted_downloads"
|
||||
cleanup_after: 72h
|
||||
scraping_interval: 12h # Run scraping every 12 hours
|
||||
Reference in New Issue
Block a user