Refactor Configuration System to Use Environment Variables

- Removed YAML configuration files for the scraper and web commands, transitioning to an environment-based configuration system.
- Updated configuration structs to utilize `env` tags for loading values from `.env` files, enhancing security and flexibility.
- Implemented reflection-based parsing for environment variables, ensuring type-safe configuration loading.
- Revised documentation to reflect the new configuration approach, including usage examples and best practices for environment variable naming conventions.
This commit is contained in:
n.nakhostin
2025-09-09 18:19:42 +03:30
parent 44d0f57082
commit 57c29dc58f
6 changed files with 348 additions and 284 deletions
-28
View File
@@ -1,28 +0,0 @@
database:
mongodb:
uri: "mongodb://localhost:27017"
name: "tender_management"
timeout: 10s
max_pool_size: 50
logging:
level: "info"
format: "json"
output: "file" # stdout, stderr, file
file:
path: "./logs/scraper.log"
max_size: 100 # Max size in MB before rotation
max_backups: 5 # Number of backup files to keep
max_age: 30 # Max age in days to keep log files
compress: true # Compress rotated files
ted:
base_url: "https://ted.europa.eu/packages/daily"
timeout: 300s # Increased to 5 minutes for large files
max_retries: 5
retry_delay: 10s
user_agent: "TenderManagement-TED-Scraper/1.0"
max_concurrency: 2 # Reduced to avoid overwhelming the server
download_dir: "./tmp/ted_downloads"
cleanup_after: 72h
scraping_interval: 12h # Run scraping every 12 hours