Implement TED Scraper and Configuration Management

- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files.
- Added configuration management for the scraper, including a new `Config` struct and YAML configuration file.
- Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles.
- Implemented comprehensive logging and error handling throughout the scraper functionality.
- Updated API routes to include tender management operations, enhancing the overall system capabilities.
This commit is contained in:
n.nakhostin
2025-08-13 16:52:24 +03:30
parent b7fa012d13
commit 96c21b8b78
16 changed files with 6970 additions and 7 deletions
+28
View File
@@ -0,0 +1,28 @@
database:
mongodb:
uri: "mongodb://localhost:27017"
name: "tender_management"
timeout: 10s
max_pool_size: 50
logging:
level: "info"
format: "json"
output: "file" # stdout, stderr, file
file:
path: "./logs/scraper.log"
max_size: 100 # Max size in MB before rotation
max_backups: 5 # Number of backup files to keep
max_age: 30 # Max age in days to keep log files
compress: true # Compress rotated files
ted:
base_url: "https://ted.europa.eu/packages/daily"
timeout: 300s # Increased to 5 minutes for large files
max_retries: 5
retry_delay: 10s
user_agent: "TenderManagement-TED-Scraper/1.0"
max_concurrency: 2 # Reduced to avoid overwhelming the server
download_dir: "./tmp/ted_downloads"
cleanup_after: 24h
scraping_interval: 12h # Run scraping every 12 hours