Implement TED Scraper and Configuration Management
- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files. - Added configuration management for the scraper, including a new `Config` struct and YAML configuration file. - Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles. - Implemented comprehensive logging and error handling throughout the scraper functionality. - Updated API routes to include tender management operations, enhancing the overall system capabilities.
This commit is contained in:
@@ -0,0 +1,28 @@
|
||||
database:
|
||||
mongodb:
|
||||
uri: "mongodb://localhost:27017"
|
||||
name: "tender_management"
|
||||
timeout: 10s
|
||||
max_pool_size: 50
|
||||
|
||||
logging:
|
||||
level: "info"
|
||||
format: "json"
|
||||
output: "file" # stdout, stderr, file
|
||||
file:
|
||||
path: "./logs/scraper.log"
|
||||
max_size: 100 # Max size in MB before rotation
|
||||
max_backups: 5 # Number of backup files to keep
|
||||
max_age: 30 # Max age in days to keep log files
|
||||
compress: true # Compress rotated files
|
||||
|
||||
ted:
|
||||
base_url: "https://ted.europa.eu/packages/daily"
|
||||
timeout: 300s # Increased to 5 minutes for large files
|
||||
max_retries: 5
|
||||
retry_delay: 10s
|
||||
user_agent: "TenderManagement-TED-Scraper/1.0"
|
||||
max_concurrency: 2 # Reduced to avoid overwhelming the server
|
||||
download_dir: "./tmp/ted_downloads"
|
||||
cleanup_after: 24h
|
||||
scraping_interval: 12h # Run scraping every 12 hours
|
||||
Reference in New Issue
Block a user