mazyar/tm_back
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor Scraper Configuration and Command Line Handling

Browse Source 
- Removed command line flag parsing for date range scraping in the scraper's main function.
- Introduced configuration fields for one-time scraping and date range (FromDate, ToDate) in the ScraperConfig struct.
- Updated the main function to utilize the new configuration fields, improving the clarity and maintainability of the scraping logic.
- This change enhances the scraper's functionality by centralizing configuration management and reducing command line complexity.
This commit is contained in:
Nima Nakhostin
2025-11-23 14:17:45 +03:30
parent afb32a9fa3
commit 10fd53af41
2 changed files with 10 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/scraper/bootstrap/config.go
+3
View File
@@ -24,4 +24,7 @@ type ScraperConfig struct {
DownloadDir string `env:"TED_DOWNLOAD_DIR" envDefault:"./downloads"`
CleanupAfter time.Duration `env:"TED_CLEANUP_AFTER" envDefault:"24h"`
ScrapingInterval string `env:"TED_SCRAPING_INTERVAL" envDefault:"* * 10 * * *"`
OneTime bool `env:"TED_ONE_TIME" envDefault:"false"`
FromDate string `env:"TED_FROM_DATE" envDefault:"02/01/2025"`
ToDate string `env:"TED_TO_DATE" envDefault:"13/09/2025"`
}