mazyar/tm_back
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update .gitignore, adjust TED scraper configuration, and enhance Tender entity structure

Browse Source 
- Added 'ted_samples/' to .gitignore to exclude TED sample files from version control.
- Updated the TED scraper configuration to extend the cleanup duration from 24 hours to 72 hours, allowing for better resource management.
- Enhanced the Tender entity by adding new fields: TenderDeadline, SubmissionDeadline, ApplicationDeadline, and SubmissionURL, improving the data model for tender management.
- Updated the TenderResponse structure to include the new fields, ensuring consistency in API responses.
- Implemented logic in the TED scraper to calculate submission and application deadlines based on tender deadlines, enhancing the scraping functionality.
This commit is contained in:
n.nakhostin
2025-08-25 13:13:40 +03:30
parent b26d77b4cf
commit f67e179ffc
5 changed files with 86 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/scraper/config.yaml
+1 -1
View File
@@ -24,5 +24,5 @@ ted:
user_agent: "TenderManagement-TED-Scraper/1.0"
max_concurrency: 2 # Reduced to avoid overwhelming the server
download_dir: "./tmp/ted_downloads"
cleanup_after: 24h
cleanup_after: 72h
scraping_interval: 12h # Run scraping every 12 hours