- Added an entrypoint script to the scraper Dockerfile to handle one-time scraping with date range support.
- Introduced a new ContentXML field in the Notice entity to store XML content, improving data structure for notices.
- Updated the scraper logic to map XML content to the tender entity, enhancing data processing capabilities.
- Added command line flags for specifying a date range and a one-time scraping mode in the scraper's main function.
- Implemented validation for the date inputs to ensure both dates are provided and correctly formatted.
- Introduced a new RunOneTimeScraping function to handle the one-time scraping logic, including context management and error handling.
- Updated the TED scraper initialization to support both scheduled and one-time scraping modes, improving flexibility in data retrieval.
- Enhanced logging to provide clear feedback on the scraping process and any errors encountered.
- Updated the worker initialization process to include the new GLM SDK, enhancing the worker's capabilities for translation tasks.
- Modified the InitWorker function and NewNoticeWorker constructor to accept the GLM service, ensuring a cohesive integration.
- Implemented the GLM service initialization and logging for successful setup, improving maintainability and usability.
- Updated the NoticeWorker to utilize the GLM SDK for translating notice titles and descriptions, enhancing functionality and user experience.
- Enhanced the GetOJS function to include error handling for reading the response body and parsing CSV records, ensuring that errors are properly returned instead of being ignored.
- Added a check to ensure that the CSV records contain data, returning a meaningful error if the calendar is empty, which improves robustness and clarity in error reporting.
- Introduced AlertMail configuration in both scraper and worker bootstrap files, allowing for customizable email notifications.
- Updated the TED scraper to utilize the AlertMail configuration for sending completion notifications, improving flexibility in notification management.
- Enhanced error logging in the worker's main function to capture issues when listing Ollama models, ensuring better visibility into potential failures.
- Refactored notification sending logic to check for a valid AlertMail before dispatching emails, ensuring notifications are only sent when configured.
- Improved overall structure and readability of the bootstrap configuration files, aligning with best practices for maintainability.
- Replaced the tender repository with a new notice repository, encapsulating notice-related data access methods.
- Introduced the Notice entity to represent tender/contract notices, including relevant fields and methods for managing notice data.
- Updated the TED scraper to utilize the new notice repository for creating and managing notices, enhancing the integration with the tender management system.
- Implemented Ollama SDK initialization in the web bootstrap process, allowing for improved AI interactions.
- Enhanced the tender service to include Ollama SDK for additional functionality, ensuring a more robust service layer.
- Updated the tender repository to create a unique index on tender_id, enhancing database query performance.
- Introduced new TED mapping functions to convert parsed TED documents into tender entities, improving integration with the tender management system.
- Added error handling for contract notice processing in the TED scraper, ensuring robust logging and error management.
- Removed deprecated eform structures to streamline the codebase and focus on essential components.
- Updated the GetOJS function to accept a base URL parameter, improving flexibility in constructing the calendar URL.
- Removed the deprecated eform.go file, streamlining the codebase by eliminating unused components.
- Introduced new eform structures for BusinessRegistrationInformationNotice, ContractNotice, ContractAwardNotice, and PriorInformationNotice, enhancing the handling of TED XML data.
- Implemented mapping functions to convert eform notices to the Tender entity, improving integration with the tender management system.
- Added a service layer to encapsulate business logic related to eform processing, adhering to Clean Architecture principles.
- Introduced a new configuration file for the TED scraper, defining database connection settings, logging preferences, and scraping parameters.
- Implemented a CronScheduler to manage scheduled tasks for TED operations, utilizing the robfig/cron library for scheduling.
- Added new entities and structures for handling TED XML data, including eForms and tender-related information, enhancing the scraper's functionality.
- Updated the main application to initialize the TED scraper with the new configuration and set up graceful shutdown handling.
- Removed the deprecated scraping implementation to streamline the codebase and focus on the new architecture.
- Deleted the TED XML parser and scraper implementation files, including parser.go, scraper.go, and ted.go, to streamline the codebase and eliminate unused components.
- Commented out the initialization code in main.go to prevent execution while maintaining the structure for future reference.
- Removed the README.md for the TED package, which contained documentation for the now-deleted parser and scraper functionalities.
- Updated the Tender entity to remove redundant comments on Unix timestamp fields, enhancing code readability.
- Renamed the AwardedEntity type to Awarded for improved clarity and consistency in naming conventions.
- Removed the transformPriorInformationNoticeToContractNotice method from TEDParser, streamlining the parser's functionality and reducing legacy code.
- Added new fields to the Tender entity for handling cancellation, award, and suspension details, improving the entity's capability to manage tender statuses.
- Implemented a DetectNoticeStatus method in the TEDParser to identify the status of TED notices from XML data, enhancing the parser's functionality.
- Updated the ParseXML method to include notice status detection, ensuring accurate status representation in parsed documents.
- Introduced upsert logic in the TED scraper to handle tender records more effectively, allowing for updates or creation based on existing ContractIDs.
- Enhanced logging and error handling throughout the scraping process to improve traceability and robustness.
- Added a comprehensive usage example document to illustrate the new features and usage patterns for the TED XML parser and scraper.
- Updated the MongoDB driver from v1.17.4 to v2.3.0, ensuring compatibility with the latest features and improvements.
- Refactored company-related entity and repository files to utilize the new driver, replacing `primitive.ObjectID` with `bson.ObjectID` for ID handling.
- Adjusted the configuration in `.drone.yml` to include the `main` branch in the trigger settings, enhancing CI/CD pipeline flexibility.
- Added 'ted_samples/' to .gitignore to exclude TED sample files from version control.
- Updated the TED scraper configuration to extend the cleanup duration from 24 hours to 72 hours, allowing for better resource management.
- Enhanced the Tender entity by adding new fields: TenderDeadline, SubmissionDeadline, ApplicationDeadline, and SubmissionURL, improving the data model for tender management.
- Updated the TenderResponse structure to include the new fields, ensuring consistency in API responses.
- Implemented logic in the TED scraper to calculate submission and application deadlines based on tender deadlines, enhancing the scraping functionality.
- Changed the location code in the `generateTenderID` method from "EURO" to "EU" to accurately reflect the country code.
- This modification enhances the clarity and correctness of the tender ID generation process, ensuring adherence to Clean Architecture principles.
- Updated the `generateTenderID` method to improve the structure and clarity of the tender ID generation process.
- Changed the source identifier from "TED" to "1" for consistency.
- Modified the handling of the Notice Publication ID, ensuring it is used correctly in the tender ID.
- Simplified the buyer code logic by commenting out the previous implementation, focusing on the essential components.
- Adjusted the date and location code handling to enhance clarity and maintainability, ensuring adherence to Clean Architecture principles.
- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files.
- Added configuration management for the scraper, including a new `Config` struct and YAML configuration file.
- Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles.
- Implemented comprehensive logging and error handling throughout the scraper functionality.
- Updated API routes to include tender management operations, enhancing the overall system capabilities.
Mazyar