- Added a new field, ScrapedTEDNotices, to the StatisticsLifetimeTotals struct to track the total number of TED notices scraped.
- Updated the Statistics method in the statistics repository to include a background process for retrieving total scraped TED notices, improving the accuracy of dashboard statistics.
- Introduced new methods in the Counter to increment and retrieve daily counts for scraped TED notices, ensuring reliable metrics for reporting.
- Modified the TEDScraper to increment the TED notice scraped counter upon successful import, enhancing the tracking of scraping activity.
This update improves the dashboard's statistics by providing detailed insights into TED notice scraping activities, contributing to better data visibility and reporting.
- Updated the `AggregateProcurementLotEstimatedValue` function to reject mixed currencies and allow empty lot currencies, ensuring accurate aggregation of estimated values.
- Introduced new unit tests in `entity_test.go` to validate the behavior of the aggregation function under various scenarios, including mixed currencies and empty lot currencies.
- Refactored budget calculation in `budget_test.go` to utilize the new aggregation logic, improving consistency in budget retrieval.
This update improves the handling of estimated values in procurement lots, enhancing the reliability of the tender management system.
- Updated the `ResolvedEstimatedValueAndCurrency` method to aggregate procurement lot values when the tender-level estimated value is not set, improving accuracy in value retrieval.
- Introduced the `AggregateProcurementLotEstimatedValue` function to sum estimated values from procurement lots and return the first found currency.
- Modified the `ToResponseWithLanguage` method to utilize the new estimated value resolution logic.
- Added unit tests for the new functionality, ensuring correct behavior for various scenarios in the `entity_test.go` and `budget_test.go` files.
This update improves the handling of estimated values in tenders, enhancing the overall reliability of the tender management system.
- Added an entrypoint script to the scraper Dockerfile to handle one-time scraping with date range support.
- Introduced a new ContentXML field in the Notice entity to store XML content, improving data structure for notices.
- Updated the scraper logic to map XML content to the tender entity, enhancing data processing capabilities.
- Added command line flags for specifying a date range and a one-time scraping mode in the scraper's main function.
- Implemented validation for the date inputs to ensure both dates are provided and correctly formatted.
- Introduced a new RunOneTimeScraping function to handle the one-time scraping logic, including context management and error handling.
- Updated the TED scraper initialization to support both scheduled and one-time scraping modes, improving flexibility in data retrieval.
- Enhanced logging to provide clear feedback on the scraping process and any errors encountered.
- Updated the worker initialization process to include the new GLM SDK, enhancing the worker's capabilities for translation tasks.
- Modified the InitWorker function and NewNoticeWorker constructor to accept the GLM service, ensuring a cohesive integration.
- Implemented the GLM service initialization and logging for successful setup, improving maintainability and usability.
- Updated the NoticeWorker to utilize the GLM SDK for translating notice titles and descriptions, enhancing functionality and user experience.
- Enhanced the GetOJS function to include error handling for reading the response body and parsing CSV records, ensuring that errors are properly returned instead of being ignored.
- Added a check to ensure that the CSV records contain data, returning a meaningful error if the calendar is empty, which improves robustness and clarity in error reporting.
- Introduced AlertMail configuration in both scraper and worker bootstrap files, allowing for customizable email notifications.
- Updated the TED scraper to utilize the AlertMail configuration for sending completion notifications, improving flexibility in notification management.
- Enhanced error logging in the worker's main function to capture issues when listing Ollama models, ensuring better visibility into potential failures.
- Refactored notification sending logic to check for a valid AlertMail before dispatching emails, ensuring notifications are only sent when configured.
- Improved overall structure and readability of the bootstrap configuration files, aligning with best practices for maintainability.
- Replaced the tender repository with a new notice repository, encapsulating notice-related data access methods.
- Introduced the Notice entity to represent tender/contract notices, including relevant fields and methods for managing notice data.
- Updated the TED scraper to utilize the new notice repository for creating and managing notices, enhancing the integration with the tender management system.
- Implemented Ollama SDK initialization in the web bootstrap process, allowing for improved AI interactions.
- Enhanced the tender service to include Ollama SDK for additional functionality, ensuring a more robust service layer.
- Updated the tender repository to create a unique index on tender_id, enhancing database query performance.
- Introduced new TED mapping functions to convert parsed TED documents into tender entities, improving integration with the tender management system.
- Added error handling for contract notice processing in the TED scraper, ensuring robust logging and error management.
- Removed deprecated eform structures to streamline the codebase and focus on essential components.
- Updated the GetOJS function to accept a base URL parameter, improving flexibility in constructing the calendar URL.
- Removed the deprecated eform.go file, streamlining the codebase by eliminating unused components.
- Introduced new eform structures for BusinessRegistrationInformationNotice, ContractNotice, ContractAwardNotice, and PriorInformationNotice, enhancing the handling of TED XML data.
- Implemented mapping functions to convert eform notices to the Tender entity, improving integration with the tender management system.
- Added a service layer to encapsulate business logic related to eform processing, adhering to Clean Architecture principles.
- Introduced a new configuration file for the TED scraper, defining database connection settings, logging preferences, and scraping parameters.
- Implemented a CronScheduler to manage scheduled tasks for TED operations, utilizing the robfig/cron library for scheduling.
- Added new entities and structures for handling TED XML data, including eForms and tender-related information, enhancing the scraper's functionality.
- Updated the main application to initialize the TED scraper with the new configuration and set up graceful shutdown handling.
- Removed the deprecated scraping implementation to streamline the codebase and focus on the new architecture.
- Deleted the TED XML parser and scraper implementation files, including parser.go, scraper.go, and ted.go, to streamline the codebase and eliminate unused components.
- Commented out the initialization code in main.go to prevent execution while maintaining the structure for future reference.
- Removed the README.md for the TED package, which contained documentation for the now-deleted parser and scraper functionalities.
- Updated the Tender entity to remove redundant comments on Unix timestamp fields, enhancing code readability.
- Renamed the AwardedEntity type to Awarded for improved clarity and consistency in naming conventions.
- Removed the transformPriorInformationNoticeToContractNotice method from TEDParser, streamlining the parser's functionality and reducing legacy code.
- Added new fields to the Tender entity for handling cancellation, award, and suspension details, improving the entity's capability to manage tender statuses.
- Implemented a DetectNoticeStatus method in the TEDParser to identify the status of TED notices from XML data, enhancing the parser's functionality.
- Updated the ParseXML method to include notice status detection, ensuring accurate status representation in parsed documents.
- Introduced upsert logic in the TED scraper to handle tender records more effectively, allowing for updates or creation based on existing ContractIDs.
- Enhanced logging and error handling throughout the scraping process to improve traceability and robustness.
- Added a comprehensive usage example document to illustrate the new features and usage patterns for the TED XML parser and scraper.
- Updated the MongoDB driver from v1.17.4 to v2.3.0, ensuring compatibility with the latest features and improvements.
- Refactored company-related entity and repository files to utilize the new driver, replacing `primitive.ObjectID` with `bson.ObjectID` for ID handling.
- Adjusted the configuration in `.drone.yml` to include the `main` branch in the trigger settings, enhancing CI/CD pipeline flexibility.
- Added 'ted_samples/' to .gitignore to exclude TED sample files from version control.
- Updated the TED scraper configuration to extend the cleanup duration from 24 hours to 72 hours, allowing for better resource management.
- Enhanced the Tender entity by adding new fields: TenderDeadline, SubmissionDeadline, ApplicationDeadline, and SubmissionURL, improving the data model for tender management.
- Updated the TenderResponse structure to include the new fields, ensuring consistency in API responses.
- Implemented logic in the TED scraper to calculate submission and application deadlines based on tender deadlines, enhancing the scraping functionality.
- Changed the location code in the `generateTenderID` method from "EURO" to "EU" to accurately reflect the country code.
- This modification enhances the clarity and correctness of the tender ID generation process, ensuring adherence to Clean Architecture principles.
- Updated the `generateTenderID` method to improve the structure and clarity of the tender ID generation process.
- Changed the source identifier from "TED" to "1" for consistency.
- Modified the handling of the Notice Publication ID, ensuring it is used correctly in the tender ID.
- Simplified the buyer code logic by commenting out the previous implementation, focusing on the essential components.
- Adjusted the date and location code handling to enhance clarity and maintainability, ensuring adherence to Clean Architecture principles.
- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files.
- Added configuration management for the scraper, including a new `Config` struct and YAML configuration file.
- Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles.
- Implemented comprehensive logging and error handling throughout the scraper functionality.
- Updated API routes to include tender management operations, enhancing the overall system capabilities.
Mazyar