- Introduced `documents_scraped_filter.go` to handle filtering tenders based on the presence of documents in MinIO, utilizing a caching mechanism to optimize performance.
- Implemented `enrichDocumentsScrapedFilter` and `cachedContractFolderIDsWithDocuments` methods in the `tenderService` for efficient retrieval of contract folder IDs with documents.
- Added unit tests in `documents_scraped_filter_test.go` to validate the new filtering logic and caching behavior, ensuring accurate results and error handling.
- Updated `SearchForm` documentation to clarify the caching aspect of `ContractFolderIDsWithDocuments`.
- Enhanced comments in the handler and repository to reflect the new filtering logic and its implications.
This update improves the tender search functionality by efficiently managing document presence checks, leading to better performance and user experience in the tender management system.
- Introduced a new `widgetCache` structure to manage caching for various dashboard widgets, improving performance and reducing redundant data retrieval.
- Updated the `ClosingSoon` and `Trend` methods in the service to utilize the new caching mechanism, ensuring efficient data access and reducing load on the database.
- Enhanced the `NoticeTypes` and `Countries` methods to implement caching, further optimizing the dashboard's responsiveness.
- Added a new `searchListCache` implementation in the tender service to cache search results, improving the efficiency of tender searches and reducing database load.
- Implemented robust error handling and logging for cache operations, ensuring better visibility into potential issues.
This update significantly enhances the performance and scalability of the dashboard and tender services by integrating effective caching strategies, leading to improved user experience and resource management.
- Eliminated the `ContentXML` field from the `Tender` struct in the entity definition, streamlining the data model.
- Removed the `FindTendersWithContentXML` method from the `TenderRepository` interface, as it is no longer necessary.
- Updated the `ToTender` method in the `NoticeWorker` to reflect these changes, ensuring consistency across the codebase.
This update simplifies the tender data structure and repository interface, improving maintainability and clarity in the tender management system.
- Introduced a new index `documents_scraped_created_at_id_idx` in the tender repository to support keyset pagination for documents marked as scraped, sorted by creation date.
- This enhancement improves the efficiency of querying scraped documents, facilitating better performance in data retrieval and management.
This update significantly optimizes the tender repository's handling of scraped documents, enhancing overall data access capabilities.
- Updated the tender search handler to include a new `include_total` query parameter, allowing clients to request total count and pagination metadata for search results.
- Refactored the `tenderSearchListProjection` to exclude heavy fields from the list response, optimizing data retrieval.
- Modified the `buildSearchFilter` method to utilize the `processing_metadata.documents_scraped` flag for filtering, improving search accuracy.
- Added unit tests for the new pagination features and search filter logic, ensuring robust validation of the search functionality.
This update significantly enhances the tender search experience by providing more flexible pagination options and improving the efficiency of data handling.
- Introduced `PickAssignedCompanyID` function to determine the appropriate company ID for company-scoped operations, improving company assignment logic.
- Added `ResolveMongoID` method in the tender service to map MongoDB IDs and AI procedure references to canonical tender IDs, enhancing ID resolution.
- Updated `ToggleTenderApproval` to handle company ID validation and improve error handling, ensuring robust processing of tender approvals.
- Enhanced logging for error scenarios in the tender approval service, providing better insights into failures during operations.
- Refactored the `GetByID` and `GetByTenderAndCompany` methods in the tender approval repository to utilize custom error types for improved clarity.
This update significantly improves the handling of company and tender operations, enhancing error management and overall service reliability.
- Updated the `markResolvedRecommendedTenderSeen` function to accept an additional `tenderRef` parameter, allowing for better tracking of tender references and preventing duplicates.
- Refactored the `GetByProcedureReferences` method in the repository to combine indexed notice lookups with contract-folder candidates, ensuring older notice references resolve correctly after tender merges.
- Introduced a new `findTendersByContractFolderIDs` method to streamline the retrieval of tenders by contract folder IDs, enhancing performance and maintainability.
- Added unit tests for new functionality in `ai_reference_test.go` and `recommendation_filter_test.go`, ensuring robust validation of tender reference mapping and deduplication logic.
This update significantly improves the handling of tender references and enhances the overall test coverage, ensuring more reliable and efficient processing of tender data.
- Introduced a context with a timeout for Redis cache retrieval in the `getCachedAIRecommendations` method, ensuring more robust handling of potential delays.
- Added a new file `recommendation_page.go` to encapsulate the logic for building recommendation pages, improving code organization and readability.
- Implemented a `buildRecommendationPage` method to handle pagination and batch processing of recommendations, enhancing performance and user experience.
- Created a new test file `recommendation_page_test.go` to validate the functionality of the recommendation page building process, ensuring correct pagination and batch resolution.
This update significantly improves the efficiency and maintainability of the tender recommendation service by optimizing caching and implementing structured pagination handling.
- Renamed the test function to `TestBuildRecommendedTenderListResponsesPreservesOrderWithoutStorage` for clarity and updated its logic to reflect changes in the recommendation response building process.
- Removed unnecessary sleep in the test to enhance performance.
- Refactored the `buildRecommendedTenderResponsesParallel` method to `buildRecommendedTenderListResponses`, simplifying the response building without relying on MinIO translation lookups.
- Updated the repository's `mapProcedureReferencesToTenders` function to improve efficiency by grouping candidates by their contract folder IDs, enhancing the mapping process.
This update streamlines the recommendation response handling and enhances the test suite for better maintainability and performance.
- Introduced a new test function `TestMapProcedureReferencesToTenders` to validate the mapping of procedure references to corresponding tenders.
- Enhanced the `GetByProcedureReferences` method in the tender repository to utilize a more efficient filtering mechanism based on unique contract folder IDs.
- Updated pagination logic to dynamically adjust limits based on the number of unique folders and references, ensuring optimal data retrieval.
This update improves the testing coverage for tender mapping functionality and optimizes the repository's data fetching strategy, enhancing overall performance and reliability.
- Updated the ListPendingTenders method to filter tenders based on their active publication submission window instead of just deadlines, enhancing the accuracy of tender listings.
- Introduced a new HasActivePublicationWindow method in the Tender entity to determine if the publication-based submission window is still open.
- Modified the GetTenderByNoticeID method to reflect the new logic for filtering tenders based on their publication status.
- Enhanced the documentation for relevant API endpoints to clarify the changes in tender retrieval criteria.
This update improves the precision of tender scraping by ensuring only relevant tenders with active submission windows are processed, contributing to better data management and scraping efficiency.
- Updated the ListPaginationOptions struct to include SkipCount and IncludeCount fields, allowing for more flexible pagination behavior.
- Modified the BuildListPagination function to handle cursor pagination with count options, improving performance by running count queries in parallel with data retrieval.
- Enhanced the FindAll method in the repository to support concurrent counting of documents, reducing overall latency for list operations.
- Added tests for pagination behavior, ensuring accurate handling of count scenarios in both offset and cursor pagination.
This update improves the efficiency and flexibility of pagination in the MongoDB repository, enhancing the overall performance of list operations.
- Introduced `ProcedureReference` struct to encapsulate AI procedure coordinates for better data management.
- Added `GetByProcedureReferences` method in the `TenderRepository` to retrieve multiple tenders based on AI procedure references in a single query.
- Updated the `resolveRecommendedTenders` method in the service layer to utilize the new repository method, improving efficiency in fetching recommended tenders.
- Enhanced error handling and logging for the new repository method to ensure robust operation.
This update improves the handling of AI procedure references, streamlining tender retrieval processes and enhancing overall system performance.
- Introduced `FormatAIProcedureRef` and `ParseAIProcedureRef` functions in the `tender` domain for handling AI service tender references.
- Added unit tests for these functions in `ai_reference_test.go` to ensure correct parsing and formatting behavior.
- Updated the `TenderResponse` struct to include a new `ProcedureRef` field for improved data representation.
- Enhanced the `GetByProcedureReference` method in the repository to retrieve tenders based on the new procedure reference format.
- Modified the `Recommend` method in the service layer to utilize the new procedure reference handling, improving the recommendation process.
This update enhances the handling of AI procedure references, ensuring better data integrity and usability in the tender management system.
- Updated the document scraper service to include a new ScrapePortalsProvider interface, allowing for dynamic retrieval of supported scraping portals.
- Modified the ListPendingTenders and GetTenderByNoticeID methods to filter tenders based on document URLs that match the configured portals.
- Introduced new error handling for cases when the scrape portals provider is not configured, returning appropriate service unavailable responses.
- Enhanced API documentation to reflect changes in tender retrieval logic and added error response details for unsupported portal scenarios.
This update improves the document scraping functionality by integrating AI portal support, enhancing the overall reliability and flexibility of the tender management system.
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.
This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
- Added `NoticePublicationID` field to the `SearchForm` for filtering tenders by TED notice publication ID.
- Updated Swagger documentation in the handler methods to include the new `notice_publication_id` parameter for relevant endpoints.
- Modified the repository's search filter to incorporate the new `NoticePublicationID` field, allowing for more precise search queries.
This update improves the search capabilities within the tender management system, enhancing user experience and data retrieval accuracy.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Updated the AI pipeline service to include a new `ScrapedDocumentMetadataSyncer` interface for persisting scraped document metadata onto tender records.
- Modified the `NewService` function to accept the new metadata syncer dependency.
- Implemented synchronization of scraped document metadata in the `ScrapeDocuments` and `Run` methods.
- Enhanced the tender service to enrich search filters based on scraped documents and added a new method for syncing scraped documents from storage.
- Updated the `SearchForm` to include `ContractFolderIDsWithDocuments` for better handling of scraped documents in queries.
This update improves the integration of scraped document handling within the AI pipeline, enhancing data consistency and operational efficiency in the tender management system.
- Removed the QueueConfig structure and related queue management files as they are no longer in use.
- Updated the worker initialization to reflect the removal of queue-related configurations.
- Cleaned up the bootstrap process by eliminating deprecated logging related to the queue system.
This update streamlines the worker's configuration and prepares the codebase for future enhancements without the legacy queue management components.
- Refactored the NoticeWorker to generate unique tender IDs using the PBL naming convention (SCDYYNNN format) and project names based on client and opportunity details.
- Introduced methods for generating tender IDs and project names, including logic for extracting client and opportunity names, ensuring consistency and clarity in naming.
- Updated the Tender entity to include a new ProjectName field, enhancing the data structure for better project identification.
- Added MongoDB indexing for the new project_name field to optimize query performance.
- Improved error handling and logging during the tender ID generation process, ensuring robustness in the worker's functionality.
- Updated the tender repository to create a unique index on tender_id, enhancing database query performance.
- Introduced new TED mapping functions to convert parsed TED documents into tender entities, improving integration with the tender management system.
- Added error handling for contract notice processing in the TED scraper, ensuring robust logging and error management.
- Removed deprecated eform structures to streamline the codebase and focus on essential components.
- Updated feedback repository and service methods to use a consistent naming convention, changing `NewTenderRepository` and similar methods to `NewRepository`.
- Refactored feedback handler methods to improve clarity by renaming methods such as `ListFeedback` to `Search` and `GetFeedback` to `Get`.
- Enhanced feedback response structures to include detailed company and tender information, improving the clarity of feedback data returned to API consumers.
- Updated Swagger documentation to reflect changes in feedback response structures and endpoint paths, ensuring accurate representation of the API for consumers.
- Updated the MongoDB driver from v1.17.4 to v2.3.0, ensuring compatibility with the latest features and improvements.
- Refactored company-related entity and repository files to utilize the new driver, replacing `primitive.ObjectID` with `bson.ObjectID` for ID handling.
- Adjusted the configuration in `.drone.yml` to include the `main` branch in the trigger settings, enhancing CI/CD pipeline flexibility.
- Changed BSON tags to JSON tags in the CustomerResponse struct to ensure proper serialization in API responses.
- Updated the comment in the buildSearchFilter method to correct a typo, enhancing clarity in the codebase.
- Renamed and restructured tender-related API endpoints for improved clarity and consistency, including changing the route from `/admin/tenders` to `/admin/v1/companies` and updating the associated methods.
- Introduced a new SearchForm structure for listing tenders with advanced filtering capabilities, enhancing the API's search functionality.
- Updated the service and repository layers to align with the new endpoint structure, ensuring proper handling of tender data.
- Enhanced Swagger and YAML documentation to reflect the changes in endpoint structure, including detailed descriptions and examples for the new search functionality.
- Improved error handling and response structures to provide clearer feedback to API consumers, ensuring a more robust and user-friendly experience.
- Changed references in API documentation from `main.HealthResponse` to `bootstrap.HealthResponse` for consistency.
- Added new query parameters `publication_from` and `publication_to` to the tender listing and recommendation endpoints to allow filtering by publication date.
- Updated the `ListTendersRequest` struct to include optional fields for `from` and `to` timestamps.
- Modified the `TenderRepository` and `TenderService` interfaces to support the new filtering parameters in the tender listing functionality.
- Enhanced the service and handler layers to process the new query parameters, improving the flexibility of tender retrieval.
- Updated the tender API to include a new endpoint for recommending public tenders based on company profiles, enhancing user experience for mobile application users.
- Modified existing routes to reflect the new structure, changing the tender details route and adding a dedicated recommendation route.
- Improved Swagger documentation to accurately represent the new endpoint, including detailed descriptions, parameters, and response formats.
- Ensured adherence to Clean Architecture principles by maintaining clear separation of concerns in the service and handler layers.
- Removed all scraping-related functionality, including routes, handlers, services, and repository methods, to streamline the tender management system.
- Updated the tender API to focus solely on tender management, enhancing endpoints for listing, retrieving, and updating tenders.
- Improved Swagger documentation to reflect the removal of scraping endpoints and the addition of new tender-related features.
- Ensured adherence to Clean Architecture principles throughout the refactoring process, maintaining a clear separation of concerns.
- Updated the tender service to include company-based matching for tenders, allowing for more relevant results based on company CPV codes.
- Modified the ListTenders endpoint to accept a company ID parameter for calculating match percentages and sorting tenders accordingly.
- Refactored the tender response structure to include match percentage and days until deadline for better client-side handling.
- Updated API documentation to reflect changes in the tender listing functionality and added new endpoints for public tender access.
- Removed deprecated customer-related methods and streamlined customer listing functionality for improved clarity and performance.
Mazyar