- Updated `documents_scraped_filter.go` to extend the caching mechanism for contract folder IDs, increasing the cache TTL to 15 minutes to prevent premature invalidation during slow scans.
- Modified the `cachedContractFolderIDsWithDocuments` method to detach from the caller's cancellation, ensuring that shared scans are not aborted by client disconnects or timeouts.
- Introduced a new test, `TestCachedContractFolderIDsSurvivesCallerCancellation`, in `documents_scraped_filter_test.go` to validate the behavior of the caching mechanism under cancellation scenarios.
- Enhanced the `scanStartedProcedureLister` to manage scan initiation and cancellation more effectively, improving the robustness of the document retrieval process.
This update improves the reliability and performance of the tender management system's document filtering capabilities, particularly in scenarios with long-running scans.
- Introduced `documents_scraped_filter.go` to handle filtering tenders based on the presence of documents in MinIO, utilizing a caching mechanism to optimize performance.
- Implemented `enrichDocumentsScrapedFilter` and `cachedContractFolderIDsWithDocuments` methods in the `tenderService` for efficient retrieval of contract folder IDs with documents.
- Added unit tests in `documents_scraped_filter_test.go` to validate the new filtering logic and caching behavior, ensuring accurate results and error handling.
- Updated `SearchForm` documentation to clarify the caching aspect of `ContractFolderIDsWithDocuments`.
- Enhanced comments in the handler and repository to reflect the new filtering logic and its implications.
This update improves the tender search functionality by efficiently managing document presence checks, leading to better performance and user experience in the tender management system.
- Added the `deadlineWithinWindowMatch` function to match raw deadline fields stored as Unix seconds or milliseconds, ensuring accurate filtering in MongoDB queries.
- Updated the `ClosingSoon` method in the repository to utilize the new deadline matching logic, improving query accuracy for upcoming deadlines.
- Introduced a new unit test, `TestDeadlineWithinWindowMatchCoversSecondsAndMilliseconds`, to validate the functionality of the deadline matching logic, ensuring both seconds and milliseconds are correctly handled.
- Enhanced comments in the `widget_cache` and `search_list_cache` files for better clarity on cache behavior and staleness.
This update improves the accuracy of deadline handling in the dashboard service and strengthens the test coverage for related functionalities.
- Introduced a new `widgetCache` structure to manage caching for various dashboard widgets, improving performance and reducing redundant data retrieval.
- Updated the `ClosingSoon` and `Trend` methods in the service to utilize the new caching mechanism, ensuring efficient data access and reducing load on the database.
- Enhanced the `NoticeTypes` and `Countries` methods to implement caching, further optimizing the dashboard's responsiveness.
- Added a new `searchListCache` implementation in the tender service to cache search results, improving the efficiency of tender searches and reducing database load.
- Implemented robust error handling and logging for cache operations, ensuring better visibility into potential issues.
This update significantly enhances the performance and scalability of the dashboard and tender services by integrating effective caching strategies, leading to improved user experience and resource management.
- Eliminated the `ContentXML` field from the `Tender` struct in the entity definition, streamlining the data model.
- Removed the `FindTendersWithContentXML` method from the `TenderRepository` interface, as it is no longer necessary.
- Updated the `ToTender` method in the `NoticeWorker` to reflect these changes, ensuring consistency across the codebase.
This update simplifies the tender data structure and repository interface, improving maintainability and clarity in the tender management system.
- Introduced a new index `documents_scraped_created_at_id_idx` in the tender repository to support keyset pagination for documents marked as scraped, sorted by creation date.
- This enhancement improves the efficiency of querying scraped documents, facilitating better performance in data retrieval and management.
This update significantly optimizes the tender repository's handling of scraped documents, enhancing overall data access capabilities.
- Updated the tender search handler to include a new `include_total` query parameter, allowing clients to request total count and pagination metadata for search results.
- Refactored the `tenderSearchListProjection` to exclude heavy fields from the list response, optimizing data retrieval.
- Modified the `buildSearchFilter` method to utilize the `processing_metadata.documents_scraped` flag for filtering, improving search accuracy.
- Added unit tests for the new pagination features and search filter logic, ensuring robust validation of the search functionality.
This update significantly enhances the tender search experience by providing more flexible pagination options and improving the efficiency of data handling.
- Introduced `PickAssignedCompanyID` function to determine the appropriate company ID for company-scoped operations, improving company assignment logic.
- Added `ResolveMongoID` method in the tender service to map MongoDB IDs and AI procedure references to canonical tender IDs, enhancing ID resolution.
- Updated `ToggleTenderApproval` to handle company ID validation and improve error handling, ensuring robust processing of tender approvals.
- Enhanced logging for error scenarios in the tender approval service, providing better insights into failures during operations.
- Refactored the `GetByID` and `GetByTenderAndCompany` methods in the tender approval repository to utilize custom error types for improved clarity.
This update significantly improves the handling of company and tender operations, enhancing error management and overall service reliability.
- Introduced a new method `isRecommendedPageCacheStale` in the `tenderService` to determine if the cached recommendations for a company are outdated based on the count of cached recommendations and the latest AI recommendations.
- Updated the `recommendFromPageCache` method to utilize the new cache staleness check, ensuring that outdated caches are not used for recommendations.
- This enhancement improves the accuracy of recommendations by ensuring that users receive the most up-to-date information, thereby enhancing the overall user experience in the tender recommendation service.
- Removed the `buildRecommendedTenderResponsesParallel` method and replaced it with `buildRecommendedTenderListResponses`, simplifying the response building process.
- Introduced `enrichRecommendedTenderResponsesParallel` to handle enrichment of tender responses in parallel, improving performance.
- Updated the `Recommend` method to utilize the new enrichment logic, ensuring timely and efficient processing of recommendations.
- Cleaned up unused functions and imports in `recommendation_page_cache.go` and `recommendation_response.go`, enhancing code clarity and maintainability.
This update streamlines the recommendation response handling and improves the overall efficiency of the tender recommendation service.
- Removed the `RecommendationPageCacheLanguages` configuration from `AISummarizerConfig` to streamline cache management.
- Updated the `companyService` and `tenderService` to utilize the new `InvalidateRecommendedTendersPageCache` method for cache invalidation, enhancing clarity and efficiency.
- Refactored the `invalidateRecommendedTendersPageCache` method to eliminate unnecessary context parameters, simplifying the function signature.
- Improved the handling of page cache refresh logic by consolidating language handling within the `tenderService`, ensuring consistent behavior across services.
- Cleaned up related tests and removed deprecated functions to maintain code quality and readability.
This update enhances the maintainability of the recommendation caching system by simplifying configuration and improving cache invalidation logic.
- Introduced a new `RecommendedTendersPageCacheRefresher` interface to manage the asynchronous refresh of recommendation pages in Redis, improving cache management.
- Updated the `companyService` to support setting page cache languages and refreshing the recommended tenders page cache based on company IDs.
- Enhanced the `tenderService` to build and invalidate recommended tenders page caches, ensuring timely updates and efficient retrieval of cached recommendations.
- Added configuration options for recommendation page cache languages in the `AISummarizerConfig`, allowing for flexible language support.
- Implemented unit tests for the new caching logic and page refresh functionality, ensuring robust validation of the recommendation caching process.
This update significantly improves the efficiency and responsiveness of the tender recommendation service by integrating enhanced caching mechanisms and page refresh capabilities.
- Updated the `markResolvedRecommendedTenderSeen` function to accept an additional `tenderRef` parameter, allowing for better tracking of tender references and preventing duplicates.
- Refactored the `GetByProcedureReferences` method in the repository to combine indexed notice lookups with contract-folder candidates, ensuring older notice references resolve correctly after tender merges.
- Introduced a new `findTendersByContractFolderIDs` method to streamline the retrieval of tenders by contract folder IDs, enhancing performance and maintainability.
- Added unit tests for new functionality in `ai_reference_test.go` and `recommendation_filter_test.go`, ensuring robust validation of tender reference mapping and deduplication logic.
This update significantly improves the handling of tender references and enhances the overall test coverage, ensuring more reliable and efficient processing of tender data.
- Introduced a context with a timeout for Redis cache retrieval in the `getCachedAIRecommendations` method, ensuring more robust handling of potential delays.
- Added a new file `recommendation_page.go` to encapsulate the logic for building recommendation pages, improving code organization and readability.
- Implemented a `buildRecommendationPage` method to handle pagination and batch processing of recommendations, enhancing performance and user experience.
- Created a new test file `recommendation_page_test.go` to validate the functionality of the recommendation page building process, ensuring correct pagination and batch resolution.
This update significantly improves the efficiency and maintainability of the tender recommendation service by optimizing caching and implementing structured pagination handling.
- Renamed the test function to `TestBuildRecommendedTenderListResponsesPreservesOrderWithoutStorage` for clarity and updated its logic to reflect changes in the recommendation response building process.
- Removed unnecessary sleep in the test to enhance performance.
- Refactored the `buildRecommendedTenderResponsesParallel` method to `buildRecommendedTenderListResponses`, simplifying the response building without relying on MinIO translation lookups.
- Updated the repository's `mapProcedureReferencesToTenders` function to improve efficiency by grouping candidates by their contract folder IDs, enhancing the mapping process.
This update streamlines the recommendation response handling and enhances the test suite for better maintainability and performance.
- Introduced a caching mechanism for translation responses in the tender recommendation service, improving efficiency by reducing redundant translation requests.
- Refactored the `enrichWithTranslation` method to support an optional translation cache, allowing for faster retrieval of stored translations.
- Implemented parallel processing for building recommended tender responses, utilizing goroutines to enhance performance and responsiveness.
- Added unit tests to validate the new caching behavior and ensure the correct application of translations in various scenarios.
This update significantly improves the performance and responsiveness of the tender recommendation service by integrating caching and parallel processing, enhancing the overall user experience.
- Introduced a new function `dedupeRecommendationResponsesByTenderID` to remove duplicate tender recommendations based on their IDs, ensuring unique entries in the recommendation list.
- Updated the `mergeRecommendedTenders` function to utilize the new deduplication logic, streamlining the merging process of recommendation lists.
- Enhanced the `fetchAIRecommendations` method to apply deduplication on the fetched recommendations, improving data integrity.
- Added unit tests for the new deduplication function to validate its behavior and ensure correct handling of various input scenarios.
This update enhances the recommendation handling by ensuring that duplicate tender entries are effectively managed, improving the overall quality of the recommendations provided to users.
- Introduced a new test file `recommendation_translation_test.go` to validate the behavior of the tender recommendation service regarding translation handling.
- Implemented tests to ensure that the service correctly applies stored translations and retains original text when no translation is available.
- Enhanced the `Recommend` method in the tender service to utilize a structured approach for managing recommended tenders, improving clarity and maintainability.
This update improves the testing coverage for translation handling in tender recommendations, ensuring accurate responses based on available translations and original content.
- Renamed `buildRejectedTenderIDSetForCompanies` to `buildRejectedTenderIDSetsForCompanies` to better reflect its functionality of returning a map of excluded tender IDs by company.
- Updated the implementation to return a nested map structure for better organization of excluded tender IDs per company.
- Added a new test function `TestIsRecommendedTenderExcludedForAllCompanies` to validate the visibility of tenders based on company-specific exclusions.
- Enhanced existing tests to ensure comprehensive coverage of the new logic for handling rejected tenders.
This update improves the clarity and functionality of the tender recommendation process by refining the handling of rejected tenders and expanding test coverage.
- Introduced a new test function `TestMapProcedureReferencesToTenders` to validate the mapping of procedure references to corresponding tenders.
- Enhanced the `GetByProcedureReferences` method in the tender repository to utilize a more efficient filtering mechanism based on unique contract folder IDs.
- Updated pagination logic to dynamically adjust limits based on the number of unique folders and references, ensuring optimal data retrieval.
This update improves the testing coverage for tender mapping functionality and optimizes the repository's data fetching strategy, enhancing overall performance and reliability.
- Introduced a new method to validate company IDs for AI recommendations, ensuring all requested companies exist before processing.
- Added caching logic to retrieve AI recommendations efficiently, reducing unnecessary calls and improving performance.
- Utilized errgroup for concurrent processing of AI recommendations across multiple companies, enhancing responsiveness.
- Updated the tender service to handle rejected tenders more effectively by incorporating concurrent fetching of rejected tender IDs.
This update significantly improves the AI recommendations process by ensuring accurate company validation and optimizing data retrieval through caching and concurrency, enhancing overall system performance and user experience.
- Added functionality to retrieve merged AI recommendations for multiple companies, improving the relevance of tender suggestions based on company-specific data.
- Introduced normalization functions to clean and deduplicate company IDs, ensuring accurate processing of recommendations.
- Enhanced the company context resolution in customer middleware to support multiple assigned companies, improving the handling of company-specific requests.
- Updated the tender recommendation logic to utilize the new merged recommendations and handle exclusions for rejected tenders accordingly.
- Added unit tests to verify the new recommendation merging logic and company ID normalization, ensuring robust functionality.
This update significantly enhances the tender recommendation process by allowing for more comprehensive and relevant suggestions based on multiple company contexts, improving user experience and satisfaction.
- Updated the tender service to include a new dependency for listing rejected tenders by company, allowing for more refined tender recommendations.
- Introduced a new field in the SearchForm to specify whether to exclude rejected tenders from the recommendation results.
- Enhanced the Recommend method to filter out rejected tenders based on the new exclusion logic, improving the relevance of AI-ranked tender recommendations.
- Added unit tests to verify the exclusion logic for rejected tenders, ensuring robust functionality.
This update improves the tender recommendation process by ensuring that company-rejected tenders are not included in the results, enhancing user experience and satisfaction.
- Updated the ListPendingTenders method to filter tenders based on their active publication submission window instead of just deadlines, enhancing the accuracy of tender listings.
- Introduced a new HasActivePublicationWindow method in the Tender entity to determine if the publication-based submission window is still open.
- Modified the GetTenderByNoticeID method to reflect the new logic for filtering tenders based on their publication status.
- Enhanced the documentation for relevant API endpoints to clarify the changes in tender retrieval criteria.
This update improves the precision of tender scraping by ensuring only relevant tenders with active submission windows are processed, contributing to better data management and scraping efficiency.
- Updated the ListPaginationOptions struct to include SkipCount and IncludeCount fields, allowing for more flexible pagination behavior.
- Modified the BuildListPagination function to handle cursor pagination with count options, improving performance by running count queries in parallel with data retrieval.
- Enhanced the FindAll method in the repository to support concurrent counting of documents, reducing overall latency for list operations.
- Added tests for pagination behavior, ensuring accurate handling of count scenarios in both offset and cursor pagination.
This update improves the efficiency and flexibility of pagination in the MongoDB repository, enhancing the overall performance of list operations.
- Added the PublicationSubmissionDeadline method to the Tender entity, which calculates the submission deadline based on the publication date and stored submission deadline.
- Updated the IsRecommendable method to utilize the new PublicationSubmissionDeadline logic for determining recommendation eligibility.
- Introduced unit tests for the PublicationSubmissionDeadline method, ensuring accurate calculation and validation of submission deadlines.
- Enhanced existing tests for the IsRecommendable method to cover new scenarios related to submission deadlines.
This update improves the accuracy of submission deadline handling and enhances the recommendation logic for tenders based on publication dates.
- Implemented the IsRecommendable method in the Tender entity to determine if a tender should be included in AI recommendations based on its status and deadline.
- Added unit tests for the IsRecommendable method to cover various scenarios, ensuring accurate recommendation logic.
- Updated the Recommend method in the tender service to utilize the new IsRecommendable method for improved clarity and functionality.
This update enhances the recommendation logic for tenders, ensuring only appropriate tenders are considered for AI recommendations based on their status and deadlines.
- Added a new endpoint in the `tender` handler for retrieving AI-ranked tender recommendations for a company, improving the functionality of the admin panel.
- Updated the `SearchForm` to include a query parameter for `only_active_deadlines`, allowing for more flexible search options.
- Enhanced API documentation for the new endpoint to provide clear usage instructions and expected parameters.
This update improves the tender management system by providing administrators with better tools for accessing relevant tender information.
- Introduced `ProcedureReference` struct to encapsulate AI procedure coordinates for better data management.
- Added `GetByProcedureReferences` method in the `TenderRepository` to retrieve multiple tenders based on AI procedure references in a single query.
- Updated the `resolveRecommendedTenders` method in the service layer to utilize the new repository method, improving efficiency in fetching recommended tenders.
- Enhanced error handling and logging for the new repository method to ensure robust operation.
This update improves the handling of AI procedure references, streamlining tender retrieval processes and enhancing overall system performance.
- Introduced `FormatAIProcedureRef` and `ParseAIProcedureRef` functions in the `tender` domain for handling AI service tender references.
- Added unit tests for these functions in `ai_reference_test.go` to ensure correct parsing and formatting behavior.
- Updated the `TenderResponse` struct to include a new `ProcedureRef` field for improved data representation.
- Enhanced the `GetByProcedureReference` method in the repository to retrieve tenders based on the new procedure reference format.
- Modified the `Recommend` method in the service layer to utilize the new procedure reference handling, improving the recommendation process.
This update enhances the handling of AI procedure references, ensuring better data integrity and usability in the tender management system.
- Changed the `Rank` field type from `string` to `int` in the `RecommendedTenderResponse` struct within the `company` domain for better data representation.
- Updated the `Rank` field type from `string` to `int` in the `TenderResponse` struct within the `tender` domain to ensure consistency in ranking data.
- Modified the `Rank` field type from `string` to `int` in the `RecommendedTender` struct within the AI summarizer package to align with the updated data structure.
This update enhances the data integrity and consistency across the tender management system by standardizing the rank representation as an integer.
- Added new fields `Rank` and `Analysis` to the `TenderResponse` struct for improved data representation.
- Updated the `RecommendTenders` handler to reflect changes in API documentation, including a new summary and description for AI-ranked tender recommendations.
- Improved error handling in the recommendation process, ensuring appropriate responses for missing company IDs and unavailable AI services.
This update enhances the tender response capabilities and improves the clarity of the recommendation API, aligning with the overall goal of providing better insights for users.
- Removed the DocumentSummarizationWorker and its related scheduling logic from the worker bootstrap.
- Updated the AI summarizer client initialization comment for clarity.
- Added a new error type for cases when tender documents have not been scraped yet, enhancing error handling in the tender service.
- Modified API documentation to reflect changes in AI summary retrieval logic, ensuring accurate descriptions of on-demand summarization behavior.
This update streamlines the AI summarization process by eliminating the document summarization worker, improving overall system efficiency and clarity in error handling.
- Updated the document scraper service to include a new ScrapePortalsProvider interface, allowing for dynamic retrieval of supported scraping portals.
- Modified the ListPendingTenders and GetTenderByNoticeID methods to filter tenders based on document URLs that match the configured portals.
- Introduced new error handling for cases when the scrape portals provider is not configured, returning appropriate service unavailable responses.
- Enhanced API documentation to reflect changes in tender retrieval logic and added error response details for unsupported portal scenarios.
This update improves the document scraping functionality by integrating AI portal support, enhancing the overall reliability and flexibility of the tender management system.
- Updated the `AggregateProcurementLotEstimatedValue` function to reject mixed currencies and allow empty lot currencies, ensuring accurate aggregation of estimated values.
- Introduced new unit tests in `entity_test.go` to validate the behavior of the aggregation function under various scenarios, including mixed currencies and empty lot currencies.
- Refactored budget calculation in `budget_test.go` to utilize the new aggregation logic, improving consistency in budget retrieval.
This update improves the handling of estimated values in procurement lots, enhancing the reliability of the tender management system.
- Updated the `ResolvedEstimatedValueAndCurrency` method to aggregate procurement lot values when the tender-level estimated value is not set, improving accuracy in value retrieval.
- Introduced the `AggregateProcurementLotEstimatedValue` function to sum estimated values from procurement lots and return the first found currency.
- Modified the `ToResponseWithLanguage` method to utilize the new estimated value resolution logic.
- Added unit tests for the new functionality, ensuring correct behavior for various scenarios in the `entity_test.go` and `budget_test.go` files.
This update improves the handling of estimated values in tenders, enhancing the overall reliability of the tender management system.
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.
This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
- Added `NoticePublicationID` field to the `SearchForm` for filtering tenders by TED notice publication ID.
- Updated Swagger documentation in the handler methods to include the new `notice_publication_id` parameter for relevant endpoints.
- Modified the repository's search filter to incorporate the new `NoticePublicationID` field, allowing for more precise search queries.
This update improves the search capabilities within the tender management system, enhancing user experience and data retrieval accuracy.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Updated the AI pipeline service to include a new `ScrapedDocumentMetadataSyncer` interface for persisting scraped document metadata onto tender records.
- Modified the `NewService` function to accept the new metadata syncer dependency.
- Implemented synchronization of scraped document metadata in the `ScrapeDocuments` and `Run` methods.
- Enhanced the tender service to enrich search filters based on scraped documents and added a new method for syncing scraped documents from storage.
- Updated the `SearchForm` to include `ContractFolderIDsWithDocuments` for better handling of scraped documents in queries.
This update improves the integration of scraped document handling within the AI pipeline, enhancing data consistency and operational efficiency in the tender management system.
- Added new routes and handlers for AI pipeline operations, including scraping documents, batch summarization, translation, and syncing with the Opplens AI service.
- Introduced request forms for handling tender references and batch operations.
- Enhanced the AI service with methods for triggering batch operations and managing pipeline runs.
- Updated Swagger documentation to reflect the new AI pipeline endpoints and their functionalities.
This update integrates comprehensive AI pipeline capabilities into the tender management system, improving operational efficiency and user experience.
- Removed the QueueConfig structure and related queue management files as they are no longer in use.
- Updated the worker initialization to reflect the removal of queue-related configurations.
- Cleaned up the bootstrap process by eliminating deprecated logging related to the queue system.
This update streamlines the worker's configuration and prepares the codebase for future enhancements without the legacy queue management components.
Paginate un-summarized tenders and mark rows missing notice/folder IDs as handled so the worker cannot spin forever. Return 400/404 for validation and not-found cases on AI summarize/analyze triggers instead of leaking internal errors as 500.
- Added a new endpoint to trigger on-demand agentic analysis for tenders via the AI service.
- Introduced `TriggerAIAnalyze` method in the `TenderHandler` to handle requests and responses for AI analysis.
- Updated the `tenderService` to include `TriggerAIAnalyze` method, which validates input and interacts with the AI summarizer client.
- Enhanced the `AIAnalyzeResponse` and `AIAnalyzeDocument` structures to support the new analysis feature.
- Refactored the `DocumentSummarizationWorker` and `TranslationWorker` to remove deprecated MinIO dependencies, streamlining the AI service interactions.
This update improves the functionality of the tender management system by allowing users to trigger AI analysis on-demand, enhancing the overall user experience and system capabilities.
- Updated `InitAISummarizerClient` to accept `mongoManager` for tracking translation success.
- Introduced new `Statistics` endpoint in the dashboard to fetch scraping and translation statistics.
- Enhanced `TranslationWorker` to utilize the new success counter for tracking successful translations.
- Added necessary data structures and query forms for statistics reporting.
This refactor improves the tracking of AI translation success and provides new insights through the dashboard statistics.
Mazyar