Compare commits

..

100 Commits

Author SHA1 Message Date
m.nazemi fa16d4080a Merge pull request 'Add unit tests for backfilling notice location from buyer' (#53) from TM-485 into develop
continuous-integration/drone/push Build is passing
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/53
Reviewed-by: Hadi Barzegar <barzagarhadi@gmail.com>
2026-07-13 14:20:08 +03:30
Mazyar e4f7c4a04c Add unit tests for backfilling notice location from buyer
- Introduced `mapper_test.go` to validate the functionality of backfilling notice location details from the buyer organization.
- Implemented two test cases: one to ensure the location fields are correctly populated from the buyer when they are empty, and another to verify that existing values are not overwritten.
- Enhanced the `mapper.go` file by adding the `backfillNoticeLocationFromBuyer` function to facilitate this logic.

This update improves the reliability of notice data handling within the tender management system by ensuring accurate location information is maintained.
2026-07-13 13:44:31 +03:30
Mazyar 51a1a6aa82 Refactor AI pipeline terminology for consistency
continuous-integration/drone/push Build is passing
- Updated comments and logging messages in the worker and related files to replace "daily-run" with "auto run" for clarity and consistency.
- Adjusted the `WorkerConfig` struct to reflect the new terminology in configuration settings.
- Renamed functions and test cases to align with the updated terminology, enhancing code readability and maintainability.

This change improves the clarity of the AI pipeline's functionality within the tender management system.
2026-07-13 11:54:58 +03:30
m.nazemi 6ee81c3581 Merge pull request 'Add documents scraped filter functionality and related tests' (#51) from documents-scraped into develop
continuous-integration/drone/push Build is passing
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/51
2026-07-12 22:03:13 +03:30
Mazyar d55b2685ca Merge branch 'documents-scraped' of https://repo.ravanertebat.com/TM/tm_back into documents-scraped 2026-07-12 22:02:43 +03:30
Mazyar 04c4102170 Enhance documents scraped filter with cancellation handling and caching improvements
- Updated `documents_scraped_filter.go` to extend the caching mechanism for contract folder IDs, increasing the cache TTL to 15 minutes to prevent premature invalidation during slow scans.
- Modified the `cachedContractFolderIDsWithDocuments` method to detach from the caller's cancellation, ensuring that shared scans are not aborted by client disconnects or timeouts.
- Introduced a new test, `TestCachedContractFolderIDsSurvivesCallerCancellation`, in `documents_scraped_filter_test.go` to validate the behavior of the caching mechanism under cancellation scenarios.
- Enhanced the `scanStartedProcedureLister` to manage scan initiation and cancellation more effectively, improving the robustness of the document retrieval process.

This update improves the reliability and performance of the tender management system's document filtering capabilities, particularly in scenarios with long-running scans.
2026-07-12 21:54:21 +03:30
Mazyar 0fde5772e2 Add documents scraped filter functionality and related tests
- Introduced `documents_scraped_filter.go` to handle filtering tenders based on the presence of documents in MinIO, utilizing a caching mechanism to optimize performance.
- Implemented `enrichDocumentsScrapedFilter` and `cachedContractFolderIDsWithDocuments` methods in the `tenderService` for efficient retrieval of contract folder IDs with documents.
- Added unit tests in `documents_scraped_filter_test.go` to validate the new filtering logic and caching behavior, ensuring accurate results and error handling.
- Updated `SearchForm` documentation to clarify the caching aspect of `ContractFolderIDsWithDocuments`.
- Enhanced comments in the handler and repository to reflect the new filtering logic and its implications.

This update improves the tender search functionality by efficiently managing document presence checks, leading to better performance and user experience in the tender management system.
2026-07-12 21:54:21 +03:30
m.nazemi c35fa331f9 Merge branch 'develop' into documents-scraped 2026-07-12 21:46:03 +03:30
m.nazemi c8616940ff Merge pull request 'Add tender submission functionality and related routes' (#52) from tender-submission into develop
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/52
2026-07-12 21:45:23 +03:30
Mazyar fa3d466579 Enhance tender approval and submission handling
- Added a new error for submission sync failures, improving error handling in the tender approval process.
- Updated the `ToggleTenderApproval` method to handle the new error, providing clearer responses for sync issues.
- Introduced a method to reopen terminal submissions when tender approvals are resubmitted, enhancing submission state management.
- Implemented optimistic locking in the repository to prevent concurrent updates, improving data integrity.
- Added unit tests for new submission handling logic, ensuring robust functionality and error management.

This update strengthens the tender approval workflow and submission handling, ensuring better error reporting and state management.
2026-07-12 21:41:53 +03:30
Mazyar 0cce9ef1b5 Add tender submission functionality and related routes
- Introduced the `tender_submission` domain, including entity, repository, service, and handler implementations for managing tender submissions.
- Added new routes for both admin and public access to tender submissions, allowing for listing, ensuring, and retrieving submissions by ID and tender.
- Enhanced the tender approval service to synchronize submission workflows with approval changes, ensuring proper state management.
- Implemented validation and response structures for tender submission operations, improving API consistency and usability.
- Added unit tests for tender submission status transitions and workflow logic, ensuring robust functionality.

This update enhances the tender management system by providing comprehensive support for tender submissions, improving overall workflow and user experience.
2026-07-12 14:54:42 +03:30
Mazyar 5c93e0f01b Add documents scraped filter functionality and related tests
- Introduced `documents_scraped_filter.go` to handle filtering tenders based on the presence of documents in MinIO, utilizing a caching mechanism to optimize performance.
- Implemented `enrichDocumentsScrapedFilter` and `cachedContractFolderIDsWithDocuments` methods in the `tenderService` for efficient retrieval of contract folder IDs with documents.
- Added unit tests in `documents_scraped_filter_test.go` to validate the new filtering logic and caching behavior, ensuring accurate results and error handling.
- Updated `SearchForm` documentation to clarify the caching aspect of `ContractFolderIDsWithDocuments`.
- Enhanced comments in the handler and repository to reflect the new filtering logic and its implications.

This update improves the tender search functionality by efficiently managing document presence checks, leading to better performance and user experience in the tender management system.
2026-07-12 00:20:39 +03:30
m.nazemi 10385e997b Merge pull request 'Remove ContentXML field from Tender entity and related repository method' (#50) from content-xml into develop
continuous-integration/drone/push Build is passing
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/50
Reviewed-by: Hadi Barzegar <barzagarhadi@gmail.com>
2026-07-11 21:47:55 +03:30
m.nazemi adfd291245 Merge branch 'develop' into content-xml 2026-07-11 21:47:44 +03:30
m.nazemi 26a593306d Merge pull request 'Enhance dashboard service with widget caching and improved query handling' (#49) from panel-widget into develop
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/49
2026-07-11 21:47:14 +03:30
Mazyar c5e62a4061 Implement deadline matching logic for dashboard repository and enhance unit tests
- Added the `deadlineWithinWindowMatch` function to match raw deadline fields stored as Unix seconds or milliseconds, ensuring accurate filtering in MongoDB queries.
- Updated the `ClosingSoon` method in the repository to utilize the new deadline matching logic, improving query accuracy for upcoming deadlines.
- Introduced a new unit test, `TestDeadlineWithinWindowMatchCoversSecondsAndMilliseconds`, to validate the functionality of the deadline matching logic, ensuring both seconds and milliseconds are correctly handled.
- Enhanced comments in the `widget_cache` and `search_list_cache` files for better clarity on cache behavior and staleness.

This update improves the accuracy of deadline handling in the dashboard service and strengthens the test coverage for related functionalities.
2026-07-11 21:44:22 +03:30
Mazyar 81b0d94ba3 Enhance dashboard service with widget caching and improved query handling
- Introduced a new `widgetCache` structure to manage caching for various dashboard widgets, improving performance and reducing redundant data retrieval.
- Updated the `ClosingSoon` and `Trend` methods in the service to utilize the new caching mechanism, ensuring efficient data access and reducing load on the database.
- Enhanced the `NoticeTypes` and `Countries` methods to implement caching, further optimizing the dashboard's responsiveness.
- Added a new `searchListCache` implementation in the tender service to cache search results, improving the efficiency of tender searches and reducing database load.
- Implemented robust error handling and logging for cache operations, ensuring better visibility into potential issues.

This update significantly enhances the performance and scalability of the dashboard and tender services by integrating effective caching strategies, leading to improved user experience and resource management.
2026-07-11 21:44:22 +03:30
m.nazemi 3e2700bc36 Merge pull request 'Refactor notification service to support multiple delivery channels' (#48) from notif-channel into develop
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/48
2026-07-11 21:12:11 +03:30
Mazyar aafae2a26f Enhance notification entity and service for improved delivery channel handling
- Introduced new event types for notifications, including email and push, to support a broader range of delivery methods.
- Added a `persistedEventType` function to map delivery channels to their corresponding event types, improving clarity and maintainability.
- Updated the `sendToRecipient` method to ensure in-app notifications are always persisted, enhancing the reliability of notification records.
- Improved error logging for in-app notification persistence failures, providing better context for debugging.

This update enhances the notification system's flexibility and reliability by supporting multiple delivery channels and improving error handling.
2026-07-11 21:07:37 +03:30
Mazyar 9d8ce12816 Refactor notification service to support multiple delivery channels
- Updated the `sendToRecipient` method to iterate over the channels specified in the notification request, allowing notifications to be persisted for each channel.
- Modified the `persistNotification` method to accept a channel parameter, enhancing the logic for handling different delivery methods (email, push, in-app).
- Improved error logging to include the channel information when persisting notifications fails, providing better context for debugging.

This update enhances the flexibility of the notification service by enabling it to handle multiple delivery channels more effectively, improving overall notification management.
2026-07-11 21:07:37 +03:30
Mazyar b388af3518 Remove ContentXML field from Tender entity and related repository method
- Eliminated the `ContentXML` field from the `Tender` struct in the entity definition, streamlining the data model.
- Removed the `FindTendersWithContentXML` method from the `TenderRepository` interface, as it is no longer necessary.
- Updated the `ToTender` method in the `NoticeWorker` to reflect these changes, ensuring consistency across the codebase.

This update simplifies the tender data structure and repository interface, improving maintainability and clarity in the tender management system.
2026-07-11 20:54:29 +03:30
Mazyar 932b0cf24e Add index for documents scraped with keyset pagination in tender repository
- Introduced a new index `documents_scraped_created_at_id_idx` in the tender repository to support keyset pagination for documents marked as scraped, sorted by creation date.
- This enhancement improves the efficiency of querying scraped documents, facilitating better performance in data retrieval and management.

This update significantly optimizes the tender repository's handling of scraped documents, enhancing overall data access capabilities.
2026-07-11 17:00:48 +03:30
Mazyar 1df44ec8ca Add translated notices scanning functionality to dashboard repository
continuous-integration/drone/push Build is passing
- Introduced a new `translatedNoticesScanner` interface for scanning MinIO for daily translated notice counts.
- Enhanced the `repository` struct to include fields for managing translated notice scope and caching.
- Implemented the `ScanTranslatedNotices` method in the `StorageClient` to retrieve daily counts and total from MinIO.
- Updated the `Statistics` method in the repository to utilize the new translated notices scope, improving data accuracy.
- Added unit tests for the translated notices functionality, ensuring robust validation of the new features.

This update significantly enhances the dashboard's ability to handle translated notice statistics, improving overall data management and reporting capabilities.
2026-07-10 17:05:38 +03:30
Mazyar 843e1508df Enhance tender search functionality with pagination and filtering improvements
continuous-integration/drone/push Build is passing
- Updated the tender search handler to include a new `include_total` query parameter, allowing clients to request total count and pagination metadata for search results.
- Refactored the `tenderSearchListProjection` to exclude heavy fields from the list response, optimizing data retrieval.
- Modified the `buildSearchFilter` method to utilize the `processing_metadata.documents_scraped` flag for filtering, improving search accuracy.
- Added unit tests for the new pagination features and search filter logic, ensuring robust validation of the search functionality.

This update significantly enhances the tender search experience by providing more flexible pagination options and improving the efficiency of data handling.
2026-07-10 16:23:46 +03:30
Mazyar 2fbf329182 Refactor tender approval logic to streamline submission mode handling
continuous-integration/drone/push Build is passing
- Removed the resetting of `SubmissionMode` in the `Reject` method of the `TenderApproval` entity, ensuring that the submission mode is preserved upon rejection.
- Updated the `GetCompanyTenderApprovalStats` method in the repository to count submissions by mode without filtering by status, enhancing the accuracy of statistics.
- Adjusted the `ToggleTenderApproval` method in the service to set the `SubmissionMode` when rejecting a tender, ensuring consistent handling of submission modes across approval states.

This update improves the clarity and functionality of tender approval processes, ensuring that submission modes are correctly managed during approval and rejection scenarios.
2026-07-08 03:02:57 +03:30
Mazyar 81a94b4879 Enhance tender and company context handling with new features and error management
continuous-integration/drone/push Build is passing
- Introduced `PickAssignedCompanyID` function to determine the appropriate company ID for company-scoped operations, improving company assignment logic.
- Added `ResolveMongoID` method in the tender service to map MongoDB IDs and AI procedure references to canonical tender IDs, enhancing ID resolution.
- Updated `ToggleTenderApproval` to handle company ID validation and improve error handling, ensuring robust processing of tender approvals.
- Enhanced logging for error scenarios in the tender approval service, providing better insights into failures during operations.
- Refactored the `GetByID` and `GetByTenderAndCompany` methods in the tender approval repository to utilize custom error types for improved clarity.

This update significantly improves the handling of company and tender operations, enhancing error management and overall service reliability.
2026-07-08 02:55:55 +03:30
Mazyar 11e0b44ebe Add cache staleness check for recommendation page
continuous-integration/drone/push Build is passing
- Introduced a new method `isRecommendedPageCacheStale` in the `tenderService` to determine if the cached recommendations for a company are outdated based on the count of cached recommendations and the latest AI recommendations.
- Updated the `recommendFromPageCache` method to utilize the new cache staleness check, ensuring that outdated caches are not used for recommendations.
- This enhancement improves the accuracy of recommendations by ensuring that users receive the most up-to-date information, thereby enhancing the overall user experience in the tender recommendation service.
2026-07-08 02:41:00 +03:30
Mazyar f68b6d7787 Refactor recommendation response handling and enhance enrichment logic
continuous-integration/drone/push Build is passing
- Removed the `buildRecommendedTenderResponsesParallel` method and replaced it with `buildRecommendedTenderListResponses`, simplifying the response building process.
- Introduced `enrichRecommendedTenderResponsesParallel` to handle enrichment of tender responses in parallel, improving performance.
- Updated the `Recommend` method to utilize the new enrichment logic, ensuring timely and efficient processing of recommendations.
- Cleaned up unused functions and imports in `recommendation_page_cache.go` and `recommendation_response.go`, enhancing code clarity and maintainability.

This update streamlines the recommendation response handling and improves the overall efficiency of the tender recommendation service.
2026-07-08 02:27:25 +03:30
Mazyar 784c3d6563 Refactor recommendation page cache handling and configuration
continuous-integration/drone/push Build is passing
- Removed the `RecommendationPageCacheLanguages` configuration from `AISummarizerConfig` to streamline cache management.
- Updated the `companyService` and `tenderService` to utilize the new `InvalidateRecommendedTendersPageCache` method for cache invalidation, enhancing clarity and efficiency.
- Refactored the `invalidateRecommendedTendersPageCache` method to eliminate unnecessary context parameters, simplifying the function signature.
- Improved the handling of page cache refresh logic by consolidating language handling within the `tenderService`, ensuring consistent behavior across services.
- Cleaned up related tests and removed deprecated functions to maintain code quality and readability.

This update enhances the maintainability of the recommendation caching system by simplifying configuration and improving cache invalidation logic.
2026-07-08 02:20:16 +03:30
Mazyar 0b74e9ad23 Enhance tender recommendation caching and page refresh logic
continuous-integration/drone/push Build is passing
- Introduced a new `RecommendedTendersPageCacheRefresher` interface to manage the asynchronous refresh of recommendation pages in Redis, improving cache management.
- Updated the `companyService` to support setting page cache languages and refreshing the recommended tenders page cache based on company IDs.
- Enhanced the `tenderService` to build and invalidate recommended tenders page caches, ensuring timely updates and efficient retrieval of cached recommendations.
- Added configuration options for recommendation page cache languages in the `AISummarizerConfig`, allowing for flexible language support.
- Implemented unit tests for the new caching logic and page refresh functionality, ensuring robust validation of the recommendation caching process.

This update significantly improves the efficiency and responsiveness of the tender recommendation service by integrating enhanced caching mechanisms and page refresh capabilities.
2026-07-08 02:05:36 +03:30
Mazyar b28bc23975 Enhance tender reference handling with improved deduplication and test coverage
- Updated the `markResolvedRecommendedTenderSeen` function to accept an additional `tenderRef` parameter, allowing for better tracking of tender references and preventing duplicates.
- Refactored the `GetByProcedureReferences` method in the repository to combine indexed notice lookups with contract-folder candidates, ensuring older notice references resolve correctly after tender merges.
- Introduced a new `findTendersByContractFolderIDs` method to streamline the retrieval of tenders by contract folder IDs, enhancing performance and maintainability.
- Added unit tests for new functionality in `ai_reference_test.go` and `recommendation_filter_test.go`, ensuring robust validation of tender reference mapping and deduplication logic.

This update significantly improves the handling of tender references and enhances the overall test coverage, ensuring more reliable and efficient processing of tender data.
2026-07-08 01:55:33 +03:30
Mazyar f108733c2a Enhance tender recommendation service with improved caching and pagination
continuous-integration/drone/push Build is passing
- Introduced a context with a timeout for Redis cache retrieval in the `getCachedAIRecommendations` method, ensuring more robust handling of potential delays.
- Added a new file `recommendation_page.go` to encapsulate the logic for building recommendation pages, improving code organization and readability.
- Implemented a `buildRecommendationPage` method to handle pagination and batch processing of recommendations, enhancing performance and user experience.
- Created a new test file `recommendation_page_test.go` to validate the functionality of the recommendation page building process, ensuring correct pagination and batch resolution.

This update significantly improves the efficiency and maintainability of the tender recommendation service by optimizing caching and implementing structured pagination handling.
2026-07-08 01:32:27 +03:30
Mazyar 2980a3beb4 Refactor tender recommendation response handling and improve test coverage
continuous-integration/drone/push Build is passing
- Renamed the test function to `TestBuildRecommendedTenderListResponsesPreservesOrderWithoutStorage` for clarity and updated its logic to reflect changes in the recommendation response building process.
- Removed unnecessary sleep in the test to enhance performance.
- Refactored the `buildRecommendedTenderResponsesParallel` method to `buildRecommendedTenderListResponses`, simplifying the response building without relying on MinIO translation lookups.
- Updated the repository's `mapProcedureReferencesToTenders` function to improve efficiency by grouping candidates by their contract folder IDs, enhancing the mapping process.

This update streamlines the recommendation response handling and enhances the test suite for better maintainability and performance.
2026-07-08 01:23:21 +03:30
Mazyar adaae4c4bc Refactor tender approval service to optimize tender detail retrieval
continuous-integration/drone/push Build is passing
- Replaced the direct fetching of tender details within the `GetTenderApprovalsByCompanyIDWithTender`, `GetTenderApprovalsByStatusWithTender`, and `GetTenderApprovalsBySubmissionModeWithTender` methods with a new approach that loads tender details in bulk using `loadTendersByIDs`.
- Introduced helper functions `fullTenderDetailsFromResponse` and `listTenderDetailsFromResponse` to streamline the mapping of tender responses to `TenderDetails`.
- Added a new file `tender_details.go` to encapsulate tender detail mapping logic, improving code organization and readability.

This update enhances the performance of the tender approval service by reducing redundant calls to fetch tender details, leading to more efficient data handling.
2026-07-08 01:17:17 +03:30
Mazyar 107469c328 Enhance tender recommendation service with caching and parallel processing
continuous-integration/drone/push Build is passing
- Introduced a caching mechanism for translation responses in the tender recommendation service, improving efficiency by reducing redundant translation requests.
- Refactored the `enrichWithTranslation` method to support an optional translation cache, allowing for faster retrieval of stored translations.
- Implemented parallel processing for building recommended tender responses, utilizing goroutines to enhance performance and responsiveness.
- Added unit tests to validate the new caching behavior and ensure the correct application of translations in various scenarios.

This update significantly improves the performance and responsiveness of the tender recommendation service by integrating caching and parallel processing, enhancing the overall user experience.
2026-07-08 00:57:52 +03:30
Mazyar 0e4fadaf29 Enhance company recommendation caching and pipeline integration
continuous-integration/drone/push Build is passing
- Updated the company service to include a new method for scheduling the refresh of cached AI recommendations after the AI pipeline execution.
- Introduced a new interface for managing cached recommendation refreshes, improving the separation of concerns within the service layer.
- Enhanced the worker initialization to include Redis client support, allowing for better management of recommendation caching.
- Added functionality to list company IDs with existing recommendation caches, ensuring efficient updates post-pipeline runs.
- Implemented unit tests to validate the new recommendation refresh logic and ensure proper handling of various scenarios.

This update significantly improves the handling of AI recommendations by integrating caching mechanisms with the AI pipeline, enhancing overall system performance and responsiveness.
2026-07-08 00:45:53 +03:30
Mazyar 4e5296d5dd Add deduplication and normalization for tender recommendations
continuous-integration/drone/push Build is passing
- Introduced a new function `dedupeRecommendationResponsesByTenderID` to remove duplicate tender recommendations based on their IDs, ensuring unique entries in the recommendation list.
- Updated the `mergeRecommendedTenders` function to utilize the new deduplication logic, streamlining the merging process of recommendation lists.
- Enhanced the `fetchAIRecommendations` method to apply deduplication on the fetched recommendations, improving data integrity.
- Added unit tests for the new deduplication function to validate its behavior and ensure correct handling of various input scenarios.

This update enhances the recommendation handling by ensuring that duplicate tender entries are effectively managed, improving the overall quality of the recommendations provided to users.
2026-07-07 23:00:16 +03:30
Mazyar 1ad0206e61 Refactor AI pipeline handling to support daily runs
continuous-integration/drone/push Build is passing
- Renamed and refactored the AI pipeline auto run functionality to a daily run, enhancing clarity and purpose.
- Introduced a new `AIPipelineDailyWorker` to manage the daily execution of the AI pipeline, replacing the previous auto run implementation.
- Updated configuration fields and logging messages to reflect the change from auto to daily run, ensuring consistent terminology throughout the codebase.
- Removed the obsolete `ai_pipeline_auto.go` file to streamline the worker structure.

This update improves the maintainability and readability of the AI pipeline management by clearly distinguishing between auto and daily run functionalities.
2026-07-07 12:17:03 +03:30
Mazyar 89faa08b1c Add functionality to manage external links for companies
continuous-integration/drone/push Build is passing
- Introduced the ability to append external links to company profiles through a new API endpoint.
- Enhanced the `Company` entity to include a `Links` field for storing external resource links.
- Created `AddLinksForm` for validating incoming link data and implemented corresponding logic in the service layer.
- Added error handling for link validation, ensuring only valid URLs are accepted and limiting the number of links to 20.
- Implemented unit tests for link management functions, including sanitization and merging of links.
- Updated relevant API documentation to reflect the new functionality.

This update significantly enhances the company management capabilities by allowing the addition of external links, improving the overall user experience and data richness in company profiles.
2026-07-06 00:05:48 +03:30
Mazyar 3aeb8630e3 Add company selection handling in customer forms with unit tests
continuous-integration/drone/push Build is passing
- Introduced new functionality in `form_companies.go` to handle company selection from various input formats, including legacy `company_ids` and company summary objects.
- Implemented `UnmarshalJSON` methods for `CreateCustomerForm`, `UpdateCustomerForm`, and `AssignCompaniesForm` to support flexible company data parsing.
- Added unit tests in `form_companies_test.go` to validate the correct unmarshalling of company IDs and summaries, ensuring robust handling of different input scenarios.
- Enhanced the `GetByCompanies` method in the customer repository to directly use string company IDs, improving data retrieval efficiency.

This update significantly improves the handling of company data in customer forms, ensuring accurate processing and validation of company selections, while expanding test coverage for these functionalities.
2026-07-05 21:20:28 +03:30
Mazyar a762430bca Add unit tests for tender recommendation translation handling
continuous-integration/drone/push Build is passing
- Introduced a new test file `recommendation_translation_test.go` to validate the behavior of the tender recommendation service regarding translation handling.
- Implemented tests to ensure that the service correctly applies stored translations and retains original text when no translation is available.
- Enhanced the `Recommend` method in the tender service to utilize a structured approach for managing recommended tenders, improving clarity and maintainability.

This update improves the testing coverage for translation handling in tender recommendations, ensuring accurate responses based on available translations and original content.
2026-07-05 21:01:42 +03:30
Mazyar 3221a31ac8 Refactor tender service methods for rejected tenders and enhance unit tests
continuous-integration/drone/push Build is passing
- Renamed `buildRejectedTenderIDSetForCompanies` to `buildRejectedTenderIDSetsForCompanies` to better reflect its functionality of returning a map of excluded tender IDs by company.
- Updated the implementation to return a nested map structure for better organization of excluded tender IDs per company.
- Added a new test function `TestIsRecommendedTenderExcludedForAllCompanies` to validate the visibility of tenders based on company-specific exclusions.
- Enhanced existing tests to ensure comprehensive coverage of the new logic for handling rejected tenders.

This update improves the clarity and functionality of the tender recommendation process by refining the handling of rejected tenders and expanding test coverage.
2026-07-04 13:56:07 +03:30
Mazyar 4c48e0bb3b Add unit test for mapping procedure references to tenders
continuous-integration/drone/push Build is passing
- Introduced a new test function `TestMapProcedureReferencesToTenders` to validate the mapping of procedure references to corresponding tenders.
- Enhanced the `GetByProcedureReferences` method in the tender repository to utilize a more efficient filtering mechanism based on unique contract folder IDs.
- Updated pagination logic to dynamically adjust limits based on the number of unique folders and references, ensuring optimal data retrieval.

This update improves the testing coverage for tender mapping functionality and optimizes the repository's data fetching strategy, enhancing overall performance and reliability.
2026-07-04 13:39:14 +03:30
Mazyar 467090e5d2 Enhance AI recommendations handling with validation and caching improvements
continuous-integration/drone/push Build is passing
- Introduced a new method to validate company IDs for AI recommendations, ensuring all requested companies exist before processing.
- Added caching logic to retrieve AI recommendations efficiently, reducing unnecessary calls and improving performance.
- Utilized errgroup for concurrent processing of AI recommendations across multiple companies, enhancing responsiveness.
- Updated the tender service to handle rejected tenders more effectively by incorporating concurrent fetching of rejected tender IDs.

This update significantly improves the AI recommendations process by ensuring accurate company validation and optimizing data retrieval through caching and concurrency, enhancing overall system performance and user experience.
2026-07-04 13:28:41 +03:30
Mazyar 1edf42187d Implement AI recommendations for multiple companies and enhance company context handling
continuous-integration/drone/push Build is passing
- Added functionality to retrieve merged AI recommendations for multiple companies, improving the relevance of tender suggestions based on company-specific data.
- Introduced normalization functions to clean and deduplicate company IDs, ensuring accurate processing of recommendations.
- Enhanced the company context resolution in customer middleware to support multiple assigned companies, improving the handling of company-specific requests.
- Updated the tender recommendation logic to utilize the new merged recommendations and handle exclusions for rejected tenders accordingly.
- Added unit tests to verify the new recommendation merging logic and company ID normalization, ensuring robust functionality.

This update significantly enhances the tender recommendation process by allowing for more comprehensive and relevant suggestions based on multiple company contexts, improving user experience and satisfaction.
2026-07-04 13:00:33 +03:30
m.nazemi f8c98da132 Merge pull request 'Enhance tender service to exclude company-rejected tenders from recommendations' (#47) from recommendation-approval into develop
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/47
2026-07-04 10:46:28 +03:30
Mazyar fe3a71ba2b Enhance tender service to exclude company-rejected tenders from recommendations
- Updated the tender service to include a new dependency for listing rejected tenders by company, allowing for more refined tender recommendations.
- Introduced a new field in the SearchForm to specify whether to exclude rejected tenders from the recommendation results.
- Enhanced the Recommend method to filter out rejected tenders based on the new exclusion logic, improving the relevance of AI-ranked tender recommendations.
- Added unit tests to verify the exclusion logic for rejected tenders, ensuring robust functionality.

This update improves the tender recommendation process by ensuring that company-rejected tenders are not included in the results, enhancing user experience and satisfaction.
2026-07-03 19:55:01 +03:30
Mazyar ec7db8b9f0 Update tender approval handler documentation and default status logic
continuous-integration/drone/push Build is passing
- Enhanced API documentation for `GetTenderApprovalByTenderAndCompany` to clarify the default behavior of the status filter, indicating it defaults to 'submitted' when omitted.
- Updated the `PublicGetTenderApprovals` method to set the default status to 'submitted' if no status is provided, ensuring consistent behavior in approval retrieval.

This update improves the clarity of the API documentation and enhances the functionality of the tender approval retrieval process by establishing a default status, thereby improving user experience.
2026-07-03 16:23:42 +03:30
Mazyar 541d08a014 Enhance dashboard summary and statistics with new fields and caching improvements
continuous-integration/drone/push Build is passing
- Added new field `Days` and `ScrapedTED` to `SummaryResponse` for tracking daily TED scrape counts.
- Updated `SummaryQuery` to include `Days` parameter for querying scraped TED data.
- Modified `Summary` handler to accept and process the new `Days` parameter.
- Refactored `Summary` method in the repository to support the new `Days` parameter and improved aggregation logic.
- Enhanced caching logic in the service layer to utilize a composite cache key based on `closingWindowSec` and `days`, improving cache management and retrieval efficiency.

This update improves the dashboard's functionality by providing more detailed insights into TED scraping activities and optimizing the caching strategy for better performance.
2026-07-01 23:03:23 +03:30
Mazyar eeafe2a625 Add inquiry status transition error handling and validation improvements
continuous-integration/drone/push Build is passing
- Introduced InvalidStatusTransitionError to handle invalid status transitions for inquiries, providing clearer error messages.
- Updated UpdateInquiryStatusForm to make the Reason field optional and added logic to set a default reason if not provided.
- Enhanced form validation tests to cover new status transition error scenarios and validation messages.
- Refactored handler methods to utilize new error handling functions for improved response management.

This update improves the robustness of inquiry status management by ensuring proper error handling and validation, enhancing user experience during status updates.
2026-07-01 22:50:44 +03:30
Mazyar 506ac01cda Update company form validation to make fields optional and enhance service logic for registration number and tax ID checks
continuous-integration/drone/push Build is passing
- Modified CompanyForm fields (RegistrationNumber, TaxID, Industry) to be optional, allowing for more flexible company creation.
- Updated the Create method in company service to check for existing companies by registration number and tax ID only if provided, improving error handling and user experience during company creation.

This update enhances the company creation process by allowing optional fields while ensuring that existing company checks are performed only when necessary.
2026-07-01 19:53:35 +03:30
Mazyar 492f9ba3c8 Implement AI pipeline auto worker functionality
continuous-integration/drone/push Build is passing
- Introduced AIPipelineAutoWorker to manage the execution of the AI pipeline auto run, including startup catch-up and scheduled tasks.
- Enhanced WorkerConfig to include AIPipelineAutoEnabled and AIPipelineAutoInterval settings for better control over AI pipeline execution.
- Added logging for AI pipeline auto run status, including success and error handling, to improve observability.
- Updated daily job tracker to include AIPipelineAutoJobName for tracking AI pipeline job completions.

This update enhances the system's capability to automate AI pipeline executions, improving efficiency and reliability in processing AI tasks.
2026-07-01 19:42:36 +03:30
Mazyar c46a8d54f4 Refactor document scraper to utilize active publication windows for tender retrieval
continuous-integration/drone/push Build is passing
- Updated the ListPendingTenders method to filter tenders based on their active publication submission window instead of just deadlines, enhancing the accuracy of tender listings.
- Introduced a new HasActivePublicationWindow method in the Tender entity to determine if the publication-based submission window is still open.
- Modified the GetTenderByNoticeID method to reflect the new logic for filtering tenders based on their publication status.
- Enhanced the documentation for relevant API endpoints to clarify the changes in tender retrieval criteria.

This update improves the precision of tender scraping by ensuring only relevant tenders with active submission windows are processed, contributing to better data management and scraping efficiency.
2026-07-01 19:22:45 +03:30
Mazyar 7a9de273bb Enhance dashboard statistics with TED notice tracking
continuous-integration/drone/push Build is passing
- Added a new field, ScrapedTEDNotices, to the StatisticsLifetimeTotals struct to track the total number of TED notices scraped.
- Updated the Statistics method in the statistics repository to include a background process for retrieving total scraped TED notices, improving the accuracy of dashboard statistics.
- Introduced new methods in the Counter to increment and retrieve daily counts for scraped TED notices, ensuring reliable metrics for reporting.
- Modified the TEDScraper to increment the TED notice scraped counter upon successful import, enhancing the tracking of scraping activity.

This update improves the dashboard's statistics by providing detailed insights into TED notice scraping activities, contributing to better data visibility and reporting.
2026-06-30 23:28:12 +03:30
Mazyar 50c018af62 Enhance company context handling in customer middleware and routing
continuous-integration/drone/push Build is passing
- Introduced CompanyContextMiddleware to resolve the active company context for customer requests, ensuring that tender recommendations and company-scoped APIs remain in sync with the database.
- Updated public routes to utilize the new CompanyContextMiddleware alongside the existing AuthMiddleware, improving the handling of company-specific requests.
- Added unit tests for the pickActiveCompanyID function to validate the logic for selecting the appropriate company context based on customer assignments and requested company IDs.

This update enhances the accuracy and reliability of company context management in the application, improving user experience and data consistency.
2026-06-30 22:30:28 +03:30
Mazyar addd616d59 Enhance dashboard statistics loading with cold-cache handling
continuous-integration/drone/push Build is passing
- Introduced a new constant, statisticsColdLoadWait, to define the wait time for cold-cache requests before serving a placeholder response.
- Updated the Statistics method to initiate a background load for statistics while allowing for a brief wait for real data, improving responsiveness during cache warming.
- Implemented a channel to handle the result of the background loading process, ensuring that users receive timely feedback while the cache is being populated.

This update optimizes the dashboard's performance by ensuring that users are served quickly, even when the statistics are being freshly loaded from the backend.
2026-06-30 22:24:04 +03:30
Mazyar 12d1cabf7e Refactor dashboard statistics retrieval to improve performance
continuous-integration/drone/push Build is passing
- Updated the Statistics method in the dashboard service to initiate a background refresh for statistics while serving a placeholder report, enhancing responsiveness during cache warming.
- Removed blocking calls to load statistics directly, allowing for faster response times.
- Improved logging to indicate when placeholder statistics are served, providing better visibility into the caching process.

This update optimizes the dashboard's performance by ensuring that users receive immediate feedback while the system prepares accurate statistics in the background.
2026-06-30 22:10:30 +03:30
Mazyar 8dbd9927b0 Enhance dashboard service with Redis caching for statistics
continuous-integration/drone/push Build is passing
- Updated the dashboard service to integrate Redis caching for improved performance in statistics retrieval.
- Modified the NewService function to accept a Redis client, enabling caching of dashboard statistics.
- Implemented logic to retrieve statistics from Redis, falling back to the database if necessary, and introduced a background process to warm the cache.
- Enhanced error handling and logging for Redis operations to ensure robust statistics management.
- Increased cache duration for scraped documents and adjusted timeout settings for MongoDB queries to optimize performance.

This update significantly improves the responsiveness and efficiency of the dashboard by leveraging Redis for caching statistics.
2026-06-30 19:03:22 +03:30
Mazyar 3002935b76 Update dashboard statistics caching and retrieval logic
continuous-integration/drone/push Build is passing
- Increased the cache duration for dashboard statistics from 60 seconds to 5 minutes, improving data freshness and reducing load on the backend.
- Introduced a stale cache mechanism that allows retrieval of stale statistics while refreshing them in the background, enhancing user experience by providing quicker access to data.
- Updated the statistics repository to handle the new caching logic, ensuring accurate and timely statistics reporting.
- Added tests to validate the new caching behavior and ensure the integrity of statistics retrieval.

This update optimizes the dashboard's performance and responsiveness by improving the caching strategy for statistics.
2026-06-30 18:50:47 +03:30
Mazyar 7aacb7dfc9 Enhance pagination and repository functionality for MongoDB
continuous-integration/drone/push Build is passing
- Updated the ListPaginationOptions struct to include SkipCount and IncludeCount fields, allowing for more flexible pagination behavior.
- Modified the BuildListPagination function to handle cursor pagination with count options, improving performance by running count queries in parallel with data retrieval.
- Enhanced the FindAll method in the repository to support concurrent counting of documents, reducing overall latency for list operations.
- Added tests for pagination behavior, ensuring accurate handling of count scenarios in both offset and cursor pagination.

This update improves the efficiency and flexibility of pagination in the MongoDB repository, enhancing the overall performance of list operations.
2026-06-30 18:35:34 +03:30
Mazyar 20ce9c53ff Enhance Tender entity with PublicationSubmissionDeadline method and corresponding tests
continuous-integration/drone/push Build is passing
- Added the PublicationSubmissionDeadline method to the Tender entity, which calculates the submission deadline based on the publication date and stored submission deadline.
- Updated the IsRecommendable method to utilize the new PublicationSubmissionDeadline logic for determining recommendation eligibility.
- Introduced unit tests for the PublicationSubmissionDeadline method, ensuring accurate calculation and validation of submission deadlines.
- Enhanced existing tests for the IsRecommendable method to cover new scenarios related to submission deadlines.

This update improves the accuracy of submission deadline handling and enhances the recommendation logic for tenders based on publication dates.
2026-06-30 18:12:55 +03:30
Mazyar ff6bdfcb09 Add IsRecommendable method and corresponding tests for Tender entity
continuous-integration/drone/push Build is passing
- Implemented the IsRecommendable method in the Tender entity to determine if a tender should be included in AI recommendations based on its status and deadline.
- Added unit tests for the IsRecommendable method to cover various scenarios, ensuring accurate recommendation logic.
- Updated the Recommend method in the tender service to utilize the new IsRecommendable method for improved clarity and functionality.

This update enhances the recommendation logic for tenders, ensuring only appropriate tenders are considered for AI recommendations based on their status and deadlines.
2026-06-30 17:22:06 +03:30
m.nazemi 86789dcd20 Merge pull request 'Implement audit logging functionality across services and middleware' (#46) from audit-log into develop
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/46
2026-06-30 13:04:22 +03:30
Mazyar 92c6c7d99a Refactor Elasticsearch client error handling and introduce bulk index action marshaling
- Updated error handling in the Elasticsearch client to utilize a new `readErrorBody` function for better diagnostics when pinging and searching.
- Refactored the `flushBatch` method to marshal bulk index actions using a dedicated `marshalBulkIndexAction` function, improving code clarity and error handling.
- Enhanced the overall structure of the Elasticsearch client for improved maintainability and readability.

This update enhances the robustness of the Elasticsearch client, ensuring more informative error messages and cleaner code organization.
2026-06-30 13:03:39 +03:30
Mazyar 241e3b5f2d Implement audit logging functionality across services and middleware
- Introduced a new `auditlog` package to handle audit logging for user actions, including creation, updates, deletions, and authentication events.
- Enhanced existing services (customer, user) to log relevant actions using the new audit logger, capturing details such as actor ID, action type, target type, and success status.
- Added middleware to enrich request context with metadata for audit logging, ensuring comprehensive tracking of user interactions.
- Integrated Elasticsearch for persistent storage of audit logs, with fallback to file-only logging if Elasticsearch is unavailable.
- Updated API documentation to include new audit log endpoints for administrative access.

This update significantly improves the system's ability to track and audit user actions, enhancing security and accountability within the application.
2026-06-29 21:11:33 +03:30
Mazyar 534213f10e Enhance notification search functionality with user ID filtering
continuous-integration/drone/push Build is passing
- Added a new `UserID` field to the `SearchForm` to allow filtering notifications by user ID.
- Implemented the `ResolvedRecipients` method to return user IDs based on the `user_id` or `recipient` query parameters, improving the flexibility of notification searches.
- Updated the `GetNotifications` method in the service layer to utilize the new recipient resolution logic, ensuring accurate retrieval of notifications based on user ID.

This update enhances the notification management capabilities, providing more granular control over notification searches.
2026-06-28 11:04:24 +03:30
Mazyar 6d5694977b Add .gitattributes file for text file handling
- Introduced a new `.gitattributes` file to ensure consistent handling of text files across different platforms by setting the `text=auto` attribute.
- This addition helps maintain cross-platform compatibility and prevents potential issues with line endings in text files.

This update lays the groundwork for better version control practices in the project, enhancing collaboration among developers working on different operating systems.
2026-06-28 00:36:10 +03:30
Mazyar 39ac76e7b0 Add scrapedDocumentsScanner interface and enhance document scanning logic
continuous-integration/drone/push Build is passing
- Introduced the `scrapedDocumentsScanner` interface to facilitate scanning of scraped documents from MinIO, returning both procedure summaries and daily document counts.
- Updated the `ListProceduresWithDocuments` method to utilize the new scanning functionality, improving data retrieval efficiency.
- Enhanced the `scrapedDocumentsPerDay` method to filter daily counts based on a specified start date, ensuring accurate reporting of document statistics.
- Added unit tests for the new scanning logic and daily counts filtering, ensuring robust functionality and error handling.

This update enhances the dashboard's document management capabilities, providing better insights into scraped documents and their daily counts.
2026-06-28 00:28:45 +03:30
Mazyar 582f8b5c02 Enhance TED scraper and worker initialization with startup catch-up logic
continuous-integration/drone/push Build is passing
- Introduced a mutex to ensure only one TED scraper run executes at a time, preventing concurrent executions during startup and scheduled runs.
- Implemented a mechanism to check if today's TED scrape has already been completed during startup, logging appropriate messages for both completed and new runs.
- Added startup catch-up logic for tender translations and unprocessed notices, ensuring that any missed tasks are executed without blocking the application startup.

This update improves the reliability and efficiency of the TED scraper and worker processes, ensuring that all necessary tasks are completed after a server restart.
2026-06-28 00:05:10 +03:30
Mazyar 69445130ce Refactor AI recommendation fetching and caching in company service
continuous-integration/drone/push Build is passing
- Introduced retry logic for fetching AI recommendations after onboarding, enhancing reliability in recommendation retrieval.
- Updated logging levels for better observability, changing cache miss logs to Info level.
- Renamed methods for clarity, replacing `refreshAIRecommendationsCacheAsync` with `scheduleRecommendationRefreshAfterOnboarding` and `fetchAndCacheAIRecommendations` with `fetchAIRecommendations`.
- Implemented a mechanism to clear the cache if no recommendations are returned, improving cache management.

This update optimizes the AI recommendation process, ensuring more robust handling of recommendation fetching and caching during onboarding.
2026-06-23 13:54:54 +03:30
Mazyar 20518e7b64 Enhance AI recommendation caching and onboarding process in company service
continuous-integration/drone/push Build is passing
- Updated `AISummarizerConfig` to allow for a default `RecommendationCacheTTL` of 0, enabling persistent caching until company updates.
- Refactored `StartAIOnboarding` to include cache invalidation and asynchronous recommendation refresh, improving responsiveness during onboarding.
- Introduced `triggerAIOnboardingAsync` method for background processing of AI onboarding and cache refresh, enhancing user experience.
- Improved logging for AI onboarding and recommendation fetching processes, providing better observability and error tracking.

This update optimizes the AI recommendation caching mechanism and onboarding workflow, ensuring a smoother and more efficient experience for users.
2026-06-23 13:43:39 +03:30
Mazyar 326e49886b Implement asynchronous AI recommendation cache refresh in company service
continuous-integration/drone/push Build is passing
- Introduced `refreshAIRecommendationsCacheAsync` method to refresh AI recommendations in the background, improving responsiveness by serving the previous cache until the refresh completes.
- Updated `StartAIOnboarding` to call the new asynchronous cache refresh method instead of invalidating the cache directly.
- Added logging for cache refresh operations, including success and error handling, to enhance observability.

This update enhances the AI recommendation caching mechanism, providing a smoother onboarding experience and reducing latency in recommendation retrieval.
2026-06-23 13:33:33 +03:30
Mazyar a2661651c9 Enhance company service with AI recommendation caching and onboarding improvements
continuous-integration/drone/push Build is passing
- Updated the `companyService` to include Redis caching for AI recommendations, improving performance and reducing redundant AI calls.
- Introduced asynchronous AI onboarding triggered after company profile updates, enhancing user experience by offloading processing.
- Added configuration for recommendation cache TTL in the `AISummarizerConfig`, allowing for flexible cache management.
- Implemented methods for caching, retrieving, and invalidating AI recommendations in the `companyService`, ensuring efficient data handling.

This update enhances the company's AI recommendation capabilities, providing faster responses and a more efficient onboarding process.
2026-06-23 13:24:45 +03:30
Mazyar a7a49fc411 Enhance tender management with recommended tenders endpoint and search form update
continuous-integration/drone/push Build is passing
- Added a new endpoint in the `tender` handler for retrieving AI-ranked tender recommendations for a company, improving the functionality of the admin panel.
- Updated the `SearchForm` to include a query parameter for `only_active_deadlines`, allowing for more flexible search options.
- Enhanced API documentation for the new endpoint to provide clear usage instructions and expected parameters.

This update improves the tender management system by providing administrators with better tools for accessing relevant tender information.
2026-06-23 13:09:27 +03:30
Mazyar 8fad771b7a Update notification response message for clarity
continuous-integration/drone/push Build is passing
- Changed the response message in the `Send` method of the `NotificationHandler` from "Notification sent successfully" to "Notification created successfully" to better reflect the action performed.

This update improves the clarity of the notification response, ensuring users receive accurate feedback on their actions.
2026-06-22 23:46:56 +03:30
Mazyar 992d41374f Update company entity and repository for document file ID handling
continuous-integration/drone/push Build is passing
- Modified the `DocumentFileIDs` field in the `Company` struct to ensure it is always persisted in BSON format, enhancing data consistency.
- Implemented logic in the `Update` method of the `companyRepository` to clear document file IDs when the slice is empty, ensuring accurate database updates.
- Added logging for errors encountered while clearing document file IDs, improving error tracking and debugging capabilities.

This update enhances the management of document file IDs within the company domain, ensuring proper handling and persistence in the database.
2026-06-22 23:38:57 +03:30
Mazyar 3ba33459e8 Update employee count validation in SearchForm for company domain
continuous-integration/drone/push Build is passing
- Modified the `EmployeeCount` field validation in `SearchForm` to use a range of 1 to 100000, ensuring more accurate input constraints.
- Updated `EmployeeCountMin` and `EmployeeCountMax` fields to reflect the same range validation, enhancing the robustness of search queries.

This update improves the validation logic for employee count fields in the company search functionality, providing better input handling and user experience.
2026-06-22 22:42:01 +03:30
Mazyar 5fbfcd4149 Enhance company document management with file ID sanitization and detachment functionality
continuous-integration/drone/push Build is passing
- Updated the `UploadDocuments` method to sanitize document file IDs before saving, ensuring only valid references are stored.
- Introduced `DetachDocumentFileID` method in the `companyService` to remove file IDs from all companies referencing a deleted file, improving data integrity.
- Enhanced the `companyRepository` with a new method to handle the removal of document file IDs from the database.
- Updated the `filestore` handler to utilize the new detachment functionality when files are deleted, ensuring consistent state across domain entities.

This update improves the management of document file IDs within the company domain, enhancing data integrity and reference handling.
2026-06-22 22:37:50 +03:30
Mazyar 3bfed0dc74 Enhance company search functionality with additional filters
continuous-integration/drone/push Build is passing
- Updated the `SearchForm` to include optional filters for `name`, `email`, `phone`, and `employee_count`, allowing for more granular search capabilities.
- Modified the `Search` method in the `companyRepository` to handle the new filters, improving the search logic with regex support for `name`, `email`, and `phone`.
- Updated API documentation in the `company` handler to reflect the new query parameters for enhanced clarity.

This update improves the search functionality within the company domain, providing users with more flexible and precise search options.
2026-06-22 22:26:26 +03:30
Mazyar 2a9682aea2 Enhance company repository search functionality
continuous-integration/drone/push Build is passing
- Updated the `Search` method in the `companyRepository` to improve search capabilities by allowing regex-based filtering on `name`, `email`, and `phone` fields.
- Added logic to handle numeric search queries for `employee_count`, enhancing the search flexibility.
- Implemented input sanitization to trim whitespace from search queries, ensuring cleaner search inputs.

This update improves the search functionality within the company repository, providing more robust and flexible search options for users.
2026-06-22 22:16:36 +03:30
Mazyar ce8a18aa8b Add unit tests for UpdateUserForm JSON unmarshalling
continuous-integration/drone/push Build is passing
- Introduced `form_test.go` to validate the behavior of the `UpdateUserForm` struct's JSON unmarshalling, specifically for the `profile_image` field.
- Added tests to ensure omitted `profile_image` is not treated as an update, null values clear the image, and valid URLs are preserved.
- Enhanced the `UpdateUserForm` and `UpdateProfileForm` structs to include logic for distinguishing between omitted and explicitly null profile images during JSON unmarshalling.

This update improves the robustness of user profile updates by ensuring correct handling of profile image data in JSON requests, enhancing overall data integrity in the user management system.
2026-06-22 22:09:30 +03:30
Mazyar d486a5e44f Enhance notification delivery system with scheduled processing
continuous-integration/drone/push Build is passing
- Introduced a new `NotificationWorker` to promote due scheduled notifications from pending to sent, improving notification management.
- Added `NotificationInterval` configuration to schedule the notification delivery worker, with a default value for flexibility.
- Implemented `MarkDueScheduledAsSent` method in the notification repository to update the status of notifications based on their delivery time.
- Updated the notification service to process due scheduled notifications during relevant operations, ensuring timely delivery.

This update enhances the notification system by automating the delivery of scheduled notifications, improving user engagement and operational efficiency.
2026-06-22 12:53:22 +03:30
Mazyar db14bfe270 Enhance tender repository and service with AI procedure reference handling
continuous-integration/drone/push Build is passing
- Introduced `ProcedureReference` struct to encapsulate AI procedure coordinates for better data management.
- Added `GetByProcedureReferences` method in the `TenderRepository` to retrieve multiple tenders based on AI procedure references in a single query.
- Updated the `resolveRecommendedTenders` method in the service layer to utilize the new repository method, improving efficiency in fetching recommended tenders.
- Enhanced error handling and logging for the new repository method to ensure robust operation.

This update improves the handling of AI procedure references, streamlining tender retrieval processes and enhancing overall system performance.
2026-06-21 15:43:58 +03:30
Mazyar fa258020f1 Add AI procedure reference formatting and parsing functions with unit tests
continuous-integration/drone/push Build is passing
- Introduced `FormatAIProcedureRef` and `ParseAIProcedureRef` functions in the `tender` domain for handling AI service tender references.
- Added unit tests for these functions in `ai_reference_test.go` to ensure correct parsing and formatting behavior.
- Updated the `TenderResponse` struct to include a new `ProcedureRef` field for improved data representation.
- Enhanced the `GetByProcedureReference` method in the repository to retrieve tenders based on the new procedure reference format.
- Modified the `Recommend` method in the service layer to utilize the new procedure reference handling, improving the recommendation process.

This update enhances the handling of AI procedure references, ensuring better data integrity and usability in the tender management system.
2026-06-21 15:36:36 +03:30
Mazyar 2b6e25f979 Update rank field type in tender and onboarding response structures
continuous-integration/drone/push Build is passing
- Changed the `Rank` field type from `string` to `int` in the `RecommendedTenderResponse` struct within the `company` domain for better data representation.
- Updated the `Rank` field type from `string` to `int` in the `TenderResponse` struct within the `tender` domain to ensure consistency in ranking data.
- Modified the `Rank` field type from `string` to `int` in the `RecommendedTender` struct within the AI summarizer package to align with the updated data structure.

This update enhances the data integrity and consistency across the tender management system by standardizing the rank representation as an integer.
2026-06-21 15:24:19 +03:30
Mazyar b671dc3fd8 Enhance notification system with in-app delivery support
continuous-integration/drone/push Build is passing
- Added `DeliveryChannelInApp` to support in-app notifications.
- Introduced `EventTypeInApp` constant for identifying in-app notifications.
- Updated `AdminNotifications` and `CustomerNotifications` handlers to set the event type to `EventTypeInApp`.
- Modified the `GetByUserID` repository method to handle filtering for in-app notifications, allowing for more flexible retrieval options.
- Updated the `persistNotification` method to include in-app as a delivery method.

This update improves the notification system by enabling in-app notifications, enhancing user engagement and notification management capabilities.
2026-06-21 14:47:15 +03:30
Mazyar 2538747768 Enhance tender response structure and update recommendation handler
continuous-integration/drone/push Build is passing
- Added new fields `Rank` and `Analysis` to the `TenderResponse` struct for improved data representation.
- Updated the `RecommendTenders` handler to reflect changes in API documentation, including a new summary and description for AI-ranked tender recommendations.
- Improved error handling in the recommendation process, ensuring appropriate responses for missing company IDs and unavailable AI services.

This update enhances the tender response capabilities and improves the clarity of the recommendation API, aligning with the overall goal of providing better insights for users.
2026-06-21 12:29:37 +03:30
Mazyar e04cdd1d10 Refactor AI summarization functionality by removing document summarization worker
continuous-integration/drone/push Build is passing
- Removed the DocumentSummarizationWorker and its related scheduling logic from the worker bootstrap.
- Updated the AI summarizer client initialization comment for clarity.
- Added a new error type for cases when tender documents have not been scraped yet, enhancing error handling in the tender service.
- Modified API documentation to reflect changes in AI summary retrieval logic, ensuring accurate descriptions of on-demand summarization behavior.

This update streamlines the AI summarization process by eliminating the document summarization worker, improving overall system efficiency and clarity in error handling.
2026-06-21 11:20:28 +03:30
Mazyar e31bccced6 Add unit tests for dashboard repository and enhance BSON handling
continuous-integration/drone/push Build is passing
2026-06-21 10:03:17 +03:30
Mazyar 45cfa24a72 Enhance document scraper service with AI portal integration and error handling
- Updated the document scraper service to include a new ScrapePortalsProvider interface, allowing for dynamic retrieval of supported scraping portals.
- Modified the ListPendingTenders and GetTenderByNoticeID methods to filter tenders based on document URLs that match the configured portals.
- Introduced new error handling for cases when the scrape portals provider is not configured, returning appropriate service unavailable responses.
- Enhanced API documentation to reflect changes in tender retrieval logic and added error response details for unsupported portal scenarios.

This update improves the document scraping functionality by integrating AI portal support, enhancing the overall reliability and flexibility of the tender management system.
2026-06-21 09:58:11 +03:30
m.nazemi 550f11a77e Merge pull request 'Enhance tender estimated value resolution and add unit tests' (#45) from est-value into develop
continuous-integration/drone/push Build is passing
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/45
2026-06-21 09:42:00 +03:30
Mazyar ee5414bc10 Enhance procurement lot estimated value aggregation and add unit tests
- Updated the `AggregateProcurementLotEstimatedValue` function to reject mixed currencies and allow empty lot currencies, ensuring accurate aggregation of estimated values.
- Introduced new unit tests in `entity_test.go` to validate the behavior of the aggregation function under various scenarios, including mixed currencies and empty lot currencies.
- Refactored budget calculation in `budget_test.go` to utilize the new aggregation logic, improving consistency in budget retrieval.

This update improves the handling of estimated values in procurement lots, enhancing the reliability of the tender management system.
2026-06-21 09:41:33 +03:30
m.nazemi a04b3fe1ec Merge branch 'develop' into est-value 2026-06-21 09:37:00 +03:30
Mazyar dfab3e17d2 Enhance dashboard repository and service with caching and aggregation improvements
- Introduced caching mechanisms for summary and statistics in the dashboard service, improving performance by reducing redundant data retrieval.
- Refactored the Summary method to utilize MongoDB aggregation for more efficient data processing and retrieval.
- Added synchronization features using singleflight to prevent duplicate processing of requests for cached data.
- Updated the repository to include a cachedScrapedTendersScope method, enhancing the efficiency of scraped document statistics retrieval.

This update significantly optimizes the dashboard's performance and data handling capabilities, ensuring faster response times and reduced load on the database.
2026-06-21 09:36:12 +03:30
Mazyar 4cfca5aa55 Enhance tender estimated value resolution and add unit tests
- Updated the `ResolvedEstimatedValueAndCurrency` method to aggregate procurement lot values when the tender-level estimated value is not set, improving accuracy in value retrieval.
- Introduced the `AggregateProcurementLotEstimatedValue` function to sum estimated values from procurement lots and return the first found currency.
- Modified the `ToResponseWithLanguage` method to utilize the new estimated value resolution logic.
- Added unit tests for the new functionality, ensuring correct behavior for various scenarios in the `entity_test.go` and `budget_test.go` files.

This update improves the handling of estimated values in tenders, enhancing the overall reliability of the tender management system.
2026-06-20 12:29:47 +03:30
Mazyar 6fb57c41c1 Refactor ScrapePortalsResponse structure and update API documentation
continuous-integration/drone/push Build is passing
- Changed the ScrapePortalsResponse type to return a slice of strings representing portal identifiers instead of a structured object.
- Updated the Swagger documentation for the GetScrapePortals endpoint to reflect the new response format, ensuring clarity in API usage.

This update simplifies the response structure for the scraping portals, enhancing the API's usability and consistency.
2026-06-18 20:56:31 +03:30
Mazyar e5fa0dfe47 Add GetScrapePortals endpoint and related service functionality
continuous-integration/drone/push Build is passing
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.

This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
2026-06-18 19:53:07 +03:30
m.nazemi 2f19ecae55 Merge pull request 'Refactor dashboard repository to integrate ProcedureDocumentsLister' (#44) from dashboard-documents into develop
continuous-integration/drone/push Build is passing
Reviewed-on: https://repo.ravanertebat.com/TM/tm_back/pulls/44
2026-06-17 15:02:23 +03:30
Mazyar 8035118f44 Enhance dashboard statistics repository with scraped tenders scope resolution
- Introduced a new `scrapedTendersScope` type to encapsulate the MongoDB filter for tenders with scraped documents, improving clarity and maintainability.
- Updated the `Statistics` method to utilize the new scope resolution, allowing for more accurate data retrieval based on the presence of scraped documents.
- Implemented multiple tests for the `resolveScrapedTendersScope` method, ensuring correct behavior for various scenarios, including empty and fallback cases.

This update enhances the dashboard's ability to manage scraped document statistics, improving overall data accuracy and system performance.
2026-06-17 15:01:52 +03:30
Mazyar 9676f99304 Refactor dashboard repository to integrate ProcedureDocumentsLister
- Introduced the ProcedureDocumentsLister interface to list contract folders with scraped documents, enhancing the accuracy of document-scrape statistics.
- Updated the dashboard repository to accept ProcedureDocumentsLister as a dependency, allowing for improved data retrieval.
- Implemented tests for the new functionality, ensuring proper handling of scraped document folder IDs and error propagation.

This update enhances the dashboard's capability to manage and report on scraped documents, improving overall system efficiency and data integrity.
2026-06-17 14:17:41 +03:30
150 changed files with 12803 additions and 1390 deletions
+1
View File
@@ -0,0 +1 @@
* text=auto
+58 -7
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"os"
"sync"
"time"
"tm/internal/notice"
"tm/pkg/config"
@@ -14,6 +15,9 @@ import (
"tm/ted"
)
// tedScraperRunMu ensures only one TED scraper run executes at a time (startup catch-up and cron).
var tedScraperRunMu sync.Mutex
// Init Application Configuration
func InitConfig() (*Config, error) {
conf, err := config.LoadConfig(".", &Config{})
@@ -101,22 +105,64 @@ func InitTEDScraper(config Config, mongoManager *mongo.ConnectionManager, appLog
// start TED scraper job with error handling
scheduler := schedule.NewCronScheduler(appLogger, mongoManager, noticeRepo)
dailyJobTracker := schedule.NewDailyJobTracker(mongoManager, appLogger)
runDailyScrape := func(trigger string) {
tedScraperRunMu.Lock()
defer tedScraperRunMu.Unlock()
function := func() {
ctx := context.Background()
today := time.Now().Local()
if trigger == "startup" {
completed, checkErr := dailyJobTracker.IsCompleted(ctx, schedule.TEDScraperJobName, today)
if checkErr != nil {
appLogger.Error("Failed to check daily scrape completion status", map[string]interface{}{
"error": checkErr.Error(),
"date": today.Format("02/01/2006"),
})
} else if completed {
appLogger.Info("Startup catch-up skipped: today's TED scrape already completed", map[string]interface{}{
"date": today.Format("02/01/2006"),
})
return
}
appLogger.Info("Running startup catch-up for today's TED scrape", map[string]interface{}{
"date": today.Format("02/01/2006"),
})
} else {
appLogger.Info("Running scheduled TED scrape", map[string]interface{}{
"date": today.Format("02/01/2006"),
})
}
err := tedScraper.Run(ctx, nil, nil)
if err != nil {
appLogger.Error("Scheduled scraper run failed", map[string]interface{}{
"error": err.Error(),
appLogger.Error("TED scraper run failed", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
"error": err.Error(),
})
// Don't exit - continue running for next scheduled execution
} else {
appLogger.Info("Scheduled scraper run completed successfully", map[string]interface{}{})
return
}
if markErr := dailyJobTracker.MarkCompleted(ctx, schedule.TEDScraperJobName, today); markErr != nil {
appLogger.Error("Failed to mark daily scrape as completed", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
"error": markErr.Error(),
})
}
appLogger.Info("TED scraper run completed successfully", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
})
}
err := scheduler.AddJob(schedule.Job{
Name: "TED Scraper Job",
Func: function,
Func: func() { runDailyScrape("scheduled") },
Expr: config.TED.ScrapingInterval,
})
if err != nil {
@@ -126,6 +172,11 @@ func InitTEDScraper(config Config, mongoManager *mongo.ConnectionManager, appLog
return nil
}
// After a server restart, resume today's scrape if it was interrupted or never ran.
go func() {
runDailyScrape("startup")
}()
// Start the scheduler
scheduler.Start()
+35 -1
View File
@@ -4,8 +4,10 @@ import (
"fmt"
"time"
ai_summarizer "tm/pkg/ai_summarizer"
"tm/pkg/audit"
"tm/pkg/authorization"
"tm/pkg/config"
"tm/pkg/elasticsearch"
"tm/pkg/filestore"
"tm/pkg/gorules"
"tm/pkg/hcaptcha"
@@ -90,6 +92,7 @@ func InitHTTPServer(conf Config, log logger.Logger) *echo.Echo {
// Add middleware
e.Use(middleware.Recover())
e.Use(middleware.RequestID())
e.Use(audit.RequestMetaMiddleware())
e.Use(middleware.Logger())
e.Use(middleware.CORSWithConfig(middleware.CORSConfig{
AllowOrigins: []string{"*"},
@@ -132,7 +135,7 @@ func InitHTTPServer(conf Config, log logger.Logger) *echo.Echo {
e.GET("/swagger*", echoSwagger.WrapHandler)
log.Info("HTTP server initialized successfully", map[string]interface{}{
"middleware": []string{"recover", "request_id", "logger", "cors", "custom_logging"},
"middleware": []string{"recover", "request_id", "audit_meta", "logger", "cors", "custom_logging"},
})
return e
@@ -402,3 +405,34 @@ func InitGoRulesClient(_ GoRulesConfig, log logger.Logger) gorules.Client {
return client
*/
}
// InitElasticsearch creates an Elasticsearch client for audit log storage.
func InitElasticsearch(conf config.ElasticsearchConfig, log logger.Logger) elasticsearch.Client {
esConfig := elasticsearch.Config{
URL: conf.URL,
Username: conf.Username,
Password: conf.Password,
IndexPrefix: conf.IndexPrefix,
Enabled: conf.Enabled,
RequestTimeout: conf.RequestTimeout,
BulkFlushPeriod: conf.BulkFlushPeriod,
QueueSize: conf.QueueSize,
}
client, err := elasticsearch.NewClient(esConfig, log)
if err != nil {
log.Warn("Elasticsearch unavailable, audit logs will be file-only", map[string]interface{}{
"error": err.Error(),
"url": conf.URL,
})
return elasticsearch.NewNoopClient()
}
return client
}
// InitAuditLogger creates a composite audit logger with optional Elasticsearch persistence.
func InitAuditLogger(log logger.Logger, esClient elasticsearch.Client) (audit.Logger, audit.Store) {
store := elasticsearch.NewAuditStore(esClient)
return audit.NewCompositeLogger(log, store), store
}
+4
View File
@@ -23,6 +23,7 @@ type Config struct {
DocumentScraper DocumentScraperConfig
AISummarizer AISummarizerConfig
GoRules GoRulesConfig
Elasticsearch config.ElasticsearchConfig
}
type ScraperConfig struct {
@@ -69,6 +70,9 @@ type AISummarizerConfig struct {
APIRetryCount int `env:"AI_SUMMARIZER_API_RETRY_COUNT" envDefault:"2"`
APIRetryDelay time.Duration `env:"AI_SUMMARIZER_API_RETRY_DELAY" envDefault:"3s"`
DefaultLanguage string `env:"AI_SUMMARIZER_DEFAULT_LANGUAGE" envDefault:"en"`
// RecommendationCacheTTL is an optional max Redis TTL for cached recommendations.
// When 0 (default), entries persist until the company is created, updated, or onboarded.
RecommendationCacheTTL time.Duration `env:"AI_RECOMMENDATION_CACHE_TTL" envDefault:"0"`
// MinIO storage settings
MinioEndpoint string `env:"AI_SUMMARIZER_MINIO_ENDPOINT" envDefault:""`
+63 -16
View File
@@ -62,6 +62,9 @@ package main
// @tag.name Admin-AI-Pipeline
// @tag.description Administrative Opplens AI pipeline operations including scrape, summarize, translate, sync, and background pipeline runs
// @tag.name Admin-AuditLogs
// @tag.description Administrative user action audit log search with filtering by actor, action, target, and time range
// @tag.name Authorization
// @tag.description Customer authentication and authorization operations for mobile application including login, logout, token refresh, and profile access
@@ -100,8 +103,9 @@ import (
"fmt"
"net/http"
"time"
"tm/internal/assets"
"tm/internal/ai_pipeline"
"tm/internal/assets"
"tm/internal/auditlog"
"tm/internal/cms"
"tm/internal/company"
"tm/internal/company_category"
@@ -115,8 +119,8 @@ import (
"tm/internal/notification"
"tm/internal/tender"
"tm/internal/tender_approval"
"tm/internal/tender_submission"
"tm/internal/user"
"tm/pkg/audit"
"tm/pkg/filestore"
"tm/pkg/security"
@@ -171,6 +175,7 @@ func main() {
// Initialize AI Summarizer service
var aiSummarizerClient tender.AISummarizerClient
var aiSummarizerStorage tender.AISummarizerStorage
var procedureDocumentsLister dashboard.ProcedureDocumentsLister
var aiRecommendationClient company.AIRecommendationClient
var aiPipelineClient ai_pipeline.Client
@@ -181,6 +186,7 @@ func main() {
}
if s := bootstrap.InitAISummarizerStorage(conf.AISummarizer, logger); s != nil {
aiSummarizerStorage = s
procedureDocumentsLister = s
}
// Initialize authorization service
@@ -196,6 +202,7 @@ func main() {
tenderRepository := tender.NewRepository(mongoManager, logger)
feedbackRepo := feedback.NewRepository(mongoManager, logger)
tenderApprovalRepository := tender_approval.NewRepository(mongoManager, logger)
tenderSubmissionRepository := tender_submission.NewRepository(mongoManager, logger)
inquiryRepository := inquiry.NewRepository(mongoManager, logger)
contactRepository := contact.NewContactRepository(mongoManager, logger)
cmsRepository := cms.NewRepository(mongoManager, logger)
@@ -205,7 +212,7 @@ func main() {
cardRepository := kanban.NewCardRepository(mongoManager, logger)
notificationRepository := notification.NewRepository(mongoManager, logger)
logger.Info("Repositories initialized successfully", map[string]interface{}{
"repositories": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "inquiry", "contact", "cms", "notice", "kanban_board", "kanban_column", "kanban_card", "notification"},
"repositories": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "tender_submission", "inquiry", "contact", "cms", "notice", "kanban_board", "kanban_column", "kanban_card", "notification"},
})
// Initialize validation services
@@ -215,26 +222,64 @@ func main() {
_ = cms.NewValidationService() // Register mediaRef validator
// Initialize services with repositories
auditLogger := audit.NewLogger(logger)
esClient := bootstrap.InitElasticsearch(conf.Elasticsearch, logger)
defer func() {
if err := esClient.Close(); err != nil {
logger.Error("Failed to close Elasticsearch client", map[string]interface{}{
"error": err.Error(),
})
}
}()
auditLogger, auditStore := bootstrap.InitAuditLogger(logger, esClient)
defer func() {
if err := auditStore.Close(); err != nil {
logger.Error("Failed to close audit store", map[string]interface{}{
"error": err.Error(),
})
}
}()
userService := user.NewService(userRepository, logger, userAuthService, userValidator, notificationSDK, redisClient, auditLogger)
categoryService := company_category.NewService(categoryRepository, logger)
companyService := company.NewService(companyRepository, categoryService, fileStoreService, aiRecommendationClient, logger)
companyService := company.NewService(companyRepository, categoryService, fileStoreService, aiRecommendationClient, aiPipelineClient, redisClient, conf.AISummarizer.RecommendationCacheTTL, logger)
customerService := customer.NewService(customerRepository, logger, customerAuthService, companyService, redisClient, notificationSDK, customerValidator, auditLogger)
tenderService := tender.NewService(tenderRepository, companyService, customerService, logger, aiSummarizerClient, aiSummarizerStorage, conf.AISummarizer.DefaultLanguage)
tenderService := tender.NewService(
tenderRepository,
companyService,
customerService,
tenderApprovalRepository,
logger,
aiSummarizerClient,
aiSummarizerStorage,
redisClient,
conf.AISummarizer.RecommendationCacheTTL,
conf.AISummarizer.DefaultLanguage,
)
if configurer, ok := companyService.(interface {
SetRecommendedTendersPageCacheRefresher(company.RecommendedTendersPageCacheRefresher)
}); ok {
if refresher, ok := tenderService.(company.RecommendedTendersPageCacheRefresher); ok {
configurer.SetRecommendedTendersPageCacheRefresher(refresher)
}
}
feedbackService := feedback.NewService(feedbackRepo, tenderService, companyService, logger)
tenderApprovalService := tender_approval.NewService(tenderApprovalRepository, tenderService, logger)
tenderSubmissionService := tender_submission.NewService(tenderSubmissionRepository, tenderApprovalRepository, tenderService, logger)
tenderApprovalService := tender_approval.NewService(tenderApprovalRepository, tenderService, logger, tenderSubmissionService)
inquiryService := inquiry.NewService(inquiryRepository, notificationSDK, logger, xssPolicy)
flagService := assets.NewService(logger, conf.Assets.FlagsPath)
notificationService := notification.NewService(notificationSDK, notificationRepository, userService, customerService, logger)
contactService := contact.NewService(contactRepository, logger, notificationSDK)
cmsService := cms.NewService(cmsRepository, logger)
kanbanService := kanban.NewService(boardRepository, columnRepository, cardRepository, goRulesClient, conf.GoRules.RuleID, logger)
documentScraperService := document_scraper.NewService(tenderRepository, logger)
dashboardRepository := dashboard.NewRepository(mongoManager, logger)
dashboardService := dashboard.NewService(dashboardRepository, logger)
aiPipelineService := ai_pipeline.NewService(aiPipelineClient, logger, tenderService)
documentScraperService := document_scraper.NewService(tenderRepository, aiPipelineClient, logger)
dashboardRepository := dashboard.NewRepository(mongoManager, logger, procedureDocumentsLister)
dashboardService := dashboard.NewService(dashboardRepository, logger, redisClient)
aiPipelineService := ai_pipeline.NewService(aiPipelineClient, logger, tenderService, companyService)
auditLogRepository := auditlog.NewRepository(auditStore)
auditLogService := auditlog.NewService(auditLogRepository, logger)
logger.Info("Services initialized successfully", map[string]interface{}{
"services": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "inquiry", "flag", "notification", "contact", "cms", "scraper", "kanban", "document_scraper", "dashboard", "ai_pipeline"},
"services": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "tender_submission", "inquiry", "flag", "notification", "contact", "cms", "scraper", "kanban", "document_scraper", "dashboard", "ai_pipeline", "auditlog"},
})
// Initialize handlers with services
@@ -245,18 +290,20 @@ func main() {
tenderHandler := tender.NewHandler(tenderService, logger)
feedbackHandler := feedback.NewHandler(feedbackService, userHandler, customerHandler, logger)
tenderApprovalHandler := tender_approval.NewHandler(tenderApprovalService, logger)
tenderSubmissionHandler := tender_submission.NewHandler(tenderSubmissionService, logger)
inquiryHandler := inquiry.NewHandler(inquiryService, logger, hcaptchaVerifier, xssPolicy)
flagHandler := assets.NewHandler(flagService, logger)
notificationHandler := notification.NewHandler(notificationService)
contactHandler := contact.NewHandler(contactService, hcaptchaVerifier)
cmsHandler := cms.NewHandler(cmsService)
kanbanHandler := kanban.NewHandler(kanbanService)
fileStoreHandler := filestore.NewHandler(fileStoreService, logger)
fileStoreHandler := filestore.NewHandler(fileStoreService, logger, companyService)
documentScraperHandler := document_scraper.NewHandler(documentScraperService, logger)
dashboardHandler := dashboard.NewHandler(dashboardService)
aiPipelineHandler := ai_pipeline.NewHandler(aiPipelineService)
auditLogHandler := auditlog.NewHandler(auditLogService)
logger.Info("Handlers initialized successfully", map[string]interface{}{
"handlers": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "inquiry", "flag", "notification", "contact", "cms", "scraper", "kanban", "filestore", "document_scraper", "dashboard", "ai_pipeline"},
"handlers": []string{"customer", "user", "company", "category", "tender", "feedback", "tender_approval", "tender_submission", "inquiry", "flag", "notification", "contact", "cms", "scraper", "kanban", "filestore", "document_scraper", "dashboard", "ai_pipeline", "auditlog"},
})
// Initialize HTTP server
@@ -265,8 +312,8 @@ func main() {
router.SetupCSPSecurity(e)
// Register routes
router.RegisterAdminRoutes(e, userHandler, companyHandler, customerHandler, tenderHandler, feedbackHandler, tenderApprovalHandler, inquiryHandler, categoryHandler, flagHandler, notificationHandler, contactHandler, cmsHandler, kanbanHandler, fileStoreHandler, dashboardHandler, aiPipelineHandler, xssPolicy)
router.RegisterPublicRoutes(e, customerHandler, tenderHandler, feedbackHandler, tenderApprovalHandler, companyHandler, inquiryHandler, categoryHandler, flagHandler, notificationHandler, contactHandler, cmsHandler, kanbanHandler, fileStoreHandler, xssPolicy)
router.RegisterAdminRoutes(e, userHandler, companyHandler, customerHandler, tenderHandler, feedbackHandler, tenderApprovalHandler, tenderSubmissionHandler, inquiryHandler, categoryHandler, flagHandler, notificationHandler, contactHandler, cmsHandler, kanbanHandler, fileStoreHandler, dashboardHandler, aiPipelineHandler, auditLogHandler, xssPolicy)
router.RegisterPublicRoutes(e, customerHandler, tenderHandler, feedbackHandler, tenderApprovalHandler, tenderSubmissionHandler, companyHandler, inquiryHandler, categoryHandler, flagHandler, notificationHandler, contactHandler, cmsHandler, kanbanHandler, fileStoreHandler, xssPolicy)
router.RegisterDocumentScraperRoutes(e, documentScraperHandler, conf.DocumentScraper.APIKey)
// Start server
+39 -6
View File
@@ -3,6 +3,7 @@ package router
import (
"tm/internal/assets"
"tm/internal/ai_pipeline"
"tm/internal/auditlog"
"tm/internal/cms"
"tm/internal/company"
"tm/internal/company_category"
@@ -16,6 +17,7 @@ import (
"tm/internal/notification"
"tm/internal/tender"
"tm/internal/tender_approval"
"tm/internal/tender_submission"
"tm/internal/user"
"tm/pkg/filestore"
"tm/pkg/security"
@@ -35,7 +37,7 @@ func SetupCSPSecurity(e *echo.Echo) {
e.POST("/api/v1/csp-report", security.CSPReportHandler)
}
func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler *company.Handler, customerHandler *customer.Handler, tenderHandler *tender.TenderHandler, feedbackHandler *feedback.Handler, tenderApprovalHandler *tender_approval.Handler, inquiryHandler *inquiry.Handler, categoryHandler *company_category.Handler, flagHandler *assets.Handler, notificationHandler *notification.NotificationHandler, contactHandler *contact.Handler, cmsHandler *cms.Handler, kanbanHandler *kanban.Handler, fileStoreHandler *filestore.Handler, dashboardHandler *dashboard.Handler, aiPipelineHandler *ai_pipeline.Handler, xssPolicy *security.XSSPolicy) {
func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler *company.Handler, customerHandler *customer.Handler, tenderHandler *tender.TenderHandler, feedbackHandler *feedback.Handler, tenderApprovalHandler *tender_approval.Handler, tenderSubmissionHandler *tender_submission.Handler, inquiryHandler *inquiry.Handler, categoryHandler *company_category.Handler, flagHandler *assets.Handler, notificationHandler *notification.NotificationHandler, contactHandler *contact.Handler, cmsHandler *cms.Handler, kanbanHandler *kanban.Handler, fileStoreHandler *filestore.Handler, dashboardHandler *dashboard.Handler, aiPipelineHandler *ai_pipeline.Handler, auditLogHandler *auditlog.Handler, xssPolicy *security.XSSPolicy) {
adminV1 := e.Group("/admin/v1")
// Admin Users Routes
@@ -51,6 +53,13 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
adminUsersGP.POST("/:id/reset-password", userHandler.ResetUserPassword, userHandler.AdminMiddleware())
}
// Admin Audit Logs Routes
auditLogsGP := adminV1.Group("/audit-logs")
{
auditLogsGP.Use(userHandler.AuthMiddleware(), userHandler.AdminMiddleware())
auditLogsGP.GET("", auditLogHandler.Search)
}
// Admin user profile
profileGP := adminV1.Group("/profile")
{
@@ -73,12 +82,14 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
companiesGP.Use(userHandler.AuthMiddleware())
companiesGP.POST("", companyHandler.Create)
companiesGP.GET("", companyHandler.Search)
companiesGP.GET("/:id/recommended-tenders", tenderHandler.AdminRecommendTenders)
companiesGP.GET("/:id", companyHandler.Get)
companiesGP.PUT("/:id", companyHandler.Update)
companiesGP.DELETE("/:id", companyHandler.Delete)
companiesGP.PATCH("/:id/status", companyHandler.UpdateStatus)
companiesGP.PATCH("/:id/verification", companyHandler.UpdateVerification)
companiesGP.POST("/:id/documents", companyHandler.UploadDocuments)
companiesGP.POST("/:id/links", companyHandler.AddLinks)
}
// Admin Categories Routes
@@ -146,6 +157,15 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
tenderApprovalGP.GET("/status/:status", tenderApprovalHandler.GetTenderApprovalsByStatus)
}
// Admin Tender-Submissions Routes
tenderSubmissionGP := adminV1.Group("/tender-submissions")
{
tenderSubmissionGP.Use(userHandler.AuthMiddleware())
tenderSubmissionGP.GET("", tenderSubmissionHandler.AdminListSubmissions)
tenderSubmissionGP.GET("/stats", tenderSubmissionHandler.AdminGetSubmissionStats)
tenderSubmissionGP.GET("/:id", tenderSubmissionHandler.AdminGetSubmissionByID)
}
// Admin Inquiry Routes
inquiryGP := adminV1.Group("/inquiries")
{
@@ -233,6 +253,7 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
aiPipelineGP.Use(userHandler.AuthMiddleware())
aiPipelineGP.POST("/scrape/documents", aiPipelineHandler.ScrapeDocuments)
aiPipelineGP.POST("/scrape/documents/batch", aiPipelineHandler.ScrapeDocumentsBatch)
aiPipelineGP.GET("/scrape/portals", aiPipelineHandler.GetScrapePortals)
aiPipelineGP.POST("/summarize/batch", aiPipelineHandler.SummarizeBatch)
aiPipelineGP.POST("/translate/batch", aiPipelineHandler.TranslateBatch)
aiPipelineGP.POST("/sync", aiPipelineHandler.Sync)
@@ -249,7 +270,7 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
}
}
func RegisterPublicRoutes(e *echo.Echo, customerHandler *customer.Handler, tenderHandler *tender.TenderHandler, feedbackHandler *feedback.Handler, tenderApprovalHandler *tender_approval.Handler, companyHandler *company.Handler, inquiryHandler *inquiry.Handler, categoryHandler *company_category.Handler, flagHandler *assets.Handler, notificationHandler *notification.NotificationHandler, contactHandler *contact.Handler, cmsHandler *cms.Handler, kanbanHandler *kanban.Handler, fileStoreHandler *filestore.Handler, xssPolicy *security.XSSPolicy) {
func RegisterPublicRoutes(e *echo.Echo, customerHandler *customer.Handler, tenderHandler *tender.TenderHandler, feedbackHandler *feedback.Handler, tenderApprovalHandler *tender_approval.Handler, tenderSubmissionHandler *tender_submission.Handler, companyHandler *company.Handler, inquiryHandler *inquiry.Handler, categoryHandler *company_category.Handler, flagHandler *assets.Handler, notificationHandler *notification.NotificationHandler, contactHandler *contact.Handler, cmsHandler *cms.Handler, kanbanHandler *kanban.Handler, fileStoreHandler *filestore.Handler, xssPolicy *security.XSSPolicy) {
v1 := e.Group("/api/v1")
customerGP := v1.Group("/profile")
@@ -271,7 +292,7 @@ func RegisterPublicRoutes(e *echo.Echo, customerHandler *customer.Handler, tende
// Public Tenders Routes
tendersGP := v1.Group("/tenders")
tendersGP.Use(customerHandler.AuthMiddleware())
tendersGP.Use(customerHandler.AuthMiddleware(), customerHandler.CompanyContextMiddleware())
{
tendersGP.GET("", tenderHandler.GetPublicTenders)
tendersGP.GET("/recommend", tenderHandler.RecommendTenders)
@@ -298,7 +319,7 @@ func RegisterPublicRoutes(e *echo.Echo, customerHandler *customer.Handler, tende
// Public Tender Approvals Routes
tenderApprovalGP := v1.Group("/tender-approvals")
{
tenderApprovalGP.Use(customerHandler.AuthMiddleware())
tenderApprovalGP.Use(customerHandler.AuthMiddleware(), customerHandler.CompanyContextMiddleware())
tenderApprovalGP.POST("", tenderApprovalHandler.ToggleTenderApproval)
tenderApprovalGP.GET("", tenderApprovalHandler.PublicGetTenderApprovals)
tenderApprovalGP.GET("/:id", tenderApprovalHandler.GetPublicTenderApproval)
@@ -306,16 +327,28 @@ func RegisterPublicRoutes(e *echo.Echo, customerHandler *customer.Handler, tende
tenderApprovalGP.GET("/stats", tenderApprovalHandler.GetCompanyTenderApprovalStats)
}
// Public Tender Submissions Routes
tenderSubmissionGP := v1.Group("/tender-submissions")
{
tenderSubmissionGP.Use(customerHandler.AuthMiddleware(), customerHandler.CompanyContextMiddleware())
tenderSubmissionGP.GET("", tenderSubmissionHandler.ListCompanySubmissions)
tenderSubmissionGP.GET("/stats", tenderSubmissionHandler.GetCompanySubmissionStats)
tenderSubmissionGP.POST("/tender/:tender_id/ensure", tenderSubmissionHandler.EnsureSubmission)
tenderSubmissionGP.GET("/tender/:tender_id", tenderSubmissionHandler.GetSubmissionByTender)
tenderSubmissionGP.PATCH("/:id/status", tenderSubmissionHandler.UpdateSubmissionStatus)
tenderSubmissionGP.GET("/:id", tenderSubmissionHandler.GetSubmissionByID)
}
// Public Company Routes
companiesGP := v1.Group("/companies")
{
companiesGP.Use(customerHandler.AuthMiddleware())
companiesGP.Use(customerHandler.AuthMiddleware(), customerHandler.CompanyContextMiddleware())
companiesGP.GET("", companyHandler.MyCompany)
}
// AI tender recommendation routes
recommendationGP := v1.Group("")
recommendationGP.Use(customerHandler.AuthMiddleware())
recommendationGP.Use(customerHandler.AuthMiddleware(), customerHandler.CompanyContextMiddleware())
{
recommendationGP.POST("/onboarding", companyHandler.StartOnboarding)
recommendationGP.POST("/recommend", companyHandler.RecommendTenders)
+194 -17
View File
@@ -1,11 +1,17 @@
package bootstrap
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"tm/cmd/worker/workers"
"tm/internal/company"
"tm/internal/company_category"
"tm/internal/notice"
notificationDomain "tm/internal/notification"
"tm/internal/tender"
ai_summarizer "tm/pkg/ai_summarizer"
"tm/pkg/config"
@@ -13,9 +19,13 @@ import (
"tm/pkg/mongo"
"tm/pkg/notification"
"tm/pkg/ollama"
"tm/pkg/redis"
"tm/pkg/schedule"
)
// aiPipelineDailyRunMu ensures only one AI pipeline auto run executes at a time (startup catch-up and cron).
var aiPipelineDailyRunMu sync.Mutex
// Init Application Configuration
func InitConfig() (*Config, error) {
conf, err := config.LoadConfig(".", &Config{})
@@ -75,8 +85,8 @@ func InitMongoDB(conf config.MongoConfig, log logger.Logger) *mongo.ConnectionMa
return connectionManager
}
// Init Worker
func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger, notify notification.SDK, aiClient *ai_summarizer.Client, aiStorage *ai_summarizer.StorageClient) *schedule.CronScheduler {
// InitWorker
func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger, notify notification.SDK, aiClient *ai_summarizer.Client, aiStorage *ai_summarizer.StorageClient, redisClient redis.Client) *schedule.CronScheduler {
// Debug: Log worker config
appLogger.Info("Worker configuration", map[string]interface{}{
"worker_interval": config.Worker.Interval,
@@ -87,28 +97,38 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
"notice_fetch_error_backoff": config.Worker.NoticeFetchErrorBackoff.String(),
"translation_enabled": config.Worker.TranslationEnabled,
"translation_interval": config.Worker.TranslationInterval,
"notification_interval": config.Worker.NotificationInterval,
"ai_pipeline_auto_enabled": config.Worker.AIPipelineAutoEnabled,
"ai_pipeline_auto_interval": config.Worker.AIPipelineAutoInterval,
})
// Initialize repositories
noticeRepo := notice.NewRepository(mongoManager, appLogger)
tenderRepo := tender.NewRepository(mongoManager, appLogger)
notificationRepo := notificationDomain.NewRepository(mongoManager, appLogger)
var recommendationRefresher company.Service
if config.Worker.RecommendationRefreshAfterPipelineEnabled && aiClient != nil && redisClient != nil {
companyRepo := company.NewRepository(mongoManager, appLogger)
categoryRepo := company_category.NewRepository(mongoManager, appLogger)
categoryService := company_category.NewService(categoryRepo, appLogger)
recommendationRefresher = company.NewService(
companyRepo,
categoryService,
nil,
aiClient,
aiClient,
redisClient,
config.AISummarizer.RecommendationCacheTTL,
appLogger,
)
appLogger.Info("Company recommendation refresh after pipeline enabled", map[string]interface{}{})
} else if config.Worker.RecommendationRefreshAfterPipelineEnabled {
appLogger.Warn("Company recommendation refresh after pipeline disabled: AI client or Redis unavailable", map[string]interface{}{})
}
// Create a single shared cron scheduler for all recurring jobs
scheduler := schedule.NewCronScheduler(appLogger, mongoManager, noticeRepo)
// Document summarization via external AI service (requires AI_SUMMARIZER_API_BASE_URL)
if aiClient != nil {
scheduler.AddJob(schedule.Job{
Name: "Document Summarization Worker Job",
Func: func() {
worker := workers.NewDocumentSummarizationWorker(mongoManager, appLogger, tenderRepo, aiClient, aiStorage)
worker.Run()
},
Expr: "0 11 * * * *", // Run at 11 AM every day (after AI pipeline scrape)
})
} else {
appLogger.Warn("Document summarization worker not scheduled: AI summarizer API is not configured", map[string]interface{}{})
}
// Schedule Notice Worker job (no blocking run at startup; first tick runs on schedule)
workerInterval := config.Worker.Interval
if workerInterval == "" {
@@ -169,12 +189,140 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
"interval": config.Worker.TranslationInterval,
"batch_size": config.Worker.TranslationBatchSize,
})
go func() {
appLogger.Info("Running startup catch-up for tender translations", map[string]interface{}{
"target_languages": targetLanguages,
})
worker := workers.NewTranslationWorker(
mongoManager,
appLogger,
tenderRepo,
aiClient,
aiStorage,
translationSuccessCounter,
targetLanguages,
config.Worker.TranslationBatchSize,
)
worker.Run()
}()
} else {
appLogger.Warn("AI summarizer client not available, tender translation worker is disabled", map[string]interface{}{})
}
// Kick off one notice-processing pass without blocking startup (cron continues on schedule)
notificationInterval := config.Worker.NotificationInterval
if notificationInterval == "" {
notificationInterval = "0 * * * * *"
appLogger.Warn("WORKER_NOTIFICATION_INTERVAL not set, using default schedule", map[string]interface{}{
"interval": notificationInterval,
})
}
scheduler.AddJob(schedule.Job{
Name: "Scheduled Notification Worker Job",
Func: func() {
worker := workers.NewNotificationWorker(notificationRepo, appLogger)
worker.Run()
},
Expr: notificationInterval,
})
appLogger.Info("Scheduled notification delivery worker", map[string]interface{}{
"interval": notificationInterval,
})
if !config.Worker.AIPipelineAutoEnabled {
appLogger.Info("AI pipeline auto worker disabled by configuration", map[string]interface{}{
"ai_pipeline_auto_enabled": false,
})
} else if aiClient != nil {
dailyJobTracker := schedule.NewDailyJobTracker(mongoManager, appLogger)
runAIPipelineDaily := func(trigger string) {
aiPipelineDailyRunMu.Lock()
defer aiPipelineDailyRunMu.Unlock()
ctx := context.Background()
today := time.Now().Local()
completed, checkErr := dailyJobTracker.IsCompleted(ctx, schedule.AIPipelineDailyJobName, today)
if checkErr != nil {
appLogger.Error("Failed to check AI pipeline auto completion status", map[string]interface{}{
"trigger": trigger,
"error": checkErr.Error(),
"date": today.Format("02/01/2006"),
})
} else if completed {
appLogger.Info("AI pipeline auto skipped: already completed today", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
})
return
}
if trigger == "startup" {
appLogger.Info("Running startup catch-up for today's AI pipeline auto run", map[string]interface{}{
"date": today.Format("02/01/2006"),
})
} else {
appLogger.Info("Running scheduled AI pipeline auto run", map[string]interface{}{
"date": today.Format("02/01/2006"),
})
}
worker := workers.NewAIPipelineDailyWorker(appLogger, aiClient)
if err := worker.Run(); err != nil {
if errors.Is(err, ai_summarizer.ErrPipelineJobRunning) {
appLogger.Info("AI pipeline auto deferred: job already running", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
})
} else {
appLogger.Error("AI pipeline auto failed", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
"error": err.Error(),
})
}
return
}
if markErr := dailyJobTracker.MarkCompleted(ctx, schedule.AIPipelineDailyJobName, today); markErr != nil {
appLogger.Error("Failed to mark AI pipeline auto as completed", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
"error": markErr.Error(),
})
}
appLogger.Info("AI pipeline auto completed successfully", map[string]interface{}{
"trigger": trigger,
"date": today.Format("02/01/2006"),
})
if recommendationRefresher != nil {
recommendationRefresher.ScheduleRefreshCachedAIRecommendationsAfterPipeline()
}
}
scheduler.AddJob(schedule.Job{
Name: "AI Pipeline Auto Job",
Func: func() { runAIPipelineDaily("scheduled") },
Expr: config.Worker.AIPipelineAutoInterval,
})
appLogger.Info("Scheduled AI pipeline auto worker", map[string]interface{}{
"interval": config.Worker.AIPipelineAutoInterval,
})
go func() {
runAIPipelineDaily("startup")
}()
} else {
appLogger.Warn("AI summarizer client not available, AI pipeline auto worker is disabled", map[string]interface{}{})
}
// After a server restart, process any unprocessed notices (including today's) without blocking startup.
go func() {
appLogger.Info("Running startup catch-up for unprocessed notices", map[string]interface{}{})
w := workers.NewNoticeWorker(
mongoManager,
appLogger,
@@ -266,6 +414,35 @@ func InitAISummarizerStorage(conf AISummarizerConfig, log logger.Logger) *ai_sum
return storage
}
// InitRedis initializes the Redis client used by worker jobs.
func InitRedis(conf config.RedisConfig, log logger.Logger) redis.Client {
connectionConfig := &redis.Config{
Host: conf.Host,
Port: conf.Port,
Password: conf.Password,
DB: conf.DB,
PoolSize: conf.PoolSize,
}
redisClient, err := redis.NewClient(connectionConfig, log)
if err != nil {
log.Error("Failed to initialize Redis connection for worker", map[string]interface{}{
"error": err.Error(),
"host": conf.Host,
"port": conf.Port,
})
panic(fmt.Sprintf("Failed to initialize Redis connection: %v", err))
}
log.Info("Redis connection initialized for worker", map[string]interface{}{
"host": conf.Host,
"port": conf.Port,
"pool_size": conf.PoolSize,
})
return redisClient
}
func parseTranslationLanguages(conf AISummarizerConfig) []string {
raw := strings.TrimSpace(conf.TranslationLanguages)
if raw == "" {
+24 -13
View File
@@ -8,6 +8,7 @@ import (
// Config defines the worker application configuration
type Config struct {
Database config.DatabaseConfig
Redis config.RedisConfig
Logging config.LoggingConfig
Notification config.NotificationConfig
Ollama config.OllamaConfig
@@ -15,7 +16,7 @@ type Config struct {
Scraper config.ScraperConfig
MinIO config.MinIOConfig
AISummarizer AISummarizerConfig
AlertMail string `env:"ALERT_MAIL" envDefault:"alerts@opplens.com"`
AlertMail string `env:"ALERT_MAIL" envDefault:"alerts@opplens.com"`
}
type WorkerConfig struct {
@@ -33,20 +34,30 @@ type WorkerConfig struct {
// TranslationEnabled schedules the automatic batch translation cron job on the worker.
// On-demand translation (admin/public tender endpoints and AI pipeline routes on web) is unaffected.
TranslationEnabled bool `env:"WORKER_TRANSLATION_ENABLED" envDefault:"true"`
// NotificationInterval schedules promotion of due scheduled notifications from pending to sent.
NotificationInterval string `env:"WORKER_NOTIFICATION_INTERVAL" envDefault:"0 * * * * *"`
// AIPipelineAutoEnabled schedules the daily AI pipeline auto run on the worker.
// On-demand pipeline routes on web are unaffected.
AIPipelineAutoEnabled bool `env:"WORKER_AI_PIPELINE_AUTO_ENABLED" envDefault:"true"`
// AIPipelineAutoInterval runs pipeline auto once per day by default at 11:00 UTC (after TED scrape at 10:00).
AIPipelineAutoInterval string `env:"WORKER_AI_PIPELINE_AUTO_INTERVAL" envDefault:"0 0 11 * * *"`
// RecommendationRefreshAfterPipelineEnabled refreshes cached company recommendations after the daily AI pipeline auto run.
RecommendationRefreshAfterPipelineEnabled bool `env:"WORKER_RECOMMENDATION_REFRESH" envDefault:"true"`
}
// AISummarizerConfig holds configuration for the external AI summarizer service.
type AISummarizerConfig struct {
APIBaseURL string `env:"AI_SUMMARIZER_API_BASE_URL" envDefault:""`
APITimeout time.Duration `env:"AI_SUMMARIZER_API_TIMEOUT" envDefault:"120s"`
APIRetryCount int `env:"AI_SUMMARIZER_API_RETRY_COUNT" envDefault:"2"`
APIRetryDelay time.Duration `env:"AI_SUMMARIZER_API_RETRY_DELAY" envDefault:"3s"`
DefaultLanguage string `env:"AI_SUMMARIZER_DEFAULT_LANGUAGE" envDefault:"en"`
TranslationLanguages string `env:"AI_SUMMARIZER_TRANSLATION_LANGUAGES" envDefault:"en"`
MinioEndpoint string `env:"AI_SUMMARIZER_MINIO_ENDPOINT" envDefault:""`
MinioAccessKey string `env:"AI_SUMMARIZER_MINIO_ACCESS_KEY" envDefault:""`
MinioSecretKey string `env:"AI_SUMMARIZER_MINIO_SECRET_KEY" envDefault:""`
MinioUseSSL bool `env:"AI_SUMMARIZER_MINIO_USE_SSL" envDefault:"false"`
MinioRegion string `env:"AI_SUMMARIZER_MINIO_REGION" envDefault:"us-east-1"`
MinioBucket string `env:"AI_SUMMARIZER_MINIO_BUCKET" envDefault:"opplens"`
APIBaseURL string `env:"AI_SUMMARIZER_API_BASE_URL" envDefault:""`
APITimeout time.Duration `env:"AI_SUMMARIZER_API_TIMEOUT" envDefault:"120s"`
APIRetryCount int `env:"AI_SUMMARIZER_API_RETRY_COUNT" envDefault:"2"`
APIRetryDelay time.Duration `env:"AI_SUMMARIZER_API_RETRY_DELAY" envDefault:"3s"`
DefaultLanguage string `env:"AI_SUMMARIZER_DEFAULT_LANGUAGE" envDefault:"en"`
TranslationLanguages string `env:"AI_SUMMARIZER_TRANSLATION_LANGUAGES" envDefault:"en"`
MinioEndpoint string `env:"AI_SUMMARIZER_MINIO_ENDPOINT" envDefault:""`
MinioAccessKey string `env:"AI_SUMMARIZER_MINIO_ACCESS_KEY" envDefault:""`
MinioSecretKey string `env:"AI_SUMMARIZER_MINIO_SECRET_KEY" envDefault:""`
MinioUseSSL bool `env:"AI_SUMMARIZER_MINIO_USE_SSL" envDefault:"false"`
MinioRegion string `env:"AI_SUMMARIZER_MINIO_REGION" envDefault:"us-east-1"`
MinioBucket string `env:"AI_SUMMARIZER_MINIO_BUCKET" envDefault:"opplens"`
RecommendationCacheTTL time.Duration `env:"AI_RECOMMENDATION_CACHE_TTL" envDefault:"0"`
}
+12 -2
View File
@@ -37,12 +37,22 @@ func main() {
// Initialize notification service
notificationService := bootstrap.InitNotificationService(config.Notification, appLogger)
// Initialize AI summarizer client (translation + document summarization workers)
// Initialize AI summarizer client (translation worker)
aiSummarizerClient := bootstrap.InitAISummarizerClient(config.AISummarizer, mongoManager, appLogger)
aiSummarizerStorage := bootstrap.InitAISummarizerStorage(config.AISummarizer, appLogger)
redisClient := bootstrap.InitRedis(config.Redis, appLogger)
defer func() {
if redisClient != nil {
if err := redisClient.Close(); err != nil {
appLogger.Error("Failed to close Redis connection", map[string]interface{}{
"error": err.Error(),
})
}
}
}()
// Initialize Worker
scheduler := bootstrap.InitWorker(*config, mongoManager, appLogger, notificationService, aiSummarizerClient, aiSummarizerStorage)
scheduler := bootstrap.InitWorker(*config, mongoManager, appLogger, notificationService, aiSummarizerClient, aiSummarizerStorage, redisClient)
// Set up signal handling for graceful shutdown
signalChan := make(chan os.Signal, 1)
+52
View File
@@ -0,0 +1,52 @@
package workers
import (
"context"
"time"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
)
// AIPipelineDailyWorker triggers the scheduled Opplens AI pipeline auto run.
type AIPipelineDailyWorker struct {
Logger logger.Logger
AIClient *ai_summarizer.Client
}
// NewAIPipelineDailyWorker creates a worker that calls POST /pipeline/auto on the AI service.
func NewAIPipelineDailyWorker(log logger.Logger, aiClient *ai_summarizer.Client) *AIPipelineDailyWorker {
return &AIPipelineDailyWorker{
Logger: log,
AIClient: aiClient,
}
}
// Run starts the pipeline auto job. A 409 response means a run is already in progress and is returned as an error
// so the daily job tracker does not mark today as completed before the run actually starts.
func (w *AIPipelineDailyWorker) Run() error {
if w.AIClient == nil {
w.Logger.Warn("AI pipeline auto worker skipped: AI client is not configured", map[string]interface{}{})
return nil
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
w.Logger.Info("AI pipeline auto worker started", map[string]interface{}{})
resp, err := w.AIClient.PipelineAuto(ctx)
if err != nil {
w.Logger.Error("AI pipeline auto worker failed", map[string]interface{}{
"error": err.Error(),
})
return err
}
w.Logger.Info("AI pipeline auto worker triggered successfully", map[string]interface{}{
"status": resp.Status,
"report": resp.Report,
})
return nil
}
+19 -1
View File
@@ -476,6 +476,16 @@ func (w *NoticeWorker) ToTender(n *notice.Notice) (*tender.Tender, error) {
awardedEntities = mergeAwardedEntitiesByLotID(t.AwardedEntities, awardedEntities)
title, description = mergeTitleAndDescriptionForMultiLot(t.Title, t.Description, title, description)
if estimatedVal == 0 && t.EstimatedValue > 0 {
estimatedVal = t.EstimatedValue
}
if awardedVal == 0 && t.AwardedValue > 0 {
awardedVal = t.AwardedValue
}
if currency == "" && t.Currency != "" {
currency = t.Currency
}
if strings.TrimSpace(previousNoticePublicationID) != "" && previousNoticePublicationID != n.NoticePublicationID {
if n.EstimatedValue > 0 {
estimatedVal = t.EstimatedValue + n.EstimatedValue
@@ -495,6 +505,15 @@ func (w *NoticeWorker) ToTender(n *notice.Notice) (*tender.Tender, error) {
}
}
if estimatedVal == 0 {
if lotValue, lotCurrency := tender.AggregateProcurementLotEstimatedValue(procurementLots); lotValue > 0 {
estimatedVal = lotValue
if currency == "" {
currency = lotCurrency
}
}
}
// Title and description stay in the notice language; English translation is handled by the AI service (translation worker).
t.Title = title
t.Description = description
@@ -546,7 +565,6 @@ func (w *NoticeWorker) ToTender(n *notice.Notice) (*tender.Tender, error) {
t.ProcurementProjectID = n.ProcurementProjectID
t.SourceFileURL = n.SourceFileURL
t.SourceFileName = n.SourceFileName
t.ContentXML = n.ContentXML
// Preserve document-scraping/summarization flags on the tender (managed by
// downstream workers), but refresh the notice-side processing metadata.
t.ProcessingMetadata.ScrapedAt = n.ProcessingMetadata.ScrapedAt
+40
View File
@@ -0,0 +1,40 @@
package workers
import (
"context"
"time"
"tm/internal/notification"
"tm/pkg/logger"
)
// NotificationWorker promotes due scheduled notifications from pending to sent.
type NotificationWorker struct {
Repository notification.Repository
Logger logger.Logger
}
// NewNotificationWorker creates a notification delivery worker.
func NewNotificationWorker(repository notification.Repository, logger logger.Logger) *NotificationWorker {
return &NotificationWorker{
Repository: repository,
Logger: logger,
}
}
// Run marks pending scheduled notifications as sent once their delivery time has passed.
func (w *NotificationWorker) Run() {
count, err := w.Repository.MarkDueScheduledAsSent(context.Background(), time.Now().Unix())
if err != nil {
w.Logger.Error("Scheduled notification worker failed", map[string]interface{}{
"error": err.Error(),
})
return
}
if count > 0 {
w.Logger.Info("Scheduled notification worker completed", map[string]interface{}{
"count": count,
})
}
}
-208
View File
@@ -1,208 +0,0 @@
package workers
import (
"context"
"strings"
"time"
"tm/internal/tender"
ai_summarizer "tm/pkg/ai_summarizer"
"tm/pkg/logger"
"tm/pkg/mongo"
)
// DocumentSummarizationWorker triggers AI summarization for scraped tenders.
// The AI service owns MinIO artefacts; this worker marks Mongo processing flags
// after the pipeline or on-demand summarize API completes.
type DocumentSummarizationWorker struct {
Mongo *mongo.ConnectionManager
Logger logger.Logger
TenderRepo tender.TenderRepository
AIClient *ai_summarizer.Client
AIStorage *ai_summarizer.StorageClient
}
// NewDocumentSummarizationWorker creates a document summarization worker.
func NewDocumentSummarizationWorker(
mongo *mongo.ConnectionManager,
logger logger.Logger,
tenderRepo tender.TenderRepository,
aiClient *ai_summarizer.Client,
aiStorage *ai_summarizer.StorageClient,
) *DocumentSummarizationWorker {
return &DocumentSummarizationWorker{
Mongo: mongo,
Logger: logger,
TenderRepo: tenderRepo,
AIClient: aiClient,
AIStorage: aiStorage,
}
}
// Run starts the document summarization process.
func (w *DocumentSummarizationWorker) Run() {
w.Logger.Info("Document summarization worker started", map[string]interface{}{})
if w.AIClient == nil {
w.Logger.Warn("AI summarizer client not configured, skipping document summarization run", map[string]interface{}{})
return
}
if refs := w.collectUnSummarizedTenderRefs(context.Background()); len(refs) > 0 {
if _, err := w.AIClient.TriggerSummarizeBatch(context.Background(), refs); err != nil {
w.Logger.Warn("Failed to trigger AI summarize batch (continuing with per-tender sync)", map[string]interface{}{
"tender_count": len(refs),
"error": err.Error(),
})
}
}
limit := 5
skip := 0
for {
tenders, totalCount, err := w.TenderRepo.GetUnSummarizedTenders(context.Background(), limit, skip)
if err != nil {
w.Logger.Error("Failed to get un-summarized tenders", map[string]interface{}{
"error": err.Error(),
})
return
}
w.Logger.Info("Found tenders for document summarization", map[string]interface{}{
"count": len(tenders),
"total_count": totalCount,
"skip": skip,
})
if len(tenders) == 0 {
break
}
for i := range tenders {
t := &tenders[i]
w.Logger.Info("Processing tender for document summarization", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
})
w.summarizeTender(t)
}
skip += len(tenders)
if int64(skip) >= totalCount {
break
}
}
w.Logger.Info("Document summarization worker completed", map[string]interface{}{})
}
func (w *DocumentSummarizationWorker) collectUnSummarizedTenderRefs(ctx context.Context) []ai_summarizer.TenderRef {
limit := 100
skip := 0
refs := make([]ai_summarizer.TenderRef, 0)
for {
tenders, totalCount, err := w.TenderRepo.GetUnSummarizedTenders(ctx, limit, skip)
if err != nil {
w.Logger.Warn("Failed to collect tenders for summarize batch", map[string]interface{}{
"error": err.Error(),
})
return refs
}
if len(tenders) == 0 {
return refs
}
for i := range tenders {
t := &tenders[i]
if strings.TrimSpace(t.ContractFolderID) == "" || strings.TrimSpace(t.NoticePublicationID) == "" {
continue
}
refs = append(refs, ai_summarizer.NewTenderRef(t.ContractFolderID, t.NoticePublicationID))
}
skip += len(tenders)
if int64(skip) >= totalCount {
return refs
}
}
}
func (w *DocumentSummarizationWorker) summarizeTender(t *tender.Tender) {
if strings.TrimSpace(t.NoticePublicationID) == "" {
w.Logger.Warn("Skipping tender summarization: missing notice_publication_id", map[string]interface{}{
"tender_id": t.ID.Hex(),
})
w.markTenderSummarized(t)
return
}
if strings.TrimSpace(t.ContractFolderID) == "" {
w.Logger.Warn("Skipping tender summarization: missing contract_folder_id", map[string]interface{}{
"tender_id": t.ID.Hex(),
})
w.markTenderSummarized(t)
return
}
ctx := context.Background()
if w.isSummaryReadyInStorage(ctx, t) {
w.markTenderSummarized(t)
return
}
req := ai_summarizer.NewSummarizeRequest(t.ContractFolderID, t.NoticePublicationID)
if _, err := w.AIClient.FetchSummaryOnDemand(ctx, req); err != nil {
w.Logger.Error("Failed to summarize tender via AI service", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
"contract_folder_id": t.ContractFolderID,
"error": err.Error(),
})
return
}
if !w.isSummaryReadyInStorage(ctx, t) {
w.Logger.Warn("AI summarize completed but storage is not ready yet", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
})
}
w.markTenderSummarized(t)
}
func (w *DocumentSummarizationWorker) isSummaryReadyInStorage(ctx context.Context, t *tender.Tender) bool {
if w.AIStorage == nil {
return false
}
ready, err := w.AIStorage.IsSummaryReady(ctx, t.ContractFolderID, t.NoticePublicationID)
if err != nil {
w.Logger.Debug("Could not check summarization status in storage", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
"error": err.Error(),
})
return false
}
return ready
}
func (w *DocumentSummarizationWorker) markTenderSummarized(t *tender.Tender) {
now := time.Now().Unix()
t.ProcessingMetadata.DocumentsSummarized = true
t.ProcessingMetadata.DocumentsSummarizedAt = now
t.UpdatedAt = now
if err := w.TenderRepo.Update(context.Background(), t); err != nil {
w.Logger.Error("Failed to update tender summarization status", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
"error": err.Error(),
})
return
}
w.Logger.Info("Tender marked as summarized", map[string]interface{}{
"tender_id": t.ID.Hex(),
"notice_publication_id": t.NoticePublicationID,
})
}
+7
View File
@@ -5,6 +5,7 @@ go 1.24
toolchain go1.24.4
require (
github.com/elastic/go-elasticsearch/v8 v8.17.1
github.com/golang-jwt/jwt/v5 v5.3.0
github.com/labstack/echo/v4 v4.13.4
github.com/microcosm-cc/bluemonday v1.0.27
@@ -28,8 +29,11 @@ require (
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/elastic/elastic-transport-go/v8 v8.6.1 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/jsonpointer v0.21.1 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/spec v0.21.0 // indirect
@@ -48,6 +52,9 @@ require (
github.com/stretchr/objx v0.5.2 // indirect
github.com/swaggo/files/v2 v2.0.2 // indirect
github.com/tinylib/msgp v1.3.0 // indirect
go.opentelemetry.io/otel v1.28.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
golang.org/x/mod v0.26.0 // indirect
golang.org/x/time v0.11.0 // indirect
golang.org/x/tools v0.35.0 // indirect
+17
View File
@@ -16,10 +16,19 @@ github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/r
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/elastic/elastic-transport-go/v8 v8.6.1 h1:h2jQRqH6eLGiBSN4eZbQnJLtL4bC5b4lfVFRjw2R4e4=
github.com/elastic/elastic-transport-go/v8 v8.6.1/go.mod h1:YLHer5cj0csTzNFXoNQ8qhtGY1GTvSqPnKWKaqQE3Hk=
github.com/elastic/go-elasticsearch/v8 v8.17.1 h1:bOXChDoCMB4TIwwGqKd031U8OXssmWLT3UrAr9EGs3Q=
github.com/elastic/go-elasticsearch/v8 v8.17.1/go.mod h1:MVJCtL+gJJ7x5jFeUmA20O7rvipX8GcQmo5iBcmaJn4=
github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-openapi/jsonpointer v0.21.1 h1:whnzv/pNXtK2FbX/W9yJfRmE2gsmkfahjMKB0fZvcic=
github.com/go-openapi/jsonpointer v0.21.1/go.mod h1:50I1STOfbY1ycR8jGz8DaMeLCdXiI6aDteEdRNNzpdk=
github.com/go-openapi/jsonreference v0.21.0 h1:Rs+Y7hSXT83Jacb7kFyjn4ijOuVGSvOdF2+tg1TRrwQ=
@@ -108,6 +117,14 @@ github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78/go.mod h1:aL8wCCfTfS
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.mongodb.org/mongo-driver/v2 v2.5.1 h1:j2U/Qp+wvueSpqitLCSZPT/+ZpVc1xzuwdHWwl7d8ro=
go.mongodb.org/mongo-driver/v2 v2.5.1/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
go.opentelemetry.io/otel v1.28.0 h1:/SqNcYk+idO0CxKEUOtKQClMK/MimZihKYMruSMViUo=
go.opentelemetry.io/otel v1.28.0/go.mod h1:q68ijF8Fc8CnMHKyzqL6akLO46ePnjkgfIMIjUIX9z4=
go.opentelemetry.io/otel/metric v1.28.0 h1:f0HGvSl1KRAU1DLgLGFjrwVyismPlnuU6JD6bOeuA5Q=
go.opentelemetry.io/otel/metric v1.28.0/go.mod h1:Fb1eVBFZmLVTMb6PPohq3TO9IIhUisDsbJoL/+uQW4s=
go.opentelemetry.io/otel/sdk v1.21.0 h1:FTt8qirL1EysG6sTQRZ5TokkU8d0ugCj8htOgThZXQ8=
go.opentelemetry.io/otel/sdk v1.21.0/go.mod h1:Nna6Yv7PWTdgJHVRD9hIYywQBRx7pbox6nwBnZIxl/E=
go.opentelemetry.io/otel/trace v1.28.0 h1:GhQ9cUuQGmNDd5BTCP2dAvv75RdMxEfTmYejp+lkx9g=
go.opentelemetry.io/otel/trace v1.28.0/go.mod h1:jPyXzNPg6da9+38HEwElrQiHlVMTnVfM3/yv2OlIHaI=
go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk=
go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo=
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
+20
View File
@@ -78,6 +78,26 @@ func (h *Handler) ScrapeDocumentsBatch(c echo.Context) error {
return response.Accepted(c, result, "Scrape batch accepted")
}
// GetScrapePortals lists document scraping portals supported by the Opplens AI service.
// @Summary List document scraping portals
// @Description Retrieve the list of document scraping portals supported by the Opplens AI service
// @Tags Admin-AI-Pipeline
// @Produce json
// @Success 200 {object} response.APIResponse{data=[]string} "Scrape portals retrieved successfully"
// @Failure 401 {object} response.APIResponse
// @Failure 500 {object} response.APIResponse
// @Failure 503 {object} response.APIResponse
// @Security BearerAuth
// @Router /admin/v1/ai-pipeline/scrape/portals [get]
func (h *Handler) GetScrapePortals(c echo.Context) error {
result, err := h.service.GetScrapePortals(c.Request().Context())
if err != nil {
return mapPipelineHTTPError(c, err, "Get scrape portals")
}
return response.Success(c, result, "Scrape portals retrieved successfully")
}
// SummarizeBatch enqueues background summarization for many tenders.
// @Summary Batch summarize tenders
// @Description Enqueue background AI summarization for multiple tenders via the Opplens AI service
+52 -7
View File
@@ -12,6 +12,7 @@ import (
type Client interface {
ScrapeDocuments(ctx context.Context, reqBody ai_summarizer.ScrapeDocumentsRequest) (*ai_summarizer.ScrapeDocumentsResponse, error)
ScrapeDocumentsBatch(ctx context.Context, tenders []ai_summarizer.TenderRef) (*ai_summarizer.BatchAcceptedResponse, error)
GetScrapePortals(ctx context.Context) (*ai_summarizer.ScrapePortalsResponse, error)
TriggerSummarizeBatch(ctx context.Context, tenders []ai_summarizer.TenderRef) (*ai_summarizer.BatchAcceptedResponse, error)
TriggerTranslateBatch(ctx context.Context, tenders []ai_summarizer.TenderRef, languages []string) (*ai_summarizer.BatchAcceptedResponse, error)
PipelineSync(ctx context.Context) (*ai_summarizer.PipelineSyncResponse, error)
@@ -31,6 +32,7 @@ type Client interface {
type Service interface {
ScrapeDocuments(ctx context.Context, form ScrapeDocumentsForm) (*ai_summarizer.ScrapeDocumentsResponse, error)
ScrapeDocumentsBatch(ctx context.Context, form TenderBatchForm) (*ai_summarizer.BatchAcceptedResponse, error)
GetScrapePortals(ctx context.Context) (*ai_summarizer.ScrapePortalsResponse, error)
SummarizeBatch(ctx context.Context, form TenderBatchForm) (*ai_summarizer.BatchAcceptedResponse, error)
TranslateBatch(ctx context.Context, form TranslateBatchForm) (*ai_summarizer.BatchAcceptedResponse, error)
Sync(ctx context.Context) (*ai_summarizer.PipelineSyncResponse, error)
@@ -46,23 +48,35 @@ type Service interface {
GetMinioStats(ctx context.Context) (ai_summarizer.PipelineMinioStatsResponse, error)
}
// CachedRecommendationRefresher schedules a background refresh of company recommendation caches.
type CachedRecommendationRefresher interface {
ScheduleRefreshCachedAIRecommendationsAfterPipeline()
}
// ScrapedDocumentMetadataSyncer persists MinIO scraped-document metadata onto tender records.
type ScrapedDocumentMetadataSyncer interface {
SyncScrapedDocumentsFromStorage(ctx context.Context, contractFolderID, noticePublicationID string) error
}
type service struct {
client Client
logger logger.Logger
metadataSyncer ScrapedDocumentMetadataSyncer
client Client
logger logger.Logger
metadataSyncer ScrapedDocumentMetadataSyncer
recommendationRefresher CachedRecommendationRefresher
}
// NewService creates an AI pipeline admin service.
func NewService(client Client, log logger.Logger, metadataSyncer ScrapedDocumentMetadataSyncer) Service {
func NewService(
client Client,
log logger.Logger,
metadataSyncer ScrapedDocumentMetadataSyncer,
recommendationRefresher CachedRecommendationRefresher,
) Service {
return &service{
client: client,
logger: log,
metadataSyncer: metadataSyncer,
client: client,
logger: log,
metadataSyncer: metadataSyncer,
recommendationRefresher: recommendationRefresher,
}
}
@@ -128,6 +142,24 @@ func (s *service) ScrapeDocumentsBatch(ctx context.Context, form TenderBatchForm
return resp, nil
}
func (s *service) GetScrapePortals(ctx context.Context) (*ai_summarizer.ScrapePortalsResponse, error) {
if err := s.requireClient(); err != nil {
return nil, err
}
s.logger.Info("Admin get scrape portals requested", map[string]interface{}{})
resp, err := s.client.GetScrapePortals(ctx)
if err != nil {
s.logger.Error("Admin get scrape portals failed", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("get scrape portals failed: %w", err)
}
return resp, nil
}
func (s *service) SummarizeBatch(ctx context.Context, form TenderBatchForm) (*ai_summarizer.BatchAcceptedResponse, error) {
if err := s.requireClient(); err != nil {
return nil, err
@@ -199,6 +231,8 @@ func (s *service) Sync(ctx context.Context) (*ai_summarizer.PipelineSyncResponse
return nil, fmt.Errorf("pipeline sync failed: %w", err)
}
s.scheduleRecommendationRefreshAfterPipeline()
return resp, nil
}
@@ -302,6 +336,8 @@ func (s *service) DailyRun(ctx context.Context) (*ai_summarizer.PipelineStartedR
return nil, fmt.Errorf("pipeline daily-run failed: %w", err)
}
s.scheduleRecommendationRefreshAfterPipeline()
return resp, nil
}
@@ -320,6 +356,8 @@ func (s *service) Auto(ctx context.Context) (*ai_summarizer.PipelineStartedRespo
return nil, fmt.Errorf("pipeline auto failed: %w", err)
}
s.scheduleRecommendationRefreshAfterPipeline()
return resp, nil
}
@@ -405,6 +443,13 @@ func (s *service) GetMinioStats(ctx context.Context) (ai_summarizer.PipelineMini
return resp, nil
}
func (s *service) scheduleRecommendationRefreshAfterPipeline() {
if s.recommendationRefresher == nil {
return
}
s.recommendationRefresher.ScheduleRefreshCachedAIRecommendationsAfterPipeline()
}
func validateTenderBatch(forms []TenderRefForm) ([]ai_summarizer.TenderRef, error) {
if len(forms) == 0 {
return nil, ErrEmptyTenderBatch
+31
View File
@@ -0,0 +1,31 @@
package auditlog
// Entry represents an audit log record returned by the API.
type Entry struct {
ActorID string `json:"actor_id"`
ActorType string `json:"actor_type"`
Action string `json:"action"`
TargetType string `json:"target_type"`
TargetID string `json:"target_id"`
At int64 `json:"at"`
IP string `json:"ip,omitempty"`
RequestID string `json:"request_id,omitempty"`
UserAgent string `json:"user_agent,omitempty"`
Success bool `json:"success"`
Metadata map[string]string `json:"metadata,omitempty"`
}
// ListResponse is the paginated audit log list response.
type ListResponse struct {
Logs []*Entry `json:"logs"`
Meta *ListMeta `json:"meta,omitempty"`
}
// ListMeta contains pagination metadata.
type ListMeta struct {
Total int64 `json:"total"`
Limit int `json:"limit"`
Offset int `json:"offset"`
Page int `json:"page"`
Pages int `json:"pages"`
}
+12
View File
@@ -0,0 +1,12 @@
package auditlog
// SearchForm defines query parameters for audit log search.
type SearchForm struct {
ActorID string `query:"actor_id" valid:"optional"`
ActorType string `query:"actor_type" valid:"optional,in(admin|customer|system)"`
Action string `query:"action" valid:"optional"`
TargetType string `query:"target_type" valid:"optional"`
TargetID string `query:"target_id" valid:"optional"`
From int64 `query:"from" valid:"optional"`
To int64 `query:"to" valid:"optional"`
}
+69
View File
@@ -0,0 +1,69 @@
package auditlog
import (
"tm/pkg/response"
"github.com/asaskevich/govalidator"
"github.com/labstack/echo/v4"
)
// Handler handles HTTP requests for audit log operations.
type Handler struct {
service Service
}
// NewHandler creates an audit log handler.
func NewHandler(service Service) *Handler {
return &Handler{service: service}
}
// Search returns paginated audit logs with optional filters.
// @Summary Search audit logs
// @Description Retrieve paginated user action audit logs with optional filtering by actor, action, target, and time range
// @Tags Admin-AuditLogs
// @Accept json
// @Produce json
// @Security BearerAuth
// @Param actor_id query string false "Filter by actor ID"
// @Param actor_type query string false "Filter by actor type" Enums(admin,customer,system)
// @Param action query string false "Filter by action (e.g. auth.login.success)"
// @Param target_type query string false "Filter by target type"
// @Param target_id query string false "Filter by target ID"
// @Param from query int64 false "Filter by timestamp from (Unix seconds)"
// @Param to query int64 false "Filter by timestamp to (Unix seconds)"
// @Param limit query int false "Number of items per page (default: 10, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0)"
// @Success 200 {object} response.APIResponse{data=ListResponse} "Audit logs retrieved successfully"
// @Failure 400 {object} response.APIResponse "Bad request"
// @Failure 401 {object} response.APIResponse "Unauthorized"
// @Failure 403 {object} response.APIResponse "Forbidden"
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Router /admin/v1/audit-logs [get]
func (h *Handler) Search(c echo.Context) error {
form, err := response.Parse[SearchForm](c)
if err != nil {
return response.ValidationError(c, "Invalid request data", err.Error())
}
if _, err := govalidator.ValidateStruct(form); err != nil {
return response.ValidationError(c, "Validation failed", err.Error())
}
pagination, err := response.NewPagination(c)
if err != nil {
return response.PaginationBadRequest(c, err)
}
result, err := h.service.Search(c.Request().Context(), *form, pagination)
if err != nil {
return response.InternalServerError(c, "Failed to retrieve audit logs")
}
return response.SuccessWithMeta(c, result, &response.Meta{
Total: int(result.Meta.Total),
Limit: result.Meta.Limit,
Offset: result.Meta.Offset,
Page: result.Meta.Page,
Pages: result.Meta.Pages,
}, "audit logs retrieved successfully")
}
+25
View File
@@ -0,0 +1,25 @@
package auditlog
import (
"context"
"tm/pkg/audit"
)
// Repository defines audit log data access.
type Repository interface {
Search(ctx context.Context, filter audit.SearchFilter) (*audit.SearchResult, error)
}
type repository struct {
store audit.Store
}
// NewRepository creates an audit log repository.
func NewRepository(store audit.Store) Repository {
return &repository{store: store}
}
func (r *repository) Search(ctx context.Context, filter audit.SearchFilter) (*audit.SearchResult, error) {
return r.store.Search(ctx, filter)
}
+88
View File
@@ -0,0 +1,88 @@
package auditlog
import (
"context"
"math"
"tm/pkg/audit"
"tm/pkg/logger"
"tm/pkg/response"
)
// Service defines audit log query operations.
type Service interface {
Search(ctx context.Context, form SearchForm, pagination *response.Pagination) (*ListResponse, error)
}
type service struct {
repository Repository
logger logger.Logger
}
// NewService creates an audit log service.
func NewService(repository Repository, log logger.Logger) Service {
return &service{
repository: repository,
logger: log,
}
}
func (s *service) Search(ctx context.Context, form SearchForm, pagination *response.Pagination) (*ListResponse, error) {
filter := audit.SearchFilter{
ActorID: form.ActorID,
ActorType: form.ActorType,
Action: form.Action,
TargetType: form.TargetType,
TargetID: form.TargetID,
From: form.From,
To: form.To,
Limit: pagination.Limit,
Offset: pagination.Offset,
}
result, err := s.repository.Search(ctx, filter)
if err != nil {
s.logger.Error("Failed to search audit logs", map[string]interface{}{
"error": err.Error(),
})
return nil, err
}
entries := make([]*Entry, 0, len(result.Items))
for _, item := range result.Items {
entries = append(entries, &Entry{
ActorID: item.ActorID,
ActorType: item.ActorType,
Action: item.Action,
TargetType: item.TargetType,
TargetID: item.TargetID,
At: item.At,
IP: item.IP,
RequestID: item.RequestID,
UserAgent: item.UserAgent,
Success: item.Success,
Metadata: item.Metadata,
})
}
total := result.TotalCount
pages := 0
if pagination.Limit > 0 {
pages = int(math.Ceil(float64(total) / float64(pagination.Limit)))
}
page := 1
if pagination.Limit > 0 {
page = (pagination.Offset / pagination.Limit) + 1
}
return &ListResponse{
Logs: entries,
Meta: &ListMeta{
Total: total,
Limit: pagination.Limit,
Offset: pagination.Offset,
Page: page,
Pages: pages,
},
}, nil
}
+41 -1
View File
@@ -75,12 +75,13 @@ func (s *companyService) UploadDocuments(ctx context.Context, companyID, uploade
"file_count": len(files),
})
return &UploadDocumentsResponse{
DocumentFileIDs: company.DocumentFileIDs,
DocumentFileIDs: s.sanitizeDocumentFileIDs(company.DocumentFileIDs),
Uploaded: uploaded,
Failed: failed,
}, errors.New("all file uploads failed")
}
company.DocumentFileIDs = s.sanitizeDocumentFileIDs(company.DocumentFileIDs)
company.DocumentFileIDs = mergeDocumentFileIDs(company.DocumentFileIDs, newIDs)
if err := s.repository.Update(ctx, company); err != nil {
s.logger.Error("Failed to save company documents", map[string]interface{}{
@@ -97,6 +98,8 @@ func (s *companyService) UploadDocuments(ctx context.Context, companyID, uploade
"documents_total": len(company.DocumentFileIDs),
})
s.triggerAIOnboardingAsync(companyID)
return &UploadDocumentsResponse{
DocumentFileIDs: company.DocumentFileIDs,
Uploaded: uploaded,
@@ -125,3 +128,40 @@ func mergeDocumentFileIDs(existing, additional []string) []string {
appendUnique(additional)
return merged
}
// DetachDocumentFileID removes a file ID from every company that still references it.
func (s *companyService) DetachDocumentFileID(ctx context.Context, fileID string) error {
if err := s.repository.RemoveDocumentFileID(ctx, fileID); err != nil {
s.logger.Error("Failed to detach document file ID from companies", map[string]interface{}{
"error": err.Error(),
"file_id": fileID,
})
return err
}
return nil
}
func (s *companyService) sanitizeDocumentFileIDs(ids []string) []string {
if len(ids) == 0 || s.fileStore == nil {
return ids
}
sanitized := make([]string, 0, len(ids))
for _, id := range ids {
if id == "" {
continue
}
exists, err := s.fileStore.Exists(id)
if err != nil {
s.logger.Warn("Failed to verify company document file", map[string]interface{}{
"error": err.Error(),
"file_id": id,
})
continue
}
if exists {
sanitized = append(sanitized, id)
}
}
return sanitized
}
+10 -1
View File
@@ -51,7 +51,10 @@ type Company struct {
Email *string `bson:"email,omitempty" json:"email,omitempty"`
// File references stored in GridFS
DocumentFileIDs []string `bson:"document_file_ids,omitempty" json:"document_file_ids,omitempty"`
DocumentFileIDs []string `bson:"document_file_ids" json:"document_file_ids,omitempty"`
// External resource links (e.g. company profile pages, portfolios)
Links []CompanyLink `bson:"links,omitempty" json:"links,omitempty"`
// Tags for categorization and search
Tags *CompanyTags `bson:"tags,omitempty" json:"tags,omitempty"`
@@ -97,6 +100,12 @@ func (c *Company) GetUpdatedAt() int64 {
return c.UpdatedAt
}
// CompanyLink represents an external URL associated with a company.
type CompanyLink struct {
URL string `bson:"url" json:"url"`
Title *string `bson:"title,omitempty" json:"title,omitempty"`
}
// Address represents a company's address
type Address struct {
Street string `bson:"street" json:"street"`
+29 -9
View File
@@ -7,9 +7,9 @@ type (
CompanyForm struct {
Name string `json:"name" valid:"required,length(2|200)" example:"Opplens"`
Type string `json:"type" valid:"required,in(private|public|government|ngo|startup)" example:"private"`
RegistrationNumber string `json:"registration_number" valid:"required,length(5|50)" example:"1234567890"`
TaxID string `json:"tax_id" valid:"required,length(5|50)" example:"1234567890"`
Industry string `json:"industry" valid:"required,length(2|100)" example:"Technology"`
RegistrationNumber string `json:"registration_number" valid:"optional,length(5|50)" example:"1234567890"`
TaxID string `json:"tax_id" valid:"optional,length(5|50)" example:"1234567890"`
Industry string `json:"industry" valid:"optional,length(2|100)" example:"Technology"`
Description *string `json:"description,omitempty" valid:"optional,length(10|1000)" example:"Opplens is a technology company"`
Website *string `json:"website,omitempty" valid:"optional,url" example:"https://opplens.com"`
@@ -24,7 +24,8 @@ type (
// Contact information
Phone *string `json:"phone,omitempty" valid:"optional,length(10|20)" example:"+1234567890"`
Email *string `json:"email,omitempty" valid:"optional,email" example:"info@opplens.com"`
DocumentFileIDs []string `json:"document_file_ids,omitempty" valid:"optional"`
DocumentFileIDs []string `json:"document_file_ids,omitempty" valid:"optional"`
Links []CompanyLinkForm `json:"links,omitempty" valid:"optional"`
// Tags for categorization and search
Tags *CompanyTagsForm `json:"tags,omitempty"`
@@ -35,6 +36,12 @@ type (
Timezone *string `json:"timezone,omitempty" valid:"optional,length(3|50)" example:"UTC"`
}
// CompanyLinkForm represents the form for a company link.
CompanyLinkForm struct {
URL string `json:"url" valid:"required,url" example:"https://example.com/profile"`
Title *string `json:"title,omitempty" valid:"optional,length(1|200)" example:"Company profile"`
}
// AddressForm represents the form for company address
AddressForm struct {
Street string `json:"street" valid:"required,length(5|200)" example:"123 Main St"`
@@ -57,6 +64,10 @@ type (
// SearchForm represents the form for searching companies with filters
type SearchForm struct {
Search *string `query:"q" valid:"optional"`
Name *string `query:"name" valid:"optional,length(1|200)"`
Email *string `query:"email" valid:"optional,length(1|254)"`
Phone *string `query:"phone" valid:"optional,length(1|20)"`
EmployeeCount *int `query:"employee_count" valid:"optional,range(1|100000)"`
Type *string `query:"type" valid:"optional,in(private|public|government|ngo|startup)"`
Status *string `query:"status" valid:"optional,in(active|inactive|suspended|pending)"`
Industry *string `query:"industry" valid:"optional"`
@@ -68,8 +79,8 @@ type SearchForm struct {
Categories []string `query:"categories" valid:"optional"`
Keywords []string `query:"keywords" valid:"optional"`
Specializations []string `query:"specializations" valid:"optional"`
EmployeeCountMin *int `query:"employee_count_min" valid:"optional,min(1)"`
EmployeeCountMax *int `query:"employee_count_max" valid:"optional,min(1)"`
EmployeeCountMin *int `query:"employee_count_min" valid:"optional,range(1|100000)"`
EmployeeCountMax *int `query:"employee_count_max" valid:"optional,range(1|100000)"`
AnnualRevenueMin *float64 `query:"annual_revenue_min" valid:"optional,min(0)"`
AnnualRevenueMax *float64 `query:"annual_revenue_max" valid:"optional,min(0)"`
FoundedYearMin *int `query:"founded_year_min" valid:"optional,range(1800|2100)"`
@@ -86,6 +97,11 @@ type StatusForm struct {
Reason *string `json:"reason" valid:"optional,length(10|500)"`
}
// AddLinksForm is the request body for appending links to a company.
type AddLinksForm struct {
Links []CompanyLinkForm `json:"links" valid:"required"`
}
// VerificationForm represents the form for updating company verification status
type VerificationForm struct {
IsVerified bool `json:"is_verified"`
@@ -124,6 +140,7 @@ type (
Phone *string `json:"phone,omitempty"`
Email *string `json:"email,omitempty"`
DocumentFileIDs []string `json:"document_file_ids,omitempty"`
Links []CompanyLink `json:"links,omitempty"`
Tags *CompanyTagsResponse `json:"tags,omitempty"`
IsVerified bool `json:"is_verified"`
IsCompliant bool `json:"is_compliant"`
@@ -163,6 +180,7 @@ func (c *Company) ToResponse(categories []*CategoryResponse) *CompanyResponse {
Phone: c.Phone,
Email: c.Email,
DocumentFileIDs: c.DocumentFileIDs,
Links: c.Links,
Tags: c.Tags.ToResponse(categories),
IsVerified: c.IsVerified,
IsCompliant: c.IsCompliant,
@@ -192,7 +210,7 @@ type OnboardingResponse struct {
// RecommendedTenderResponse is one ranked tender recommendation from the AI service.
type RecommendedTenderResponse struct {
Rank string `json:"rank"`
Rank int `json:"rank"`
TenderID string `json:"tender_id"`
Analysis string `json:"analysis"`
}
@@ -204,8 +222,9 @@ type CompanyProfileResponse struct {
RegistrationNumber string `json:"registration_number"`
Industry string `json:"industry"`
FoundedYear *int `json:"founded_year,omitempty"`
DocumentFileIDs []string `json:"document_file_ids,omitempty"`
CreatedAt int64 `json:"created_at"`
DocumentFileIDs []string `json:"document_file_ids,omitempty"`
Links []CompanyLink `json:"links,omitempty"`
CreatedAt int64 `json:"created_at"`
}
// ToResponse converts Company to CompanyResponse
@@ -217,6 +236,7 @@ func (c *Company) ToProfileResponse() *CompanyProfileResponse {
Industry: c.Industry,
FoundedYear: c.FoundedYear,
DocumentFileIDs: c.DocumentFileIDs,
Links: c.Links,
CreatedAt: c.CreatedAt,
}
}
+73
View File
@@ -101,6 +101,21 @@ func (h *Handler) StartOnboarding(c echo.Context) error {
// @Security BearerAuth
// @Router /api/v1/recommend [post]
func (h *Handler) RecommendTenders(c echo.Context) error {
if companyIDs, ok := c.Get("company_ids").([]string); ok && len(companyIDs) > 0 {
result, err := h.service.GetMergedAIRecommendations(c.Request().Context(), companyIDs)
if err != nil {
if errors.Is(err, ErrAIRecommendationNotConfigured) {
return response.ServiceUnavailable(c, "AI recommendation service is not configured")
}
if err.Error() == "company not found" {
return response.NotFound(c, "Company not found")
}
return response.InternalServerError(c, "Failed to retrieve AI recommendations")
}
return response.Success(c, result, "AI recommendations retrieved successfully")
}
companyID, err := user.GetCompanyIDFromContext(c)
if err != nil {
return response.Unauthorized(c, "User not authenticated")
@@ -153,6 +168,9 @@ func (h *Handler) Create(c echo.Context) error {
err.Error() == "failed to validate categories" {
return response.BadRequest(c, err.Error(), "Invalid category data")
}
if err.Error() == "invalid link URL" || err.Error() == "too many links provided" {
return response.BadRequest(c, err.Error(), "")
}
return response.InternalServerError(c, "Failed to create company")
}
@@ -223,6 +241,9 @@ func (h *Handler) Update(c echo.Context) error {
err.Error() == "failed to validate categories" {
return response.BadRequest(c, err.Error(), "Invalid category data")
}
if err.Error() == "invalid link URL" || err.Error() == "too many links provided" {
return response.BadRequest(c, err.Error(), "")
}
return response.InternalServerError(c, "Failed to update company")
}
@@ -265,6 +286,10 @@ func (h *Handler) Delete(c echo.Context) error {
// @Accept json
// @Produce json
// @Param q query string false "Search query"
// @Param name query string false "Filter by company name"
// @Param email query string false "Filter by email"
// @Param phone query string false "Filter by phone"
// @Param employee_count query integer false "Filter by exact employee count"
// @Param cpv_codes query array false "CPV codes filter"
// @Param categories query array false "Categories filter"
// @Param keywords query array false "Keywords filter"
@@ -436,6 +461,54 @@ func (h *Handler) UploadDocuments(c echo.Context) error {
return response.Success(c, result, message)
}
// AddLinks appends external links to a company (Web Panel)
// @Summary Add company links
// @Description Append one or more external URLs to the company (max 20 links per request)
// @Tags Admin-Companies
// @Accept json
// @Produce json
// @Param id path string true "Company ID"
// @Param links body AddLinksForm true "Links to append"
// @Success 200 {object} response.APIResponse{data=AddLinksResponse} "Links added successfully"
// @Failure 400 {object} response.APIResponse "Bad request - No links or invalid URLs"
// @Failure 404 {object} response.APIResponse "Not found - Company not found"
// @Failure 422 {object} response.APIResponse "Validation error - Invalid request data"
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Security BearerAuth
// @Router /admin/v1/companies/{id}/links [post]
func (h *Handler) AddLinks(c echo.Context) error {
id := c.Param("id")
form, err := response.Parse[AddLinksForm](c)
if err != nil {
return response.ValidationError(c, "Invalid request data", err.Error())
}
if len(form.Links) > maxCompanyLinks {
return response.ValidationError(c, "Too many links", "Maximum 20 links per request")
}
result, err := h.service.AddLinks(c.Request().Context(), id, form)
if err != nil {
switch err.Error() {
case "company not found":
return response.NotFound(c, "Company not found")
case "no links provided", "no valid links provided", "invalid link URL":
return response.BadRequest(c, err.Error(), "")
case "too many links in request", "company link limit exceeded":
return response.ValidationError(c, err.Error(), "")
default:
return response.InternalServerError(c, "Failed to add company links")
}
}
message := "Company links added successfully"
if len(result.Added) == 0 {
message = "No new company links were added"
}
return response.Success(c, result, message)
}
func collectCompanyUploadFiles(form *multipart.Form) []*multipart.FileHeader {
if form == nil {
return nil
+164
View File
@@ -0,0 +1,164 @@
package company
import (
"context"
"errors"
"strings"
"github.com/asaskevich/govalidator"
)
const maxCompanyLinks = 20
// AddLinksResponse is returned after appending links to a company.
type AddLinksResponse struct {
Links []CompanyLink `json:"links"`
Added []CompanyLink `json:"added"`
}
// AddLinks appends validated links to a company.
func (s *companyService) AddLinks(ctx context.Context, companyID string, form *AddLinksForm) (*AddLinksResponse, error) {
if form == nil || len(form.Links) == 0 {
return nil, errors.New("no links provided")
}
if len(form.Links) > maxCompanyLinks {
return nil, errors.New("too many links in request")
}
company, err := s.repository.GetByID(ctx, companyID)
if err != nil {
return nil, errors.New("company not found")
}
newLinks, err := sanitizeCompanyLinks(convertCompanyLinkForms(form.Links))
if err != nil {
return nil, err
}
if len(newLinks) == 0 {
return nil, errors.New("no valid links provided")
}
existing := sanitizeStoredCompanyLinks(company.Links)
merged, added := mergeCompanyLinks(existing, newLinks)
if len(merged) > maxCompanyLinks {
return nil, errors.New("company link limit exceeded")
}
company.Links = merged
if err := s.repository.Update(ctx, company); err != nil {
s.logger.Error("Failed to save company links", map[string]interface{}{
"error": err.Error(),
"company_id": companyID,
})
return nil, errors.New("failed to update company links")
}
s.logger.Info("Company links added", map[string]interface{}{
"company_id": companyID,
"added_count": len(added),
"links_total": len(company.Links),
})
s.triggerAIOnboardingAsync(companyID)
return &AddLinksResponse{
Links: company.Links,
Added: added,
}, nil
}
func convertCompanyLinkForms(forms []CompanyLinkForm) []CompanyLink {
if len(forms) == 0 {
return nil
}
links := make([]CompanyLink, 0, len(forms))
for _, form := range forms {
links = append(links, CompanyLink{
URL: form.URL,
Title: form.Title,
})
}
return links
}
func sanitizeCompanyLinks(links []CompanyLink) ([]CompanyLink, error) {
if len(links) == 0 {
return nil, nil
}
sanitized := make([]CompanyLink, 0, len(links))
for _, link := range links {
url := strings.TrimSpace(link.URL)
if url == "" {
continue
}
if !govalidator.IsURL(url) {
return nil, errors.New("invalid link URL")
}
var title *string
if link.Title != nil {
trimmed := strings.TrimSpace(*link.Title)
if trimmed != "" {
title = &trimmed
}
}
sanitized = append(sanitized, CompanyLink{
URL: url,
Title: title,
})
}
return sanitized, nil
}
func sanitizeStoredCompanyLinks(links []CompanyLink) []CompanyLink {
sanitized, err := sanitizeCompanyLinks(links)
if err != nil || len(sanitized) == 0 {
return nil
}
return sanitized
}
func mergeCompanyLinks(existing, additional []CompanyLink) ([]CompanyLink, []CompanyLink) {
seen := make(map[string]struct{}, len(existing)+len(additional))
merged := make([]CompanyLink, 0, len(existing)+len(additional))
added := make([]CompanyLink, 0, len(additional))
appendUnique := func(items []CompanyLink, trackAdded bool) {
for _, item := range items {
key := strings.ToLower(strings.TrimSpace(item.URL))
if key == "" {
continue
}
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
merged = append(merged, item)
if trackAdded {
added = append(added, item)
}
}
}
appendUnique(existing, false)
appendUnique(additional, true)
return merged, added
}
func companyLinkURLs(links []CompanyLink) []string {
if len(links) == 0 {
return nil
}
urls := make([]string, 0, len(links))
for _, link := range links {
url := strings.TrimSpace(link.URL)
if url != "" {
urls = append(urls, url)
}
}
return urls
}
+70
View File
@@ -0,0 +1,70 @@
package company
import "testing"
func TestMergeCompanyLinks(t *testing.T) {
existing := []CompanyLink{
{URL: "https://example.com/a"},
{URL: "https://example.com/b"},
}
additional := []CompanyLink{
{URL: " https://example.com/b "},
{URL: "https://Example.com/C"},
{URL: " "},
}
merged, added := mergeCompanyLinks(existing, additional)
if len(merged) != 3 {
t.Fatalf("mergeCompanyLinks() merged len = %d, want 3", len(merged))
}
if len(added) != 1 {
t.Fatalf("mergeCompanyLinks() added len = %d, want 1", len(added))
}
if added[0].URL != "https://Example.com/C" {
t.Fatalf("added link = %+v, want https://Example.com/C", added[0])
}
}
func TestSanitizeCompanyLinks(t *testing.T) {
title := " Profile "
got, err := sanitizeCompanyLinks([]CompanyLink{
{URL: " https://example.com ", Title: &title},
{URL: "not-a-url"},
})
if err == nil {
t.Fatal("sanitizeCompanyLinks() expected invalid URL error")
}
got, err = sanitizeCompanyLinks([]CompanyLink{
{URL: "https://example.com", Title: &title},
})
if err != nil {
t.Fatalf("sanitizeCompanyLinks() unexpected error: %v", err)
}
if len(got) != 1 {
t.Fatalf("sanitizeCompanyLinks() len = %d, want 1", len(got))
}
if got[0].URL != "https://example.com" {
t.Fatalf("sanitized URL = %q, want https://example.com", got[0].URL)
}
if got[0].Title == nil || *got[0].Title != "Profile" {
t.Fatalf("sanitized title = %+v, want Profile", got[0].Title)
}
}
func TestCompanyLinkURLs(t *testing.T) {
got := companyLinkURLs([]CompanyLink{
{URL: " https://a.test "},
{URL: " "},
{URL: "https://b.test"},
})
want := []string{"https://a.test", "https://b.test"}
if len(got) != len(want) {
t.Fatalf("companyLinkURLs() len = %d, want %d", len(got), len(want))
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("companyLinkURLs()[%d] = %q, want %q", i, got[i], want[i])
}
}
}
+8
View File
@@ -0,0 +1,8 @@
package company
// RecommendedTendersPageCacheRefresher pre-builds resolved recommendation pages in Redis.
// Implemented by the tender service on the web API; optional on the worker.
type RecommendedTendersPageCacheRefresher interface {
ScheduleRefreshRecommendedTendersPageCache(companyID string)
InvalidateRecommendedTendersPageCache(companyID string)
}
+376 -8
View File
@@ -8,12 +8,52 @@ import (
"fmt"
"io"
"strings"
"time"
"github.com/redis/go-redis/v9"
"golang.org/x/sync/errgroup"
"tm/pkg/ai_summarizer"
)
var ErrAIRecommendationNotConfigured = errors.New("AI recommendation service is not configured")
const aiRecommendationCacheKeyPrefix = "ai:recommendations:"
const (
recommendationFetchMaxAttempts = 6
recommendationFetchInitialDelay = 10 * time.Second
)
func normalizeRecommendationCompanyIDs(companyIDs []string) []string {
normalized := make([]string, 0, len(companyIDs))
seen := make(map[string]struct{}, len(companyIDs))
for _, companyID := range companyIDs {
clean := strings.TrimSpace(companyID)
if clean == "" {
continue
}
if _, ok := seen[clean]; ok {
continue
}
seen[clean] = struct{}{}
normalized = append(normalized, clean)
}
return normalized
}
func mergeRecommendedTenders(recommendationLists ...[]RecommendedTenderResponse) []RecommendedTenderResponse {
merged := make([]RecommendedTenderResponse, 0)
for _, recommendations := range recommendationLists {
merged = append(merged, recommendations...)
}
return dedupeRecommendationResponsesByTenderID(merged)
}
func aiRecommendationCacheKey(companyID string) string {
return aiRecommendationCacheKeyPrefix + companyID
}
// AIRecommendationClient defines the interface for company tender recommendation AI operations.
type AIRecommendationClient interface {
StartOnboarding(ctx context.Context, reqBody ai_summarizer.OnboardingRequest) (*ai_summarizer.OnboardingResponse, error)
@@ -31,10 +71,67 @@ func (s *companyService) StartAIOnboarding(ctx context.Context, companyID string
if s.aiRecommendationClient == nil {
return nil, ErrAIRecommendationNotConfigured
}
if strings.TrimSpace(companyID) == "" {
companyID = strings.TrimSpace(companyID)
if companyID == "" {
return nil, errors.New("company ID is required")
}
s.invalidateAIRecommendationCache(ctx, companyID)
result, err := s.startAIOnboardingInternal(ctx, companyID)
if err != nil {
return nil, err
}
s.scheduleRecommendationRefreshAfterOnboarding(companyID)
return result, nil
}
// triggerAIOnboardingAsync re-indexes the company on the AI service and refreshes the recommendation cache.
// This is the only path (besides StartAIOnboarding) that calls the AI recommend endpoint.
func (s *companyService) triggerAIOnboardingAsync(companyID string) {
if s.aiRecommendationClient == nil {
return
}
companyID = strings.TrimSpace(companyID)
if companyID == "" {
return
}
go func() {
ctx := context.Background()
s.logger.Info("Starting async AI company onboarding and recommendation refresh", map[string]interface{}{
"company_id": companyID,
})
s.invalidateAIRecommendationCache(ctx, companyID)
if _, err := s.startAIOnboardingInternal(ctx, companyID); err != nil {
s.logger.Error("Async AI company onboarding failed", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
return
}
if err := s.fetchAndCacheAIRecommendationsWithRetry(ctx, companyID); err != nil {
s.logger.Error("Failed to refresh AI recommendations after onboarding", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
return
}
s.logger.Info("Async AI company onboarding and recommendation refresh completed", map[string]interface{}{
"company_id": companyID,
})
}()
}
func (s *companyService) startAIOnboardingInternal(ctx context.Context, companyID string) (*OnboardingResponse, error) {
company, err := s.repository.GetByID(ctx, companyID)
if err != nil {
s.logger.Error("Failed to load company for AI onboarding", map[string]interface{}{
@@ -59,15 +156,17 @@ func (s *companyService) StartAIOnboarding(ctx context.Context, companyID string
}
s.logger.Info("Starting AI company onboarding", map[string]interface{}{
"company_id": companyID,
"documents_count": len(company.DocumentFileIDs),
"has_website_url": websiteURL != "",
"company_id": companyID,
"documents_count": len(company.DocumentFileIDs),
"links_count": len(company.Links),
"has_website_url": websiteURL != "",
})
result, err := s.aiRecommendationClient.StartOnboarding(ctx, ai_summarizer.OnboardingRequest{
CompanyID: companyID,
Documents: documents,
WebsiteURL: websiteURL,
Links: companyLinkURLs(company.Links),
})
if err != nil {
s.logger.Error("AI onboarding request failed", map[string]interface{}{
@@ -80,15 +179,22 @@ func (s *companyService) StartAIOnboarding(ctx context.Context, companyID string
return &OnboardingResponse{Status: result.Status}, nil
}
// GetAIRecommendations retrieves ranked tender recommendations from the AI team.
// GetAIRecommendations returns cached recommendations only. The AI service is not called on read;
// cache is populated when a company is created, updated, or onboarded.
func (s *companyService) GetAIRecommendations(ctx context.Context, companyID string) ([]RecommendedTenderResponse, error) {
if s.aiRecommendationClient == nil {
return nil, ErrAIRecommendationNotConfigured
}
if strings.TrimSpace(companyID) == "" {
companyID = strings.TrimSpace(companyID)
if companyID == "" {
return nil, errors.New("company ID is required")
}
if cached, ok := s.getCachedAIRecommendations(ctx, companyID); ok {
return cached, nil
}
if _, err := s.repository.GetByID(ctx, companyID); err != nil {
s.logger.Error("Failed to load company for AI recommendations", map[string]interface{}{
"company_id": companyID,
@@ -97,7 +203,157 @@ func (s *companyService) GetAIRecommendations(ctx context.Context, companyID str
return nil, errors.New("company not found")
}
s.logger.Info("Fetching AI tender recommendations", map[string]interface{}{
s.logger.Debug("AI recommendations cache miss; returning empty list", map[string]interface{}{
"company_id": companyID,
})
return []RecommendedTenderResponse{}, nil
}
func (s *companyService) validateRecommendationCompanyIDs(ctx context.Context, companyIDs []string) error {
companies, err := s.repository.GetByIDs(ctx, companyIDs)
if err != nil {
s.logger.Error("Failed to load companies for merged AI recommendations", map[string]interface{}{
"company_ids": companyIDs,
"error": err.Error(),
})
return errors.New("company not found")
}
if len(companies) != len(companyIDs) {
s.logger.Error("Some companies were not found for merged AI recommendations", map[string]interface{}{
"requested_count": len(companyIDs),
"loaded_count": len(companies),
})
return errors.New("company not found")
}
return nil
}
func (s *companyService) getCachedAIRecommendationsOrEmpty(ctx context.Context, companyID string) []RecommendedTenderResponse {
if cached, ok := s.getCachedAIRecommendations(ctx, companyID); ok {
return cached
}
s.logger.Debug("AI recommendations cache miss; returning empty list", map[string]interface{}{
"company_id": companyID,
})
return []RecommendedTenderResponse{}
}
// GetMergedAIRecommendations unions cached AI recommendations across multiple companies.
func (s *companyService) GetMergedAIRecommendations(ctx context.Context, companyIDs []string) ([]RecommendedTenderResponse, error) {
if s.aiRecommendationClient == nil {
return nil, ErrAIRecommendationNotConfigured
}
normalized := normalizeRecommendationCompanyIDs(companyIDs)
if len(normalized) == 0 {
return []RecommendedTenderResponse{}, nil
}
recommendationLists := make([][]RecommendedTenderResponse, len(normalized))
group, groupCtx := errgroup.WithContext(ctx)
group.Go(func() error {
return s.validateRecommendationCompanyIDs(groupCtx, normalized)
})
for i, companyID := range normalized {
i := i
companyID := companyID
group.Go(func() error {
recommendationLists[i] = s.getCachedAIRecommendationsOrEmpty(groupCtx, companyID)
return nil
})
}
if err := group.Wait(); err != nil {
return nil, err
}
return mergeRecommendedTenders(recommendationLists...), nil
}
func (s *companyService) scheduleRecommendationRefreshAfterOnboarding(companyID string) {
if s.aiRecommendationClient == nil {
return
}
companyID = strings.TrimSpace(companyID)
if companyID == "" {
return
}
go func() {
ctx := context.Background()
if err := s.fetchAndCacheAIRecommendationsWithRetry(ctx, companyID); err != nil {
s.logger.Error("Failed to refresh AI recommendation cache after onboarding", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
}
}()
}
// fetchAndCacheAIRecommendationsWithRetry waits for the AI service to finish indexing after onboarding.
// Empty results are not cached so reads keep returning empty until real recommendations exist.
func (s *companyService) fetchAndCacheAIRecommendationsWithRetry(ctx context.Context, companyID string) error {
delay := recommendationFetchInitialDelay
for attempt := 1; attempt <= recommendationFetchMaxAttempts; attempt++ {
if attempt > 1 {
s.logger.Info("Retrying AI recommendation fetch after onboarding", map[string]interface{}{
"company_id": companyID,
"attempt": attempt,
"delay_sec": int(delay.Seconds()),
})
} else {
s.logger.Info("Waiting before first AI recommendation fetch after onboarding", map[string]interface{}{
"company_id": companyID,
"delay_sec": int(delay.Seconds()),
})
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(delay):
}
if attempt > 1 {
delay *= 2
}
responses, err := s.fetchAIRecommendations(ctx, companyID)
if err != nil {
if attempt == recommendationFetchMaxAttempts {
return err
}
continue
}
if len(responses) == 0 {
if attempt == recommendationFetchMaxAttempts {
s.logger.Warn("AI returned no recommendations after onboarding retries; cache not updated", map[string]interface{}{
"company_id": companyID,
"attempts": attempt,
})
return nil
}
continue
}
s.cacheAIRecommendations(ctx, companyID, responses)
s.logger.Info("AI recommendation cache populated after onboarding", map[string]interface{}{
"company_id": companyID,
"count": len(responses),
"attempt": attempt,
})
return nil
}
return nil
}
func (s *companyService) fetchAIRecommendations(ctx context.Context, companyID string) ([]RecommendedTenderResponse, error) {
s.logger.Info("Fetching AI tender recommendations from AI service", map[string]interface{}{
"company_id": companyID,
})
@@ -112,6 +368,10 @@ func (s *companyService) GetAIRecommendations(ctx context.Context, companyID str
return nil, fmt.Errorf("failed to fetch AI recommendations: %w", err)
}
return dedupeRecommendationResponsesByTenderID(itemsToRecommendationResponses(items)), nil
}
func itemsToRecommendationResponses(items []ai_summarizer.RecommendedTender) []RecommendedTenderResponse {
responses := make([]RecommendedTenderResponse, 0, len(items))
for _, item := range items {
responses = append(responses, RecommendedTenderResponse{
@@ -120,8 +380,116 @@ func (s *companyService) GetAIRecommendations(ctx context.Context, companyID str
Analysis: item.Analysis,
})
}
return responses
}
return responses, nil
func dedupeRecommendationResponsesByTenderID(responses []RecommendedTenderResponse) []RecommendedTenderResponse {
seen := make(map[string]struct{}, len(responses))
out := make([]RecommendedTenderResponse, 0, len(responses))
for _, recommendation := range responses {
tenderID := strings.TrimSpace(recommendation.TenderID)
if tenderID == "" {
continue
}
if _, ok := seen[tenderID]; ok {
continue
}
seen[tenderID] = struct{}{}
recommendation.TenderID = tenderID
recommendation.Analysis = strings.TrimSpace(recommendation.Analysis)
out = append(out, recommendation)
}
return out
}
func (s *companyService) getCachedAIRecommendations(ctx context.Context, companyID string) ([]RecommendedTenderResponse, bool) {
if s.redisClient == nil {
return nil, false
}
readCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
raw, err := s.redisClient.Get(readCtx, aiRecommendationCacheKey(companyID))
if err != nil {
if err != redis.Nil {
s.logger.Warn("Failed to read AI recommendation cache", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
}
return nil, false
}
var responses []RecommendedTenderResponse
if err := json.Unmarshal([]byte(raw), &responses); err != nil {
s.logger.Warn("Failed to decode AI recommendation cache", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
_ = s.redisClient.Del(ctx, aiRecommendationCacheKey(companyID))
return nil, false
}
if len(responses) == 0 {
_ = s.redisClient.Del(ctx, aiRecommendationCacheKey(companyID))
return nil, false
}
s.logger.Debug("AI tender recommendations served from cache", map[string]interface{}{
"company_id": companyID,
"count": len(responses),
})
return responses, true
}
func (s *companyService) cacheAIRecommendations(ctx context.Context, companyID string, responses []RecommendedTenderResponse) {
if s.redisClient == nil || len(responses) == 0 {
return
}
encoded, err := json.Marshal(responses)
if err != nil {
s.logger.Warn("Failed to encode AI recommendations for cache", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
return
}
expiration := s.recommendationCacheExpiration()
if err := s.redisClient.Set(ctx, aiRecommendationCacheKey(companyID), string(encoded), expiration); err != nil {
s.logger.Warn("Failed to store AI recommendation cache", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
return
}
s.invalidateRecommendedTendersPageCache(companyID)
s.scheduleRecommendedTendersPageCacheRefresh(companyID)
}
func (s *companyService) recommendationCacheExpiration() time.Duration {
if s.recommendationCacheTTL > 0 {
return s.recommendationCacheTTL
}
return 0
}
func (s *companyService) invalidateAIRecommendationCache(ctx context.Context, companyID string) {
if s.redisClient == nil {
return
}
if err := s.redisClient.Del(ctx, aiRecommendationCacheKey(companyID)); err != nil {
s.logger.Warn("Failed to invalidate AI recommendation cache", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
}
s.invalidateRecommendedTendersPageCache(companyID)
}
func (s *companyService) buildOnboardingDocuments(fileIDs []string) (string, error) {
@@ -0,0 +1,20 @@
package company
func (s *companyService) invalidateRecommendedTendersPageCache(companyID string) {
if s.pageCacheRefresher == nil {
return
}
s.pageCacheRefresher.InvalidateRecommendedTendersPageCache(companyID)
}
func (s *companyService) scheduleRecommendedTendersPageCacheRefresh(companyID string) {
if s.pageCacheRefresher == nil {
return
}
s.pageCacheRefresher.ScheduleRefreshRecommendedTendersPageCache(companyID)
}
// SetRecommendedTendersPageCacheRefresher wires async page-cache rebuilds from the tender service.
func (s *companyService) SetRecommendedTendersPageCacheRefresher(refresher RecommendedTendersPageCacheRefresher) {
s.pageCacheRefresher = refresher
}
@@ -0,0 +1,221 @@
package company
import (
"context"
"strings"
"sync/atomic"
"time"
"golang.org/x/sync/errgroup"
"tm/pkg/ai_summarizer"
)
// AIPipelineStatusClient reports pipeline job status from the AI service.
type AIPipelineStatusClient interface {
GetPipelineLastAutoRun(ctx context.Context) (*ai_summarizer.PipelineReportResponse, error)
}
const (
recommendationPipelineWaitInitialDelay = 5 * time.Minute
recommendationPipelineWaitMaxAttempts = 24
recommendationRefreshConcurrency = 3
)
// ScheduleRefreshCachedAIRecommendationsAfterPipeline waits for the AI pipeline auto run to finish,
// then re-fetches ranked tenders for every company that already has a recommendation cache.
func (s *companyService) ScheduleRefreshCachedAIRecommendationsAfterPipeline() {
if s.aiRecommendationClient == nil {
return
}
go func() {
ctx := context.Background()
s.logger.Info("Scheduling AI recommendation cache refresh after pipeline sync", map[string]interface{}{})
if err := s.refreshCachedAIRecommendationsAfterPipeline(ctx); err != nil {
s.logger.Error("AI recommendation cache refresh after pipeline failed", map[string]interface{}{
"error": err.Error(),
})
return
}
s.logger.Info("AI recommendation cache refresh after pipeline completed", map[string]interface{}{})
}()
}
func (s *companyService) refreshCachedAIRecommendationsAfterPipeline(ctx context.Context) error {
if err := s.waitForPipelineAutoCompletion(ctx); err != nil {
return err
}
companyIDs, err := s.listCompanyIDsWithRecommendationCache(ctx)
if err != nil {
return err
}
if len(companyIDs) == 0 {
s.logger.Info("No companies with recommendation cache to refresh after pipeline", map[string]interface{}{})
return nil
}
s.logger.Info("Refreshing AI recommendation caches after pipeline", map[string]interface{}{
"company_count": len(companyIDs),
})
group, groupCtx := errgroup.WithContext(ctx)
group.SetLimit(recommendationRefreshConcurrency)
var refreshed atomic.Int32
var failed atomic.Int32
for _, companyID := range companyIDs {
companyID := companyID
group.Go(func() error {
if err := s.refreshAIRecommendationsCache(groupCtx, companyID); err != nil {
s.logger.Error("Failed to refresh AI recommendation cache for company", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
failed.Add(1)
return nil
}
refreshed.Add(1)
return nil
})
}
if err := group.Wait(); err != nil {
return err
}
s.logger.Info("AI recommendation cache refresh summary", map[string]interface{}{
"company_count": len(companyIDs),
"refreshed": refreshed.Load(),
"failed": failed.Load(),
})
return nil
}
func (s *companyService) waitForPipelineAutoCompletion(ctx context.Context) error {
if s.aiPipelineStatusClient == nil {
s.logger.Info("AI pipeline status client not configured; waiting before recommendation refresh", map[string]interface{}{
"delay_sec": int(recommendationPipelineWaitInitialDelay.Seconds()),
})
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(recommendationPipelineWaitInitialDelay):
}
return nil
}
delay := recommendationPipelineWaitInitialDelay
for attempt := 1; attempt <= recommendationPipelineWaitMaxAttempts; attempt++ {
if attempt == 1 {
s.logger.Info("Waiting before first AI pipeline status check", map[string]interface{}{
"delay_sec": int(delay.Seconds()),
})
} else {
s.logger.Info("Retrying AI pipeline status check", map[string]interface{}{
"attempt": attempt,
"delay_sec": int(delay.Seconds()),
})
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(delay):
}
if attempt > 1 {
delay *= 2
if delay > 30*time.Minute {
delay = 30 * time.Minute
}
}
report, err := s.aiPipelineStatusClient.GetPipelineLastAutoRun(ctx)
if err != nil {
s.logger.Warn("Failed to read AI pipeline last-auto-run status", map[string]interface{}{
"attempt": attempt,
"error": err.Error(),
})
if attempt == recommendationPipelineWaitMaxAttempts {
s.logger.Warn("Proceeding with recommendation refresh without pipeline status", map[string]interface{}{})
return nil
}
continue
}
if !isPipelineRunInProgress(report) {
s.logger.Info("AI pipeline auto run finished; refreshing recommendation caches", map[string]interface{}{
"attempt": attempt,
"status": strings.TrimSpace(report.Status),
})
return nil
}
if attempt == recommendationPipelineWaitMaxAttempts {
s.logger.Warn("AI pipeline still in progress after max wait; refreshing recommendation caches anyway", map[string]interface{}{
"attempts": attempt,
"status": strings.TrimSpace(report.Status),
})
return nil
}
}
return nil
}
func isPipelineRunInProgress(report *ai_summarizer.PipelineReportResponse) bool {
if report == nil {
return true
}
status := strings.ToLower(strings.TrimSpace(report.Status))
switch status {
case "", "running", "in_progress", "started", "processing", "pending", "queued":
return true
default:
return false
}
}
func (s *companyService) listCompanyIDsWithRecommendationCache(ctx context.Context) ([]string, error) {
allIDs, err := s.repository.ListIDs(ctx)
if err != nil {
s.logger.Error("Failed to list companies for recommendation refresh", map[string]interface{}{
"error": err.Error(),
})
return nil, err
}
cached := make([]string, 0, len(allIDs))
for _, companyID := range allIDs {
if _, ok := s.getCachedAIRecommendations(ctx, companyID); ok {
cached = append(cached, companyID)
}
}
return cached, nil
}
// refreshAIRecommendationsCache updates Redis when the AI service returns a non-empty ranked list.
// Empty or failed fetches leave the existing cache untouched.
func (s *companyService) refreshAIRecommendationsCache(ctx context.Context, companyID string) error {
responses, err := s.fetchAIRecommendations(ctx, companyID)
if err != nil {
return err
}
if len(responses) == 0 {
s.logger.Info("Skipping recommendation cache update after pipeline: AI returned empty list", map[string]interface{}{
"company_id": companyID,
})
return nil
}
s.cacheAIRecommendations(ctx, companyID, responses)
s.logger.Info("AI recommendation cache refreshed after pipeline", map[string]interface{}{
"company_id": companyID,
"count": len(responses),
})
return nil
}
@@ -0,0 +1,30 @@
package company
import (
"testing"
"tm/pkg/ai_summarizer"
)
func TestIsPipelineRunInProgress(t *testing.T) {
tests := []struct {
name string
report *ai_summarizer.PipelineReportResponse
want bool
}{
{name: "nil report", report: nil, want: true},
{name: "empty status", report: &ai_summarizer.PipelineReportResponse{}, want: true},
{name: "running", report: &ai_summarizer.PipelineReportResponse{Status: "running"}, want: true},
{name: "in progress", report: &ai_summarizer.PipelineReportResponse{Status: "in_progress"}, want: true},
{name: "completed", report: &ai_summarizer.PipelineReportResponse{Status: "completed"}, want: false},
{name: "failed", report: &ai_summarizer.PipelineReportResponse{Status: "failed"}, want: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isPipelineRunInProgress(tt.report); got != tt.want {
t.Fatalf("isPipelineRunInProgress() = %v, want %v", got, tt.want)
}
})
}
}
+64
View File
@@ -0,0 +1,64 @@
package company
import "testing"
func TestDedupeRecommendationResponsesByTenderID(t *testing.T) {
input := []RecommendedTenderResponse{
{Rank: 1, TenderID: " PROC-1 ", Analysis: " first "},
{Rank: 2, TenderID: "PROC-2", Analysis: "second"},
{Rank: 1, TenderID: "PROC-2", Analysis: "duplicate"},
{Rank: 4, TenderID: " ", Analysis: "ignored"},
}
got := dedupeRecommendationResponsesByTenderID(input)
if len(got) != 2 {
t.Fatalf("dedupeRecommendationResponsesByTenderID() len = %d, want 2", len(got))
}
if got[0].TenderID != "PROC-1" || got[0].Analysis != "first" {
t.Fatalf("first recommendation = %+v, want trimmed PROC-1/first", got[0])
}
if got[1].TenderID != "PROC-2" || got[1].Rank != 2 {
t.Fatalf("second recommendation = %+v, want original PROC-2 rank 2", got[1])
}
}
func TestMergeRecommendedTenders(t *testing.T) {
first := []RecommendedTenderResponse{
{Rank: 1, TenderID: " PROC-1 ", Analysis: " first "},
{Rank: 2, TenderID: "PROC-2", Analysis: "second"},
}
second := []RecommendedTenderResponse{
{Rank: 1, TenderID: "PROC-2", Analysis: "duplicate"},
{Rank: 3, TenderID: "PROC-3", Analysis: "third"},
{Rank: 4, TenderID: " ", Analysis: "ignored"},
}
got := mergeRecommendedTenders(first, second)
if len(got) != 3 {
t.Fatalf("mergeRecommendedTenders() len = %d, want 3", len(got))
}
if got[0].TenderID != "PROC-1" || got[0].Analysis != "first" {
t.Fatalf("first recommendation = %+v, want trimmed PROC-1/first", got[0])
}
if got[1].TenderID != "PROC-2" || got[1].Rank != 2 {
t.Fatalf("second recommendation = %+v, want original PROC-2 rank 2", got[1])
}
if got[2].TenderID != "PROC-3" {
t.Fatalf("third recommendation = %+v, want PROC-3", got[2])
}
}
func TestNormalizeRecommendationCompanyIDs(t *testing.T) {
got := normalizeRecommendationCompanyIDs([]string{" company-a ", "", "company-b", "company-a"})
want := []string{"company-a", "company-b"}
if len(got) != len(want) {
t.Fatalf("normalizeRecommendationCompanyIDs() len = %d, want %d", len(got), len(want))
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("normalizeRecommendationCompanyIDs()[%d] = %q, want %q", i, got[i], want[i])
}
}
}
+148 -6
View File
@@ -3,12 +3,15 @@ package company
import (
"context"
"errors"
"regexp"
"strings"
"time"
"tm/pkg/logger"
orm "tm/pkg/mongo"
"tm/pkg/response"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
)
// Repository defines the interface for company data operations
@@ -21,15 +24,18 @@ type Repository interface {
GetByRegistrationNumber(ctx context.Context, registrationNumber string) (*Company, error)
GetByTaxID(ctx context.Context, taxID string) (*Company, error)
Search(ctx context.Context, form *SearchForm, pagination *response.Pagination) (*orm.PaginatedResult[Company], error)
ListIDs(ctx context.Context) ([]string, error)
Delete(ctx context.Context, id string) error
UpdateStatus(ctx context.Context, id string, status CompanyStatus) error
UpdateVerification(ctx context.Context, id string, isVerified, isCompliant bool, complianceNotes *string) error
RemoveDocumentFileID(ctx context.Context, fileID string) error
}
// companyRepository implements the Repository interface using the MongoDB ORM
type companyRepository struct {
ormRepo orm.Repository[Company]
logger logger.Logger
ormRepo orm.Repository[Company]
collection *mongo.Collection
logger logger.Logger
}
// NewCompanyRepository creates a new company repository
@@ -47,12 +53,15 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Re
})
}
collection := mongoManager.GetCollection("companies")
// Create ORM repository
ormRepo := orm.NewRepository[Company](mongoManager.GetCollection("companies"), logger)
ormRepo := orm.NewRepository[Company](collection, logger)
return &companyRepository{
ormRepo: ormRepo,
logger: logger,
ormRepo: ormRepo,
collection: collection,
logger: logger,
}
}
@@ -161,6 +170,9 @@ func (r *companyRepository) Update(ctx context.Context, company *Company) error
// Set updated timestamp using Unix timestamp
company.SetUpdatedAt(time.Now().Unix())
clearDocuments := company.DocumentFileIDs != nil && len(company.DocumentFileIDs) == 0
clearLinks := company.Links != nil && len(company.Links) == 0
// Use ORM to update company
err := r.ormRepo.Update(ctx, company)
if err != nil {
@@ -171,6 +183,30 @@ func (r *companyRepository) Update(ctx context.Context, company *Company) error
return err
}
// BSON omitempty can skip empty slices in $set, so persist explicit empty arrays.
if clearDocuments || clearLinks {
setFields := bson.M{
"updated_at": company.UpdatedAt,
}
if clearDocuments {
setFields["document_file_ids"] = []string{}
}
if clearLinks {
setFields["links"] = []CompanyLink{}
}
_, err = r.collection.UpdateOne(ctx, bson.M{"_id": company.ID}, bson.M{
"$set": setFields,
})
if err != nil {
r.logger.Error("Failed to clear company collection fields", map[string]interface{}{
"error": err.Error(),
"company_id": company.ID,
})
return err
}
}
r.logger.Info("Company updated successfully", map[string]interface{}{
"company_id": company.ID,
"name": company.Name,
@@ -213,6 +249,36 @@ func (r *companyRepository) Search(ctx context.Context, form *SearchForm, pagina
filter["$text"] = bson.M{"$search": *form.Search}
}
if form.Name != nil {
name := strings.TrimSpace(*form.Name)
if name != "" {
filter["name"] = bson.M{
"$regex": regexp.QuoteMeta(name),
"$options": "i",
}
}
}
if form.Email != nil {
email := strings.TrimSpace(*form.Email)
if email != "" {
filter["email"] = bson.M{
"$regex": regexp.QuoteMeta(email),
"$options": "i",
}
}
}
if form.Phone != nil {
phone := strings.TrimSpace(*form.Phone)
if phone != "" {
filter["phone"] = bson.M{
"$regex": regexp.QuoteMeta(phone),
"$options": "i",
}
}
}
if form.Type != nil {
filter["type"] = *form.Type
}
@@ -259,7 +325,9 @@ func (r *companyRepository) Search(ctx context.Context, form *SearchForm, pagina
}
// Range filters
if form.EmployeeCountMin != nil || form.EmployeeCountMax != nil {
if form.EmployeeCount != nil {
filter["employee_count"] = *form.EmployeeCount
} else if form.EmployeeCountMin != nil || form.EmployeeCountMax != nil {
rangeFilter := bson.M{}
if form.EmployeeCountMin != nil {
rangeFilter["$gte"] = *form.EmployeeCountMin
@@ -316,6 +384,49 @@ func (r *companyRepository) Search(ctx context.Context, form *SearchForm, pagina
return result, nil
}
// ListIDs returns every company document ID.
func (r *companyRepository) ListIDs(ctx context.Context) ([]string, error) {
const pageSize = 500
ids := make([]string, 0)
skip := 0
for {
result, err := r.ormRepo.FindAll(ctx, bson.M{}, orm.Pagination{
Limit: pageSize,
Skip: skip,
SortField: "_id",
SortOrder: 1,
SkipCount: true,
Projection: bson.M{"_id": 1},
})
if err != nil {
r.logger.Error("Failed to list company IDs", map[string]interface{}{
"error": err.Error(),
"skip": skip,
})
return nil, err
}
if len(result.Items) == 0 {
break
}
for i := range result.Items {
id := strings.TrimSpace(result.Items[i].GetID())
if id != "" {
ids = append(ids, id)
}
}
if len(result.Items) < pageSize {
break
}
skip += pageSize
}
return ids, nil
}
// UpdateStatus updates company status
func (r *companyRepository) UpdateStatus(ctx context.Context, id string, status CompanyStatus) error {
// Get company first
@@ -379,3 +490,34 @@ func (r *companyRepository) UpdateVerification(ctx context.Context, id string, i
return nil
}
// RemoveDocumentFileID removes a file ID from all companies that reference it.
func (r *companyRepository) RemoveDocumentFileID(ctx context.Context, fileID string) error {
if fileID == "" {
return nil
}
filter := bson.M{"document_file_ids": fileID}
update := bson.M{
"$pull": bson.M{"document_file_ids": fileID},
"$set": bson.M{"updated_at": time.Now().Unix()},
}
result, err := r.collection.UpdateMany(ctx, filter, update)
if err != nil {
r.logger.Error("Failed to remove document file ID from companies", map[string]interface{}{
"error": err.Error(),
"file_id": fileID,
})
return err
}
if result.ModifiedCount > 0 {
r.logger.Info("Removed document file ID from companies", map[string]interface{}{
"file_id": fileID,
"modified_count": result.ModifiedCount,
})
}
return nil
}
+63 -12
View File
@@ -3,12 +3,13 @@ package company
import (
"context"
"errors"
"mime/multipart"
"time"
"tm/internal/company_category"
"tm/pkg/filestore"
"tm/pkg/logger"
"tm/pkg/redis"
"tm/pkg/response"
)
@@ -35,11 +36,21 @@ type Service interface {
// UploadDocuments uploads files and attaches them to a company
UploadDocuments(ctx context.Context, companyID, uploadedBy string, files []*multipart.FileHeader) (*UploadDocumentsResponse, error)
// AddLinks appends external URLs to a company
AddLinks(ctx context.Context, companyID string, form *AddLinksForm) (*AddLinksResponse, error)
// DetachDocumentFileID removes a file ID from companies when the file is deleted
DetachDocumentFileID(ctx context.Context, fileID string) error
// StartAIOnboarding sends company data to the AI team for recommendation onboarding
StartAIOnboarding(ctx context.Context, companyID string) (*OnboardingResponse, error)
// GetAIRecommendations returns ranked tender recommendations from the AI team
GetAIRecommendations(ctx context.Context, companyID string) ([]RecommendedTenderResponse, error)
GetMergedAIRecommendations(ctx context.Context, companyIDs []string) ([]RecommendedTenderResponse, error)
// ScheduleRefreshCachedAIRecommendationsAfterPipeline re-fetches cached company recommendations after tender sync.
ScheduleRefreshCachedAIRecommendationsAfterPipeline()
}
// companyService implements the Service interface
@@ -48,6 +59,10 @@ type companyService struct {
categoryService company_category.Service
fileStore filestore.FileStoreService
aiRecommendationClient AIRecommendationClient
aiPipelineStatusClient AIPipelineStatusClient
pageCacheRefresher RecommendedTendersPageCacheRefresher
redisClient redis.Client
recommendationCacheTTL time.Duration
logger logger.Logger
}
@@ -57,6 +72,9 @@ func NewService(
categoryService company_category.Service,
fileStore filestore.FileStoreService,
aiRecommendationClient AIRecommendationClient,
aiPipelineStatusClient AIPipelineStatusClient,
redisClient redis.Client,
recommendationCacheTTL time.Duration,
logger logger.Logger,
) Service {
return &companyService{
@@ -64,6 +82,9 @@ func NewService(
categoryService: categoryService,
fileStore: fileStore,
aiRecommendationClient: aiRecommendationClient,
aiPipelineStatusClient: aiPipelineStatusClient,
redisClient: redisClient,
recommendationCacheTTL: recommendationCacheTTL,
logger: logger,
}
}
@@ -93,16 +114,26 @@ func (s *companyService) Create(ctx context.Context, form *CompanyForm) (*Compan
return nil, errors.New("company with this name already exists")
}
// Check if registration number already exists
existingCompany, _ = s.repository.GetByRegistrationNumber(ctx, form.RegistrationNumber)
if existingCompany != nil {
return nil, errors.New("company with this registration number already exists")
if form.RegistrationNumber != "" {
existingCompany, _ = s.repository.GetByRegistrationNumber(ctx, form.RegistrationNumber)
if existingCompany != nil {
return nil, errors.New("company with this registration number already exists")
}
}
// Check if tax ID already exists
existingCompany, _ = s.repository.GetByTaxID(ctx, form.TaxID)
if existingCompany != nil {
return nil, errors.New("company with this tax ID already exists")
if form.TaxID != "" {
existingCompany, _ = s.repository.GetByTaxID(ctx, form.TaxID)
if existingCompany != nil {
return nil, errors.New("company with this tax ID already exists")
}
}
links, err := sanitizeCompanyLinks(convertCompanyLinkForms(form.Links))
if err != nil {
return nil, err
}
if len(links) > maxCompanyLinks {
return nil, errors.New("too many links provided")
}
// Create company
@@ -121,7 +152,8 @@ func (s *companyService) Create(ctx context.Context, form *CompanyForm) (*Compan
Address: s.convertAddressForm(form.Address),
Phone: form.Phone,
Email: form.Email,
DocumentFileIDs: form.DocumentFileIDs,
DocumentFileIDs: s.sanitizeDocumentFileIDs(form.DocumentFileIDs),
Links: links,
Tags: s.convertCompanyTagsForm(form.Tags),
IsVerified: false,
IsCompliant: false,
@@ -131,7 +163,7 @@ func (s *companyService) Create(ctx context.Context, form *CompanyForm) (*Compan
}
// Save to database
err := s.repository.Create(ctx, company)
err = s.repository.Create(ctx, company)
if err != nil {
s.logger.Error("Failed to create company", map[string]interface{}{
"error": err.Error(),
@@ -147,6 +179,8 @@ func (s *companyService) Create(ctx context.Context, form *CompanyForm) (*Compan
"type": company.Type,
})
s.triggerAIOnboardingAsync(company.GetID())
return company.ToResponse(nil), nil
}
@@ -183,6 +217,8 @@ func (s *companyService) GetByID(ctx context.Context, id string) (*CompanyRespon
})
}
company.DocumentFileIDs = s.sanitizeDocumentFileIDs(company.DocumentFileIDs)
return company.ToResponse(convertedCategories), nil
}
@@ -284,7 +320,18 @@ func (s *companyService) Update(ctx context.Context, id string, form *CompanyFor
}
if form.DocumentFileIDs != nil {
company.DocumentFileIDs = form.DocumentFileIDs
company.DocumentFileIDs = s.sanitizeDocumentFileIDs(form.DocumentFileIDs)
}
if form.Links != nil {
links, linkErr := sanitizeCompanyLinks(convertCompanyLinkForms(form.Links))
if linkErr != nil {
return nil, linkErr
}
if len(links) > maxCompanyLinks {
return nil, errors.New("too many links provided")
}
company.Links = links
}
if form.Tags != nil {
@@ -318,6 +365,8 @@ func (s *companyService) Update(ctx context.Context, id string, form *CompanyFor
"name": company.Name,
})
s.triggerAIOnboardingAsync(company.GetID())
return company.ToResponse(nil), nil
}
@@ -497,6 +546,8 @@ func (s *companyService) GetMyCompany(ctx context.Context, id string) (*CompanyP
return nil, errors.New("company not found")
}
company.DocumentFileIDs = s.sanitizeDocumentFileIDs(company.DocumentFileIDs)
return company.ToProfileResponse(), nil
}
@@ -88,9 +88,11 @@ func (s *customerService) AdminResetPassword(ctx context.Context, actorID, targe
s.auditLogger.Log(ctx, audit.Entry{
ActorID: actorID,
ActorType: audit.ActorTypeAdmin,
TargetType: audit.TargetTypeCustomer,
TargetID: targetID,
Action: audit.ActionPasswordReset,
Success: true,
})
s.logger.Info("Customer password reset completed", map[string]interface{}{
+103
View File
@@ -0,0 +1,103 @@
package customer
import (
"context"
"errors"
"strings"
)
// ErrCompanyNotAssigned is returned when the requested company is not assigned to the customer.
var ErrCompanyNotAssigned = errors.New("company is not assigned to customer")
var errCompanyNotAssigned = ErrCompanyNotAssigned
func normalizeAssignedCompanyIDs(assigned []string) []string {
normalized := make([]string, 0, len(assigned))
seen := make(map[string]struct{}, len(assigned))
for _, id := range assigned {
clean := strings.TrimSpace(id)
if clean == "" {
continue
}
if _, ok := seen[clean]; ok {
continue
}
seen[clean] = struct{}{}
normalized = append(normalized, clean)
}
return normalized
}
// pickActiveCompanyID chooses the company context for a customer request.
// When requestedCompanyID is set it must be in assigned; otherwise a still-valid
// token company is kept; if the token company was removed, the first assignment is used.
func pickActiveCompanyID(assigned []string, tokenCompanyID, requestedCompanyID string) (string, error) {
assigned = normalizeAssignedCompanyIDs(assigned)
if len(assigned) == 0 {
return "", nil
}
requested := strings.TrimSpace(requestedCompanyID)
if requested != "" {
for _, id := range assigned {
if id == requested {
return requested, nil
}
}
return "", errCompanyNotAssigned
}
token := strings.TrimSpace(tokenCompanyID)
if token != "" {
for _, id := range assigned {
if id == token {
return token, nil
}
}
}
return assigned[0], nil
}
// PickAssignedCompanyID chooses the company for a company-scoped write (e.g. tender approval).
// When requestedCompanyID is set it must be in assigned; otherwise activeCompanyID is used.
func PickAssignedCompanyID(activeCompanyID string, assigned []string, requestedCompanyID string) (string, error) {
activeCompanyID = strings.TrimSpace(activeCompanyID)
requestedCompanyID = strings.TrimSpace(requestedCompanyID)
if requestedCompanyID != "" {
for _, id := range normalizeAssignedCompanyIDs(assigned) {
if id == requestedCompanyID {
return requestedCompanyID, nil
}
}
return "", ErrCompanyNotAssigned
}
if activeCompanyID == "" {
return "", errors.New("company ID is required")
}
return activeCompanyID, nil
}
// ResolveCompanyContext resolves the active company and assigned companies for an authenticated customer.
func (s *customerService) ResolveCompanyContext(ctx context.Context, customerID, tokenCompanyID, requestedCompanyID string) (string, []string, error) {
customer, err := s.repository.GetByID(ctx, customerID)
if err != nil {
return "", nil, errors.New("customer not found")
}
assigned := normalizeAssignedCompanyIDs(customer.Companies)
companyID, err := pickActiveCompanyID(assigned, tokenCompanyID, requestedCompanyID)
if err != nil {
return "", nil, err
}
return companyID, assigned, nil
}
// ResolveActiveCompanyID resolves the active company for an authenticated customer.
func (s *customerService) ResolveActiveCompanyID(ctx context.Context, customerID, tokenCompanyID, requestedCompanyID string) (string, error) {
companyID, _, err := s.ResolveCompanyContext(ctx, customerID, tokenCompanyID, requestedCompanyID)
return companyID, err
}
+77
View File
@@ -0,0 +1,77 @@
package customer
import "testing"
func TestPickActiveCompanyID(t *testing.T) {
assigned := []string{"company-a", "company-b"}
tests := []struct {
name string
assigned []string
token string
requested string
want string
wantErr bool
wantErrType error
}{
{
name: "uses requested company when assigned",
assigned: assigned,
token: "company-a",
requested: "company-b",
want: "company-b",
},
{
name: "rejects requested company that is not assigned",
assigned: assigned,
token: "company-a",
requested: "company-c",
wantErr: true,
wantErrType: errCompanyNotAssigned,
},
{
name: "keeps valid token company",
assigned: assigned,
token: "company-b",
want: "company-b",
},
{
name: "falls back to first assignment when token company was removed",
assigned: []string{"company-b"},
token: "company-a",
want: "company-b",
},
{
name: "returns empty when customer has no companies",
assigned: nil,
token: "company-a",
want: "",
},
{
name: "uses first assignment when token is empty",
assigned: assigned,
want: "company-a",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := pickActiveCompanyID(tt.assigned, tt.token, tt.requested)
if tt.wantErr {
if err == nil {
t.Fatalf("expected error, got nil")
}
if tt.wantErrType != nil && err != tt.wantErrType {
t.Fatalf("expected error %v, got %v", tt.wantErrType, err)
}
return
}
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != tt.want {
t.Fatalf("pickActiveCompanyID() = %q, want %q", got, tt.want)
}
})
}
}
+152
View File
@@ -0,0 +1,152 @@
package customer
import (
"encoding/json"
"fmt"
"strings"
)
type companySelectionObject struct {
ID string `json:"id"`
}
// decodeCompanySelection parses company IDs sent as strings, legacy company_ids,
// or company summary objects returned by GET customer.
func decodeCompanySelection(raw json.RawMessage) ([]string, error) {
if len(raw) == 0 {
return nil, fmt.Errorf("empty company selection")
}
if string(raw) == "null" {
return []string{}, nil
}
var ids []string
if err := json.Unmarshal(raw, &ids); err == nil {
return ids, nil
}
var objects []companySelectionObject
if err := json.Unmarshal(raw, &objects); err == nil {
result := make([]string, 0, len(objects))
for _, object := range objects {
if id := strings.TrimSpace(object.ID); id != "" {
result = append(result, id)
}
}
return result, nil
}
return nil, fmt.Errorf("companies must be an array of IDs or objects with id")
}
func mergeCompanySelectionFields(raw map[string]json.RawMessage) (json.RawMessage, bool, error) {
if value, ok := raw["companies"]; ok {
return value, true, nil
}
if value, ok := raw["company_ids"]; ok {
return value, true, nil
}
return nil, false, nil
}
func unmarshalFormWithoutCompanyFields(data []byte, raw map[string]json.RawMessage, target any) error {
delete(raw, "companies")
delete(raw, "company_ids")
trimmed, err := json.Marshal(raw)
if err != nil {
return err
}
return json.Unmarshal(trimmed, target)
}
// UnmarshalJSON accepts companies, legacy company_ids, and company summary objects.
func (f *CreateCustomerForm) UnmarshalJSON(data []byte) error {
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
selectionRaw, present, err := mergeCompanySelectionFields(raw)
if err != nil {
return err
}
type alias CreateCustomerForm
if err := unmarshalFormWithoutCompanyFields(data, raw, (*alias)(f)); err != nil {
return err
}
if !present {
return nil
}
companies, err := decodeCompanySelection(selectionRaw)
if err != nil {
return err
}
f.Companies = companies
return nil
}
// UnmarshalJSON accepts companies, legacy company_ids, and company summary objects.
func (f *UpdateCustomerForm) UnmarshalJSON(data []byte) error {
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
selectionRaw, present, err := mergeCompanySelectionFields(raw)
if err != nil {
return err
}
type alias UpdateCustomerForm
if err := unmarshalFormWithoutCompanyFields(data, raw, (*alias)(f)); err != nil {
return err
}
if !present {
return nil
}
companies, err := decodeCompanySelection(selectionRaw)
if err != nil {
return err
}
f.Companies = &companies
return nil
}
// UnmarshalJSON accepts companies, legacy company_ids, and company summary objects.
func (f *AssignCompaniesForm) UnmarshalJSON(data []byte) error {
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
selectionRaw, present, err := mergeCompanySelectionFields(raw)
if err != nil {
return err
}
type alias AssignCompaniesForm
if err := unmarshalFormWithoutCompanyFields(data, raw, (*alias)(f)); err != nil {
return err
}
if !present {
return nil
}
companies, err := decodeCompanySelection(selectionRaw)
if err != nil {
return err
}
f.Companies = companies
return nil
}
+92
View File
@@ -0,0 +1,92 @@
package customer
import (
"encoding/json"
"testing"
)
func TestCreateCustomerForm_UnmarshalJSON_CompanyIDs(t *testing.T) {
payload := `{
"type": "individual",
"role": "analyst",
"username": "testuser",
"email": "test@example.com",
"password": "Test!1234",
"company_ids": ["507f1f77bcf86cd799439011", "507f1f77bcf86cd799439012"]
}`
var form CreateCustomerForm
if err := json.Unmarshal([]byte(payload), &form); err != nil {
t.Fatalf("unexpected error: %v", err)
}
want := []string{"507f1f77bcf86cd799439011", "507f1f77bcf86cd799439012"}
if len(form.Companies) != len(want) {
t.Fatalf("Companies len = %d, want %d", len(form.Companies), len(want))
}
for i, id := range want {
if form.Companies[i] != id {
t.Fatalf("Companies[%d] = %q, want %q", i, form.Companies[i], id)
}
}
}
func TestUpdateCustomerForm_UnmarshalJSON_CompanySummaries(t *testing.T) {
payload := `{
"type": "individual",
"role": "analyst",
"username": "testuser",
"email": "test@example.com",
"companies": [
{"id": "507f1f77bcf86cd799439011", "name": "Acme"},
{"id": "507f1f77bcf86cd799439012", "name": "Globex"}
]
}`
var form UpdateCustomerForm
if err := json.Unmarshal([]byte(payload), &form); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if form.Companies == nil {
t.Fatal("Companies should be set")
}
want := []string{"507f1f77bcf86cd799439011", "507f1f77bcf86cd799439012"}
if len(*form.Companies) != len(want) {
t.Fatalf("Companies len = %d, want %d", len(*form.Companies), len(want))
}
for i, id := range want {
if (*form.Companies)[i] != id {
t.Fatalf("Companies[%d] = %q, want %q", i, (*form.Companies)[i], id)
}
}
}
func TestUpdateCustomerForm_UnmarshalJSON_OmittedCompanies(t *testing.T) {
payload := `{
"type": "individual",
"role": "analyst",
"username": "testuser",
"email": "test@example.com"
}`
var form UpdateCustomerForm
if err := json.Unmarshal([]byte(payload), &form); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if form.Companies != nil {
t.Fatalf("Companies = %v, want nil when omitted", form.Companies)
}
}
func TestAssignCompaniesForm_UnmarshalJSON_LegacyField(t *testing.T) {
payload := `{"company_ids": ["507f1f77bcf86cd799439011"]}`
var form AssignCompaniesForm
if err := json.Unmarshal([]byte(payload), &form); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(form.Companies) != 1 || form.Companies[0] != "507f1f77bcf86cd799439011" {
t.Fatalf("Companies = %v", form.Companies)
}
}
+38
View File
@@ -1,8 +1,10 @@
package customer
import (
"errors"
"net/http"
"strings"
"tm/pkg/audit"
"tm/pkg/response"
"github.com/labstack/echo/v4"
@@ -52,12 +54,48 @@ func (h *Handler) AuthMiddleware() echo.MiddlewareFunc {
c.Set("company_id", validationResult.Claims.CompanyID)
c.Set("access_token", tokenString)
audit.EnrichEchoContext(c, audit.ActorTypeCustomer)
// Continue to next handler
return next(c)
}
}
}
// CompanyContextMiddleware resolves company_id from current customer assignments.
// JWT company_id can be stale after admin reassigns companies; this keeps tender
// recommendations and other company-scoped APIs in sync with the database.
func (h *Handler) CompanyContextMiddleware() echo.MiddlewareFunc {
return func(next echo.HandlerFunc) echo.HandlerFunc {
return func(c echo.Context) error {
customerID, err := GetCustomerIDFromContext(c)
if err != nil {
return response.Unauthorized(c, "User not authenticated")
}
tokenCompanyID, _ := c.Get("company_id").(string)
requestedCompanyID := strings.TrimSpace(c.QueryParam("company_id"))
companyID, companyIDs, err := h.service.ResolveCompanyContext(
c.Request().Context(),
customerID,
tokenCompanyID,
requestedCompanyID,
)
if err != nil {
if errors.Is(err, errCompanyNotAssigned) {
return response.Forbidden(c, "Company is not assigned to customer")
}
return response.Unauthorized(c, "User not authenticated")
}
c.Set("company_id", companyID)
c.Set("company_ids", companyIDs)
return next(c)
}
}
}
// GetCustomerIDFromContext extracts customer ID from Echo context
func GetCustomerIDFromContext(c echo.Context) (string, error) {
customerID, ok := c.Get("customer_id").(string)
+6 -11
View File
@@ -249,16 +249,11 @@ func (r *customerRepository) GetByRole(ctx context.Context, role CustomerRole) (
// GetByCompanies retrieves customers by companies
func (r *customerRepository) GetByCompanies(ctx context.Context, companies []string) ([]Customer, error) {
companyIDs := make([]bson.ObjectID, 0)
for _, company := range companies {
id, err := bson.ObjectIDFromHex(company)
if err != nil {
return nil, err
}
companyIDs = append(companyIDs, id)
if len(companies) == 0 {
return []Customer{}, nil
}
customers, err := r.ormRepo.FindAll(ctx, bson.M{"companies": bson.M{"$in": companyIDs}}, orm.NewPaginationBuilder().Build())
customers, err := r.ormRepo.FindAll(ctx, bson.M{"companies": bson.M{"$in": companies}}, orm.NewPaginationBuilder().Build())
if err != nil {
return nil, err
}
@@ -274,10 +269,10 @@ func (r *customerRepository) GetByCompanyID(ctx context.Context, companyID strin
SortBy(pagination.SortBy, pagination.SortOrder).
Build()
// Filter by company ID and active status
// Filter by assigned company ID and active status
filter := bson.M{
"company_id": companyID,
"status": bson.M{"$ne": CustomerStatusInactive},
"companies": companyID,
"status": bson.M{"$ne": CustomerStatusInactive},
}
// Use ORM to find customers
+148 -24
View File
@@ -53,6 +53,13 @@ type Service interface {
// Role assignment operations
AssignRole(ctx context.Context, customerID string, form *AssignRoleForm) (*AssignRoleResponse, error)
AssignCompanies(ctx context.Context, customerID string, Companies []string) error
// ResolveCompanyContext returns the active company and all assigned companies
// for an authenticated customer request.
ResolveCompanyContext(ctx context.Context, customerID, tokenCompanyID, requestedCompanyID string) (string, []string, error)
// ResolveActiveCompanyID returns the company context for an authenticated customer.
ResolveActiveCompanyID(ctx context.Context, customerID, tokenCompanyID, requestedCompanyID string) (string, error)
}
// customerService implements the Service interface
@@ -181,6 +188,14 @@ func (s *customerService) Register(ctx context.Context, form *CreateCustomerForm
"company_ids": Companies,
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerCreate,
TargetType: audit.TargetTypeCustomer,
TargetID: customer.ID.Hex(),
Success: true,
Metadata: map[string]string{"role": string(customer.Role)},
})
go s.notification.SendEmail(context.Background(), notification.Model{
Recipient: customer.Email,
Title: "Welcome to Opplens",
@@ -198,6 +213,13 @@ func (s *customerService) GetByID(ctx context.Context, id string) (*CustomerResp
customer, err := s.repository.GetByID(ctx, id)
if err != nil {
if err.Error() == "customer not found" {
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerRead,
TargetType: audit.TargetTypeCustomer,
TargetID: id,
Success: false,
Metadata: map[string]string{"reason": "not_found"},
})
return nil, errors.New("customer not found")
}
s.logger.Error("Failed to get customer by ID", map[string]interface{}{
@@ -223,6 +245,13 @@ func (s *customerService) GetByID(ctx context.Context, id string) (*CustomerResp
"email": customer.Email,
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerRead,
TargetType: audit.TargetTypeCustomer,
TargetID: id,
Success: true,
})
return customer.ToResponse(companies), nil
}
@@ -249,12 +278,37 @@ func (s *customerService) Search(ctx context.Context, form *SearchCustomersForm,
customerResponses = append(customerResponses, customer.ToResponse(companies))
}
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerList,
Success: true,
Metadata: audit.MergeMetadata(
customerSearchFilterMetadata(form),
audit.PaginationMetadata(pagination.Limit, pagination.Offset, page.TotalCount),
),
})
return &CustomerListResponse{
Customers: customerResponses,
Meta: pagination.ListMeta(page.TotalCount, page.NextCursor, page.HasMore, page.PageOffset),
}, nil
}
func customerSearchFilterMetadata(form *SearchCustomersForm) map[string]string {
if form == nil {
return nil
}
return audit.MergeMetadata(
audit.FlagMetadata("has_search", form.Search != nil && *form.Search != ""),
audit.FlagMetadata("has_full_name", form.FullName != nil && *form.FullName != ""),
audit.FlagMetadata("has_email_filter", form.Email != nil && *form.Email != ""),
audit.OptionalStringMetadata("status", form.Status),
audit.OptionalStringMetadata("role", form.Role),
audit.OptionalStringMetadata("type", form.Type),
audit.OptionalStringMetadata("company_id", form.CompanyID),
)
}
func (s *customerService) SearchNotification(ctx context.Context, form *SearchCustomersForm, pagination *response.Pagination) ([]*CustomerNotificationResponse, error) {
page, err := s.repository.Search(ctx, form, pagination)
if err != nil {
@@ -452,6 +506,13 @@ func (s *customerService) Update(ctx context.Context, id string, form *UpdateCus
"email": customer.Email,
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerUpdate,
TargetType: audit.TargetTypeCustomer,
TargetID: id,
Success: true,
})
responseCompanies, err := s.loadCompaniesForCustomer(ctx, customer)
if err != nil {
s.logger.Error("Failed to load companies for customer response", map[string]interface{}{
@@ -490,6 +551,13 @@ func (s *customerService) Delete(ctx context.Context, id string) error {
"customer_id": id,
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerDelete,
TargetType: audit.TargetTypeCustomer,
TargetID: id,
Success: true,
})
return nil
}
@@ -518,6 +586,14 @@ func (s *customerService) UpdateStatus(ctx context.Context, id string, form *Upd
"status": form.Status,
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionCustomerStatusChange,
TargetType: audit.TargetTypeCustomer,
TargetID: id,
Success: true,
Metadata: map[string]string{"status": form.Status},
})
go s.notification.SendEmail(context.Background(), notification.Model{
Recipient: customer.Email,
Title: "Account Status Updated",
@@ -682,6 +758,12 @@ func (s *customerService) Login(ctx context.Context, form *LoginForm) (*AuthResp
s.logger.Warn("Customer login failed - username not found", map[string]interface{}{
"username": strings.ToLower(form.Username),
})
s.auditLogger.Log(ctx, audit.Entry{
Action: audit.ActionAuthLoginFailure,
ActorType: audit.ActorTypeCustomer,
TargetType: audit.TargetTypeCustomer,
Success: false,
})
return nil, errors.New("invalid credentials")
}
@@ -692,6 +774,15 @@ func (s *customerService) Login(ctx context.Context, form *LoginForm) (*AuthResp
"customer_id": customer.ID,
"status": string(customer.Status),
})
s.auditLogger.Log(ctx, audit.Entry{
ActorID: customer.ID.Hex(),
ActorType: audit.ActorTypeCustomer,
TargetType: audit.TargetTypeCustomer,
TargetID: customer.ID.Hex(),
Action: audit.ActionAuthLoginFailure,
Success: false,
Metadata: map[string]string{"reason": "inactive"},
})
return nil, errors.New("account is not active")
}
@@ -702,13 +793,20 @@ func (s *customerService) Login(ctx context.Context, form *LoginForm) (*AuthResp
"customer_id": customer.ID,
"email": customer.Email,
})
s.auditLogger.Log(ctx, audit.Entry{
ActorID: customer.ID.Hex(),
ActorType: audit.ActorTypeCustomer,
TargetType: audit.TargetTypeCustomer,
TargetID: customer.ID.Hex(),
Action: audit.ActionAuthLoginFailure,
Success: false,
})
return nil, errors.New("invalid credentials")
}
// Generate JWT tokens using authorization service
companyID := ""
if len(customer.Companies) > 0 {
companyID = customer.Companies[0] // Use first company ID for JWT token
companyID, err := pickActiveCompanyID(customer.Companies, "", "")
if err != nil {
return nil, errors.New("failed to resolve customer company")
}
tokenPair, err := s.authService.GenerateTokenPair(
@@ -747,6 +845,16 @@ func (s *customerService) Login(ctx context.Context, form *LoginForm) (*AuthResp
"customer_type": string(customer.Type),
})
s.auditLogger.Log(ctx, audit.Entry{
ActorID: customer.ID.Hex(),
ActorType: audit.ActorTypeCustomer,
TargetType: audit.TargetTypeCustomer,
TargetID: customer.ID.Hex(),
Action: audit.ActionAuthLoginSuccess,
Success: true,
Metadata: map[string]string{"role": string(customer.Role)},
})
return &AuthResponse{
Customer: customer.ToResponse(companies),
AccessToken: tokenPair.AccessToken,
@@ -761,32 +869,20 @@ func (s *customerService) RefreshToken(ctx context.Context, refreshToken string)
"refresh_token": refreshToken[:10] + "...", // Log only first 10 chars for security
})
// Validate refresh token and generate new access token
newAccessToken, err := s.authService.RefreshAccessToken(refreshToken)
validationResult, err := s.authService.ValidateRefreshToken(refreshToken)
if err != nil {
s.logger.Warn("Failed to refresh access token", map[string]interface{}{
s.logger.Warn("Failed to validate refresh token", map[string]interface{}{
"error": err.Error(),
})
return nil, errors.New("invalid or expired refresh token")
}
// Parse the new access token to get customer information
validationResult, err := s.authService.ValidateAccessToken(newAccessToken)
if err != nil {
s.logger.Error("Failed to validate new access token", map[string]interface{}{
"error": err.Error(),
})
return nil, errors.New("failed to generate valid access token")
}
if !validationResult.Valid {
s.logger.Error("New access token validation failed", map[string]interface{}{
s.logger.Warn("Invalid refresh token provided", map[string]interface{}{
"error": validationResult.Error,
})
return nil, errors.New("failed to generate valid access token")
return nil, errors.New("invalid or expired refresh token")
}
// Get customer information
customerID, err := bson.ObjectIDFromHex(validationResult.Claims.UserID)
if err != nil {
s.logger.Error("Failed to parse customer ID from token", map[string]interface{}{
@@ -804,11 +900,29 @@ func (s *customerService) RefreshToken(ctx context.Context, refreshToken string)
return nil, errors.New("customer not found")
}
// Check if customer is still active
if customer.Status != CustomerStatusActive {
return nil, errors.New("customer account is inactive")
}
companyID, err := pickActiveCompanyID(customer.Companies, validationResult.Claims.CompanyID, "")
if err != nil {
return nil, errors.New("failed to resolve customer company")
}
tokenPair, err := s.authService.GenerateTokenPair(
customer.ID.Hex(),
customer.Email,
string(customer.Role),
companyID,
)
if err != nil {
s.logger.Error("Failed to generate JWT tokens during refresh", map[string]interface{}{
"error": err.Error(),
"customer_id": customer.ID,
})
return nil, errors.New("failed to generate authentication tokens")
}
companies, err := s.loadCompaniesForCustomer(ctx, customer)
if err != nil {
s.logger.Error("Failed to load companies for customer", map[string]interface{}{
@@ -820,13 +934,14 @@ func (s *customerService) RefreshToken(ctx context.Context, refreshToken string)
s.logger.Info("Access token refreshed successfully", map[string]interface{}{
"customer_id": customer.ID,
"customer_type": string(customer.Type),
"company_id": companyID,
})
return &AuthResponse{
Customer: customer.ToResponse(companies),
AccessToken: newAccessToken,
RefreshToken: refreshToken, // Return the same refresh token
ExpiresAt: time.Now().Add(15 * time.Minute).Unix(), // 15 minutes from now
AccessToken: tokenPair.AccessToken,
RefreshToken: tokenPair.RefreshToken,
ExpiresAt: time.Now().Add(15 * time.Minute).Unix(),
}, nil
}
@@ -942,6 +1057,15 @@ func (s *customerService) Logout(ctx context.Context, customerID string, accessT
"customer_id": customerID,
})
s.auditLogger.Log(ctx, audit.Entry{
ActorID: customerID,
ActorType: audit.ActorTypeCustomer,
TargetType: audit.TargetTypeCustomer,
TargetID: customerID,
Action: audit.ActionAuthLogout,
Success: true,
})
return nil
}
+13 -11
View File
@@ -2,15 +2,17 @@ package dashboard
// SummaryResponse powers hero pills and stat cards.
type SummaryResponse struct {
TotalTenders int64 `json:"total_tenders"`
ActiveTenders int64 `json:"active_tenders"`
AwardedTenders int64 `json:"awarded_tenders"`
ExpiredTenders int64 `json:"expired_tenders"`
CancelledTenders int64 `json:"cancelled_tenders"`
ClosingSoon int64 `json:"closing_soon"`
TotalEstimatedValue float64 `json:"total_estimated_value"`
ValueCurrency string `json:"value_currency"`
GeneratedAt int64 `json:"generated_at"`
TotalTenders int64 `json:"total_tenders"`
ActiveTenders int64 `json:"active_tenders"`
AwardedTenders int64 `json:"awarded_tenders"`
ExpiredTenders int64 `json:"expired_tenders"`
CancelledTenders int64 `json:"cancelled_tenders"`
ClosingSoon int64 `json:"closing_soon"`
TotalEstimatedValue float64 `json:"total_estimated_value"`
ValueCurrency string `json:"value_currency"`
Days int `json:"days"`
ScrapedTED []TrendPoint `json:"scraped_ted"`
GeneratedAt int64 `json:"generated_at"`
}
// TrendResponse powers the tender flow chart.
@@ -96,8 +98,7 @@ type StatisticsReportResponse struct {
// StatisticsDailySeries contains per-day chart series.
type StatisticsDailySeries struct {
ScrapedTED []TrendPoint `json:"scraped_ted"`
ScrapedDocuments []TrendPoint `json:"scraped_documents"`
ScrapedDocuments []TrendPoint `json:"scraped_documents"`
TranslatedNotices []TrendPoint `json:"translated_notices"`
}
@@ -105,4 +106,5 @@ type StatisticsDailySeries struct {
type StatisticsLifetimeTotals struct {
ScrapedDocuments int64 `json:"scraped_documents"`
TranslatedTenders int64 `json:"translated_tenders"`
ScrapedTEDNotices int64 `json:"scraped_ted_notices"`
}
+1
View File
@@ -3,6 +3,7 @@ package dashboard
// SummaryQuery binds query params for GET /dashboard/summary.
type SummaryQuery struct {
ClosingWindow int `query:"closing_window"`
Days int `query:"days"`
}
// TrendQuery binds query params for GET /dashboard/trend.
+6 -4
View File
@@ -19,10 +19,11 @@ func NewHandler(service Service) *Handler {
// Summary returns top-level dashboard counters.
// @Summary Dashboard summary
// @Description Top-level counters for hero pills and stat cards
// @Description Top-level counters for hero pills and stat cards, including daily TED scrape counts
// @Tags Admin-Dashboard
// @Produce json
// @Param closing_window query int false "Hours considered closing soon (default 168)"
// @Param days query int false "Days back for scraped TED daily series (default 14, max 90)"
// @Success 200 {object} response.APIResponse{data=SummaryResponse}
// @Failure 500 {object} response.APIResponse
// @Router /admin/v1/dashboard/summary [get]
@@ -30,6 +31,7 @@ func NewHandler(service Service) *Handler {
func (h *Handler) Summary(c echo.Context) error {
query := SummaryQuery{
ClosingWindow: parseIntQuery(c, "closing_window", 0),
Days: parseIntQuery(c, "days", 0),
}
out, err := h.service.Summary(c.Request().Context(), query)
@@ -37,7 +39,7 @@ func (h *Handler) Summary(c echo.Context) error {
return response.InternalServerError(c, "Failed to load dashboard summary")
}
setPrivateCache(c, 60)
setPrivateCache(c, 300)
return response.Success(c, out, "Dashboard summary retrieved successfully")
}
@@ -163,7 +165,7 @@ func (h *Handler) Recent(c echo.Context) error {
// Statistics returns scraping and translation statistics for charts.
// @Summary Dashboard statistics report
// @Description Daily and lifetime statistics for TED scraping, document scraping, and AI translation
// @Description Daily and lifetime statistics for document scraping and AI translation
// @Tags Admin-Dashboard
// @Produce json
// @Param days query int false "Days back for daily series (default 14, max 90)"
@@ -181,7 +183,7 @@ func (h *Handler) Statistics(c echo.Context) error {
return response.InternalServerError(c, "Failed to load dashboard statistics")
}
setPrivateCache(c, 60)
setPrivateCache(c, 300)
return response.Success(c, out, "Dashboard statistics retrieved successfully")
}
+276 -105
View File
@@ -4,20 +4,39 @@ import (
"context"
"fmt"
"strings"
"sync"
"time"
"tm/internal/tender"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
orm "tm/pkg/mongo"
"go.mongodb.org/mongo-driver/v2/bson"
"go.mongodb.org/mongo-driver/v2/mongo"
"golang.org/x/sync/singleflight"
)
// ProcedureDocumentsLister lists contract folders that have scraped documents in MinIO.
// This is the source of truth for document-scrape statistics, matching tender panel search.
type ProcedureDocumentsLister interface {
ListProceduresWithDocuments(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error)
}
// scrapedDocumentsScanner optionally scans MinIO for per-day document counts in the same pass.
type scrapedDocumentsScanner interface {
ScanScrapedDocuments(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, map[string]int64, error)
}
// translatedNoticesScanner optionally scans MinIO for per-day translated notice counts.
type translatedNoticesScanner interface {
ScanTranslatedNotices(ctx context.Context) (map[string]int64, int64, error)
}
const defaultValueCurrency = "EUR"
// Repository defines dashboard data access.
type Repository interface {
Summary(ctx context.Context, closingWindowSec int64) (*SummaryResponse, error)
Summary(ctx context.Context, closingWindowSec int64, days int) (*SummaryResponse, error)
Trend(ctx context.Context, days int, metric string, startUnix int64) (map[string]int64, error)
Countries(ctx context.Context, limit int) (*CountriesResponse, error)
NoticeTypes(ctx context.Context) (*NoticeTypesResponse, error)
@@ -27,19 +46,31 @@ type Repository interface {
}
type repository struct {
ormRepo orm.Repository[tender.Tender]
mongoManager *orm.ConnectionManager
metricsCounter *orm.Counter
logger logger.Logger
ormRepo orm.Repository[tender.Tender]
mongoManager *orm.ConnectionManager
metricsCounter *orm.Counter
procedureLister ProcedureDocumentsLister
logger logger.Logger
scrapedScopeMu sync.Mutex
scrapedScope scrapedTendersScope
scrapedScopeExpiry time.Time
scrapedScopeCached bool
scrapedScopeGroup singleflight.Group
translatedScopeMu sync.Mutex
translatedScope translatedNoticesScope
translatedScopeExpiry time.Time
translatedScopeCached bool
translatedScopeGroup singleflight.Group
}
// NewRepository creates a dashboard repository backed by the tenders collection.
func NewRepository(mongoManager *orm.ConnectionManager, log logger.Logger) Repository {
func NewRepository(mongoManager *orm.ConnectionManager, log logger.Logger, procedureLister ProcedureDocumentsLister) Repository {
return &repository{
ormRepo: orm.NewRepository[tender.Tender](mongoManager.GetCollection(tenderCollectionName()), log),
mongoManager: mongoManager,
metricsCounter: orm.NewCounter(mongoManager),
logger: log,
ormRepo: orm.NewRepository[tender.Tender](mongoManager.GetCollection(tenderCollectionName()), log),
mongoManager: mongoManager,
metricsCounter: orm.NewCounter(mongoManager),
procedureLister: procedureLister,
logger: log,
}
}
@@ -47,110 +78,162 @@ func tenderCollectionName() string {
return "tenders"
}
func (r *repository) Summary(ctx context.Context, closingWindowSec int64) (*SummaryResponse, error) {
func (r *repository) Summary(ctx context.Context, closingWindowSec int64, days int) (*SummaryResponse, error) {
now := time.Now().Unix()
windowEnd := now + closingWindowSec
total, err := r.ormRepo.Count(ctx, bson.M{})
if err != nil {
return nil, fmt.Errorf("count total tenders: %w", err)
nowUTC := time.Now().UTC()
endDay := time.Date(nowUTC.Year(), nowUTC.Month(), nowUTC.Day(), 0, 0, 0, 0, time.UTC)
startDay := endDay.AddDate(0, 0, -(days - 1))
var (
statusCounts summaryStatusCounts
valueCurrency string
totalValue float64
scrapedTED map[string]int64
wg sync.WaitGroup
mu sync.Mutex
firstErr error
)
recordErr := func(section string, err error) {
mu.Lock()
defer mu.Unlock()
if firstErr == nil {
firstErr = fmt.Errorf("%s: %w", section, err)
}
}
active, err := r.ormRepo.Count(ctx, bson.M{"status": tender.TenderStatusActive})
if err != nil {
return nil, fmt.Errorf("count active tenders: %w", err)
}
wg.Add(3)
go func() {
defer wg.Done()
counts, err := r.summaryStatusCounts(ctx, now, windowEnd)
if err != nil {
recordErr("status_counts", err)
return
}
statusCounts = counts
}()
awarded, err := r.ormRepo.Count(ctx, bson.M{"status": tender.TenderStatusAwarded})
if err != nil {
return nil, fmt.Errorf("count awarded tenders: %w", err)
}
go func() {
defer wg.Done()
currency, value, err := r.summaryCurrencyValue(ctx)
if err != nil {
recordErr("currency_value", err)
return
}
valueCurrency, totalValue = currency, value
}()
expired, err := r.countExpired(ctx, now)
if err != nil {
return nil, err
}
go func() {
defer wg.Done()
qctx, cancel := context.WithTimeout(ctx, statisticsMongoQueryTimeout)
defer cancel()
cancelled, err := r.ormRepo.Count(ctx, bson.M{"status": tender.TenderStatusCancelled})
if err != nil {
return nil, fmt.Errorf("count cancelled tenders: %w", err)
}
counts, err := r.metricsCounter.GetTEDNoticeScrapedDailyCounts(qctx, startDay, endDay)
if err != nil {
if r.logger != nil {
r.logger.Warn("Dashboard summary scraped TED daily counts failed", map[string]interface{}{
"error": err.Error(),
})
}
scrapedTED = map[string]int64{}
return
}
scrapedTED = counts
}()
closingSoon, err := r.countClosingSoon(ctx, now, windowEnd)
if err != nil {
return nil, err
}
valueCurrency, totalValue, err := r.dominantCurrencyValue(ctx)
if err != nil {
return nil, err
wg.Wait()
if firstErr != nil {
return nil, fmt.Errorf("dashboard summary aggregation: %w", firstErr)
}
return &SummaryResponse{
TotalTenders: total,
ActiveTenders: active,
AwardedTenders: awarded,
ExpiredTenders: expired,
CancelledTenders: cancelled,
ClosingSoon: closingSoon,
TotalTenders: statusCounts.total,
ActiveTenders: statusCounts.active,
AwardedTenders: statusCounts.awarded,
ExpiredTenders: statusCounts.expired,
CancelledTenders: statusCounts.cancelled,
ClosingSoon: statusCounts.closingSoon,
TotalEstimatedValue: totalValue,
ValueCurrency: valueCurrency,
Days: days,
ScrapedTED: fillTrendSeries(days, scrapedTED),
GeneratedAt: now,
}, nil
}
func (r *repository) countExpired(ctx context.Context, now int64) (int64, error) {
type summaryStatusCounts struct {
total int64
active int64
awarded int64
expired int64
cancelled int64
closingSoon int64
}
func (r *repository) summaryStatusCounts(ctx context.Context, now, windowEnd int64) (summaryStatusCounts, error) {
pipeline := mongo.Pipeline{
{{Key: "$addFields", Value: bson.M{"expiry_deadline": expiryDeadlineExpr()}}},
{{Key: "$match", Value: bson.M{
"status": bson.M{"$nin": []tender.TenderStatus{
tender.TenderStatusAwarded,
tender.TenderStatusCancelled,
}},
"$or": []bson.M{
{"status": tender.TenderStatusExpired},
{
"expiry_deadline": bson.M{"$gt": 0, "$lte": now},
},
},
{{Key: "$addFields", Value: bson.M{
"expiry_deadline": expiryDeadlineExpr(),
"effective_deadline": effectiveDeadlineExpr(),
}}},
{{Key: "$group", Value: bson.M{
"_id": nil,
"total": bson.M{"$sum": 1},
"active": bson.M{"$sum": bson.M{"$cond": bson.A{
bson.M{"$eq": bson.A{"$status", tender.TenderStatusActive}}, 1, 0,
}}},
"awarded": bson.M{"$sum": bson.M{"$cond": bson.A{
bson.M{"$eq": bson.A{"$status", tender.TenderStatusAwarded}}, 1, 0,
}}},
"cancelled": bson.M{"$sum": bson.M{"$cond": bson.A{
bson.M{"$eq": bson.A{"$status", tender.TenderStatusCancelled}}, 1, 0,
}}},
"expired": bson.M{"$sum": bson.M{"$cond": bson.A{
bson.M{"$and": bson.A{
bson.M{"$not": bson.M{"$in": bson.A{
"$status",
bson.A{tender.TenderStatusAwarded, tender.TenderStatusCancelled},
}}},
bson.M{"$or": bson.A{
bson.M{"$eq": bson.A{"$status", tender.TenderStatusExpired}},
bson.M{"$and": bson.A{
bson.M{"$gt": bson.A{"$expiry_deadline", 0}},
bson.M{"$lte": bson.A{"$expiry_deadline", now}},
}},
}},
}}, 1, 0,
}}},
"closing_soon": bson.M{"$sum": bson.M{"$cond": bson.A{
bson.M{"$and": bson.A{
bson.M{"$gt": bson.A{"$effective_deadline", now}},
bson.M{"$lte": bson.A{"$effective_deadline", windowEnd}},
}}, 1, 0,
}}},
}}},
{{Key: "$count", Value: "count"}},
}
results, err := r.ormRepo.Aggregate(ctx, pipeline)
if err != nil {
return 0, fmt.Errorf("count expired tenders: %w", err)
return summaryStatusCounts{}, err
}
if len(results) == 0 {
return 0, nil
return summaryStatusCounts{}, nil
}
return aggregateCount(results[0], "count"), nil
row := results[0]
return summaryStatusCounts{
total: aggregateCount(row, "total"),
active: aggregateCount(row, "active"),
awarded: aggregateCount(row, "awarded"),
expired: aggregateCount(row, "expired"),
cancelled: aggregateCount(row, "cancelled"),
closingSoon: aggregateCount(row, "closing_soon"),
}, nil
}
func (r *repository) countClosingSoon(ctx context.Context, now, windowEnd int64) (int64, error) {
pipeline := mongo.Pipeline{
{{Key: "$addFields", Value: bson.M{"effective_deadline": effectiveDeadlineExpr()}}},
{{Key: "$match", Value: bson.M{
"effective_deadline": bson.M{"$gt": now, "$lte": windowEnd},
}}},
{{Key: "$count", Value: "count"}},
}
results, err := r.ormRepo.Aggregate(ctx, pipeline)
if err != nil {
return 0, fmt.Errorf("count closing soon: %w", err)
}
if len(results) == 0 {
return 0, nil
}
return aggregateCount(results[0], "count"), nil
}
func (r *repository) dominantCurrencyValue(ctx context.Context) (string, float64, error) {
func (r *repository) summaryCurrencyValue(ctx context.Context) (string, float64, error) {
pipeline := mongo.Pipeline{
{{Key: "$match", Value: bson.M{
"estimated_value": bson.M{"$gt": 0},
@@ -167,19 +250,71 @@ func (r *repository) dominantCurrencyValue(ctx context.Context) (string, float64
results, err := r.ormRepo.Aggregate(ctx, pipeline)
if err != nil {
return defaultValueCurrency, 0, fmt.Errorf("dominant currency value: %w", err)
return defaultValueCurrency, 0, err
}
if len(results) == 0 {
return defaultValueCurrency, 0, nil
}
currency := defaultValueCurrency
if c, ok := results[0]["_id"].(string); ok && c != "" {
currency = strings.ToUpper(c)
currency, total := facetCurrencyValue(bson.M{"currency_value": bson.A{results[0]}})
return currency, total, nil
}
func facetCount(facet bson.M, key string) int64 {
rows := facetRows(facet, key)
if len(rows) == 0 {
return 0
}
return aggregateCount(asBSONMap(rows[0]), "count")
}
func facetCurrencyValue(facet bson.M) (string, float64) {
rows := facetRows(facet, "currency_value")
if len(rows) == 0 {
return defaultValueCurrency, 0
}
row := asBSONMap(rows[0])
if row == nil {
return defaultValueCurrency, 0
}
return currency, toFloat64(results[0]["total"]), nil
currency := defaultValueCurrency
if c, ok := row["_id"].(string); ok && c != "" {
currency = strings.ToUpper(c)
}
return currency, toFloat64(row["total"])
}
func facetRows(facet bson.M, key string) []interface{} {
raw, ok := facet[key]
if !ok {
return nil
}
switch rows := raw.(type) {
case bson.A:
return []interface{}(rows)
case []interface{}:
return rows
default:
return nil
}
}
func asBSONMap(v interface{}) bson.M {
switch doc := v.(type) {
case bson.M:
return doc
case bson.D:
out := make(bson.M, len(doc))
for _, elem := range doc {
out[elem.Key] = elem.Value
}
return out
case map[string]interface{}:
return doc
default:
return nil
}
}
func (r *repository) Trend(ctx context.Context, days int, metric string, startUnix int64) (map[string]int64, error) {
@@ -306,6 +441,22 @@ func (r *repository) ClosingSoon(ctx context.Context, limit int, windowSec int64
windowEnd := now + windowSec
pipeline := mongo.Pipeline{
{{Key: "$match", Value: bson.M{
"$or": bson.A{
bson.M{
"$and": bson.A{
bson.M{"submission_deadline": bson.M{"$gt": 0}},
deadlineWithinWindowMatch("submission_deadline", now, windowEnd),
},
},
bson.M{
"$and": bson.A{
noSubmissionDeadlineClause(),
deadlineWithinWindowMatch("tender_deadline", now, windowEnd),
},
},
},
}}},
{{Key: "$addFields", Value: bson.M{"effective_deadline": effectiveDeadlineExpr()}}},
{{Key: "$match", Value: bson.M{
"effective_deadline": bson.M{"$gt": now, "$lte": windowEnd},
@@ -313,16 +464,16 @@ func (r *repository) ClosingSoon(ctx context.Context, limit int, windowSec int64
{{Key: "$sort", Value: bson.M{"effective_deadline": 1}}},
{{Key: "$limit", Value: limit}},
{{Key: "$project", Value: bson.M{
"_id": 1,
"title": 1,
"country_code": 1,
"buyer_organization": 1,
"submission_deadline": 1,
"tender_deadline": 1,
"status": 1,
"estimated_value": 1,
"currency": 1,
"effective_deadline": 1,
"_id": 1,
"title": 1,
"country_code": 1,
"buyer_organization": 1,
"submission_deadline": 1,
"tender_deadline": 1,
"status": 1,
"estimated_value": 1,
"currency": 1,
"effective_deadline": 1,
}}},
}
@@ -351,12 +502,12 @@ func (r *repository) Recent(ctx context.Context, limit int, cursor string) ([]Re
filter,
orm.ListPaginationOptions{
Projection: bson.M{
"title": 1,
"country_code": 1,
"status": 1,
"title": 1,
"country_code": 1,
"status": 1,
"buyer_organization": 1,
"created_at": 1,
"publication_date": 1,
"created_at": 1,
"publication_date": 1,
},
},
)
@@ -445,6 +596,26 @@ func trendTimestampField(metric string) string {
}
}
// deadlineWithinWindowMatch matches raw deadline fields stored as Unix seconds or
// milliseconds. normalizeTimestampExpr treats values > 1e12 as ms; indexed pre-filters
// must accept both encodings to stay equivalent to filtering on normalized deadlines.
func deadlineWithinWindowMatch(field string, now, windowEnd int64) bson.M {
return bson.M{
"$or": bson.A{
bson.M{field: bson.M{"$gt": now, "$lte": windowEnd}},
bson.M{field: bson.M{"$gt": now * 1000, "$lte": windowEnd * 1000}},
},
}
}
func noSubmissionDeadlineClause() bson.M {
return bson.M{"$or": bson.A{
bson.M{"submission_deadline": bson.M{"$exists": false}},
bson.M{"submission_deadline": nil},
bson.M{"submission_deadline": bson.M{"$lte": 0}},
}}
}
// effectiveDeadlineExpr is used for closing-soon (submission first, per dashboard-api.md).
func effectiveDeadlineExpr() bson.M {
return normalizeDeadlineFieldExpr(
+71
View File
@@ -0,0 +1,71 @@
package dashboard
import (
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
)
func TestFacetCountDecodesBSOND(t *testing.T) {
facet := bson.M{
"total": bson.A{
bson.D{{Key: "count", Value: int32(42)}},
},
"active": bson.A{
bson.D{{Key: "count", Value: int64(7)}},
},
}
if got := facetCount(facet, "total"); got != 42 {
t.Fatalf("expected total 42, got %d", got)
}
if got := facetCount(facet, "active"); got != 7 {
t.Fatalf("expected active 7, got %d", got)
}
}
func TestFacetCurrencyValueDecodesBSOND(t *testing.T) {
facet := bson.M{
"currency_value": bson.A{
bson.D{
{Key: "_id", Value: "eur"},
{Key: "total", Value: float64(12345.67)},
},
},
}
currency, total := facetCurrencyValue(facet)
if currency != "EUR" {
t.Fatalf("expected EUR, got %q", currency)
}
if total != 12345.67 {
t.Fatalf("expected 12345.67, got %f", total)
}
}
func TestDeadlineWithinWindowMatchCoversSecondsAndMilliseconds(t *testing.T) {
now := int64(1_700_000_000)
windowEnd := now + 86_400
match := deadlineWithinWindowMatch("submission_deadline", now, windowEnd)
orClause, ok := match["$or"].(bson.A)
if !ok || len(orClause) != 2 {
t.Fatalf("expected two range branches, got %#v", match)
}
secRange, ok := orClause[0].(bson.M)["submission_deadline"].(bson.M)
if !ok {
t.Fatalf("expected seconds range on submission_deadline, got %#v", orClause[0])
}
if secRange["$gt"] != now || secRange["$lte"] != windowEnd {
t.Fatalf("unexpected seconds range: %#v", secRange)
}
msRange, ok := orClause[1].(bson.M)["submission_deadline"].(bson.M)
if !ok {
t.Fatalf("expected milliseconds range on submission_deadline, got %#v", orClause[1])
}
if msRange["$gt"] != now*1000 || msRange["$lte"] != windowEnd*1000 {
t.Fatalf("unexpected milliseconds range: %#v", msRange)
}
}
+598 -104
View File
@@ -2,11 +2,17 @@ package dashboard
import (
"context"
"encoding/json"
"fmt"
"strconv"
"strings"
"sync"
"time"
"tm/pkg/logger"
"tm/pkg/redis"
goredis "github.com/redis/go-redis/v9"
"golang.org/x/sync/singleflight"
)
const (
@@ -16,7 +22,15 @@ const (
defaultCountriesLimit = 6
defaultListLimit = 5
maxListLimit = 20
statisticsCacheTTL = 60 * time.Second
summaryCacheTTL = 5 * time.Minute
summaryStaleGraceTTL = 15 * time.Minute
statisticsCacheTTL = 5 * time.Minute
statisticsStaleGraceTTL = 15 * time.Minute
// statisticsColdLoadWait bounds how long a cold-cache request waits for a fresh
// load before falling back to a placeholder. Mongo-only loads (MinIO scope
// already cached) comfortably finish within this window; a first-ever MinIO
// bucket scan does not, so we don't block the request for that.
statisticsColdLoadWait = 4 * time.Second
)
// Service defines dashboard business operations.
@@ -30,145 +44,230 @@ type Service interface {
Statistics(ctx context.Context, query StatisticsQuery) (*StatisticsReportResponse, error)
}
type statisticsCacheEntry struct {
expiresAt time.Time
value *StatisticsReportResponse
type cacheEntry[T any] struct {
expiresAt time.Time
staleUntil time.Time
value T
}
type service struct {
repo Repository
logger logger.Logger
statisticsMu sync.Mutex
statistics map[int]statisticsCacheEntry
repo Repository
logger logger.Logger
redis redis.Client
summaryMu sync.Mutex
summary map[string]cacheEntry[*SummaryResponse]
summaryGroup singleflight.Group
statisticsMu sync.Mutex
statistics map[int]cacheEntry[*StatisticsReportResponse]
statisticsGroup singleflight.Group
trendCache *widgetCache[*TrendResponse]
countriesCache *widgetCache[*CountriesResponse]
noticeTypesCache *widgetCache[*NoticeTypesResponse]
closingSoonCache *widgetCache[*ClosingSoonResponse]
recentCache *widgetCache[*RecentResponse]
}
// NewService creates a dashboard service.
func NewService(repo Repository, log logger.Logger) Service {
return &service{
repo: repo,
logger: log,
statistics: make(map[int]statisticsCacheEntry),
func NewService(repo Repository, log logger.Logger, redisClient redis.Client) Service {
s := &service{
repo: repo,
logger: log,
redis: redisClient,
summary: make(map[string]cacheEntry[*SummaryResponse]),
statistics: make(map[int]cacheEntry[*StatisticsReportResponse]),
trendCache: newWidgetCache[*TrendResponse](redisClient, log, "trend"),
countriesCache: newWidgetCache[*CountriesResponse](redisClient, log, "countries"),
noticeTypesCache: newWidgetCache[*NoticeTypesResponse](redisClient, log, "notice-types"),
closingSoonCache: newWidgetCache[*ClosingSoonResponse](redisClient, log, "closing-soon"),
recentCache: newWidgetCache[*RecentResponse](redisClient, log, "recent"),
}
go s.warmStatisticsCache()
go s.warmSummaryCache()
go s.warmWidgetCaches()
return s
}
func (s *service) Summary(ctx context.Context, query SummaryQuery) (*SummaryResponse, error) {
windowHours := closingWindowHours(query.ClosingWindow)
windowSec := int64(windowHours) * 3600
days := trendDays(query.Days)
cacheKey := summaryCacheKey(windowSec, days)
s.logger.Info("Fetching dashboard summary", map[string]interface{}{
"closing_window_hours": windowHours,
})
out, err := s.repo.Summary(ctx, windowSec)
if err != nil {
s.logger.Error("Failed to fetch dashboard summary", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard summary: %w", err)
if cached, ok := s.cachedSummary(cacheKey); ok {
return cached, nil
}
return out, nil
if stale, ok := s.staleSummary(cacheKey); ok {
go s.refreshSummary(windowSec, days)
return stale, nil
}
if redisCached, ok := s.getRedisSummary(ctx, cacheKey); ok {
s.storeSummary(cacheKey, redisCached)
go s.refreshSummary(windowSec, days)
return redisCached, nil
}
out, err, _ := s.summaryGroup.Do(cacheKey, func() (interface{}, error) {
if cached, ok := s.cachedSummary(cacheKey); ok {
return cached, nil
}
s.logger.Info("Fetching dashboard summary", map[string]interface{}{
"closing_window_hours": windowHours,
"days": days,
})
result, err := s.repo.Summary(context.WithoutCancel(ctx), windowSec, days)
if err != nil {
s.logger.Error("Failed to fetch dashboard summary", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard summary: %w", err)
}
s.storeSummary(cacheKey, result)
s.storeRedisSummary(context.Background(), cacheKey, result)
return result, nil
})
if err != nil {
return nil, err
}
return out.(*SummaryResponse), nil
}
func (s *service) Trend(ctx context.Context, query TrendQuery) (*TrendResponse, error) {
days := trendDays(query.Days)
metric := normalizeTrendMetric(query.Metric)
startUnix := trendStartUnix(days)
cacheKey := fmt.Sprintf("%d:%s", days, metric)
s.logger.Info("Fetching dashboard trend", map[string]interface{}{
"days": days,
"metric": metric,
})
return s.trendCache.Get(ctx, cacheKey, func(loadCtx context.Context) (*TrendResponse, error) {
startUnix := trendStartUnix(days)
counts, err := s.repo.Trend(ctx, days, metric, startUnix)
if err != nil {
s.logger.Error("Failed to fetch dashboard trend", map[string]interface{}{
"error": err.Error(),
s.logger.Info("Fetching dashboard trend", map[string]interface{}{
"days": days,
"metric": metric,
})
return nil, fmt.Errorf("dashboard trend: %w", err)
}
series := fillTrendSeries(days, counts)
counts, err := s.repo.Trend(loadCtx, days, metric, startUnix)
if err != nil {
s.logger.Error("Failed to fetch dashboard trend", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard trend: %w", err)
}
return &TrendResponse{
Metric: metric,
Days: days,
Series: series,
}, nil
return &TrendResponse{
Metric: metric,
Days: days,
Series: fillTrendSeries(days, counts),
}, nil
})
}
func (s *service) Countries(ctx context.Context, query CountriesQuery) (*CountriesResponse, error) {
limit := countriesLimit(query.Limit)
cacheKey := strconv.Itoa(limit)
s.logger.Info("Fetching dashboard countries", map[string]interface{}{
"limit": limit,
})
out, err := s.repo.Countries(ctx, limit)
if err != nil {
s.logger.Error("Failed to fetch dashboard countries", map[string]interface{}{
"error": err.Error(),
return s.countriesCache.Get(ctx, cacheKey, func(loadCtx context.Context) (*CountriesResponse, error) {
s.logger.Info("Fetching dashboard countries", map[string]interface{}{
"limit": limit,
})
return nil, fmt.Errorf("dashboard countries: %w", err)
}
return out, nil
out, err := s.repo.Countries(loadCtx, limit)
if err != nil {
s.logger.Error("Failed to fetch dashboard countries", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard countries: %w", err)
}
return out, nil
})
}
func (s *service) NoticeTypes(ctx context.Context) (*NoticeTypesResponse, error) {
s.logger.Info("Fetching dashboard notice types", nil)
return s.noticeTypesCache.Get(ctx, "all", func(loadCtx context.Context) (*NoticeTypesResponse, error) {
s.logger.Info("Fetching dashboard notice types", nil)
out, err := s.repo.NoticeTypes(ctx)
if err != nil {
s.logger.Error("Failed to fetch dashboard notice types", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard notice types: %w", err)
}
out, err := s.repo.NoticeTypes(loadCtx)
if err != nil {
s.logger.Error("Failed to fetch dashboard notice types", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard notice types: %w", err)
}
return out, nil
return out, nil
})
}
func (s *service) ClosingSoon(ctx context.Context, query ClosingSoonQuery) (*ClosingSoonResponse, error) {
limit := listLimit(query.Limit)
windowHours := closingWindowHours(query.Window)
windowSec := int64(windowHours) * 3600
cacheKey := fmt.Sprintf("%d:%d", limit, windowSec)
s.logger.Info("Fetching dashboard closing soon", map[string]interface{}{
"limit": limit,
"window": windowHours,
})
items, err := s.repo.ClosingSoon(ctx, limit, windowSec)
if err != nil {
s.logger.Error("Failed to fetch dashboard closing soon", map[string]interface{}{
"error": err.Error(),
return s.closingSoonCache.Get(ctx, cacheKey, func(loadCtx context.Context) (*ClosingSoonResponse, error) {
s.logger.Info("Fetching dashboard closing soon", map[string]interface{}{
"limit": limit,
"window": windowHours,
})
return nil, fmt.Errorf("dashboard closing soon: %w", err)
}
return &ClosingSoonResponse{Items: items}, nil
items, err := s.repo.ClosingSoon(loadCtx, limit, windowSec)
if err != nil {
s.logger.Error("Failed to fetch dashboard closing soon", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard closing soon: %w", err)
}
return &ClosingSoonResponse{Items: items}, nil
})
}
func (s *service) Recent(ctx context.Context, query RecentQuery) (*RecentResponse, error) {
limit := listLimit(query.Limit)
s.logger.Info("Fetching dashboard recent tenders", map[string]interface{}{
"limit": limit,
})
items, nextCursor, err := s.repo.Recent(ctx, limit, strings.TrimSpace(query.Cursor))
if err != nil {
s.logger.Error("Failed to fetch dashboard recent tenders", map[string]interface{}{
"error": err.Error(),
cursor := strings.TrimSpace(query.Cursor)
if cursor != "" {
s.logger.Info("Fetching dashboard recent tenders", map[string]interface{}{
"limit": limit,
})
return nil, fmt.Errorf("dashboard recent: %w", err)
items, nextCursor, err := s.repo.Recent(ctx, limit, cursor)
if err != nil {
s.logger.Error("Failed to fetch dashboard recent tenders", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard recent: %w", err)
}
return &RecentResponse{
Items: items,
NextCursor: nextCursor,
}, nil
}
return &RecentResponse{
Items: items,
NextCursor: nextCursor,
}, nil
cacheKey := strconv.Itoa(limit)
return s.recentCache.Get(ctx, cacheKey, func(loadCtx context.Context) (*RecentResponse, error) {
s.logger.Info("Fetching dashboard recent tenders", map[string]interface{}{
"limit": limit,
})
items, nextCursor, err := s.repo.Recent(loadCtx, limit, "")
if err != nil {
s.logger.Error("Failed to fetch dashboard recent tenders", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard recent: %w", err)
}
return &RecentResponse{
Items: items,
NextCursor: nextCursor,
}, nil
})
}
func (s *service) Statistics(ctx context.Context, query StatisticsQuery) (*StatisticsReportResponse, error) {
@@ -177,22 +276,344 @@ func (s *service) Statistics(ctx context.Context, query StatisticsQuery) (*Stati
return cached, nil
}
startUnix := trendStartUnix(days)
s.logger.Info("Fetching dashboard statistics report", map[string]interface{}{
"days": days,
})
out, err := s.repo.Statistics(ctx, days, startUnix)
if err != nil {
s.logger.Error("Failed to fetch dashboard statistics report", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard statistics: %w", err)
if stale, ok := s.staleStatistics(days); ok {
go s.refreshStatistics(days)
return stale, nil
}
s.storeStatistics(days, out)
return out, nil
if redisCached, ok := s.getRedisStatistics(ctx, days); ok {
s.storeStatistics(days, redisCached)
go s.refreshStatistics(days)
return redisCached, nil
}
// Cold cache: the underlying load performs a full MinIO bucket scan plus Mongo
// aggregations that can take minutes on a first run. Start the load in the
// background (deduped via singleflight, and detached from this request's
// context so it keeps running even if we give up waiting) and wait briefly for
// it. If it finishes in time we return real data; otherwise we serve a
// placeholder so the endpoint stays fast, and the load keeps warming the cache
// for the next request.
resultCh := make(chan *StatisticsReportResponse, 1)
go func() {
result, err := s.loadStatistics(context.Background(), days)
if err != nil {
resultCh <- nil
return
}
resultCh <- result
}()
select {
case result := <-resultCh:
if result != nil {
return result, nil
}
case <-time.After(statisticsColdLoadWait):
s.logger.Info("Serving placeholder dashboard statistics while cache warms", map[string]interface{}{
"days": days,
})
}
return emptyStatisticsReport(days), nil
}
func (s *service) warmStatisticsCache() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if cached, ok := s.getRedisStatistics(ctx, defaultTrendDays); ok {
s.storeStatistics(defaultTrendDays, cached)
s.logger.Info("Dashboard statistics cache warmed from Redis", map[string]interface{}{
"days": defaultTrendDays,
})
return
}
go s.refreshStatistics(defaultTrendDays)
}
func emptyStatisticsReport(days int) *StatisticsReportResponse {
return &StatisticsReportResponse{
Days: days,
GeneratedAt: time.Now().UTC().Unix(),
Daily: StatisticsDailySeries{
ScrapedDocuments: fillTrendSeries(days, nil),
TranslatedNotices: fillTrendSeries(days, nil),
},
}
}
func (s *service) refreshStatistics(days int) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
_, _ = s.loadStatistics(ctx, days)
}
func (s *service) loadStatistics(ctx context.Context, days int) (*StatisticsReportResponse, error) {
cacheKey := strconv.Itoa(days)
out, err, _ := s.statisticsGroup.Do(cacheKey, func() (interface{}, error) {
if cached, ok := s.cachedStatistics(days); ok {
return cached, nil
}
startUnix := trendStartUnix(days)
s.logger.Info("Fetching dashboard statistics report", map[string]interface{}{
"days": days,
})
result, err := s.repo.Statistics(context.WithoutCancel(ctx), days, startUnix)
if err != nil {
s.logger.Error("Failed to fetch dashboard statistics report", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard statistics: %w", err)
}
s.storeStatistics(days, result)
s.storeRedisStatistics(context.Background(), days, result)
return result, nil
})
if err != nil {
return nil, err
}
return out.(*StatisticsReportResponse), nil
}
func (s *service) warmWidgetCaches() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
defaultWindowSec := int64(defaultClosingWindowHours) * 3600
warmed := 0
if s.trendCache.WarmFromRedis(ctx, fmt.Sprintf("%d:%s", defaultTrendDays, "created")) {
warmed++
}
if s.countriesCache.WarmFromRedis(ctx, strconv.Itoa(defaultCountriesLimit)) {
warmed++
}
if s.noticeTypesCache.WarmFromRedis(ctx, "all") {
warmed++
}
if s.closingSoonCache.WarmFromRedis(ctx, fmt.Sprintf("%d:%d", defaultListLimit, defaultWindowSec)) {
warmed++
}
if s.recentCache.WarmFromRedis(ctx, strconv.Itoa(defaultListLimit)) {
warmed++
}
if warmed > 0 {
s.logger.Info("Dashboard widget caches warmed from Redis", map[string]interface{}{
"warmed": warmed,
})
}
go s.trendCache.Reload(fmt.Sprintf("%d:%s", defaultTrendDays, "created"), func(loadCtx context.Context) (*TrendResponse, error) {
return s.loadTrend(loadCtx, TrendQuery{Days: defaultTrendDays, Metric: "created"})
})
go s.countriesCache.Reload(strconv.Itoa(defaultCountriesLimit), func(loadCtx context.Context) (*CountriesResponse, error) {
return s.repo.Countries(loadCtx, defaultCountriesLimit)
})
go s.noticeTypesCache.Reload("all", func(loadCtx context.Context) (*NoticeTypesResponse, error) {
return s.repo.NoticeTypes(loadCtx)
})
go s.closingSoonCache.Reload(fmt.Sprintf("%d:%d", defaultListLimit, defaultWindowSec), func(loadCtx context.Context) (*ClosingSoonResponse, error) {
items, err := s.repo.ClosingSoon(loadCtx, defaultListLimit, defaultWindowSec)
if err != nil {
return nil, err
}
return &ClosingSoonResponse{Items: items}, nil
})
go s.recentCache.Reload(strconv.Itoa(defaultListLimit), func(loadCtx context.Context) (*RecentResponse, error) {
items, nextCursor, err := s.repo.Recent(loadCtx, defaultListLimit, "")
if err != nil {
return nil, err
}
return &RecentResponse{
Items: items,
NextCursor: nextCursor,
}, nil
})
}
func (s *service) loadTrend(ctx context.Context, query TrendQuery) (*TrendResponse, error) {
days := trendDays(query.Days)
metric := normalizeTrendMetric(query.Metric)
startUnix := trendStartUnix(days)
counts, err := s.repo.Trend(ctx, days, metric, startUnix)
if err != nil {
return nil, err
}
return &TrendResponse{
Metric: metric,
Days: days,
Series: fillTrendSeries(days, counts),
}, nil
}
func (s *service) warmSummaryCache() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
windowSec := int64(defaultClosingWindowHours) * 3600
cacheKey := summaryCacheKey(windowSec, defaultTrendDays)
if cached, ok := s.getRedisSummary(ctx, cacheKey); ok {
s.storeSummary(cacheKey, cached)
s.logger.Info("Dashboard summary cache warmed from Redis", map[string]interface{}{
"closing_window_hours": defaultClosingWindowHours,
"days": defaultTrendDays,
})
return
}
go s.refreshSummary(windowSec, defaultTrendDays)
}
func (s *service) refreshSummary(windowSec int64, days int) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
_, _ = s.loadSummary(ctx, windowSec, days)
}
func (s *service) loadSummary(ctx context.Context, windowSec int64, days int) (*SummaryResponse, error) {
cacheKey := summaryCacheKey(windowSec, days)
out, err, _ := s.summaryGroup.Do(cacheKey, func() (interface{}, error) {
if cached, ok := s.cachedSummary(cacheKey); ok {
return cached, nil
}
s.logger.Info("Refreshing dashboard summary cache", map[string]interface{}{
"closing_window_sec": windowSec,
"days": days,
})
result, err := s.repo.Summary(context.WithoutCancel(ctx), windowSec, days)
if err != nil {
s.logger.Error("Failed to refresh dashboard summary", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("dashboard summary: %w", err)
}
s.storeSummary(cacheKey, result)
s.storeRedisSummary(context.Background(), cacheKey, result)
return result, nil
})
if err != nil {
return nil, err
}
return out.(*SummaryResponse), nil
}
func summaryCacheKey(windowSec int64, days int) string {
return fmt.Sprintf("%d:%d", windowSec, days)
}
func summaryRedisKey(cacheKey string) string {
return fmt.Sprintf("dashboard:summary:%s", cacheKey)
}
func (s *service) cachedSummary(cacheKey string) (*SummaryResponse, bool) {
now := time.Now()
s.summaryMu.Lock()
defer s.summaryMu.Unlock()
entry, ok := s.summary[cacheKey]
if !ok || now.After(entry.expiresAt) {
return nil, false
}
return entry.value, true
}
func (s *service) staleSummary(cacheKey string) (*SummaryResponse, bool) {
now := time.Now()
s.summaryMu.Lock()
defer s.summaryMu.Unlock()
entry, ok := s.summary[cacheKey]
if !ok || now.After(entry.staleUntil) {
if ok {
delete(s.summary, cacheKey)
}
return nil, false
}
return entry.value, true
}
func (s *service) storeSummary(cacheKey string, value *SummaryResponse) {
now := time.Now()
s.summaryMu.Lock()
defer s.summaryMu.Unlock()
s.summary[cacheKey] = cacheEntry[*SummaryResponse]{
expiresAt: now.Add(summaryCacheTTL),
staleUntil: now.Add(summaryCacheTTL + summaryStaleGraceTTL),
value: value,
}
}
func (s *service) getRedisSummary(ctx context.Context, cacheKey string) (*SummaryResponse, bool) {
if s.redis == nil {
return nil, false
}
raw, err := s.redis.Get(ctx, summaryRedisKey(cacheKey))
if err != nil {
if err != goredis.Nil {
s.logger.Warn("Failed to read dashboard summary cache from Redis", map[string]interface{}{
"cache_key": cacheKey,
"error": err.Error(),
})
}
return nil, false
}
var report SummaryResponse
if err := json.Unmarshal([]byte(raw), &report); err != nil {
s.logger.Warn("Failed to decode dashboard summary cache from Redis", map[string]interface{}{
"cache_key": cacheKey,
"error": err.Error(),
})
_ = s.redis.Del(ctx, summaryRedisKey(cacheKey))
return nil, false
}
return &report, true
}
func (s *service) storeRedisSummary(ctx context.Context, cacheKey string, value *SummaryResponse) {
if s.redis == nil || value == nil {
return
}
encoded, err := json.Marshal(value)
if err != nil {
s.logger.Warn("Failed to encode dashboard summary for Redis", map[string]interface{}{
"cache_key": cacheKey,
"error": err.Error(),
})
return
}
ttl := summaryCacheTTL + summaryStaleGraceTTL
if err := s.redis.Set(ctx, summaryRedisKey(cacheKey), string(encoded), ttl); err != nil {
s.logger.Warn("Failed to store dashboard summary in Redis", map[string]interface{}{
"cache_key": cacheKey,
"error": err.Error(),
})
}
}
func (s *service) cachedStatistics(days int) (*StatisticsReportResponse, bool) {
@@ -203,6 +624,20 @@ func (s *service) cachedStatistics(days int) (*StatisticsReportResponse, bool) {
entry, ok := s.statistics[days]
if !ok || now.After(entry.expiresAt) {
return nil, false
}
return entry.value, true
}
func (s *service) staleStatistics(days int) (*StatisticsReportResponse, bool) {
now := time.Now()
s.statisticsMu.Lock()
defer s.statisticsMu.Unlock()
entry, ok := s.statistics[days]
if !ok || now.After(entry.staleUntil) {
if ok {
delete(s.statistics, days)
}
@@ -213,12 +648,71 @@ func (s *service) cachedStatistics(days int) (*StatisticsReportResponse, bool) {
}
func (s *service) storeStatistics(days int, value *StatisticsReportResponse) {
now := time.Now()
s.statisticsMu.Lock()
defer s.statisticsMu.Unlock()
s.statistics[days] = statisticsCacheEntry{
expiresAt: time.Now().Add(statisticsCacheTTL),
value: value,
s.statistics[days] = cacheEntry[*StatisticsReportResponse]{
expiresAt: now.Add(statisticsCacheTTL),
staleUntil: now.Add(statisticsCacheTTL + statisticsStaleGraceTTL),
value: value,
}
}
func statisticsRedisKey(days int) string {
return fmt.Sprintf("dashboard:statistics:%d", days)
}
func (s *service) getRedisStatistics(ctx context.Context, days int) (*StatisticsReportResponse, bool) {
if s.redis == nil {
return nil, false
}
raw, err := s.redis.Get(ctx, statisticsRedisKey(days))
if err != nil {
if err != goredis.Nil {
s.logger.Warn("Failed to read dashboard statistics cache from Redis", map[string]interface{}{
"days": days,
"error": err.Error(),
})
}
return nil, false
}
var report StatisticsReportResponse
if err := json.Unmarshal([]byte(raw), &report); err != nil {
s.logger.Warn("Failed to decode dashboard statistics cache from Redis", map[string]interface{}{
"days": days,
"error": err.Error(),
})
_ = s.redis.Del(ctx, statisticsRedisKey(days))
return nil, false
}
return &report, true
}
func (s *service) storeRedisStatistics(ctx context.Context, days int, value *StatisticsReportResponse) {
if s.redis == nil || value == nil {
return
}
encoded, err := json.Marshal(value)
if err != nil {
s.logger.Warn("Failed to encode dashboard statistics for Redis", map[string]interface{}{
"days": days,
"error": err.Error(),
})
return
}
ttl := statisticsCacheTTL + statisticsStaleGraceTTL
if err := s.redis.Set(ctx, statisticsRedisKey(days), string(encoded), ttl); err != nil {
s.logger.Warn("Failed to store dashboard statistics in Redis", map[string]interface{}{
"days": days,
"error": err.Error(),
})
}
}
+364 -109
View File
@@ -2,151 +2,222 @@ package dashboard
import (
"context"
"fmt"
"strings"
"sync"
"time"
"tm/internal/notice"
"tm/pkg/ai_summarizer"
orm "tm/pkg/mongo"
"go.mongodb.org/mongo-driver/v2/bson"
mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
"golang.org/x/sync/errgroup"
)
const noticesCollectionName = "notices"
// scrapedScopeCacheTTL avoids listing every MinIO object on each statistics request while
// keeping counts aligned with the tender panel (MinIO is the source of truth for scraped docs).
const scrapedScopeCacheTTL = 15 * time.Minute
const (
scrapedScopeResolveTimeout = 3 * time.Minute
statisticsMongoQueryTimeout = 90 * time.Second
maxScrapedFolderInClause = 5000
)
// scrapedTendersScope is the resolved Mongo filter for tenders with scraped documents.
// zero is set when MinIO reports no procedure folders; callers must return zero without querying.
type scrapedTendersScope struct {
match bson.M
zero bool
fromMinIO bool
dailyCounts map[string]int64 // non-nil when scope was resolved from MinIO
totalTenderCount int64 // >0 when MinIO procedure count should be used instead of Mongo Count
}
// translatedNoticesScope holds MinIO-backed translation statistics when available.
type translatedNoticesScope struct {
fromMinIO bool
dailyCounts map[string]int64
totalCount int64
}
func (r *repository) Statistics(ctx context.Context, days int, startUnix int64) (*StatisticsReportResponse, error) {
_ = ctx
now := time.Now().UTC()
endDay := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC)
startDay := endDay.AddDate(0, 0, -(days - 1))
var (
scrapedTED map[string]int64
scrapedDocuments map[string]int64
translatedNotices map[string]int64
totalDocuments int64
totalTranslated int64
totalScrapedTED int64
scrapedScope scrapedTendersScope
translatedScope translatedNoticesScope
)
g, gctx := errgroup.WithContext(ctx)
var wg sync.WaitGroup
var mu sync.Mutex
g.Go(func() error {
// Use created_at (first ingest time). processing_metadata.scraped_at is reset on
// notice refresh merges, which would collapse historical scrapes onto the latest day.
counts, err := r.dailyCountByTimestamp(gctx, noticesCollectionName, bson.M{
"source": notice.TenderSourceTEDScraper,
"created_at": bson.M{"$gte": startUnix, "$gt": 0},
}, "created_at")
recordFailure := func(section string, err error) {
mu.Lock()
defer mu.Unlock()
r.logger.Warn("Dashboard statistics section failed", map[string]interface{}{
"section": section,
"error": err.Error(),
})
}
wg.Add(1)
go func() {
defer wg.Done()
qctx, cancel := context.WithTimeout(context.Background(), statisticsMongoQueryTimeout)
defer cancel()
total, err := r.metricsCounter.Get(qctx, orm.TEDNoticeScrapedCounterKey)
if err != nil {
return fmt.Errorf("scraped ted per day: %w", err)
recordFailure("scraped_ted_total", err)
return
}
scrapedTED = counts
return nil
})
totalScrapedTED = total
}()
g.Go(func() error {
counts, err := r.scrapedDocumentsPerDay(gctx, startUnix)
wg.Add(1)
go func() {
defer wg.Done()
scope, err := r.cachedTranslatedNoticesScope(context.Background())
if err != nil {
return fmt.Errorf("scraped documents per day: %w", err)
recordFailure("translated_scope", err)
return
}
scrapedDocuments = counts
return nil
})
g.Go(func() error {
counts, err := r.translatedNoticesPerDay(gctx, startDay, endDay)
if err != nil {
return fmt.Errorf("translated notices per day: %w", err)
translatedScope = scope
if scope.fromMinIO {
return
}
translatedNotices = counts
return nil
})
g.Go(func() error {
total, err := r.totalScrapedDocuments(gctx)
qctx, cancel := context.WithTimeout(context.Background(), statisticsMongoQueryTimeout)
defer cancel()
counts, err := r.translatedNoticesPerDayFromMetrics(qctx, startDay, endDay)
if err != nil {
return fmt.Errorf("total scraped documents: %w", err)
recordFailure("translated_notices", err)
translatedNotices = map[string]int64{}
} else {
translatedNotices = counts
}
totalDocuments = total
return nil
})
g.Go(func() error {
total, err := r.metricsCounter.Get(gctx, orm.AITranslationSuccessCounterKey)
total, err := r.metricsCounter.Get(qctx, orm.AITranslationSuccessCounterKey)
if err != nil {
return fmt.Errorf("total translated tenders: %w", err)
recordFailure("translated_total", err)
return
}
totalTranslated = total
return nil
})
}()
if err := g.Wait(); err != nil {
return nil, err
wg.Add(1)
go func() {
defer wg.Done()
scope, err := r.cachedScrapedTendersScope(context.Background())
if err != nil {
recordFailure("scraped_scope", err)
scrapedScope = mongoScrapedTendersScope()
return
}
scrapedScope = scope
}()
wg.Wait()
if translatedScope.fromMinIO {
translatedNotices = filterDailyCountsSince(translatedScope.dailyCounts, startUnix)
totalTranslated = translatedScope.totalCount
}
wg.Add(1)
go func() {
defer wg.Done()
qctx, cancel := context.WithTimeout(context.Background(), statisticsMongoQueryTimeout)
defer cancel()
counts, err := r.scrapedDocumentsPerDay(qctx, startUnix, scrapedScope)
if err != nil {
recordFailure("scraped_documents", err)
scrapedDocuments = map[string]int64{}
return
}
scrapedDocuments = counts
}()
wg.Add(1)
go func() {
defer wg.Done()
qctx, cancel := context.WithTimeout(context.Background(), statisticsMongoQueryTimeout)
defer cancel()
total, err := r.totalScrapedTenders(qctx, scrapedScope)
if err != nil {
recordFailure("scraped_documents_total", err)
return
}
totalDocuments = total
}()
wg.Wait()
return &StatisticsReportResponse{
Days: days,
GeneratedAt: now.Unix(),
Daily: StatisticsDailySeries{
ScrapedTED: fillTrendSeries(days, scrapedTED),
ScrapedDocuments: fillTrendSeries(days, scrapedDocuments),
TranslatedNotices: fillTrendSeries(days, translatedNotices),
},
Totals: StatisticsLifetimeTotals{
ScrapedDocuments: totalDocuments,
TranslatedTenders: totalTranslated,
ScrapedTEDNotices: totalScrapedTED,
},
}, nil
}
func (r *repository) dailyCountByTimestamp(ctx context.Context, collection string, match bson.M, timestampField string) (map[string]int64, error) {
pipeline := mongodriver.Pipeline{
{{Key: "$match", Value: match}},
{{Key: "$addFields", Value: bson.M{
"metric_ts": normalizeTimestampExpr(timestampField),
}}},
{{Key: "$group", Value: bson.M{
"_id": bson.M{
"$dateToString": bson.M{
"format": "%Y-%m-%d",
"date": bson.M{"$toDate": bson.M{"$multiply": bson.A{"$metric_ts", 1000}}},
"timezone": "UTC",
},
},
"count": bson.M{"$sum": 1},
}}},
func (r *repository) scrapedDocumentsPerDay(ctx context.Context, startUnix int64, scope scrapedTendersScope) (map[string]int64, error) {
if scope.zero {
return map[string]int64{}, nil
}
cursor, err := r.mongoManager.GetCollection(collection).Aggregate(ctx, pipeline)
if err != nil {
return nil, err
if scope.dailyCounts != nil {
return filterDailyCountsSince(scope.dailyCounts, startUnix), nil
}
defer cursor.Close(ctx)
return decodeDailyCounts(ctx, cursor)
return r.scrapedDocumentsPerDayFromMongo(ctx, startUnix, scope.match)
}
func (r *repository) scrapedDocumentsPerDay(ctx context.Context, startUnix int64) (map[string]int64, error) {
func filterDailyCountsSince(counts map[string]int64, startUnix int64) map[string]int64 {
if len(counts) == 0 {
return map[string]int64{}
}
startDay := time.Unix(startUnix, 0).UTC().Format("2006-01-02")
filtered := make(map[string]int64)
for date, count := range counts {
if date >= startDay {
filtered[date] = count
}
}
return filtered
}
func (r *repository) scrapedDocumentsPerDayFromMongo(ctx context.Context, startUnix int64, match bson.M) (map[string]int64, error) {
combinedMatch := bson.M{
"processing_metadata.documents_scraped_at": bson.M{"$gte": startUnix, "$gt": 0},
}
for key, value := range match {
combinedMatch[key] = value
}
pipeline := mongodriver.Pipeline{
{{Key: "$match", Value: bson.M{
"processing_metadata.documents_scraped": true,
"scraped_documents": bson.M{"$exists": true, "$ne": bson.A{}},
}}},
{{Key: "$unwind", Value: "$scraped_documents"}},
{{Key: "$match", Value: combinedMatch}},
{{Key: "$addFields", Value: bson.M{
"metric_ts": bson.M{
"$cond": bson.A{
bson.M{"$gt": bson.A{"$scraped_documents.scraped_at", 0}},
"$scraped_documents.scraped_at",
"$processing_metadata.documents_scraped_at",
},
},
}}},
{{Key: "$addFields", Value: bson.M{
"metric_ts": normalizeTimestampExpr("metric_ts"),
}}},
{{Key: "$match", Value: bson.M{
"metric_ts": bson.M{"$gte": startUnix, "$gt": 0},
"metric_ts": normalizeTimestampExpr("processing_metadata.documents_scraped_at"),
}}},
{{Key: "$group", Value: bson.M{
"_id": bson.M{
@@ -169,41 +240,225 @@ func (r *repository) scrapedDocumentsPerDay(ctx context.Context, startUnix int64
return decodeDailyCounts(ctx, cursor)
}
func (r *repository) translatedNoticesPerDay(ctx context.Context, startDay, endDay time.Time) (map[string]int64, error) {
func (r *repository) translatedNoticesPerDayFromMetrics(ctx context.Context, startDay, endDay time.Time) (map[string]int64, error) {
return r.metricsCounter.GetDailyCounts(ctx, startDay, endDay)
}
func (r *repository) totalScrapedDocuments(ctx context.Context) (int64, error) {
pipeline := mongodriver.Pipeline{
{{Key: "$match", Value: bson.M{
"processing_metadata.documents_scraped": true,
"scraped_documents": bson.M{"$exists": true, "$ne": bson.A{}},
}}},
{{Key: "$project", Value: bson.M{
"doc_count": bson.M{"$size": "$scraped_documents"},
}}},
{{Key: "$group", Value: bson.M{
"_id": nil,
"total": bson.M{"$sum": "$doc_count"},
}}},
}
func (r *repository) cachedTranslatedNoticesScope(_ context.Context) (translatedNoticesScope, error) {
now := time.Now()
cursor, err := r.mongoManager.GetCollection(tenderCollectionName()).Aggregate(ctx, pipeline)
r.translatedScopeMu.Lock()
if r.translatedScopeCached && now.Before(r.translatedScopeExpiry) {
scope := r.translatedScope
r.translatedScopeMu.Unlock()
return scope, nil
}
r.translatedScopeMu.Unlock()
v, err, _ := r.translatedScopeGroup.Do("translated-scope", func() (interface{}, error) {
r.translatedScopeMu.Lock()
if r.translatedScopeCached && time.Now().Before(r.translatedScopeExpiry) {
scope := r.translatedScope
r.translatedScopeMu.Unlock()
return scope, nil
}
r.translatedScopeMu.Unlock()
resolveCtx, cancel := context.WithTimeout(context.Background(), scrapedScopeResolveTimeout)
defer cancel()
scope, resolveErr := r.resolveTranslatedNoticesScope(resolveCtx)
if resolveErr != nil {
if r.logger != nil {
r.logger.Warn("MinIO translation scope unavailable, using metrics counter fallback for dashboard statistics", map[string]interface{}{
"error": resolveErr.Error(),
})
}
return translatedNoticesScope{}, nil
}
r.translatedScopeMu.Lock()
r.translatedScope = scope
r.translatedScopeExpiry = time.Now().Add(scrapedScopeCacheTTL)
r.translatedScopeCached = true
r.translatedScopeMu.Unlock()
return scope, nil
})
if err != nil {
return 0, err
return translatedNoticesScope{}, err
}
defer cursor.Close(ctx)
if !cursor.Next(ctx) {
return v.(translatedNoticesScope), nil
}
func (r *repository) resolveTranslatedNoticesScope(ctx context.Context) (translatedNoticesScope, error) {
if r.procedureLister == nil {
return translatedNoticesScope{}, nil
}
scanner, ok := r.procedureLister.(translatedNoticesScanner)
if !ok {
return translatedNoticesScope{}, nil
}
dailyCounts, total, err := scanner.ScanTranslatedNotices(ctx)
if err != nil {
return translatedNoticesScope{}, err
}
return translatedNoticesScope{
fromMinIO: true,
dailyCounts: dailyCounts,
totalCount: total,
}, nil
}
func (r *repository) totalScrapedTenders(ctx context.Context, scope scrapedTendersScope) (int64, error) {
if scope.zero {
return 0, nil
}
var row bson.M
if err := cursor.Decode(&row); err != nil {
return 0, err
if scope.totalTenderCount > 0 {
return scope.totalTenderCount, nil
}
return aggregateCount(row, "total"), nil
return r.ormRepo.Count(ctx, scope.match)
}
func mongoScrapedTendersScope() scrapedTendersScope {
return scrapedTendersScope{
match: bson.M{
"processing_metadata.documents_scraped": true,
"scraped_documents": bson.M{"$exists": true, "$ne": bson.A{}},
},
}
}
func (r *repository) cachedScrapedTendersScope(_ context.Context) (scrapedTendersScope, error) {
now := time.Now()
r.scrapedScopeMu.Lock()
if r.scrapedScopeCached && now.Before(r.scrapedScopeExpiry) {
scope := r.scrapedScope
r.scrapedScopeMu.Unlock()
return scope, nil
}
r.scrapedScopeMu.Unlock()
v, err, _ := r.scrapedScopeGroup.Do("scraped-scope", func() (interface{}, error) {
r.scrapedScopeMu.Lock()
if r.scrapedScopeCached && time.Now().Before(r.scrapedScopeExpiry) {
scope := r.scrapedScope
r.scrapedScopeMu.Unlock()
return scope, nil
}
r.scrapedScopeMu.Unlock()
resolveCtx, cancel := context.WithTimeout(context.Background(), scrapedScopeResolveTimeout)
defer cancel()
scope, resolveErr := r.resolveScrapedTendersScope(resolveCtx)
if resolveErr != nil {
if r.logger != nil {
r.logger.Warn("MinIO scraped scope unavailable, using Mongo fallback for dashboard statistics", map[string]interface{}{
"error": resolveErr.Error(),
})
}
return mongoScrapedTendersScope(), nil
}
r.scrapedScopeMu.Lock()
r.scrapedScope = scope
r.scrapedScopeExpiry = time.Now().Add(scrapedScopeCacheTTL)
r.scrapedScopeCached = true
r.scrapedScopeMu.Unlock()
return scope, nil
})
if err != nil {
return scrapedTendersScope{}, err
}
return v.(scrapedTendersScope), nil
}
func (r *repository) resolveScrapedTendersScope(ctx context.Context) (scrapedTendersScope, error) {
procedures, dailyCounts, err := r.listScrapedProcedures(ctx)
if err != nil {
return scrapedTendersScope{}, err
}
folderIDs := folderIDsFromProcedures(procedures)
if len(folderIDs) > 0 {
totalTenders := int64(len(folderIDs))
if len(folderIDs) > maxScrapedFolderInClause {
if r.logger != nil {
r.logger.Warn("Too many MinIO procedure folders for dashboard statistics, using MinIO counts without Mongo $in filter", map[string]interface{}{
"folder_count": len(folderIDs),
"max_in_clause": maxScrapedFolderInClause,
})
}
return scrapedTendersScope{
fromMinIO: true,
dailyCounts: dailyCounts,
totalTenderCount: totalTenders,
}, nil
}
return scrapedTendersScope{
match: bson.M{"contract_folder_id": bson.M{"$in": folderIDs}},
fromMinIO: true,
dailyCounts: dailyCounts,
totalTenderCount: totalTenders,
}, nil
}
return mongoScrapedTendersScope(), nil
}
func (r *repository) listScrapedProcedures(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, map[string]int64, error) {
if r.procedureLister == nil {
return nil, nil, nil
}
if scanner, ok := r.procedureLister.(scrapedDocumentsScanner); ok {
return scanner.ScanScrapedDocuments(ctx)
}
procedures, err := r.procedureLister.ListProceduresWithDocuments(ctx)
if err != nil {
return nil, nil, err
}
return procedures, nil, nil
}
func folderIDsFromProcedures(procedures []ai_summarizer.ProcedureDocumentsSummary) []string {
folderIDs := make([]string, 0, len(procedures))
seen := make(map[string]struct{}, len(procedures))
for _, proc := range procedures {
if proc.DocumentCount <= 0 {
continue
}
id := strings.TrimSpace(proc.ContractFolderID)
if id == "" {
continue
}
if _, ok := seen[id]; ok {
continue
}
seen[id] = struct{}{}
folderIDs = append(folderIDs, id)
}
return folderIDs
}
func (r *repository) scrapedDocumentsFolderIDs(ctx context.Context) ([]string, error) {
if r.procedureLister == nil {
return nil, nil
}
procedures, _, err := r.listScrapedProcedures(ctx)
if err != nil {
return nil, err
}
return folderIDsFromProcedures(procedures), nil
}
func decodeDailyCounts(ctx context.Context, cursor *mongodriver.Cursor) (map[string]int64, error) {
@@ -0,0 +1,310 @@
package dashboard
import (
"context"
"errors"
"fmt"
"testing"
"time"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
"go.mongodb.org/mongo-driver/v2/bson"
)
type noopTestLogger struct{}
func (noopTestLogger) Debug(string, map[string]interface{}) {}
func (noopTestLogger) Info(string, map[string]interface{}) {}
func (noopTestLogger) Warn(string, map[string]interface{}) {}
func (noopTestLogger) Error(string, map[string]interface{}) {}
func (noopTestLogger) Fatal(string, map[string]interface{}) {}
func (l noopTestLogger) WithFields(map[string]interface{}) logger.Logger {
return l
}
func (noopTestLogger) Sync() error { return nil }
type mockProcedureDocumentsLister struct {
procedures []ai_summarizer.ProcedureDocumentsSummary
dailyCounts map[string]int64
err error
translatedDailyCounts map[string]int64
translatedTotal int64
translatedErr error
}
func (m *mockProcedureDocumentsLister) ListProceduresWithDocuments(context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error) {
if m.err != nil {
return nil, m.err
}
return m.procedures, nil
}
func (m *mockProcedureDocumentsLister) ScanScrapedDocuments(context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, map[string]int64, error) {
if m.err != nil {
return nil, nil, m.err
}
return m.procedures, m.dailyCounts, nil
}
func (m *mockProcedureDocumentsLister) ScanTranslatedNotices(context.Context) (map[string]int64, int64, error) {
if m.translatedErr != nil {
return nil, 0, m.translatedErr
}
return m.translatedDailyCounts, m.translatedTotal, nil
}
func TestScrapedDocumentsFolderIDsUsesMinIOProcedures(t *testing.T) {
repo := &repository{
procedureLister: &mockProcedureDocumentsLister{
procedures: []ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "PROC-1", DocumentCount: 3},
{ContractFolderID: "PROC-2", DocumentCount: 0},
{ContractFolderID: " PROC-3 ", DocumentCount: 1},
{ContractFolderID: "PROC-1", DocumentCount: 2},
},
},
}
got, err := repo.scrapedDocumentsFolderIDs(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
want := []string{"PROC-1", "PROC-3"}
if len(got) != len(want) {
t.Fatalf("expected %d folder IDs, got %d: %v", len(want), len(got), got)
}
for i, id := range want {
if got[i] != id {
t.Fatalf("expected folder ID %q at index %d, got %q", id, i, got[i])
}
}
}
func TestScrapedDocumentsFolderIDsWithoutLister(t *testing.T) {
repo := &repository{}
got, err := repo.scrapedDocumentsFolderIDs(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got != nil {
t.Fatalf("expected nil folder IDs without lister, got %v", got)
}
}
func TestScrapedDocumentsFolderIDsPropagatesError(t *testing.T) {
repo := &repository{
procedureLister: &mockProcedureDocumentsLister{
err: errors.New("minio unavailable"),
},
}
_, err := repo.scrapedDocumentsFolderIDs(context.Background())
if err == nil {
t.Fatal("expected error")
}
}
func TestResolveScrapedTendersScopeUsesMinIOFolders(t *testing.T) {
repo := &repository{
procedureLister: &mockProcedureDocumentsLister{
procedures: []ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "PROC-1", DocumentCount: 2},
},
dailyCounts: map[string]int64{"2026-06-28": 2},
},
}
scope, err := repo.resolveScrapedTendersScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if scope.zero {
t.Fatal("expected non-zero scope")
}
if !scope.fromMinIO {
t.Fatal("expected MinIO-backed scope")
}
if scope.totalTenderCount != 1 {
t.Fatalf("expected total tender count 1, got %d", scope.totalTenderCount)
}
if scope.dailyCounts == nil {
t.Fatal("expected MinIO daily counts")
}
in, ok := scope.match["contract_folder_id"].(bson.M)
if !ok {
t.Fatalf("expected contract_folder_id filter, got %#v", scope.match)
}
ids, ok := in["$in"].([]string)
if !ok || len(ids) != 1 || ids[0] != "PROC-1" {
t.Fatalf("unexpected folder IDs: %#v", in["$in"])
}
}
func TestResolveScrapedTendersScopeKeepsMinIOCountsWhenTooManyFolders(t *testing.T) {
procedures := make([]ai_summarizer.ProcedureDocumentsSummary, maxScrapedFolderInClause+1)
for i := range procedures {
procedures[i] = ai_summarizer.ProcedureDocumentsSummary{
ContractFolderID: fmt.Sprintf("PROC-%d", i),
DocumentCount: 1,
}
}
repo := &repository{
logger: noopTestLogger{},
procedureLister: &mockProcedureDocumentsLister{
procedures: procedures,
dailyCounts: map[string]int64{"2026-06-28": 3},
},
}
scope, err := repo.resolveScrapedTendersScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if scope.zero {
t.Fatal("expected non-zero scope")
}
if !scope.fromMinIO {
t.Fatal("expected MinIO-backed scope")
}
if scope.totalTenderCount != int64(len(procedures)) {
t.Fatalf("expected total tender count %d, got %d", len(procedures), scope.totalTenderCount)
}
if scope.dailyCounts == nil {
t.Fatal("expected MinIO daily counts to be preserved")
}
if scope.match != nil {
t.Fatalf("expected no Mongo $in filter for large MinIO scope, got %#v", scope.match)
}
}
func TestResolveScrapedTendersScopeMongoFallbackWhenMinIOEmpty(t *testing.T) {
repo := &repository{
procedureLister: &mockProcedureDocumentsLister{},
}
scope, err := repo.resolveScrapedTendersScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if scope.zero {
t.Fatal("expected mongo fallback scope, not zero scope")
}
if scope.match["processing_metadata.documents_scraped"] != true {
t.Fatalf("expected mongo fallback filter, got %#v", scope.match)
}
}
func TestResolveScrapedTendersScopeMongoFallbackWithoutLister(t *testing.T) {
repo := &repository{}
scope, err := repo.resolveScrapedTendersScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if scope.zero {
t.Fatal("expected mongo fallback scope")
}
if scope.match["processing_metadata.documents_scraped"] != true {
t.Fatalf("expected mongo fallback filter, got %#v", scope.match)
}
}
func TestFilterDailyCountsSince(t *testing.T) {
counts := map[string]int64{
"2026-06-01": 2,
"2026-06-15": 5,
"2026-06-28": 1,
}
startUnix := time.Date(2026, 6, 15, 0, 0, 0, 0, time.UTC).Unix()
got := filterDailyCountsSince(counts, startUnix)
want := map[string]int64{
"2026-06-15": 5,
"2026-06-28": 1,
}
if len(got) != len(want) {
t.Fatalf("expected %d dates, got %d: %v", len(want), len(got), got)
}
for date, count := range want {
if got[date] != count {
t.Fatalf("expected %d on %s, got %d", count, date, got[date])
}
}
}
func TestScrapedDocumentsPerDayUsesMinIODailyCounts(t *testing.T) {
repo := &repository{}
scope := scrapedTendersScope{
dailyCounts: map[string]int64{
"2026-06-01": 2,
"2026-06-28": 3,
},
}
startUnix := time.Date(2026, 6, 15, 0, 0, 0, 0, time.UTC).Unix()
got, err := repo.scrapedDocumentsPerDay(context.Background(), startUnix, scope)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got["2026-06-01"] != 0 {
t.Fatalf("expected old date filtered out, got %v", got)
}
if got["2026-06-28"] != 3 {
t.Fatalf("expected 3 on 2026-06-28, got %v", got)
}
}
func TestMongoScrapedTendersScopeMatchesStatisticsFilter(t *testing.T) {
scope := mongoScrapedTendersScope()
if scope.zero {
t.Fatal("expected non-zero mongo statistics scope")
}
if scope.match["processing_metadata.documents_scraped"] != true {
t.Fatalf("expected documents_scraped filter, got %#v", scope.match)
}
}
func TestResolveTranslatedNoticesScopeUsesMinIOCounts(t *testing.T) {
repo := &repository{
procedureLister: &mockProcedureDocumentsLister{
translatedDailyCounts: map[string]int64{
"2026-07-01": 4,
"2026-07-09": 2,
},
translatedTotal: 6,
},
}
scope, err := repo.resolveTranslatedNoticesScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !scope.fromMinIO {
t.Fatal("expected MinIO-backed translation scope")
}
if scope.totalCount != 6 {
t.Fatalf("expected total 6, got %d", scope.totalCount)
}
if scope.dailyCounts["2026-07-09"] != 2 {
t.Fatalf("expected 2 on 2026-07-09, got %d", scope.dailyCounts["2026-07-09"])
}
}
func TestResolveTranslatedNoticesScopeWithoutLister(t *testing.T) {
repo := &repository{}
scope, err := repo.resolveTranslatedNoticesScope(context.Background())
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if scope.fromMinIO {
t.Fatal("expected metrics fallback without lister")
}
}
+230
View File
@@ -0,0 +1,230 @@
package dashboard
import (
"context"
"encoding/json"
"fmt"
"sync"
"time"
"tm/pkg/logger"
"tm/pkg/redis"
goredis "github.com/redis/go-redis/v9"
"golang.org/x/sync/singleflight"
)
const (
// widgetCacheTTL is the fresh window before a background reload is triggered.
widgetCacheTTL = 60 * time.Second
// widgetStaleGraceTTL is how long stale entries may be served while reloading.
// Dashboard widgets have no write-side invalidation; expect up to this lag after
// data changes (e.g. a new tender may not appear in cached widgets immediately).
widgetStaleGraceTTL = 5 * time.Minute
widgetReloadTimeout = 2 * time.Minute
)
type widgetLoader[T any] func(ctx context.Context) (T, error)
type widgetCache[T any] struct {
mu sync.Mutex
entries map[string]cacheEntry[T]
group singleflight.Group
redis redis.Client
logger logger.Logger
keyPrefix string
ttl time.Duration
staleGrace time.Duration
}
func newWidgetCache[T any](redisClient redis.Client, log logger.Logger, keyPrefix string) *widgetCache[T] {
return &widgetCache[T]{
entries: make(map[string]cacheEntry[T]),
redis: redisClient,
logger: log,
keyPrefix: keyPrefix,
ttl: widgetCacheTTL,
staleGrace: widgetStaleGraceTTL,
}
}
func (c *widgetCache[T]) Get(ctx context.Context, key string, load widgetLoader[T]) (T, error) {
var zero T
// Returned values are shared across concurrent callers via singleflight and stale
// hits. Callers must treat them as read-only through JSON serialization.
if value, ok := c.fresh(key); ok {
return value, nil
}
if value, ok := c.stale(key); ok {
go c.Reload(key, load)
return value, nil
}
if value, ok := c.fromRedis(ctx, key); ok {
c.store(key, value)
go c.Reload(key, load)
return value, nil
}
out, err, _ := c.group.Do(key, func() (interface{}, error) {
if value, ok := c.fresh(key); ok {
return value, nil
}
result, err := load(context.WithoutCancel(ctx))
if err != nil {
return zero, err
}
c.store(key, result)
c.storeRedis(context.Background(), key, result)
return result, nil
})
if err != nil {
return zero, err
}
return out.(T), nil
}
func (c *widgetCache[T]) Reload(key string, load widgetLoader[T]) {
ctx, cancel := context.WithTimeout(context.Background(), widgetReloadTimeout)
defer cancel()
_, _, _ = c.group.Do("reload:"+key, func() (interface{}, error) {
result, err := load(ctx)
if err != nil {
if c.logger != nil {
c.logger.Warn("Failed to refresh dashboard widget cache", map[string]interface{}{
"widget": c.keyPrefix,
"key": key,
"error": err.Error(),
})
}
return nil, err
}
c.store(key, result)
c.storeRedis(context.Background(), key, result)
return result, nil
})
}
func (c *widgetCache[T]) WarmFromRedis(ctx context.Context, key string) bool {
value, ok := c.fromRedis(ctx, key)
if !ok {
return false
}
c.store(key, value)
return true
}
func (c *widgetCache[T]) fresh(key string) (T, bool) {
var zero T
now := time.Now()
c.mu.Lock()
defer c.mu.Unlock()
entry, ok := c.entries[key]
if !ok || now.After(entry.expiresAt) {
return zero, false
}
return entry.value, true
}
func (c *widgetCache[T]) stale(key string) (T, bool) {
var zero T
now := time.Now()
c.mu.Lock()
defer c.mu.Unlock()
entry, ok := c.entries[key]
if !ok || now.After(entry.staleUntil) {
if ok {
delete(c.entries, key)
}
return zero, false
}
return entry.value, true
}
func (c *widgetCache[T]) store(key string, value T) {
now := time.Now()
c.mu.Lock()
defer c.mu.Unlock()
c.entries[key] = cacheEntry[T]{
expiresAt: now.Add(c.ttl),
staleUntil: now.Add(c.ttl + c.staleGrace),
value: value,
}
}
func (c *widgetCache[T]) redisKey(key string) string {
return fmt.Sprintf("dashboard:%s:%s", c.keyPrefix, key)
}
func (c *widgetCache[T]) fromRedis(ctx context.Context, key string) (T, bool) {
var zero T
if c.redis == nil {
return zero, false
}
raw, err := c.redis.Get(ctx, c.redisKey(key))
if err != nil {
if err != goredis.Nil && c.logger != nil {
c.logger.Warn("Failed to read dashboard widget cache from Redis", map[string]interface{}{
"widget": c.keyPrefix,
"key": key,
"error": err.Error(),
})
}
return zero, false
}
var value T
if err := json.Unmarshal([]byte(raw), &value); err != nil {
if c.logger != nil {
c.logger.Warn("Failed to decode dashboard widget cache from Redis", map[string]interface{}{
"widget": c.keyPrefix,
"key": key,
"error": err.Error(),
})
}
_ = c.redis.Del(ctx, c.redisKey(key))
return zero, false
}
return value, true
}
func (c *widgetCache[T]) storeRedis(ctx context.Context, key string, value T) {
if c.redis == nil {
return
}
encoded, err := json.Marshal(value)
if err != nil {
if c.logger != nil {
c.logger.Warn("Failed to encode dashboard widget cache for Redis", map[string]interface{}{
"widget": c.keyPrefix,
"key": key,
"error": err.Error(),
})
}
return
}
ttl := c.ttl + c.staleGrace
if err := c.redis.Set(ctx, c.redisKey(key), string(encoded), ttl); err != nil && c.logger != nil {
c.logger.Warn("Failed to store dashboard widget cache in Redis", map[string]interface{}{
"widget": c.keyPrefix,
"key": key,
"error": err.Error(),
})
}
}
+10
View File
@@ -0,0 +1,10 @@
package document_scraper
import "errors"
var (
// ErrInvalidDateRange is returned when created_at_from is after created_at_to.
ErrInvalidDateRange = errors.New("invalid date range: created_at_from must be before or equal to created_at_to")
// ErrScrapePortalsUnavailable is returned when the scrape portals provider is not configured.
ErrScrapePortalsUnavailable = errors.New("scrape portals provider is not configured")
)
+18 -2
View File
@@ -2,8 +2,24 @@ package document_scraper
// DocumentScraperListRequest represents the request to list tenders pending document scraping
type DocumentScraperListRequest struct {
Limit int `query:"limit" valid:"optional,range(1|100)" default:"20"`
Offset int `query:"offset" valid:"optional,range(0|1000000)" default:"0"`
Limit int `query:"limit" valid:"optional,range(1|100)"`
Offset int `query:"offset" valid:"optional,range(0|1000000)"`
CreatedAtFrom *int64 `query:"created_at_from" valid:"optional"`
CreatedAtTo *int64 `query:"created_at_to" valid:"optional"`
IncludeExpired bool `query:"include_expired" valid:"optional"`
}
// Normalize applies default pagination bounds.
func (r *DocumentScraperListRequest) Normalize() {
if r.Limit <= 0 {
r.Limit = 20
}
if r.Limit > 100 {
r.Limit = 100
}
if r.Offset < 0 {
r.Offset = 0
}
}
// DocumentScraperGetRequest represents the request to get a specific tender by notice publication ID
+24 -25
View File
@@ -1,7 +1,7 @@
package document_scraper
import (
"strconv"
"errors"
"tm/pkg/logger"
"tm/pkg/response"
@@ -24,36 +24,35 @@ func NewHandler(service Service, logger logger.Logger) *Handler {
// ListPendingTenders retrieves tenders that need document scraping
// @Summary List pending tenders for document scraping
// @Description Retrieve tenders that have not yet been scraped for documents. Each item includes contract_folder_id, notice_publication_id, document_url, title, and description.
// @Description Retrieve tenders that have not yet been scraped for documents and whose document URL matches a portal supported by the AI scraping service. Each item includes contract_folder_id, notice_publication_id, document_url, title, and description. Optionally filter by created_at range (Unix timestamps in seconds). By default only tenders with an active publication submission window are returned (submission_deadline, or 48 working hours after publication_date); set include_expired=true to include expired tenders.
// @Tags Document-Scraper
// @Produce json
// @Param limit query int false "Number of items per page (default: 20, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0)"
// @Param created_at_from query int64 false "Filter by created_at lower bound (Unix timestamp, seconds)"
// @Param created_at_to query int64 false "Filter by created_at upper bound (Unix timestamp, seconds)"
// @Param include_expired query bool false "When true, include tenders whose deadline has passed (default: false)"
// @Success 200 {object} response.APIResponse{data=DocumentScraperListResponse} "List of pending tenders"
// @Failure 400 {object} response.APIResponse "Invalid query parameters"
// @Failure 401 {object} response.APIResponse "Invalid or missing API key"
// @Failure 503 {object} response.APIResponse "Scrape portals service is not configured"
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Security ApiKeyAuth
// @Router /api/v1/scraper/tenders [get]
func (h *Handler) ListPendingTenders(c echo.Context) error {
// Parse pagination parameters
limit, err := strconv.Atoi(c.QueryParam("limit"))
if err != nil || limit <= 0 {
limit = 20
}
if limit > 100 {
limit = 100
}
offset, err := strconv.Atoi(c.QueryParam("offset"))
if err != nil || offset < 0 {
offset = 0
}
tenders, meta, err := h.service.ListPendingTenders(c.Request().Context(), limit, offset)
form, err := response.Parse[DocumentScraperListRequest](c)
if err != nil {
h.logger.Error("Failed to list pending tenders", map[string]interface{}{
"error": err.Error(),
})
return response.ValidationError(c, "Invalid request data", err.Error())
}
tenders, meta, err := h.service.ListPendingTenders(c.Request().Context(), *form)
if err != nil {
if errors.Is(err, ErrInvalidDateRange) {
return response.BadRequest(c, "Invalid date range", err.Error())
}
if errors.Is(err, ErrScrapePortalsUnavailable) {
return response.ServiceUnavailable(c, "Scrape portals service is not configured")
}
return response.InternalServerError(c, "Failed to list pending tenders")
}
@@ -62,7 +61,7 @@ func (h *Handler) ListPendingTenders(c echo.Context) error {
// GetTenderByNoticeID retrieves a specific tender by its notice publication ID
// @Summary Get tender by notice ID for document scraping
// @Description Retrieve a specific tender by its notice publication ID (contract_folder_id, notice_publication_id, document_url, title, description).
// @Description Retrieve a specific tender whose publication submission window is still open and whose document URL matches a supported scrape portal.
// @Tags Document-Scraper
// @Produce json
// @Param notice_id path string true "Notice Publication ID"
@@ -70,6 +69,7 @@ func (h *Handler) ListPendingTenders(c echo.Context) error {
// @Failure 400 {object} response.APIResponse "Invalid notice ID"
// @Failure 401 {object} response.APIResponse "Invalid or missing API key"
// @Failure 404 {object} response.APIResponse "Tender not found"
// @Failure 503 {object} response.APIResponse "Scrape portals service is not configured"
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Security ApiKeyAuth
// @Router /api/v1/scraper/tenders/{notice_id} [get]
@@ -81,13 +81,12 @@ func (h *Handler) GetTenderByNoticeID(c echo.Context) error {
tender, err := h.service.GetTenderByNoticeID(c.Request().Context(), noticeID)
if err != nil {
if errors.Is(err, ErrScrapePortalsUnavailable) {
return response.ServiceUnavailable(c, "Scrape portals service is not configured")
}
if err.Error() == "tender not found" {
return response.NotFound(c, "Tender not found")
}
h.logger.Error("Failed to get tender by notice ID", map[string]interface{}{
"notice_id": noticeID,
"error": err.Error(),
})
return response.InternalServerError(c, "Failed to get tender")
}
+33
View File
@@ -0,0 +1,33 @@
package document_scraper
import (
"strings"
"tm/internal/tender"
)
// documentURL returns the URL used for portal matching and the scraper API payload.
func documentURL(t *tender.Tender) string {
if t == nil {
return ""
}
if u := strings.TrimSpace(t.DocumentURI); u != "" {
return u
}
return strings.TrimSpace(t.TenderURL)
}
// matchesScrapePortals reports whether url contains any supported portal identifier.
func matchesScrapePortals(url string, portals []string) bool {
if url == "" || len(portals) == 0 {
return false
}
urlLower := strings.ToLower(url)
for _, portal := range portals {
p := strings.TrimSpace(portal)
if p != "" && strings.Contains(urlLower, strings.ToLower(p)) {
return true
}
}
return false
}
+113 -31
View File
@@ -3,55 +3,113 @@ package document_scraper
import (
"context"
"fmt"
"strings"
"time"
"tm/internal/tender"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
"tm/pkg/response"
)
var documentScraperCountryCodes = []string{"SWE", "FIN", "NOR"}
func isDocumentScraperCountry(countryCode string) bool {
for _, code := range documentScraperCountryCodes {
if countryCode == code {
return true
}
}
return false
// ScrapePortalsProvider fetches supported document scraping portal identifiers.
type ScrapePortalsProvider interface {
GetScrapePortals(ctx context.Context) (*ai_summarizer.ScrapePortalsResponse, error)
}
// Service defines business logic for the document scraper API
type Service interface {
// ListPendingTenders retrieves tenders that need document scraping
ListPendingTenders(ctx context.Context, limit, offset int) (*DocumentScraperListResponse, *response.Meta, error)
ListPendingTenders(ctx context.Context, req DocumentScraperListRequest) (*DocumentScraperListResponse, *response.Meta, error)
// GetTenderByNoticeID retrieves a specific tender by its notice publication ID
GetTenderByNoticeID(ctx context.Context, noticeID string) (*DocumentScraperTenderResponse, error)
}
type service struct {
tenderRepo tender.TenderRepository
logger logger.Logger
tenderRepo tender.TenderRepository
portalsProvider ScrapePortalsProvider
logger logger.Logger
}
// NewService creates a new document scraper service
func NewService(tenderRepo tender.TenderRepository, logger logger.Logger) Service {
func NewService(tenderRepo tender.TenderRepository, portalsProvider ScrapePortalsProvider, logger logger.Logger) Service {
return &service{
tenderRepo: tenderRepo,
logger: logger,
tenderRepo: tenderRepo,
portalsProvider: portalsProvider,
logger: logger,
}
}
func (s *service) fetchScrapePortals(ctx context.Context) ([]string, error) {
if s.portalsProvider == nil {
s.logger.Warn("Scrape portals provider not configured", map[string]interface{}{})
return nil, ErrScrapePortalsUnavailable
}
resp, err := s.portalsProvider.GetScrapePortals(ctx)
if err != nil {
s.logger.Error("Failed to fetch scrape portals", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("fetch scrape portals: %w", err)
}
if resp == nil {
return nil, nil
}
portals := make([]string, 0, len(*resp))
for _, portal := range *resp {
if p := strings.TrimSpace(portal); p != "" {
portals = append(portals, p)
}
}
return portals, nil
}
// ListPendingTenders retrieves tenders that have not yet been scraped for documents.
// Only returns tenders from Sweden, Finland, and Norway with active deadlines (deadline not yet reached).
func (s *service) ListPendingTenders(ctx context.Context, limit, offset int) (*DocumentScraperListResponse, *response.Meta, error) {
// Only returns tenders whose document URL matches a portal supported by the AI service.
// By default only tenders with an active publication submission window are included; set IncludeExpired to include expired tenders.
func (s *service) ListPendingTenders(ctx context.Context, req DocumentScraperListRequest) (*DocumentScraperListResponse, *response.Meta, error) {
req.Normalize()
if req.CreatedAtFrom != nil && req.CreatedAtTo != nil && *req.CreatedAtFrom > *req.CreatedAtTo {
return nil, nil, ErrInvalidDateRange
}
portals, err := s.fetchScrapePortals(ctx)
if err != nil {
return nil, nil, err
}
if len(portals) == 0 {
s.logger.Info("No scrape portals configured; returning empty pending tender list", map[string]interface{}{})
meta := &response.Meta{Total: 0, Limit: req.Limit, Offset: req.Offset, HasMore: false}
if meta.Limit > 0 {
meta.Page = (req.Offset / meta.Limit) + 1
}
return &DocumentScraperListResponse{Tenders: []DocumentScraperTenderResponse{}}, meta, nil
}
s.logger.Info("Listing pending tenders for document scraper", map[string]interface{}{
"limit": limit,
"offset": offset,
"limit": req.Limit,
"offset": req.Offset,
"created_at_from": req.CreatedAtFrom,
"created_at_to": req.CreatedAtTo,
"include_expired": req.IncludeExpired,
"portal_count": len(portals),
})
now := time.Now().Unix()
filter := tender.PendingDocumentScrapeFilter{
Portals: portals,
CreatedAtFrom: req.CreatedAtFrom,
CreatedAtTo: req.CreatedAtTo,
Limit: req.Limit,
Skip: req.Offset,
}
if !req.IncludeExpired {
now := time.Now().Unix()
filter.MinWindowEnd = &now
}
tenders, hasMore, err := s.tenderRepo.GetPendingDocumentScrapeTenders(ctx, documentScraperCountryCodes, now, limit, offset)
tenders, hasMore, err := s.tenderRepo.GetPendingDocumentScrapeTenders(ctx, filter)
if err != nil {
s.logger.Error("Failed to list pending tenders", map[string]interface{}{
"error": err.Error(),
@@ -59,6 +117,11 @@ func (s *service) ListPendingTenders(ctx context.Context, limit, offset int) (*D
return nil, nil, fmt.Errorf("failed to list pending tenders: %w", err)
}
if !req.IncludeExpired {
now := time.Now().Unix()
tenders = filterActivePublicationWindow(tenders, now)
}
result := &DocumentScraperListResponse{
Tenders: make([]DocumentScraperTenderResponse, len(tenders)),
}
@@ -69,12 +132,12 @@ func (s *service) ListPendingTenders(ctx context.Context, limit, offset int) (*D
meta := &response.Meta{
Total: len(tenders),
Limit: limit,
Offset: offset,
Limit: req.Limit,
Offset: req.Offset,
HasMore: hasMore,
}
if meta.Limit > 0 {
meta.Page = (offset / meta.Limit) + 1
meta.Page = (req.Offset / meta.Limit) + 1
}
s.logger.Info("Pending tenders retrieved successfully", map[string]interface{}{
@@ -86,12 +149,17 @@ func (s *service) ListPendingTenders(ctx context.Context, limit, offset int) (*D
}
// GetTenderByNoticeID retrieves a specific tender by its notice publication ID.
// Only returns tenders from Sweden, Finland, and Norway with active deadlines (deadline not yet reached).
// Only returns tenders with an active publication submission window whose document URL matches a supported scrape portal.
func (s *service) GetTenderByNoticeID(ctx context.Context, noticeID string) (*DocumentScraperTenderResponse, error) {
s.logger.Info("Getting tender by notice ID for document scraper", map[string]interface{}{
"notice_id": noticeID,
})
portals, err := s.fetchScrapePortals(ctx)
if err != nil {
return nil, err
}
t, err := s.tenderRepo.GetByNoticePublicationID(ctx, noticeID)
if err != nil {
s.logger.Error("Failed to get tender by notice ID", map[string]interface{}{
@@ -106,12 +174,13 @@ func (s *service) GetTenderByNoticeID(ctx context.Context, noticeID string) (*Do
}
now := time.Now().Unix()
if !isDocumentScraperCountry(t.CountryCode) || t.TenderDeadline <= now {
s.logger.Info("Tender filtered out: unsupported country or deadline passed", map[string]interface{}{
"notice_id": noticeID,
"country_code": t.CountryCode,
"tender_deadline": t.TenderDeadline,
"now": now,
if !t.HasActivePublicationWindow(now) || !matchesScrapePortals(documentURL(t), portals) {
s.logger.Info("Tender filtered out: unsupported portal or publication window closed", map[string]interface{}{
"notice_id": noticeID,
"publication_date": t.PublicationDate,
"submission_deadline": t.SubmissionDeadline,
"publication_deadline": t.PublicationSubmissionDeadline(),
"now": now,
})
return nil, fmt.Errorf("tender not found")
}
@@ -125,3 +194,16 @@ func (s *service) GetTenderByNoticeID(ctx context.Context, noticeID string) (*Do
return result, nil
}
func filterActivePublicationWindow(tenders []tender.Tender, now int64) []tender.Tender {
if len(tenders) == 0 {
return tenders
}
active := make([]tender.Tender, 0, len(tenders))
for i := range tenders {
if tenders[i].HasActivePublicationWindow(now) {
active = append(active, tenders[i])
}
}
return active
}
+2 -11
View File
@@ -1,10 +1,6 @@
package document_scraper
import (
"strings"
"tm/internal/tender"
)
import "tm/internal/tender"
// BuildDocumentScraperTenderResponse maps a stored tender to the scraper API contract.
func BuildDocumentScraperTenderResponse(t *tender.Tender) *DocumentScraperTenderResponse {
@@ -12,15 +8,10 @@ func BuildDocumentScraperTenderResponse(t *tender.Tender) *DocumentScraperTender
return &DocumentScraperTenderResponse{}
}
documentURL := strings.TrimSpace(t.DocumentURI)
if documentURL == "" {
documentURL = strings.TrimSpace(t.TenderURL)
}
return &DocumentScraperTenderResponse{
ContractFolderID: t.ContractFolderID,
NoticePublicationID: t.NoticePublicationID,
DocumentURL: documentURL,
DocumentURL: documentURL(t),
Title: t.Title,
Description: t.Description,
}
+33 -1
View File
@@ -1,6 +1,10 @@
package inquiry
import "errors"
import (
"errors"
"fmt"
"strings"
)
// Inquiry-specific errors
var (
@@ -25,6 +29,34 @@ func (e *DuplicateInquiryError) Error() string {
}
// NewDuplicateInquiryError creates a new duplicate inquiry error
// InvalidStatusTransitionError is returned when an inquiry status change is not allowed.
type InvalidStatusTransitionError struct {
CurrentStatus string
NewStatus string
Allowed []string
}
func (e *InvalidStatusTransitionError) Error() string {
if len(e.Allowed) == 0 {
return fmt.Sprintf("Inquiry status is already %s and cannot be changed", e.CurrentStatus)
}
return fmt.Sprintf(
"Cannot change inquiry status from %s to %s. Allowed next statuses: %s",
e.CurrentStatus,
e.NewStatus,
strings.Join(e.Allowed, ", "),
)
}
// NewInvalidStatusTransitionError creates a status transition error with allowed next statuses.
func NewInvalidStatusTransitionError(currentStatus, newStatus string, allowed []string) *InvalidStatusTransitionError {
return &InvalidStatusTransitionError{
CurrentStatus: currentStatus,
NewStatus: newStatus,
Allowed: allowed,
}
}
func NewDuplicateInquiryError(inquiryType, existingID, status string, createdAt int64) *DuplicateInquiryError {
var message string
if inquiryType == "email" {
+7 -1
View File
@@ -1,6 +1,7 @@
package inquiry
import (
"strings"
"tm/pkg/response"
"tm/pkg/security"
@@ -21,7 +22,7 @@ type CreateInquiryForm struct {
// UpdateInquiryStatusForm represents the data required to update inquiry status
type UpdateInquiryStatusForm struct {
Status string `json:"status" valid:"required,in(pending|reviewed|approved|rejected)" example:"reviewed"` // New status
Reason string `json:"reason" valid:"required,length(5|500)" example:"Initial review completed"` // Reason for status change
Reason string `json:"reason" valid:"optional,length(1|500)" example:"Initial review completed"` // Reason for status change
Description string `json:"description" valid:"optional,length(0|1000)" example:"Additional notes about the review"` // Optional description
}
@@ -125,6 +126,11 @@ func (f *CreateInquiryForm) ValidateAndSanitize(xssPolicy *security.XSSPolicy) e
// ValidateAndSanitize validates and sanitizes the status update form
func (f *UpdateInquiryStatusForm) ValidateAndSanitize(xssPolicy *security.XSSPolicy) error {
f.Status = strings.ToLower(strings.TrimSpace(f.Status))
if strings.TrimSpace(f.Reason) == "" {
f.Reason = "Status updated to " + f.Status
}
var err error
// Sanitize Reason
+35
View File
@@ -0,0 +1,35 @@
package inquiry
import (
"encoding/json"
"testing"
"tm/pkg/security"
)
func TestUpdateInquiryStatusFormValidation(t *testing.T) {
xssPolicy := security.NewXSSPolicy()
cases := []struct {
name string
body string
wantErr bool
}{
{"status only", `{"status":"reviewed"}`, false},
{"capitalized status", `{"status":"Reviewed"}`, false},
{"valid with reason", `{"status":"reviewed","reason":"Initial review completed"}`, false},
{"user payload", `{"status":"reviewed","reason":"test","description":"test"}`, false},
{"invalid status", `{"status":"done"}`, true},
}
for _, tc := range cases {
var form UpdateInquiryStatusForm
if err := json.Unmarshal([]byte(tc.body), &form); err != nil {
t.Fatalf("%s: unmarshal failed: %v", tc.name, err)
}
err := form.ValidateAndSanitize(xssPolicy)
if (err != nil) != tc.wantErr {
t.Errorf("%s: got err=%v wantErr=%v", tc.name, err, tc.wantErr)
}
}
}
+18 -21
View File
@@ -42,14 +42,13 @@ func NewHandler(service Service, logger logger.Logger, hcaptchaVerifier hcaptcha
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Router /api/v1/inquiries [post]
func (h *Handler) CreateInquiry(c echo.Context) error {
form, err := response.Parse[CreateInquiryForm](c)
if err != nil {
return response.BadRequest(c, "Invalid request format", "")
var form CreateInquiryForm
if err := c.Bind(&form); err != nil {
return handleBindError(c, err)
}
// Validate and sanitize form
if err := form.ValidateAndSanitize(h.xssPolicy); err != nil {
return response.ValidationError(c, err.Error(), "")
return handleValidationError(c, err)
}
// Verify hCaptcha token
@@ -64,7 +63,7 @@ func (h *Handler) CreateInquiry(c echo.Context) error {
}
// Call service
inquiry, err := h.service.Create(c.Request().Context(), form)
inquiry, err := h.service.Create(c.Request().Context(), &form)
if err != nil {
// Check if it's a duplicate inquiry error
if duplicateErr, ok := err.(*DuplicateInquiryError); ok {
@@ -110,7 +109,7 @@ func (h *Handler) GetInquiryByID(c echo.Context) error {
inquiry, err := h.service.GetByID(c.Request().Context(), idStr)
if err != nil {
return response.NotFound(c, "Inquiry not found")
return handleServiceError(c, err, "Failed to retrieve inquiry")
}
return response.Success(c, inquiry, "Inquiry retrieved successfully")
@@ -133,14 +132,13 @@ func (h *Handler) GetInquiryByID(c echo.Context) error {
func (h *Handler) UpdateInquiryStatus(c echo.Context) error {
idStr := c.Param("id")
form, err := response.Parse[UpdateInquiryStatusForm](c)
if err != nil {
return response.BadRequest(c, "Invalid request format", "")
var form UpdateInquiryStatusForm
if err := c.Bind(&form); err != nil {
return handleBindError(c, err)
}
// Validate and sanitize form
if err := form.ValidateAndSanitize(h.xssPolicy); err != nil {
return response.ValidationError(c, err.Error(), "")
return handleValidationError(c, err)
}
// Get user ID from context (assuming it's set by auth middleware)
@@ -148,9 +146,9 @@ func (h *Handler) UpdateInquiryStatus(c echo.Context) error {
// changedBy := c.Get("user_id").(string)
// Call service
inquiry, err := h.service.UpdateStatus(c.Request().Context(), idStr, form, changedBy)
inquiry, err := h.service.UpdateStatus(c.Request().Context(), idStr, &form, changedBy)
if err != nil {
return response.BadRequest(c, err.Error(), "")
return handleServiceError(c, err, "Failed to update inquiry status")
}
return response.Success(c, inquiry, "Inquiry status updated successfully")
@@ -174,14 +172,13 @@ func (h *Handler) UpdateInquiryStatus(c echo.Context) error {
// @Failure 500 {object} response.APIResponse "Internal server error"
// @Router /admin/v1/inquiries [get]
func (h *Handler) SearchInquiries(c echo.Context) error {
form, err := response.Parse[SearchInquiriesForm](c)
if err != nil {
return response.BadRequest(c, "Invalid request format", "")
var form SearchInquiriesForm
if err := c.Bind(&form); err != nil {
return handleBindError(c, err)
}
// Validate and sanitize form
if err := form.ValidateAndSanitize(h.xssPolicy); err != nil {
return response.ValidationError(c, err.Error(), "")
return handleValidationError(c, err)
}
// Call service
@@ -190,7 +187,7 @@ func (h *Handler) SearchInquiries(c echo.Context) error {
return response.PaginationBadRequest(c, err)
}
inquiries, err := h.service.Search(c.Request().Context(), form, pagination)
inquiries, err := h.service.Search(c.Request().Context(), &form, pagination)
if err != nil {
if response.IsListPaginationError(err) {
return response.PaginationBadRequest(c, err)
@@ -220,7 +217,7 @@ func (h *Handler) DeleteInquiry(c echo.Context) error {
err := h.service.Delete(c.Request().Context(), id)
if err != nil {
return response.BadRequest(c, err.Error(), "")
return handleServiceError(c, err, "Failed to delete inquiry")
}
return response.Success(c, map[string]interface{}{
+38
View File
@@ -0,0 +1,38 @@
package inquiry
import (
"errors"
"strings"
"tm/pkg/response"
"github.com/labstack/echo/v4"
)
func handleBindError(c echo.Context, err error) error {
return response.BadRequest(c, "Invalid request format", err.Error())
}
func handleValidationError(c echo.Context, err error) error {
return response.ValidationError(c, "Please fix the validation errors", FormatValidationErrors(err))
}
func handleServiceError(c echo.Context, err error, fallback string) error {
if err == nil {
return nil
}
if errors.Is(err, ErrInquiryNotFound) {
return response.NotFound(c, "Inquiry not found")
}
var transitionErr *InvalidStatusTransitionError
if errors.As(err, &transitionErr) {
return response.BadRequest(c, transitionErr.Error(), "")
}
if strings.Contains(err.Error(), "cannot be updated") {
return response.BadRequest(c, err.Error(), "")
}
return response.InternalServerError(c, fallback)
}
+1 -1
View File
@@ -105,7 +105,7 @@ func (r *inquiryRepository) GetByID(ctx context.Context, id string) (*Inquiry, e
inquiry, err := r.ormRepo.FindByID(ctx, id)
if err != nil {
if errors.Is(err, orm.ErrDocumentNotFound) {
return nil, errors.New("inquiry not found")
return nil, ErrInquiryNotFound
}
r.logger.Error("Failed to get inquiry by ID", map[string]interface{}{
"error": err.Error(),
+2 -3
View File
@@ -251,17 +251,16 @@ func (s *inquiryService) validateStatusTransition(ctx context.Context, id string
allowedNextStatuses, exists := allowedTransitions[currentStatus]
if !exists {
return errors.New("invalid current status")
return fmt.Errorf("inquiry has unknown status %q and cannot be updated", currentStatus)
}
// Check if transition is allowed
for _, allowedStatus := range allowedNextStatuses {
if allowedStatus == newStatus {
return nil
}
}
return errors.New("invalid status transition: cannot change from " + currentStatus + " to " + newStatus)
return NewInvalidStatusTransitionError(currentStatus, newStatus, allowedNextStatuses)
}
// Search searches inquiries with search and filters
+120
View File
@@ -0,0 +1,120 @@
package inquiry
import (
"fmt"
"regexp"
"strings"
"github.com/asaskevich/govalidator"
)
var (
lengthRulePattern = regexp.MustCompile(`length\((\d+)\|(\d+)\)`)
inRulePattern = regexp.MustCompile(`in\(([^)]+)\)`)
)
// FormatValidationErrors converts govalidator errors into user-facing messages.
func FormatValidationErrors(err error) string {
if err == nil {
return ""
}
errs, ok := err.(govalidator.Errors)
if !ok {
return humanizeValidationError(err.Error())
}
messages := make([]string, 0, len(errs))
for _, item := range errs {
messages = append(messages, humanizeValidationError(item.Error()))
}
return strings.Join(messages, "; ")
}
func humanizeValidationError(raw string) string {
field, rule, ok := parseGovalidatorError(raw)
if !ok {
return raw
}
label := inquiryFieldLabel(field)
lowerRule := strings.ToLower(rule)
switch {
case strings.Contains(lowerRule, "required"):
return label + " is required"
case strings.Contains(lowerRule, "email"):
return label + " must be a valid email address"
case strings.Contains(lowerRule, "length("):
if min, max, ok := parseLengthRule(rule); ok {
if min == max {
return fmt.Sprintf("%s must be exactly %d characters", label, min)
}
return fmt.Sprintf("%s must be between %d and %d characters", label, min, max)
}
return label + " has an invalid length"
case strings.Contains(lowerRule, "in("):
if values, ok := parseInRule(rule); ok {
return fmt.Sprintf("%s must be one of: %s", label, strings.Join(values, ", "))
}
return label + " has an invalid value"
case strings.Contains(lowerRule, "range("):
return label + " is out of the allowed range"
default:
return label + " is invalid"
}
}
func parseGovalidatorError(raw string) (field, rule string, ok bool) {
parts := strings.SplitN(raw, ": ", 2)
if len(parts) != 2 {
return "", "", false
}
return parts[0], parts[1], true
}
func parseLengthRule(rule string) (min, max int, ok bool) {
matches := lengthRulePattern.FindStringSubmatch(rule)
if len(matches) != 3 {
return 0, 0, false
}
_, err := fmt.Sscanf(matches[1]+" "+matches[2], "%d %d", &min, &max)
return min, max, err == nil
}
func parseInRule(rule string) (values []string, ok bool) {
matches := inRulePattern.FindStringSubmatch(rule)
if len(matches) != 2 {
return nil, false
}
for _, value := range strings.Split(matches[1], "|") {
values = append(values, value)
}
return values, len(values) > 0
}
func inquiryFieldLabel(field string) string {
field = strings.TrimPrefix(field, "CreateInquiryForm.")
field = strings.TrimPrefix(field, "UpdateInquiryStatusForm.")
field = strings.TrimPrefix(field, "SearchInquiriesForm.")
labels := map[string]string{
"FullName": "Full name",
"CompanyName": "Company name",
"WorkEmail": "Work email",
"PhoneNumber": "Phone number",
"HCaptchaToken": "hCaptcha token",
"Status": "Status",
"Reason": "Reason",
"Description": "Description",
"Search": "Search query",
"Limit": "Limit",
"Offset": "Offset",
"SortBy": "Sort field",
"SortOrder": "Sort order",
}
if label, exists := labels[field]; exists {
return label
}
return field
}
+39
View File
@@ -0,0 +1,39 @@
package inquiry
import (
"testing"
"github.com/asaskevich/govalidator"
)
func TestFormatValidationErrors(t *testing.T) {
form := UpdateInquiryStatusForm{
Status: "done",
Reason: "",
}
_, err := govalidator.ValidateStruct(form)
if err == nil {
t.Fatal("expected validation error")
}
message := FormatValidationErrors(err)
if message == "" {
t.Fatal("expected formatted validation message")
}
if message == err.Error() {
t.Fatalf("expected humanized message, got raw error: %s", message)
}
}
func TestInvalidStatusTransitionErrorMessage(t *testing.T) {
err := NewInvalidStatusTransitionError(StatusPending, StatusApproved, []string{StatusReviewed, StatusRejected})
expected := "Cannot change inquiry status from pending to approved. Allowed next statuses: reviewed, rejected"
if err.Error() != expected {
t.Fatalf("got %q, want %q", err.Error(), expected)
}
finalErr := NewInvalidStatusTransitionError(StatusApproved, StatusReviewed, nil)
if finalErr.Error() != "Inquiry status is already approved and cannot be changed" {
t.Fatalf("unexpected final status message: %s", finalErr.Error())
}
}
+22
View File
@@ -25,8 +25,30 @@ type DeliveryChannel string
const (
DeliveryChannelPush DeliveryChannel = "push"
DeliveryChannelEmail DeliveryChannel = "email"
DeliveryChannelInApp DeliveryChannel = "in_app"
)
// Persisted event types for stored notifications (admin list / notification center).
const (
EventTypeInApp = "in_app"
EventTypeEmail = "email"
EventTypePush = "push"
)
// persistedEventType maps a delivery channel to the event_type stored in MongoDB.
func persistedEventType(channel DeliveryChannel) (string, bool) {
switch channel {
case DeliveryChannelEmail:
return EventTypeEmail, true
case DeliveryChannelPush:
return EventTypePush, true
case DeliveryChannelInApp:
return EventTypeInApp, true
default:
return "", false
}
}
// DeliveryStatus represents the delivery status of notification
type NotificationStatus string
+9
View File
@@ -30,6 +30,7 @@ type SearchForm struct {
EventType string `query:"event_type" valid:"optional"`
Type string `query:"type" valid:"optional"`
Recipient []string `query:"recipient" json:"recipient" valid:"optional"`
UserID string `query:"user_id" valid:"optional"`
Seen *bool `query:"seen" valid:"optional"`
Priority string `query:"priority" valid:"optional"`
}
@@ -45,6 +46,14 @@ func (f *SearchForm) ResolvedSearch() string {
return ""
}
// ResolvedRecipients returns the user IDs to filter by from user_id or recipient query params.
func (f *SearchForm) ResolvedRecipients() []string {
if id := strings.TrimSpace(f.UserID); id != "" {
return []string{id}
}
return f.Recipient
}
type NotificationListResponse struct {
Notifications []*NotificationResponse `json:"notifications"`
Meta *response.Meta `json:"meta"`
+6 -1
View File
@@ -49,7 +49,7 @@ func (h *NotificationHandler) Send(c echo.Context) error {
// Send notification
go h.service.Send(context.Background(), &req)
return response.Created(c, nil, "Notification sent successfully")
return response.Created(c, nil, "Notification created successfully")
}
// GetNotifications gets notifications
@@ -65,6 +65,7 @@ func (h *NotificationHandler) Send(c echo.Context) error {
// @Param event_type query string false "Event type filter"
// @Param type query string false "Type filter"
// @Param recipient query string false "Recipient filter"
// @Param user_id query string false "User ID filter"
// @Param seen query bool false "Seen filter"
// @Param priority query string false "Priority filter"
// @Param limit query int false "Limit results"
@@ -147,6 +148,8 @@ func (h *NotificationHandler) AdminNotifications(c echo.Context) error {
form.Recipient = []string{userID}
form.Status = "sent"
form.EventType = EventTypeInApp
form.Method = ""
pagination, err := response.NewPagination(c)
if err != nil {
@@ -265,6 +268,8 @@ func (h *NotificationHandler) CustomerNotifications(c echo.Context) error {
form.Recipient = []string{userID}
form.Status = "sent"
form.EventType = EventTypeInApp
form.Method = ""
pagination, err := response.NewPagination(c)
if err != nil {
+62 -5
View File
@@ -58,6 +58,7 @@ type Repository interface {
Update(ctx context.Context, notification *Notification) error
MarkAsSeen(ctx context.Context, notificationID, userID string) error
MarkAllAsSeen(ctx context.Context, userID string) error
MarkDueScheduledAsSent(ctx context.Context, dueBefore int64) (int64, error)
}
type notificationRepository struct {
@@ -74,6 +75,11 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Re
*orm.NewIndex("status_idx", bson.D{{Key: "status", Value: 1}}),
*orm.NewIndex("seen_idx", bson.D{{Key: "seen", Value: 1}}),
*orm.NewIndex("created_at_idx", bson.D{{Key: "created_at", Value: -1}}),
*orm.NewIndex("scheduled_delivery_idx", bson.D{
{Key: "status", Value: 1},
{Key: "is_scheduled", Value: 1},
{Key: "schedule_at", Value: 1},
}),
}
err := mongoManager.CreateIndexes("notifications", indexes)
@@ -119,13 +125,16 @@ func (r *notificationRepository) GetByID(ctx context.Context, id string) (*Notif
// GetByUserID retrieves notifications with optional user filters
func (r *notificationRepository) GetByUserID(ctx context.Context, userIDs []string, search, status string, seen *bool, priority, eventType, notificationType string, pagination *response.Pagination) (*orm.PaginatedResult[Notification], error) {
filter := bson.M{}
andConditions := bson.A{}
if q := strings.TrimSpace(search); q != "" {
pattern := regexp.QuoteMeta(q)
filter["$or"] = bson.A{
bson.M{"title": bson.M{"$regex": pattern, "$options": "i"}},
bson.M{"message": bson.M{"$regex": pattern, "$options": "i"}},
}
andConditions = append(andConditions, bson.M{
"$or": bson.A{
bson.M{"title": bson.M{"$regex": pattern, "$options": "i"}},
bson.M{"message": bson.M{"$regex": pattern, "$options": "i"}},
},
})
}
if len(userIDs) == 1 {
@@ -147,13 +156,31 @@ func (r *notificationRepository) GetByUserID(ctx context.Context, userIDs []stri
}
if eventType != "" {
filter["event_type"] = eventType
if eventType == EventTypeInApp {
andConditions = append(andConditions, bson.M{
"$or": bson.A{
bson.M{"event_type": EventTypeInApp},
bson.M{"event_type": ""},
bson.M{"event_type": bson.M{"$exists": false}},
},
})
} else {
filter["event_type"] = eventType
}
}
if notificationType != "" {
filter["type"] = notificationType
}
if len(andConditions) == 1 {
for key, value := range andConditions[0].(bson.M) {
filter[key] = value
}
} else if len(andConditions) > 1 {
filter["$and"] = andConditions
}
mongoPagination, err := orm.BuildListPagination(
pagination.Limit,
pagination.Offset,
@@ -258,3 +285,33 @@ func (r *notificationRepository) MarkAllAsSeen(ctx context.Context, userID strin
return nil
}
// MarkDueScheduledAsSent updates pending scheduled notifications whose delivery time has passed.
func (r *notificationRepository) MarkDueScheduledAsSent(ctx context.Context, dueBefore int64) (int64, error) {
filter := bson.M{
"status": string(DeliveryStatusPending),
"is_scheduled": true,
"schedule_at": bson.M{
"$gt": 0,
"$lte": dueBefore,
},
}
update := bson.M{
"$set": bson.M{
"status": string(DeliveryStatusSent),
"updated_at": time.Now().Unix(),
},
}
result, err := r.collection.UpdateMany(ctx, filter, update)
if err != nil {
r.logger.Error("Failed to mark due scheduled notifications as sent", map[string]interface{}{
"error": err.Error(),
"due_before": dueBefore,
})
return 0, err
}
return result.ModifiedCount, nil
}
+57 -8
View File
@@ -5,6 +5,7 @@ import (
"errors"
"slices"
"strings"
"time"
"tm/internal/customer"
"tm/internal/user"
@@ -98,13 +99,27 @@ func (s *notificationService) Send(ctx context.Context, req *NotificationRequest
}
func (s *notificationService) sendToRecipient(ctx context.Context, recipient notificationRecipient, req *NotificationRequest, link, image string) {
if err := s.persistNotification(ctx, recipient, req, link, image); err != nil {
// Always persist an in-app record so every send appears in the notification center.
if err := s.persistNotification(ctx, recipient, req, link, image, DeliveryChannelInApp); err != nil {
s.logger.Error("Failed to persist in-app notification", map[string]interface{}{
"error": err.Error(),
"user_id": recipient.UserID,
})
}
for _, channel := range req.Channels {
if channel == DeliveryChannelInApp {
continue
}
if err := s.persistNotification(ctx, recipient, req, link, image, channel); err != nil {
s.logger.Error("Failed to persist notification", map[string]interface{}{
"error": err.Error(),
"user_id": recipient.UserID,
"channel": channel,
})
}
}
metadata := map[string]any{
"tender": strings.TrimSpace(req.Tender),
}
@@ -148,13 +163,26 @@ func (s *notificationService) sendToRecipient(ctx context.Context, recipient not
}
}
func (s *notificationService) persistNotification(ctx context.Context, recipient notificationRecipient, req *NotificationRequest, link, image string) error {
methods := make(map[string]string)
if slices.Contains(req.Channels, DeliveryChannelEmail) && recipient.Email != "" {
methods["email"] = recipient.Email
func (s *notificationService) persistNotification(ctx context.Context, recipient notificationRecipient, req *NotificationRequest, link, image string, channel DeliveryChannel) error {
eventType, ok := persistedEventType(channel)
if !ok {
return nil
}
if slices.Contains(req.Channels, DeliveryChannelPush) && len(recipient.DeviceTokens) > 0 {
methods := map[string]string{}
switch channel {
case DeliveryChannelEmail:
if recipient.Email == "" {
return nil
}
methods["email"] = recipient.Email
case DeliveryChannelPush:
if len(recipient.DeviceTokens) == 0 {
return nil
}
methods["push"] = recipient.DeviceTokens[0]
case DeliveryChannelInApp:
methods["in_app"] = "true"
}
status := string(DeliveryStatusSent)
@@ -170,6 +198,7 @@ func (s *notificationService) persistNotification(ctx context.Context, recipient
Image: image,
Priority: string(req.Priority),
Type: string(req.Type),
EventType: eventType,
Status: status,
Methods: methods,
Metadata: map[string]any{
@@ -183,6 +212,8 @@ func (s *notificationService) persistNotification(ctx context.Context, recipient
}
func (s *notificationService) GetNotifications(ctx context.Context, req *SearchForm, pagination *response.Pagination) (*NotificationListResponse, error) {
s.processDueScheduled(ctx)
search := req.ResolvedSearch()
s.logger.Info("Getting notifications", map[string]interface{}{
@@ -191,14 +222,14 @@ func (s *notificationService) GetNotifications(ctx context.Context, req *SearchF
"method": req.Method,
"event_type": req.EventType,
"type": req.Type,
"recipient": req.Recipient,
"recipient": req.ResolvedRecipients(),
"seen": req.Seen,
"priority": req.Priority,
"limit": pagination.Limit,
"offset": pagination.Offset,
})
page, err := s.repository.GetByUserID(ctx, req.Recipient, search, req.Status, req.Seen, req.Priority, req.EventType, req.Type, pagination)
page, err := s.repository.GetByUserID(ctx, req.ResolvedRecipients(), search, req.Status, req.Seen, req.Priority, req.EventType, req.Type, pagination)
if err != nil {
s.logger.Error("Failed to get notifications", map[string]interface{}{
"error": err.Error(),
@@ -261,6 +292,8 @@ func (s *notificationService) GetNotifications(ctx context.Context, req *SearchF
}
func (s *notificationService) GetNotification(ctx context.Context, notificationID string) (*NotificationResponse, error) {
s.processDueScheduled(ctx)
s.logger.Info("Getting notification", map[string]interface{}{
"notification_id": notificationID,
})
@@ -464,3 +497,19 @@ func (s *notificationService) getCustomers(ctx context.Context, values []string,
return recipients, nil
}
func (s *notificationService) processDueScheduled(ctx context.Context) {
count, err := s.repository.MarkDueScheduledAsSent(ctx, time.Now().Unix())
if err != nil {
s.logger.Error("Failed to process due scheduled notifications", map[string]interface{}{
"error": err.Error(),
})
return
}
if count > 0 {
s.logger.Info("Processed due scheduled notifications", map[string]interface{}{
"count": count,
})
}
}
+78
View File
@@ -0,0 +1,78 @@
package tender
import (
"strings"
"go.mongodb.org/mongo-driver/v2/bson"
)
const aiProcedurePrefix = "PROC_"
// FormatAIProcedureRef builds the AI service tender reference:
// PROC_{contract_folder_id}/{notice_publication_id}
func FormatAIProcedureRef(contractFolderID, noticePublicationID string) string {
contractFolderID = strings.TrimSpace(contractFolderID)
noticePublicationID = strings.TrimSpace(noticePublicationID)
if contractFolderID == "" || noticePublicationID == "" {
return ""
}
return aiProcedurePrefix + contractFolderID + "/" + noticePublicationID
}
// ProcedureReference identifies a tender using AI procedure coordinates.
type ProcedureReference struct {
Ref string
ContractFolderID string
NoticePublicationID string
}
// ParseAIProcedureRef parses PROC_{contract_folder_id}/{notice_publication_id}.
func ParseAIProcedureRef(ref string) (contractFolderID, noticePublicationID string, ok bool) {
ref = strings.TrimSpace(ref)
if ref == "" {
return "", "", false
}
withoutPrefix := ref
if len(ref) >= len(aiProcedurePrefix) && strings.EqualFold(ref[:len(aiProcedurePrefix)], aiProcedurePrefix) {
withoutPrefix = ref[len(aiProcedurePrefix):]
}
slash := strings.Index(withoutPrefix, "/")
if slash <= 0 || slash >= len(withoutPrefix)-1 {
return "", "", false
}
contractFolderID = strings.TrimSpace(withoutPrefix[:slash])
noticePublicationID = strings.TrimSpace(withoutPrefix[slash+1:])
if contractFolderID == "" || noticePublicationID == "" {
return "", "", false
}
return contractFolderID, noticePublicationID, true
}
func tenderMatchesProcedureRef(t Tender, contractFolderID, noticePublicationID string) bool {
if strings.TrimSpace(t.ContractFolderID) != strings.TrimSpace(contractFolderID) {
return false
}
if t.NoticePublicationID == noticePublicationID {
return true
}
for _, related := range t.RelatedNoticePublicationIDs {
if related == noticePublicationID {
return true
}
}
return false
}
// IsMongoObjectIDRef reports whether ref is a 24-char hex MongoDB ObjectId.
func IsMongoObjectIDRef(ref string) bool {
ref = strings.TrimSpace(ref)
if len(ref) != 24 {
return false
}
_, err := bson.ObjectIDFromHex(ref)
return err == nil
}
+124
View File
@@ -0,0 +1,124 @@
package tender
import (
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
)
func TestParseAIProcedureRef(t *testing.T) {
tests := []struct {
ref string
folder string
notice string
ok bool
}{
{
ref: "PROC_055329e0-3d8f-4eeb-946a-85a451f2e36b/00423458-2026",
folder: "055329e0-3d8f-4eeb-946a-85a451f2e36b",
notice: "00423458-2026",
ok: true,
},
{
ref: "proc_d3d34785-e10f-4947-8d00-701816777bf5/00332053-2026",
folder: "d3d34785-e10f-4947-8d00-701816777bf5",
notice: "00332053-2026",
ok: true,
},
{ref: "", ok: false},
{ref: "507f1f77bcf86cd799439011", ok: false},
{ref: "PROC_no-slash", ok: false},
}
for _, tt := range tests {
folder, notice, ok := ParseAIProcedureRef(tt.ref)
if ok != tt.ok {
t.Fatalf("ParseAIProcedureRef(%q) ok=%v want %v", tt.ref, ok, tt.ok)
}
if !tt.ok {
continue
}
if folder != tt.folder || notice != tt.notice {
t.Fatalf("ParseAIProcedureRef(%q) = (%q, %q) want (%q, %q)", tt.ref, folder, notice, tt.folder, tt.notice)
}
}
}
func TestFormatAIProcedureRef(t *testing.T) {
got := FormatAIProcedureRef("055329e0-3d8f-4eeb-946a-85a451f2e36b", "00423458-2026")
want := "PROC_055329e0-3d8f-4eeb-946a-85a451f2e36b/00423458-2026"
if got != want {
t.Fatalf("FormatAIProcedureRef() = %q want %q", got, want)
}
}
func TestMapProcedureReferencesToTenders(t *testing.T) {
candidates := []Tender{
{
ContractFolderID: "folder-1",
NoticePublicationID: "notice-new",
RelatedNoticePublicationIDs: []string{"notice-old"},
},
{
ContractFolderID: "folder-1",
NoticePublicationID: "notice-legacy",
},
{
ContractFolderID: "folder-2",
NoticePublicationID: "notice-2",
},
}
refs := []ProcedureReference{
{Ref: "PROC_folder-1/notice-old", ContractFolderID: "folder-1", NoticePublicationID: "notice-old"},
{Ref: "PROC_folder-1/notice-legacy", ContractFolderID: "folder-1", NoticePublicationID: "notice-legacy"},
{Ref: "PROC_folder-2/notice-2", ContractFolderID: "folder-2", NoticePublicationID: "notice-2"},
}
got := mapProcedureReferencesToTenders(candidates, refs)
if len(got) != 3 {
t.Fatalf("mapProcedureReferencesToTenders() len = %d, want 3", len(got))
}
if got["PROC_folder-1/notice-old"].NoticePublicationID != "notice-new" {
t.Fatalf("expected notice-old ref to resolve to latest merged tender, got %+v", got["PROC_folder-1/notice-old"])
}
if got["PROC_folder-1/notice-legacy"].NoticePublicationID != "notice-legacy" {
t.Fatalf("expected legacy ref to resolve to legacy tender, got %+v", got["PROC_folder-1/notice-legacy"])
}
if got["PROC_folder-2/notice-2"].NoticePublicationID != "notice-2" {
t.Fatalf("expected folder-2 ref to resolve, got %+v", got["PROC_folder-2/notice-2"])
}
}
func TestMapProcedureReferencesToTendersUsesFolderCandidatesWhenNoticeLookupMisses(t *testing.T) {
// Notice-old is only present on the merged tender loaded via contract_folder_id.
candidates := []Tender{
{
ContractFolderID: "folder-1",
NoticePublicationID: "notice-new",
RelatedNoticePublicationIDs: []string{"notice-old"},
},
}
refs := []ProcedureReference{
{Ref: "PROC_folder-1/notice-old", ContractFolderID: "folder-1", NoticePublicationID: "notice-old"},
}
got := mapProcedureReferencesToTenders(candidates, refs)
if len(got) != 1 {
t.Fatalf("mapProcedureReferencesToTenders() len = %d, want 1", len(got))
}
if got["PROC_folder-1/notice-old"].NoticePublicationID != "notice-new" {
t.Fatalf("expected folder candidate to resolve notice-old ref, got %+v", got["PROC_folder-1/notice-old"])
}
}
func TestMergeTenderCandidatesDedupesByMongoID(t *testing.T) {
first := Tender{ContractFolderID: "folder-1", NoticePublicationID: "notice-a"}
first.ID, _ = bson.ObjectIDFromHex("507f1f77bcf86cd799439011")
second := Tender{ContractFolderID: "folder-1", NoticePublicationID: "notice-b"}
second.ID = first.ID
got := mergeTenderCandidates([]Tender{first}, []Tender{second})
if len(got) != 1 {
t.Fatalf("mergeTenderCandidates() len = %d, want 1", len(got))
}
}
+120
View File
@@ -0,0 +1,120 @@
package tender
import (
"context"
"errors"
"fmt"
"strings"
"time"
"tm/pkg/ai_summarizer"
)
const (
// contractFoldersWithDocsCacheTTL avoids scanning MinIO on every documents_scraped list request.
// Kept >= resolve timeout so a slow scan is not immediately invalidated for waiters.
contractFoldersWithDocsCacheTTL = 15 * time.Minute
// contractFoldersWithDocsResolveTimeout bounds a single MinIO procedure scan.
contractFoldersWithDocsResolveTimeout = 3 * time.Minute
)
type contractFoldersWithDocsCacheEntry struct {
expiresAt time.Time
folderIDs []string
valid bool
}
// enrichDocumentsScrapedFilter resolves contract_folder_id values that currently have
// documents in MinIO and stores them on the search form for the repository filter.
func (s *tenderService) enrichDocumentsScrapedFilter(ctx context.Context, form *SearchForm) error {
if form == nil || !form.DocumentsScraped {
return nil
}
folderIDs, err := s.cachedContractFolderIDsWithDocuments(ctx)
if err != nil {
s.logger.Error("Failed to resolve tenders with scraped documents from MinIO", map[string]interface{}{
"error": err.Error(),
})
if errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
return fmt.Errorf("failed to resolve tenders with scraped documents: MinIO connection unavailable: %w", err)
}
return fmt.Errorf("failed to resolve tenders with scraped documents: %w", err)
}
form.ContractFolderIDsWithDocuments = folderIDs
return nil
}
func (s *tenderService) cachedContractFolderIDsWithDocuments(ctx context.Context) ([]string, error) {
now := time.Now()
s.contractFoldersCacheMu.Lock()
if s.contractFoldersCache.valid && now.Before(s.contractFoldersCache.expiresAt) {
ids := append([]string(nil), s.contractFoldersCache.folderIDs...)
s.contractFoldersCacheMu.Unlock()
return ids, nil
}
s.contractFoldersCacheMu.Unlock()
v, err, _ := s.contractFoldersCacheGroup.Do("contract-folders-with-docs", func() (interface{}, error) {
s.contractFoldersCacheMu.Lock()
if s.contractFoldersCache.valid && time.Now().Before(s.contractFoldersCache.expiresAt) {
ids := append([]string(nil), s.contractFoldersCache.folderIDs...)
s.contractFoldersCacheMu.Unlock()
return ids, nil
}
s.contractFoldersCacheMu.Unlock()
// Detach from the caller's cancellation so a client disconnect or proxy timeout
// does not abort the shared singleflight scan for every waiter.
resolveCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), contractFoldersWithDocsResolveTimeout)
defer cancel()
procedures, listErr := s.listProceduresWithDocuments(resolveCtx)
if listErr != nil {
return nil, listErr
}
folderIDs := contractFolderIDsFromProcedures(procedures)
s.contractFoldersCacheMu.Lock()
s.contractFoldersCache = contractFoldersWithDocsCacheEntry{
expiresAt: time.Now().Add(contractFoldersWithDocsCacheTTL),
folderIDs: folderIDs,
valid: true,
}
s.contractFoldersCacheMu.Unlock()
return append([]string(nil), folderIDs...), nil
})
if err != nil {
return nil, err
}
ids, ok := v.([]string)
if !ok {
return nil, fmt.Errorf("unexpected contract folder cache value type %T", v)
}
return append([]string(nil), ids...), nil
}
func contractFolderIDsFromProcedures(procedures []ai_summarizer.ProcedureDocumentsSummary) []string {
folderIDs := make([]string, 0, len(procedures))
seen := make(map[string]struct{}, len(procedures))
for _, proc := range procedures {
if proc.DocumentCount <= 0 {
continue
}
id := strings.TrimSpace(proc.ContractFolderID)
if id == "" {
continue
}
if _, ok := seen[id]; ok {
continue
}
seen[id] = struct{}{}
folderIDs = append(folderIDs, id)
}
return folderIDs
}
@@ -0,0 +1,241 @@
package tender
import (
"context"
"errors"
"fmt"
"io"
"sync/atomic"
"testing"
"time"
"go.mongodb.org/mongo-driver/v2/bson"
"tm/pkg/ai_summarizer"
)
type mockProcedureDocumentLister struct {
procedures []ai_summarizer.ProcedureDocumentsSummary
err error
calls atomic.Int32
}
func (m *mockProcedureDocumentLister) ListProceduresWithDocuments(context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error) {
m.calls.Add(1)
if m.err != nil {
return nil, m.err
}
return m.procedures, nil
}
type stubProcedureDocumentStorage struct {
lister interface {
ListProceduresWithDocuments(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error)
}
}
func (s stubProcedureDocumentStorage) GetSummaryFromStorage(context.Context, string, string) (string, error) {
return "", ai_summarizer.ErrObjectNotFound
}
func (s stubProcedureDocumentStorage) GetDocumentSummariesFromStorage(context.Context, string, string) ([]ai_summarizer.DocumentSummary, error) {
return nil, nil
}
func (s stubProcedureDocumentStorage) GetTranslationFromStorage(context.Context, string, string, string) (ai_summarizer.StoredTranslation, error) {
return ai_summarizer.StoredTranslation{}, ai_summarizer.ErrTranslationNotReady
}
func (s stubProcedureDocumentStorage) ListDocuments(context.Context, string, string) ([]ai_summarizer.StoredDocument, error) {
return nil, nil
}
func (s stubProcedureDocumentStorage) DownloadDocument(context.Context, string) (io.ReadCloser, error) {
return nil, ai_summarizer.ErrObjectNotFound
}
func (s stubProcedureDocumentStorage) ListProceduresWithDocuments(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error) {
return s.lister.ListProceduresWithDocuments(ctx)
}
func TestBuildSearchFilterDocumentsScrapedUsesMinIOFolderIDs(t *testing.T) {
repo := &tenderRepository{}
filter := repo.buildSearchFilter(&SearchForm{
DocumentsScraped: true,
ContractFolderIDsWithDocuments: []string{"folder-a", "folder-b"},
})
in, ok := filter["contract_folder_id"].(bson.M)["$in"].([]string)
if !ok {
t.Fatalf("expected contract_folder_id $in filter, got %#v", filter)
}
if len(in) != 2 || in[0] != "folder-a" || in[1] != "folder-b" {
t.Fatalf("unexpected folder ids: %#v", in)
}
if filter["processing_metadata.documents_scraped"] != nil {
t.Fatalf("expected no stale mongo documents_scraped flag, got %#v", filter)
}
}
func TestContractFolderIDsSearchClauseChunksLargeLists(t *testing.T) {
folderIDs := make([]string, maxContractFolderInClause+2)
for i := range folderIDs {
folderIDs[i] = fmt.Sprintf("folder-%d", i)
}
clause := contractFolderIDsSearchClause(folderIDs)
or, ok := clause["$or"].([]bson.M)
if !ok || len(or) != 2 {
t.Fatalf("expected chunked $or filter, got %#v", clause)
}
}
func TestCachedContractFolderIDsWithDocumentsUsesCache(t *testing.T) {
lister := &mockProcedureDocumentLister{
procedures: []ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "folder-1", DocumentCount: 2},
{ContractFolderID: "folder-2", DocumentCount: 0},
{ContractFolderID: "folder-3", DocumentCount: 1},
},
}
svc := &tenderService{
aiSummarizerStorage: stubProcedureDocumentStorage{lister: lister},
}
first, err := svc.cachedContractFolderIDsWithDocuments(context.Background())
if err != nil {
t.Fatalf("first lookup failed: %v", err)
}
if len(first) != 2 || first[0] != "folder-1" || first[1] != "folder-3" {
t.Fatalf("unexpected folder ids: %#v", first)
}
if lister.calls.Load() != 1 {
t.Fatalf("expected one MinIO scan, got %d", lister.calls.Load())
}
second, err := svc.cachedContractFolderIDsWithDocuments(context.Background())
if err != nil {
t.Fatalf("second lookup failed: %v", err)
}
if len(second) != 2 {
t.Fatalf("unexpected cached folder ids: %#v", second)
}
if lister.calls.Load() != 1 {
t.Fatalf("expected cache hit without extra MinIO scan, got %d calls", lister.calls.Load())
}
}
func TestEnrichDocumentsScrapedFilterPropagatesMinIOError(t *testing.T) {
svc := &tenderService{
aiSummarizerStorage: stubProcedureDocumentStorage{
lister: &mockProcedureDocumentLister{
err: ai_summarizer.ErrMinIOUnavailable,
},
},
logger: noopTenderTestLogger{},
}
form := &SearchForm{DocumentsScraped: true}
err := svc.enrichDocumentsScrapedFilter(context.Background(), form)
if err == nil {
t.Fatal("expected MinIO error")
}
if !errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
t.Fatalf("expected wrapped MinIO unavailable error, got %v", err)
}
}
func TestContractFolderIDsFromProceduresDedupesAndSkipsEmpty(t *testing.T) {
ids := contractFolderIDsFromProcedures([]ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "a", DocumentCount: 1},
{ContractFolderID: "a", DocumentCount: 2},
{ContractFolderID: " ", DocumentCount: 1},
{ContractFolderID: "b", DocumentCount: 0},
})
if len(ids) != 1 || ids[0] != "a" {
t.Fatalf("unexpected ids: %#v", ids)
}
}
type scanStartedProcedureLister struct {
mockProcedureDocumentLister
started chan struct{}
}
func (m *scanStartedProcedureLister) ListProceduresWithDocuments(ctx context.Context) ([]ai_summarizer.ProcedureDocumentsSummary, error) {
if m.started != nil {
select {
case <-m.started:
default:
close(m.started)
}
}
time.Sleep(50 * time.Millisecond)
if err := ctx.Err(); err != nil {
return nil, err
}
return m.mockProcedureDocumentLister.ListProceduresWithDocuments(ctx)
}
func TestCachedContractFolderIDsSurvivesCallerCancellation(t *testing.T) {
started := make(chan struct{})
lister := &scanStartedProcedureLister{
mockProcedureDocumentLister: mockProcedureDocumentLister{
procedures: []ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "folder-1", DocumentCount: 1},
},
},
started: started,
}
svc := &tenderService{aiSummarizerStorage: stubProcedureDocumentStorage{lister: lister}}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
errCh := make(chan error, 1)
var ids []string
go func() {
var err error
ids, err = svc.cachedContractFolderIDsWithDocuments(ctx)
errCh <- err
}()
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("MinIO scan did not start")
}
cancel()
if err := <-errCh; err != nil {
t.Fatalf("expected shared scan to finish despite caller cancel: %v", err)
}
if len(ids) != 1 || ids[0] != "folder-1" {
t.Fatalf("unexpected folder ids: %#v", ids)
}
}
func TestContractFoldersWithDocsCacheExpires(t *testing.T) {
lister := &mockProcedureDocumentLister{
procedures: []ai_summarizer.ProcedureDocumentsSummary{
{ContractFolderID: "folder-1", DocumentCount: 1},
},
}
svc := &tenderService{aiSummarizerStorage: stubProcedureDocumentStorage{lister: lister}}
if _, err := svc.cachedContractFolderIDsWithDocuments(context.Background()); err != nil {
t.Fatalf("lookup failed: %v", err)
}
svc.contractFoldersCacheMu.Lock()
svc.contractFoldersCache.expiresAt = time.Now().Add(-time.Second)
svc.contractFoldersCacheMu.Unlock()
if _, err := svc.cachedContractFolderIDsWithDocuments(context.Background()); err != nil {
t.Fatalf("lookup after expiry failed: %v", err)
}
if lister.calls.Load() != 2 {
t.Fatalf("expected cache refresh after expiry, got %d calls", lister.calls.Load())
}
}
+89 -7
View File
@@ -5,7 +5,9 @@ import (
"strconv"
"strings"
"time"
"tm/pkg/mongo"
"tm/ted"
)
// DocumentSummary represents a summary of a tender document
@@ -86,7 +88,6 @@ type Tender struct {
Source TenderSource `bson:"source" json:"source"`
SourceFileURL string `bson:"source_file_url" json:"source_file_url"`
SourceFileName string `bson:"source_file_name" json:"source_file_name"`
ContentXML string `bson:"content_xml,omitempty" json:"content_xml,omitempty"`
ProcessingMetadata ProcessingMetadata `bson:"processing_metadata" json:"processing_metadata"`
// AIOverallSummary optional denormalized overall summary for search/list (e.g. synced from AI pipeline storage).
AIOverallSummary string `bson:"ai_overall_summary,omitempty" json:"ai_overall_summary,omitempty"`
@@ -262,11 +263,38 @@ func (t *Tender) IsExpired() bool {
return false
}
// Check if deadline (in milliseconds) is in the past
now := time.Now().Unix()
return t.TenderDeadline < now
}
// PublicationSubmissionDeadline returns the submission window end derived from publication_date.
// Stored submission_deadline is preferred; otherwise it is calculated as 48 working hours after publication.
func (t *Tender) PublicationSubmissionDeadline() int64 {
if t.SubmissionDeadline > 0 {
return t.SubmissionDeadline
}
if t.PublicationDate <= 0 {
return 0
}
return ted.CalculateSubmissionDeadline(time.Unix(t.PublicationDate, 0)).Unix()
}
// HasActivePublicationWindow reports whether the publication-based submission window is still open.
func (t *Tender) HasActivePublicationWindow(now int64) bool {
deadline := t.PublicationSubmissionDeadline()
return deadline > 0 && deadline > now
}
// IsRecommendable reports whether a tender should appear in customer AI recommendation results.
// Recommendations use the publication-based submission window, not tender_deadline.
func (t *Tender) IsRecommendable(now int64) bool {
switch t.Status {
case TenderStatusExpired, TenderStatusCancelled, TenderStatusAwarded, TenderStatusClosed:
return false
}
return t.HasActivePublicationWindow(now)
}
// GetTenderURL returns the TED tender URL
func (t *Tender) GetTenderURL() string {
if t.TenderURL != "" {
@@ -305,20 +333,74 @@ func (t *Tender) GetMainOrganization() *Organization {
// HasEstimatedValue returns true if the tender has an estimated value
func (t *Tender) HasEstimatedValue() bool {
return t.EstimatedValue > 0
value, _ := t.ResolvedEstimatedValueAndCurrency()
return value > 0
}
// AggregateProcurementLotEstimatedValue sums lot-level estimated values when all non-empty
// lot currencies match. Conflicting currencies are not aggregated.
func AggregateProcurementLotEstimatedValue(lots []ProcurementLot) (value float64, currency string) {
var refCurrency string
for _, lot := range lots {
if lot.EstimatedValue <= 0 {
continue
}
c := strings.TrimSpace(lot.Currency)
if c == "" {
continue
}
if refCurrency == "" {
refCurrency = c
continue
}
if c != refCurrency {
return 0, ""
}
}
for _, lot := range lots {
if lot.EstimatedValue <= 0 {
continue
}
value += lot.EstimatedValue
}
if value <= 0 {
return 0, ""
}
return value, refCurrency
}
// ResolvedEstimatedValueAndCurrency returns the tender-level estimated value when set,
// otherwise aggregates procurement lot values for API and display use.
func (t *Tender) ResolvedEstimatedValueAndCurrency() (float64, string) {
if t == nil {
return 0, ""
}
if t.EstimatedValue > 0 {
return t.EstimatedValue, strings.TrimSpace(t.Currency)
}
value, currency := AggregateProcurementLotEstimatedValue(t.ProcurementLots)
if value > 0 {
if currency == "" {
currency = strings.TrimSpace(t.Currency)
}
return value, currency
}
return 0, strings.TrimSpace(t.Currency)
}
// GetFormattedEstimatedValue returns formatted estimated value with currency
func (t *Tender) GetFormattedEstimatedValue() string {
if !t.HasEstimatedValue() {
value, currency := t.ResolvedEstimatedValueAndCurrency()
if value <= 0 {
return ""
}
if t.Currency != "" {
return fmt.Sprintf("%.2f %s", t.EstimatedValue, t.Currency)
if currency != "" {
return fmt.Sprintf("%.2f %s", value, currency)
}
return fmt.Sprintf("%.2f", t.EstimatedValue)
return fmt.Sprintf("%.2f", value)
}
// UpdateStatus updates the tender status and updated timestamp
+197
View File
@@ -0,0 +1,197 @@
package tender
import "testing"
func TestAggregateProcurementLotEstimatedValue(t *testing.T) {
value, currency := AggregateProcurementLotEstimatedValue([]ProcurementLot{
{EstimatedValue: 100000, Currency: "EUR"},
{EstimatedValue: 250000, Currency: "EUR"},
})
if value != 350000 {
t.Fatalf("expected total 350000, got %v", value)
}
if currency != "EUR" {
t.Fatalf("expected currency EUR, got %q", currency)
}
}
func TestAggregateProcurementLotEstimatedValueRejectsMixedCurrencies(t *testing.T) {
value, currency := AggregateProcurementLotEstimatedValue([]ProcurementLot{
{EstimatedValue: 100000, Currency: "EUR"},
{EstimatedValue: 250000, Currency: "USD"},
})
if value != 0 || currency != "" {
t.Fatalf("expected no aggregation for mixed currencies, got value=%v currency=%q", value, currency)
}
}
func TestAggregateProcurementLotEstimatedValueAllowsEmptyLotCurrency(t *testing.T) {
value, currency := AggregateProcurementLotEstimatedValue([]ProcurementLot{
{EstimatedValue: 100000, Currency: "EUR"},
{EstimatedValue: 250000},
})
if value != 350000 {
t.Fatalf("expected total 350000, got %v", value)
}
if currency != "EUR" {
t.Fatalf("expected currency EUR, got %q", currency)
}
}
func TestResolvedEstimatedValueAndCurrencyUsesLotsWhenTopLevelMissing(t *testing.T) {
tender := &Tender{
Currency: "EUR",
ProcurementLots: []ProcurementLot{
{LotID: "LOT-0001", EstimatedValue: 500000, Currency: "EUR"},
},
}
value, currency := tender.ResolvedEstimatedValueAndCurrency()
if value != 500000 {
t.Fatalf("expected 500000, got %v", value)
}
if currency != "EUR" {
t.Fatalf("expected EUR, got %q", currency)
}
}
func TestResolvedEstimatedValueAndCurrencyPrefersTopLevel(t *testing.T) {
tender := &Tender{
EstimatedValue: 900000,
Currency: "USD",
ProcurementLots: []ProcurementLot{
{EstimatedValue: 100000, Currency: "EUR"},
},
}
value, currency := tender.ResolvedEstimatedValueAndCurrency()
if value != 900000 {
t.Fatalf("expected 900000, got %v", value)
}
if currency != "USD" {
t.Fatalf("expected USD, got %q", currency)
}
}
func TestTenderPublicationSubmissionDeadline(t *testing.T) {
pubDate := int64(1_700_000_000)
tender := Tender{PublicationDate: pubDate}
got := tender.PublicationSubmissionDeadline()
if got <= pubDate {
t.Fatalf("expected submission deadline after publication_date, got %d (publication_date=%d)", got, pubDate)
}
withStored := Tender{PublicationDate: pubDate, SubmissionDeadline: pubDate + 999}
if deadline := withStored.PublicationSubmissionDeadline(); deadline != pubDate+999 {
t.Fatalf("expected stored submission_deadline, got %d", deadline)
}
}
func TestTenderHasActivePublicationWindow(t *testing.T) {
now := int64(1_700_000_000)
tests := []struct {
name string
tender Tender
want bool
}{
{
name: "future submission deadline",
tender: Tender{SubmissionDeadline: now + 3600},
want: true,
},
{
name: "past submission deadline",
tender: Tender{SubmissionDeadline: now - 3600},
want: false,
},
{
name: "future tender deadline only",
tender: Tender{TenderDeadline: now + 86400},
want: false,
},
{
name: "publication date only with open window",
tender: Tender{PublicationDate: now - 3600},
want: true,
},
{
name: "no publication or submission deadline",
tender: Tender{},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.tender.HasActivePublicationWindow(now); got != tt.want {
t.Fatalf("HasActivePublicationWindow() = %v, want %v", got, tt.want)
}
})
}
}
func TestTenderIsRecommendable(t *testing.T) {
now := int64(1_700_000_000)
tests := []struct {
name string
tender Tender
want bool
}{
{
name: "active with future submission deadline",
tender: Tender{Status: TenderStatusActive, SubmissionDeadline: now + 3600},
want: true,
},
{
name: "active with past submission deadline",
tender: Tender{Status: TenderStatusActive, SubmissionDeadline: now - 3600},
want: false,
},
{
name: "active with future tender deadline but past submission deadline",
tender: Tender{Status: TenderStatusActive, TenderDeadline: now + 86400, SubmissionDeadline: now - 3600},
want: false,
},
{
name: "active with future tender deadline only",
tender: Tender{Status: TenderStatusActive, TenderDeadline: now + 86400},
want: false,
},
{
name: "active with submission deadline at now",
tender: Tender{Status: TenderStatusActive, SubmissionDeadline: now},
want: false,
},
{
name: "active without publication or submission deadline",
tender: Tender{Status: TenderStatusActive},
want: false,
},
{
name: "active with publication date only and open submission window",
tender: Tender{Status: TenderStatusActive, PublicationDate: now - 3600},
want: true,
},
{
name: "expired status with future submission deadline",
tender: Tender{Status: TenderStatusExpired, SubmissionDeadline: now + 3600},
want: false,
},
{
name: "awarded with future submission deadline",
tender: Tender{Status: TenderStatusAwarded, SubmissionDeadline: now + 3600},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.tender.IsRecommendable(now); got != tt.want {
t.Fatalf("IsRecommendable() = %v, want %v", got, tt.want)
}
})
}
}
+1
View File
@@ -9,5 +9,6 @@ var (
ErrTenderNotFound = errors.New("tender: not found")
ErrTenderMissingNoticeID = errors.New("tender: missing notice publication id")
ErrTenderMissingContractFolderID = errors.New("tender: missing contract folder id")
ErrTenderDocumentsNotScraped = errors.New("tender: documents not scraped")
ErrAINotConfigured = errors.New("tender: ai service not configured")
)
+86 -4
View File
@@ -32,6 +32,7 @@ type SearchForm struct {
CountryCodes []string `query:"country_codes" valid:"optional"`
RegionCodes []string `query:"region_codes" valid:"optional"`
CompanyID *string `query:"company_id" valid:"optional"`
CompanyIDs []string `query:"-" valid:"optional"`
CustomerID *string `query:"customer_id" valid:"optional"`
Status []string `query:"status" valid:"optional"`
MainClassification *string `query:"main_classification" valid:"optional"`
@@ -60,11 +61,86 @@ type SearchForm struct {
BuyerOrganizationID *string `query:"buyer_organization_id" valid:"optional"`
NoticePublicationID *string `query:"notice_publication_id" valid:"optional"`
Languages []string `query:"languages" valid:"optional"`
OnlyActiveDeadlines bool `query:"-" valid:"optional"`
OnlyActiveDeadlines bool `query:"only_active_deadlines" valid:"optional"`
Language *string `query:"lang" valid:"optional"`
DocumentsScraped bool `query:"documents_scraped" valid:"optional"`
// ContractFolderIDsWithDocuments is populated by the service from MinIO when DocumentsScraped is true.
// ContractFolderIDsWithDocuments is populated by the service from a cached MinIO scan when DocumentsScraped is true.
ContractFolderIDsWithDocuments []string `query:"-" valid:"optional"`
// ExcludeRejectedTenders hides company-rejected tenders from recommendation results.
ExcludeRejectedTenders bool `query:"-" valid:"optional"`
}
// HasListFilters reports whether the form applies any MongoDB list filters.
func (f *SearchForm) HasListFilters() bool {
if f == nil {
return false
}
if f.DocumentsScraped || f.OnlyActiveDeadlines || f.ExcludeRejectedTenders {
return true
}
if f.Search != nil && strings.TrimSpace(*f.Search) != "" {
return true
}
if f.Title != nil && strings.TrimSpace(*f.Title) != "" {
return true
}
if f.Description != nil && strings.TrimSpace(*f.Description) != "" {
return true
}
if f.NoticeType != nil && strings.TrimSpace(*f.NoticeType) != "" {
return true
}
if len(f.NoticeTypes) > 0 || len(f.FormTypes) > 0 {
return true
}
if f.ProcurementType != nil && strings.TrimSpace(*f.ProcurementType) != "" {
return true
}
if f.Country != nil && strings.TrimSpace(*f.Country) != "" {
return true
}
if f.CountryCode != nil && strings.TrimSpace(*f.CountryCode) != "" {
return true
}
if len(f.CountryCodes) > 0 || len(f.RegionCodes) > 0 {
return true
}
if f.MainClassification != nil && strings.TrimSpace(*f.MainClassification) != "" {
return true
}
if len(f.Classifications) > 0 || len(f.CpvCodes) > 0 {
return true
}
if f.MinEstimatedValue != nil || f.MaxEstimatedValue != nil || f.Currency != "" {
return true
}
if f.CreatedAt != nil || f.CreatedAtFrom != nil || f.CreatedAtTo != nil {
return true
}
if f.TenderDeadline != nil || f.TenderDeadlineFrom != nil || f.TenderDeadlineTo != nil ||
f.DeadlineFrom != nil || f.DeadlineTo != nil {
return true
}
if f.PublicationDate != nil || f.PublicationDateFrom != nil || f.PublicationDateTo != nil {
return true
}
if f.SubmissionDeadline != nil || f.SubmissionDateAliasFrom != nil || f.SubmissionDateAliasTo != nil ||
f.SubmissionDateFrom != nil || f.SubmissionDateTo != nil {
return true
}
if len(f.Status) > 0 || len(f.Source) > 0 || len(f.Languages) > 0 {
return true
}
if f.BuyerOrganizationID != nil && strings.TrimSpace(*f.BuyerOrganizationID) != "" {
return true
}
if f.NoticePublicationID != nil && strings.TrimSpace(*f.NoticePublicationID) != "" {
return true
}
if len(f.ContractFolderIDsWithDocuments) > 0 {
return true
}
return false
}
// SearchResponse represents the response for listing tenders
@@ -233,6 +309,10 @@ type TenderResponse struct {
OverallSummary string `json:"overall_summary,omitempty"`
Language string `json:"language,omitempty"`
Rank int `json:"rank,omitempty"`
Analysis string `json:"analysis,omitempty"`
ProcedureRef string `json:"procedure_ref,omitempty"`
}
// TenderDocumentResponse represents a scraped tender document available for download.
@@ -277,6 +357,8 @@ func (t *Tender) ToResponseWithLanguage(language string) *TenderResponse {
documentURL = strings.TrimSpace(t.TenderURL)
}
estimatedValue, currency := t.ResolvedEstimatedValueAndCurrency()
response := &TenderResponse{
ID: t.ID.Hex(),
NoticePublicationID: t.NoticePublicationID,
@@ -307,8 +389,8 @@ func (t *Tender) ToResponseWithLanguage(language string) *TenderResponse {
MainClassificationDescription: mainDesc,
MainClassificationDisplay: classificationDisplay(mainCode, mainDesc),
AdditionalClassifications: append([]string(nil), t.AdditionalClassifications...),
EstimatedValue: t.EstimatedValue,
Currency: strings.TrimSpace(t.Currency),
EstimatedValue: estimatedValue,
Currency: currency,
EstimatedValueVATBasis: vatBasisUnspecified,
Duration: t.Duration,
+86 -49
View File
@@ -7,6 +7,7 @@ import (
"strconv"
"strings"
"time"
"tm/internal/company"
"tm/pkg/logger"
orm "tm/pkg/mongo"
"tm/pkg/response"
@@ -159,10 +160,11 @@ func (h *TenderHandler) Delete(c echo.Context) error {
// @Param languages query []string false "Filter by languages (comma-separated)"
// @Param buyer_organization_id query string false "Filter by buyer organization ID"
// @Param notice_publication_id query string false "Filter by TED notice publication ID"
// @Param documents_scraped query bool false "When true, only tenders whose contract_folder_id has documents in MinIO"
// @Param documents_scraped query bool false "When true, only tenders whose contract_folder_id has documents in MinIO (cached scan)"
// @Param limit query int false "Number of items per page (default: 20, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0). Prefer cursor for deep pages."
// @Param cursor query string false "Opaque cursor from previous page meta.next_cursor"
// @Param include_total query bool false "When true, include meta.total and meta.pages (slower on large collections)"
// @Param sort_by query string false "Sort by field"
// @Param sort_order query string false "Sort order (asc or desc)"
// @Success 200 {object} response.APIResponse{data=map[string]interface{}}
@@ -197,6 +199,44 @@ func (h *TenderHandler) Search(c echo.Context) error {
return response.SuccessWithMeta(c, tenders, tenders.Metadata, "Tenders retrieved successfully")
}
// AdminRecommendTenders returns AI-ranked tender recommendations for a company (Web Panel)
// @Summary Get recommended tenders for a company
// @Description Retrieve AI-ranked tenders with full tender details for the given company. Unlike the customer API, active-only filters are not applied unless requested via query parameters.
// @Tags Admin-Companies
// @Produce json
// @Param id path string true "Company ID"
// @Param limit query int false "Number of items per page (default: 10, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0)"
// @Param lang query string false "Response language code (defaults to en)"
// @Param status query []string false "Filter by status (comma-separated)"
// @Param only_active_deadlines query bool false "When true, exclude tenders past their deadline"
// @Success 200 {object} response.APIResponse{data=SearchResponse}
// @Failure 400 {object} response.APIResponse
// @Failure 404 {object} response.APIResponse
// @Failure 503 {object} response.APIResponse
// @Failure 500 {object} response.APIResponse
// @Security BearerAuth
// @Router /admin/v1/companies/{id}/recommended-tenders [get]
func (h *TenderHandler) AdminRecommendTenders(c echo.Context) error {
companyID := strings.TrimSpace(c.Param("id"))
if companyID == "" {
return response.BadRequest(c, "Company ID is required", "")
}
form, err := response.Parse[SearchForm](c)
if err != nil {
return response.BadRequest(c, "Invalid request format", "")
}
if _, err := govalidator.ValidateStruct(form); err != nil {
return response.ValidationError(c, err.Error(), "")
}
form.CompanyID = &companyID
return h.serveRecommendTenders(c, form)
}
// *****************************************************
// * PUBLIC HANDLERS *
// *****************************************************
@@ -231,10 +271,11 @@ func (h *TenderHandler) Search(c echo.Context) error {
// @Param languages query []string false "Filter by languages (comma-separated)"
// @Param buyer_organization_id query string false "Filter by buyer organization ID"
// @Param notice_publication_id query string false "Filter by TED notice publication ID"
// @Param documents_scraped query bool false "When true, only tenders whose contract_folder_id has documents in MinIO"
// @Param documents_scraped query bool false "When true, only tenders whose contract_folder_id has documents in MinIO (cached scan)"
// @Param limit query int false "Number of items per page (default: 20, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0). Prefer cursor for deep pages."
// @Param cursor query string false "Opaque cursor from previous page meta.next_cursor"
// @Param include_total query bool false "When true, include meta.total and meta.pages (slower on large collections)"
// @Param sort_by query string false "Sort by field"
// @Param sort_order query string false "Sort order (asc or desc)"
// @Success 200 {object} response.APIResponse{data=SearchResponse}
@@ -285,44 +326,18 @@ func (h *TenderHandler) GetPublicTenders(c echo.Context) error {
return response.SuccessWithMeta(c, result, result.Metadata, "Tenders retrieved successfully")
}
// GetPublicTenders retrieves public tenders for mobile app
// @Summary Get public tenders
// @Description Retrieve active tenders for mobile application users with automatic company-based matching from customer profile
// RecommendTenders returns AI-ranked tender recommendations for the authenticated customer's company
// @Summary Get recommended tenders
// @Description Retrieve AI-ranked tenders with full tender details. Only active tenders whose publication-based submission window has not passed are returned (submission_deadline, or 48 working hours after publication_date). Company-rejected tenders are excluded. AI tender_id values use PROC_{contract_folder_id}/{notice_publication_id}; each item includes rank, analysis, procedure_ref, and the standard tender fields.
// @Tags Tenders
// @Produce json
// @Param q query string false "Keyword search across title, description, AI document summaries, optional stored overall summary"
// @Param notice_type query string false "Filter by TED notice type code (notice_type_code)"
// @Param notice_types query []string false "Multiple notice type codes"
// @Param form_types query []string false "TED form types: competition, result, planning, contract-award, prior-information, etc."
// @Param procurement_type query string false "Filter by procurement type code"
// @Param country query string false "Single ISO country code"
// @Param country_codes query []string false "Filter by country codes"
// @Param status query []string false "Filter by status (comma-separated)"
// @Param source query []string false "Filter by source (comma-separated)"
// @Param classifications query []string false "CPV / classification codes"
// @Param cpv_codes query []string false "One or more CPV codes"
// @Param min_estimated_value query number false "Minimum estimated value"
// @Param max_estimated_value query number false "Maximum estimated value"
// @Param currency query string false "Filter by currency"
// @Param created_at_from query number false "Created at from (Unix seconds)"
// @Param created_at_to query number false "Created at to (Unix seconds)"
// @Param deadline_from query number false "Tender deadline from (Unix seconds)"
// @Param deadline_to query number false "Tender deadline to (Unix seconds)"
// @Param publication_date_from query number false "Publication date from (Unix seconds)"
// @Param publication_date_to query number false "Publication date to (Unix seconds)"
// @Param submission_date_from query number false "Submission deadline from (Unix seconds)"
// @Param submission_date_to query number false "Submission deadline to (Unix seconds)"
// @Param languages query []string false "Filter by languages (comma-separated)"
// @Param buyer_organization_id query string false "Filter by buyer organization ID"
// @Param notice_publication_id query string false "Filter by TED notice publication ID"
// @Param documents_scraped query bool false "When true, only tenders whose contract_folder_id has documents in MinIO"
// @Param limit query int false "Number of items per page (default: 20, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0). Prefer cursor for deep pages."
// @Param cursor query string false "Opaque cursor from previous page meta.next_cursor"
// @Param sort_by query string false "Sort by field"
// @Param sort_order query string false "Sort order (asc or desc)"
// @Param limit query int false "Number of items per page (default: 10, max: 100)"
// @Param offset query int false "Number of items to skip (default: 0)"
// @Param lang query string false "Response language code (defaults to en)"
// @Success 200 {object} response.APIResponse{data=SearchResponse}
// @Failure 400 {object} response.APIResponse
// @Failure 404 {object} response.APIResponse
// @Failure 503 {object} response.APIResponse
// @Failure 500 {object} response.APIResponse
// @Security BearerAuth
// @Router /api/v1/tenders/recommend [get]
@@ -339,19 +354,31 @@ func (h *TenderHandler) RecommendTenders(c echo.Context) error {
form.Status = []string{string(TenderStatusActive)}
form.OnlyActiveDeadlines = true
form.ExcludeRejectedTenders = true
var companyID *string
if customerCompanyID, ok := c.Get("company_id").(string); ok {
companyID = &customerCompanyID
if customerIDStr, ok := c.Get("customer_id").(string); ok && customerIDStr != "" {
form.CustomerID = &customerIDStr
}
form.CompanyID = companyID
var customerID *string
if customerIDStr, ok := c.Get("customer_id").(string); ok {
customerID = &customerIDStr
requestedCompanyID := ""
if form.CompanyID != nil {
requestedCompanyID = strings.TrimSpace(*form.CompanyID)
}
form.CustomerID = customerID
if requestedCompanyID != "" {
if customerCompanyID, ok := c.Get("company_id").(string); ok && customerCompanyID != "" {
form.CompanyID = &customerCompanyID
}
} else if customerCompanyIDs, ok := c.Get("company_ids").([]string); ok && len(customerCompanyIDs) > 0 {
form.CompanyIDs = append([]string(nil), customerCompanyIDs...)
} else if customerCompanyID, ok := c.Get("company_id").(string); ok && customerCompanyID != "" {
form.CompanyID = &customerCompanyID
}
return h.serveRecommendTenders(c, form)
}
func (h *TenderHandler) serveRecommendTenders(c echo.Context, form *SearchForm) error {
pagination, err := response.NewPagination(c)
if err != nil {
return response.PaginationBadRequest(c, err)
@@ -362,10 +389,19 @@ func (h *TenderHandler) RecommendTenders(c echo.Context) error {
if response.IsListPaginationError(err) {
return response.PaginationBadRequest(c, err)
}
return response.InternalServerError(c, "Failed to retrieve tenders")
if errors.Is(err, company.ErrAIRecommendationNotConfigured) {
return response.ServiceUnavailable(c, "AI recommendation service is not configured")
}
if err.Error() == "company ID is required" {
return response.BadRequest(c, "Company ID is required", "")
}
if err.Error() == "company not found" {
return response.NotFound(c, "Company not found")
}
return response.InternalServerError(c, "Failed to retrieve recommended tenders")
}
return response.SuccessWithMeta(c, result, result.Metadata, "Tenders retrieved successfully")
return response.SuccessWithMeta(c, result, result.Metadata, "Recommended tenders retrieved successfully")
}
// GetPublicTenderDetails retrieves public tender details for mobile app
@@ -537,7 +573,7 @@ func (h *TenderHandler) streamDocumentDownload(c echo.Context, tenderID, filenam
// GetAISummary retrieves the AI-generated overall summary for a tender
// @Summary Get AI-generated tender summary
// @Description Retrieve the AI-generated overall summary for a tender from the AI pipeline storage
// @Description Retrieve the AI-generated overall summary for a tender. When documents are scraped but no summary exists yet, on-demand summarization is triggered automatically.
// @Tags Admin-Tenders
// @Produce json
// @Param id path string true "Tender ID"
@@ -637,6 +673,8 @@ func mapAITriggerHTTPError(c echo.Context, err error, action string) error {
return response.NotFound(c, "Tender not found")
case errors.Is(err, ErrTenderMissingNoticeID), errors.Is(err, ErrTenderMissingContractFolderID):
return response.BadRequest(c, "Tender not eligible", "Tender is missing required identifiers for AI processing")
case errors.Is(err, ErrTenderDocumentsNotScraped):
return response.BadRequest(c, "Tender not eligible", "Tender documents have not been scraped yet")
case errors.Is(err, ErrAINotConfigured):
return response.InternalServerError(c, "AI service is not available")
default:
@@ -683,7 +721,7 @@ func (h *TenderHandler) TriggerAITranslate(c echo.Context) error {
// GetPublicAISummary retrieves the AI-generated overall summary for a tender (public endpoint)
// @Summary Get AI-generated tender summary (public)
// @Description Retrieve the AI-generated overall summary for a tender from the AI pipeline storage
// @Description Retrieve the AI-generated overall summary for a tender. When documents are scraped but no summary exists yet, on-demand summarization is triggered automatically.
// @Tags Tenders
// @Produce json
// @Param id path string true "Tender ID"
@@ -799,4 +837,3 @@ func (h *TenderHandler) GetAllAISummaries(c echo.Context) error {
return response.Success(c, summaries, "AI summaries retrieved successfully")
}
+158
View File
@@ -0,0 +1,158 @@
package tender
import (
"context"
"fmt"
"strings"
"golang.org/x/sync/errgroup"
)
// CompanyRejectedTenderLister returns tender IDs the company has rejected.
type CompanyRejectedTenderLister interface {
ListRejectedTenderIDsByCompanyID(ctx context.Context, companyID string) ([]string, error)
}
func normalizeRecommendationCompanyScope(companyIDs []string) []string {
normalized := make([]string, 0, len(companyIDs))
seen := make(map[string]struct{}, len(companyIDs))
for _, companyID := range companyIDs {
clean := strings.TrimSpace(companyID)
if clean == "" {
continue
}
if _, ok := seen[clean]; ok {
continue
}
seen[clean] = struct{}{}
normalized = append(normalized, clean)
}
return normalized
}
func (s *tenderService) buildRejectedTenderIDSet(ctx context.Context, companyID string) (map[string]struct{}, error) {
excluded := make(map[string]struct{})
if s.rejectedTenderLister == nil {
return excluded, nil
}
ids, err := s.rejectedTenderLister.ListRejectedTenderIDsByCompanyID(ctx, companyID)
if err != nil {
return nil, fmt.Errorf("failed to list rejected tender exclusions: %w", err)
}
addTenderIDsToSet(excluded, ids)
return excluded, nil
}
func (s *tenderService) buildRejectedTenderIDSetsForCompanies(ctx context.Context, companyIDs []string) (map[string]map[string]struct{}, error) {
excludedByCompany := make(map[string]map[string]struct{})
if s.rejectedTenderLister == nil {
return excludedByCompany, nil
}
normalized := normalizeRecommendationCompanyScope(companyIDs)
if len(normalized) == 0 {
return excludedByCompany, nil
}
results := make([][]string, len(normalized))
group, groupCtx := errgroup.WithContext(ctx)
for i, companyID := range normalized {
i := i
companyID := companyID
group.Go(func() error {
ids, err := s.rejectedTenderLister.ListRejectedTenderIDsByCompanyID(groupCtx, companyID)
if err != nil {
return fmt.Errorf("failed to list rejected tender exclusions: %w", err)
}
results[i] = ids
return nil
})
}
if err := group.Wait(); err != nil {
return nil, err
}
for i, companyID := range normalized {
excluded := make(map[string]struct{})
addTenderIDsToSet(excluded, results[i])
excludedByCompany[companyID] = excluded
}
return excludedByCompany, nil
}
func addTenderIDsToSet(set map[string]struct{}, ids []string) {
for _, id := range ids {
id = strings.TrimSpace(id)
if id == "" {
continue
}
set[id] = struct{}{}
}
}
// markResolvedRecommendedTenderSeen records a resolved tender and reports whether it was already seen.
// Recommendations may reference the same tender via different AI refs (procedure ref vs Mongo ID,
// or multiple notice publication IDs for one merged tender).
func markResolvedRecommendedTenderSeen(seen map[string]struct{}, tender Tender, tenderRef string) bool {
if seen == nil {
return false
}
key := strings.TrimSpace(tender.GetID())
if key == "" {
key = strings.TrimSpace(tenderRef)
}
if key == "" {
return false
}
if _, exists := seen[key]; exists {
return true
}
seen[key] = struct{}{}
return false
}
func isRecommendedTenderExcluded(tender Tender, tenderRef string, excluded map[string]struct{}) bool {
if len(excluded) == 0 {
return false
}
if _, ok := excluded[tender.GetID()]; ok {
return true
}
tenderRef = strings.TrimSpace(tenderRef)
if tenderRef != "" {
if _, ok := excluded[tenderRef]; ok {
return true
}
}
procRef := FormatAIProcedureRef(tender.ContractFolderID, tender.NoticePublicationID)
if procRef != "" {
if _, ok := excluded[procRef]; ok {
return true
}
}
return false
}
func isRecommendedTenderExcludedForAllCompanies(tender Tender, tenderRef string, excludedByCompany map[string]map[string]struct{}, companyIDs []string) bool {
normalized := normalizeRecommendationCompanyScope(companyIDs)
if len(normalized) == 0 {
return false
}
for _, companyID := range normalized {
if !isRecommendedTenderExcluded(tender, tenderRef, excludedByCompany[companyID]) {
return false
}
}
return true
}
@@ -0,0 +1,149 @@
package tender
import (
"context"
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
"tm/pkg/mongo"
)
type stubRejectedTenderLister struct {
byCompany map[string][]string
}
func (s stubRejectedTenderLister) ListRejectedTenderIDsByCompanyID(_ context.Context, companyID string) ([]string, error) {
return s.byCompany[companyID], nil
}
func TestIsRecommendedTenderExcluded(t *testing.T) {
tender := Tender{
Model: mongo.Model{},
ContractFolderID: "055329e0-3d8f-4eeb-946a-85a451f2e36b",
NoticePublicationID: "00423458-2026",
}
tender.ID, _ = bson.ObjectIDFromHex("507f1f77bcf86cd799439011")
procRef := FormatAIProcedureRef(tender.ContractFolderID, tender.NoticePublicationID)
tests := []struct {
name string
excluded map[string]struct{}
tenderRef string
want bool
}{
{
name: "empty exclusions",
excluded: map[string]struct{}{},
want: false,
},
{
name: "excluded by mongo id",
excluded: map[string]struct{}{tender.GetID(): {}},
want: true,
},
{
name: "excluded by procedure ref",
excluded: map[string]struct{}{procRef: {}},
want: true,
},
{
name: "excluded by recommendation ref",
excluded: map[string]struct{}{procRef: {}},
tenderRef: procRef,
want: true,
},
{
name: "not excluded",
excluded: map[string]struct{}{"other-tender-id": {}},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := isRecommendedTenderExcluded(tender, tt.tenderRef, tt.excluded)
if got != tt.want {
t.Fatalf("isRecommendedTenderExcluded() = %v want %v", got, tt.want)
}
})
}
}
func TestBuildRejectedTenderIDSetsForCompanies(t *testing.T) {
service := &tenderService{
rejectedTenderLister: stubRejectedTenderLister{
byCompany: map[string][]string{
"company-a": {" tender-1 ", "tender-2"},
"company-b": {"tender-2", "tender-3"},
},
},
}
got, err := service.buildRejectedTenderIDSetsForCompanies(context.Background(), []string{"company-a", "company-b", "company-a"})
if err != nil {
t.Fatalf("buildRejectedTenderIDSetsForCompanies() error = %v", err)
}
if _, ok := got["company-a"]["tender-1"]; !ok {
t.Fatalf("expected company-a exclusions to contain tender-1")
}
if _, ok := got["company-a"]["tender-2"]; !ok {
t.Fatalf("expected company-a exclusions to contain tender-2")
}
if _, ok := got["company-b"]["tender-2"]; !ok {
t.Fatalf("expected company-b exclusions to contain tender-2")
}
if _, ok := got["company-b"]["tender-3"]; !ok {
t.Fatalf("expected company-b exclusions to contain tender-3")
}
if len(got) != 2 {
t.Fatalf("buildRejectedTenderIDSetsForCompanies() len = %d, want 2", len(got))
}
}
func TestIsRecommendedTenderExcludedForAllCompanies(t *testing.T) {
tender := Tender{
Model: mongo.Model{},
ContractFolderID: "055329e0-3d8f-4eeb-946a-85a451f2e36b",
NoticePublicationID: "00423458-2026",
}
tender.ID, _ = bson.ObjectIDFromHex("507f1f77bcf86cd799439011")
procRef := FormatAIProcedureRef(tender.ContractFolderID, tender.NoticePublicationID)
excludedByCompany := map[string]map[string]struct{}{
"company-a": {procRef: {}},
"company-b": {},
}
if isRecommendedTenderExcludedForAllCompanies(tender, procRef, excludedByCompany, []string{"company-a", "company-b"}) {
t.Fatalf("expected tender to remain visible when not all companies rejected it")
}
excludedByCompany["company-b"][procRef] = struct{}{}
if !isRecommendedTenderExcludedForAllCompanies(tender, procRef, excludedByCompany, []string{"company-a", "company-b"}) {
t.Fatalf("expected tender to be hidden when all companies rejected it")
}
}
func TestMarkResolvedRecommendedTenderSeen(t *testing.T) {
sample := Tender{}
sample.ID, _ = bson.ObjectIDFromHex("507f1f77bcf86cd799439011")
seen := make(map[string]struct{})
first := markResolvedRecommendedTenderSeen(seen, sample, "PROC_folder/notice")
second := markResolvedRecommendedTenderSeen(seen, sample, "PROC_folder/notice")
if first {
t.Fatalf("first sighting should not be skipped, id=%q seen=%v", sample.GetID(), seen)
}
if !second {
t.Fatalf("duplicate sighting should be skipped, id=%q seen=%v", sample.GetID(), seen)
}
otherRefSameTender := Tender{}
otherRefSameTender.ID = sample.ID
if !markResolvedRecommendedTenderSeen(seen, otherRefSameTender, "PROC_folder/other-notice") {
t.Fatalf("duplicate tender via different ref should be skipped")
}
}
+94
View File
@@ -0,0 +1,94 @@
package tender
import (
"context"
"strings"
"tm/internal/company"
)
const recommendationResolveBatchSize = 100
type recommendationPageResult struct {
items []rankedRecommendedTender
total int64
}
func (s *tenderService) buildRecommendationPage(
ctx context.Context,
recommendations []company.RecommendedTenderResponse,
form *SearchForm,
companyIDs []string,
excluded map[string]struct{},
excludedByCompany map[string]map[string]struct{},
now int64,
offset int,
limit int,
) (recommendationPageResult, error) {
seenTenderIDs := make(map[string]struct{}, len(recommendations))
pageStart := offset
pageEnd := offset + limit
if pageStart < 0 {
pageStart = 0
}
if pageEnd < pageStart {
pageEnd = pageStart
}
result := recommendationPageResult{
items: make([]rankedRecommendedTender, 0, limit),
}
for start := 0; start < len(recommendations); start += recommendationResolveBatchSize {
end := start + recommendationResolveBatchSize
if end > len(recommendations) {
end = len(recommendations)
}
tenderByRef, err := s.resolveRecommendedTenders(ctx, recommendations[start:end])
if err != nil {
return recommendationPageResult{}, err
}
for _, rec := range recommendations[start:end] {
tenderRef := strings.TrimSpace(rec.TenderID)
tender, ok := tenderByRef[tenderRef]
if !ok {
continue
}
if len(companyIDs) > 0 {
if isRecommendedTenderExcludedForAllCompanies(tender, tenderRef, excludedByCompany, companyIDs) {
continue
}
} else if isRecommendedTenderExcluded(tender, tenderRef, excluded) {
continue
}
if form.OnlyActiveDeadlines && !tender.IsRecommendable(now) {
continue
}
if len(form.Status) > 0 && !statusAllowed(tender.Status, form.Status) {
continue
}
if markResolvedRecommendedTenderSeen(seenTenderIDs, tender, tenderRef) {
continue
}
result.total++
if int(result.total) <= pageStart || len(result.items) >= limit {
continue
}
if int(result.total) > pageEnd {
continue
}
result.items = append(result.items, rankedRecommendedTender{
tender: tender,
rank: rec.Rank,
analysis: rec.Analysis,
tenderRef: tenderRef,
})
}
}
return result, nil
}
@@ -0,0 +1,355 @@
package tender
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/redis/go-redis/v9"
"tm/internal/company"
"tm/pkg/response"
)
const (
recommendedTendersPageCacheKeyPrefix = "ai:recommendations:page:"
recommendationPageCacheReadTimeout = 3 * time.Second
recommendationPageCacheBuildTimeout = 5 * time.Minute
recommendationPageCacheConcurrency = 8
)
type cachedRecommendedTendersPage struct {
BuiltAt int64 `json:"built_at"`
CompanyID string `json:"company_id"`
Language string `json:"language"`
Items []TenderResponse `json:"items"`
}
func recommendedTendersPageCacheKey(companyID, language string) string {
return recommendedTendersPageCacheKeyPrefix +
strings.TrimSpace(companyID) + ":" +
strings.ToLower(strings.TrimSpace(language))
}
// ScheduleRefreshRecommendedTendersPageCache rebuilds the page cache for the service default language.
func (s *tenderService) ScheduleRefreshRecommendedTendersPageCache(companyID string) {
s.schedulePageCacheRefreshForLanguage(companyID, s.defaultLanguage)
}
// InvalidateRecommendedTendersPageCache drops the cached page for the service default language.
func (s *tenderService) InvalidateRecommendedTendersPageCache(companyID string) {
companyID = strings.TrimSpace(companyID)
if companyID == "" || s.redisClient == nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), recommendationPageCacheReadTimeout)
defer cancel()
s.deleteRecommendedTendersPageCache(ctx, companyID, s.defaultLanguage)
}
func (s *tenderService) schedulePageCacheRefreshForLanguage(companyID, language string) {
companyID = strings.TrimSpace(companyID)
language = strings.ToLower(strings.TrimSpace(language))
if companyID == "" || language == "" || s.redisClient == nil {
return
}
go func() {
ctx, cancel := context.WithTimeout(context.Background(), recommendationPageCacheBuildTimeout)
defer cancel()
if err := s.buildRecommendedTendersPageCache(ctx, companyID, language); err != nil {
s.logger.Warn("Failed to refresh recommended tenders page cache", map[string]interface{}{
"company_id": companyID,
"language": language,
"error": err.Error(),
})
}
}()
}
func (s *tenderService) buildRecommendedTendersPageCache(ctx context.Context, companyID, language string) error {
recommendations, err := s.companyService.GetAIRecommendations(ctx, companyID)
if err != nil {
return err
}
if len(recommendations) == 0 {
s.deleteRecommendedTendersPageCache(ctx, companyID, language)
return nil
}
items, err := s.buildAllRankedRecommendations(ctx, recommendations)
if err != nil {
return err
}
if len(items) == 0 {
s.deleteRecommendedTendersPageCache(ctx, companyID, language)
return nil
}
responses := s.buildRecommendedTenderListResponses(items, language)
page := cachedRecommendedTendersPage{
BuiltAt: time.Now().Unix(),
CompanyID: companyID,
Language: language,
Items: responses,
}
encoded, err := json.Marshal(page)
if err != nil {
return fmt.Errorf("encode recommended tenders page cache: %w", err)
}
writeCtx, cancel := context.WithTimeout(ctx, recommendationPageCacheReadTimeout)
defer cancel()
if err := s.redisClient.Set(writeCtx, recommendedTendersPageCacheKey(companyID, language), string(encoded), s.pageCacheTTL); err != nil {
return fmt.Errorf("store recommended tenders page cache: %w", err)
}
s.logger.Info("Recommended tenders page cache built", map[string]interface{}{
"company_id": companyID,
"language": language,
"count": len(responses),
})
return nil
}
func (s *tenderService) getRecommendedTendersPageCache(ctx context.Context, companyID, language string) ([]TenderResponse, bool) {
if s.redisClient == nil {
return nil, false
}
readCtx, cancel := context.WithTimeout(ctx, recommendationPageCacheReadTimeout)
defer cancel()
raw, err := s.redisClient.Get(readCtx, recommendedTendersPageCacheKey(companyID, language))
if err != nil {
if err != redis.Nil {
s.logger.Warn("Failed to read recommended tenders page cache", map[string]interface{}{
"company_id": companyID,
"language": language,
"error": err.Error(),
})
}
return nil, false
}
var page cachedRecommendedTendersPage
if err := json.Unmarshal([]byte(raw), &page); err != nil {
s.logger.Warn("Failed to decode recommended tenders page cache", map[string]interface{}{
"company_id": companyID,
"language": language,
"error": err.Error(),
})
s.deleteRecommendedTendersPageCache(ctx, companyID, language)
return nil, false
}
if len(page.Items) == 0 {
return nil, false
}
return page.Items, true
}
func (s *tenderService) deleteRecommendedTendersPageCache(ctx context.Context, companyID, language string) {
if s.redisClient == nil {
return
}
_ = s.redisClient.Del(ctx, recommendedTendersPageCacheKey(companyID, language))
}
func (s *tenderService) buildAllRankedRecommendations(
ctx context.Context,
recommendations []company.RecommendedTenderResponse,
) ([]rankedRecommendedTender, error) {
seenTenderIDs := make(map[string]struct{}, len(recommendations))
out := make([]rankedRecommendedTender, 0, len(recommendations))
for start := 0; start < len(recommendations); start += recommendationResolveBatchSize {
end := start + recommendationResolveBatchSize
if end > len(recommendations) {
end = len(recommendations)
}
tenderByRef, err := s.resolveRecommendedTenders(ctx, recommendations[start:end])
if err != nil {
return nil, err
}
for _, rec := range recommendations[start:end] {
tenderRef := strings.TrimSpace(rec.TenderID)
tender, ok := tenderByRef[tenderRef]
if !ok {
continue
}
if markResolvedRecommendedTenderSeen(seenTenderIDs, tender, tenderRef) {
continue
}
out = append(out, rankedRecommendedTender{
tender: tender,
rank: rec.Rank,
analysis: rec.Analysis,
tenderRef: tenderRef,
})
}
}
return out, nil
}
func (s *tenderService) isRecommendedPageCacheStale(ctx context.Context, companyID string, cachedCount int) bool {
companyID = strings.TrimSpace(companyID)
if companyID == "" || cachedCount == 0 {
return false
}
recommendations, err := s.companyService.GetAIRecommendations(ctx, companyID)
if err != nil || len(recommendations) == 0 {
return false
}
return cachedCount < len(recommendations)
}
func (s *tenderService) recommendFromPageCache(
ctx context.Context,
companyID string,
companyIDs []string,
form *SearchForm,
pagination *response.Pagination,
lang string,
) (*SearchResponse, bool) {
var cached []TenderResponse
if len(companyIDs) > 0 {
cached = s.mergeCachedRecommendationPages(ctx, companyIDs, lang)
} else {
items, ok := s.getRecommendedTendersPageCache(ctx, companyID, lang)
if !ok {
return nil, false
}
cached = items
}
if len(cached) == 0 {
return nil, false
}
if len(companyIDs) == 0 && s.isRecommendedPageCacheStale(ctx, companyID, len(cached)) {
return nil, false
}
excluded, excludedByCompany, err := s.loadRecommendationExclusions(ctx, form, companyID, companyIDs)
if err != nil {
return nil, false
}
now := time.Now().Unix()
filtered := filterCachedRecommendedTenders(cached, form, companyIDs, excluded, excludedByCompany, now)
total := int64(len(filtered))
start := pagination.Offset
if start > len(filtered) {
start = len(filtered)
}
end := start + pagination.Limit
if end > len(filtered) {
end = len(filtered)
}
paged := append([]TenderResponse(nil), filtered[start:end]...)
paged = s.enrichRecommendedTenderResponsesParallel(ctx, paged, lang)
return &SearchResponse{
Tenders: paged,
Metadata: pagination.ListMeta(total, "", end < len(filtered), start),
}, true
}
func (s *tenderService) mergeCachedRecommendationPages(ctx context.Context, companyIDs []string, lang string) []TenderResponse {
normalized := normalizeRecommendationCompanyScope(companyIDs)
if len(normalized) == 0 {
return nil
}
lists := make([][]TenderResponse, 0, len(normalized))
for _, companyID := range normalized {
items, ok := s.getRecommendedTendersPageCache(ctx, companyID, lang)
if !ok || len(items) == 0 {
return nil
}
lists = append(lists, items)
}
merged := make([]TenderResponse, 0)
for _, list := range lists {
merged = append(merged, list...)
}
return dedupeCachedRecommendedTendersByID(merged)
}
func dedupeCachedRecommendedTendersByID(items []TenderResponse) []TenderResponse {
seen := make(map[string]struct{}, len(items))
out := make([]TenderResponse, 0, len(items))
for _, item := range items {
key := strings.TrimSpace(item.ID)
if key == "" {
key = strings.TrimSpace(item.ProcedureRef)
}
if key == "" {
continue
}
if _, exists := seen[key]; exists {
continue
}
seen[key] = struct{}{}
out = append(out, item)
}
return out
}
func filterCachedRecommendedTenders(
items []TenderResponse,
form *SearchForm,
companyIDs []string,
excluded map[string]struct{},
excludedByCompany map[string]map[string]struct{},
now int64,
) []TenderResponse {
if len(items) == 0 {
return nil
}
out := make([]TenderResponse, 0, len(items))
for _, item := range items {
tender := tenderFromRecommendationResponse(item)
tenderRef := strings.TrimSpace(item.ProcedureRef)
if len(companyIDs) > 0 {
if isRecommendedTenderExcludedForAllCompanies(tender, tenderRef, excludedByCompany, companyIDs) {
continue
}
} else if isRecommendedTenderExcluded(tender, tenderRef, excluded) {
continue
}
if form.OnlyActiveDeadlines && !tender.IsRecommendable(now) {
continue
}
if len(form.Status) > 0 && !statusAllowed(tender.Status, form.Status) {
continue
}
out = append(out, item)
}
return out
}
// RecommendedTendersPageCacheKeyPrefix is shared with company invalidation logic.
const RecommendedTendersPageCacheKeyPrefix = recommendedTendersPageCacheKeyPrefix
// RecommendedTendersPageCacheKey builds the Redis key for a cached recommendation page.
func RecommendedTendersPageCacheKey(companyID, language string) string {
return recommendedTendersPageCacheKey(companyID, language)
}
@@ -0,0 +1,44 @@
package tender
import (
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
)
func TestFilterCachedRecommendedTendersAppliesActiveAndRejectedFilters(t *testing.T) {
activeID, _ := bson.ObjectIDFromHex("507f1f77bcf86cd799439011")
expiredID, _ := bson.ObjectIDFromHex("507f1f77bcf86cd799439012")
items := []TenderResponse{
{
ID: activeID.Hex(),
Status: TenderStatusActive,
PublicationDate: 1,
SubmissionDeadline: 9999999999,
ProcedureRef: "PROC_folder-a/notice-a",
Rank: 1,
},
{
ID: expiredID.Hex(),
Status: TenderStatusExpired,
ProcedureRef: "PROC_folder-b/notice-b",
Rank: 2,
},
}
form := &SearchForm{
OnlyActiveDeadlines: true,
ExcludeRejectedTenders: true,
Status: []string{string(TenderStatusActive)},
}
excluded := map[string]struct{}{expiredID.Hex(): {}}
got := filterCachedRecommendedTenders(items, form, nil, excluded, nil, 100)
if len(got) != 1 {
t.Fatalf("len = %d, want 1", len(got))
}
if got[0].ID != activeID.Hex() {
t.Fatalf("got ID %q, want active tender", got[0].ID)
}
}
@@ -0,0 +1,86 @@
package tender
import (
"context"
"fmt"
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
"tm/internal/company"
)
type stubRecommendTenderRepository struct {
TenderRepository
batchCalls int
}
func (s *stubRecommendTenderRepository) GetByProcedureReferences(_ context.Context, refs []ProcedureReference) (map[string]Tender, error) {
s.batchCalls++
out := make(map[string]Tender, len(refs))
for i, ref := range refs {
tender := Tender{
ContractFolderID: ref.ContractFolderID,
NoticePublicationID: ref.NoticePublicationID,
Title: ref.Ref,
Status: TenderStatusActive,
}
tender.ID = bsonObjectIDFromCounter(i + s.batchCalls*1000)
out[ref.Ref] = tender
}
return out, nil
}
func bsonObjectIDFromCounter(n int) bson.ObjectID {
var id [12]byte
id[11] = byte(n)
id[10] = byte(n >> 8)
return bson.ObjectID(id)
}
func TestBuildRecommendationPageResolvesInBatchesAndPaginates(t *testing.T) {
repo := &stubRecommendTenderRepository{}
service := &tenderService{
repository: repo,
logger: noopTenderTestLogger{},
}
recommendations := make([]company.RecommendedTenderResponse, 0, 120)
for i := 0; i < 120; i++ {
recommendations = append(recommendations, company.RecommendedTenderResponse{
Rank: i + 1,
TenderID: fmt.Sprintf("PROC_folder/notice-%d", i),
Analysis: "fit",
})
}
form := &SearchForm{
OnlyActiveDeadlines: false,
ExcludeRejectedTenders: false,
Status: []string{string(TenderStatusActive)},
}
got, err := service.buildRecommendationPage(
context.Background(),
recommendations,
form,
nil,
nil,
nil,
0,
10,
10,
)
if err != nil {
t.Fatalf("buildRecommendationPage() error = %v", err)
}
if repo.batchCalls != 2 {
t.Fatalf("batchCalls = %d, want 2", repo.batchCalls)
}
if got.total != 120 {
t.Fatalf("total = %d, want 120", got.total)
}
if len(got.items) != 10 {
t.Fatalf("len(items) = %d, want 10", len(got.items))
}
}
+163
View File
@@ -0,0 +1,163 @@
package tender
import (
"context"
"fmt"
"strings"
"sync"
"go.mongodb.org/mongo-driver/v2/bson"
"golang.org/x/sync/errgroup"
"tm/pkg/ai_summarizer"
)
type rankedRecommendedTender struct {
tender Tender
rank int
analysis string
tenderRef string
}
type recommendationTranslationCache struct {
mu sync.Mutex
items map[string]ai_summarizer.StoredTranslation
}
func newRecommendationTranslationCache() *recommendationTranslationCache {
return &recommendationTranslationCache{
items: make(map[string]ai_summarizer.StoredTranslation),
}
}
func recommendationTranslationCacheKey(tenderID, language string) string {
return strings.TrimSpace(tenderID) + "|" + strings.TrimSpace(language)
}
func (c *recommendationTranslationCache) get(tenderID, language string) (ai_summarizer.StoredTranslation, bool) {
if c == nil {
return ai_summarizer.StoredTranslation{}, false
}
c.mu.Lock()
defer c.mu.Unlock()
stored, ok := c.items[recommendationTranslationCacheKey(tenderID, language)]
return stored, ok
}
func (c *recommendationTranslationCache) set(tenderID, language string, stored ai_summarizer.StoredTranslation) {
if c == nil {
return
}
c.mu.Lock()
defer c.mu.Unlock()
c.items[recommendationTranslationCacheKey(tenderID, language)] = stored
}
// buildRecommendedTenderListResponses builds list responses without MinIO translation lookups.
// Recommendation pages use MongoDB text; tender detail endpoints still enrich from storage.
func (s *tenderService) buildRecommendedTenderListResponses(items []rankedRecommendedTender, language string) []TenderResponse {
if len(items) == 0 {
return nil
}
out := make([]TenderResponse, len(items))
for i, item := range items {
resp := item.tender.ToResponseWithLanguage(language)
resp.Rank = item.rank
resp.Analysis = item.analysis
resp.ProcedureRef = item.tenderRef
out[i] = *resp
}
return out
}
func (s *tenderService) enrichRecommendedTenderResponsesParallel(ctx context.Context, responses []TenderResponse, language string) []TenderResponse {
if len(responses) == 0 || s.aiSummarizerStorage == nil {
return responses
}
out := make([]TenderResponse, len(responses))
if len(responses) == 1 {
out[0] = s.enrichRecommendedTenderResponse(ctx, responses[0], language, newRecommendationTranslationCache())
return out
}
translationCache := newRecommendationTranslationCache()
group, groupCtx := errgroup.WithContext(ctx)
group.SetLimit(recommendationPageCacheConcurrency)
for i := range responses {
i := i
group.Go(func() error {
out[i] = s.enrichRecommendedTenderResponse(groupCtx, responses[i], language, translationCache)
return nil
})
}
if err := group.Wait(); err != nil {
for i, resp := range responses {
out[i] = s.enrichRecommendedTenderResponse(ctx, resp, language, translationCache)
}
}
return out
}
func (s *tenderService) enrichRecommendedTenderResponse(
ctx context.Context,
resp TenderResponse,
language string,
cache *recommendationTranslationCache,
) TenderResponse {
enriched := resp
tender := tenderFromRecommendationResponse(resp)
s.enrichWithTranslationTimeout(
ctx,
&tender,
&enriched,
language,
cache,
recommendTranslationEnrichmentTimeout,
)
return enriched
}
func tenderFromRecommendationResponse(item TenderResponse) Tender {
var tender Tender
if id, err := bson.ObjectIDFromHex(strings.TrimSpace(item.ID)); err == nil {
tender.ID = id
}
tender.ContractFolderID = item.ContractFolderID
tender.NoticePublicationID = item.NoticePublicationID
tender.RelatedNoticePublicationIDs = append([]string(nil), item.RelatedNoticePublicationIDs...)
tender.Status = item.Status
tender.PublicationDate = item.PublicationDate
tender.SubmissionDeadline = item.SubmissionDeadline
tender.TenderDeadline = item.TenderDeadline
return tender
}
func (s *tenderService) loadRecommendationExclusions(
ctx context.Context,
form *SearchForm,
companyID string,
companyIDs []string,
) (map[string]struct{}, map[string]map[string]struct{}, error) {
if !form.ExcludeRejectedTenders {
return nil, nil, nil
}
if len(companyIDs) > 0 {
excludedByCompany, err := s.buildRejectedTenderIDSetsForCompanies(ctx, companyIDs)
if err != nil {
return nil, nil, fmt.Errorf("failed to build recommendation exclusions: %w", err)
}
return nil, excludedByCompany, nil
}
excluded, err := s.buildRejectedTenderIDSet(ctx, companyID)
if err != nil {
return nil, nil, fmt.Errorf("failed to build recommendation exclusions: %w", err)
}
return excluded, nil, nil
}
@@ -0,0 +1,88 @@
package tender
import (
"context"
"io"
"sync/atomic"
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
"tm/pkg/ai_summarizer"
)
type countingTranslationStorage struct {
calls atomic.Int32
}
func (s *countingTranslationStorage) GetSummaryFromStorage(context.Context, string, string) (string, error) {
return "", ai_summarizer.ErrSummaryNotReady
}
func (s *countingTranslationStorage) GetDocumentSummariesFromStorage(context.Context, string, string) ([]ai_summarizer.DocumentSummary, error) {
return nil, nil
}
func (s *countingTranslationStorage) GetTranslationFromStorage(context.Context, string, string, string) (ai_summarizer.StoredTranslation, error) {
s.calls.Add(1)
return ai_summarizer.StoredTranslation{
Title: "Translated title",
Description: "Translated description",
}, nil
}
func (s *countingTranslationStorage) ListDocuments(context.Context, string, string) ([]ai_summarizer.StoredDocument, error) {
return nil, nil
}
func (s *countingTranslationStorage) DownloadDocument(context.Context, string) (io.ReadCloser, error) {
return nil, ai_summarizer.ErrObjectNotFound
}
func TestBuildRecommendedTenderListResponsesPreservesOrderWithoutStorage(t *testing.T) {
storage := &countingTranslationStorage{}
service := &tenderService{
aiSummarizerStorage: storage,
defaultLanguage: "en",
logger: noopTenderTestLogger{},
}
items := make([]rankedRecommendedTender, 0, 4)
ids := []string{
"507f1f77bcf86cd799439011",
"507f1f77bcf86cd799439012",
"507f1f77bcf86cd799439013",
"507f1f77bcf86cd799439014",
}
for i, idHex := range ids {
tender := Tender{
ContractFolderID: "folder",
NoticePublicationID: "notice",
Title: "Original",
}
tender.ID, _ = bson.ObjectIDFromHex(idHex)
items = append(items, rankedRecommendedTender{
tender: tender,
rank: i + 1,
analysis: "fit",
tenderRef: "PROC_folder/notice",
})
}
got := service.buildRecommendedTenderListResponses(items, "fr")
if len(got) != len(items) {
t.Fatalf("len = %d, want %d", len(got), len(items))
}
for i, resp := range got {
wantRank := i + 1
if resp.Rank != wantRank {
t.Fatalf("got[%d].Rank = %d, want %d", i, resp.Rank, wantRank)
}
if resp.Title != "Original" {
t.Fatalf("got[%d].Title = %q, want MongoDB title without MinIO lookup", i, resp.Title)
}
}
if storage.calls.Load() != 0 {
t.Fatalf("translation lookups = %d, want 0", storage.calls.Load())
}
}
@@ -0,0 +1,136 @@
package tender
import (
"context"
"io"
"testing"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
)
type noopTenderTestLogger struct{}
func (noopTenderTestLogger) Debug(string, map[string]interface{}) {}
func (noopTenderTestLogger) Info(string, map[string]interface{}) {}
func (noopTenderTestLogger) Warn(string, map[string]interface{}) {}
func (noopTenderTestLogger) Error(string, map[string]interface{}) {}
func (noopTenderTestLogger) Fatal(string, map[string]interface{}) {}
func (l noopTenderTestLogger) WithFields(map[string]interface{}) logger.Logger {
return l
}
func (noopTenderTestLogger) Sync() error { return nil }
type stubRecommendationTranslationStorage struct {
translations map[string]ai_summarizer.StoredTranslation
}
func (s stubRecommendationTranslationStorage) GetSummaryFromStorage(context.Context, string, string) (string, error) {
return "", ai_summarizer.ErrSummaryNotReady
}
func (s stubRecommendationTranslationStorage) GetDocumentSummariesFromStorage(context.Context, string, string) ([]ai_summarizer.DocumentSummary, error) {
return nil, nil
}
func (s stubRecommendationTranslationStorage) GetTranslationFromStorage(_ context.Context, contractFolderID, noticePublicationID, language string) (ai_summarizer.StoredTranslation, error) {
key := contractFolderID + "|" + noticePublicationID + "|" + language
if translation, ok := s.translations[key]; ok {
return translation, nil
}
return ai_summarizer.StoredTranslation{}, ai_summarizer.ErrTranslationNotReady
}
func (s stubRecommendationTranslationStorage) ListDocuments(context.Context, string, string) ([]ai_summarizer.StoredDocument, error) {
return nil, nil
}
func (s stubRecommendationTranslationStorage) DownloadDocument(context.Context, string) (io.ReadCloser, error) {
return nil, ai_summarizer.ErrObjectNotFound
}
func TestBuildRecommendedTenderResponseAppliesStoredTranslation(t *testing.T) {
service := &tenderService{
aiSummarizerStorage: stubRecommendationTranslationStorage{
translations: map[string]ai_summarizer.StoredTranslation{
"folder-1|00123456-2026|fr": {
Title: "Titre traduit",
Description: "Description traduite",
},
},
},
defaultLanguage: "en",
logger: noopTenderTestLogger{},
}
tender := Tender{
ContractFolderID: "folder-1",
NoticePublicationID: "00123456-2026",
Title: "Original title",
Description: "Original description",
}
resp := service.buildRecommendedTenderResponse(
context.Background(),
tender,
3,
"Strong fit",
"PROC_folder-1/00123456-2026",
"fr",
nil,
)
if resp.Title != "Titre traduit" {
t.Fatalf("expected translated title, got %q", resp.Title)
}
if resp.Description != "Description traduite" {
t.Fatalf("expected translated description, got %q", resp.Description)
}
if resp.Language != "fr" {
t.Fatalf("expected response language fr, got %q", resp.Language)
}
if resp.Rank != 3 {
t.Fatalf("expected rank 3, got %d", resp.Rank)
}
if resp.Analysis != "Strong fit" {
t.Fatalf("expected analysis to be preserved, got %q", resp.Analysis)
}
if resp.ProcedureRef != "PROC_folder-1/00123456-2026" {
t.Fatalf("expected procedure ref to be preserved, got %q", resp.ProcedureRef)
}
}
func TestBuildRecommendedTenderResponseKeepsOriginalTextWithoutStoredTranslation(t *testing.T) {
service := &tenderService{
aiSummarizerStorage: stubRecommendationTranslationStorage{},
defaultLanguage: "en",
logger: noopTenderTestLogger{},
}
tender := Tender{
ContractFolderID: "folder-1",
NoticePublicationID: "00123456-2026",
Title: "Original title",
Description: "Original description",
}
resp := service.buildRecommendedTenderResponse(
context.Background(),
tender,
1,
"Fallback",
"PROC_folder-1/00123456-2026",
"fr",
nil,
)
if resp.Title != "Original title" {
t.Fatalf("expected original title, got %q", resp.Title)
}
if resp.Description != "Original description" {
t.Fatalf("expected original description, got %q", resp.Description)
}
if resp.Language != "" {
t.Fatalf("expected empty language when no translation exists, got %q", resp.Language)
}
}
+394 -60
View File
@@ -13,6 +13,9 @@ import (
"go.mongodb.org/mongo-driver/v2/mongo"
)
// maxContractFolderInClause caps each MongoDB $in batch for documents_scraped MinIO cross-check filters.
const maxContractFolderInClause = 5000
// TenderRepository interface defines tender data access methods
type TenderRepository interface {
// Tender CRUD operations
@@ -21,17 +24,18 @@ type TenderRepository interface {
GetByIDs(ctx context.Context, ids []string) ([]Tender, error)
GetByContractNoticeID(ctx context.Context, contractNoticeID string) (*Tender, error)
GetByNoticePublicationID(ctx context.Context, noticePublicationID string) (*Tender, error)
GetByProcedureReference(ctx context.Context, contractFolderID, noticePublicationID string) (*Tender, error)
GetByProcedureReferences(ctx context.Context, refs []ProcedureReference) (map[string]Tender, error)
GetByContractFolderID(ctx context.Context, contractFolderID string) (*Tender, error)
// GetLatestByContractFolderIDs returns the most recently updated tender per distinct contract_folder_id.
GetLatestByContractFolderIDs(ctx context.Context, contractFolderIDs []string) (map[string]*Tender, error)
GetByProcurementProjectID(ctx context.Context, procurementProjectID string) (*Tender, error)
FindTendersWithContentXML(ctx context.Context, limit, skip int) ([]Tender, error)
GetTenderCountByCountry(ctx context.Context) (map[string]int64, error)
GetTenderCountByType(ctx context.Context) (map[string]int64, error)
GetTenderCountByClassification(ctx context.Context) (map[string]int64, error)
Search(ctx context.Context, form *SearchForm, pagination *response.Pagination) (*SearchPageResult, error)
GetExpiredTenders(ctx context.Context, beforeDate int64) ([]Tender, error)
GetPendingDocumentScrapeTenders(ctx context.Context, countryCodes []string, minDeadline int64, limit, skip int) ([]Tender, bool, error)
GetPendingDocumentScrapeTenders(ctx context.Context, filter PendingDocumentScrapeFilter) ([]Tender, bool, error)
GetUnSummarizedTenders(ctx context.Context, limit, skip int) ([]Tender, int64, error)
GetUnTranslatedTenders(ctx context.Context, language string, limit, skip int) ([]Tender, int64, error)
Update(ctx context.Context, tender *Tender) error
@@ -62,6 +66,20 @@ func tenderSearchListProjection() bson.M {
"modifications": 0,
"source_file_url": 0,
"source_file_name": 0,
// Detail-only or bulky fields; list rows use title and summary metadata only.
"description": 0,
"ai_overall_summary": 0,
"procurement_lots": 0,
"awarded": 0,
"cancellation_reason": 0,
"suspension_reason": 0,
"translations": 0,
"organizations": 0,
"review_organization": 0,
"processing_metadata.enrichment_data": 0,
"processing_metadata.translated_data": 0,
"processing_metadata.parsing_errors": 0,
"processing_metadata.validation_errors": 0,
}
}
@@ -74,6 +92,8 @@ func documentScraperListProjection() bson.M {
"tender_url": 1,
"title": 1,
"description": 1,
"publication_date": 1,
"submission_deadline": 1,
}
}
@@ -112,6 +132,15 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Te
*orm.NewIndex("main_classification_idx", bson.D{{Key: "main_classification", Value: 1}}),
*orm.NewIndex("publication_date_idx", bson.D{{Key: "publication_date", Value: -1}}),
*orm.NewIndex("tender_deadline_idx", bson.D{{Key: "tender_deadline", Value: 1}}),
*orm.NewIndex("submission_deadline_idx", bson.D{{Key: "submission_deadline", Value: 1}}),
*orm.NewIndex("closing_soon_submission_idx", bson.D{
{Key: "submission_deadline", Value: 1},
{Key: "status", Value: 1},
}),
*orm.NewIndex("closing_soon_tender_deadline_idx", bson.D{
{Key: "tender_deadline", Value: 1},
{Key: "status", Value: 1},
}),
*orm.NewIndex("estimated_value_idx", bson.D{{Key: "estimated_value", Value: -1}}),
*orm.NewIndex("created_at_idx", bson.D{{Key: "created_at", Value: -1}}),
// Stable keyset pagination for default admin list (created_at desc).
@@ -154,6 +183,13 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Te
{Key: "processing_metadata.documents_scraped", Value: 1},
{Key: "processing_metadata.documents_scraped_at", Value: -1},
}),
// Panel list: documents_scraped=true sorted by created_at desc (keyset pagination).
*orm.NewIndex("documents_scraped_created_at_id_idx", bson.D{
{Key: "created_at", Value: -1},
{Key: "_id", Value: -1},
}).WithPartialFilterExpression(bson.M{
"processing_metadata.documents_scraped": true,
}),
*orm.NewIndex("scraped_documents_scraped_at_idx", bson.D{
{Key: "scraped_documents.scraped_at", Value: 1},
}),
@@ -227,25 +263,49 @@ func (r *tenderRepository) GetByIDs(ctx context.Context, ids []string) ([]Tender
return nil, nil
}
objectIDs := make([]bson.ObjectID, 0, len(ids))
for _, id := range ids {
objectID, err := bson.ObjectIDFromHex(id)
const batchSize = 500
out := make([]Tender, 0, len(ids))
for start := 0; start < len(ids); start += batchSize {
end := start + batchSize
if end > len(ids) {
end = len(ids)
}
batch := ids[start:end]
objectIDs := make([]bson.ObjectID, 0, len(batch))
for _, id := range batch {
objectID, err := bson.ObjectIDFromHex(strings.TrimSpace(id))
if err != nil {
r.logger.Warn("Skipping invalid tender ID in batch lookup", map[string]interface{}{
"tender_id": id,
"error": err.Error(),
})
continue
}
objectIDs = append(objectIDs, objectID)
}
if len(objectIDs) == 0 {
continue
}
pagination := orm.Pagination{
Limit: len(objectIDs),
SortField: "_id",
SortOrder: -1,
SkipCount: true,
}
result, err := r.ormRepo.FindAll(ctx, bson.M{"_id": bson.M{"$in": objectIDs}}, pagination)
if err != nil {
r.logger.Error("Failed to get tenders by IDs", map[string]interface{}{
"count": len(batch),
"error": err.Error(),
})
return nil, err
}
objectIDs = append(objectIDs, objectID)
out = append(out, result.Items...)
}
result, err := r.ormRepo.FindAll(ctx, bson.M{"_id": bson.M{"$in": objectIDs}}, orm.NewPaginationBuilder().Build())
if err != nil {
r.logger.Error("Failed to get tenders by IDs", map[string]interface{}{
"count": len(ids),
"error": err.Error(),
})
return nil, err
}
return result.Items, nil
return out, nil
}
// GetByContractNoticeID retrieves a tender by contract notice ID
@@ -297,6 +357,204 @@ func (r *tenderRepository) GetByNoticePublicationID(ctx context.Context, noticeP
return &result.Items[0], nil
}
// GetByProcedureReference retrieves a tender by contract folder and notice publication id.
// Matches canonical notice_publication_id and related_notice_publication_ids.
func (r *tenderRepository) GetByProcedureReference(ctx context.Context, contractFolderID, noticePublicationID string) (*Tender, error) {
contractFolderID = strings.TrimSpace(contractFolderID)
noticePublicationID = strings.TrimSpace(noticePublicationID)
if contractFolderID == "" || noticePublicationID == "" {
return nil, orm.ErrDocumentNotFound
}
filter := bson.M{
"contract_folder_id": contractFolderID,
"$or": []bson.M{
{"notice_publication_id": noticePublicationID},
{"related_notice_publication_ids": noticePublicationID},
},
}
pagination := orm.Pagination{Limit: 1, SortField: "updated_at", SortOrder: -1}
result, err := r.ormRepo.FindAll(ctx, filter, pagination)
if err != nil {
r.logger.Error("Failed to get tender by procedure reference", map[string]interface{}{
"contract_folder_id": contractFolderID,
"notice_publication_id": noticePublicationID,
"error": err.Error(),
})
return nil, err
}
if len(result.Items) == 0 {
return nil, orm.ErrDocumentNotFound
}
return &result.Items[0], nil
}
const procedureReferenceLookupBatchSize = 250
const contractFolderLookupBatchSize = 250
// GetByProcedureReferences loads tenders for multiple AI procedure refs.
// It combines indexed notice lookups with contract-folder candidates so older notice
// references still resolve after tender merges (same behavior as before the perf refactor).
func (r *tenderRepository) GetByProcedureReferences(ctx context.Context, refs []ProcedureReference) (map[string]Tender, error) {
out := make(map[string]Tender)
if len(refs) == 0 {
return out, nil
}
folderIDs := make([]string, 0, len(refs))
noticeIDs := make([]string, 0, len(refs))
for _, ref := range refs {
folder := strings.TrimSpace(ref.ContractFolderID)
notice := strings.TrimSpace(ref.NoticePublicationID)
if folder == "" || notice == "" {
continue
}
folderIDs = append(folderIDs, folder)
noticeIDs = append(noticeIDs, notice)
}
uniqueNotices := uniqueNonEmptyTrimmedStrings(noticeIDs)
noticeCandidates := make([]Tender, 0, len(uniqueNotices))
for start := 0; start < len(uniqueNotices); start += procedureReferenceLookupBatchSize {
end := start + procedureReferenceLookupBatchSize
if end > len(uniqueNotices) {
end = len(uniqueNotices)
}
batch, err := r.findTendersByNoticePublicationIDs(ctx, uniqueNotices[start:end])
if err != nil {
return nil, err
}
noticeCandidates = append(noticeCandidates, batch...)
}
folderCandidates, err := r.findTendersByContractFolderIDs(ctx, folderIDs)
if err != nil {
return nil, err
}
candidates := mergeTenderCandidates(noticeCandidates, folderCandidates)
return mapProcedureReferencesToTenders(candidates, refs), nil
}
func (r *tenderRepository) findTendersByContractFolderIDs(ctx context.Context, folderIDs []string) ([]Tender, error) {
uniqueFolders := uniqueNonEmptyTrimmedStrings(folderIDs)
if len(uniqueFolders) == 0 {
return nil, nil
}
out := make([]Tender, 0, len(uniqueFolders))
for start := 0; start < len(uniqueFolders); start += contractFolderLookupBatchSize {
end := start + contractFolderLookupBatchSize
if end > len(uniqueFolders) {
end = len(uniqueFolders)
}
batch := uniqueFolders[start:end]
findLimit := len(batch) * 5
if minLimit := len(batch) + 1; findLimit < minLimit {
findLimit = minLimit
}
filter := bson.M{"contract_folder_id": bson.M{"$in": batch}}
pagination := orm.Pagination{
Limit: findLimit,
SortField: "updated_at",
SortOrder: -1,
SkipCount: true,
}
result, err := r.ormRepo.FindAll(ctx, filter, pagination)
if err != nil {
r.logger.Error("Failed to get tenders by contract folder IDs", map[string]interface{}{
"count": len(batch),
"error": err.Error(),
})
return nil, err
}
out = append(out, result.Items...)
}
return out, nil
}
func mergeTenderCandidates(groups ...[]Tender) []Tender {
seen := make(map[string]struct{})
out := make([]Tender, 0)
for _, group := range groups {
for i := range group {
tenderID := strings.TrimSpace(group[i].GetID())
if tenderID == "" {
out = append(out, group[i])
continue
}
if _, exists := seen[tenderID]; exists {
continue
}
seen[tenderID] = struct{}{}
out = append(out, group[i])
}
}
return out
}
func (r *tenderRepository) findTendersByNoticePublicationIDs(ctx context.Context, noticeIDs []string) ([]Tender, error) {
if len(noticeIDs) == 0 {
return nil, nil
}
filter := bson.M{
"$or": []bson.M{
{"notice_publication_id": bson.M{"$in": noticeIDs}},
{"related_notice_publication_ids": bson.M{"$in": noticeIDs}},
},
}
pagination := orm.Pagination{
Limit: len(noticeIDs) * 3,
SortField: "updated_at",
SortOrder: -1,
SkipCount: true,
}
result, err := r.ormRepo.FindAll(ctx, filter, pagination)
if err != nil {
r.logger.Error("Failed to get tenders by notice publication IDs", map[string]interface{}{
"count": len(noticeIDs),
"error": err.Error(),
})
return nil, err
}
return result.Items, nil
}
func mapProcedureReferencesToTenders(candidates []Tender, refs []ProcedureReference) map[string]Tender {
byFolder := make(map[string][]Tender)
for i := range candidates {
folder := strings.TrimSpace(candidates[i].ContractFolderID)
if folder == "" {
continue
}
byFolder[folder] = append(byFolder[folder], candidates[i])
}
out := make(map[string]Tender, len(refs))
for _, ref := range refs {
folder := strings.TrimSpace(ref.ContractFolderID)
for _, candidate := range byFolder[folder] {
if tenderMatchesProcedureRef(candidate, ref.ContractFolderID, ref.NoticePublicationID) {
out[ref.Ref] = candidate
break
}
}
}
return out
}
// GetByContractFolderID retrieves a tender by its procedure-level identifier (ContractFolderID).
// TED groups multiple notice publications (e.g. competition notice + result notice) under the same
// ContractFolderID, so this lookup is the authoritative way to find the existing tender for a procedure
@@ -420,27 +678,6 @@ func (r *tenderRepository) GetByProcurementProjectID(ctx context.Context, procur
return &result.Items[0], nil
}
// FindTendersWithContentXML returns tenders that still have TED XML (for backfill after notices were deleted).
func (r *tenderRepository) FindTendersWithContentXML(ctx context.Context, limit, skip int) ([]Tender, error) {
filter := bson.M{"content_xml": bson.M{"$exists": true, "$ne": ""}}
pagination := orm.Pagination{
Limit: limit,
Skip: skip,
SortField: "_id",
SortOrder: 1,
}
result, err := r.ormRepo.FindAll(ctx, filter, pagination)
if err != nil {
r.logger.Error("Failed to list tenders with content_xml", map[string]interface{}{
"error": err.Error(),
})
return nil, err
}
return result.Items, nil
}
// Update updates an existing tender
func (r *tenderRepository) Update(ctx context.Context, tender *Tender) error {
tender.UpdatedAt = time.Now().Unix()
@@ -493,6 +730,15 @@ func (r *tenderRepository) Search(ctx context.Context, form *SearchForm, paginat
sortOrder = pagination.SortOrder
}
listOpts := orm.ListPaginationOptions{Projection: tenderSearchListProjection()}
if pagination.IncludeTotal {
if pagination.Cursor != "" {
listOpts.IncludeCount = true
}
} else {
listOpts.SkipCount = true
}
mongoPagination, err := orm.BuildListPagination(
pagination.Limit,
pagination.Offset,
@@ -500,7 +746,7 @@ func (r *tenderRepository) Search(ctx context.Context, form *SearchForm, paginat
sort,
sortOrder,
filter,
orm.ListPaginationOptions{Projection: tenderSearchListProjection()},
listOpts,
)
if err != nil {
return nil, err
@@ -555,37 +801,101 @@ func (r *tenderRepository) GetExpiredTenders(ctx context.Context, beforeDate int
return result.Items, nil
}
// GetPendingDocumentScrapeTenders returns unscraped tenders for the given countries with an active deadline.
func (r *tenderRepository) GetPendingDocumentScrapeTenders(ctx context.Context, countryCodes []string, minDeadline int64, limit, skip int) ([]Tender, bool, error) {
if len(countryCodes) == 0 {
// PendingDocumentScrapeFilter defines query criteria for pending document scrape tenders.
type PendingDocumentScrapeFilter struct {
Portals []string
MinWindowEnd *int64 // when set, only tenders with an active publication submission window
CreatedAtFrom *int64
CreatedAtTo *int64
Limit int
Skip int
}
func scrapePortalURLFilter(portals []string) bson.M {
conditions := make([]bson.M, 0, len(portals)*2)
for _, portal := range portals {
portal = strings.TrimSpace(portal)
if portal == "" {
continue
}
regex := bson.M{"$regex": regexp.QuoteMeta(portal), "$options": "i"}
conditions = append(conditions,
bson.M{"document_uri": regex},
bson.M{"tender_url": regex},
)
}
if len(conditions) == 0 {
return nil
}
return bson.M{"$or": conditions}
}
func publicationWindowActiveBSON(now int64) bson.M {
return bson.M{
"$or": []bson.M{
{"submission_deadline": bson.M{"$gt": now}},
{
"$and": []bson.M{
{"$or": []bson.M{
{"submission_deadline": bson.M{"$exists": false}},
{"submission_deadline": 0},
}},
{"publication_date": bson.M{"$gt": 0}},
},
},
},
}
}
// GetPendingDocumentScrapeTenders returns unscraped tenders whose document URL matches a supported portal.
// When MinWindowEnd is set, only tenders whose publication-based submission window is still open are returned.
func (r *tenderRepository) GetPendingDocumentScrapeTenders(ctx context.Context, filter PendingDocumentScrapeFilter) ([]Tender, bool, error) {
portalFilter := scrapePortalURLFilter(filter.Portals)
if portalFilter == nil {
return nil, false, nil
}
filter := bson.M{
"country_code": bson.M{"$in": countryCodes},
"tender_deadline": bson.M{
"$gt": minDeadline,
},
"processing_metadata.documents_scraped": bson.M{"$ne": true},
andFilters := []bson.M{
{"processing_metadata.documents_scraped": bson.M{"$ne": true}},
portalFilter,
}
if filter.MinWindowEnd != nil {
andFilters = append(andFilters, publicationWindowActiveBSON(*filter.MinWindowEnd))
}
if filter.CreatedAtFrom != nil || filter.CreatedAtTo != nil {
createdFilter := bson.M{}
if filter.CreatedAtFrom != nil {
createdFilter["$gte"] = *filter.CreatedAtFrom
}
if filter.CreatedAtTo != nil {
createdFilter["$lte"] = *filter.CreatedAtTo
}
andFilters = append(andFilters, bson.M{"created_at": createdFilter})
}
query := bson.M{"$and": andFilters}
pagination := orm.Pagination{
Limit: limit,
Skip: skip,
Limit: filter.Limit,
Skip: filter.Skip,
SortField: "created_at",
SortOrder: 1,
SkipCount: true,
Projection: documentScraperListProjection(),
}
result, err := r.ormRepo.FindAll(ctx, filter, pagination)
result, err := r.ormRepo.FindAll(ctx, query, pagination)
if err != nil {
r.logger.Error("Failed to get pending document scrape tenders", map[string]interface{}{
"country_codes": countryCodes,
"min_deadline": minDeadline,
"limit": limit,
"skip": skip,
"error": err.Error(),
"portals": filter.Portals,
"min_window_end": filter.MinWindowEnd,
"created_at_from": filter.CreatedAtFrom,
"created_at_to": filter.CreatedAtTo,
"limit": filter.Limit,
"skip": filter.Skip,
"error": err.Error(),
})
return nil, false, err
}
@@ -1073,11 +1383,7 @@ func (r *tenderRepository) buildSearchFilter(form *SearchForm) bson.M {
}
if form.DocumentsScraped {
folderIDs := dedupeNonEmptyStrings(form.ContractFolderIDsWithDocuments)
scrapedClause := bson.M{"contract_folder_id": bson.M{"$in": folderIDs}}
if len(folderIDs) == 0 {
scrapedClause = bson.M{"_id": bson.M{"$exists": false}}
}
scrapedClause := documentsScrapedSearchClause(form)
if len(filter) == 0 {
return scrapedClause
}
@@ -1096,3 +1402,31 @@ func (r *tenderRepository) buildSearchFilter(form *SearchForm) bson.M {
return filter
}
// documentsScrapedSearchClause restricts list queries to tenders whose contract_folder_id
// currently has documents in MinIO (folder IDs are populated by the service layer).
func documentsScrapedSearchClause(form *SearchForm) bson.M {
if form == nil {
return bson.M{"contract_folder_id": bson.M{"$in": []string{}}}
}
return contractFolderIDsSearchClause(form.ContractFolderIDsWithDocuments)
}
func contractFolderIDsSearchClause(folderIDs []string) bson.M {
if len(folderIDs) == 0 {
return bson.M{"contract_folder_id": bson.M{"$in": []string{}}}
}
if len(folderIDs) <= maxContractFolderInClause {
return bson.M{"contract_folder_id": bson.M{"$in": folderIDs}}
}
or := make([]bson.M, 0, (len(folderIDs)+maxContractFolderInClause-1)/maxContractFolderInClause)
for i := 0; i < len(folderIDs); i += maxContractFolderInClause {
end := i + maxContractFolderInClause
if end > len(folderIDs) {
end = len(folderIDs)
}
or = append(or, bson.M{"contract_folder_id": bson.M{"$in": folderIDs[i:end]}})
}
return bson.M{"$or": or}
}
+32
View File
@@ -0,0 +1,32 @@
package tender
import (
"testing"
"go.mongodb.org/mongo-driver/v2/bson"
)
func TestBuildSearchFilterDocumentsScrapedWithoutFolderIDsMatchesNothing(t *testing.T) {
repo := &tenderRepository{}
filter := repo.buildSearchFilter(&SearchForm{DocumentsScraped: true})
in, ok := filter["contract_folder_id"].(bson.M)["$in"].([]string)
if !ok || len(in) != 0 {
t.Fatalf("expected empty contract_folder_id $in filter, got %#v", filter)
}
}
func TestTenderSearchListProjectionExcludesHeavyFields(t *testing.T) {
projection := tenderSearchListProjection()
for _, field := range []string{
"content_xml",
"description",
"ai_overall_summary",
"procurement_lots",
"document_summaries",
} {
if projection[field] != 0 {
t.Fatalf("expected %s to be excluded from list projection", field)
}
}
}
+240
View File
@@ -0,0 +1,240 @@
package tender
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
goredis "github.com/redis/go-redis/v9"
"tm/pkg/response"
)
const (
searchListCacheTTL = 60 * time.Second
// searchListCacheStaleTTL bounds how long the default unfiltered list may lag after
// writes. Only isCacheableSearchList queries use this cache (~5 min max staleness).
searchListCacheStaleTTL = 5 * time.Minute
searchListCacheKeyPrefix = "tender:search:list:"
)
type searchListCacheEntry struct {
expiresAt time.Time
staleUntil time.Time
value *SearchResponse
}
type cachedSearchListPayload struct {
Tenders []TenderResponse `json:"tenders"`
Metadata *response.Meta `json:"metadata,omitempty"`
}
func searchResponseToCachePayload(resp *SearchResponse) *cachedSearchListPayload {
if resp == nil {
return nil
}
return &cachedSearchListPayload{
Tenders: resp.Tenders,
Metadata: resp.Metadata,
}
}
func cachePayloadToSearchResponse(payload *cachedSearchListPayload) *SearchResponse {
if payload == nil {
return nil
}
return &SearchResponse{
Tenders: payload.Tenders,
Metadata: payload.Metadata,
}
}
func isCacheableSearchList(form *SearchForm, pagination *response.Pagination) bool {
if form == nil || pagination == nil {
return false
}
if form.HasListFilters() {
return false
}
if form.CompanyID != nil && strings.TrimSpace(*form.CompanyID) != "" {
return false
}
if form.CustomerID != nil && strings.TrimSpace(*form.CustomerID) != "" {
return false
}
if len(form.CompanyIDs) > 0 {
return false
}
if pagination.Cursor != "" || pagination.IncludeTotal || pagination.Offset != 0 {
return false
}
if pagination.Limit <= 0 || pagination.Limit > 20 {
return false
}
sortBy := pagination.SortBy
if sortBy == "" {
sortBy = "created_at"
}
sortOrder := pagination.SortOrder
if sortOrder == "" {
sortOrder = "desc"
}
return sortBy == "created_at" && sortOrder == "desc"
}
func searchListCacheKey(pagination *response.Pagination, language string) string {
return fmt.Sprintf("%d:%s", pagination.Limit, strings.ToLower(strings.TrimSpace(language)))
}
func (s *tenderService) cachedSearchList(ctx context.Context, form *SearchForm, pagination *response.Pagination, language string, load func(context.Context) (*SearchResponse, error)) (*SearchResponse, error) {
if !isCacheableSearchList(form, pagination) {
return load(ctx)
}
// Cached *SearchResponse values (including the Tenders slice) are shared across
// concurrent callers via singleflight and stale hits; treat as read-only.
cacheKey := searchListCacheKey(pagination, language)
redisKey := searchListCacheKeyPrefix + cacheKey
if cached, ok := s.getFreshSearchListCache(cacheKey); ok {
return cached, nil
}
if stale, ok := s.getStaleSearchListCache(cacheKey); ok {
go s.refreshSearchListCache(cacheKey, redisKey, load)
return stale, nil
}
if cached, ok := s.getRedisSearchListCache(ctx, redisKey); ok {
s.storeSearchListCache(cacheKey, cached)
go s.refreshSearchListCache(cacheKey, redisKey, load)
return cached, nil
}
out, err, _ := s.searchListCacheGroup.Do(cacheKey, func() (interface{}, error) {
if cached, ok := s.getFreshSearchListCache(cacheKey); ok {
return cached, nil
}
result, err := load(context.WithoutCancel(ctx))
if err != nil {
return nil, err
}
s.storeSearchListCache(cacheKey, result)
s.storeRedisSearchListCache(context.Background(), redisKey, result)
return result, nil
})
if err != nil {
return nil, err
}
return out.(*SearchResponse), nil
}
func (s *tenderService) refreshSearchListCache(cacheKey, redisKey string, load func(context.Context) (*SearchResponse, error)) {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
result, err := load(ctx)
if err != nil {
s.logger.Warn("Failed to refresh tender search list cache", map[string]interface{}{
"cache_key": cacheKey,
"error": err.Error(),
})
return
}
s.storeSearchListCache(cacheKey, result)
s.storeRedisSearchListCache(context.Background(), redisKey, result)
}
func (s *tenderService) getFreshSearchListCache(cacheKey string) (*SearchResponse, bool) {
s.searchListCacheMu.Lock()
defer s.searchListCacheMu.Unlock()
entry, ok := s.searchListCache[cacheKey]
if !ok || time.Now().After(entry.expiresAt) {
return nil, false
}
return entry.value, true
}
func (s *tenderService) getStaleSearchListCache(cacheKey string) (*SearchResponse, bool) {
s.searchListCacheMu.Lock()
defer s.searchListCacheMu.Unlock()
entry, ok := s.searchListCache[cacheKey]
if !ok || time.Now().After(entry.staleUntil) {
if ok {
delete(s.searchListCache, cacheKey)
}
return nil, false
}
return entry.value, true
}
func (s *tenderService) storeSearchListCache(cacheKey string, value *SearchResponse) {
now := time.Now()
s.searchListCacheMu.Lock()
defer s.searchListCacheMu.Unlock()
s.searchListCache[cacheKey] = searchListCacheEntry{
expiresAt: now.Add(searchListCacheTTL),
staleUntil: now.Add(searchListCacheTTL + searchListCacheStaleTTL),
value: value,
}
}
func (s *tenderService) getRedisSearchListCache(ctx context.Context, redisKey string) (*SearchResponse, bool) {
if s.redisClient == nil {
return nil, false
}
raw, err := s.redisClient.Get(ctx, redisKey)
if err != nil {
if err != goredis.Nil {
s.logger.Warn("Failed to read tender search list cache from Redis", map[string]interface{}{
"cache_key": redisKey,
"error": err.Error(),
})
}
return nil, false
}
var cached cachedSearchListPayload
if err := json.Unmarshal([]byte(raw), &cached); err != nil {
s.logger.Warn("Failed to decode tender search list cache from Redis", map[string]interface{}{
"cache_key": redisKey,
"error": err.Error(),
})
_ = s.redisClient.Del(ctx, redisKey)
return nil, false
}
return cachePayloadToSearchResponse(&cached), true
}
func (s *tenderService) storeRedisSearchListCache(ctx context.Context, redisKey string, value *SearchResponse) {
if s.redisClient == nil || value == nil {
return
}
encoded, err := json.Marshal(searchResponseToCachePayload(value))
if err != nil {
s.logger.Warn("Failed to encode tender search list cache for Redis", map[string]interface{}{
"cache_key": redisKey,
"error": err.Error(),
})
return
}
if err := s.redisClient.Set(ctx, redisKey, string(encoded), searchListCacheTTL+searchListCacheStaleTTL); err != nil {
s.logger.Warn("Failed to store tender search list cache in Redis", map[string]interface{}{
"cache_key": redisKey,
"error": err.Error(),
})
}
}
+431 -204
View File
@@ -10,13 +10,18 @@ import (
"path"
"path/filepath"
"strings"
"sync"
"time"
"tm/internal/company"
"tm/internal/customer"
"tm/pkg/ai_summarizer"
"tm/pkg/logger"
orm "tm/pkg/mongo"
"tm/pkg/redis"
"tm/pkg/response"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/singleflight"
)
// AISummarizerClient defines the interface for on-demand AI operations.
@@ -52,6 +57,8 @@ type Service interface {
// Tender operations
GetByID(ctx context.Context, id string, language string) (*TenderResponse, error)
GetByIDs(ctx context.Context, ids []string) (map[string]*TenderResponse, error)
// ResolveMongoID maps a MongoDB id or AI procedure ref to the canonical tender MongoDB id.
ResolveMongoID(ctx context.Context, tenderRef string) (string, error)
Update(ctx context.Context, req UpdateTenderRequest) (*TenderResponse, error)
Delete(ctx context.Context, id string) error
Search(ctx context.Context, req *SearchForm, pagination *response.Pagination) (*SearchResponse, error)
@@ -80,18 +87,29 @@ type Service interface {
// tenderService implements TenderService interface
type tenderService struct {
repository TenderRepository
companyService company.Service
customerService customer.Service
logger logger.Logger
aiSummarizerClient AISummarizerClient
aiSummarizerStorage AISummarizerStorage
defaultLanguage string
repository TenderRepository
companyService company.Service
customerService customer.Service
rejectedTenderLister CompanyRejectedTenderLister
logger logger.Logger
aiSummarizerClient AISummarizerClient
aiSummarizerStorage AISummarizerStorage
redisClient redis.Client
pageCacheTTL time.Duration
defaultLanguage string
searchListCacheMu sync.Mutex
searchListCache map[string]searchListCacheEntry
searchListCacheGroup singleflight.Group
contractFoldersCacheMu sync.Mutex
contractFoldersCache contractFoldersWithDocsCacheEntry
contractFoldersCacheGroup singleflight.Group
}
const (
aiSummaryEnrichmentTimeout = 2 * time.Second
aiTranslationEnrichmentTimeout = 2 * time.Second
aiSummaryEnrichmentTimeout = 2 * time.Second
aiTranslationEnrichmentTimeout = 2 * time.Second
recommendTranslationEnrichmentTimeout = 5 * time.Second
)
var (
@@ -100,19 +118,34 @@ var (
)
// NewService creates a new tender service
func NewService(repository TenderRepository, companyService company.Service, customerService customer.Service, logger logger.Logger, aiClient AISummarizerClient, aiStorage AISummarizerStorage, defaultLanguage string) Service {
func NewService(
repository TenderRepository,
companyService company.Service,
customerService customer.Service,
rejectedTenderLister CompanyRejectedTenderLister,
logger logger.Logger,
aiClient AISummarizerClient,
aiStorage AISummarizerStorage,
redisClient redis.Client,
recommendationPageCacheTTL time.Duration,
defaultLanguage string,
) Service {
if strings.TrimSpace(defaultLanguage) == "" {
defaultLanguage = "en"
}
return &tenderService{
repository: repository,
companyService: companyService,
customerService: customerService,
logger: logger,
aiSummarizerClient: aiClient,
aiSummarizerStorage: aiStorage,
defaultLanguage: strings.ToLower(defaultLanguage),
repository: repository,
companyService: companyService,
customerService: customerService,
rejectedTenderLister: rejectedTenderLister,
logger: logger,
aiSummarizerClient: aiClient,
aiSummarizerStorage: aiStorage,
redisClient: redisClient,
pageCacheTTL: recommendationPageCacheTTL,
defaultLanguage: strings.ToLower(defaultLanguage),
searchListCache: make(map[string]searchListCacheEntry),
}
}
@@ -160,57 +193,80 @@ func (s *tenderService) GetByIDs(ctx context.Context, ids []string) (map[string]
return out, nil
}
// ResolveMongoID maps a MongoDB id or AI procedure ref (PROC_{folder}/{notice}) to the canonical tender id.
func (s *tenderService) ResolveMongoID(ctx context.Context, tenderRef string) (string, error) {
tenderRef = strings.TrimSpace(tenderRef)
if tenderRef == "" {
return "", fmt.Errorf("tender reference is required")
}
if IsMongoObjectIDRef(tenderRef) {
if _, err := s.repository.GetByID(ctx, tenderRef); err != nil {
return "", fmt.Errorf("tender not found")
}
return tenderRef, nil
}
if folder, notice, ok := ParseAIProcedureRef(tenderRef); ok {
t, err := s.repository.GetByProcedureReference(ctx, folder, notice)
if err != nil || t == nil {
return "", fmt.Errorf("tender not found")
}
return t.GetID(), nil
}
if _, err := s.repository.GetByID(ctx, tenderRef); err == nil {
return tenderRef, nil
}
return "", fmt.Errorf("tender not found")
}
// buildDetailResponse maps a tender to API form and enriches translation (MinIO) and summary.
func (s *tenderService) buildDetailResponse(ctx context.Context, tender *Tender, language string) *TenderResponse {
resp := tender.ToResponseWithLanguage("")
s.enrichWithTranslation(ctx, tender, resp, language)
s.enrichWithTranslation(ctx, tender, resp, language, nil)
s.enrichWithAISummary(ctx, tender, resp)
return resp
}
// enrichWithAISummary loads overall_summary from MinIO (PROC_<contractFolderID>/<noticePublicationID>/tender.json)
// and attaches it to the response. The primary notice publication id is tried first, then related ids.
// A previously stored ai_overall_summary on the tender (already mapped in ToResponse) is kept when MinIO
// has no ready summary. On success the tender document is updated so list/search can return it without
// another MinIO round-trip.
// enrichWithAISummary attaches the overall AI summary to a tender detail response.
// It reads from MinIO when available; for scraped tenders without a summary it triggers
// on-demand summarization via the AI service (same path as GetAISummary).
func (s *tenderService) enrichWithAISummary(ctx context.Context, t *Tender, resp *TenderResponse) {
if t == nil || resp == nil {
return
}
if s.aiSummarizerStorage == nil || strings.TrimSpace(t.ContractFolderID) == "" {
return
}
enrichCtx, cancel := context.WithTimeout(ctx, aiSummaryEnrichmentTimeout)
defer cancel()
summary, usedNotice, err := s.lookupSummaryForTender(enrichCtx, t)
result, err := s.resolveAISummary(ctx, t)
if err != nil {
s.logger.Debug("AI summary not available for tender", map[string]interface{}{
"tender_id": resp.ID,
"contract_folder_id": t.ContractFolderID,
"notice_id": t.NoticePublicationID,
"error": err.Error(),
s.logger.Debug("AI summary resolution failed for tender detail", map[string]interface{}{
"tender_id": resp.ID,
"error": err.Error(),
})
return
}
resp.OverallSummary = summary
if usedNotice != t.NoticePublicationID {
s.logger.Debug("AI summary resolved from related notice", map[string]interface{}{
"tender_id": resp.ID,
"contract_folder_id": t.ContractFolderID,
"primary_notice_id": t.NoticePublicationID,
"used_notice_id": usedNotice,
})
if result == nil || strings.TrimSpace(result.OverallSummary) == "" {
return
}
s.persistAIOverallSummary(enrichCtx, t, summary)
resp.OverallSummary = result.OverallSummary
}
// enrichWithTranslation overlays title/description from MinIO when the pipeline has finished
// for the requested language. Does not call the AI translate API on GET.
func (s *tenderService) enrichWithTranslation(ctx context.Context, t *Tender, resp *TenderResponse, language string) {
func (s *tenderService) enrichWithTranslation(ctx context.Context, t *Tender, resp *TenderResponse, language string, cache *recommendationTranslationCache) {
s.enrichWithTranslationTimeout(ctx, t, resp, language, cache, aiTranslationEnrichmentTimeout)
}
func (s *tenderService) enrichWithTranslationTimeout(
ctx context.Context,
t *Tender,
resp *TenderResponse,
language string,
cache *recommendationTranslationCache,
enrichmentTimeout time.Duration,
) {
if t == nil || resp == nil {
return
}
@@ -219,7 +275,18 @@ func (s *tenderService) enrichWithTranslation(ctx context.Context, t *Tender, re
}
targetLanguage := s.resolveDetailLanguage(language)
enrichCtx, cancel := context.WithTimeout(ctx, aiTranslationEnrichmentTimeout)
if cache != nil {
if stored, ok := cache.get(t.GetID(), targetLanguage); ok {
if strings.TrimSpace(stored.Title) != "" {
resp.Title = stored.Title
resp.Description = stored.Description
resp.Language = targetLanguage
}
return
}
}
enrichCtx, cancel := context.WithTimeout(ctx, enrichmentTimeout)
defer cancel()
stored, usedNotice, err := s.lookupTranslationForTender(enrichCtx, t, targetLanguage)
@@ -237,6 +304,10 @@ func (s *tenderService) enrichWithTranslation(ctx context.Context, t *Tender, re
return
}
if cache != nil {
cache.set(t.GetID(), targetLanguage, stored)
}
resp.Title = stored.Title
resp.Description = stored.Description
resp.Language = targetLanguage
@@ -861,33 +932,13 @@ func (s *tenderService) Search(ctx context.Context, form *SearchForm, pagination
"to": form.DeadlineTo,
})
// If customer ID is provided, get customer keywords and add to search
if form.CustomerID != nil && *form.CustomerID != "" {
customerResp, err := s.customerService.GetByID(ctx, *form.CustomerID)
if err == nil && customerResp != nil {
// Only use keywords if status is ready
if customerResp.KeywordsStatus == "ready" && len(customerResp.Keywords) > 0 {
// Combine customer keywords with existing search query
keywordsQuery := strings.Join(customerResp.Keywords, " ")
if form.Search != nil && *form.Search != "" {
combinedSearch := *form.Search + " " + keywordsQuery
form.Search = &combinedSearch
} else {
form.Search = &keywordsQuery
}
s.logger.Info("Using customer keywords for search", map[string]interface{}{
"customer_id": *form.CustomerID,
"keywords": customerResp.Keywords,
})
} else {
s.logger.Info("Customer keywords not ready, skipping keyword-based search", map[string]interface{}{
"customer_id": *form.CustomerID,
"keywords_status": customerResp.KeywordsStatus,
})
}
}
}
language := s.pickResponseLanguage(form.Language)
return s.cachedSearchList(ctx, form, pagination, language, func(loadCtx context.Context) (*SearchResponse, error) {
return s.searchUncached(loadCtx, form, pagination, language)
})
}
func (s *tenderService) searchUncached(ctx context.Context, form *SearchForm, pagination *response.Pagination, language string) (*SearchResponse, error) {
if form.DocumentsScraped {
if err := s.enrichDocumentsScrapedFilter(ctx, form); err != nil {
return nil, err
@@ -909,7 +960,7 @@ func (s *tenderService) Search(ctx context.Context, form *SearchForm, pagination
if form.CompanyID == nil || *form.CompanyID == "" {
tenderResponses := make([]TenderResponse, 0, len(page.Items))
for _, tender := range page.Items {
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(s.pickResponseLanguage(form.Language)))
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(language))
}
return &SearchResponse{
@@ -928,7 +979,7 @@ func (s *tenderService) Search(ctx context.Context, form *SearchForm, pagination
// Continue without match percentage if company not found
tenderResponses := make([]TenderResponse, 0, len(page.Items))
for _, tender := range page.Items {
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(s.pickResponseLanguage(form.Language)))
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(language))
}
return &SearchResponse{
@@ -951,7 +1002,7 @@ func (s *tenderService) Search(ctx context.Context, form *SearchForm, pagination
tenderResponses := make([]TenderResponse, 0, len(page.Items))
for _, tender := range page.Items {
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(s.pickResponseLanguage(form.Language)))
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(language))
}
return &SearchResponse{
@@ -961,81 +1012,106 @@ func (s *tenderService) Search(ctx context.Context, form *SearchForm, pagination
nil
}
// Recommend retrieves tenders with pagination and filtering
// Recommend retrieves AI-ranked tenders for the authenticated customer's company.
func (s *tenderService) Recommend(ctx context.Context, form *SearchForm, pagination *response.Pagination) (*SearchResponse, error) {
s.logger.Info("Recommend tenders", map[string]interface{}{
"company_id": form.CompanyID,
"customer_id": form.CustomerID,
companyIDs := normalizeRecommendationCompanyScope(form.CompanyIDs)
companyID := ""
if form.CompanyID != nil {
companyID = strings.TrimSpace(*form.CompanyID)
}
if companyID == "" && len(companyIDs) == 0 {
return nil, errors.New("company ID is required")
}
if pagination.Cursor != "" {
return nil, fmt.Errorf("%w: cursor pagination is not supported for recommendations", response.ErrInvalidPagination)
}
s.logger.Debug("Recommend tenders request", map[string]interface{}{
"company_id": companyID,
"company_ids": companyIDs,
})
form.Status = []string{string(TenderStatusActive)}
// Priority: Use customer keywords if available, otherwise use company CPV codes
if form.CustomerID != nil && *form.CustomerID != "" {
customerResp, err := s.customerService.GetByID(ctx, *form.CustomerID)
if err == nil && customerResp != nil {
// Only use keywords if status is ready
if customerResp.KeywordsStatus == "ready" && len(customerResp.Keywords) > 0 {
// Use customer keywords for recommendation
keywordsQuery := strings.Join(customerResp.Keywords, " ")
form.Search = &keywordsQuery
s.logger.Info("Using customer keywords for recommendation", map[string]interface{}{
"customer_id": *form.CustomerID,
"keywords": customerResp.Keywords,
})
} else {
s.logger.Info("Customer keywords not ready, falling back to company CPV codes", map[string]interface{}{
"customer_id": *form.CustomerID,
"keywords_status": customerResp.KeywordsStatus,
})
}
}
lang := s.pickResponseLanguage(form.Language)
if result, ok := s.recommendFromPageCache(ctx, companyID, companyIDs, form, pagination, lang); ok {
return result, nil
}
// Fallback to company CPV codes if no customer keywords
if form.Search == nil || *form.Search == "" {
if form.CompanyID != nil && *form.CompanyID != "" {
company, err := s.companyService.GetByID(ctx, *form.CompanyID)
if err != nil {
s.logger.Error("Failed to get company for CPV codes, skipping CPV-based filtering", map[string]interface{}{
"company_id": form.CompanyID,
"error": err.Error(),
})
// Continue without CPV-based filtering instead of failing
} else if company.Tags != nil && len(company.Tags.CPVCodes) > 0 {
form.Classifications = company.Tags.CPVCodes
}
}
var (
recommendations []company.RecommendedTenderResponse
err error
)
if len(companyIDs) > 0 {
recommendations, err = s.companyService.GetMergedAIRecommendations(ctx, companyIDs)
} else {
recommendations, err = s.companyService.GetAIRecommendations(ctx, companyID)
}
if form.DocumentsScraped {
if err := s.enrichDocumentsScrapedFilter(ctx, form); err != nil {
return nil, err
}
if len(form.ContractFolderIDsWithDocuments) == 0 {
return s.emptySearchResponse(pagination), nil
}
}
page, err := s.repository.Search(ctx, form, pagination)
if err != nil {
s.logger.Error("Failed to list tenders", map[string]interface{}{
"error": err.Error(),
})
return nil, fmt.Errorf("failed to list tenders: %w", err)
return nil, err
}
if len(recommendations) == 0 {
return s.emptySearchResponse(pagination), nil
}
tenderResponses := make([]TenderResponse, 0, len(page.Items))
for _, tender := range page.Items {
tenderResponses = append(tenderResponses, *tender.ToResponseWithLanguage(s.pickResponseLanguage(form.Language)))
if len(companyIDs) == 0 && companyID != "" {
s.schedulePageCacheRefreshForLanguage(companyID, lang)
}
excluded, excludedByCompany, err := s.loadRecommendationExclusions(ctx, form, companyID, companyIDs)
if err != nil {
if form.ExcludeRejectedTenders {
s.logger.Error("Failed to build recommendation exclusions", map[string]interface{}{
"company_id": companyID,
"company_ids": companyIDs,
"error": err.Error(),
})
}
return nil, err
}
now := time.Now().Unix()
pageResult, err := s.buildRecommendationPage(
ctx,
recommendations,
form,
companyIDs,
excluded,
excludedByCompany,
now,
pagination.Offset,
pagination.Limit,
)
if err != nil {
return nil, err
}
paged := s.buildRecommendedTenderListResponses(pageResult.items, lang)
paged = s.enrichRecommendedTenderResponsesParallel(ctx, paged, lang)
return &SearchResponse{
Tenders: tenderResponses,
Metadata: pagination.ListMeta(page.TotalCount, page.NextCursor, page.HasMore, page.PageOffset),
Tenders: paged,
Metadata: pagination.ListMeta(pageResult.total, "", int(pagination.Offset)+len(paged) < int(pageResult.total), pagination.Offset),
}, nil
}
func (s *tenderService) buildRecommendedTenderResponse(ctx context.Context, tender Tender, rank int, analysis, tenderRef, language string, cache *recommendationTranslationCache) TenderResponse {
resp := tender.ToResponseWithLanguage(language)
s.enrichWithTranslation(ctx, &tender, resp, language, cache)
resp.Rank = rank
resp.Analysis = analysis
resp.ProcedureRef = tenderRef
return *resp
}
func statusAllowed(status TenderStatus, allowed []string) bool {
for _, s := range allowed {
if string(status) == strings.TrimSpace(s) {
return true
}
}
return false
}
// Delete deletes a tender
// GetTenderStatistics retrieves tender statistics
@@ -1131,8 +1207,8 @@ func (s *tenderService) CleanupOldData(ctx context.Context, olderThan time.Durat
}
// GetAISummary retrieves the AI-generated summary for a tender.
// It loads from MinIO storage (async pipeline result). When no summary exists yet,
// the pipeline is still running, or storage is unavailable, it returns 200-compatible
// It reads from MinIO when available; for scraped tenders without a summary it triggers
// on-demand summarization. When no summary can be produced it returns 200-compatible
// data with an empty overall_summary and a Source hint instead of an error.
func (s *tenderService) GetAISummary(ctx context.Context, id string) (*AISummaryResponse, error) {
if id == "" {
@@ -1151,6 +1227,12 @@ func (s *tenderService) GetAISummary(ctx context.Context, id string) (*AISummary
return nil, fmt.Errorf("tender not found")
}
return s.resolveAISummary(ctx, t)
}
// resolveAISummary returns the overall AI summary for a tender, triggering on-demand
// summarization when documents are scraped but no summary exists in storage yet.
func (s *tenderService) resolveAISummary(ctx context.Context, t *Tender) (*AISummaryResponse, error) {
noticeID := strings.TrimSpace(t.NoticePublicationID)
emptyResp := func(source string) *AISummaryResponse {
return &AISummaryResponse{
@@ -1164,45 +1246,146 @@ func (s *tenderService) GetAISummary(ctx context.Context, id string) (*AISummary
return emptyResp("unavailable"), nil
}
if s.aiSummarizerStorage == nil {
s.logger.Debug("AI summarizer storage not configured; returning empty AI summary", map[string]interface{}{
"tender_id": id,
if s.aiSummarizerStorage == nil && s.aiSummarizerClient == nil {
s.logger.Debug("AI summarizer not configured; returning empty AI summary", map[string]interface{}{
"tender_id": t.ID.Hex(),
})
return emptyResp("unavailable"), nil
}
summary, usedNotice, err := s.lookupSummaryForTender(ctx, t)
if err == nil {
return &AISummaryResponse{
NoticeID: usedNotice,
OverallSummary: summary,
Source: "storage",
}, nil
lookupCtx := ctx
if deadline, ok := ctx.Deadline(); !ok || time.Until(deadline) > aiSummaryEnrichmentTimeout {
var cancel context.CancelFunc
lookupCtx, cancel = context.WithTimeout(ctx, aiSummaryEnrichmentTimeout)
defer cancel()
}
s.logger.Info("AI summary not available from storage", map[string]interface{}{
"tender_id": id,
"contract_folder_id": t.ContractFolderID,
"notice_id": noticeID,
"error": err.Error(),
})
if s.aiSummarizerStorage != nil {
summary, usedNotice, err := s.lookupSummaryForTender(lookupCtx, t)
if err == nil {
s.persistAIOverallSummary(ctx, t, summary)
return &AISummaryResponse{
NoticeID: usedNotice,
OverallSummary: summary,
Source: "storage",
}, nil
}
if errors.Is(err, ai_summarizer.ErrSummaryNotReady) {
return emptyResp("pending"), nil
if !errors.Is(err, ai_summarizer.ErrSummaryNotReady) &&
!errors.Is(err, ai_summarizer.ErrObjectNotFound) &&
!errors.Is(err, ai_summarizer.ErrNoOverallSummary) {
if errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
s.logger.Error("Cannot retrieve AI summary: MinIO connection unavailable", map[string]interface{}{
"tender_id": t.ID.Hex(),
"contract_folder_id": t.ContractFolderID,
"notice_id": noticeID,
"error": err.Error(),
})
return nil, fmt.Errorf("failed to get AI summary: MinIO connection unavailable: %w", err)
}
return nil, fmt.Errorf("failed to get AI summary: %w", err)
}
}
if errors.Is(err, ai_summarizer.ErrObjectNotFound) || errors.Is(err, ai_summarizer.ErrNoOverallSummary) {
if !s.tenderHasScrapedDocuments(t) {
return emptyResp("unavailable"), nil
}
if errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
s.logger.Error("Cannot retrieve AI summary: MinIO connection unavailable", map[string]interface{}{
"tender_id": id,
if s.aiSummarizerClient == nil {
return emptyResp("unavailable"), nil
}
resp, err := s.fetchAISummaryOnDemand(ctx, t)
if err != nil {
s.logger.Error("On-demand AI summarization on tender read failed", map[string]interface{}{
"tender_id": t.ID.Hex(),
"contract_folder_id": t.ContractFolderID,
"notice_id": noticeID,
"error": err.Error(),
})
return nil, fmt.Errorf("failed to get AI summary: MinIO connection unavailable: %w", err)
return emptyResp("pending"), nil
}
return nil, fmt.Errorf("failed to get AI summary: %w", err)
overall := derefString(resp.OverallSummary)
usedNotice := resp.ResolvedNoticePublicationID()
if usedNotice == "" {
usedNotice = noticeID
}
if overall == "" && s.aiSummarizerStorage != nil {
if summary, notice, lookupErr := s.lookupSummaryForTender(ctx, t); lookupErr == nil {
overall = summary
usedNotice = notice
}
}
if overall != "" {
s.persistAIOverallSummary(ctx, t, overall)
return &AISummaryResponse{
NoticeID: usedNotice,
OverallSummary: overall,
Source: "generated",
}, nil
}
return emptyResp("pending"), nil
}
func (s *tenderService) tenderHasScrapedDocuments(t *Tender) bool {
if t == nil {
return false
}
if t.ProcessingMetadata.DocumentsScraped {
return true
}
return len(t.ScrapedDocuments) > 0
}
func (s *tenderService) fetchAISummaryOnDemand(ctx context.Context, t *Tender) (*ai_summarizer.SummarizeResponse, error) {
if s.aiSummarizerClient == nil {
return nil, ErrAINotConfigured
}
if strings.TrimSpace(t.NoticePublicationID) == "" {
return nil, ErrTenderMissingNoticeID
}
if strings.TrimSpace(t.ContractFolderID) == "" {
return nil, ErrTenderMissingContractFolderID
}
reqBody := ai_summarizer.NewSummarizeRequest(t.ContractFolderID, t.NoticePublicationID)
s.logger.Info("Triggering on-demand AI summarization", map[string]interface{}{
"tender_id": t.ID.Hex(),
"contract_folder_id": t.ContractFolderID,
"notice_publication_id": t.NoticePublicationID,
})
resp, err := s.aiSummarizerClient.FetchSummaryOnDemand(ctx, reqBody)
if err != nil {
return nil, fmt.Errorf("AI summarization request failed: %w", err)
}
s.markTenderSummarized(ctx, t)
return resp, nil
}
func (s *tenderService) markTenderSummarized(ctx context.Context, t *Tender) {
now := time.Now().Unix()
t.ProcessingMetadata.DocumentsSummarized = true
t.ProcessingMetadata.DocumentsSummarizedAt = now
t.UpdatedAt = now
if err := s.repository.Update(ctx, t); err != nil {
s.logger.Warn("Failed to update tender summarization status", map[string]interface{}{
"tender_id": t.ID.Hex(),
"error": err.Error(),
})
}
}
func derefString(value *string) string {
if value == nil {
return ""
}
return strings.TrimSpace(*value)
}
// TriggerAISummarize triggers on-demand AI summarization for a tender.
@@ -1216,7 +1399,6 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
return nil, ErrAINotConfigured
}
// Get tender to build the request
t, err := s.repository.GetByID(ctx, id)
if err != nil {
s.logger.Error("Failed to get tender for on-demand summarization", map[string]interface{}{
@@ -1236,23 +1418,11 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
return nil, ErrTenderNotFound
}
if t.NoticePublicationID == "" {
return nil, ErrTenderMissingNoticeID
}
if strings.TrimSpace(t.ContractFolderID) == "" {
return nil, ErrTenderMissingContractFolderID
if !s.tenderHasScrapedDocuments(t) {
return nil, ErrTenderDocumentsNotScraped
}
reqBody := ai_summarizer.NewSummarizeRequest(t.ContractFolderID, t.NoticePublicationID)
s.logger.Info("Triggering on-demand AI summarization", map[string]interface{}{
"tender_id": id,
"contract_folder_id": t.ContractFolderID,
"notice_publication_id": t.NoticePublicationID,
})
// Call the AI service
resp, err := s.aiSummarizerClient.FetchSummaryOnDemand(ctx, reqBody)
resp, err := s.fetchAISummaryOnDemand(ctx, t)
if err != nil {
s.logger.Error("On-demand AI summarization failed", map[string]interface{}{
"tender_id": id,
@@ -1260,18 +1430,17 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
"notice_publication_id": t.NoticePublicationID,
"error": err.Error(),
})
return nil, fmt.Errorf("AI summarization request failed: %w", err)
return nil, err
}
// Map the response to our response type
summaries := make([]AIDocumentSummary, 0, len(resp.Summaries))
for _, s := range resp.Summaries {
for _, item := range resp.Summaries {
summaries = append(summaries, AIDocumentSummary{
DocumentName: s.DocumentName,
DocumentType: s.DocumentType,
Summary: s.Summary,
SummaryModel: s.SummaryModel,
Error: s.Error,
DocumentName: item.DocumentName,
DocumentType: item.DocumentType,
Summary: item.Summary,
SummaryModel: item.SummaryModel,
Error: item.Error,
})
}
@@ -1291,6 +1460,10 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
contractFolderID = t.ContractFolderID
}
if overall := derefString(resp.OverallSummary); overall != "" {
s.persistAIOverallSummary(ctx, t, overall)
}
return &AISummarizeResponse{
ContractFolderID: contractFolderID,
NoticePublicationID: noticePublicationID,
@@ -1599,29 +1772,83 @@ func (s *tenderService) listProceduresWithDocuments(ctx context.Context) ([]ai_s
return procedures, nil
}
func (s *tenderService) enrichDocumentsScrapedFilter(ctx context.Context, form *SearchForm) error {
procedures, err := s.listProceduresWithDocuments(ctx)
if err != nil {
s.logger.Error("Failed to resolve tenders with scraped documents from MinIO", map[string]interface{}{
"error": err.Error(),
})
if errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
return fmt.Errorf("failed to resolve tenders with scraped documents: MinIO connection unavailable: %w", err)
}
return fmt.Errorf("failed to resolve tenders with scraped documents: %w", err)
}
func (s *tenderService) resolveRecommendedTenders(ctx context.Context, recommendations []company.RecommendedTenderResponse) (map[string]Tender, error) {
out := make(map[string]Tender)
procRefs := make([]ProcedureReference, 0, len(recommendations))
mongoIDs := make([]string, 0)
folderIDs := make([]string, 0, len(procedures))
for _, proc := range procedures {
if proc.DocumentCount <= 0 {
for _, rec := range recommendations {
ref := strings.TrimSpace(rec.TenderID)
if ref == "" {
continue
}
if id := strings.TrimSpace(proc.ContractFolderID); id != "" {
folderIDs = append(folderIDs, id)
if folder, notice, ok := ParseAIProcedureRef(ref); ok {
procRefs = append(procRefs, ProcedureReference{
Ref: ref,
ContractFolderID: folder,
NoticePublicationID: notice,
})
continue
}
if IsMongoObjectIDRef(ref) {
mongoIDs = append(mongoIDs, ref)
continue
}
s.logger.Warn("Skipping unrecognized recommended tender reference", map[string]interface{}{
"tender_ref": ref,
})
}
form.ContractFolderIDsWithDocuments = folderIDs
return nil
var (
byProcedure = make(map[string]Tender)
byMongoID = make(map[string]Tender)
)
group, groupCtx := errgroup.WithContext(ctx)
if len(procRefs) > 0 {
group.Go(func() error {
byProcedureResult, err := s.repository.GetByProcedureReferences(groupCtx, procRefs)
if err != nil {
return fmt.Errorf("failed to load recommended tenders by procedure reference: %w", err)
}
for ref, tender := range byProcedureResult {
byProcedure[ref] = tender
}
return nil
})
}
if len(mongoIDs) > 0 {
group.Go(func() error {
tenders, err := s.repository.GetByIDs(groupCtx, mongoIDs)
if err != nil {
s.logger.Error("Failed to load recommended tenders by Mongo ID", map[string]interface{}{
"error": err.Error(),
})
return fmt.Errorf("failed to load recommended tenders: %w", err)
}
for i := range tenders {
byMongoID[tenders[i].GetID()] = tenders[i]
}
return nil
})
}
if err := group.Wait(); err != nil {
return nil, err
}
for ref, tender := range byProcedure {
out[ref] = tender
}
for ref, tender := range byMongoID {
out[ref] = tender
}
return out, nil
}
func (s *tenderService) emptySearchResponse(pagination *response.Pagination) *SearchResponse {
-1
View File
@@ -148,7 +148,6 @@ func (ta *TenderApproval) Submit() {
// Reject rejects the tender with a reason
func (ta *TenderApproval) Reject() {
ta.Status = ApprovalStatusRejected
ta.SubmissionMode = ""
ta.CreatedAt = time.Now().Unix()
}
+1
View File
@@ -14,4 +14,5 @@ var (
ErrInvalidCompanyID = errors.New("invalid company ID")
ErrUnauthorized = errors.New("unauthorized to perform this action")
ErrInvalidData = errors.New("invalid tender approval data")
ErrSubmissionSyncFailed = errors.New("failed to sync tender submission workflow")
)
+1
View File
@@ -81,6 +81,7 @@ type CompanyTenderApprovalStatsResponse struct {
// ToggleTenderApprovalForm represents the form for toggling a tender approval
type ToggleTenderApprovalForm struct {
TenderID string `json:"tender_id" valid:"required"`
CompanyID *string `json:"company_id,omitempty" valid:"optional"`
Status ApprovalStatus `json:"status" valid:"required,in(submitted|rejected)"`
SubmissionMode SubmissionMode `json:"submission_mode" valid:"required,in(self-apply|partnership)"`
}
+25 -6
View File
@@ -1,7 +1,10 @@
package tender_approval
import (
"errors"
"strconv"
"strings"
"tm/internal/customer"
"tm/pkg/logger"
"tm/pkg/response"
@@ -42,14 +45,30 @@ func (h *Handler) ToggleTenderApproval(c echo.Context) error {
return response.ValidationError(c, "Invalid request data", err.Error())
}
// Get company ID from context
companyID := c.Get("company_id").(string)
if companyID == "" {
return response.Unauthorized(c, "Unauthorized")
requestedCompanyID := strings.TrimSpace(c.QueryParam("company_id"))
if form.CompanyID != nil && strings.TrimSpace(*form.CompanyID) != "" {
requestedCompanyID = strings.TrimSpace(*form.CompanyID)
}
tenderApproval, err := h.service.ToggleTenderApproval(c.Request().Context(), form, companyID)
activeCompanyID, _ := c.Get("company_id").(string)
assignedCompanyIDs, _ := c.Get("company_ids").([]string)
companyID, err := customer.PickAssignedCompanyID(activeCompanyID, assignedCompanyIDs, requestedCompanyID)
if err != nil {
if errors.Is(err, customer.ErrCompanyNotAssigned) {
return response.Forbidden(c, "Company is not assigned to customer")
}
return response.BadRequest(c, "Company ID is required", "")
}
customerID, _ := customer.GetCustomerIDFromContext(c)
tenderApproval, err := h.service.ToggleTenderApproval(c.Request().Context(), form, companyID, customerID)
if err != nil {
if errors.Is(err, ErrTenderNotFound) {
return response.NotFound(c, "Tender not found")
}
if errors.Is(err, ErrSubmissionSyncFailed) {
return response.InternalServerError(c, "Failed to sync tender submission workflow")
}
if err.Error() == "tender approval already exists for this tender and company" {
return response.Conflict(c, err.Error())
}
@@ -132,7 +151,7 @@ func (h *Handler) GetTenderApprovalByTenderAndCompany(c echo.Context) error {
// @Param limit query integer false "Number of approvals per page (1-100)" minimum(1) maximum(100) default(20)
// @Param offset query integer false "Number of approvals to skip for pagination" minimum(0) default(0)
// @Param submission_mode query string false "Filter by submission mode Enums(self-apply, partnership)"
// @Param status query string false "Filter by status Enums(submitted, rejected)"
// @Param status query string false "Filter by status Enums(submitted, rejected). Defaults to submitted when omitted."
// @Param created_from query integer false "Filter by created date from (Unix timestamp)"
// @Param created_to query integer false "Filter by created date to (Unix timestamp)"
// @Success 200 {object} response.APIResponse{data=TenderApprovalWithTenderListResponse} "Tender approvals retrieved successfully"
+43 -5
View File
@@ -3,11 +3,13 @@ package tender_approval
import (
"context"
"errors"
"strings"
"time"
"tm/pkg/logger"
"tm/pkg/mongo"
"go.mongodb.org/mongo-driver/v2/bson"
mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
)
// Repository defines the interface for tender approval data operations
@@ -30,6 +32,7 @@ type Repository interface {
GetTenderApprovalStats(ctx context.Context) (*TenderApprovalStatsResponse, error)
GetCompanyTenderApprovalStats(ctx context.Context, companyID string) (*CompanyTenderApprovalStatsResponse, error)
SearchByCriteria(ctx context.Context, criteria *TenderApprovalSearchCriteria) ([]*TenderApproval, error)
ListRejectedTenderIDsByCompanyID(ctx context.Context, companyID string) ([]string, error)
}
// tenderApprovalRepository implements the Repository interface using the MongoDB ORM
@@ -101,7 +104,7 @@ func (r *tenderApprovalRepository) GetByID(ctx context.Context, id string) (*Ten
tenderApproval, err := r.ormRepo.FindByID(ctx, id)
if err != nil {
if errors.Is(err, mongo.ErrDocumentNotFound) {
return nil, errors.New("tender approval not found")
return nil, ErrTenderApprovalNotFound
}
r.logger.Error("Failed to get tender approval by ID", map[string]interface{}{
"error": err.Error(),
@@ -123,7 +126,7 @@ func (r *tenderApprovalRepository) GetByTenderAndCompany(ctx context.Context, te
tenderApproval, err := r.ormRepo.FindOne(ctx, filter)
if err != nil {
if errors.Is(err, mongo.ErrDocumentNotFound) {
return nil, errors.New("tender approval not found")
return nil, ErrTenderApprovalNotFound
}
r.logger.Error("Failed to get tender approval by tender and company", map[string]interface{}{
"error": err.Error(),
@@ -513,11 +516,11 @@ func (r *tenderApprovalRepository) GetCompanyTenderApprovalStats(ctx context.Con
rejectedFilter := bson.M{"company_id": companyID, "status": string(ApprovalStatusRejected)}
rejectedTenders, _ := r.ormRepo.Count(ctx, rejectedFilter)
// Count by submission mode for this company
selfApplyFilter := bson.M{"company_id": companyID, "status": string(ApprovalStatusSubmitted), "submission_mode": string(SubmissionModeSelfApply)}
// Count by submission mode for this company (includes submitted and rejected decisions).
selfApplyFilter := bson.M{"company_id": companyID, "submission_mode": string(SubmissionModeSelfApply)}
selfApplyCount, _ := r.ormRepo.Count(ctx, selfApplyFilter)
partnershipFilter := bson.M{"company_id": companyID, "status": string(ApprovalStatusSubmitted), "submission_mode": string(SubmissionModePartnership)}
partnershipFilter := bson.M{"company_id": companyID, "submission_mode": string(SubmissionModePartnership)}
partnershipCount, _ := r.ormRepo.Count(ctx, partnershipFilter)
stats := &CompanyTenderApprovalStatsResponse{
@@ -533,6 +536,28 @@ func (r *tenderApprovalRepository) GetCompanyTenderApprovalStats(ctx context.Con
return stats, nil
}
// ListRejectedTenderIDsByCompanyID returns distinct tender IDs rejected by a company.
func (r *tenderApprovalRepository) ListRejectedTenderIDsByCompanyID(ctx context.Context, companyID string) ([]string, error) {
pipeline := mongodriver.Pipeline{
{{Key: "$match", Value: bson.M{
"company_id": companyID,
"status": string(ApprovalStatusRejected),
}}},
{{Key: "$group", Value: bson.M{"_id": "$tender_id"}}},
}
results, err := r.ormRepo.Aggregate(ctx, pipeline)
if err != nil {
r.logger.Error("Failed to list rejected tender IDs by company", map[string]interface{}{
"company_id": companyID,
"error": err.Error(),
})
return nil, err
}
return aggregateStringIDs(results), nil
}
// SearchByCriteria searches tender approvals using search criteria
func (r *tenderApprovalRepository) SearchByCriteria(ctx context.Context, criteria *TenderApprovalSearchCriteria) ([]*TenderApproval, error) {
// Convert criteria to search parameters
@@ -549,3 +574,16 @@ func (r *tenderApprovalRepository) SearchByCriteria(ctx context.Context, criteri
// Use the search method
return r.Search(ctx, criteria.TenderID, criteria.CompanyID, status, submissionMode, criteria.CreatedAtFrom, criteria.CreatedAtTo, criteria.Limit, criteria.Offset, "created_at", "desc")
}
func aggregateStringIDs(results []bson.M) []string {
ids := make([]string, 0, len(results))
for _, result := range results {
switch id := result["_id"].(type) {
case string:
if strings.TrimSpace(id) != "" {
ids = append(ids, id)
}
}
}
return ids
}

Some files were not shown because too many files have changed in this diff Show More