- Removed the DocumentSummarizationWorker and its related scheduling logic from the worker bootstrap.
- Updated the AI summarizer client initialization comment for clarity.
- Added a new error type for cases when tender documents have not been scraped yet, enhancing error handling in the tender service.
- Modified API documentation to reflect changes in AI summary retrieval logic, ensuring accurate descriptions of on-demand summarization behavior.
This update streamlines the AI summarization process by eliminating the document summarization worker, improving overall system efficiency and clarity in error handling.
- Updated the document scraper service to include a new ScrapePortalsProvider interface, allowing for dynamic retrieval of supported scraping portals.
- Modified the ListPendingTenders and GetTenderByNoticeID methods to filter tenders based on document URLs that match the configured portals.
- Introduced new error handling for cases when the scrape portals provider is not configured, returning appropriate service unavailable responses.
- Enhanced API documentation to reflect changes in tender retrieval logic and added error response details for unsupported portal scenarios.
This update improves the document scraping functionality by integrating AI portal support, enhancing the overall reliability and flexibility of the tender management system.
- Updated the `AggregateProcurementLotEstimatedValue` function to reject mixed currencies and allow empty lot currencies, ensuring accurate aggregation of estimated values.
- Introduced new unit tests in `entity_test.go` to validate the behavior of the aggregation function under various scenarios, including mixed currencies and empty lot currencies.
- Refactored budget calculation in `budget_test.go` to utilize the new aggregation logic, improving consistency in budget retrieval.
This update improves the handling of estimated values in procurement lots, enhancing the reliability of the tender management system.
- Introduced caching mechanisms for summary and statistics in the dashboard service, improving performance by reducing redundant data retrieval.
- Refactored the Summary method to utilize MongoDB aggregation for more efficient data processing and retrieval.
- Added synchronization features using singleflight to prevent duplicate processing of requests for cached data.
- Updated the repository to include a cachedScrapedTendersScope method, enhancing the efficiency of scraped document statistics retrieval.
This update significantly optimizes the dashboard's performance and data handling capabilities, ensuring faster response times and reduced load on the database.
- Updated the `ResolvedEstimatedValueAndCurrency` method to aggregate procurement lot values when the tender-level estimated value is not set, improving accuracy in value retrieval.
- Introduced the `AggregateProcurementLotEstimatedValue` function to sum estimated values from procurement lots and return the first found currency.
- Modified the `ToResponseWithLanguage` method to utilize the new estimated value resolution logic.
- Added unit tests for the new functionality, ensuring correct behavior for various scenarios in the `entity_test.go` and `budget_test.go` files.
This update improves the handling of estimated values in tenders, enhancing the overall reliability of the tender management system.
- Changed the ScrapePortalsResponse type to return a slice of strings representing portal identifiers instead of a structured object.
- Updated the Swagger documentation for the GetScrapePortals endpoint to reflect the new response format, ensuring clarity in API usage.
This update simplifies the response structure for the scraping portals, enhancing the API's usability and consistency.
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.
This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
- Introduced a new `scrapedTendersScope` type to encapsulate the MongoDB filter for tenders with scraped documents, improving clarity and maintainability.
- Updated the `Statistics` method to utilize the new scope resolution, allowing for more accurate data retrieval based on the presence of scraped documents.
- Implemented multiple tests for the `resolveScrapedTendersScope` method, ensuring correct behavior for various scenarios, including empty and fallback cases.
This update enhances the dashboard's ability to manage scraped document statistics, improving overall data accuracy and system performance.
- Introduced the ProcedureDocumentsLister interface to list contract folders with scraped documents, enhancing the accuracy of document-scrape statistics.
- Updated the dashboard repository to accept ProcedureDocumentsLister as a dependency, allowing for improved data retrieval.
- Implemented tests for the new functionality, ensuring proper handling of scraped document folder IDs and error propagation.
This update enhances the dashboard's capability to manage and report on scraped documents, improving overall system efficiency and data integrity.
- Updated the `Statistics` method to utilize `created_at` instead of `processing_metadata.scraped_at` for fetching daily counts, ensuring accurate historical data representation.
- Removed redundant conditions in the `scrapedDocumentsPerDay` method, streamlining the query logic for better performance and clarity.
- Added a new index on `source` and `created_at` to optimize database queries related to scraped documents.
This update enhances the accuracy of data retrieval in the dashboard statistics, improving the overall efficiency of the tender management system.
- Added `NoticePublicationID` field to the `SearchForm` for filtering tenders by TED notice publication ID.
- Updated Swagger documentation in the handler methods to include the new `notice_publication_id` parameter for relevant endpoints.
- Modified the repository's search filter to incorporate the new `NoticePublicationID` field, allowing for more precise search queries.
This update improves the search capabilities within the tender management system, enhancing user experience and data retrieval accuracy.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Updated the AI pipeline service to include a new `ScrapedDocumentMetadataSyncer` interface for persisting scraped document metadata onto tender records.
- Modified the `NewService` function to accept the new metadata syncer dependency.
- Implemented synchronization of scraped document metadata in the `ScrapeDocuments` and `Run` methods.
- Enhanced the tender service to enrich search filters based on scraped documents and added a new method for syncing scraped documents from storage.
- Updated the `SearchForm` to include `ContractFolderIDsWithDocuments` for better handling of scraped documents in queries.
This update improves the integration of scraped document handling within the AI pipeline, enhancing data consistency and operational efficiency in the tender management system.
- Added `TranslationEnabled` and `TranslationInterval` fields to the worker configuration to manage automatic translation scheduling.
- Updated the worker initialization to log when the translation worker is disabled.
- Improved error handling in the AI summarizer client by introducing `APIStatusError` for better context on API failures, replacing direct error messages with structured error responses.
This update enhances the configurability of the worker and improves error reporting for AI service interactions, contributing to better maintainability and user experience.
- Added new routes and handlers for AI pipeline operations, including scraping documents, batch summarization, translation, and syncing with the Opplens AI service.
- Introduced request forms for handling tender references and batch operations.
- Enhanced the AI service with methods for triggering batch operations and managing pipeline runs.
- Updated Swagger documentation to reflect the new AI pipeline endpoints and their functionalities.
This update integrates comprehensive AI pipeline capabilities into the tender management system, improving operational efficiency and user experience.
- Revised the Gap Analysis Report to provide detailed counts of implemented, partial, and not implemented controls, enhancing clarity on compliance status.
- Updated the ISO27001 Roadmap to include a status column for deliverables, improving tracking of progress towards certification.
- Adjusted the Statement of Applicability to reconcile summary counts with control rows and clarify the applicability of outsourced development.
These updates strengthen the documentation framework, ensuring it accurately reflects the current state of security controls and compliance efforts within the Tender Management System.
- Introduced a comprehensive suite of security documents to support ISO/IEC 27001 certification, including the ISMS Foundation, Risk Assessment Matrix, Gap Analysis Report, Statement of Applicability, and ISO27001 Roadmap.
- Updated the README to include links to the new security documentation, enhancing the project's compliance framework and providing clear guidance on security policies and procedures.
This addition strengthens the overall security posture of the Tender Management System and aligns with industry standards for information security management.
- Added `search` and `q` query parameters to the `SearchForm` for improved notification searching capabilities.
- Implemented `ResolvedSearch` method to prioritize search term resolution from the new parameters.
- Updated `GetByUserID` repository method to support searching notifications by title and message using regex.
- Enhanced logging in the `GetNotifications` service method to include search parameters.
- Updated Swagger documentation to reflect the new search parameters for the notification API.
This update improves the user experience by allowing more flexible and efficient searching of notifications.
- Added new AI onboarding and recommendation endpoints in the company handler for starting onboarding and retrieving ranked tender recommendations.
- Introduced `StartAIOnboarding` and `GetAIRecommendations` methods in the company service to handle AI interactions.
- Updated the company service constructor to include the AI recommendation client.
- Enhanced the AI summarizer client with methods for onboarding and fetching recommendations.
- Added response structures for onboarding and recommended tenders in the company form.
This update enhances the tender management system by integrating AI capabilities for onboarding and tender recommendations, improving user experience and operational efficiency.
- Updated `.drone.yml` to remove lingering legacy worker source files after git deletion, ensuring a clean build environment.
- Modified `Makefile` to include a new `build-worker` target that removes deprecated scraper and queue source files, streamlining the worker build process.
This change enhances the build process by eliminating outdated components and ensuring that the worker is built without legacy dependencies.
- Removed the QueueConfig structure and related queue management files as they are no longer in use.
- Updated the worker initialization to reflect the removal of queue-related configurations.
- Cleaned up the bootstrap process by eliminating deprecated logging related to the queue system.
This update streamlines the worker's configuration and prepares the codebase for future enhancements without the legacy queue management components.
Paginate un-summarized tenders and mark rows missing notice/folder IDs as handled so the worker cannot spin forever. Return 400/404 for validation and not-found cases on AI summarize/analyze triggers instead of leaking internal errors as 500.
- Added a new endpoint to trigger on-demand agentic analysis for tenders via the AI service.
- Introduced `TriggerAIAnalyze` method in the `TenderHandler` to handle requests and responses for AI analysis.
- Updated the `tenderService` to include `TriggerAIAnalyze` method, which validates input and interacts with the AI summarizer client.
- Enhanced the `AIAnalyzeResponse` and `AIAnalyzeDocument` structures to support the new analysis feature.
- Refactored the `DocumentSummarizationWorker` and `TranslationWorker` to remove deprecated MinIO dependencies, streamlining the AI service interactions.
This update improves the functionality of the tender management system by allowing users to trigger AI analysis on-demand, enhancing the overall user experience and system capabilities.
- Improved the `IsDuplicateKeyError` function to utilize structured error messages for better accuracy in identifying duplicate key errors.
- Introduced `duplicateKeyMessages` and `hasStructuredWriteException` functions to parse and extract relevant information from MongoDB error messages.
- Updated `DuplicateKeyMatchesField` to match duplicate key errors against specific fields by analyzing error messages rather than relying on substring matching.
This refactor enhances the clarity and reliability of duplicate key error handling, ensuring more precise identification of conflicts in MongoDB operations.
- Updated `InitAISummarizerClient` to accept `mongoManager` for tracking translation success.
- Introduced new `Statistics` endpoint in the dashboard to fetch scraping and translation statistics.
- Enhanced `TranslationWorker` to utilize the new success counter for tracking successful translations.
- Added necessary data structures and query forms for statistics reporting.
This refactor improves the tracking of AI translation success and provides new insights through the dashboard statistics.
Map E11000 from the per-collection phone index to friendly conflict
errors and skip redundant GetByPhone when the phone is unchanged.
Not blocking: IAT test DB may need a one-off dedupe for index build;
phone uniqueness is per collection (users vs customers), not global.
Added functionality to check for existing customers and users by phone number during registration and updates. Updated the handler, service, and repository layers to include phone number checks, ensuring unique phone numbers across customers and users.
Mazyar
mazyar