- Introduced the `scrapedDocumentsScanner` interface to facilitate scanning of scraped documents from MinIO, returning both procedure summaries and daily document counts.
- Updated the `ListProceduresWithDocuments` method to utilize the new scanning functionality, improving data retrieval efficiency.
- Enhanced the `scrapedDocumentsPerDay` method to filter daily counts based on a specified start date, ensuring accurate reporting of document statistics.
- Added unit tests for the new scanning logic and daily counts filtering, ensuring robust functionality and error handling.
This update enhances the dashboard's document management capabilities, providing better insights into scraped documents and their daily counts.
- Introduced a mutex to ensure only one TED scraper run executes at a time, preventing concurrent executions during startup and scheduled runs.
- Implemented a mechanism to check if today's TED scrape has already been completed during startup, logging appropriate messages for both completed and new runs.
- Added startup catch-up logic for tender translations and unprocessed notices, ensuring that any missed tasks are executed without blocking the application startup.
This update improves the reliability and efficiency of the TED scraper and worker processes, ensuring that all necessary tasks are completed after a server restart.
- Introduced retry logic for fetching AI recommendations after onboarding, enhancing reliability in recommendation retrieval.
- Updated logging levels for better observability, changing cache miss logs to Info level.
- Renamed methods for clarity, replacing `refreshAIRecommendationsCacheAsync` with `scheduleRecommendationRefreshAfterOnboarding` and `fetchAndCacheAIRecommendations` with `fetchAIRecommendations`.
- Implemented a mechanism to clear the cache if no recommendations are returned, improving cache management.
This update optimizes the AI recommendation process, ensuring more robust handling of recommendation fetching and caching during onboarding.
- Updated `AISummarizerConfig` to allow for a default `RecommendationCacheTTL` of 0, enabling persistent caching until company updates.
- Refactored `StartAIOnboarding` to include cache invalidation and asynchronous recommendation refresh, improving responsiveness during onboarding.
- Introduced `triggerAIOnboardingAsync` method for background processing of AI onboarding and cache refresh, enhancing user experience.
- Improved logging for AI onboarding and recommendation fetching processes, providing better observability and error tracking.
This update optimizes the AI recommendation caching mechanism and onboarding workflow, ensuring a smoother and more efficient experience for users.
- Introduced `refreshAIRecommendationsCacheAsync` method to refresh AI recommendations in the background, improving responsiveness by serving the previous cache until the refresh completes.
- Updated `StartAIOnboarding` to call the new asynchronous cache refresh method instead of invalidating the cache directly.
- Added logging for cache refresh operations, including success and error handling, to enhance observability.
This update enhances the AI recommendation caching mechanism, providing a smoother onboarding experience and reducing latency in recommendation retrieval.
- Updated the `companyService` to include Redis caching for AI recommendations, improving performance and reducing redundant AI calls.
- Introduced asynchronous AI onboarding triggered after company profile updates, enhancing user experience by offloading processing.
- Added configuration for recommendation cache TTL in the `AISummarizerConfig`, allowing for flexible cache management.
- Implemented methods for caching, retrieving, and invalidating AI recommendations in the `companyService`, ensuring efficient data handling.
This update enhances the company's AI recommendation capabilities, providing faster responses and a more efficient onboarding process.
- Added a new endpoint in the `tender` handler for retrieving AI-ranked tender recommendations for a company, improving the functionality of the admin panel.
- Updated the `SearchForm` to include a query parameter for `only_active_deadlines`, allowing for more flexible search options.
- Enhanced API documentation for the new endpoint to provide clear usage instructions and expected parameters.
This update improves the tender management system by providing administrators with better tools for accessing relevant tender information.
- Changed the response message in the `Send` method of the `NotificationHandler` from "Notification sent successfully" to "Notification created successfully" to better reflect the action performed.
This update improves the clarity of the notification response, ensuring users receive accurate feedback on their actions.
- Modified the `DocumentFileIDs` field in the `Company` struct to ensure it is always persisted in BSON format, enhancing data consistency.
- Implemented logic in the `Update` method of the `companyRepository` to clear document file IDs when the slice is empty, ensuring accurate database updates.
- Added logging for errors encountered while clearing document file IDs, improving error tracking and debugging capabilities.
This update enhances the management of document file IDs within the company domain, ensuring proper handling and persistence in the database.
- Modified the `EmployeeCount` field validation in `SearchForm` to use a range of 1 to 100000, ensuring more accurate input constraints.
- Updated `EmployeeCountMin` and `EmployeeCountMax` fields to reflect the same range validation, enhancing the robustness of search queries.
This update improves the validation logic for employee count fields in the company search functionality, providing better input handling and user experience.
- Updated the `UploadDocuments` method to sanitize document file IDs before saving, ensuring only valid references are stored.
- Introduced `DetachDocumentFileID` method in the `companyService` to remove file IDs from all companies referencing a deleted file, improving data integrity.
- Enhanced the `companyRepository` with a new method to handle the removal of document file IDs from the database.
- Updated the `filestore` handler to utilize the new detachment functionality when files are deleted, ensuring consistent state across domain entities.
This update improves the management of document file IDs within the company domain, enhancing data integrity and reference handling.
- Updated the `SearchForm` to include optional filters for `name`, `email`, `phone`, and `employee_count`, allowing for more granular search capabilities.
- Modified the `Search` method in the `companyRepository` to handle the new filters, improving the search logic with regex support for `name`, `email`, and `phone`.
- Updated API documentation in the `company` handler to reflect the new query parameters for enhanced clarity.
This update improves the search functionality within the company domain, providing users with more flexible and precise search options.
- Updated the `Search` method in the `companyRepository` to improve search capabilities by allowing regex-based filtering on `name`, `email`, and `phone` fields.
- Added logic to handle numeric search queries for `employee_count`, enhancing the search flexibility.
- Implemented input sanitization to trim whitespace from search queries, ensuring cleaner search inputs.
This update improves the search functionality within the company repository, providing more robust and flexible search options for users.
- Introduced `form_test.go` to validate the behavior of the `UpdateUserForm` struct's JSON unmarshalling, specifically for the `profile_image` field.
- Added tests to ensure omitted `profile_image` is not treated as an update, null values clear the image, and valid URLs are preserved.
- Enhanced the `UpdateUserForm` and `UpdateProfileForm` structs to include logic for distinguishing between omitted and explicitly null profile images during JSON unmarshalling.
This update improves the robustness of user profile updates by ensuring correct handling of profile image data in JSON requests, enhancing overall data integrity in the user management system.
- Introduced a new `NotificationWorker` to promote due scheduled notifications from pending to sent, improving notification management.
- Added `NotificationInterval` configuration to schedule the notification delivery worker, with a default value for flexibility.
- Implemented `MarkDueScheduledAsSent` method in the notification repository to update the status of notifications based on their delivery time.
- Updated the notification service to process due scheduled notifications during relevant operations, ensuring timely delivery.
This update enhances the notification system by automating the delivery of scheduled notifications, improving user engagement and operational efficiency.
- Introduced `ProcedureReference` struct to encapsulate AI procedure coordinates for better data management.
- Added `GetByProcedureReferences` method in the `TenderRepository` to retrieve multiple tenders based on AI procedure references in a single query.
- Updated the `resolveRecommendedTenders` method in the service layer to utilize the new repository method, improving efficiency in fetching recommended tenders.
- Enhanced error handling and logging for the new repository method to ensure robust operation.
This update improves the handling of AI procedure references, streamlining tender retrieval processes and enhancing overall system performance.
- Introduced `FormatAIProcedureRef` and `ParseAIProcedureRef` functions in the `tender` domain for handling AI service tender references.
- Added unit tests for these functions in `ai_reference_test.go` to ensure correct parsing and formatting behavior.
- Updated the `TenderResponse` struct to include a new `ProcedureRef` field for improved data representation.
- Enhanced the `GetByProcedureReference` method in the repository to retrieve tenders based on the new procedure reference format.
- Modified the `Recommend` method in the service layer to utilize the new procedure reference handling, improving the recommendation process.
This update enhances the handling of AI procedure references, ensuring better data integrity and usability in the tender management system.
- Changed the `Rank` field type from `string` to `int` in the `RecommendedTenderResponse` struct within the `company` domain for better data representation.
- Updated the `Rank` field type from `string` to `int` in the `TenderResponse` struct within the `tender` domain to ensure consistency in ranking data.
- Modified the `Rank` field type from `string` to `int` in the `RecommendedTender` struct within the AI summarizer package to align with the updated data structure.
This update enhances the data integrity and consistency across the tender management system by standardizing the rank representation as an integer.
- Added `DeliveryChannelInApp` to support in-app notifications.
- Introduced `EventTypeInApp` constant for identifying in-app notifications.
- Updated `AdminNotifications` and `CustomerNotifications` handlers to set the event type to `EventTypeInApp`.
- Modified the `GetByUserID` repository method to handle filtering for in-app notifications, allowing for more flexible retrieval options.
- Updated the `persistNotification` method to include in-app as a delivery method.
This update improves the notification system by enabling in-app notifications, enhancing user engagement and notification management capabilities.
- Added new fields `Rank` and `Analysis` to the `TenderResponse` struct for improved data representation.
- Updated the `RecommendTenders` handler to reflect changes in API documentation, including a new summary and description for AI-ranked tender recommendations.
- Improved error handling in the recommendation process, ensuring appropriate responses for missing company IDs and unavailable AI services.
This update enhances the tender response capabilities and improves the clarity of the recommendation API, aligning with the overall goal of providing better insights for users.
- Removed the DocumentSummarizationWorker and its related scheduling logic from the worker bootstrap.
- Updated the AI summarizer client initialization comment for clarity.
- Added a new error type for cases when tender documents have not been scraped yet, enhancing error handling in the tender service.
- Modified API documentation to reflect changes in AI summary retrieval logic, ensuring accurate descriptions of on-demand summarization behavior.
This update streamlines the AI summarization process by eliminating the document summarization worker, improving overall system efficiency and clarity in error handling.
- Updated the document scraper service to include a new ScrapePortalsProvider interface, allowing for dynamic retrieval of supported scraping portals.
- Modified the ListPendingTenders and GetTenderByNoticeID methods to filter tenders based on document URLs that match the configured portals.
- Introduced new error handling for cases when the scrape portals provider is not configured, returning appropriate service unavailable responses.
- Enhanced API documentation to reflect changes in tender retrieval logic and added error response details for unsupported portal scenarios.
This update improves the document scraping functionality by integrating AI portal support, enhancing the overall reliability and flexibility of the tender management system.
- Updated the `AggregateProcurementLotEstimatedValue` function to reject mixed currencies and allow empty lot currencies, ensuring accurate aggregation of estimated values.
- Introduced new unit tests in `entity_test.go` to validate the behavior of the aggregation function under various scenarios, including mixed currencies and empty lot currencies.
- Refactored budget calculation in `budget_test.go` to utilize the new aggregation logic, improving consistency in budget retrieval.
This update improves the handling of estimated values in procurement lots, enhancing the reliability of the tender management system.
- Introduced caching mechanisms for summary and statistics in the dashboard service, improving performance by reducing redundant data retrieval.
- Refactored the Summary method to utilize MongoDB aggregation for more efficient data processing and retrieval.
- Added synchronization features using singleflight to prevent duplicate processing of requests for cached data.
- Updated the repository to include a cachedScrapedTendersScope method, enhancing the efficiency of scraped document statistics retrieval.
This update significantly optimizes the dashboard's performance and data handling capabilities, ensuring faster response times and reduced load on the database.
- Updated the `ResolvedEstimatedValueAndCurrency` method to aggregate procurement lot values when the tender-level estimated value is not set, improving accuracy in value retrieval.
- Introduced the `AggregateProcurementLotEstimatedValue` function to sum estimated values from procurement lots and return the first found currency.
- Modified the `ToResponseWithLanguage` method to utilize the new estimated value resolution logic.
- Added unit tests for the new functionality, ensuring correct behavior for various scenarios in the `entity_test.go` and `budget_test.go` files.
This update improves the handling of estimated values in tenders, enhancing the overall reliability of the tender management system.
- Changed the ScrapePortalsResponse type to return a slice of strings representing portal identifiers instead of a structured object.
- Updated the Swagger documentation for the GetScrapePortals endpoint to reflect the new response format, ensuring clarity in API usage.
This update simplifies the response structure for the scraping portals, enhancing the API's usability and consistency.
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.
This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
- Introduced a new `scrapedTendersScope` type to encapsulate the MongoDB filter for tenders with scraped documents, improving clarity and maintainability.
- Updated the `Statistics` method to utilize the new scope resolution, allowing for more accurate data retrieval based on the presence of scraped documents.
- Implemented multiple tests for the `resolveScrapedTendersScope` method, ensuring correct behavior for various scenarios, including empty and fallback cases.
This update enhances the dashboard's ability to manage scraped document statistics, improving overall data accuracy and system performance.
- Introduced the ProcedureDocumentsLister interface to list contract folders with scraped documents, enhancing the accuracy of document-scrape statistics.
- Updated the dashboard repository to accept ProcedureDocumentsLister as a dependency, allowing for improved data retrieval.
- Implemented tests for the new functionality, ensuring proper handling of scraped document folder IDs and error propagation.
This update enhances the dashboard's capability to manage and report on scraped documents, improving overall system efficiency and data integrity.
- Updated the `Statistics` method to utilize `created_at` instead of `processing_metadata.scraped_at` for fetching daily counts, ensuring accurate historical data representation.
- Removed redundant conditions in the `scrapedDocumentsPerDay` method, streamlining the query logic for better performance and clarity.
- Added a new index on `source` and `created_at` to optimize database queries related to scraped documents.
This update enhances the accuracy of data retrieval in the dashboard statistics, improving the overall efficiency of the tender management system.
- Added `NoticePublicationID` field to the `SearchForm` for filtering tenders by TED notice publication ID.
- Updated Swagger documentation in the handler methods to include the new `notice_publication_id` parameter for relevant endpoints.
- Modified the repository's search filter to incorporate the new `NoticePublicationID` field, allowing for more precise search queries.
This update improves the search capabilities within the tender management system, enhancing user experience and data retrieval accuracy.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Updated the AI pipeline service to include a new `ScrapedDocumentMetadataSyncer` interface for persisting scraped document metadata onto tender records.
- Modified the `NewService` function to accept the new metadata syncer dependency.
- Implemented synchronization of scraped document metadata in the `ScrapeDocuments` and `Run` methods.
- Enhanced the tender service to enrich search filters based on scraped documents and added a new method for syncing scraped documents from storage.
- Updated the `SearchForm` to include `ContractFolderIDsWithDocuments` for better handling of scraped documents in queries.
This update improves the integration of scraped document handling within the AI pipeline, enhancing data consistency and operational efficiency in the tender management system.
- Added `TranslationEnabled` and `TranslationInterval` fields to the worker configuration to manage automatic translation scheduling.
- Updated the worker initialization to log when the translation worker is disabled.
- Improved error handling in the AI summarizer client by introducing `APIStatusError` for better context on API failures, replacing direct error messages with structured error responses.
This update enhances the configurability of the worker and improves error reporting for AI service interactions, contributing to better maintainability and user experience.
- Added new routes and handlers for AI pipeline operations, including scraping documents, batch summarization, translation, and syncing with the Opplens AI service.
- Introduced request forms for handling tender references and batch operations.
- Enhanced the AI service with methods for triggering batch operations and managing pipeline runs.
- Updated Swagger documentation to reflect the new AI pipeline endpoints and their functionalities.
This update integrates comprehensive AI pipeline capabilities into the tender management system, improving operational efficiency and user experience.
- Revised the Gap Analysis Report to provide detailed counts of implemented, partial, and not implemented controls, enhancing clarity on compliance status.
- Updated the ISO27001 Roadmap to include a status column for deliverables, improving tracking of progress towards certification.
- Adjusted the Statement of Applicability to reconcile summary counts with control rows and clarify the applicability of outsourced development.
These updates strengthen the documentation framework, ensuring it accurately reflects the current state of security controls and compliance efforts within the Tender Management System.
Mazyar
mazyar