- Updated the tender search handler to include a new `include_total` query parameter, allowing clients to request total count and pagination metadata for search results.
- Refactored the `tenderSearchListProjection` to exclude heavy fields from the list response, optimizing data retrieval.
- Modified the `buildSearchFilter` method to utilize the `processing_metadata.documents_scraped` flag for filtering, improving search accuracy.
- Added unit tests for the new pagination features and search filter logic, ensuring robust validation of the search functionality.
This update significantly enhances the tender search experience by providing more flexible pagination options and improving the efficiency of data handling.
- Renamed and refactored the AI pipeline auto run functionality to a daily run, enhancing clarity and purpose.
- Introduced a new `AIPipelineDailyWorker` to manage the daily execution of the AI pipeline, replacing the previous auto run implementation.
- Updated configuration fields and logging messages to reflect the change from auto to daily run, ensuring consistent terminology throughout the codebase.
- Removed the obsolete `ai_pipeline_auto.go` file to streamline the worker structure.
This update improves the maintainability and readability of the AI pipeline management by clearly distinguishing between auto and daily run functionalities.
- Introduced the ability to append external links to company profiles through a new API endpoint.
- Enhanced the `Company` entity to include a `Links` field for storing external resource links.
- Created `AddLinksForm` for validating incoming link data and implemented corresponding logic in the service layer.
- Added error handling for link validation, ensuring only valid URLs are accepted and limiting the number of links to 20.
- Implemented unit tests for link management functions, including sanitization and merging of links.
- Updated relevant API documentation to reflect the new functionality.
This update significantly enhances the company management capabilities by allowing the addition of external links, improving the overall user experience and data richness in company profiles.
- Introduced AIPipelineAutoWorker to manage the execution of the AI pipeline auto run, including startup catch-up and scheduled tasks.
- Enhanced WorkerConfig to include AIPipelineAutoEnabled and AIPipelineAutoInterval settings for better control over AI pipeline execution.
- Added logging for AI pipeline auto run status, including success and error handling, to improve observability.
- Updated daily job tracker to include AIPipelineAutoJobName for tracking AI pipeline job completions.
This update enhances the system's capability to automate AI pipeline executions, improving efficiency and reliability in processing AI tasks.
- Added a new field, ScrapedTEDNotices, to the StatisticsLifetimeTotals struct to track the total number of TED notices scraped.
- Updated the Statistics method in the statistics repository to include a background process for retrieving total scraped TED notices, improving the accuracy of dashboard statistics.
- Introduced new methods in the Counter to increment and retrieve daily counts for scraped TED notices, ensuring reliable metrics for reporting.
- Modified the TEDScraper to increment the TED notice scraped counter upon successful import, enhancing the tracking of scraping activity.
This update improves the dashboard's statistics by providing detailed insights into TED notice scraping activities, contributing to better data visibility and reporting.
- Updated the ListPaginationOptions struct to include SkipCount and IncludeCount fields, allowing for more flexible pagination behavior.
- Modified the BuildListPagination function to handle cursor pagination with count options, improving performance by running count queries in parallel with data retrieval.
- Enhanced the FindAll method in the repository to support concurrent counting of documents, reducing overall latency for list operations.
- Added tests for pagination behavior, ensuring accurate handling of count scenarios in both offset and cursor pagination.
This update improves the efficiency and flexibility of pagination in the MongoDB repository, enhancing the overall performance of list operations.
- Updated error handling in the Elasticsearch client to utilize a new `readErrorBody` function for better diagnostics when pinging and searching.
- Refactored the `flushBatch` method to marshal bulk index actions using a dedicated `marshalBulkIndexAction` function, improving code clarity and error handling.
- Enhanced the overall structure of the Elasticsearch client for improved maintainability and readability.
This update enhances the robustness of the Elasticsearch client, ensuring more informative error messages and cleaner code organization.
- Introduced a new `auditlog` package to handle audit logging for user actions, including creation, updates, deletions, and authentication events.
- Enhanced existing services (customer, user) to log relevant actions using the new audit logger, capturing details such as actor ID, action type, target type, and success status.
- Added middleware to enrich request context with metadata for audit logging, ensuring comprehensive tracking of user interactions.
- Integrated Elasticsearch for persistent storage of audit logs, with fallback to file-only logging if Elasticsearch is unavailable.
- Updated API documentation to include new audit log endpoints for administrative access.
This update significantly improves the system's ability to track and audit user actions, enhancing security and accountability within the application.
- Introduced the `scrapedDocumentsScanner` interface to facilitate scanning of scraped documents from MinIO, returning both procedure summaries and daily document counts.
- Updated the `ListProceduresWithDocuments` method to utilize the new scanning functionality, improving data retrieval efficiency.
- Enhanced the `scrapedDocumentsPerDay` method to filter daily counts based on a specified start date, ensuring accurate reporting of document statistics.
- Added unit tests for the new scanning logic and daily counts filtering, ensuring robust functionality and error handling.
This update enhances the dashboard's document management capabilities, providing better insights into scraped documents and their daily counts.
- Introduced a mutex to ensure only one TED scraper run executes at a time, preventing concurrent executions during startup and scheduled runs.
- Implemented a mechanism to check if today's TED scrape has already been completed during startup, logging appropriate messages for both completed and new runs.
- Added startup catch-up logic for tender translations and unprocessed notices, ensuring that any missed tasks are executed without blocking the application startup.
This update improves the reliability and efficiency of the TED scraper and worker processes, ensuring that all necessary tasks are completed after a server restart.
- Updated the `UploadDocuments` method to sanitize document file IDs before saving, ensuring only valid references are stored.
- Introduced `DetachDocumentFileID` method in the `companyService` to remove file IDs from all companies referencing a deleted file, improving data integrity.
- Enhanced the `companyRepository` with a new method to handle the removal of document file IDs from the database.
- Updated the `filestore` handler to utilize the new detachment functionality when files are deleted, ensuring consistent state across domain entities.
This update improves the management of document file IDs within the company domain, enhancing data integrity and reference handling.
- Changed the `Rank` field type from `string` to `int` in the `RecommendedTenderResponse` struct within the `company` domain for better data representation.
- Updated the `Rank` field type from `string` to `int` in the `TenderResponse` struct within the `tender` domain to ensure consistency in ranking data.
- Modified the `Rank` field type from `string` to `int` in the `RecommendedTender` struct within the AI summarizer package to align with the updated data structure.
This update enhances the data integrity and consistency across the tender management system by standardizing the rank representation as an integer.
- Changed the ScrapePortalsResponse type to return a slice of strings representing portal identifiers instead of a structured object.
- Updated the Swagger documentation for the GetScrapePortals endpoint to reflect the new response format, ensuring clarity in API usage.
This update simplifies the response structure for the scraping portals, enhancing the API's usability and consistency.
- Introduced the GetScrapePortals method in the AI pipeline handler to list document scraping portals supported by the Opplens AI service.
- Updated the service layer to include GetScrapePortals, which retrieves the portals from the client and handles errors appropriately.
- Enhanced the routes to register the new endpoint for retrieving scrape portals.
- Added a new error type for invalid date ranges in the document scraper, improving validation and error handling.
This update expands the AI pipeline capabilities, allowing for better management of document scraping portals within the tender management system.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Added `TranslationEnabled` and `TranslationInterval` fields to the worker configuration to manage automatic translation scheduling.
- Updated the worker initialization to log when the translation worker is disabled.
- Improved error handling in the AI summarizer client by introducing `APIStatusError` for better context on API failures, replacing direct error messages with structured error responses.
This update enhances the configurability of the worker and improves error reporting for AI service interactions, contributing to better maintainability and user experience.
- Added new routes and handlers for AI pipeline operations, including scraping documents, batch summarization, translation, and syncing with the Opplens AI service.
- Introduced request forms for handling tender references and batch operations.
- Enhanced the AI service with methods for triggering batch operations and managing pipeline runs.
- Updated Swagger documentation to reflect the new AI pipeline endpoints and their functionalities.
This update integrates comprehensive AI pipeline capabilities into the tender management system, improving operational efficiency and user experience.
- Added new AI onboarding and recommendation endpoints in the company handler for starting onboarding and retrieving ranked tender recommendations.
- Introduced `StartAIOnboarding` and `GetAIRecommendations` methods in the company service to handle AI interactions.
- Updated the company service constructor to include the AI recommendation client.
- Enhanced the AI summarizer client with methods for onboarding and fetching recommendations.
- Added response structures for onboarding and recommended tenders in the company form.
This update enhances the tender management system by integrating AI capabilities for onboarding and tender recommendations, improving user experience and operational efficiency.
- Removed the QueueConfig structure and related queue management files as they are no longer in use.
- Updated the worker initialization to reflect the removal of queue-related configurations.
- Cleaned up the bootstrap process by eliminating deprecated logging related to the queue system.
This update streamlines the worker's configuration and prepares the codebase for future enhancements without the legacy queue management components.
- Added a new endpoint to trigger on-demand agentic analysis for tenders via the AI service.
- Introduced `TriggerAIAnalyze` method in the `TenderHandler` to handle requests and responses for AI analysis.
- Updated the `tenderService` to include `TriggerAIAnalyze` method, which validates input and interacts with the AI summarizer client.
- Enhanced the `AIAnalyzeResponse` and `AIAnalyzeDocument` structures to support the new analysis feature.
- Refactored the `DocumentSummarizationWorker` and `TranslationWorker` to remove deprecated MinIO dependencies, streamlining the AI service interactions.
This update improves the functionality of the tender management system by allowing users to trigger AI analysis on-demand, enhancing the overall user experience and system capabilities.
- Improved the `IsDuplicateKeyError` function to utilize structured error messages for better accuracy in identifying duplicate key errors.
- Introduced `duplicateKeyMessages` and `hasStructuredWriteException` functions to parse and extract relevant information from MongoDB error messages.
- Updated `DuplicateKeyMatchesField` to match duplicate key errors against specific fields by analyzing error messages rather than relying on substring matching.
This refactor enhances the clarity and reliability of duplicate key error handling, ensuring more precise identification of conflicts in MongoDB operations.
- Updated `InitAISummarizerClient` to accept `mongoManager` for tracking translation success.
- Introduced new `Statistics` endpoint in the dashboard to fetch scraping and translation statistics.
- Enhanced `TranslationWorker` to utilize the new success counter for tracking successful translations.
- Added necessary data structures and query forms for statistics reporting.
This refactor improves the tracking of AI translation success and provides new insights through the dashboard statistics.
Map E11000 from the per-collection phone index to friendly conflict
errors and skip redundant GetByPhone when the phone is unchanged.
Not blocking: IAT test DB may need a one-off dedupe for index build;
phone uniqueness is per collection (users vs customers), not global.
Frontend sends nested address field names for sorting; accept them alongside the existing country and state aliases.
Co-authored-by: Cursor <cursoragent@cursor.com>
Mazyar