- Updated the `Statistics` method to utilize `created_at` instead of `processing_metadata.scraped_at` for fetching daily counts, ensuring accurate historical data representation.
- Removed redundant conditions in the `scrapedDocumentsPerDay` method, streamlining the query logic for better performance and clarity.
- Added a new index on `source` and `created_at` to optimize database queries related to scraped documents.
This update enhances the accuracy of data retrieval in the dashboard statistics, improving the overall efficiency of the tender management system.
- Changed the default MinIO bucket name from "opplens-documents" to "opplens" across multiple configuration files.
- Introduced caching for dashboard statistics in the service layer to improve performance and reduce redundant data fetching.
- Implemented a mutex for thread-safe access to cached statistics, ensuring data integrity during concurrent requests.
This update streamlines the configuration for the AI summarizer and optimizes the dashboard service, enhancing overall system efficiency.
- Added new indexes for the notices collection in the notice repository to improve query performance.
- Introduced indexes for `processing_metadata.processed` and `main_classification`, optimizing data retrieval and management.
- Modified the GetUnProcessedNotices method in the notice repository to include a filter for main_classification set to "72000000".
- Changed the sort order from ascending to descending for the created_at field, improving the retrieval of unprocessed notices.
- Added an entrypoint script to the scraper Dockerfile to handle one-time scraping with date range support.
- Introduced a new ContentXML field in the Notice entity to store XML content, improving data structure for notices.
- Updated the scraper logic to map XML content to the tender entity, enhancing data processing capabilities.
- Updated the worker initialization process to include the new GLM SDK, enhancing the worker's capabilities for translation tasks.
- Modified the InitWorker function and NewNoticeWorker constructor to accept the GLM service, ensuring a cohesive integration.
- Implemented the GLM service initialization and logging for successful setup, improving maintainability and usability.
- Updated the NoticeWorker to utilize the GLM SDK for translating notice titles and descriptions, enhancing functionality and user experience.
- Introduced a new worker service with a dedicated main entry point for handling background tasks, including MongoDB connection management and notification service initialization.
- Added a bootstrap package to manage application configuration, logging, and service initialization for the worker.
- Implemented a NoticeWorker to process unprocessed notices and create corresponding tender entities, enhancing the integration of notice management within the tender system.
- Refactored the tender service to remove the Ollama SDK dependency, streamlining the service initialization in the main application.
- Enhanced the notice repository with a method to retrieve unprocessed notices, improving data handling capabilities.
- Replaced the tender repository with a new notice repository, encapsulating notice-related data access methods.
- Introduced the Notice entity to represent tender/contract notices, including relevant fields and methods for managing notice data.
- Updated the TED scraper to utilize the new notice repository for creating and managing notices, enhancing the integration with the tender management system.
- Implemented Ollama SDK initialization in the web bootstrap process, allowing for improved AI interactions.
- Enhanced the tender service to include Ollama SDK for additional functionality, ensuring a more robust service layer.
Mazyar