- Introduced a mutex to ensure only one TED scraper run executes at a time, preventing concurrent executions during startup and scheduled runs.
- Implemented a mechanism to check if today's TED scrape has already been completed during startup, logging appropriate messages for both completed and new runs.
- Added startup catch-up logic for tender translations and unprocessed notices, ensuring that any missed tasks are executed without blocking the application startup.
This update improves the reliability and efficiency of the TED scraper and worker processes, ensuring that all necessary tasks are completed after a server restart.
- Updated the InitTEDScraper function to return a scheduler instance, allowing for better error handling during initialization.
- Added error logging for failed scheduler initialization and job scheduling, improving robustness.
- Implemented graceful shutdown for the scheduler in the main function, ensuring proper resource management during application termination.
- Removed the entrypoint.sh script from the scraper Dockerfile to streamline the build process.
- Updated the Dockerfile to directly set the ENTRYPOINT to the scraper build executable, simplifying the container startup.
- This change enhances maintainability by reducing the number of files and improving clarity in the Docker configuration.
- Removed command line flag parsing for date range scraping in the scraper's main function.
- Introduced configuration fields for one-time scraping and date range (FromDate, ToDate) in the ScraperConfig struct.
- Updated the main function to utilize the new configuration fields, improving the clarity and maintainability of the scraping logic.
- This change enhances the scraper's functionality by centralizing configuration management and reducing command line complexity.
- Replaced inline entrypoint script creation in the Dockerfile with a separate entrypoint.sh file for better maintainability and clarity.
- The new entrypoint script handles one-time scraping with date range support, improving the scraper's functionality and usability.
- Added an entrypoint script to the scraper Dockerfile to handle one-time scraping with date range support.
- Introduced a new ContentXML field in the Notice entity to store XML content, improving data structure for notices.
- Updated the scraper logic to map XML content to the tender entity, enhancing data processing capabilities.
- Added command line flags for specifying a date range and a one-time scraping mode in the scraper's main function.
- Implemented validation for the date inputs to ensure both dates are provided and correctly formatted.
- Introduced a new RunOneTimeScraping function to handle the one-time scraping logic, including context management and error handling.
- Updated the TED scraper initialization to support both scheduled and one-time scraping modes, improving flexibility in data retrieval.
- Enhanced logging to provide clear feedback on the scraping process and any errors encountered.
- Introduced AlertMail configuration in both scraper and worker bootstrap files, allowing for customizable email notifications.
- Updated the TED scraper to utilize the AlertMail configuration for sending completion notifications, improving flexibility in notification management.
- Enhanced error logging in the worker's main function to capture issues when listing Ollama models, ensuring better visibility into potential failures.
- Refactored notification sending logic to check for a valid AlertMail before dispatching emails, ensuring notifications are only sent when configured.
- Improved overall structure and readability of the bootstrap configuration files, aligning with best practices for maintainability.
- Moved the initialization of the notification service to follow the MongoDB connection setup for better organization.
- Added comments to clarify the initialization sequence of services, including the TED scraper.
- Removed commented-out code related to stopping the scheduler to clean up the main function.
- Enhanced the overall readability and structure of the main application entry point.
- Replaced the tender repository with a new notice repository, encapsulating notice-related data access methods.
- Introduced the Notice entity to represent tender/contract notices, including relevant fields and methods for managing notice data.
- Updated the TED scraper to utilize the new notice repository for creating and managing notices, enhancing the integration with the tender management system.
- Implemented Ollama SDK initialization in the web bootstrap process, allowing for improved AI interactions.
- Enhanced the tender service to include Ollama SDK for additional functionality, ensuring a more robust service layer.
- Introduced a new configuration file for the TED scraper, defining database connection settings, logging preferences, and scraping parameters.
- Implemented a CronScheduler to manage scheduled tasks for TED operations, utilizing the robfig/cron library for scheduling.
- Added new entities and structures for handling TED XML data, including eForms and tender-related information, enhancing the scraper's functionality.
- Updated the main application to initialize the TED scraper with the new configuration and set up graceful shutdown handling.
- Removed the deprecated scraping implementation to streamline the codebase and focus on the new architecture.
- Deleted the TED XML parser and scraper implementation files, including parser.go, scraper.go, and ted.go, to streamline the codebase and eliminate unused components.
- Commented out the initialization code in main.go to prevent execution while maintaining the structure for future reference.
- Removed the README.md for the TED package, which contained documentation for the now-deleted parser and scraper functionalities.
- Updated feedback repository and service methods to use a consistent naming convention, changing `NewTenderRepository` and similar methods to `NewRepository`.
- Refactored feedback handler methods to improve clarity by renaming methods such as `ListFeedback` to `Search` and `GetFeedback` to `Get`.
- Enhanced feedback response structures to include detailed company and tender information, improving the clarity of feedback data returned to API consumers.
- Updated Swagger documentation to reflect changes in feedback response structures and endpoint paths, ensuring accurate representation of the API for consumers.
- Removed YAML configuration files for the scraper and web commands, transitioning to an environment-based configuration system.
- Updated configuration structs to utilize `env` tags for loading values from `.env` files, enhancing security and flexibility.
- Implemented reflection-based parsing for environment variables, ensuring type-safe configuration loading.
- Revised documentation to reflect the new configuration approach, including usage examples and best practices for environment variable naming conventions.
- Deleted the docker-compose.yml file to streamline the project structure and reduce complexity in managing services.
- Updated the entry point in the scraper Dockerfile to use the correct path for the scraper binary, ensuring proper execution within the container.
- Changed the working directory in the Dockerfile to /app for better organization.
- Removed the iat.yaml configuration file to simplify the container setup and reduce complexity.
- Adjusted the exposed port to 80, aligning with standard HTTP practices for the scraper application.
- Updated the CI configuration in .drone.yml to reflect these changes, ensuring a streamlined build process.
- Introduced iat.yaml configuration files for both the scraper and web applications, defining essential settings such as database connections, logging configurations, and scraping parameters.
- Updated the Dockerfiles for both applications to include the new iat.yaml files, ensuring they are available in the container environment for runtime configuration.
- This enhancement improves the maintainability and configurability of both applications, allowing for easier adjustments to settings without code changes.
- Added a new Dockerfile for the scraper application located at cmd/scraper/Dockerfile to streamline the build process.
- Updated the CI configuration in .drone.yml to correctly build the scraper application with the new Dockerfile.
- Refactored the main application entry point to utilize a new bootstrap package for configuration, logging, and MongoDB initialization.
- Enhanced the scraper's configuration management by defining a dedicated Config struct for better organization and maintainability.
- Added 'ted_samples/' to .gitignore to exclude TED sample files from version control.
- Updated the TED scraper configuration to extend the cleanup duration from 24 hours to 72 hours, allowing for better resource management.
- Enhanced the Tender entity by adding new fields: TenderDeadline, SubmissionDeadline, ApplicationDeadline, and SubmissionURL, improving the data model for tender management.
- Updated the TenderResponse structure to include the new fields, ensuring consistency in API responses.
- Implemented logic in the TED scraper to calculate submission and application deadlines based on tender deadlines, enhancing the scraping functionality.
- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files.
- Added configuration management for the scraper, including a new `Config` struct and YAML configuration file.
- Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles.
- Implemented comprehensive logging and error handling throughout the scraper functionality.
- Updated API routes to include tender management operations, enhancing the overall system capabilities.
Mazyar