- Removed YAML configuration files for the scraper and web commands, transitioning to an environment-based configuration system.
- Updated configuration structs to utilize `env` tags for loading values from `.env` files, enhancing security and flexibility.
- Implemented reflection-based parsing for environment variables, ensuring type-safe configuration loading.
- Revised documentation to reflect the new configuration approach, including usage examples and best practices for environment variable naming conventions.
- Deleted the docker-compose.yml file to streamline the project structure and reduce complexity in managing services.
- Updated the entry point in the scraper Dockerfile to use the correct path for the scraper binary, ensuring proper execution within the container.
- Changed the working directory in the Dockerfile to /app for better organization.
- Removed the iat.yaml configuration file to simplify the container setup and reduce complexity.
- Adjusted the exposed port to 80, aligning with standard HTTP practices for the scraper application.
- Updated the CI configuration in .drone.yml to reflect these changes, ensuring a streamlined build process.
- Introduced iat.yaml configuration files for both the scraper and web applications, defining essential settings such as database connections, logging configurations, and scraping parameters.
- Updated the Dockerfiles for both applications to include the new iat.yaml files, ensuring they are available in the container environment for runtime configuration.
- This enhancement improves the maintainability and configurability of both applications, allowing for easier adjustments to settings without code changes.
- Added a new Dockerfile for the scraper application located at cmd/scraper/Dockerfile to streamline the build process.
- Updated the CI configuration in .drone.yml to correctly build the scraper application with the new Dockerfile.
- Refactored the main application entry point to utilize a new bootstrap package for configuration, logging, and MongoDB initialization.
- Enhanced the scraper's configuration management by defining a dedicated Config struct for better organization and maintainability.
- Added 'ted_samples/' to .gitignore to exclude TED sample files from version control.
- Updated the TED scraper configuration to extend the cleanup duration from 24 hours to 72 hours, allowing for better resource management.
- Enhanced the Tender entity by adding new fields: TenderDeadline, SubmissionDeadline, ApplicationDeadline, and SubmissionURL, improving the data model for tender management.
- Updated the TenderResponse structure to include the new fields, ensuring consistency in API responses.
- Implemented logic in the TED scraper to calculate submission and application deadlines based on tender deadlines, enhancing the scraping functionality.
- Introduced a new TED scraper in `cmd/scraper` to handle downloading and parsing TED XML files.
- Added configuration management for the scraper, including a new `Config` struct and YAML configuration file.
- Created necessary handler, service, and repository layers for tender management, adhering to Clean Architecture principles.
- Implemented comprehensive logging and error handling throughout the scraper functionality.
- Updated API routes to include tender management operations, enhancing the overall system capabilities.
n.nakhostin