Add TED Scraper Configuration and Cron Scheduler Implementation
- Introduced a new configuration file for the TED scraper, defining database connection settings, logging preferences, and scraping parameters. - Implemented a CronScheduler to manage scheduled tasks for TED operations, utilizing the robfig/cron library for scheduling. - Added new entities and structures for handling TED XML data, including eForms and tender-related information, enhancing the scraper's functionality. - Updated the main application to initialize the TED scraper with the new configuration and set up graceful shutdown handling. - Removed the deprecated scraping implementation to streamline the codebase and focus on the new architecture.
This commit is contained in:
n.nakhostin
@@ -0,0 +1,349 @@
|
||||
# TED XML Scraper
|
||||
|
||||
A comprehensive TED (Tenders Electronic Daily) XML scraper that downloads and parses daily calendar XML files, converting them to tender entities and storing them in MongoDB.
|
||||
|
||||
## Features
|
||||
|
||||
- **Daily Scheduled Scraping**: Runs automatically at 10:00 AM daily using cron jobs
|
||||
- **TED XML Format Support**: Handles eForms XML, Contract Notices, and TED Calendar formats
|
||||
- **XML Parser Integration**: Uses the custom `xmlparser` package with generic type support
|
||||
- **Robust Error Handling**: Comprehensive error handling, retry mechanisms, and logging
|
||||
- **Data Conversion**: Converts TED XML structures to internal tender entity models
|
||||
- **MongoDB Integration**: Stores processed data with proper indexing and validation
|
||||
- **Progress Tracking**: Job tracking, status monitoring, and statistics
|
||||
- **Configurable**: Environment variables and YAML configuration support
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
cmd/scraper/
|
||||
├── main.go # Application entry point
|
||||
├── config.yaml # Default configuration
|
||||
├── bootstrap/
|
||||
│ ├── bootstrap.go # Application initialization
|
||||
│ └── config.go # Configuration structures
|
||||
└── README.md # This file
|
||||
|
||||
internal/scraper/
|
||||
├── service.go # Core scraping service logic
|
||||
├── repository.go # MongoDB operations for scraping jobs
|
||||
└── scheduler.go # Cron job scheduling
|
||||
|
||||
pkg/xmlparser/ted/
|
||||
├── models.go # TED XML data structures
|
||||
└── converter.go # XML to entity conversion
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
1. **Prerequisites**:
|
||||
- Go 1.21+
|
||||
- MongoDB 4.4+
|
||||
|
||||
2. **Dependencies**:
|
||||
```bash
|
||||
cd cmd/scraper
|
||||
go mod tidy
|
||||
```
|
||||
|
||||
3. **Configuration**:
|
||||
Create a `config.yaml` file or set environment variables:
|
||||
```yaml
|
||||
database:
|
||||
mongodb:
|
||||
uri: "mongodb://localhost:27017"
|
||||
name: "tender_management"
|
||||
|
||||
ted:
|
||||
base_url: "https://ted.europa.eu"
|
||||
timeout: 30s
|
||||
max_retries: 3
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Running the Scraper
|
||||
|
||||
```bash
|
||||
# Run with default configuration
|
||||
go run main.go
|
||||
|
||||
# Run with custom config file
|
||||
CONFIG_PATH=./custom-config.yaml go run main.go
|
||||
|
||||
# Run with environment variables
|
||||
TED_BASE_URL=https://ted.europa.eu go run main.go
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `TED_BASE_URL` | `https://ted.europa.eu` | TED website base URL |
|
||||
| `TED_TIMEOUT` | `30s` | HTTP request timeout |
|
||||
| `TED_MAX_RETRIES` | `3` | Maximum retry attempts |
|
||||
| `TED_RETRY_DELAY` | `5s` | Delay between retries |
|
||||
| `TED_USER_AGENT` | `TM-TED-Scraper/1.0` | HTTP User-Agent header |
|
||||
| `TED_MAX_CONCURRENCY` | `5` | Maximum concurrent downloads |
|
||||
| `TED_DOWNLOAD_DIR` | `./downloads` | Download directory |
|
||||
| `TED_CLEANUP_AFTER` | `24h` | File cleanup interval |
|
||||
|
||||
### Manual Trigger
|
||||
|
||||
To manually trigger a scraping job outside the schedule:
|
||||
|
||||
```bash
|
||||
# Using the scheduler API (if implemented)
|
||||
curl -X POST http://localhost:8080/api/scraper/trigger
|
||||
|
||||
# Or restart the application to trigger immediate run
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Daily Calendar Download
|
||||
|
||||
The scraper downloads TED daily packages using this URL pattern:
|
||||
```
|
||||
https://ted.europa.eu/packages/daily/{yyyynnnnn}
|
||||
```
|
||||
|
||||
Where `{yyyynnnnn}` is the year + 5-digit day of year (e.g., `202500001` for January 1st, 2025).
|
||||
|
||||
### 2. XML Parsing
|
||||
|
||||
The scraper supports multiple TED XML formats:
|
||||
|
||||
- **Contract Notice**: Individual tender announcements
|
||||
- **Tender Calendar**: Daily collections of multiple notices
|
||||
- **eForms**: New European standard format
|
||||
- **Legacy TED Schema**: Older format support
|
||||
|
||||
### 3. Data Conversion
|
||||
|
||||
XML data is converted to internal tender entities with:
|
||||
|
||||
- **Field Mapping**: TED XML fields → Tender entity fields
|
||||
- **Data Validation**: Required field validation and sanitization
|
||||
- **Unix Timestamps**: All dates converted to Unix timestamps
|
||||
- **Text Normalization**: Clean up whitespace and formatting
|
||||
- **URL Generation**: Generate TED website URLs for notices
|
||||
|
||||
### 4. Storage
|
||||
|
||||
Processed data is stored in MongoDB with:
|
||||
|
||||
- **Indexes**: Optimized for search and filtering
|
||||
- **Deduplication**: Based on contract notice ID
|
||||
- **Audit Trail**: Creation and update timestamps
|
||||
- **Error Tracking**: Processing errors and warnings
|
||||
|
||||
## TED XML Format Analysis
|
||||
|
||||
### Contract Notice Structure
|
||||
|
||||
```xml
|
||||
<ContractNotice xmlns="urn:oasis:names:specification:ubl:schema:xsd:ContractNotice-2">
|
||||
<ID>123456-2025</ID>
|
||||
<ContractFolderID>folder-123</ContractFolderID>
|
||||
<IssueDate>2025-01-09+02:00</IssueDate>
|
||||
<NoticeTypeCode>cn-standard</NoticeTypeCode>
|
||||
<NoticeLanguageCode>ENG</NoticeLanguageCode>
|
||||
<!-- ... procurement project, buyer information, etc. -->
|
||||
</ContractNotice>
|
||||
```
|
||||
|
||||
### Key XML Elements Mapped
|
||||
|
||||
| XML Element | Tender Field | Description |
|
||||
|-------------|--------------|-------------|
|
||||
| `ID` | `contract_notice_id` | Unique notice identifier |
|
||||
| `ContractFolderID` | `contract_folder_id` | Contract folder reference |
|
||||
| `IssueDate` | `issue_date` | Notice publication date |
|
||||
| `ProcurementProject/Name` | `title` | Tender title |
|
||||
| `ProcurementProject/Description` | `description` | Tender description |
|
||||
| `ContractingParty` | `buyer_organization` | Buyer/contracting authority |
|
||||
| `TenderSubmissionDeadlinePeriod` | `tender_deadline` | Submission deadline |
|
||||
|
||||
## Scheduling
|
||||
|
||||
### Cron Schedule
|
||||
|
||||
- **Main Job**: `0 0 10 * * *` (10:00 AM daily UTC)
|
||||
- **Cleanup Job**: `0 0 2 * * *` (2:00 AM daily UTC)
|
||||
|
||||
### Job Configuration
|
||||
|
||||
```go
|
||||
schedulerConfig := scraper.SchedulerConfig{
|
||||
CronExpression: "0 0 10 * * *", // 10:00 AM daily (seconds, minutes, hours, day, month, weekday)
|
||||
Timezone: "UTC",
|
||||
JobTimeout: 2 * time.Hour,
|
||||
RetryAttempts: 3,
|
||||
RetryDelay: 30 * time.Minute,
|
||||
}
|
||||
```
|
||||
|
||||
### Retry Logic
|
||||
|
||||
- **Automatic Retries**: Up to 3 attempts on failure
|
||||
- **Exponential Backoff**: 30-minute delay between retries
|
||||
- **Context Timeout**: 2-hour maximum per job
|
||||
- **Graceful Shutdown**: Waits for running jobs to complete
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Job Status
|
||||
|
||||
Track scraping job progress and status:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "job-123",
|
||||
"type": "ted_scraper",
|
||||
"status": "running",
|
||||
"progress": {
|
||||
"processed_count": 150,
|
||||
"total_count": 500,
|
||||
"success_count": 145,
|
||||
"error_count": 5,
|
||||
"percentage": 30.0,
|
||||
"current_step": "processing_tenders"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Statistics
|
||||
|
||||
```json
|
||||
{
|
||||
"period": "Last 7 days",
|
||||
"total_jobs": 7,
|
||||
"completed_jobs": 6,
|
||||
"failed_jobs": 1,
|
||||
"total_processed": 3500,
|
||||
"total_created": 2100,
|
||||
"total_updated": 1200,
|
||||
"total_errors": 200
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Types
|
||||
|
||||
1. **Download Errors**: Network issues, server errors
|
||||
2. **Parsing Errors**: Invalid XML, schema mismatches
|
||||
3. **Validation Errors**: Missing required fields
|
||||
4. **Database Errors**: Connection issues, write failures
|
||||
|
||||
### Error Recovery
|
||||
|
||||
- **Retry Mechanism**: Automatic retries with backoff
|
||||
- **Partial Success**: Continue processing valid notices
|
||||
- **Error Logging**: Detailed error tracking and reporting
|
||||
- **Graceful Degradation**: Skip problematic entries
|
||||
|
||||
### Example Error Response
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job-123",
|
||||
"status": "completed",
|
||||
"results": {
|
||||
"total_processed": 100,
|
||||
"total_created": 85,
|
||||
"total_errors": 15,
|
||||
"processing_errors": [
|
||||
"notice 23: validation failed: title is required",
|
||||
"notice 45: failed to parse deadline: invalid format"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Performance
|
||||
|
||||
- **Batch Processing**: Process tenders in configurable batches (default: 10)
|
||||
- **Connection Pooling**: MongoDB connection management
|
||||
- **Memory Management**: Stream large XML files
|
||||
- **Concurrent Downloads**: Configurable concurrency limits
|
||||
|
||||
### Data Quality
|
||||
|
||||
- **Validation**: Comprehensive field validation
|
||||
- **Sanitization**: Text cleaning and normalization
|
||||
- **Deduplication**: Prevent duplicate tender entries
|
||||
- **Audit Trail**: Track all changes and updates
|
||||
|
||||
### Reliability
|
||||
|
||||
- **Health Checks**: Monitor service health
|
||||
- **Graceful Shutdown**: Clean exit on signals
|
||||
- **Resource Cleanup**: Automatic file cleanup
|
||||
- **State Persistence**: Track scraping state across restarts
|
||||
|
||||
## Development
|
||||
|
||||
### Adding New XML Formats
|
||||
|
||||
1. **Extend TED Models**: Add new XML structures in `pkg/xmlparser/ted/models.go`
|
||||
2. **Update Converter**: Add conversion logic in `pkg/xmlparser/ted/converter.go`
|
||||
3. **Add Parser Support**: Register new parsers in service
|
||||
4. **Test Thoroughly**: Add comprehensive test cases
|
||||
|
||||
### Custom Validation
|
||||
|
||||
```go
|
||||
// Add custom validators in converter.go
|
||||
func (c *Converter) ValidateCustomField(value string) error {
|
||||
if len(value) > 1000 {
|
||||
return fmt.Errorf("field too long: %d characters", len(value))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### Extending Scheduling
|
||||
|
||||
```go
|
||||
// Add new scheduled jobs in scheduler.go
|
||||
_, err = s.cron.AddFunc("0 */6 * * *", s.runStatisticsJob) // Every 6 hours
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Connection Errors**: Check MongoDB connection and credentials
|
||||
2. **Parsing Failures**: Verify TED XML format changes
|
||||
3. **Schedule Issues**: Check timezone and cron expression
|
||||
4. **Memory Usage**: Monitor for large XML files
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Enable debug logging:
|
||||
|
||||
```yaml
|
||||
logging:
|
||||
level: "debug"
|
||||
```
|
||||
|
||||
### Manual Testing
|
||||
|
||||
```bash
|
||||
# Test XML parsing
|
||||
go run main.go --test-xml ./sample.xml
|
||||
|
||||
# Test specific date
|
||||
go run main.go --test-date 2025-01-09
|
||||
|
||||
# Validate configuration
|
||||
go run main.go --validate-config
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
This project is part of the Tender Management system and follows the same licensing terms.
|
||||
+167
-123
@@ -1,140 +1,184 @@
|
||||
package bootstrap
|
||||
|
||||
// import (
|
||||
// "context"
|
||||
// "fmt"
|
||||
// "os"
|
||||
// "os/signal"
|
||||
// "syscall"
|
||||
// "time"
|
||||
// "tm/internal/tender"
|
||||
// "tm/pkg/config"
|
||||
// "tm/pkg/logger"
|
||||
// "tm/pkg/mongo"
|
||||
// "tm/ted"
|
||||
// )
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
"tm/internal/tender"
|
||||
"tm/pkg/config"
|
||||
"tm/pkg/logger"
|
||||
"tm/pkg/mongo"
|
||||
"tm/pkg/notification"
|
||||
"tm/pkg/schedule"
|
||||
"tm/ted"
|
||||
)
|
||||
|
||||
// // Init Application Configuration
|
||||
// func InitConfig() (*Config, error) {
|
||||
// conf, err := config.LoadConfig(".", &Config{})
|
||||
// if err != nil {
|
||||
// return nil, fmt.Errorf("failed to load config: %v", err)
|
||||
// }
|
||||
// Init Application Configuration
|
||||
func InitConfig() (*Config, error) {
|
||||
conf, err := config.LoadConfig(".", &Config{})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to load config: %v", err)
|
||||
}
|
||||
|
||||
// return conf, nil
|
||||
// }
|
||||
return conf, nil
|
||||
}
|
||||
|
||||
// // Init Logger Service
|
||||
// func InitLogger(conf config.LoggingConfig) logger.Logger {
|
||||
// return logger.NewLogger(&logger.Config{
|
||||
// Level: conf.Level,
|
||||
// Format: conf.Format,
|
||||
// Output: conf.Output,
|
||||
// File: logger.FileConfig{
|
||||
// Path: conf.File.Path,
|
||||
// MaxSize: conf.File.MaxSize,
|
||||
// MaxBackups: conf.File.MaxBackups,
|
||||
// MaxAge: conf.File.MaxAge,
|
||||
// Compress: conf.File.Compress,
|
||||
// },
|
||||
// })
|
||||
// }
|
||||
// Init Logger Service
|
||||
func InitLogger(conf config.LoggingConfig) logger.Logger {
|
||||
return logger.NewLogger(&logger.Config{
|
||||
Level: conf.Level,
|
||||
Format: conf.Format,
|
||||
Output: conf.Output,
|
||||
File: logger.FileConfig{
|
||||
Path: conf.File.Path,
|
||||
MaxSize: conf.File.MaxSize,
|
||||
MaxBackups: conf.File.MaxBackups,
|
||||
MaxAge: conf.File.MaxAge,
|
||||
Compress: conf.File.Compress,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// // Init MongoDB Connection Manager
|
||||
// func InitMongoDB(conf config.MongoConfig, log logger.Logger) *mongo.ConnectionManager {
|
||||
// // Convert infra.MongoConfig to mongo.ConnectionConfig
|
||||
// connectionConfig := mongo.ConnectionConfig{
|
||||
// URI: conf.URI,
|
||||
// Database: conf.Name,
|
||||
// MaxPoolSize: uint64(conf.MaxPoolSize),
|
||||
// MinPoolSize: 5, // Default minimum pool size
|
||||
// MaxConnIdleTime: 30 * time.Minute, // Default idle time
|
||||
// ConnectTimeout: conf.Timeout,
|
||||
// ServerSelectionTimeout: 5 * time.Second, // Default server selection timeout
|
||||
// }
|
||||
// Init MongoDB Connection Manager
|
||||
func InitMongoDB(conf config.MongoConfig, log logger.Logger) *mongo.ConnectionManager {
|
||||
// Convert infra.MongoConfig to mongo.ConnectionConfig
|
||||
connectionConfig := mongo.ConnectionConfig{
|
||||
URI: conf.URI,
|
||||
Database: conf.Name,
|
||||
MaxPoolSize: uint64(conf.MaxPoolSize),
|
||||
MinPoolSize: 5, // Default minimum pool size
|
||||
MaxConnIdleTime: 30 * time.Minute, // Default idle time
|
||||
ConnectTimeout: conf.Timeout,
|
||||
ServerSelectionTimeout: 5 * time.Second, // Default server selection timeout
|
||||
}
|
||||
|
||||
// // Create MongoDB connection manager
|
||||
// connectionManager, err := mongo.NewConnectionManager(connectionConfig, log)
|
||||
// if err != nil {
|
||||
// log.Error("Failed to initialize MongoDB connection", map[string]interface{}{
|
||||
// "error": err.Error(),
|
||||
// "uri": conf.URI,
|
||||
// "db": conf.Name,
|
||||
// })
|
||||
// panic(fmt.Sprintf("Failed to initialize MongoDB connection: %v", err))
|
||||
// }
|
||||
// Create MongoDB connection manager
|
||||
connectionManager, err := mongo.NewConnectionManager(connectionConfig, log)
|
||||
if err != nil {
|
||||
log.Error("Failed to initialize MongoDB connection", map[string]interface{}{
|
||||
"error": err.Error(),
|
||||
"uri": conf.URI,
|
||||
"db": conf.Name,
|
||||
})
|
||||
panic(fmt.Sprintf("Failed to initialize MongoDB connection: %v", err))
|
||||
}
|
||||
|
||||
// log.Info("MongoDB connection manager initialized successfully", map[string]interface{}{
|
||||
// "database": conf.Name,
|
||||
// "uri": conf.URI,
|
||||
// "pool_size": conf.MaxPoolSize,
|
||||
// })
|
||||
log.Info("MongoDB connection manager initialized successfully", map[string]interface{}{
|
||||
"database": conf.Name,
|
||||
"uri": conf.URI,
|
||||
"pool_size": conf.MaxPoolSize,
|
||||
})
|
||||
|
||||
// return connectionManager
|
||||
// }
|
||||
return connectionManager
|
||||
}
|
||||
|
||||
// // init TED scraper
|
||||
// func InitTEDScraper(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger) {
|
||||
// // Initialize tender repository
|
||||
// tenderRepo := tender.NewRepository(mongoManager, appLogger)
|
||||
// init TED scraper
|
||||
func InitTEDScraper(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger, notify notification.SDK) {
|
||||
// Initialize tender repository
|
||||
tenderRepo := tender.NewRepository(mongoManager, appLogger)
|
||||
|
||||
// // Initialize TED scraper
|
||||
// tedScraper := ted.NewTEDScraper(
|
||||
// &ted.Config{
|
||||
// BaseURL: config.TED.BaseURL,
|
||||
// Timeout: config.TED.Timeout,
|
||||
// MaxRetries: config.TED.MaxRetries,
|
||||
// RetryDelay: config.TED.RetryDelay,
|
||||
// UserAgent: config.TED.UserAgent,
|
||||
// MaxConcurrency: config.TED.MaxConcurrency,
|
||||
// DownloadDir: config.TED.DownloadDir,
|
||||
// CleanupAfter: config.TED.CleanupAfter,
|
||||
// ScrapingInterval: config.TED.ScrapingInterval,
|
||||
// },
|
||||
// appLogger,
|
||||
// mongoManager,
|
||||
// tenderRepo,
|
||||
// )
|
||||
// Initialize TED scraper
|
||||
tedScraper := ted.NewTEDScraper(
|
||||
&ted.Config{
|
||||
BaseURL: config.TED.BaseURL,
|
||||
Timeout: config.TED.Timeout,
|
||||
MaxRetries: config.TED.MaxRetries,
|
||||
RetryDelay: config.TED.RetryDelay,
|
||||
UserAgent: config.TED.UserAgent,
|
||||
MaxConcurrency: config.TED.MaxConcurrency,
|
||||
DownloadDir: config.TED.DownloadDir,
|
||||
CleanupAfter: config.TED.CleanupAfter,
|
||||
ScrapingInterval: config.TED.ScrapingInterval,
|
||||
},
|
||||
appLogger,
|
||||
mongoManager,
|
||||
tenderRepo,
|
||||
notify,
|
||||
)
|
||||
|
||||
// // Create context with cancellation
|
||||
// ctx, cancel := context.WithCancel(context.Background())
|
||||
// defer cancel()
|
||||
// start TED scraper job
|
||||
schedule.NewCronScheduler(appLogger, mongoManager, tenderRepo).AddJob(schedule.Job{
|
||||
Name: "TED Scraper Job",
|
||||
Func: func() {
|
||||
_ = tedScraper.Run(context.Background())
|
||||
},
|
||||
Expr: config.TED.ScrapingInterval,
|
||||
})
|
||||
|
||||
// // Set up signal handling for graceful shutdown
|
||||
// signalChan := make(chan os.Signal, 1)
|
||||
// signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
|
||||
// Create context with cancellation
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
// // Start the scraper in a goroutine
|
||||
// scraperDone := make(chan error, 1)
|
||||
// go func() {
|
||||
// appLogger.Info("Starting TED scraper", map[string]interface{}{
|
||||
// "interval": config.TED.ScrapingInterval.String(),
|
||||
// })
|
||||
// scraperDone <- tedScraper.RunPeriodicScraping(ctx)
|
||||
// }()
|
||||
// Set up signal handling for graceful shutdown
|
||||
signalChan := make(chan os.Signal, 1)
|
||||
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
|
||||
|
||||
// // Wait for shutdown signal or scraper completion
|
||||
// select {
|
||||
// case <-signalChan:
|
||||
// appLogger.Info("Received shutdown signal, stopping scraper...", map[string]interface{}{})
|
||||
// cancel()
|
||||
// Start the scraper in a goroutine
|
||||
scraperDone := make(chan error, 1)
|
||||
go func() {
|
||||
appLogger.Info("Starting TED scraper", map[string]interface{}{
|
||||
"interval": config.TED.ScrapingInterval,
|
||||
})
|
||||
scraperDone <- tedScraper.Run(ctx)
|
||||
}()
|
||||
|
||||
// // Wait for scraper to finish gracefully with timeout
|
||||
// shutdownTimeout := time.NewTimer(30 * time.Second)
|
||||
// select {
|
||||
// case <-scraperDone:
|
||||
// appLogger.Info("Scraper stopped gracefully", map[string]interface{}{})
|
||||
// case <-shutdownTimeout.C:
|
||||
// appLogger.Warn("Scraper shutdown timeout, forcing exit", map[string]interface{}{})
|
||||
// }
|
||||
// Wait for shutdown signal or scraper completion
|
||||
select {
|
||||
case <-signalChan:
|
||||
appLogger.Info("Received shutdown signal, stopping scraper...", map[string]interface{}{})
|
||||
cancel()
|
||||
|
||||
// case err := <-scraperDone:
|
||||
// if err != nil {
|
||||
// appLogger.Error("Scraper finished with error", map[string]interface{}{
|
||||
// "error": err.Error(),
|
||||
// })
|
||||
// os.Exit(1)
|
||||
// }
|
||||
// appLogger.Info("Scraper finished successfully", map[string]interface{}{})
|
||||
// }
|
||||
// }
|
||||
// Wait for scraper to finish gracefully with timeout
|
||||
shutdownTimeout := time.NewTimer(30 * time.Second)
|
||||
select {
|
||||
case <-scraperDone:
|
||||
appLogger.Info("Scraper stopped gracefully", map[string]interface{}{})
|
||||
case <-shutdownTimeout.C:
|
||||
appLogger.Warn("Scraper shutdown timeout, forcing exit", map[string]interface{}{})
|
||||
}
|
||||
|
||||
case err := <-scraperDone:
|
||||
|
||||
if err != nil {
|
||||
appLogger.Error("Scraper finished with error", map[string]interface{}{
|
||||
"error": err.Error(),
|
||||
})
|
||||
os.Exit(1)
|
||||
}
|
||||
appLogger.Info("Scraper finished successfully", map[string]interface{}{})
|
||||
}
|
||||
}
|
||||
|
||||
// Init Notification Service
|
||||
func InitNotificationService(conf config.NotificationConfig, log logger.Logger) notification.SDK {
|
||||
notificationConfig := ¬ification.Config{
|
||||
BaseURL: conf.BaseURL,
|
||||
Timeout: conf.Timeout,
|
||||
RetryAttempts: conf.RetryAttempts,
|
||||
RetryDelay: conf.RetryDelay,
|
||||
EnableLogging: conf.EnableLogging,
|
||||
UserAgent: conf.UserAgent,
|
||||
}
|
||||
|
||||
notificationSDK, err := notification.New(notificationConfig, log)
|
||||
if err != nil {
|
||||
log.Error("Failed to initialize Notification SDK", map[string]interface{}{
|
||||
"error": err.Error(),
|
||||
})
|
||||
fmt.Printf("Failed to initialize Notification SDK: %v\n", err.Error())
|
||||
}
|
||||
|
||||
log.Info("Notification SDK initialized successfully", map[string]interface{}{
|
||||
"base_url": conf.BaseURL,
|
||||
"timeout": conf.Timeout,
|
||||
"retry_attempts": conf.RetryAttempts,
|
||||
"retry_delay": conf.RetryDelay,
|
||||
"enable_logging": conf.EnableLogging,
|
||||
"user_agent": conf.UserAgent,
|
||||
})
|
||||
|
||||
return *notificationSDK
|
||||
}
|
||||
|
||||
@@ -7,19 +7,20 @@ import (
|
||||
|
||||
// Config defines the scraper application configuration
|
||||
type Config struct {
|
||||
Database config.DatabaseConfig
|
||||
Logging config.LoggingConfig
|
||||
TED ScraperConfig
|
||||
Database config.DatabaseConfig
|
||||
Logging config.LoggingConfig
|
||||
TED ScraperConfig
|
||||
Notification config.NotificationConfig
|
||||
}
|
||||
|
||||
type ScraperConfig struct {
|
||||
BaseURL string `env:"TED_BASE_URL"`
|
||||
Timeout time.Duration `env:"TED_TIMEOUT"`
|
||||
MaxRetries int `env:"TED_MAX_RETRIES"`
|
||||
RetryDelay time.Duration `env:"TED_RETRY_DELAY"`
|
||||
UserAgent string `env:"TED_USER_AGENT"`
|
||||
MaxConcurrency int `env:"TED_MAX_CONCURRENCY"`
|
||||
DownloadDir string `env:"TED_DOWNLOAD_DIR"`
|
||||
CleanupAfter time.Duration `env:"TED_CLEANUP_AFTER"`
|
||||
ScrapingInterval time.Duration `env:"TED_SCRAPING_INTERVAL"`
|
||||
BaseURL string `env:"TED_BASE_URL" envDefault:"https://ted.europa.eu"`
|
||||
Timeout time.Duration `env:"TED_TIMEOUT" envDefault:"30s"`
|
||||
MaxRetries int `env:"TED_MAX_RETRIES" envDefault:"3"`
|
||||
RetryDelay time.Duration `env:"TED_RETRY_DELAY" envDefault:"5s"`
|
||||
UserAgent string `env:"TED_USER_AGENT" envDefault:"TM-TED-Scraper/1.0"`
|
||||
MaxConcurrency int `env:"TED_MAX_CONCURRENCY" envDefault:"5"`
|
||||
DownloadDir string `env:"TED_DOWNLOAD_DIR" envDefault:"./downloads"`
|
||||
CleanupAfter time.Duration `env:"TED_CLEANUP_AFTER" envDefault:"24h"`
|
||||
ScrapingInterval string `env:"TED_SCRAPING_INTERVAL" envDefault:"* * 10 * * *"`
|
||||
}
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# TED Scraper Configuration
|
||||
database:
|
||||
mongodb:
|
||||
uri: "mongodb://localhost:27017"
|
||||
name: "tender_management"
|
||||
max_pool_size: 100
|
||||
timeout: 30s
|
||||
|
||||
logging:
|
||||
level: "info"
|
||||
format: "json"
|
||||
output: "stdout"
|
||||
file:
|
||||
path: "./logs/scraper.log"
|
||||
max_size: 100
|
||||
max_backups: 10
|
||||
max_age: 30
|
||||
compress: true
|
||||
|
||||
ted:
|
||||
base_url: "https://ted.europa.eu"
|
||||
timeout: 30s
|
||||
max_retries: 3
|
||||
retry_delay: 5s
|
||||
user_agent: "TM-TED-Scraper/1.0"
|
||||
max_concurrency: 5
|
||||
download_dir: "./downloads"
|
||||
cleanup_after: 24h
|
||||
scraping_interval: 24h
|
||||
+46
-23
@@ -1,31 +1,54 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
"tm/cmd/scraper/bootstrap"
|
||||
)
|
||||
|
||||
func main() {
|
||||
// // Load configuration
|
||||
// config, err := bootstrap.InitConfig()
|
||||
// if err != nil {
|
||||
// panic(err)
|
||||
// }
|
||||
// Load configuration
|
||||
config, err := bootstrap.InitConfig()
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
|
||||
// // Initialize logger
|
||||
// appLogger := bootstrap.InitLogger(config.Logging)
|
||||
// appLogger.Info("Starting TED scraper application", map[string]interface{}{
|
||||
// "version": "1.0.0",
|
||||
// })
|
||||
// Initialize logger
|
||||
appLogger := bootstrap.InitLogger(config.Logging)
|
||||
appLogger.Info("Starting TED scraper application", map[string]interface{}{
|
||||
"version": "1.0.0",
|
||||
})
|
||||
|
||||
// // Initialize MongoDB connection
|
||||
// mongoManager := bootstrap.InitMongoDB(config.Database.MongoDB, appLogger)
|
||||
// defer func() {
|
||||
// ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
// defer cancel()
|
||||
// if err := mongoManager.Close(ctx); err != nil {
|
||||
// appLogger.Error("Failed to close MongoDB connection", map[string]interface{}{
|
||||
// "error": err.Error(),
|
||||
// })
|
||||
// }
|
||||
// }()
|
||||
// Initialize notification service
|
||||
|
||||
// bootstrap.InitTEDScraper(*config, mongoManager, appLogger)
|
||||
// Initialize MongoDB connection
|
||||
mongoManager := bootstrap.InitMongoDB(config.Database.MongoDB, appLogger)
|
||||
defer func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := mongoManager.Close(ctx); err != nil {
|
||||
appLogger.Error("Failed to close MongoDB connection", map[string]interface{}{
|
||||
"error": err.Error(),
|
||||
})
|
||||
}
|
||||
}()
|
||||
|
||||
// appLogger.Info("TED scraper application stopped", map[string]interface{}{})
|
||||
notificationService := bootstrap.InitNotificationService(config.Notification, appLogger)
|
||||
bootstrap.InitTEDScraper(*config, mongoManager, appLogger, notificationService)
|
||||
|
||||
// Set up signal handling for graceful shutdown
|
||||
signalChan := make(chan os.Signal, 1)
|
||||
signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
|
||||
|
||||
// Wait for shutdown signal
|
||||
<-signalChan
|
||||
appLogger.Info("Received shutdown signal, stopping scheduler...", map[string]interface{}{})
|
||||
|
||||
// Stop the scheduler
|
||||
// scheduler.Stop()
|
||||
|
||||
appLogger.Info("TED scraper application stopped", map[string]interface{}{})
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user