tm_back/cmd/scraper/README.md

# TED XML Scraper

A comprehensive TED (Tenders Electronic Daily) XML scraper that downloads and parses daily calendar XML files, converting them to tender entities and storing them in MongoDB.

## Features

- **Daily Scheduled Scraping**: Runs automatically at 10:00 AM daily using cron jobs
- **TED XML Format Support**: Handles eForms XML, Contract Notices, and TED Calendar formats
- **XML Parser Integration**: Uses the custom `xmlparser` package with generic type support
- **Robust Error Handling**: Comprehensive error handling, retry mechanisms, and logging
- **Data Conversion**: Converts TED XML structures to internal tender entity models
- **MongoDB Integration**: Stores processed data with proper indexing and validation
- **Progress Tracking**: Job tracking, status monitoring, and statistics
- **Configurable**: Environment variables and YAML configuration support

## Architecture

```
cmd/scraper/
├── main.go                 # Application entry point
├── config.yaml            # Default configuration
├── bootstrap/
│   ├── bootstrap.go        # Application initialization
│   └── config.go          # Configuration structures
└── README.md              # This file

internal/scraper/
├── service.go             # Core scraping service logic
├── repository.go          # MongoDB operations for scraping jobs
└── scheduler.go           # Cron job scheduling

pkg/xmlparser/ted/
├── models.go              # TED XML data structures
└── converter.go           # XML to entity conversion
```

## Installation

1. **Prerequisites**:
   - Go 1.21+
   - MongoDB 4.4+

2. **Dependencies**:
   ```bash
   cd cmd/scraper
   go mod tidy
   ```

3. **Configuration**:
   Create a `config.yaml` file or set environment variables:
   ```yaml
   database:
     mongodb:
       uri: "mongodb://localhost:27017"
       name: "tender_management"
   
   ted:
     base_url: "https://ted.europa.eu"
     timeout: 30s
     max_retries: 3
   ```

## Usage

### Running the Scraper

```bash
# Run with default configuration
go run main.go

# Run with custom config file
CONFIG_PATH=./custom-config.yaml go run main.go

# Run with environment variables
TED_BASE_URL=https://ted.europa.eu go run main.go
```

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `TED_BASE_URL` | `https://ted.europa.eu` | TED website base URL |
| `TED_TIMEOUT` | `30s` | HTTP request timeout |
| `TED_MAX_RETRIES` | `3` | Maximum retry attempts |
| `TED_RETRY_DELAY` | `5s` | Delay between retries |
| `TED_USER_AGENT` | `TM-TED-Scraper/1.0` | HTTP User-Agent header |
| `TED_MAX_CONCURRENCY` | `5` | Maximum concurrent downloads |
| `TED_DOWNLOAD_DIR` | `./downloads` | Download directory |
| `TED_CLEANUP_AFTER` | `24h` | File cleanup interval |

### Manual Trigger

To manually trigger a scraping job outside the schedule:

```bash
# Using the scheduler API (if implemented)
curl -X POST http://localhost:8080/api/scraper/trigger

# Or restart the application to trigger immediate run
```

## How It Works

### 1. Daily Calendar Download

The scraper downloads TED daily packages using this URL pattern:
```
https://ted.europa.eu/packages/daily/{yyyynnnnn}
```

Where `{yyyynnnnn}` is the year + 5-digit day of year (e.g., `202500001` for January 1st, 2025).

### 2. XML Parsing

The scraper supports multiple TED XML formats:

- **Contract Notice**: Individual tender announcements
- **Tender Calendar**: Daily collections of multiple notices
- **eForms**: New European standard format
- **Legacy TED Schema**: Older format support

### 3. Data Conversion

XML data is converted to internal tender entities with:

- **Field Mapping**: TED XML fields → Tender entity fields
- **Data Validation**: Required field validation and sanitization
- **Unix Timestamps**: All dates converted to Unix timestamps
- **Text Normalization**: Clean up whitespace and formatting
- **URL Generation**: Generate TED website URLs for notices

### 4. Storage

Processed data is stored in MongoDB with:

- **Indexes**: Optimized for search and filtering
- **Deduplication**: Based on contract notice ID
- **Audit Trail**: Creation and update timestamps
- **Error Tracking**: Processing errors and warnings

## TED XML Format Analysis

### Contract Notice Structure

```xml
<ContractNotice xmlns="urn:oasis:names:specification:ubl:schema:xsd:ContractNotice-2">
    <ID>123456-2025</ID>
    <ContractFolderID>folder-123</ContractFolderID>
    <IssueDate>2025-01-09+02:00</IssueDate>
    <NoticeTypeCode>cn-standard</NoticeTypeCode>
    <NoticeLanguageCode>ENG</NoticeLanguageCode>
    <!-- ... procurement project, buyer information, etc. -->
</ContractNotice>
```

### Key XML Elements Mapped

| XML Element | Tender Field | Description |
|-------------|--------------|-------------|
| `ID` | `contract_notice_id` | Unique notice identifier |
| `ContractFolderID` | `contract_folder_id` | Contract folder reference |
| `IssueDate` | `issue_date` | Notice publication date |
| `ProcurementProject/Name` | `title` | Tender title |
| `ProcurementProject/Description` | `description` | Tender description |
| `ContractingParty` | `buyer_organization` | Buyer/contracting authority |
| `TenderSubmissionDeadlinePeriod` | `tender_deadline` | Submission deadline |

## Scheduling

### Cron Schedule

- **Main Job**: `0 0 10 * * *` (10:00 AM daily UTC)
- **Cleanup Job**: `0 0 2 * * *` (2:00 AM daily UTC)

### Job Configuration

```go
schedulerConfig := scraper.SchedulerConfig{
    CronExpression: "0 0 10 * * *",    // 10:00 AM daily (seconds, minutes, hours, day, month, weekday)
    Timezone:       "UTC",
    JobTimeout:     2 * time.Hour,
    RetryAttempts:  3,
    RetryDelay:     30 * time.Minute,
}
```

### Retry Logic

- **Automatic Retries**: Up to 3 attempts on failure
- **Exponential Backoff**: 30-minute delay between retries
- **Context Timeout**: 2-hour maximum per job
- **Graceful Shutdown**: Waits for running jobs to complete

## Monitoring

### Job Status

Track scraping job progress and status:

```json
{
  "id": "job-123",
  "type": "ted_scraper",
  "status": "running",
  "progress": {
    "processed_count": 150,
    "total_count": 500,
    "success_count": 145,
    "error_count": 5,
    "percentage": 30.0,
    "current_step": "processing_tenders"
  }
}
```

### Statistics

```json
{
  "period": "Last 7 days",
  "total_jobs": 7,
  "completed_jobs": 6,
  "failed_jobs": 1,
  "total_processed": 3500,
  "total_created": 2100,
  "total_updated": 1200,
  "total_errors": 200
}
```

## Error Handling

### Error Types

1. **Download Errors**: Network issues, server errors
2. **Parsing Errors**: Invalid XML, schema mismatches
3. **Validation Errors**: Missing required fields
4. **Database Errors**: Connection issues, write failures

### Error Recovery

- **Retry Mechanism**: Automatic retries with backoff
- **Partial Success**: Continue processing valid notices
- **Error Logging**: Detailed error tracking and reporting
- **Graceful Degradation**: Skip problematic entries

### Example Error Response

```json
{
  "job_id": "job-123",
  "status": "completed",
  "results": {
    "total_processed": 100,
    "total_created": 85,
    "total_errors": 15,
    "processing_errors": [
      "notice 23: validation failed: title is required",
      "notice 45: failed to parse deadline: invalid format"
    ]
  }
}
```

## Best Practices

### Performance

- **Batch Processing**: Process tenders in configurable batches (default: 10)
- **Connection Pooling**: MongoDB connection management
- **Memory Management**: Stream large XML files
- **Concurrent Downloads**: Configurable concurrency limits

### Data Quality

- **Validation**: Comprehensive field validation
- **Sanitization**: Text cleaning and normalization
- **Deduplication**: Prevent duplicate tender entries
- **Audit Trail**: Track all changes and updates

### Reliability

- **Health Checks**: Monitor service health
- **Graceful Shutdown**: Clean exit on signals
- **Resource Cleanup**: Automatic file cleanup
- **State Persistence**: Track scraping state across restarts

## Development

### Adding New XML Formats

1. **Extend TED Models**: Add new XML structures in `pkg/xmlparser/ted/models.go`
2. **Update Converter**: Add conversion logic in `pkg/xmlparser/ted/converter.go`
3. **Add Parser Support**: Register new parsers in service
4. **Test Thoroughly**: Add comprehensive test cases

### Custom Validation

```go
// Add custom validators in converter.go
func (c *Converter) ValidateCustomField(value string) error {
    if len(value) > 1000 {
        return fmt.Errorf("field too long: %d characters", len(value))
    }
    return nil
}
```

### Extending Scheduling

```go
// Add new scheduled jobs in scheduler.go
_, err = s.cron.AddFunc("0 */6 * * *", s.runStatisticsJob) // Every 6 hours
```

## Troubleshooting

### Common Issues

1. **Connection Errors**: Check MongoDB connection and credentials
2. **Parsing Failures**: Verify TED XML format changes
3. **Schedule Issues**: Check timezone and cron expression
4. **Memory Usage**: Monitor for large XML files

### Debug Mode

Enable debug logging:

```yaml
logging:
  level: "debug"
```

### Manual Testing

```bash
# Test XML parsing
go run main.go --test-xml ./sample.xml

# Test specific date
go run main.go --test-date 2025-01-09

# Validate configuration
go run main.go --validate-config
```

## License

This project is part of the Tender Management system and follows the same licensing terms.