tm_back/cmd/scraper

TED XML Scraper

A comprehensive TED (Tenders Electronic Daily) XML scraper that downloads and parses daily calendar XML files, converting them to tender entities and storing them in MongoDB.

Features

• Daily Scheduled Scraping: Runs automatically at 10:00 AM daily using cron jobs
• TED XML Format Support: Handles eForms XML, Contract Notices, and TED Calendar formats
• XML Parser Integration: Uses the custom xmlparser package with generic type support
• Robust Error Handling: Comprehensive error handling, retry mechanisms, and logging
• Data Conversion: Converts TED XML structures to internal tender entity models
• MongoDB Integration: Stores processed data with proper indexing and validation
• Progress Tracking: Job tracking, status monitoring, and statistics
• Configurable: Environment variables and YAML configuration support

Architecture

cmd/scraper/
├── main.go                 # Application entry point
├── config.yaml            # Default configuration
├── bootstrap/
│   ├── bootstrap.go        # Application initialization
│   └── config.go          # Configuration structures
└── README.md              # This file

internal/scraper/
├── service.go             # Core scraping service logic
├── repository.go          # MongoDB operations for scraping jobs
└── scheduler.go           # Cron job scheduling

pkg/xmlparser/ted/
├── models.go              # TED XML data structures
└── converter.go           # XML to entity conversion

Installation

1. Prerequisites:

   • Go 1.21+
   • MongoDB 4.4+

2. Dependencies:

   cd cmd/scraper
   go mod tidy
   

3. Configuration: Create a config.yaml file or set environment variables:

   database:
     mongodb:
       uri: "mongodb://localhost:27017"
       name: "tender_management"
   
   ted:
     base_url: "https://ted.europa.eu"
     timeout: 30s
     max_retries: 3
   

Usage

Running the Scraper

# Run with default configuration
go run main.go

# Run with custom config file
CONFIG_PATH=./custom-config.yaml go run main.go

# Run with environment variables
TED_BASE_URL=https://ted.europa.eu go run main.go

Environment Variables

Variable	Default	Description
TED_BASE_URL	https://ted.europa.eu	TED website base URL
TED_TIMEOUT	30s	HTTP request timeout
TED_MAX_RETRIES	3	Maximum retry attempts
TED_RETRY_DELAY	5s	Delay between retries
TED_USER_AGENT	TM-TED-Scraper/1.0	HTTP User-Agent header
TED_MAX_CONCURRENCY	5	Maximum concurrent downloads
TED_DOWNLOAD_DIR	./downloads	Download directory
TED_CLEANUP_AFTER	24h	File cleanup interval

Manual Trigger

To manually trigger a scraping job outside the schedule:

# Using the scheduler API (if implemented)
curl -X POST http://localhost:8080/api/scraper/trigger

# Or restart the application to trigger immediate run

How It Works

1. Daily Calendar Download

The scraper downloads TED daily packages using this URL pattern:

https://ted.europa.eu/packages/daily/{yyyynnnnn}

Where {yyyynnnnn} is the year + 5-digit day of year (e.g., 202500001 for January 1st, 2025).

2. XML Parsing

The scraper supports multiple TED XML formats:

• Contract Notice: Individual tender announcements
• Tender Calendar: Daily collections of multiple notices
• eForms: New European standard format
• Legacy TED Schema: Older format support

3. Data Conversion

XML data is converted to internal tender entities with:

• Field Mapping: TED XML fields → Tender entity fields
• Data Validation: Required field validation and sanitization
• Unix Timestamps: All dates converted to Unix timestamps
• Text Normalization: Clean up whitespace and formatting
• URL Generation: Generate TED website URLs for notices

4. Storage

Processed data is stored in MongoDB with:

• Indexes: Optimized for search and filtering
• Deduplication: Based on contract notice ID
• Audit Trail: Creation and update timestamps
• Error Tracking: Processing errors and warnings

TED XML Format Analysis

Contract Notice Structure

<ContractNotice xmlns="urn:oasis:names:specification:ubl:schema:xsd:ContractNotice-2">
    <ID>123456-2025</ID>
    <ContractFolderID>folder-123</ContractFolderID>
    <IssueDate>2025-01-09+02:00</IssueDate>
    <NoticeTypeCode>cn-standard</NoticeTypeCode>
    <NoticeLanguageCode>ENG</NoticeLanguageCode>
    <!-- ... procurement project, buyer information, etc. -->
</ContractNotice>

Key XML Elements Mapped

XML Element	Tender Field	Description
ID	contract_notice_id	Unique notice identifier
ContractFolderID	contract_folder_id	Contract folder reference
IssueDate	issue_date	Notice publication date
ProcurementProject/Name	title	Tender title
ProcurementProject/Description	description	Tender description
ContractingParty	buyer_organization	Buyer/contracting authority
TenderSubmissionDeadlinePeriod	tender_deadline	Submission deadline

Scheduling

Cron Schedule

• Main Job: 0 0 10 * * * (10:00 AM daily UTC)
• Cleanup Job: 0 0 2 * * * (2:00 AM daily UTC)

Job Configuration

schedulerConfig := scraper.SchedulerConfig{
    CronExpression: "0 0 10 * * *",    // 10:00 AM daily (seconds, minutes, hours, day, month, weekday)
    Timezone:       "UTC",
    JobTimeout:     2 * time.Hour,
    RetryAttempts:  3,
    RetryDelay:     30 * time.Minute,
}

Retry Logic

• Automatic Retries: Up to 3 attempts on failure
• Exponential Backoff: 30-minute delay between retries
• Context Timeout: 2-hour maximum per job
• Graceful Shutdown: Waits for running jobs to complete

Monitoring

Job Status

Track scraping job progress and status:

{
  "id": "job-123",
  "type": "ted_scraper",
  "status": "running",
  "progress": {
    "processed_count": 150,
    "total_count": 500,
    "success_count": 145,
    "error_count": 5,
    "percentage": 30.0,
    "current_step": "processing_tenders"
  }
}

Statistics

{
  "period": "Last 7 days",
  "total_jobs": 7,
  "completed_jobs": 6,
  "failed_jobs": 1,
  "total_processed": 3500,
  "total_created": 2100,
  "total_updated": 1200,
  "total_errors": 200
}

Error Handling

Error Types

1. Download Errors: Network issues, server errors
2. Parsing Errors: Invalid XML, schema mismatches
3. Validation Errors: Missing required fields
4. Database Errors: Connection issues, write failures

Error Recovery

• Retry Mechanism: Automatic retries with backoff
• Partial Success: Continue processing valid notices
• Error Logging: Detailed error tracking and reporting
• Graceful Degradation: Skip problematic entries

Example Error Response

{
  "job_id": "job-123",
  "status": "completed",
  "results": {
    "total_processed": 100,
    "total_created": 85,
    "total_errors": 15,
    "processing_errors": [
      "notice 23: validation failed: title is required",
      "notice 45: failed to parse deadline: invalid format"
    ]
  }
}

Best Practices

Performance

• Batch Processing: Process tenders in configurable batches (default: 10)
• Connection Pooling: MongoDB connection management
• Memory Management: Stream large XML files
• Concurrent Downloads: Configurable concurrency limits

Data Quality

• Validation: Comprehensive field validation
• Sanitization: Text cleaning and normalization
• Deduplication: Prevent duplicate tender entries
• Audit Trail: Track all changes and updates

Reliability

• Health Checks: Monitor service health
• Graceful Shutdown: Clean exit on signals
• Resource Cleanup: Automatic file cleanup
• State Persistence: Track scraping state across restarts

Development

Adding New XML Formats

1. Extend TED Models: Add new XML structures in pkg/xmlparser/ted/models.go
2. Update Converter: Add conversion logic in pkg/xmlparser/ted/converter.go
3. Add Parser Support: Register new parsers in service
4. Test Thoroughly: Add comprehensive test cases

Custom Validation

// Add custom validators in converter.go
func (c *Converter) ValidateCustomField(value string) error {
    if len(value) > 1000 {
        return fmt.Errorf("field too long: %d characters", len(value))
    }
    return nil
}

Extending Scheduling

// Add new scheduled jobs in scheduler.go
_, err = s.cron.AddFunc("0 */6 * * *", s.runStatisticsJob) // Every 6 hours

Troubleshooting

Common Issues

1. Connection Errors: Check MongoDB connection and credentials
2. Parsing Failures: Verify TED XML format changes
3. Schedule Issues: Check timezone and cron expression
4. Memory Usage: Monitor for large XML files

Debug Mode

Enable debug logging:

logging:
  level: "debug"

Manual Testing

# Test XML parsing
go run main.go --test-xml ./sample.xml

# Test specific date
go run main.go --test-date 2025-01-09

# Validate configuration
go run main.go --validate-config

License

This project is part of the Tender Management system and follows the same licensing terms.