tm_back/LOGGING_UPGRADE.md

Logging System Upgrade: Logrus → Zap + Lumberjack

Overview

The tender management system has been upgraded from Logrus to Zap with Lumberjack for high-performance structured logging with automatic log rotation.

🚀 Improvements

Performance

• 10x faster logging performance with Zap
• Zero-allocation logging in hot paths
• Structured field processing optimized for speed

Log Rotation

• Automatic log rotation based on file size, age, and backup count
• Compression of rotated log files to save disk space
• Configurable retention policies

Flexibility

• Multi-output logging (file + stdout simultaneously)
• Multiple formats (JSON, console, text)
• Environment-specific configuration

📁 Configuration

YAML Configuration

logging:
  level: "info"              # debug, info, warn, error, fatal
  format: "json"             # json, console, text
  output: "file"             # stdout, stderr, file
  file:
    path: "./logs/app.log"   # Path to log file
    max_size: 100            # Max size in MB before rotation
    max_backups: 5           # Number of backup files to keep
    max_age: 30              # Max age in days to keep log files
    compress: true           # Compress rotated files

Environment Variables

export TM_LOGGING_LEVEL="info"
export TM_LOGGING_OUTPUT="file"
export TM_LOGGING_FILE_PATH="./logs/app.log"
export TM_LOGGING_FILE_MAX_SIZE=100
export TM_LOGGING_FILE_MAX_BACKUPS=5
export TM_LOGGING_FILE_MAX_AGE=30
export TM_LOGGING_FILE_COMPRESS=true

🔧 Usage

Same Interface

The logger interface remains the same - no code changes required in existing handlers and services:

// All existing code continues to work
log.Info("User logged in", map[string]interface{}{
    "user_id": user.ID.String(),
    "email":   user.Email,
})

// Structured logging with fields
logger := log.WithFields(map[string]interface{}{
    "component": "auth",
    "request_id": requestID,
})
logger.Error("Authentication failed", map[string]interface{}{
    "error": err.Error(),
})

New Features

// Flush logs before shutdown
logger.Sync()

// Or use the global function
import "tm/pkg/logger"
logger.Sync()

📊 Log Rotation Behavior

File Naming Pattern

logs/
├── app.log              # Current log file
├── app.log.1            # Most recent backup
├── app.log.2            # Older backup
├── app.log.3.gz         # Compressed older backup
├── app.log.4.gz         # Compressed older backup
└── app.log.5.gz         # Oldest backup (will be deleted on next rotation)

Rotation Triggers

• Size-based: When app.log reaches 100MB
• Time-based: Daily rotation (if configured)
• Manual: Can be triggered programmatically

Cleanup

• Keeps maximum 5 backup files
• Deletes files older than 30 days
• Compresses old files to save space

🛠️ Development Commands

Log Management

# View live logs
make logs

# Follow logs in real-time
make logs-live

# Clean all log files
make logs-clean

# Clean everything including logs
make clean-all

Log File Locations

• Development: ./logs/app.log
• Docker: /logs/app.log (mounted volume)
• Production: Configurable via environment variables

🐳 Docker Integration

Volume Mounting

# docker-compose.yml
volumes:
  - ./logs:/logs  # Host logs directory mounted to container

Log Persistence

• Logs persist on the host machine
• Rotation works across container restarts
• Easy log analysis and monitoring

🔍 Log Format Examples

JSON Format (Default)

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "info",
  "caller": "handler/auth_handler.go:45",
  "msg": "User logged in successfully",
  "user_id": "123e4567-e89b-12d3-a456-426614174000",
  "email": "user@example.com",
  "request_id": "req_abc123"
}

Console Format (Development)

2024-01-15T10:30:45.123Z  INFO  handler/auth_handler.go:45  User logged in successfully
  user_id=123e4567-e89b-12d3-a456-426614174000
  email=user@example.com
  request_id=req_abc123

🚨 Migration Notes

Breaking Changes

• None - The logger interface remains compatible

Dependencies Updated

// Removed
github.com/sirupsen/logrus v1.9.3

// Added
go.uber.org/zap v1.26.0
gopkg.in/natefinch/lumberjack.v2 v2.2.1

Configuration Changes

• Added file section to logging configuration
• output: "file" now enables file logging with rotation

📈 Performance Comparison

Metric	Logrus	Zap	Improvement
Throughput	~100k/sec	~1M+/sec	10x faster
Allocations	High	Zero (hot path)	Much lower memory
CPU Usage	Higher	Lower	Better efficiency
Structured Fields	Reflection-based	Type-safe	Faster processing

🔧 Advanced Configuration

Custom Log Levels

// Set different levels for different components
config := &infrastructure.LoggingConfig{
    Level: "debug",  // Global level
    // Component-specific levels can be added later
}

Multiple Outputs

// Logs go to both file and stdout when output is "file"
// Perfect for development - see logs in terminal AND save to file

Programmatic Control

// Flush logs before application shutdown
defer logger.Cleanup()

// Or manually sync
if err := logger.Sync(); err != nil {
    // Handle sync error
}

🎯 Best Practices

1. Use structured fields for better log analysis
2. Set appropriate log levels (info for production, debug for development)
3. Monitor log file sizes in production
4. Use log aggregation tools like ELK stack for production
5. Flush logs before application shutdown

📚 Resources

• Zap Documentation
• Lumberjack Documentation
• Go Logging Best Practices