tm_back/pkg/filestore

MongoDB GridFS File Store Service

A comprehensive file storage service for the Opplens backend using MongoDB GridFS for scalable, distributed file storage.

Overview

The file store service provides a complete solution for managing file uploads, downloads, and deletions using MongoDB's GridFS. It supports:

• Secure file uploads with validation and size limits
• Metadata storage for tracking file information
• File categorization and tagging
• List and search functionality
• Stream-based downloads for efficient memory usage
• Access control and user tracking

Key Features

✅ MongoDB GridFS Backend - Native MongoDB file storage without external dependencies ✅ File Validation - Filename, MIME type, and size validation ✅ Metadata Support - Store rich metadata with files ✅ Category & Tags - Organize files with categories and tags ✅ Stream Processing - Efficient memory usage for large files ✅ Error Handling - Comprehensive error codes and messages ✅ Logging - Detailed operation logging ✅ Thread-Safe - Concurrent operations support

Architecture

┌─────────────────────────────────────┐
│       HTTP Handler (Echo)           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│      File Handler Service           │
│  - Request validation               │
│  - Response formatting              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│  GridFS Service Interface           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│   MongoDB GridFS Implementation     │
│  - Bucket operations                │
│  - Metadata management              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│     MongoDB Database                │
│  - fs.files collection              │
│  - fs.chunks collection             │
└─────────────────────────────────────┘

Installation

The service is automatically initialized in the bootstrap:

// In cmd/web/bootstrap/bootstrap.go
fileStoreService, err := filestore.NewGridFSService(mongoManager, logger)
if err != nil {
    logger.Fatal("Failed to initialize file store service", map[string]interface{}{
        "error": err.Error(),
    })
}

API Endpoints

Upload File

POST /api/v1/files/upload

Upload a file to GridFS storage.

Request:

• Content-Type: multipart/form-data
• Parameters:
  • file (required): The file to upload
  • category (optional): File category for organization
  • tags (optional, array): Tags for the file
  • description (optional): File description

Response:

{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "message": "File uploaded successfully"
}

Error Responses:

• 400 Bad Request - No file provided
• 413 Payload Too Large - File exceeds size limit
• 500 Internal Server Error - Upload failed

Download File

GET /api/v1/files/{file_id}/download

Download a file by its ID.

Parameters:

• file_id (path, required): The file ID

Response:

• Returns file content with appropriate MIME type and Content-Disposition header

Error Responses:

• 404 Not Found - File not found
• 500 Internal Server Error - Download failed

Get File Info

GET /api/v1/files/{file_id}/info

Retrieve metadata for a file.

Parameters:

• file_id (path, required): The file ID

Response:

{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "uploaded_by": "user_123",
  "uploaded_at": "2024-04-19T10:30:00Z",
  "category": "profile_images",
  "tags": ["user", "profile"],
  "description": "User profile picture"
}

List Files

GET /api/v1/files

List files with optional filtering.

Query Parameters:

• category (optional): Filter by category
• tags (optional, array): Filter by tags
• limit (optional, default 50, max 1000): Number of files to return
• skip (optional, default 0): Number of files to skip

Response:

{
  "files": [
    {
      "file_id": "507f1f77bcf86cd799439011",
      "filename": "profile.jpg",
      "content_type": "image/jpeg",
      "size": 102400,
      "uploaded_by": "user_123",
      "uploaded_at": "2024-04-19T10:30:00Z",
      "category": "profile_images",
      "tags": ["user", "profile"]
    }
  ],
  "total": 150,
  "limit": 50,
  "skip": 0
}

Delete File

DELETE /api/v1/files/{file_id}

Delete a file by its ID.

Parameters:

• file_id (path, required): The file ID

Response:

{
  "message": "File deleted successfully"
}

Error Responses:

• 404 Not Found - File not found
• 500 Internal Server Error - Deletion failed

Usage Examples

Upload Profile Image

curl -X POST \
  -F "file=@profile.jpg" \
  -F "category=profile_images" \
  -F "tags=user" \
  -F "tags=profile" \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/upload

Download File

curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \
  -o downloaded_file.jpg

Get File Information

curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info

List User Profile Images

curl -X GET \
  -H "Authorization: Bearer <token>" \
  "http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20"

File Validation

The service includes built-in validation for file uploads:

Default Validators

Standard File Validator

• Supported MIME types: Images, PDFs, Office documents, compressed files
• Maximum size: 100 MB
• Filename validation and sanitization

Profile Image Validator

• Supported MIME types: JPEG, PNG, GIF, WebP
• Maximum size: 5 MB
• Extension validation

Custom Validation

Create a custom validator:

validator := filestore.NewFileValidator(
    10 * 1024 * 1024, // 10 MB max
    map[string]bool{
        "application/pdf": true,
        "image/jpeg":      true,
        "image/png":       true,
    },
)

// Validate file
if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil {
    // Handle validation error
}

Metadata Storage

Files are stored with rich metadata:

{
  "filename": "profile.jpg",
  "uploadDate": "2024-04-19T10:30:00Z",
  "metadata": {
    "uploaded_by": "user_123",
    "category": "profile_images",
    "tags": ["user", "profile"],
    "description": "User profile picture",
    "content_type": "image/jpeg",
    "extra": {
      "user_id": "user_123",
      "department": "marketing"
    }
  }
}

Error Handling

The service uses a comprehensive error handling system:

Error Codes

• FILE_NOT_FOUND - File does not exist
• UPLOAD_FAILED - Upload operation failed
• DOWNLOAD_FAILED - Download operation failed
• DELETE_FAILED - Delete operation failed
• INVALID_INPUT - Invalid input parameters
• FILE_TOO_LARGE - File exceeds size limit
• UNSUPPORTED_FILE_TYPE - MIME type not supported
• DUPLICATE_FILE - File already exists
• ACCESS_DENIED - User does not have access

Error Response Format

{
  "error": "File is too large",
  "code": "FILE_TOO_LARGE",
  "message": "Maximum file size is 100 MB"
}

Database Schema

fs.files Collection

{
  _id: ObjectId,
  filename: String,
  uploadDate: Date,
  chunkSize: Number,
  length: Number,
  md5: String,
  metadata: {
    uploaded_by: String,
    category: String,
    tags: [String],
    description: String,
    content_type: String,
    extra: {
      user_id: String,
      // ... additional fields
    }
  }
}

fs.chunks Collection

{
  _id: ObjectId,
  files_id: ObjectId,
  n: Number,
  data: BinData
}

Integration with User Service

To integrate file uploads with user profiles:

1. Update User Entity - Add FileID field:

type User struct {
    // ... existing fields
    ProfileImageFileID *string `bson:"profile_image_file_id"`
}

2. Add Upload Endpoint - In user handler:

func (h *Handler) UploadProfileImage(c echo.Context) error {
    // Get user ID from context
    userID := c.Get("user_id").(string)
    
    // Get file
    file, err := c.FormFile("profile_image")
    // ... upload logic
    
    // Update user with file ID
    // ... update user
}

3. Add Download Endpoint - Serve user's profile image:

func (h *Handler) GetProfileImage(c echo.Context) error {
    userID := c.Param("user_id")
    
    // Get user
    user, err := h.service.GetUserByID(userID)
    
    // Download file
    reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID)
    // ... stream response
}

Performance Considerations

1. Chunk Size - GridFS default is 255 KB. For large files, consider increasing this.
2. Indexing - Create indexes on frequently queried metadata:

db.fs.files.createIndex({ "metadata.category": 1 })
db.fs.files.createIndex({ "metadata.tags": 1 })
db.fs.files.createIndex({ "metadata.uploaded_by": 1 })

3. Cleanup - Implement regular cleanup for unused files:

// Delete file and associated chunks
func (s *GridFSService) Delete(fileID string) error {
    // Automatically handled by MongoDB
}

Security Considerations

1. File Type Validation - Validate MIME types and extensions
2. Filename Sanitization - Remove dangerous characters:

sanitized := filestore.SanitizeFilename(filename)

3. Size Limits - Enforce maximum file sizes
4. Access Control - Implement proper authorization checks
5. Malware Scanning - Consider integrating antivirus scanning
6. Rate Limiting - Implement rate limiting on upload endpoints

Testing

Unit tests are included for all operations:

go test ./pkg/filestore -v

Test coverage includes:

• Upload operations
• Download operations
• Delete operations
• File validation
• Error handling
• Metadata storage and retrieval

Troubleshooting

MongoDB Connection Failed

Error: MongoDB connection not available
Solution: Ensure MongoDB is running and configured correctly

File Not Found

Error: file not found
Solution: Verify the file ID is correct

File Too Large

Error: file size exceeds maximum limit
Solution: Check MaxSize configuration and file size

Invalid MIME Type

Error: unsupported file type
Solution: Verify the file type is in the allowed MIME types list

Configuration

Configure file store limits in bootstrap.go:

// Maximum file sizes
const (
    MaxUploadSize         = 100 * 1024 * 1024 // 100 MB
    MaxProfileImageSize   = 5 * 1024 * 1024   // 5 MB
)

// Supported MIME types
var AllowedImageTypes = map[string]bool{
    "image/jpeg":  true,
    "image/png":   true,
    "image/gif":   true,
    "image/webp":  true,
}

Contributing

When extending the file store service:

1. Maintain backward compatibility
2. Add appropriate error handling
3. Update logging for operations
4. Include unit tests for new features
5. Update documentation

License

This file store service is part of the Opplens backend system.