✅ MongoDB GridFS Backend - Native MongoDB file storage without external dependencies ✅ File Validation - Filename, MIME type, and size validation ✅ Metadata Support - Store rich metadata with files ✅ Category & Tags - Organize files with categories and tags ✅ Stream Processing - Efficient memory usage for large files ✅ Error Handling - Comprehensive error codes and messages ✅ Logging - Detailed operation logging ✅ Thread-Safe - Concurrent operations support

Architecture

┌─────────────────────────────────────┐
│       HTTP Handler (Echo)           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│      File Handler Service           │
│  - Request validation               │
│  - Response formatting              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│  GridFS Service Interface           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│   MongoDB GridFS Implementation     │
│  - Bucket operations                │
│  - Metadata management              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│     MongoDB Database                │
│  - fs.files collection              │
│  - fs.chunks collection             │
└─────────────────────────────────────┘

Installation

The service is automatically initialized in the bootstrap:

// In cmd/web/bootstrap/bootstrap.go
fileStoreService, err := filestore.NewGridFSService(mongoManager, logger)
if err != nil {
    logger.Fatal("Failed to initialize file store service", map[string]interface{}{
        "error": err.Error(),
    })
}

API Endpoints

Upload File

POST /api/v1/files/upload

Upload a file to GridFS storage.

Request:

Content-Type: multipart/form-data
Parameters:
- file (required): The file to upload
- category (optional): File category for organization
- tags (optional, array): Tags for the file
- description (optional): File description

Response:

{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "message": "File uploaded successfully"
}

Error Responses:

400 Bad Request - No file provided
413 Payload Too Large - File exceeds size limit
500 Internal Server Error - Upload failed

Download File

GET /api/v1/files/{file_id}/download

Download a file by its ID.

Parameters:

file_id (path, required): The file ID

Response:

Returns file content with appropriate MIME type and Content-Disposition header

Error Responses:

404 Not Found - File not found
500 Internal Server Error - Download failed

Get File Info

GET /api/v1/files/{file_id}/info

Retrieve metadata for a file.

Parameters:

file_id (path, required): The file ID

Response:

{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "uploaded_by": "user_123",
  "uploaded_at": "2024-04-19T10:30:00Z",
  "category": "profile_images",
  "tags": ["user", "profile"],
  "description": "User profile picture"
}

List Files

GET /api/v1/files

List files with optional filtering.

Query Parameters:

category (optional): Filter by category
tags (optional, array): Filter by tags
limit (optional, default 50, max 1000): Number of files to return
skip (optional, default 0): Number of files to skip

Response:

{
  "files": [
    {
      "file_id": "507f1f77bcf86cd799439011",
      "filename": "profile.jpg",
      "content_type": "image/jpeg",
      "size": 102400,
      "uploaded_by": "user_123",
      "uploaded_at": "2024-04-19T10:30:00Z",
      "category": "profile_images",
      "tags": ["user", "profile"]
    }
  ],
  "total": 150,
  "limit": 50,
  "skip": 0
}

Delete File

DELETE /api/v1/files/{file_id}

Delete a file by its ID.

Parameters:

file_id (path, required): The file ID

Response:

{
  "message": "File deleted successfully"
}

Error Responses:

404 Not Found - File not found
500 Internal Server Error - Deletion failed

Usage Examples

Upload Profile Image

curl -X POST \
  -F "file=@profile.jpg" \
  -F "category=profile_images" \
  -F "tags=user" \
  -F "tags=profile" \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/upload

Download File

curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \
  -o downloaded_file.jpg

Get File Information

curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info

List User Profile Images

curl -X GET \
  -H "Authorization: Bearer <token>" \
  "http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20"

File Validation

The service includes built-in validation for file uploads:

Default Validators

Standard File Validator

Supported MIME types: Images, PDFs, Office documents, compressed files
Maximum size: 100 MB
Filename validation and sanitization

Profile Image Validator

Supported MIME types: JPEG, PNG, GIF, WebP
Maximum size: 5 MB
Extension validation

Custom Validation

Create a custom validator:

validator := filestore.NewFileValidator(
    10 * 1024 * 1024, // 10 MB max
    map[string]bool{
        "application/pdf": true,
        "image/jpeg":      true,
        "image/png":       true,
    },
)

// Validate file
if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil {
    // Handle validation error
}

Metadata Storage

Files are stored with rich metadata:

{
  "filename": "profile.jpg",
  "uploadDate": "2024-04-19T10:30:00Z",
  "metadata": {
    "uploaded_by": "user_123",
    "category": "profile_images",
    "tags": ["user", "profile"],
    "description": "User profile picture",
    "content_type": "image/jpeg",
    "extra": {
      "user_id": "user_123",
      "department": "marketing"
    }
  }
}

Error Handling

The service uses a comprehensive error handling system:

Error Codes

FILE_NOT_FOUND - File does not exist
UPLOAD_FAILED - Upload operation failed
DOWNLOAD_FAILED - Download operation failed
DELETE_FAILED - Delete operation failed
INVALID_INPUT - Invalid input parameters
FILE_TOO_LARGE - File exceeds size limit
UNSUPPORTED_FILE_TYPE - MIME type not supported
DUPLICATE_FILE - File already exists
ACCESS_DENIED - User does not have access

Error Response Format

{
  "error": "File is too large",
  "code": "FILE_TOO_LARGE",
  "message": "Maximum file size is 100 MB"
}

Database Schema

fs.files Collection

{
  _id: ObjectId,
  filename: String,
  uploadDate: Date,
  chunkSize: Number,
  length: Number,
  md5: String,
  metadata: {
    uploaded_by: String,
    category: String,
    tags: [String],
    description: String,
    content_type: String,
    extra: {
      user_id: String,
      // ... additional fields
    }
  }
}

fs.chunks Collection

{
  _id: ObjectId,
  files_id: ObjectId,
  n: Number,
  data: BinData
}

Integration with User Service

To integrate file uploads with user profiles:

Update User Entity - Add FileID field:

type User struct {
    // ... existing fields
    ProfileImageFileID *string `bson:"profile_image_file_id"`
}

Add Upload Endpoint - In user handler:

func (h *Handler) UploadProfileImage(c echo.Context) error {
    // Get user ID from context
    userID := c.Get("user_id").(string)
    
    // Get file
    file, err := c.FormFile("profile_image")
    // ... upload logic
    
    // Update user with file ID
    // ... update user
}

Add Download Endpoint - Serve user's profile image:

func (h *Handler) GetProfileImage(c echo.Context) error {
    userID := c.Param("user_id")
    
    // Get user
    user, err := h.service.GetUserByID(userID)
    
    // Download file
    reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID)
    // ... stream response
}

Performance Considerations

Chunk Size - GridFS default is 255 KB. For large files, consider increasing this.
Indexing - Create indexes on frequently queried metadata:

db.fs.files.createIndex({ "metadata.category": 1 })
db.fs.files.createIndex({ "metadata.tags": 1 })
db.fs.files.createIndex({ "metadata.uploaded_by": 1 })

Cleanup - Implement regular cleanup for unused files:

// Delete file and associated chunks
func (s *GridFSService) Delete(fileID string) error {
    // Automatically handled by MongoDB
}

Security Considerations

File Type Validation - Validate MIME types and extensions
Filename Sanitization - Remove dangerous characters:

sanitized := filestore.SanitizeFilename(filename)

Size Limits - Enforce maximum file sizes
Access Control - Implement proper authorization checks
Malware Scanning - Consider integrating antivirus scanning
Rate Limiting - Implement rate limiting on upload endpoints

Testing

Unit tests are included for all operations:

go test ./pkg/filestore -v

Test coverage includes:

Upload operations
Download operations
Delete operations
File validation
Error handling
Metadata storage and retrieval

Troubleshooting

MongoDB Connection Failed

Error: MongoDB connection not available
Solution: Ensure MongoDB is running and configured correctly

File Not Found

Error: file not found
Solution: Verify the file ID is correct

File Too Large

Error: file size exceeds maximum limit
Solution: Check MaxSize configuration and file size

Invalid MIME Type

Error: unsupported file type
Solution: Verify the file type is in the allowed MIME types list

Configuration

Configure file store limits in bootstrap.go:

// Maximum file sizes
const (
    MaxUploadSize         = 100 * 1024 * 1024 // 100 MB
    MaxProfileImageSize   = 5 * 1024 * 1024   // 5 MB
)

// Supported MIME types
var AllowedImageTypes = map[string]bool{
    "image/jpeg":  true,
    "image/png":   true,
    "image/gif":   true,
    "image/webp":  true,
}

Contributing

When extending the file store service:

Maintain backward compatibility
Add appropriate error handling
Update logging for operations
Include unit tests for new features
Update documentation

License

This file store service is part of the Opplens backend system.