# MongoDB GridFS File Store Service A comprehensive file storage service for the Opplens backend using MongoDB GridFS for scalable, distributed file storage. ## Overview The file store service provides a complete solution for managing file uploads, downloads, and deletions using MongoDB's GridFS. It supports: - **Secure file uploads** with validation and size limits - **Metadata storage** for tracking file information - **File categorization** and tagging - **List and search** functionality - **Stream-based downloads** for efficient memory usage - **Access control** and user tracking ## Key Features ✅ **MongoDB GridFS Backend** - Native MongoDB file storage without external dependencies ✅ **File Validation** - Filename, MIME type, and size validation ✅ **Metadata Support** - Store rich metadata with files ✅ **Category & Tags** - Organize files with categories and tags ✅ **Stream Processing** - Efficient memory usage for large files ✅ **Error Handling** - Comprehensive error codes and messages ✅ **Logging** - Detailed operation logging ✅ **Thread-Safe** - Concurrent operations support ## Architecture ``` ┌─────────────────────────────────────┐ │ HTTP Handler (Echo) │ │ - Upload, Download, Delete, List │ └────────────┬────────────────────────┘ │ ┌────────────▼────────────────────────┐ │ File Handler Service │ │ - Request validation │ │ - Response formatting │ └────────────┬────────────────────────┘ │ ┌────────────▼────────────────────────┐ │ GridFS Service Interface │ │ - Upload, Download, Delete, List │ └────────────┬────────────────────────┘ │ ┌────────────▼────────────────────────┐ │ MongoDB GridFS Implementation │ │ - Bucket operations │ │ - Metadata management │ └────────────┬────────────────────────┘ │ ┌────────────▼────────────────────────┐ │ MongoDB Database │ │ - fs.files collection │ │ - fs.chunks collection │ └─────────────────────────────────────┘ ``` ## Installation The service is automatically initialized in the bootstrap: ```go // In cmd/web/bootstrap/bootstrap.go fileStoreService, err := filestore.NewGridFSService(mongoManager, logger) if err != nil { logger.Fatal("Failed to initialize file store service", map[string]interface{}{ "error": err.Error(), }) } ``` ## API Endpoints ### Upload File **POST** `/api/v1/files/upload` Upload a file to GridFS storage. **Request:** - Content-Type: `multipart/form-data` - Parameters: - `file` (required): The file to upload - `category` (optional): File category for organization - `tags` (optional, array): Tags for the file - `description` (optional): File description **Response:** ```json { "file_id": "507f1f77bcf86cd799439011", "filename": "profile.jpg", "content_type": "image/jpeg", "size": 102400, "message": "File uploaded successfully" } ``` **Error Responses:** - `400 Bad Request` - No file provided - `413 Payload Too Large` - File exceeds size limit - `500 Internal Server Error` - Upload failed ### Download File **GET** `/api/v1/files/{file_id}/download` Download a file by its ID. **Parameters:** - `file_id` (path, required): The file ID **Response:** - Returns file content with appropriate MIME type and `Content-Disposition` header **Error Responses:** - `404 Not Found` - File not found - `500 Internal Server Error` - Download failed ### Get File Info **GET** `/api/v1/files/{file_id}/info` Retrieve metadata for a file. **Parameters:** - `file_id` (path, required): The file ID **Response:** ```json { "file_id": "507f1f77bcf86cd799439011", "filename": "profile.jpg", "content_type": "image/jpeg", "size": 102400, "uploaded_by": "user_123", "uploaded_at": "2024-04-19T10:30:00Z", "category": "profile_images", "tags": ["user", "profile"], "description": "User profile picture" } ``` ### List Files **GET** `/api/v1/files` List files with optional filtering. **Query Parameters:** - `category` (optional): Filter by category - `tags` (optional, array): Filter by tags - `limit` (optional, default 50, max 1000): Number of files to return - `skip` (optional, default 0): Number of files to skip **Response:** ```json { "files": [ { "file_id": "507f1f77bcf86cd799439011", "filename": "profile.jpg", "content_type": "image/jpeg", "size": 102400, "uploaded_by": "user_123", "uploaded_at": "2024-04-19T10:30:00Z", "category": "profile_images", "tags": ["user", "profile"] } ], "total": 150, "limit": 50, "skip": 0 } ``` ### Delete File **DELETE** `/api/v1/files/{file_id}` Delete a file by its ID. **Parameters:** - `file_id` (path, required): The file ID **Response:** ```json { "message": "File deleted successfully" } ``` **Error Responses:** - `404 Not Found` - File not found - `500 Internal Server Error` - Deletion failed ## Usage Examples ### Upload Profile Image ```bash curl -X POST \ -F "file=@profile.jpg" \ -F "category=profile_images" \ -F "tags=user" \ -F "tags=profile" \ -H "Authorization: Bearer " \ http://localhost:8080/api/v1/files/upload ``` ### Download File ```bash curl -X GET \ -H "Authorization: Bearer " \ http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \ -o downloaded_file.jpg ``` ### Get File Information ```bash curl -X GET \ -H "Authorization: Bearer " \ http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info ``` ### List User Profile Images ```bash curl -X GET \ -H "Authorization: Bearer " \ "http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20" ``` ## File Validation The service includes built-in validation for file uploads: ### Default Validators **Standard File Validator** - Supported MIME types: Images, PDFs, Office documents, compressed files - Maximum size: 100 MB - Filename validation and sanitization **Profile Image Validator** - Supported MIME types: JPEG, PNG, GIF, WebP - Maximum size: 5 MB - Extension validation ### Custom Validation Create a custom validator: ```go validator := filestore.NewFileValidator( 10 * 1024 * 1024, // 10 MB max map[string]bool{ "application/pdf": true, "image/jpeg": true, "image/png": true, }, ) // Validate file if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil { // Handle validation error } ``` ## Metadata Storage Files are stored with rich metadata: ```json { "filename": "profile.jpg", "uploadDate": "2024-04-19T10:30:00Z", "metadata": { "uploaded_by": "user_123", "category": "profile_images", "tags": ["user", "profile"], "description": "User profile picture", "content_type": "image/jpeg", "extra": { "user_id": "user_123", "department": "marketing" } } } ``` ## Error Handling The service uses a comprehensive error handling system: ### Error Codes - `FILE_NOT_FOUND` - File does not exist - `UPLOAD_FAILED` - Upload operation failed - `DOWNLOAD_FAILED` - Download operation failed - `DELETE_FAILED` - Delete operation failed - `INVALID_INPUT` - Invalid input parameters - `FILE_TOO_LARGE` - File exceeds size limit - `UNSUPPORTED_FILE_TYPE` - MIME type not supported - `DUPLICATE_FILE` - File already exists - `ACCESS_DENIED` - User does not have access ### Error Response Format ```json { "error": "File is too large", "code": "FILE_TOO_LARGE", "message": "Maximum file size is 100 MB" } ``` ## Database Schema ### fs.files Collection ```javascript { _id: ObjectId, filename: String, uploadDate: Date, chunkSize: Number, length: Number, md5: String, metadata: { uploaded_by: String, category: String, tags: [String], description: String, content_type: String, extra: { user_id: String, // ... additional fields } } } ``` ### fs.chunks Collection ```javascript { _id: ObjectId, files_id: ObjectId, n: Number, data: BinData } ``` ## Integration with User Service To integrate file uploads with user profiles: 1. **Update User Entity** - Add FileID field: ```go type User struct { // ... existing fields ProfileImageFileID *string `bson:"profile_image_file_id"` } ``` 2. **Add Upload Endpoint** - In user handler: ```go func (h *Handler) UploadProfileImage(c echo.Context) error { // Get user ID from context userID := c.Get("user_id").(string) // Get file file, err := c.FormFile("profile_image") // ... upload logic // Update user with file ID // ... update user } ``` 3. **Add Download Endpoint** - Serve user's profile image: ```go func (h *Handler) GetProfileImage(c echo.Context) error { userID := c.Param("user_id") // Get user user, err := h.service.GetUserByID(userID) // Download file reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID) // ... stream response } ``` ## Performance Considerations 1. **Chunk Size** - GridFS default is 255 KB. For large files, consider increasing this. 2. **Indexing** - Create indexes on frequently queried metadata: ```javascript db.fs.files.createIndex({ "metadata.category": 1 }) db.fs.files.createIndex({ "metadata.tags": 1 }) db.fs.files.createIndex({ "metadata.uploaded_by": 1 }) ``` 3. **Cleanup** - Implement regular cleanup for unused files: ```go // Delete file and associated chunks func (s *GridFSService) Delete(fileID string) error { // Automatically handled by MongoDB } ``` ## Security Considerations 1. **File Type Validation** - Validate MIME types and extensions 2. **Filename Sanitization** - Remove dangerous characters: ```go sanitized := filestore.SanitizeFilename(filename) ``` 3. **Size Limits** - Enforce maximum file sizes 4. **Access Control** - Implement proper authorization checks 5. **Malware Scanning** - Consider integrating antivirus scanning 6. **Rate Limiting** - Implement rate limiting on upload endpoints ## Testing Unit tests are included for all operations: ```bash go test ./pkg/filestore -v ``` Test coverage includes: - Upload operations - Download operations - Delete operations - File validation - Error handling - Metadata storage and retrieval ## Troubleshooting ### MongoDB Connection Failed ``` Error: MongoDB connection not available Solution: Ensure MongoDB is running and configured correctly ``` ### File Not Found ``` Error: file not found Solution: Verify the file ID is correct ``` ### File Too Large ``` Error: file size exceeds maximum limit Solution: Check MaxSize configuration and file size ``` ### Invalid MIME Type ``` Error: unsupported file type Solution: Verify the file type is in the allowed MIME types list ``` ## Configuration Configure file store limits in `bootstrap.go`: ```go // Maximum file sizes const ( MaxUploadSize = 100 * 1024 * 1024 // 100 MB MaxProfileImageSize = 5 * 1024 * 1024 // 5 MB ) // Supported MIME types var AllowedImageTypes = map[string]bool{ "image/jpeg": true, "image/png": true, "image/gif": true, "image/webp": true, } ``` ## Contributing When extending the file store service: 1. Maintain backward compatibility 2. Add appropriate error handling 3. Update logging for operations 4. Include unit tests for new features 5. Update documentation ## License This file store service is part of the Opplens backend system.