m.nazemi
511 lines
12 KiB
Markdown
511 lines
12 KiB
Markdown
# MongoDB GridFS File Store Service
|
|
|
|
A comprehensive file storage service for the Opplens backend using MongoDB GridFS for scalable, distributed file storage.
|
|
|
|
## Overview
|
|
|
|
The file store service provides a complete solution for managing file uploads, downloads, and deletions using MongoDB's GridFS. It supports:
|
|
|
|
- **Secure file uploads** with validation and size limits
|
|
- **Metadata storage** for tracking file information
|
|
- **File categorization** and tagging
|
|
- **List and search** functionality
|
|
- **Stream-based downloads** for efficient memory usage
|
|
- **Access control** and user tracking
|
|
|
|
## Key Features
|
|
|
|
✅ **MongoDB GridFS Backend** - Native MongoDB file storage without external dependencies
|
|
✅ **File Validation** - Filename, MIME type, and size validation
|
|
✅ **Metadata Support** - Store rich metadata with files
|
|
✅ **Category & Tags** - Organize files with categories and tags
|
|
✅ **Stream Processing** - Efficient memory usage for large files
|
|
✅ **Error Handling** - Comprehensive error codes and messages
|
|
✅ **Logging** - Detailed operation logging
|
|
✅ **Thread-Safe** - Concurrent operations support
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────┐
|
|
│ HTTP Handler (Echo) │
|
|
│ - Upload, Download, Delete, List │
|
|
└────────────┬────────────────────────┘
|
|
│
|
|
┌────────────▼────────────────────────┐
|
|
│ File Handler Service │
|
|
│ - Request validation │
|
|
│ - Response formatting │
|
|
└────────────┬────────────────────────┘
|
|
│
|
|
┌────────────▼────────────────────────┐
|
|
│ GridFS Service Interface │
|
|
│ - Upload, Download, Delete, List │
|
|
└────────────┬────────────────────────┘
|
|
│
|
|
┌────────────▼────────────────────────┐
|
|
│ MongoDB GridFS Implementation │
|
|
│ - Bucket operations │
|
|
│ - Metadata management │
|
|
└────────────┬────────────────────────┘
|
|
│
|
|
┌────────────▼────────────────────────┐
|
|
│ MongoDB Database │
|
|
│ - fs.files collection │
|
|
│ - fs.chunks collection │
|
|
└─────────────────────────────────────┘
|
|
```
|
|
|
|
## Installation
|
|
|
|
The service is automatically initialized in the bootstrap:
|
|
|
|
```go
|
|
// In cmd/web/bootstrap/bootstrap.go
|
|
fileStoreService, err := filestore.NewGridFSService(mongoManager, logger)
|
|
if err != nil {
|
|
logger.Fatal("Failed to initialize file store service", map[string]interface{}{
|
|
"error": err.Error(),
|
|
})
|
|
}
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Upload File
|
|
|
|
**POST** `/api/v1/files/upload`
|
|
|
|
Upload a file to GridFS storage.
|
|
|
|
**Request:**
|
|
- Content-Type: `multipart/form-data`
|
|
- Parameters:
|
|
- `file` (required): The file to upload
|
|
- `category` (optional): File category for organization
|
|
- `tags` (optional, array): Tags for the file
|
|
- `description` (optional): File description
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"file_id": "507f1f77bcf86cd799439011",
|
|
"filename": "profile.jpg",
|
|
"content_type": "image/jpeg",
|
|
"size": 102400,
|
|
"message": "File uploaded successfully"
|
|
}
|
|
```
|
|
|
|
**Error Responses:**
|
|
- `400 Bad Request` - No file provided
|
|
- `413 Payload Too Large` - File exceeds size limit
|
|
- `500 Internal Server Error` - Upload failed
|
|
|
|
### Download File
|
|
|
|
**GET** `/api/v1/files/{file_id}/download`
|
|
|
|
Download a file by its ID.
|
|
|
|
**Parameters:**
|
|
- `file_id` (path, required): The file ID
|
|
|
|
**Response:**
|
|
- Returns file content with appropriate MIME type and `Content-Disposition` header
|
|
|
|
**Error Responses:**
|
|
- `404 Not Found` - File not found
|
|
- `500 Internal Server Error` - Download failed
|
|
|
|
### Get File Info
|
|
|
|
**GET** `/api/v1/files/{file_id}/info`
|
|
|
|
Retrieve metadata for a file.
|
|
|
|
**Parameters:**
|
|
- `file_id` (path, required): The file ID
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"file_id": "507f1f77bcf86cd799439011",
|
|
"filename": "profile.jpg",
|
|
"content_type": "image/jpeg",
|
|
"size": 102400,
|
|
"uploaded_by": "user_123",
|
|
"uploaded_at": "2024-04-19T10:30:00Z",
|
|
"category": "profile_images",
|
|
"tags": ["user", "profile"],
|
|
"description": "User profile picture"
|
|
}
|
|
```
|
|
|
|
### List Files
|
|
|
|
**GET** `/api/v1/files`
|
|
|
|
List files with optional filtering.
|
|
|
|
**Query Parameters:**
|
|
- `category` (optional): Filter by category
|
|
- `tags` (optional, array): Filter by tags
|
|
- `limit` (optional, default 50, max 1000): Number of files to return
|
|
- `skip` (optional, default 0): Number of files to skip
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"files": [
|
|
{
|
|
"file_id": "507f1f77bcf86cd799439011",
|
|
"filename": "profile.jpg",
|
|
"content_type": "image/jpeg",
|
|
"size": 102400,
|
|
"uploaded_by": "user_123",
|
|
"uploaded_at": "2024-04-19T10:30:00Z",
|
|
"category": "profile_images",
|
|
"tags": ["user", "profile"]
|
|
}
|
|
],
|
|
"total": 150,
|
|
"limit": 50,
|
|
"skip": 0
|
|
}
|
|
```
|
|
|
|
### Delete File
|
|
|
|
**DELETE** `/api/v1/files/{file_id}`
|
|
|
|
Delete a file by its ID.
|
|
|
|
**Parameters:**
|
|
- `file_id` (path, required): The file ID
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"message": "File deleted successfully"
|
|
}
|
|
```
|
|
|
|
**Error Responses:**
|
|
- `404 Not Found` - File not found
|
|
- `500 Internal Server Error` - Deletion failed
|
|
|
|
## Usage Examples
|
|
|
|
### Upload Profile Image
|
|
|
|
```bash
|
|
curl -X POST \
|
|
-F "file=@profile.jpg" \
|
|
-F "category=profile_images" \
|
|
-F "tags=user" \
|
|
-F "tags=profile" \
|
|
-H "Authorization: Bearer <token>" \
|
|
http://localhost:8080/api/v1/files/upload
|
|
```
|
|
|
|
### Download File
|
|
|
|
```bash
|
|
curl -X GET \
|
|
-H "Authorization: Bearer <token>" \
|
|
http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \
|
|
-o downloaded_file.jpg
|
|
```
|
|
|
|
### Get File Information
|
|
|
|
```bash
|
|
curl -X GET \
|
|
-H "Authorization: Bearer <token>" \
|
|
http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info
|
|
```
|
|
|
|
### List User Profile Images
|
|
|
|
```bash
|
|
curl -X GET \
|
|
-H "Authorization: Bearer <token>" \
|
|
"http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20"
|
|
```
|
|
|
|
## File Validation
|
|
|
|
The service includes built-in validation for file uploads:
|
|
|
|
### Default Validators
|
|
|
|
**Standard File Validator**
|
|
- Supported MIME types: Images, PDFs, Office documents, compressed files
|
|
- Maximum size: 100 MB
|
|
- Filename validation and sanitization
|
|
|
|
**Profile Image Validator**
|
|
- Supported MIME types: JPEG, PNG, GIF, WebP
|
|
- Maximum size: 5 MB
|
|
- Extension validation
|
|
|
|
### Custom Validation
|
|
|
|
Create a custom validator:
|
|
|
|
```go
|
|
validator := filestore.NewFileValidator(
|
|
10 * 1024 * 1024, // 10 MB max
|
|
map[string]bool{
|
|
"application/pdf": true,
|
|
"image/jpeg": true,
|
|
"image/png": true,
|
|
},
|
|
)
|
|
|
|
// Validate file
|
|
if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil {
|
|
// Handle validation error
|
|
}
|
|
```
|
|
|
|
## Metadata Storage
|
|
|
|
Files are stored with rich metadata:
|
|
|
|
```json
|
|
{
|
|
"filename": "profile.jpg",
|
|
"uploadDate": "2024-04-19T10:30:00Z",
|
|
"metadata": {
|
|
"uploaded_by": "user_123",
|
|
"category": "profile_images",
|
|
"tags": ["user", "profile"],
|
|
"description": "User profile picture",
|
|
"content_type": "image/jpeg",
|
|
"extra": {
|
|
"user_id": "user_123",
|
|
"department": "marketing"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
The service uses a comprehensive error handling system:
|
|
|
|
### Error Codes
|
|
|
|
- `FILE_NOT_FOUND` - File does not exist
|
|
- `UPLOAD_FAILED` - Upload operation failed
|
|
- `DOWNLOAD_FAILED` - Download operation failed
|
|
- `DELETE_FAILED` - Delete operation failed
|
|
- `INVALID_INPUT` - Invalid input parameters
|
|
- `FILE_TOO_LARGE` - File exceeds size limit
|
|
- `UNSUPPORTED_FILE_TYPE` - MIME type not supported
|
|
- `DUPLICATE_FILE` - File already exists
|
|
- `ACCESS_DENIED` - User does not have access
|
|
|
|
### Error Response Format
|
|
|
|
```json
|
|
{
|
|
"error": "File is too large",
|
|
"code": "FILE_TOO_LARGE",
|
|
"message": "Maximum file size is 100 MB"
|
|
}
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
### fs.files Collection
|
|
|
|
```javascript
|
|
{
|
|
_id: ObjectId,
|
|
filename: String,
|
|
uploadDate: Date,
|
|
chunkSize: Number,
|
|
length: Number,
|
|
md5: String,
|
|
metadata: {
|
|
uploaded_by: String,
|
|
category: String,
|
|
tags: [String],
|
|
description: String,
|
|
content_type: String,
|
|
extra: {
|
|
user_id: String,
|
|
// ... additional fields
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### fs.chunks Collection
|
|
|
|
```javascript
|
|
{
|
|
_id: ObjectId,
|
|
files_id: ObjectId,
|
|
n: Number,
|
|
data: BinData
|
|
}
|
|
```
|
|
|
|
## Integration with User Service
|
|
|
|
To integrate file uploads with user profiles:
|
|
|
|
1. **Update User Entity** - Add FileID field:
|
|
|
|
```go
|
|
type User struct {
|
|
// ... existing fields
|
|
ProfileImageFileID *string `bson:"profile_image_file_id"`
|
|
}
|
|
```
|
|
|
|
2. **Add Upload Endpoint** - In user handler:
|
|
|
|
```go
|
|
func (h *Handler) UploadProfileImage(c echo.Context) error {
|
|
// Get user ID from context
|
|
userID := c.Get("user_id").(string)
|
|
|
|
// Get file
|
|
file, err := c.FormFile("profile_image")
|
|
// ... upload logic
|
|
|
|
// Update user with file ID
|
|
// ... update user
|
|
}
|
|
```
|
|
|
|
3. **Add Download Endpoint** - Serve user's profile image:
|
|
|
|
```go
|
|
func (h *Handler) GetProfileImage(c echo.Context) error {
|
|
userID := c.Param("user_id")
|
|
|
|
// Get user
|
|
user, err := h.service.GetUserByID(userID)
|
|
|
|
// Download file
|
|
reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID)
|
|
// ... stream response
|
|
}
|
|
```
|
|
|
|
## Performance Considerations
|
|
|
|
1. **Chunk Size** - GridFS default is 255 KB. For large files, consider increasing this.
|
|
2. **Indexing** - Create indexes on frequently queried metadata:
|
|
|
|
```javascript
|
|
db.fs.files.createIndex({ "metadata.category": 1 })
|
|
db.fs.files.createIndex({ "metadata.tags": 1 })
|
|
db.fs.files.createIndex({ "metadata.uploaded_by": 1 })
|
|
```
|
|
|
|
3. **Cleanup** - Implement regular cleanup for unused files:
|
|
|
|
```go
|
|
// Delete file and associated chunks
|
|
func (s *GridFSService) Delete(fileID string) error {
|
|
// Automatically handled by MongoDB
|
|
}
|
|
```
|
|
|
|
## Security Considerations
|
|
|
|
1. **File Type Validation** - Validate MIME types and extensions
|
|
2. **Filename Sanitization** - Remove dangerous characters:
|
|
|
|
```go
|
|
sanitized := filestore.SanitizeFilename(filename)
|
|
```
|
|
|
|
3. **Size Limits** - Enforce maximum file sizes
|
|
4. **Access Control** - Implement proper authorization checks
|
|
5. **Malware Scanning** - Consider integrating antivirus scanning
|
|
6. **Rate Limiting** - Implement rate limiting on upload endpoints
|
|
|
|
## Testing
|
|
|
|
Unit tests are included for all operations:
|
|
|
|
```bash
|
|
go test ./pkg/filestore -v
|
|
```
|
|
|
|
Test coverage includes:
|
|
- Upload operations
|
|
- Download operations
|
|
- Delete operations
|
|
- File validation
|
|
- Error handling
|
|
- Metadata storage and retrieval
|
|
|
|
## Troubleshooting
|
|
|
|
### MongoDB Connection Failed
|
|
```
|
|
Error: MongoDB connection not available
|
|
Solution: Ensure MongoDB is running and configured correctly
|
|
```
|
|
|
|
### File Not Found
|
|
```
|
|
Error: file not found
|
|
Solution: Verify the file ID is correct
|
|
```
|
|
|
|
### File Too Large
|
|
```
|
|
Error: file size exceeds maximum limit
|
|
Solution: Check MaxSize configuration and file size
|
|
```
|
|
|
|
### Invalid MIME Type
|
|
```
|
|
Error: unsupported file type
|
|
Solution: Verify the file type is in the allowed MIME types list
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configure file store limits in `bootstrap.go`:
|
|
|
|
```go
|
|
// Maximum file sizes
|
|
const (
|
|
MaxUploadSize = 100 * 1024 * 1024 // 100 MB
|
|
MaxProfileImageSize = 5 * 1024 * 1024 // 5 MB
|
|
)
|
|
|
|
// Supported MIME types
|
|
var AllowedImageTypes = map[string]bool{
|
|
"image/jpeg": true,
|
|
"image/png": true,
|
|
"image/gif": true,
|
|
"image/webp": true,
|
|
}
|
|
```
|
|
|
|
## Contributing
|
|
|
|
When extending the file store service:
|
|
|
|
1. Maintain backward compatibility
|
|
2. Add appropriate error handling
|
|
3. Update logging for operations
|
|
4. Include unit tests for new features
|
|
5. Update documentation
|
|
|
|
## License
|
|
|
|
This file store service is part of the Opplens backend system.
|