implemented mongo gridFS for file upload
This commit is contained in:
m.nazemi
@@ -0,0 +1,510 @@
|
||||
# MongoDB GridFS File Store Service
|
||||
|
||||
A comprehensive file storage service for the Opplens backend using MongoDB GridFS for scalable, distributed file storage.
|
||||
|
||||
## Overview
|
||||
|
||||
The file store service provides a complete solution for managing file uploads, downloads, and deletions using MongoDB's GridFS. It supports:
|
||||
|
||||
- **Secure file uploads** with validation and size limits
|
||||
- **Metadata storage** for tracking file information
|
||||
- **File categorization** and tagging
|
||||
- **List and search** functionality
|
||||
- **Stream-based downloads** for efficient memory usage
|
||||
- **Access control** and user tracking
|
||||
|
||||
## Key Features
|
||||
|
||||
✅ **MongoDB GridFS Backend** - Native MongoDB file storage without external dependencies
|
||||
✅ **File Validation** - Filename, MIME type, and size validation
|
||||
✅ **Metadata Support** - Store rich metadata with files
|
||||
✅ **Category & Tags** - Organize files with categories and tags
|
||||
✅ **Stream Processing** - Efficient memory usage for large files
|
||||
✅ **Error Handling** - Comprehensive error codes and messages
|
||||
✅ **Logging** - Detailed operation logging
|
||||
✅ **Thread-Safe** - Concurrent operations support
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ HTTP Handler (Echo) │
|
||||
│ - Upload, Download, Delete, List │
|
||||
└────────────┬────────────────────────┘
|
||||
│
|
||||
┌────────────▼────────────────────────┐
|
||||
│ File Handler Service │
|
||||
│ - Request validation │
|
||||
│ - Response formatting │
|
||||
└────────────┬────────────────────────┘
|
||||
│
|
||||
┌────────────▼────────────────────────┐
|
||||
│ GridFS Service Interface │
|
||||
│ - Upload, Download, Delete, List │
|
||||
└────────────┬────────────────────────┘
|
||||
│
|
||||
┌────────────▼────────────────────────┐
|
||||
│ MongoDB GridFS Implementation │
|
||||
│ - Bucket operations │
|
||||
│ - Metadata management │
|
||||
└────────────┬────────────────────────┘
|
||||
│
|
||||
┌────────────▼────────────────────────┐
|
||||
│ MongoDB Database │
|
||||
│ - fs.files collection │
|
||||
│ - fs.chunks collection │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
The service is automatically initialized in the bootstrap:
|
||||
|
||||
```go
|
||||
// In cmd/web/bootstrap/bootstrap.go
|
||||
fileStoreService, err := filestore.NewGridFSService(mongoManager, logger)
|
||||
if err != nil {
|
||||
logger.Fatal("Failed to initialize file store service", map[string]interface{}{
|
||||
"error": err.Error(),
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Upload File
|
||||
|
||||
**POST** `/api/v1/files/upload`
|
||||
|
||||
Upload a file to GridFS storage.
|
||||
|
||||
**Request:**
|
||||
- Content-Type: `multipart/form-data`
|
||||
- Parameters:
|
||||
- `file` (required): The file to upload
|
||||
- `category` (optional): File category for organization
|
||||
- `tags` (optional, array): Tags for the file
|
||||
- `description` (optional): File description
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"file_id": "507f1f77bcf86cd799439011",
|
||||
"filename": "profile.jpg",
|
||||
"content_type": "image/jpeg",
|
||||
"size": 102400,
|
||||
"message": "File uploaded successfully"
|
||||
}
|
||||
```
|
||||
|
||||
**Error Responses:**
|
||||
- `400 Bad Request` - No file provided
|
||||
- `413 Payload Too Large` - File exceeds size limit
|
||||
- `500 Internal Server Error` - Upload failed
|
||||
|
||||
### Download File
|
||||
|
||||
**GET** `/api/v1/files/{file_id}/download`
|
||||
|
||||
Download a file by its ID.
|
||||
|
||||
**Parameters:**
|
||||
- `file_id` (path, required): The file ID
|
||||
|
||||
**Response:**
|
||||
- Returns file content with appropriate MIME type and `Content-Disposition` header
|
||||
|
||||
**Error Responses:**
|
||||
- `404 Not Found` - File not found
|
||||
- `500 Internal Server Error` - Download failed
|
||||
|
||||
### Get File Info
|
||||
|
||||
**GET** `/api/v1/files/{file_id}/info`
|
||||
|
||||
Retrieve metadata for a file.
|
||||
|
||||
**Parameters:**
|
||||
- `file_id` (path, required): The file ID
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"file_id": "507f1f77bcf86cd799439011",
|
||||
"filename": "profile.jpg",
|
||||
"content_type": "image/jpeg",
|
||||
"size": 102400,
|
||||
"uploaded_by": "user_123",
|
||||
"uploaded_at": "2024-04-19T10:30:00Z",
|
||||
"category": "profile_images",
|
||||
"tags": ["user", "profile"],
|
||||
"description": "User profile picture"
|
||||
}
|
||||
```
|
||||
|
||||
### List Files
|
||||
|
||||
**GET** `/api/v1/files`
|
||||
|
||||
List files with optional filtering.
|
||||
|
||||
**Query Parameters:**
|
||||
- `category` (optional): Filter by category
|
||||
- `tags` (optional, array): Filter by tags
|
||||
- `limit` (optional, default 50, max 1000): Number of files to return
|
||||
- `skip` (optional, default 0): Number of files to skip
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"files": [
|
||||
{
|
||||
"file_id": "507f1f77bcf86cd799439011",
|
||||
"filename": "profile.jpg",
|
||||
"content_type": "image/jpeg",
|
||||
"size": 102400,
|
||||
"uploaded_by": "user_123",
|
||||
"uploaded_at": "2024-04-19T10:30:00Z",
|
||||
"category": "profile_images",
|
||||
"tags": ["user", "profile"]
|
||||
}
|
||||
],
|
||||
"total": 150,
|
||||
"limit": 50,
|
||||
"skip": 0
|
||||
}
|
||||
```
|
||||
|
||||
### Delete File
|
||||
|
||||
**DELETE** `/api/v1/files/{file_id}`
|
||||
|
||||
Delete a file by its ID.
|
||||
|
||||
**Parameters:**
|
||||
- `file_id` (path, required): The file ID
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"message": "File deleted successfully"
|
||||
}
|
||||
```
|
||||
|
||||
**Error Responses:**
|
||||
- `404 Not Found` - File not found
|
||||
- `500 Internal Server Error` - Deletion failed
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Upload Profile Image
|
||||
|
||||
```bash
|
||||
curl -X POST \
|
||||
-F "file=@profile.jpg" \
|
||||
-F "category=profile_images" \
|
||||
-F "tags=user" \
|
||||
-F "tags=profile" \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
http://localhost:8080/api/v1/files/upload
|
||||
```
|
||||
|
||||
### Download File
|
||||
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \
|
||||
-o downloaded_file.jpg
|
||||
```
|
||||
|
||||
### Get File Information
|
||||
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info
|
||||
```
|
||||
|
||||
### List User Profile Images
|
||||
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Bearer <token>" \
|
||||
"http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20"
|
||||
```
|
||||
|
||||
## File Validation
|
||||
|
||||
The service includes built-in validation for file uploads:
|
||||
|
||||
### Default Validators
|
||||
|
||||
**Standard File Validator**
|
||||
- Supported MIME types: Images, PDFs, Office documents, compressed files
|
||||
- Maximum size: 100 MB
|
||||
- Filename validation and sanitization
|
||||
|
||||
**Profile Image Validator**
|
||||
- Supported MIME types: JPEG, PNG, GIF, WebP
|
||||
- Maximum size: 5 MB
|
||||
- Extension validation
|
||||
|
||||
### Custom Validation
|
||||
|
||||
Create a custom validator:
|
||||
|
||||
```go
|
||||
validator := filestore.NewFileValidator(
|
||||
10 * 1024 * 1024, // 10 MB max
|
||||
map[string]bool{
|
||||
"application/pdf": true,
|
||||
"image/jpeg": true,
|
||||
"image/png": true,
|
||||
},
|
||||
)
|
||||
|
||||
// Validate file
|
||||
if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil {
|
||||
// Handle validation error
|
||||
}
|
||||
```
|
||||
|
||||
## Metadata Storage
|
||||
|
||||
Files are stored with rich metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"filename": "profile.jpg",
|
||||
"uploadDate": "2024-04-19T10:30:00Z",
|
||||
"metadata": {
|
||||
"uploaded_by": "user_123",
|
||||
"category": "profile_images",
|
||||
"tags": ["user", "profile"],
|
||||
"description": "User profile picture",
|
||||
"content_type": "image/jpeg",
|
||||
"extra": {
|
||||
"user_id": "user_123",
|
||||
"department": "marketing"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The service uses a comprehensive error handling system:
|
||||
|
||||
### Error Codes
|
||||
|
||||
- `FILE_NOT_FOUND` - File does not exist
|
||||
- `UPLOAD_FAILED` - Upload operation failed
|
||||
- `DOWNLOAD_FAILED` - Download operation failed
|
||||
- `DELETE_FAILED` - Delete operation failed
|
||||
- `INVALID_INPUT` - Invalid input parameters
|
||||
- `FILE_TOO_LARGE` - File exceeds size limit
|
||||
- `UNSUPPORTED_FILE_TYPE` - MIME type not supported
|
||||
- `DUPLICATE_FILE` - File already exists
|
||||
- `ACCESS_DENIED` - User does not have access
|
||||
|
||||
### Error Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"error": "File is too large",
|
||||
"code": "FILE_TOO_LARGE",
|
||||
"message": "Maximum file size is 100 MB"
|
||||
}
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
### fs.files Collection
|
||||
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId,
|
||||
filename: String,
|
||||
uploadDate: Date,
|
||||
chunkSize: Number,
|
||||
length: Number,
|
||||
md5: String,
|
||||
metadata: {
|
||||
uploaded_by: String,
|
||||
category: String,
|
||||
tags: [String],
|
||||
description: String,
|
||||
content_type: String,
|
||||
extra: {
|
||||
user_id: String,
|
||||
// ... additional fields
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### fs.chunks Collection
|
||||
|
||||
```javascript
|
||||
{
|
||||
_id: ObjectId,
|
||||
files_id: ObjectId,
|
||||
n: Number,
|
||||
data: BinData
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with User Service
|
||||
|
||||
To integrate file uploads with user profiles:
|
||||
|
||||
1. **Update User Entity** - Add FileID field:
|
||||
|
||||
```go
|
||||
type User struct {
|
||||
// ... existing fields
|
||||
ProfileImageFileID *string `bson:"profile_image_file_id"`
|
||||
}
|
||||
```
|
||||
|
||||
2. **Add Upload Endpoint** - In user handler:
|
||||
|
||||
```go
|
||||
func (h *Handler) UploadProfileImage(c echo.Context) error {
|
||||
// Get user ID from context
|
||||
userID := c.Get("user_id").(string)
|
||||
|
||||
// Get file
|
||||
file, err := c.FormFile("profile_image")
|
||||
// ... upload logic
|
||||
|
||||
// Update user with file ID
|
||||
// ... update user
|
||||
}
|
||||
```
|
||||
|
||||
3. **Add Download Endpoint** - Serve user's profile image:
|
||||
|
||||
```go
|
||||
func (h *Handler) GetProfileImage(c echo.Context) error {
|
||||
userID := c.Param("user_id")
|
||||
|
||||
// Get user
|
||||
user, err := h.service.GetUserByID(userID)
|
||||
|
||||
// Download file
|
||||
reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID)
|
||||
// ... stream response
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
1. **Chunk Size** - GridFS default is 255 KB. For large files, consider increasing this.
|
||||
2. **Indexing** - Create indexes on frequently queried metadata:
|
||||
|
||||
```javascript
|
||||
db.fs.files.createIndex({ "metadata.category": 1 })
|
||||
db.fs.files.createIndex({ "metadata.tags": 1 })
|
||||
db.fs.files.createIndex({ "metadata.uploaded_by": 1 })
|
||||
```
|
||||
|
||||
3. **Cleanup** - Implement regular cleanup for unused files:
|
||||
|
||||
```go
|
||||
// Delete file and associated chunks
|
||||
func (s *GridFSService) Delete(fileID string) error {
|
||||
// Automatically handled by MongoDB
|
||||
}
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **File Type Validation** - Validate MIME types and extensions
|
||||
2. **Filename Sanitization** - Remove dangerous characters:
|
||||
|
||||
```go
|
||||
sanitized := filestore.SanitizeFilename(filename)
|
||||
```
|
||||
|
||||
3. **Size Limits** - Enforce maximum file sizes
|
||||
4. **Access Control** - Implement proper authorization checks
|
||||
5. **Malware Scanning** - Consider integrating antivirus scanning
|
||||
6. **Rate Limiting** - Implement rate limiting on upload endpoints
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests are included for all operations:
|
||||
|
||||
```bash
|
||||
go test ./pkg/filestore -v
|
||||
```
|
||||
|
||||
Test coverage includes:
|
||||
- Upload operations
|
||||
- Download operations
|
||||
- Delete operations
|
||||
- File validation
|
||||
- Error handling
|
||||
- Metadata storage and retrieval
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### MongoDB Connection Failed
|
||||
```
|
||||
Error: MongoDB connection not available
|
||||
Solution: Ensure MongoDB is running and configured correctly
|
||||
```
|
||||
|
||||
### File Not Found
|
||||
```
|
||||
Error: file not found
|
||||
Solution: Verify the file ID is correct
|
||||
```
|
||||
|
||||
### File Too Large
|
||||
```
|
||||
Error: file size exceeds maximum limit
|
||||
Solution: Check MaxSize configuration and file size
|
||||
```
|
||||
|
||||
### Invalid MIME Type
|
||||
```
|
||||
Error: unsupported file type
|
||||
Solution: Verify the file type is in the allowed MIME types list
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Configure file store limits in `bootstrap.go`:
|
||||
|
||||
```go
|
||||
// Maximum file sizes
|
||||
const (
|
||||
MaxUploadSize = 100 * 1024 * 1024 // 100 MB
|
||||
MaxProfileImageSize = 5 * 1024 * 1024 // 5 MB
|
||||
)
|
||||
|
||||
// Supported MIME types
|
||||
var AllowedImageTypes = map[string]bool{
|
||||
"image/jpeg": true,
|
||||
"image/png": true,
|
||||
"image/gif": true,
|
||||
"image/webp": true,
|
||||
}
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When extending the file store service:
|
||||
|
||||
1. Maintain backward compatibility
|
||||
2. Add appropriate error handling
|
||||
3. Update logging for operations
|
||||
4. Include unit tests for new features
|
||||
5. Update documentation
|
||||
|
||||
## License
|
||||
|
||||
This file store service is part of the Opplens backend system.
|
||||
Reference in New Issue
Block a user