# MongoDB GridFS File Store Service

A comprehensive file storage service for the Opplens backend using MongoDB GridFS for scalable, distributed file storage.

## Overview

The file store service provides a complete solution for managing file uploads, downloads, and deletions using MongoDB's GridFS. It supports:

- **Secure file uploads** with validation and size limits
- **Metadata storage** for tracking file information
- **File categorization** and tagging
- **List and search** functionality
- **Stream-based downloads** for efficient memory usage
- **Access control** and user tracking

## Key Features

✅ **MongoDB GridFS Backend** - Native MongoDB file storage without external dependencies
✅ **File Validation** - Filename, MIME type, and size validation
✅ **Metadata Support** - Store rich metadata with files
✅ **Category & Tags** - Organize files with categories and tags
✅ **Stream Processing** - Efficient memory usage for large files
✅ **Error Handling** - Comprehensive error codes and messages
✅ **Logging** - Detailed operation logging
✅ **Thread-Safe** - Concurrent operations support

## Architecture

```
┌─────────────────────────────────────┐
│       HTTP Handler (Echo)           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│      File Handler Service           │
│  - Request validation               │
│  - Response formatting              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│  GridFS Service Interface           │
│  - Upload, Download, Delete, List  │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│   MongoDB GridFS Implementation     │
│  - Bucket operations                │
│  - Metadata management              │
└────────────┬────────────────────────┘
             │
┌────────────▼────────────────────────┐
│     MongoDB Database                │
│  - fs.files collection              │
│  - fs.chunks collection             │
└─────────────────────────────────────┘
```

## Installation

The service is automatically initialized in the bootstrap:

```go
// In cmd/web/bootstrap/bootstrap.go
fileStoreService, err := filestore.NewGridFSService(mongoManager, logger)
if err != nil {
    logger.Fatal("Failed to initialize file store service", map[string]interface{}{
        "error": err.Error(),
    })
}
```

## API Endpoints

### Upload File

**POST** `/api/v1/files/upload`

Upload a file to GridFS storage.

**Request:**
- Content-Type: `multipart/form-data`
- Parameters:
  - `file` (required): The file to upload
  - `category` (optional): File category for organization
  - `tags` (optional, array): Tags for the file
  - `description` (optional): File description

**Response:**
```json
{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "message": "File uploaded successfully"
}
```

**Error Responses:**
- `400 Bad Request` - No file provided
- `413 Payload Too Large` - File exceeds size limit
- `500 Internal Server Error` - Upload failed

### Download File

**GET** `/api/v1/files/{file_id}/download`

Download a file by its ID.

**Parameters:**
- `file_id` (path, required): The file ID

**Response:**
- Returns file content with appropriate MIME type and `Content-Disposition` header

**Error Responses:**
- `404 Not Found` - File not found
- `500 Internal Server Error` - Download failed

### Get File Info

**GET** `/api/v1/files/{file_id}/info`

Retrieve metadata for a file.

**Parameters:**
- `file_id` (path, required): The file ID

**Response:**
```json
{
  "file_id": "507f1f77bcf86cd799439011",
  "filename": "profile.jpg",
  "content_type": "image/jpeg",
  "size": 102400,
  "uploaded_by": "user_123",
  "uploaded_at": "2024-04-19T10:30:00Z",
  "category": "profile_images",
  "tags": ["user", "profile"],
  "description": "User profile picture"
}
```

### List Files

**GET** `/api/v1/files`

List files with optional filtering.

**Query Parameters:**
- `category` (optional): Filter by category
- `tags` (optional, array): Filter by tags
- `limit` (optional, default 50, max 1000): Number of files to return
- `skip` (optional, default 0): Number of files to skip

**Response:**
```json
{
  "files": [
    {
      "file_id": "507f1f77bcf86cd799439011",
      "filename": "profile.jpg",
      "content_type": "image/jpeg",
      "size": 102400,
      "uploaded_by": "user_123",
      "uploaded_at": "2024-04-19T10:30:00Z",
      "category": "profile_images",
      "tags": ["user", "profile"]
    }
  ],
  "total": 150,
  "limit": 50,
  "skip": 0
}
```

### Delete File

**DELETE** `/api/v1/files/{file_id}`

Delete a file by its ID.

**Parameters:**
- `file_id` (path, required): The file ID

**Response:**
```json
{
  "message": "File deleted successfully"
}
```

**Error Responses:**
- `404 Not Found` - File not found
- `500 Internal Server Error` - Deletion failed

## Usage Examples

### Upload Profile Image

```bash
curl -X POST \
  -F "file=@profile.jpg" \
  -F "category=profile_images" \
  -F "tags=user" \
  -F "tags=profile" \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/upload
```

### Download File

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/download \
  -o downloaded_file.jpg
```

### Get File Information

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  http://localhost:8080/api/v1/files/507f1f77bcf86cd799439011/info
```

### List User Profile Images

```bash
curl -X GET \
  -H "Authorization: Bearer <token>" \
  "http://localhost:8080/api/v1/files?category=profile_images&tags=user&limit=20"
```

## File Validation

The service includes built-in validation for file uploads:

### Default Validators

**Standard File Validator**
- Supported MIME types: Images, PDFs, Office documents, compressed files
- Maximum size: 100 MB
- Filename validation and sanitization

**Profile Image Validator**
- Supported MIME types: JPEG, PNG, GIF, WebP
- Maximum size: 5 MB
- Extension validation

### Custom Validation

Create a custom validator:

```go
validator := filestore.NewFileValidator(
    10 * 1024 * 1024, // 10 MB max
    map[string]bool{
        "application/pdf": true,
        "image/jpeg":      true,
        "image/png":       true,
    },
)

// Validate file
if err := validator.ValidateFile("document.pdf", "application/pdf", fileSize); err != nil {
    // Handle validation error
}
```

## Metadata Storage

Files are stored with rich metadata:

```json
{
  "filename": "profile.jpg",
  "uploadDate": "2024-04-19T10:30:00Z",
  "metadata": {
    "uploaded_by": "user_123",
    "category": "profile_images",
    "tags": ["user", "profile"],
    "description": "User profile picture",
    "content_type": "image/jpeg",
    "extra": {
      "user_id": "user_123",
      "department": "marketing"
    }
  }
}
```

## Error Handling

The service uses a comprehensive error handling system:

### Error Codes

- `FILE_NOT_FOUND` - File does not exist
- `UPLOAD_FAILED` - Upload operation failed
- `DOWNLOAD_FAILED` - Download operation failed
- `DELETE_FAILED` - Delete operation failed
- `INVALID_INPUT` - Invalid input parameters
- `FILE_TOO_LARGE` - File exceeds size limit
- `UNSUPPORTED_FILE_TYPE` - MIME type not supported
- `DUPLICATE_FILE` - File already exists
- `ACCESS_DENIED` - User does not have access

### Error Response Format

```json
{
  "error": "File is too large",
  "code": "FILE_TOO_LARGE",
  "message": "Maximum file size is 100 MB"
}
```

## Database Schema

### fs.files Collection

```javascript
{
  _id: ObjectId,
  filename: String,
  uploadDate: Date,
  chunkSize: Number,
  length: Number,
  md5: String,
  metadata: {
    uploaded_by: String,
    category: String,
    tags: [String],
    description: String,
    content_type: String,
    extra: {
      user_id: String,
      // ... additional fields
    }
  }
}
```

### fs.chunks Collection

```javascript
{
  _id: ObjectId,
  files_id: ObjectId,
  n: Number,
  data: BinData
}
```

## Integration with User Service

To integrate file uploads with user profiles:

1. **Update User Entity** - Add FileID field:

```go
type User struct {
    // ... existing fields
    ProfileImageFileID *string `bson:"profile_image_file_id"`
}
```

2. **Add Upload Endpoint** - In user handler:

```go
func (h *Handler) UploadProfileImage(c echo.Context) error {
    // Get user ID from context
    userID := c.Get("user_id").(string)
    
    // Get file
    file, err := c.FormFile("profile_image")
    // ... upload logic
    
    // Update user with file ID
    // ... update user
}
```

3. **Add Download Endpoint** - Serve user's profile image:

```go
func (h *Handler) GetProfileImage(c echo.Context) error {
    userID := c.Param("user_id")
    
    // Get user
    user, err := h.service.GetUserByID(userID)
    
    // Download file
    reader, fileInfo, err := h.fileStoreService.Download(*user.ProfileImageFileID)
    // ... stream response
}
```

## Performance Considerations

1. **Chunk Size** - GridFS default is 255 KB. For large files, consider increasing this.
2. **Indexing** - Create indexes on frequently queried metadata:

```javascript
db.fs.files.createIndex({ "metadata.category": 1 })
db.fs.files.createIndex({ "metadata.tags": 1 })
db.fs.files.createIndex({ "metadata.uploaded_by": 1 })
```

3. **Cleanup** - Implement regular cleanup for unused files:

```go
// Delete file and associated chunks
func (s *GridFSService) Delete(fileID string) error {
    // Automatically handled by MongoDB
}
```

## Security Considerations

1. **File Type Validation** - Validate MIME types and extensions
2. **Filename Sanitization** - Remove dangerous characters:

```go
sanitized := filestore.SanitizeFilename(filename)
```

3. **Size Limits** - Enforce maximum file sizes
4. **Access Control** - Implement proper authorization checks
5. **Malware Scanning** - Consider integrating antivirus scanning
6. **Rate Limiting** - Implement rate limiting on upload endpoints

## Testing

Unit tests are included for all operations:

```bash
go test ./pkg/filestore -v
```

Test coverage includes:
- Upload operations
- Download operations
- Delete operations
- File validation
- Error handling
- Metadata storage and retrieval

## Troubleshooting

### MongoDB Connection Failed
```
Error: MongoDB connection not available
Solution: Ensure MongoDB is running and configured correctly
```

### File Not Found
```
Error: file not found
Solution: Verify the file ID is correct
```

### File Too Large
```
Error: file size exceeds maximum limit
Solution: Check MaxSize configuration and file size
```

### Invalid MIME Type
```
Error: unsupported file type
Solution: Verify the file type is in the allowed MIME types list
```

## Configuration

Configure file store limits in `bootstrap.go`:

```go
// Maximum file sizes
const (
    MaxUploadSize         = 100 * 1024 * 1024 // 100 MB
    MaxProfileImageSize   = 5 * 1024 * 1024   // 5 MB
)

// Supported MIME types
var AllowedImageTypes = map[string]bool{
    "image/jpeg":  true,
    "image/png":   true,
    "image/gif":   true,
    "image/webp":  true,
}
```

## Contributing

When extending the file store service:

1. Maintain backward compatibility
2. Add appropriate error handling
3. Update logging for operations
4. Include unit tests for new features
5. Update documentation

## License

This file store service is part of the Opplens backend system.
