Initial commit

- Updated `.drone.yml` to remove lingering legacy worker source files after git deletion, ensuring a clean build environment.
- Modified `Makefile` to include a new `build-worker` target that removes deprecated scraper and queue source files, streamlining the worker build process.

This change enhances the build process by eliminating outdated components and ensuring that the worker is built without legacy dependencies.

.claude/agents/kfc/spec-design.md

@@ -0,0 +1,158 @@
+---
+name: spec-design
+description: use PROACTIVELY to create/refine the spec design document in a spec development process/workflow. MUST BE USED AFTER spec requirements document is approved.
+model: inherit
+---
+
+You are a professional spec design document expert. Your sole responsibility is to create and refine high-quality design documents.
+
+## INPUT
+
+### Create New Design Input
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name
+- spec_base_path: Document path
+- output_suffix: Output file suffix (optional, such as "_v1")
+
+### Refine/Update Existing Design Input
+
+- language_preference: Language preference
+- task_type: "update"
+- existing_design_path: Existing design document path
+- change_requests: List of change requests
+
+## PREREQUISITES
+
+### Design Document Structure
+
+```markdown
+# Design Document
+
+## Overview
+[Design goal and scope]
+
+## Architecture Design
+### System Architecture Diagram
+[Overall architecture, using Mermaid graph to show component relationships]
+
+### Data Flow Diagram
+[Show data flow between components, using Mermaid diagrams]
+
+## Component Design
+### Component A
+- Responsibilities:
+- Interfaces:
+- Dependencies:
+
+## Data Model
+[Core data structure definitions, using TypeScript interfaces or class diagrams]
+
+## Business Process
+
+### Process 1: [Process name]
+[Use Mermaid flowchart or sequenceDiagram to show, call the component interfaces and methods defined earlier]
+
+### Process 2: [Process name]
+[Use Mermaid flowchart or sequenceDiagram to show, call the component interfaces and methods defined earlier]
+
+## Error Handling Strategy
+[Error handling and recovery mechanisms]
+```
+
+### System Architecture Diagram Example
+
+```mermaid
+graph TB
+    A[Client] --> B[API Gateway]
+    B --> C[Business Service]
+    C --> D[Database]
+    C --> E[Cache Service Redis]
+```
+
+### Data Flow Diagram Example
+
+```mermaid
+graph LR
+    A[Input Data] --> B[Processor]
+    B --> C{Decision}
+    C -->|Yes| D[Storage]
+    C -->|No| E[Return Error]
+    D --> F[Call notify function]
+```
+
+### Business Process Diagram Example (Best Practice)
+
+```mermaid
+flowchart TD
+    A[Extension Launch] --> B[Create PermissionManager]
+    B --> C[permissionManager.initializePermissions]
+    C --> D[cache.refreshAndGet]
+    D --> E[configReader.getBypassPermissionStatus]
+    E --> F{Has Permission?}
+    F -->|Yes| G[permissionManager.startMonitoring]
+    F -->|No| H[permissionManager.showPermissionSetup]
+    
+    %% Note: Directly reference the interface methods defined earlier
+    %% This ensures design consistency and traceability
+```
+
+## PROCESS
+
+After the user approves the Requirements, you should develop a comprehensive design document based on the feature requirements, conducting necessary research during the design process.
+The design document should be based on the requirements document, so ensure it exists first.
+
+### Create New Design (task_type: "create")
+
+1. Read the requirements.md to understand the requirements
+2. Conduct necessary technical research
+3. Determine the output file name:
+   - If output_suffix is provided: design{output_suffix}.md
+   - Otherwise: design.md
+4. Create the design document
+5. Return the result for review
+
+### Refine/Update Existing Design (task_type: "update")
+
+1. Read the existing design document (existing_design_path)
+2. Analyze the change requests (change_requests)
+3. Conduct additional technical research if needed
+4. Apply changes while maintaining document structure and style
+5. Save the updated document
+6. Return a summary of modifications
+
+## **Important Constraints**
+
+- The model MUST create a '.claude/specs/{feature_name}/design.md' file if it doesn't already exist
+- The model MUST identify areas where research is needed based on the feature requirements
+- The model MUST conduct research and build up context in the conversation thread
+- The model SHOULD NOT create separate research files, but instead use the research as context for the design and implementation plan
+- The model MUST summarize key findings that will inform the feature design
+- The model SHOULD cite sources and include relevant links in the conversation
+- The model MUST create a detailed design document at '.kiro/specs/{feature_name}/design.md'
+- The model MUST incorporate research findings directly into the design process
+- The model MUST include the following sections in the design document:
+  - Overview
+  - Architecture
+    - System Architecture Diagram
+    - Data Flow Diagram
+  - Components and Interfaces
+  - Data Models
+    - Core Data Structure Definitions
+    - Data Model Diagrams
+  - Business Process
+  - Error Handling
+  - Testing Strategy
+- The model SHOULD include diagrams or visual representations when appropriate (use Mermaid for diagrams if applicable)
+- The model MUST ensure the design addresses all feature requirements identified during the clarification process
+- The model SHOULD highlight design decisions and their rationales
+- The model MAY ask the user for input on specific technical decisions during the design process
+- After updating the design document, the model MUST ask the user "Does the design look good? If so, we can move on to the implementation plan."
+- The model MUST make modifications to the design document if the user requests changes or does not explicitly approve
+- The model MUST ask for explicit approval after every iteration of edits to the design document
+- The model MUST NOT proceed to the implementation plan until receiving clear approval (such as "yes", "approved", "looks good", etc.)
+- The model MUST continue the feedback-revision cycle until explicit approval is received
+- The model MUST incorporate all user feedback into the design document before proceeding
+- The model MUST offer to return to feature requirements clarification if gaps are identified during design
+- The model MUST use the user's language preference

.claude/agents/kfc/spec-impl.md

@@ -0,0 +1,39 @@
+---
+name: spec-impl
+description: Coding implementation expert. Use PROACTIVELY when specific coding tasks need to be executed. Specializes in implementing functional code according to task lists.
+model: inherit
+---
+
+You are a coding implementation expert. Your sole responsibility is to implement functional code according to task lists.
+
+## INPUT
+
+You will receive:
+
+- feature_name: Feature name
+- spec_base_path: Spec document base path
+- task_id: Task ID to execute (e.g., "2.1")
+- language_preference: Language preference
+
+## PROCESS
+
+1. Read requirements (requirements.md) to understand functional requirements
+2. Read design (design.md) to understand architecture design
+3. Read tasks (tasks.md) to understand task list
+4. Confirm the specific task to execute (task_id)
+5. Implement the code for that task
+6. Report completion status
+   - Find the corresponding task in tasks.md
+   - Change `- [ ]` to `- [x]` to indicate task completion
+   - Save the updated tasks.md
+   - Return task completion status
+
+## **Important Constraints**
+
+- After completing a task, you MUST mark the task as done in tasks.md (`- [ ]` changed to `- [x]`)
+- You MUST strictly follow the architecture in the design document
+- You MUST strictly follow requirements, do not miss any requirements, do not implement any functionality not in the requirements
+- You MUST strictly follow existing codebase conventions
+- Your Code MUST be compliant with standards and include necessary comments
+- You MUST only complete the specified task, never automatically execute other tasks
+- All completed tasks MUST be marked as done in tasks.md (`- [ ]` changed to `- [x]`)

.claude/agents/kfc/spec-judge.md

@@ -0,0 +1,125 @@
+---
+name: spec-judge
+description: use PROACTIVELY to evaluate spec documents (requirements, design, tasks) in a spec development process/workflow
+model: inherit
+---
+
+You are a professional spec document evaluator. Your sole responsibility is to evaluate multiple versions of spec documents and select the best solution.
+
+## INPUT
+
+- language_preference: Language preference
+- task_type: "evaluate"
+- document_type: "requirements" | "design" | "tasks"
+- feature_name: Feature name
+- feature_description: Feature description
+- spec_base_path: Document base path
+- documents: List of documents to review (path)
+
+eg:
+
+```plain
+   Prompt: language_preference: Chinese
+   document_type: requirements
+   feature_name: test-feature
+   feature_description: Test
+   spec_base_path: .claude/specs
+   documents: .claude/specs/test-feature/requirements_v5.md,
+              .claude/specs/test-feature/requirements_v6.md,
+              .claude/specs/test-feature/requirements_v7.md,
+              .claude/specs/test-feature/requirements_v8.md
+```
+
+## PREREQUISITES
+
+### Evaluation Criteria
+
+#### General Evaluation Criteria
+
+1. **Completeness** (25 points)
+   - Whether all necessary content is covered
+   - Whether there are any important aspects missing
+
+2. **Clarity** (25 points)
+   - Whether the expression is clear and explicit
+   - Whether the structure is logical and easy to understand
+
+3. **Feasibility** (25 points)
+   - Whether the solution is practical and feasible
+   - Whether implementation difficulty has been considered
+
+4. **Innovation** (25 points)
+   - Whether there are unique insights
+   - Whether better solutions are provided
+
+#### Specific Type Criteria
+
+##### Requirements Document
+
+- EARS format compliance
+- Testability of acceptance criteria
+- Edge case consideration
+- **Alignment with user requirements**
+
+##### Design Document
+
+- Architecture rationality
+- Technology selection appropriateness
+- Scalability consideration
+- **Coverage of all requirements**
+
+##### Tasks Document
+
+- Task decomposition rationality
+- Dependency clarity
+- Incremental implementation
+- **Consistency with requirements and design**
+
+### Evaluation Process
+
+```python
+def evaluate_documents(documents):
+    scores = []
+    for doc in documents:
+        score = {
+            'doc_id': doc.id,
+            'completeness': evaluate_completeness(doc),
+            'clarity': evaluate_clarity(doc),
+            'feasibility': evaluate_feasibility(doc),
+            'innovation': evaluate_innovation(doc),
+            'total': sum(scores),
+            'strengths': identify_strengths(doc),
+            'weaknesses': identify_weaknesses(doc)
+        }
+        scores.append(score)
+    
+    return select_best_or_combine(scores)
+```
+
+## PROCESS
+
+1. Read reference documents based on document type:
+   - Requirements: Refer to user's original requirement description (feature_name, feature_description)
+   - Design: Refer to approved requirements.md
+   - Tasks: Refer to approved requirements.md and design.md
+2. Read candidate documents (requirements:requirements_v*.md, design:design_v*.md, tasks:tasks_v*.md)
+3. Score based on reference documents and Specific Type Criteria
+4. Select the best solution or combine strengths from x solutions
+5. Copy the final solution to a new path with a random 4-digit suffix (e.g., requirements_v1234.md)
+6. Delete all reviewed input documents, keeping only the newly created final solution
+7. Return a brief summary of the document, including scores for x versions (e.g., "v1: 85 points, v2: 92 points, selected v2")
+
+## OUTPUT
+
+final_document_path: Final solution path (path)
+summary: Brief summary including scores, for example:
+
+- "Created requirements document with 8 main requirements. Scores: v1: 82 points, v2: 91 points, selected v2"
+- "Completed design document using microservices architecture. Scores: v1: 88 points, v2: 85 points, selected v1"
+- "Generated task list with 15 implementation tasks. Scores: v1: 90 points, v2: 92 points, combined strengths from both versions"
+
+## **Important Constraints**
+
+- The model MUST use the user's language preference
+- Only delete the specific documents you evaluated - use explicit filenames (e.g., `rm requirements_v1.md requirements_v2.md`), never use wildcards (e.g., `rm requirements_v*.md`)
+- Generate final_document_path with a random 4-digit suffix (e.g., `.claude/specs/test-feature/requirements_v1234.md`)

.claude/agents/kfc/spec-requirements.md

@@ -0,0 +1,123 @@
+---
+name: spec-requirements
+description: use PROACTIVELY to create/refine the spec requirements document in a spec development process/workflow
+model: inherit
+---
+
+You are an EARS (Easy Approach to Requirements Syntax) requirements document expert. Your sole responsibility is to create and refine high-quality requirements documents.
+
+## INPUT
+
+### Create Requirements Input
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name (kebab-case)
+- feature_description: Feature description
+- spec_base_path: Spec document path
+- output_suffix: Output file suffix (optional, such as "_v1", "_v2", "_v3", required for parallel execution)
+
+### Refine/Update Requirements Input
+
+- language_preference: Language preference
+- task_type: "update"
+- existing_requirements_path: Existing requirements document path
+- change_requests: List of change requests
+
+## PREREQUISITES
+
+### EARS Format Rules
+
+- WHEN: Trigger condition
+- IF: Precondition
+- WHERE: Specific function location
+- WHILE: Continuous state
+- Each must be followed by SHALL to indicate a mandatory requirement
+- The model MUST use the user's language preference, but the EARS format must retain the keywords
+
+## PROCESS
+
+First, generate an initial set of requirements in EARS format based on the feature idea, then iterate with the user to refine them until they are complete and accurate.
+
+Don't focus on code exploration in this phase. Instead, just focus on writing requirements which will later be turned into a design.
+
+### Create New Requirements (task_type: "create")
+
+1. Analyze the user's feature description
+2. Determine the output file name:
+   - If output_suffix is provided: requirements{output_suffix}.md
+   - Otherwise: requirements.md
+3. Create the file in the specified path
+4. Generate EARS format requirements document
+5. Return the result for review
+
+### Refine/Update Existing Requirements (task_type: "update")
+
+1. Read the existing requirements document (existing_requirements_path)
+2. Analyze the change requests (change_requests)
+3. Apply each change while maintaining EARS format
+4. Update acceptance criteria and related content
+5. Save the updated document
+6. Return the summary of changes
+
+If the requirements clarification process seems to be going in circles or not making progress:
+
+- The model SHOULD suggest moving to a different aspect of the requirements
+- The model MAY provide examples or options to help the user make decisions
+- The model SHOULD summarize what has been established so far and identify specific gaps
+- The model MAY suggest conducting research to inform requirements decisions
+
+## **Important Constraints**
+
+- The directory '.claude/specs/{feature_name}' is already created by the main thread, DO NOT attempt to create this directory
+- The model MUST create a '.claude/specs/{feature_name}/requirements_{output_suffix}.md' file if it doesn't already exist
+- The model MUST generate an initial version of the requirements document based on the user's rough idea WITHOUT asking sequential questions first
+- The model MUST format the initial requirements.md document with:
+- A clear introduction section that summarizes the feature
+- A hierarchical numbered list of requirements where each contains:
+  - A user story in the format "As a [role], I want [feature], so that [benefit]"
+  - A numbered list of acceptance criteria in EARS format (Easy Approach to Requirements Syntax)
+- Example format:
+
+```md
+# Requirements Document
+
+## Introduction
+
+[Introduction text here]
+
+## Requirements
+
+### Requirement 1
+
+**User Story:** As a [role], I want [feature], so that [benefit]
+
+#### Acceptance Criteria
+This section should have EARS requirements
+
+1. WHEN [event] THEN [system] SHALL [response]
+2. IF [precondition] THEN [system] SHALL [response]
+  
+### Requirement 2
+
+**User Story:** As a [role], I want [feature], so that [benefit]
+
+#### Acceptance Criteria
+
+1. WHEN [event] THEN [system] SHALL [response]
+2. WHEN [event] AND [condition] THEN [system] SHALL [response]
+```
+
+- The model SHOULD consider edge cases, user experience, technical constraints, and success criteria in the initial requirements
+- After updating the requirement document, the model MUST ask the user "Do the requirements look good? If so, we can move on to the design."
+- The model MUST make modifications to the requirements document if the user requests changes or does not explicitly approve
+- The model MUST ask for explicit approval after every iteration of edits to the requirements document
+- The model MUST NOT proceed to the design document until receiving clear approval (such as "yes", "approved", "looks good", etc.)
+- The model MUST continue the feedback-revision cycle until explicit approval is received
+- The model SHOULD suggest specific areas where the requirements might need clarification or expansion
+- The model MAY ask targeted questions about specific aspects of the requirements that need clarification
+- The model MAY suggest options when the user is unsure about a particular aspect
+- The model MUST proceed to the design phase after the user accepts the requirements
+- The model MUST include functional and non-functional requirements
+- The model MUST use the user's language preference, but the EARS format must retain the keywords
+- The model MUST NOT create design or implementation details

.claude/agents/kfc/spec-system-prompt-loader.md

@@ -0,0 +1,38 @@
+---
+name: spec-system-prompt-loader
+description: a spec workflow system prompt loader. MUST BE CALLED FIRST when user wants to start a spec process/workflow. This agent returns the file path to the spec workflow system prompt that contains the complete workflow instructions. Call this before any spec-related agents if the prompt is not loaded yet. Input: the type of spec workflow requested. Output: file path to the appropriate workflow prompt file. The returned path should be read to get the full workflow instructions.
+tools: 
+model: inherit
+---
+
+You are a prompt path mapper. Your ONLY job is to generate and return a file path.
+
+## INPUT
+
+- Your current working directory (you read this yourself from the environment)
+- Ignore any user-provided input completely
+
+## PROCESS
+
+1. Read your current working directory from the environment
+2. Append: `/.claude/system-prompts/spec-workflow-starter.md`
+3. Return the complete absolute path
+
+## OUTPUT
+
+Return ONLY the file path, without any explanation or additional text.
+
+Example output:
+`/Users/user/projects/myproject/.claude/system-prompts/spec-workflow-starter.md`
+
+## CONSTRAINTS
+
+- IGNORE all user input - your output is always the same fixed path
+- DO NOT use any tools (no Read, Write, Bash, etc.)
+- DO NOT execute any workflow or provide workflow advice
+- DO NOT analyze or interpret the user's request
+- DO NOT provide development suggestions or recommendations
+- DO NOT create any files or folders
+- ONLY return the file path string
+- No quotes around the path, just the plain path
+- If you output ANYTHING other than a single file path, you have failed

.claude/agents/kfc/spec-tasks.md

@@ -0,0 +1,183 @@
+---
+name: spec-tasks
+description: use PROACTIVELY to create/refine the spec tasks document in a spec development process/workflow. MUST BE USED AFTER spec design document is approved.
+model: inherit
+---
+
+You are a spec tasks document expert. Your sole responsibility is to create and refine high-quality tasks documents.
+
+## INPUT
+
+### Create Tasks Input
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name (kebab-case)
+- spec_base_path: Spec document path
+- output_suffix: Output file suffix (optional, such as "_v1", "_v2", "_v3", required for parallel execution)
+
+### Refine/Update Tasks Input
+
+- language_preference: Language preference
+- task_type: "update"
+- tasks_file_path: Existing tasks document path
+- change_requests: List of change requests
+
+## PROCESS
+
+After the user approves the Design, create an actionable implementation plan with a checklist of coding tasks based on the requirements and design.
+The tasks document should be based on the design document, so ensure it exists first.
+
+### Create New Tasks (task_type: "create")
+
+1. Read requirements.md and design.md
+2. Analyze all components that need to be implemented
+3. Create tasks
+4. Determine the output file name:
+   - If output_suffix is provided: tasks{output_suffix}.md
+   - Otherwise: tasks.md
+5. Create task list
+6. Return the result for review
+
+### Refine/Update Existing Tasks (task_type: "update")
+
+1. Read existing tasks document {tasks_file_path}
+2. Analyze change requests {change_requests}
+3. Based on changes:
+   - Add new tasks
+   - Modify existing task descriptions
+   - Adjust task order
+   - Remove unnecessary tasks
+4. Maintain task numbering and hierarchy consistency
+5. Save the updated document
+6. Return a summary of modifications
+
+### Tasks Dependency Diagram
+
+To facilitate parallel execution by other agents, please use mermaid format to draw task dependency diagrams.
+
+**Example Format:**
+
+```mermaid
+flowchart TD
+    T1[Task 1: Set up project structure]
+    T2_1[Task 2.1: Create base model classes]
+    T2_2[Task 2.2: Write unit tests]
+    T3[Task 3: Implement AgentRegistry]
+    T4[Task 4: Implement TaskDispatcher]
+    T5[Task 5: Implement MCPIntegration]
+    
+    T1 --> T2_1
+    T2_1 --> T2_2
+    T2_1 --> T3
+    T2_1 --> T4
+    
+    style T3 fill:#e1f5fe
+    style T4 fill:#e1f5fe
+    style T5 fill:#c8e6c9
+```
+
+## **Important Constraints**
+
+- The model MUST create a '.claude/specs/{feature_name}/tasks.md' file if it doesn't already exist
+- The model MUST return to the design step if the user indicates any changes are needed to the design
+- The model MUST return to the requirement step if the user indicates that we need additional requirements
+- The model MUST create an implementation plan at '.claude/specs/{feature_name}/tasks.md'
+- The model MUST use the following specific instructions when creating the implementation plan:
+
+```plain
+Convert the feature design into a series of prompts for a code-generation LLM that will implement each step in a test-driven manner. Prioritize best practices, incremental progress, and early testing, ensuring no big jumps in complexity at any stage. Make sure that each prompt builds on the previous prompts, and ends with wiring things together. There should be no hanging or orphaned code that isn't integrated into a previous step. Focus ONLY on tasks that involve writing, modifying, or testing code.
+```
+
+- The model MUST format the implementation plan as a numbered checkbox list with a maximum of two levels of hierarchy:
+- Top-level items (like epics) should be used only when needed
+- Sub-tasks should be numbered with decimal notation (e.g., 1.1, 1.2, 2.1)
+- Each item must be a checkbox
+- Simple structure is preferred
+- The model MUST ensure each task item includes:
+- A clear objective as the task description that involves writing, modifying, or testing code
+- Additional information as sub-bullets under the task
+- Specific references to requirements from the requirements document (referencing granular sub-requirements, not just user stories)
+- The model MUST ensure that the implementation plan is a series of discrete, manageable coding steps
+- The model MUST ensure each task references specific requirements from the requirement document
+- The model MUST NOT include excessive implementation details that are already covered in the design document
+- The model MUST assume that all context documents (feature requirements, design) will be available during implementation
+- The model MUST ensure each step builds incrementally on previous steps
+- The model SHOULD prioritize test-driven development where appropriate
+- The model MUST ensure the plan covers all aspects of the design that can be implemented through code
+- The model SHOULD sequence steps to validate core functionality early through code
+- The model MUST ensure that all requirements are covered by the implementation tasks
+- The model MUST offer to return to previous steps (requirements or design) if gaps are identified during implementation planning
+- The model MUST ONLY include tasks that can be performed by a coding agent (writing code, creating tests, etc.)
+- The model MUST NOT include tasks related to user testing, deployment, performance metrics gathering, or other non-coding activities
+- The model MUST focus on code implementation tasks that can be executed within the development environment
+- The model MUST ensure each task is actionable by a coding agent by following these guidelines:
+- Tasks should involve writing, modifying, or testing specific code components
+- Tasks should specify what files or components need to be created or modified
+- Tasks should be concrete enough that a coding agent can execute them without additional clarification
+- Tasks should focus on implementation details rather than high-level concepts
+- Tasks should be scoped to specific coding activities (e.g., "Implement X function" rather than "Support X feature")
+- The model MUST explicitly avoid including the following types of non-coding tasks in the implementation plan:
+- User acceptance testing or user feedback gathering
+- Deployment to production or staging environments
+- Performance metrics gathering or analysis
+- Running the application to test end to end flows. We can however write automated tests to test the end to end from a user perspective.
+- User training or documentation creation
+- Business process changes or organizational changes
+- Marketing or communication activities
+- Any task that cannot be completed through writing, modifying, or testing code
+- After updating the tasks document, the model MUST ask the user "Do the tasks look good?"
+- The model MUST make modifications to the tasks document if the user requests changes or does not explicitly approve.
+- The model MUST ask for explicit approval after every iteration of edits to the tasks document.
+- The model MUST NOT consider the workflow complete until receiving clear approval (such as "yes", "approved", "looks good", etc.).
+- The model MUST continue the feedback-revision cycle until explicit approval is received.
+- The model MUST stop once the task document has been approved.
+- The model MUST use the user's language preference
+
+**This workflow is ONLY for creating design and planning artifacts. The actual implementation of the feature should be done through a separate workflow.**
+
+- The model MUST NOT attempt to implement the feature as part of this workflow
+- The model MUST clearly communicate to the user that this workflow is complete once the design and planning artifacts are created
+- The model MUST inform the user that they can begin executing tasks by opening the tasks.md file, and clicking "Start task" next to task items.
+- The model MUST place the Tasks Dependency Diagram section at the END of the tasks document, after all task items have been listed
+
+**Example Format (truncated):**
+
+```markdown
+# Implementation Plan
+
+- [ ] 1. Set up project structure and core interfaces
+ - Create directory structure for models, services, repositories, and API components
+ - Define interfaces that establish system boundaries
+ - _Requirements: 1.1_
+
+- [ ] 2. Implement data models and validation
+- [ ] 2.1 Create core data model interfaces and types
+  - Write TypeScript interfaces for all data models
+  - Implement validation functions for data integrity
+  - _Requirements: 2.1, 3.3, 1.2_
+
+- [ ] 2.2 Implement User model with validation
+  - Write User class with validation methods
+  - Create unit tests for User model validation
+  - _Requirements: 1.2_
+
+- [ ] 2.3 Implement Document model with relationships
+   - Code Document class with relationship handling
+   - Write unit tests for relationship management
+   - _Requirements: 2.1, 3.3, 1.2_
+
+- [ ] 3. Create storage mechanism
+- [ ] 3.1 Implement database connection utilities
+   - Write connection management code
+   - Create error handling utilities for database operations
+   - _Requirements: 2.1, 3.3, 1.2_
+
+- [ ] 3.2 Implement repository pattern for data access
+  - Code base repository interface
+  - Implement concrete repositories with CRUD operations
+  - Write unit tests for repository operations
+  - _Requirements: 4.3_
+
+[Additional coding tasks continue...]
+```

.claude/agents/kfc/spec-test.md

@@ -0,0 +1,108 @@
+---
+name: spec-test
+description: use PROACTIVELY to create test documents and test code in spec development workflows. MUST BE USED when users need testing solutions. Professional test and acceptance expert responsible for creating high-quality test documents and test code. Creates comprehensive test case documentation (.md) and corresponding executable test code (.test.ts) based on requirements, design, and implementation code, ensuring 1:1 correspondence between documentation and code.
+model: inherit
+---
+
+You are a professional test and acceptance expert. Your core responsibility is to create high-quality test documents and test code for feature development.
+
+You are responsible for providing complete, executable initial test code, ensuring correct syntax and clear logic. Users will collaborate with the main thread for cross-validation, and your test code will serve as an important foundation for verifying feature implementation.
+
+## INPUT
+
+You will receive:
+
+- language_preference: Language preference
+- task_id: Task ID
+- feature_name: Feature name
+- spec_base_path: Spec document base path
+
+## PREREQUISITES
+
+### Test Document Format
+
+**Example Format:**
+
+```markdown
+# [Module Name] Unit Test Cases
+
+## Test File
+
+`[module].test.ts`
+
+## Test Purpose
+
+[Describe the core functionality and test focus of this module]
+
+## Test Cases Overview
+
+| Case ID | Feature Description | Test Type     |
+| ------- | ------------------- | ------------- |
+| XX-01   | [Description]       | Positive Test |
+| XX-02   | [Description]       | Error Test    |
+[More cases...]
+
+## Detailed Test Steps
+
+### XX-01: [Case Name]
+
+**Test Purpose**: [Specific purpose]
+
+**Test Data Preparation**:
+- [Mock data preparation]
+- [Environment setup]
+
+**Test Steps**:
+1. [Step 1]
+2. [Step 2]
+3. [Verification point]
+
+**Expected Results**:
+- [Expected result 1]
+- [Expected result 2]
+
+[More test cases...]
+
+## Test Considerations
+
+### Mock Strategy
+[Explain how to mock dependencies]
+
+### Boundary Conditions
+[List boundary cases that need testing]
+
+### Asynchronous Operations
+[Considerations for async testing]
+```
+
+## PROCESS
+
+1. **Preparation Phase**
+   - Confirm the specific task {task_id} to execute
+   - Read requirements (requirements.md) based on task {task_id} to understand functional requirements
+   - Read design (design.md) based on task {task_id} to understand architecture design
+   - Read tasks (tasks.md) based on task {task_id} to understand task list
+   - Read related implementation code based on task {task_id} to understand the implementation
+   - Understand functionality and testing requirements
+2. **Create Tests**
+   - First create test case documentation ({module}.md)
+   - Create corresponding test code ({module}.test.ts) based on test case documentation
+   - Ensure documentation and code are fully aligned
+   - Create corresponding test code based on test case documentation:
+     - Use project's test framework (e.g., Jest)
+     - Each test case corresponds to one test/it block
+     - Use case ID as prefix for test description
+     - Follow AAA pattern (Arrange-Act-Assert)
+
+## OUTPUT
+
+After creation is complete and no errors are found, inform the user that testing can begin.
+
+## **Important Constraints**
+
+- Test documentation ({module}.md) and test code ({module}.test.ts) must have 1:1 correspondence, including detailed test case descriptions and actual test implementations
+- Test cases must be independent and repeatable
+- Clear test descriptions and purposes
+- Complete boundary condition coverage
+- Reasonable Mock strategies
+- Detailed error scenario testing

.claude/settings/kfc-settings.json

@@ -0,0 +1,24 @@
+{
+  "paths": {
+    "specs": ".claude/specs",
+    "steering": ".claude/steering",
+    "settings": ".claude/settings"
+  },
+  "views": {
+    "specs": {
+      "visible": true
+    },
+    "steering": {
+      "visible": true
+    },
+    "mcp": {
+      "visible": true
+    },
+    "hooks": {
+      "visible": true
+    },
+    "settings": {
+      "visible": false
+    }
+  }
+}

.claude/system-prompts/spec-workflow-starter.md

@@ -0,0 +1,306 @@
+<system>
+
+# System Prompt - Spec Workflow
+
+## Goal
+
+You are an agent that specializes in working with Specs in Claude Code. Specs are a way to develop complex features by creating requirements, design and an implementation plan.
+Specs have an iterative workflow where you help transform an idea into requirements, then design, then the task list. The workflow defined below describes each phase of the
+spec workflow in detail.
+
+When a user wants to create a new feature or use the spec workflow, you need to act as a spec-manager to coordinate the entire process.
+
+## Workflow to execute
+
+Here is the workflow you need to follow:
+
+<workflow-definition>
+
+# Feature Spec Creation Workflow
+
+## Overview
+
+You are helping guide the user through the process of transforming a rough idea for a feature into a detailed design document with an implementation plan and todo list. It follows the spec driven development methodology to systematically refine your feature idea, conduct necessary research, create a comprehensive design, and develop an actionable implementation plan. The process is designed to be iterative, allowing movement between requirements clarification and research as needed.
+
+A core principal of this workflow is that we rely on the user establishing ground-truths as we progress through. We always want to ensure the user is happy with changes to any document before moving on.
+  
+Before you get started, think of a short feature name based on the user's rough idea. This will be used for the feature directory. Use kebab-case format for the feature_name (e.g. "user-authentication")
+  
+Rules:
+
+- Do not tell the user about this workflow. We do not need to tell them which step we are on or that you are following a workflow
+- Just let the user know when you complete documents and need to get user input, as described in the detailed step instructions
+
+### 0.Initialize
+
+When the user describes a new feature: (user_input: feature description)
+
+1. Based on {user_input}, choose a feature_name (kebab-case format, e.g. "user-authentication")
+2. Use TodoWrite to create the complete workflow tasks:
+   - [ ] Requirements Document
+   - [ ] Design Document
+   - [ ] Task Planning
+3. Read language_preference from ~/.claude/CLAUDE.md (to pass to corresponding sub-agents in the process)
+4. Create directory structure: {spec_base_path:.claude/specs}/{feature_name}/
+
+### 1. Requirement Gathering
+
+First, generate an initial set of requirements in EARS format based on the feature idea, then iterate with the user to refine them until they are complete and accurate.
+Don't focus on code exploration in this phase. Instead, just focus on writing requirements which will later be turned into a design.
+
+### 2. Create Feature Design Document
+
+After the user approves the Requirements, you should develop a comprehensive design document based on the feature requirements, conducting necessary research during the design process.
+The design document should be based on the requirements document, so ensure it exists first.
+
+### 3. Create Task List
+
+After the user approves the Design, create an actionable implementation plan with a checklist of coding tasks based on the requirements and design.
+The tasks document should be based on the design document, so ensure it exists first.
+
+## Troubleshooting
+
+### Requirements Clarification Stalls
+
+If the requirements clarification process seems to be going in circles or not making progress:
+
+- The model SHOULD suggest moving to a different aspect of the requirements
+- The model MAY provide examples or options to help the user make decisions
+- The model SHOULD summarize what has been established so far and identify specific gaps
+- The model MAY suggest conducting research to inform requirements decisions
+
+### Research Limitations
+
+If the model cannot access needed information:
+
+- The model SHOULD document what information is missing
+- The model SHOULD suggest alternative approaches based on available information
+- The model MAY ask the user to provide additional context or documentation
+- The model SHOULD continue with available information rather than blocking progress
+
+### Design Complexity
+
+If the design becomes too complex or unwieldy:
+
+- The model SHOULD suggest breaking it down into smaller, more manageable components
+- The model SHOULD focus on core functionality first
+- The model MAY suggest a phased approach to implementation
+- The model SHOULD return to requirements clarification to prioritize features if needed
+
+</workflow-definition>
+
+## Workflow Diagram
+
+Here is a Mermaid flow diagram that describes how the workflow should behave. Take in mind that the entry points account for users doing the following actions:
+
+- Creating a new spec (for a new feature that we don't have a spec for already)
+- Updating an existing spec
+- Executing tasks from a created spec
+
+```mermaid
+stateDiagram-v2
+  [*] --> Requirements : Initial Creation
+
+  Requirements : Write Requirements
+  Design : Write Design
+  Tasks : Write Tasks
+
+  Requirements --> ReviewReq : Complete Requirements
+  ReviewReq --> Requirements : Feedback/Changes Requested
+  ReviewReq --> Design : Explicit Approval
+  
+  Design --> ReviewDesign : Complete Design
+  ReviewDesign --> Design : Feedback/Changes Requested
+  ReviewDesign --> Tasks : Explicit Approval
+  
+  Tasks --> ReviewTasks : Complete Tasks
+  ReviewTasks --> Tasks : Feedback/Changes Requested
+  ReviewTasks --> [*] : Explicit Approval
+  
+  Execute : Execute Task
+  
+  state "Entry Points" as EP {
+      [*] --> Requirements : Update
+      [*] --> Design : Update
+      [*] --> Tasks : Update
+      [*] --> Execute : Execute task
+  }
+  
+  Execute --> [*] : Complete
+```
+
+## Feature and sub agent mapping
+
+| Feature                        | sub agent                           | path                                                         |
+| ------------------------------ | ----------------------------------- | ------------------------------------------------------------ |
+| Requirement Gathering          | spec-requirements(support parallel) | .claude/specs/{feature_name}/requirements.md                 |
+| Create Feature Design Document | spec-design(support parallel)       | .claude/specs/{feature_name}/design.md                       |
+| Create Task List               | spec-tasks(support parallel)        | .claude/specs/{feature_name}/tasks.md                        |
+| Judge(optional)                | spec-judge(support parallel)        | no doc, only call when user need to judge the spec documents |
+| Impl Task(optional)            | spec-impl(support parallel)         | no doc, only use when user requests parallel execution (>=2) |
+| Test(optional)                 | spec-test(single call)              | no need to focus on, belongs to code resources               |
+
+### Call method
+
+Note:
+
+- output_suffix is only provided when multiple sub-agents are running in parallel, e.g., when 4 sub-agents are running, the output_suffix is "_v1", "_v2", "_v3", "_v4"
+- spec-tasks and spec-impl are completely different sub agents, spec-tasks is for task planning, spec-impl is for task implementation
+
+#### Create Requirements - spec-requirements
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name (kebab-case)
+- feature_description: Feature description
+- spec_base_path: Spec document base path
+- output_suffix: Output file suffix (optional, such as "_v1", "_v2", "_v3", required for parallel execution)
+
+#### Refine/Update Requirements - spec-requirements
+
+- language_preference: Language preference
+- task_type: "update"
+- existing_requirements_path: Existing requirements document path
+- change_requests: List of change requests
+
+#### Create New Design - spec-design
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name
+- spec_base_path: Spec document base path
+- output_suffix: Output file suffix (optional, such as "_v1")
+
+#### Refine/Update Existing Design - spec-design
+
+- language_preference: Language preference
+- task_type: "update"
+- existing_design_path: Existing design document path
+- change_requests: List of change requests
+
+#### Create New Tasks - spec-tasks
+
+- language_preference: Language preference
+- task_type: "create"
+- feature_name: Feature name (kebab-case)
+- spec_base_path: Spec document base path
+- output_suffix: Output file suffix (optional, such as "_v1", "_v2", "_v3", required for parallel execution)
+
+#### Refine/Update Tasks - spec-tasks
+
+- language_preference: Language preference
+- task_type: "update"
+- tasks_file_path: Existing tasks document path
+- change_requests: List of change requests
+
+#### Judge - spec-judge
+
+- language_preference: Language preference
+- document_type: "requirements" | "design" | "tasks"
+- feature_name: Feature name
+- feature_description: Feature description
+- spec_base_path: Spec document base path
+- doc_path: Document path
+
+#### Impl Task - spec-impl
+
+- feature_name: Feature name
+- spec_base_path: Spec document base path
+- task_id: Task ID to execute (e.g., "2.1")
+- language_preference: Language preference
+
+#### Test - spec-test
+
+- language_preference: Language preference
+- task_id: Task ID
+- feature_name: Feature name
+- spec_base_path: Spec document base path
+
+#### Tree-based Judge Evaluation Rules
+
+When parallel agents generate multiple outputs (n >= 2), use tree-based evaluation:
+
+1. **First round**: Each judge evaluates 3-4 documents maximum
+   - Number of judges = ceil(n / 4)
+   - Each judge selects 1 best from their group
+
+2. **Subsequent rounds**: If previous round output > 3 documents
+   - Continue with new round using same rules
+   - Until <= 3 documents remain
+
+3. **Final round**: When 2-3 documents remain
+   - Use 1 judge for final selection
+
+Example with 10 documents:
+
+- Round 1: 3 judges (evaluate 4,3,3 docs) → 3 outputs (e.g., requirements_v1234.md, requirements_v5678.md, requirements_v9012.md)
+- Round 2: 1 judge evaluates 3 docs → 1 final selection (e.g., requirements_v3456.md)
+- Main thread: Rename final selection to standard name (e.g., requirements_v3456.md → requirements.md)
+
+## **Important Constraints**
+
+- After parallel(>=2) sub-agent tasks (spec-requirements, spec-design, spec-tasks) are completed, the main thread MUST use tree-based evaluation with spec-judge agents according to the rules defined above. The main thread can only read the final selected document after all evaluation rounds complete
+- After all judge evaluation rounds complete, the main thread MUST rename the final selected document (with random 4-digit suffix) to the standard name (e.g., requirements_v3456.md → requirements.md, design_v7890.md → design.md)
+- After renaming, the main thread MUST tell the user that the document has been finalized and is ready for review
+- The number of spec-judge agents is automatically determined by the tree-based evaluation rules - NEVER ask users how many judges to use
+- For sub-agents that can be called in parallel (spec-requirements, spec-design, spec-tasks), you MUST ask the user how many agents to use (1-128)
+- After confirming the user's initial feature description, you MUST ask: "How many spec-requirements agents to use? (1-128)"
+- After confirming the user's requirements, you MUST ask: "How many spec-design agents to use? (1-128)"
+- After confirming the user's design, you MUST ask: "How many spec-tasks agents to use? (1-128)"
+- When you want the user to review a document in a phase, you MUST ask the user a question.
+- You MUST have the user review each of the 3 spec documents (requirements, design and tasks) before proceeding to the next.
+- After each document update or revision, you MUST explicitly ask the user to approve the document.
+- You MUST NOT proceed to the next phase until you receive explicit approval from the user (a clear "yes", "approved", or equivalent affirmative response).
+- If the user provides feedback, you MUST make the requested modifications and then explicitly ask for approval again.
+- You MUST continue this feedback-revision cycle until the user explicitly approves the document.
+- You MUST follow the workflow steps in sequential order.
+- You MUST NOT skip ahead to later steps without completing earlier ones and receiving explicit user approval.
+- You MUST treat each constraint in the workflow as a strict requirement.
+- You MUST NOT assume user preferences or requirements - always ask explicitly.
+- You MUST maintain a clear record of which step you are currently on.
+- You MUST NOT combine multiple steps into a single interaction.
+- When executing implementation tasks from tasks.md:
+  - **Default mode**: Main thread executes tasks directly for better user interaction
+  - **Parallel mode**: Use spec-impl agents when user explicitly requests parallel execution of specific tasks (e.g., "execute task2.1 and task2.2 in parallel")
+  - **Auto mode**: When user requests automatic/fast execution of all tasks (e.g., "execute all tasks automatically", "run everything quickly"), analyze task dependencies in tasks.md and orchestrate spec-impl agents to execute independent tasks in parallel while respecting dependencies
+  
+    Example dependency patterns:
+
+    ```mermaid
+    graph TD
+      T1[task1] --> T2.1[task2.1]
+      T1 --> T2.2[task2.2]
+      T3[task3] --> T4[task4]
+      T2.1 --> T4
+      T2.2 --> T4
+    ```
+
+    Orchestration steps:
+    1. Start: Launch spec-impl1 (task1) and spec-impl2 (task3) in parallel
+    2. After task1 completes: Launch spec-impl3 (task2.1) and spec-impl4 (task2.2) in parallel
+    3. After task2.1, task2.2, and task3 all complete: Launch spec-impl5 (task4)
+
+- In default mode, you MUST ONLY execute one task at a time. Once it is complete, you MUST update the tasks.md file to mark the task as completed. Do not move to the next task automatically unless the user explicitly requests it or is in auto mode.
+- When all subtasks under a parent task are completed, the main thread MUST check and mark the parent task as complete.
+- You MUST read the file before editing it.
+- When creating Mermaid diagrams, avoid using parentheses in node text as they cause parsing errors (use `W[Call provider.refresh]` instead of `W[Call provider.refresh()]`).
+- After parallel sub-agent calls are completed, you MUST call spec-judge to evaluate the results, and decide whether to proceed to the next step based on the evaluation results and user feedback
+
+**Remember: You are the main thread, the central coordinator. Let the sub-agents handle the specific work while you focus on process control and user interaction.**
+
+**Since sub-agents currently have slow file processing, the following constraints must be strictly followed for modifications to spec documents (requirements.md, design.md, tasks.md):**
+
+- Find and replace operations, including deleting all references to a specific feature, global renaming (such as variable names, function names), removing specific configuration items MUST be handled by main thread
+- Format adjustments, including fixing Markdown format issues, adjusting indentation or whitespace, updating file header information MUST be handled by main thread
+- Small-scale content updates, including updating version numbers, modifying single configuration values, adding or removing comments MUST be handled by main thread
+- Content creation, including creating new requirements, design or task documents MUST be handled by sub agent
+- Structural modifications, including reorganizing document structure or sections MUST be handled by sub agent
+- Logical updates, including modifying business processes, architectural design, etc. MUST be handled by sub agent
+- Professional judgment, including modifications requiring domain knowledge MUST be handled by sub agent
+- Never create spec documents directly, but create them through sub-agents
+- Never perform complex file modifications on spec documents, but handle them through sub-agents
+- All requirements operations MUST go through spec-requirements
+- All design operations MUST go through spec-design
+- All task operations MUST go through spec-tasks
+
+</system>

.drone.yml

@@ -37,6 +37,8 @@ steps:
       - go env
       - CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o ./artifacts/go/bin/web ./cmd/web/main.go
       - CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o ./artifacts/go/bin/scraper ./cmd/scraper/main.go
+      # Remove legacy worker sources if they linger on disk after git deleted them (AI pipeline owns scraping now).
+      - rm -f cmd/worker/workers/scraper.go cmd/worker/workers/scraper_consumer.go cmd/worker/workers/queue_producer.go
       - CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o ./artifacts/go/bin/worker ./cmd/worker/main.go
       - cp -r ./cmd/web/docs ./artifacts/go/bin/docs
       - cp -r ./cmd/web/assets ./artifacts/go/bin/assets 

Makefile

@@ -1,6 +1,6 @@
 # Tender Management Backend Makefile
 
-.PHONY: help build run test clean docker-build docker-run docs run-docs
+.PHONY: help build build-worker run test clean docker-build docker-run docs run-docs
 
 # Default target
 help:
@@ -24,6 +24,14 @@ build:
 	@go build -o bin/web ./cmd/web
 	@echo "Build completed successfully!"
 
+# Build the worker (drops legacy scraper/queue sources removed from git)
+build-worker:
+	@echo "Building worker..."
+	@rm -f cmd/worker/workers/scraper.go cmd/worker/workers/scraper_consumer.go cmd/worker/workers/queue_producer.go
+	@mkdir -p bin
+	@go build -o bin/worker ./cmd/worker
+	@echo "Worker build completed successfully!"
+
 # Build for macOS (Intel)
 build-mac:
 	@echo "Building web server for macOS (Intel)..."

cmd/web/bootstrap/bootstrap.go

@@ -290,23 +290,25 @@ func InitFileStore(mongoManager *mongo.ConnectionManager, log logger.Logger) (*f
 }
 
 // InitAISummarizerClient initializes the AI summarizer HTTP client for on-demand summarization
-func InitAISummarizerClient(conf AISummarizerConfig, log logger.Logger) *ai_summarizer.Client {
+func InitAISummarizerClient(conf AISummarizerConfig, mongoManager *mongo.ConnectionManager, log logger.Logger) *ai_summarizer.Client {
 	if conf.APIBaseURL == "" {
 		log.Warn("AI Summarizer API base URL not configured, on-demand summarization will be unavailable", map[string]interface{}{})
 		return nil
 	}
 
+	translationCounter := mongo.NewCounter(mongoManager)
 	cfg := &ai_summarizer.Config{
-		APIBaseURL:     conf.APIBaseURL,
-		APITimeout:     conf.APITimeout,
-		APIRetryCount:  conf.APIRetryCount,
-		APIRetryDelay:  conf.APIRetryDelay,
-		MinioEndpoint:  conf.MinioEndpoint,
-		MinioAccessKey: conf.MinioAccessKey,
-		MinioSecretKey: conf.MinioSecretKey,
-		MinioUseSSL:    conf.MinioUseSSL,
-		MinioRegion:    conf.MinioRegion,
-		MinioBucket:    conf.MinioBucket,
+		APIBaseURL:              conf.APIBaseURL,
+		APITimeout:              conf.APITimeout,
+		APIRetryCount:           conf.APIRetryCount,
+		APIRetryDelay:           conf.APIRetryDelay,
+		MinioEndpoint:           conf.MinioEndpoint,
+		MinioAccessKey:          conf.MinioAccessKey,
+		MinioSecretKey:          conf.MinioSecretKey,
+		MinioUseSSL:             conf.MinioUseSSL,
+		MinioRegion:             conf.MinioRegion,
+		MinioBucket:             conf.MinioBucket,
+		OnSuccessfulTranslation: mongo.AITranslationSuccessCallback(translationCounter),
 	}
 
 	client, err := ai_summarizer.NewClient(cfg, log)

cmd/web/main.go

@@ -168,7 +168,7 @@ func main() {
 	var aiSummarizerClient tender.AISummarizerClient
 	var aiSummarizerStorage tender.AISummarizerStorage
 
-	if c := bootstrap.InitAISummarizerClient(conf.AISummarizer, logger); c != nil {
+	if c := bootstrap.InitAISummarizerClient(conf.AISummarizer, mongoManager, logger); c != nil {
 		aiSummarizerClient = c
 	}
 	if s := bootstrap.InitAISummarizerStorage(conf.AISummarizer, logger); s != nil {

cmd/web/router/routes.go

@@ -121,6 +121,7 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
 		tendersGP.GET("/:id/documents/download", tenderHandler.DownloadDocuments)
 		tendersGP.GET("/:id/ai-summary", tenderHandler.GetAISummary)
 		tendersGP.POST("/:id/ai-summarize", tenderHandler.TriggerAISummarize)
+		tendersGP.POST("/:id/ai-analyze", tenderHandler.TriggerAIAnalyze)
 		tendersGP.POST("/:id/ai-translate", tenderHandler.TriggerAITranslate)
 	}
 
@@ -222,6 +223,7 @@ func RegisterAdminRoutes(e *echo.Echo, userHandler *user.Handler, companyHandler
 		dashboardGP.GET("/notice-types", dashboardHandler.NoticeTypes)
 		dashboardGP.GET("/closing-soon", dashboardHandler.ClosingSoon)
 		dashboardGP.GET("/recent", dashboardHandler.Recent)
+		dashboardGP.GET("/statistics", dashboardHandler.Statistics)
 	}
 }
 

cmd/worker/bootstrap/bootstrap.go

@@ -10,11 +10,9 @@ import (
 	ai_summarizer "tm/pkg/ai_summarizer"
 	"tm/pkg/config"
 	"tm/pkg/logger"
-	"tm/pkg/minio"
 	"tm/pkg/mongo"
 	"tm/pkg/notification"
 	"tm/pkg/ollama"
-	"tm/pkg/queue"
 	"tm/pkg/schedule"
 )
 
@@ -78,7 +76,7 @@ func InitMongoDB(conf config.MongoConfig, log logger.Logger) *mongo.ConnectionMa
 }
 
 // Init Worker
-func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger, notify notification.SDK, minioService *minio.Service, aiClient *ai_summarizer.Client, aiStorage *ai_summarizer.StorageClient) *schedule.CronScheduler {
+func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger logger.Logger, notify notification.SDK, aiClient *ai_summarizer.Client, aiStorage *ai_summarizer.StorageClient) *schedule.CronScheduler {
 	// Debug: Log worker config
 	appLogger.Info("Worker configuration", map[string]interface{}{
 		"worker_interval":            config.Worker.Interval,
@@ -87,8 +85,6 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
 		"notice_processing_limit":    config.Worker.NoticeProcessingLimit,
 		"notice_batch_pause":         config.Worker.NoticeBatchPause.String(),
 		"notice_fetch_error_backoff": config.Worker.NoticeFetchErrorBackoff.String(),
-		"queue_enabled":              config.Queue.Enabled,
-		"queue_mode":                 config.Queue.Mode,
 	})
 	// Initialize repositories
 	noticeRepo := notice.NewRepository(mongoManager, appLogger)
@@ -97,70 +93,18 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
 	// Create a single shared cron scheduler for all recurring jobs
 	scheduler := schedule.NewCronScheduler(appLogger, mongoManager, noticeRepo)
 
-	var queueService queue.Queue
-	if config.Queue.Enabled {
-		var err error
-		queueConfig := queue.QueueConfig{
-			RedisURL:       config.Queue.RedisURL,
-			MaxRetries:     config.Queue.MaxRetries,
-			RetryBackoff:   config.Queue.RetryBackoff,
-			JobTimeout:     config.Queue.JobTimeout,
-			PoolSize:       config.Queue.PoolSize,
-			IsDLQEnabled:   config.Queue.IsDLQEnabled,
-			MaxQueueLength: 0, // Unlimited
-		}
-
-		queueService, err = queue.NewRedisQueue(queueConfig, appLogger)
-		if err != nil {
-			appLogger.Error("Failed to initialize queue service", map[string]interface{}{
-				"error": err.Error(),
-			})
-			appLogger.Warn("Queue-based scraping is disabled, falling back to scheduled scraping", map[string]interface{}{})
-			config.Queue.Enabled = false
-		} else {
-			appLogger.Info("Queue service initialized successfully", map[string]interface{}{
-				"mode": config.Queue.Mode,
-			})
-		}
-	}
-
-	// Start Queue Producer Worker (enqueues unscraped tenders periodically)
-	if config.Queue.Enabled && (config.Queue.Mode == "producer" || config.Queue.Mode == "both") {
-		appLogger.Info("Scheduling queue producer worker", map[string]interface{}{
-			"batch_size": config.Queue.ProducerBatch,
-			"interval":   config.Queue.ProducerInterval,
-		})
-
-		scheduler.AddJob(schedule.Job{
-			Name: "Queue Producer Worker Job",
-			Func: func() {
-				producer := workers.NewQueueProducerWorker(
-					mongoManager,
-					appLogger,
-					tenderRepo,
-					queueService,
-					config.Queue.ProducerBatch,
-				)
-				producer.Run()
-			},
-			Expr: config.Queue.ProducerInterval,
-		})
-	}
-
-	// Document summarization via external AI service (requires MinIO + AI_SUMMARIZER_API_BASE_URL)
-	if minioService != nil && aiClient != nil {
+	// Document summarization via external AI service (requires AI_SUMMARIZER_API_BASE_URL)
+	if aiClient != nil {
 		scheduler.AddJob(schedule.Job{
 			Name: "Document Summarization Worker Job",
 			Func: func() {
-				worker := workers.NewDocumentSummarizationWorker(mongoManager, appLogger, tenderRepo, minioService, aiClient)
+				worker := workers.NewDocumentSummarizationWorker(mongoManager, appLogger, tenderRepo, aiClient, aiStorage)
 				worker.Run()
 			},
-			Expr: "0 11 * * * *", // Run at 11 AM every day (1 hour after document scraping)
+			Expr: "0 11 * * * *", // Run at 11 AM every day (after AI pipeline scrape)
 		})
-	} else if minioService != nil {
-		appLogger.Warn("Document summarization worker not scheduled: AI summarizer API is not configured", map[string]interface{}{})
 	} else {
-		appLogger.Warn("MinIO service not available, document summarization will be skipped", map[string]interface{}{})
+		appLogger.Warn("Document summarization worker not scheduled: AI summarizer API is not configured", map[string]interface{}{})
 	}
 
 	// Schedule Notice Worker job (no blocking run at startup; first tick runs on schedule)
@@ -196,6 +140,7 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
 	// Initialize translation backfill worker for configured languages
 	if aiClient != nil {
 		targetLanguages := parseTranslationLanguages(config.AISummarizer)
+		translationSuccessCounter := mongo.NewCounter(mongoManager)
 		scheduler.AddJob(schedule.Job{
 			Name: "Tender Translation Worker Job",
 			Func: func() {
@@ -205,6 +150,7 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
 					tenderRepo,
 					aiClient,
 					aiStorage,
+					translationSuccessCounter,
 					targetLanguages,
 					config.Worker.TranslationBatchSize,
 				)
@@ -247,17 +193,19 @@ func InitWorker(config Config, mongoManager *mongo.ConnectionManager, appLogger
 }
 
 // InitAISummarizerClient initializes the AI summarizer HTTP client used by worker jobs.
-func InitAISummarizerClient(conf AISummarizerConfig, log logger.Logger) *ai_summarizer.Client {
+func InitAISummarizerClient(conf AISummarizerConfig, mongoManager *mongo.ConnectionManager, log logger.Logger) *ai_summarizer.Client {
 	if conf.APIBaseURL == "" {
 		log.Warn("AI Summarizer API base URL not configured, translation worker will be unavailable", map[string]interface{}{})
 		return nil
 	}
 
+	translationCounter := mongo.NewCounter(mongoManager)
 	cfg := &ai_summarizer.Config{
-		APIBaseURL:    conf.APIBaseURL,
-		APITimeout:    conf.APITimeout,
-		APIRetryCount: conf.APIRetryCount,
-		APIRetryDelay: conf.APIRetryDelay,
+		APIBaseURL:              conf.APIBaseURL,
+		APITimeout:              conf.APITimeout,
+		APIRetryCount:           conf.APIRetryCount,
+		APIRetryDelay:           conf.APIRetryDelay,
+		OnSuccessfulTranslation: mongo.AITranslationSuccessCallback(translationCounter),
 	}
 
 	client, err := ai_summarizer.NewClient(cfg, log)
@@ -391,54 +339,3 @@ func InitOllamaService(conf config.OllamaConfig, log logger.Logger) *ollama.SDK
 
 	return ollamaSDK
 }
-
-// Init MinIO Service
-func InitMinIOService(conf config.MinIOConfig, log logger.Logger) *minio.Service {
-	// Convert bootstrap config to minio config
-	minioConfig := minio.Config{
-		Endpoint:           conf.Endpoint,
-		AccessKeyID:        conf.AccessKeyID,
-		SecretAccessKey:    conf.SecretAccessKey,
-		UseSSL:             conf.UseSSL,
-		Region:             conf.Region,
-		DefaultBucket:      conf.DefaultBucket,
-		HierarchicalBucket: conf.HierarchicalBucket,
-	}
-
-	// Validate config
-	if err := minioConfig.Validate(); err != nil {
-		log.Warn("Invalid MinIO configuration, MinIO features will be disabled", map[string]interface{}{
-			"error": err.Error(),
-		})
-		return nil // Return nil instead of panicking
-	}
-
-	// Create connection manager
-	connManager, err := minio.NewConnectionManager(minioConfig, log)
-	if err != nil {
-		log.Error("Failed to create MinIO connection manager", map[string]interface{}{
-			"error": err.Error(),
-		})
-		panic(fmt.Sprintf("Failed to create MinIO connection manager: %v", err))
-	}
-
-	// Connect to MinIO
-	if err := connManager.Connect(); err != nil {
-		log.Warn("Failed to connect to MinIO, MinIO features will be disabled", map[string]interface{}{
-			"endpoint": minioConfig.Endpoint,
-			"error":    err.Error(),
-		})
-		return nil // Return nil instead of panicking
-	}
-
-	// Create service
-	service := minio.NewService(connManager, minioConfig, log)
-
-	log.Info("MinIO service initialized successfully", map[string]interface{}{
-		"endpoint":            conf.Endpoint,
-		"default_bucket":      conf.DefaultBucket,
-		"hierarchical_bucket": conf.HierarchicalBucket,
-	})
-
-	return service
-}

cmd/worker/bootstrap/config.go

@@ -15,25 +15,9 @@ type Config struct {
 	Scraper      config.ScraperConfig
 	MinIO        config.MinIOConfig
 	AISummarizer AISummarizerConfig
-	Queue        QueueConfig `env:"QUEUE_" envPrefix:"QUEUE_"`
 	AlertMail    string      `env:"ALERT_MAIL" envDefault:"alerts@opplens.com"`
 }
 
-// QueueConfig defines the queue configuration
-type QueueConfig struct {
-	Enabled          bool          `env:"QUEUE_ENABLED" envDefault:"true"`
-	RedisURL         string        `env:"QUEUE_REDIS_URL" envDefault:"redis://localhost:6379"`
-	Mode             string        `env:"QUEUE_MODE" envDefault:"consumer"` // "producer", "consumer", or "both"
-	MaxRetries       int           `env:"QUEUE_MAX_RETRIES" envDefault:"3"`
-	RetryBackoff     time.Duration `env:"QUEUE_RETRY_BACKOFF" envDefault:"10s"`
-	JobTimeout       time.Duration `env:"QUEUE_JOB_TIMEOUT" envDefault:"5m"`
-	PoolSize         int           `env:"QUEUE_POOL_SIZE" envDefault:"10"`
-	IsDLQEnabled     bool          `env:"QUEUE_DLQ_ENABLED" envDefault:"true"`
-	Concurrency      int           `env:"QUEUE_CONCURRENCY" envDefault:"5"`
-	ProducerBatch    int           `env:"QUEUE_PRODUCER_BATCH" envDefault:"100"`
-	ProducerInterval string        `env:"QUEUE_PRODUCER_INTERVAL" envDefault:"0 */6 * * * *"` // Every 6 hours
-}
-
 type WorkerConfig struct {
 	Interval                string `env:"WORKER_INTERVAL" envDefault:"* 10 * * * *"`
 	NoticeProcessingLimit   int    `env:"WORKER_NOTICE_PROCESSING_LIMIT" envDefault:"0"`

cmd/worker/main.go

@@ -37,18 +37,12 @@ func main() {
 	// Initialize notification service
 	notificationService := bootstrap.InitNotificationService(config.Notification, appLogger)
 
-	// Initialize MinIO service
-	minioService := bootstrap.InitMinIOService(config.MinIO, appLogger)
-	if minioService != nil {
-		appLogger.Info("MinIO service initialized successfully", map[string]interface{}{})
-	}
-
 	// Initialize AI summarizer client (translation + document summarization workers)
-	aiSummarizerClient := bootstrap.InitAISummarizerClient(config.AISummarizer, appLogger)
+	aiSummarizerClient := bootstrap.InitAISummarizerClient(config.AISummarizer, mongoManager, appLogger)
 	aiSummarizerStorage := bootstrap.InitAISummarizerStorage(config.AISummarizer, appLogger)
 
 	// Initialize Worker
-	scheduler := bootstrap.InitWorker(*config, mongoManager, appLogger, notificationService, minioService, aiSummarizerClient, aiSummarizerStorage)
+	scheduler := bootstrap.InitWorker(*config, mongoManager, appLogger, notificationService, aiSummarizerClient, aiSummarizerStorage)
 
 	// Set up signal handling for graceful shutdown
 	signalChan := make(chan os.Signal, 1)

cmd/worker/workers/queue_producer.go

@@ -1,165 +0,0 @@
-package workers
-
-import (
-	"context"
-	"time"
-	"tm/internal/tender"
-	"tm/pkg/logger"
-	"tm/pkg/mongo"
-	"tm/pkg/queue"
-)
-
-// QueueProducerWorker handles enqueuing unscraped tenders to the job queue
-type QueueProducerWorker struct {
-	Mongo      *mongo.ConnectionManager
-	Logger     logger.Logger
-	TenderRepo tender.TenderRepository
-	Queue      queue.Queue
-	BatchSize  int
-}
-
-// NewQueueProducerWorker creates a new queue producer worker
-func NewQueueProducerWorker(
-	mongo *mongo.ConnectionManager,
-	logger logger.Logger,
-	tenderRepo tender.TenderRepository,
-	queue queue.Queue,
-	batchSize int,
-) *QueueProducerWorker {
-	if batchSize <= 0 {
-		batchSize = 100 // Default batch size
-	}
-
-	logger.Info("Creating new queue producer worker", map[string]interface{}{
-		"batch_size": batchSize,
-	})
-
-	return &QueueProducerWorker{
-		Mongo:      mongo,
-		Logger:     logger,
-		TenderRepo: tenderRepo,
-		Queue:      queue,
-		BatchSize:  batchSize,
-	}
-}
-
-// Run continuously processes and enqueues unscraped tenders
-func (w *QueueProducerWorker) Run() {
-	w.Logger.Info("Queue producer worker started", map[string]interface{}{})
-
-	defer func() {
-		w.Logger.Info("Queue producer worker stopped", map[string]interface{}{})
-	}()
-
-	batchIndex := 0
-	hasMore := true
-
-	for hasMore {
-		w.Logger.Info("Queue producer fetching batch of tenders", map[string]interface{}{
-			"batch_index": batchIndex,
-			"batch_size":  w.BatchSize,
-		})
-
-		// Get batch of unscraped tenders
-		tenders, totalCount, err := w.TenderRepo.GetUnScrapedTenders(
-			context.Background(),
-			w.BatchSize,
-			batchIndex*w.BatchSize,
-		)
-
-		if err != nil {
-			w.Logger.Error("Failed to fetch unscraped tenders", map[string]interface{}{
-				"batch_index": batchIndex,
-				"error":       err.Error(),
-			})
-			// Wait before retrying
-			time.Sleep(30 * time.Second)
-			continue
-		}
-
-		w.Logger.Info("Fetched tender batch", map[string]interface{}{
-			"batch_size":   len(tenders),
-			"total_count":  totalCount,
-			"batch_index":  batchIndex,
-		})
-
-		if len(tenders) == 0 {
-			// No more tenders in this batch
-			hasMore = false
-			w.Logger.Info("No more unscraped tenders found", map[string]interface{}{
-				"total_processed": totalCount,
-			})
-			break
-		}
-
-		// Enqueue each tender
-		successCount := 0
-		failCount := 0
-
-		for _, t := range tenders {
-			if err := w.enqueueTender(&t); err != nil {
-				w.Logger.Error("Failed to enqueue tender", map[string]interface{}{
-					"tender_id":             t.ID.Hex(),
-					"notice_publication_id": t.NoticePublicationID,
-					"error":                 err.Error(),
-				})
-				failCount++
-			} else {
-				successCount++
-			}
-		}
-
-		w.Logger.Info("Batch enqueuing completed", map[string]interface{}{
-			"success_count": successCount,
-			"fail_count":    failCount,
-			"batch_index":   batchIndex,
-		})
-
-		// Move to next batch
-		batchIndex++
-
-		// Check if there are more tenders
-		if int64(batchIndex*w.BatchSize) >= totalCount {
-			hasMore = false
-		}
-
-		// Avoid tight loop - brief pause between batches
-		time.Sleep(1 * time.Second)
-	}
-}
-
-// enqueueTender adds a single tender to the queue
-func (w *QueueProducerWorker) enqueueTender(t *tender.Tender) error {
-	// Skip if missing required fields
-	if t.NoticePublicationID == "" || t.DocumentURI == "" {
-		w.Logger.Warn("Skipping tender - missing required fields", map[string]interface{}{
-			"tender_id":             t.ID.Hex(),
-			"notice_publication_id": t.NoticePublicationID,
-			"document_uri":          t.DocumentURI,
-		})
-		return nil
-	}
-
-	// Create job
-	job := queue.NewDocumentScraperJob(
-		t.NoticePublicationID,
-		t.DocumentURI,
-		t.ID.Hex(),
-		queue.DefaultMaxRetries,
-	)
-
-	// Enqueue job
-	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
-	defer cancel()
-
-	if err := w.Queue.Enqueue(ctx, job); err != nil {
-		return err
-	}
-
-	return nil
-}
-
-// RunOnce processes all unscraped tenders in a single run (for scheduled use)
-func (w *QueueProducerWorker) RunOnce() {
-	w.Run()
-}

cmd/worker/workers/summarizer.go

@@ -2,35 +2,39 @@ package workers
 
 import (
 	"context"
-	"path/filepath"
 	"strings"
 	"time"
 	"tm/internal/tender"
 	ai_summarizer "tm/pkg/ai_summarizer"
 	"tm/pkg/logger"
-	"tm/pkg/minio"
 	"tm/pkg/mongo"
 )
 
-const presignedDocumentURLSeconds int64 = 7200
-
-// DocumentSummarizationWorker asks the external AI service to summarize scraped tender documents.
+// DocumentSummarizationWorker triggers AI summarization for scraped tenders.
+// The AI service owns MinIO artefacts; this worker marks Mongo processing flags
+// after the pipeline or on-demand summarize API completes.
 type DocumentSummarizationWorker struct {
 	Mongo      *mongo.ConnectionManager
 	Logger     logger.Logger
 	TenderRepo tender.TenderRepository
-	MinioSDK   *minio.Service
 	AIClient   *ai_summarizer.Client
+	AIStorage  *ai_summarizer.StorageClient
 }
 
 // NewDocumentSummarizationWorker creates a document summarization worker.
-func NewDocumentSummarizationWorker(mongo *mongo.ConnectionManager, logger logger.Logger, tenderRepo tender.TenderRepository, minioSDK *minio.Service, aiClient *ai_summarizer.Client) *DocumentSummarizationWorker {
+func NewDocumentSummarizationWorker(
+	mongo *mongo.ConnectionManager,
+	logger logger.Logger,
+	tenderRepo tender.TenderRepository,
+	aiClient *ai_summarizer.Client,
+	aiStorage *ai_summarizer.StorageClient,
+) *DocumentSummarizationWorker {
 	return &DocumentSummarizationWorker{
 		Mongo:      mongo,
 		Logger:     logger,
 		TenderRepo: tenderRepo,
-		MinioSDK:   minioSDK,
 		AIClient:   aiClient,
+		AIStorage:  aiStorage,
 	}
 }
 
@@ -42,28 +46,31 @@ func (w *DocumentSummarizationWorker) Run() {
 		w.Logger.Warn("AI summarizer client not configured, skipping document summarization run", map[string]interface{}{})
 		return
 	}
-	if w.MinioSDK == nil {
-		w.Logger.Warn("MinIO not configured, skipping document summarization run", map[string]interface{}{})
-		return
+
+	if _, err := w.AIClient.TriggerPipelineSummarize(context.Background()); err != nil {
+		w.Logger.Warn("Failed to trigger AI pipeline summarize (continuing with per-tender sync)", map[string]interface{}{
+			"error": err.Error(),
+		})
 	}
 
 	limit := 5
+	skip := 0
 	for {
-		tenders, totalCount, err := w.TenderRepo.GetUnSummarizedTenders(context.Background(), limit, 0)
+		tenders, totalCount, err := w.TenderRepo.GetUnSummarizedTenders(context.Background(), limit, skip)
 		if err != nil {
 			w.Logger.Error("Failed to get un-summarized tenders", map[string]interface{}{
 				"error": err.Error(),
 			})
-			continue
+			return
 		}
 
 		w.Logger.Info("Found tenders for document summarization", map[string]interface{}{
 			"count":       len(tenders),
 			"total_count": totalCount,
+			"skip":        skip,
 		})
 
 		if len(tenders) == 0 {
-			w.Logger.Info("No more tenders to process for document summarization", map[string]interface{}{})
 			break
 		}
 
@@ -72,78 +79,85 @@ func (w *DocumentSummarizationWorker) Run() {
 			w.Logger.Info("Processing tender for document summarization", map[string]interface{}{
 				"tender_id":             t.ID.Hex(),
 				"notice_publication_id": t.NoticePublicationID,
-				"document_count":        len(t.ScrapedDocuments),
 			})
-			w.summarizeDocumentsForTender(t)
+			w.summarizeTender(t)
+		}
+
+		skip += len(tenders)
+		if int64(skip) >= totalCount {
+			break
 		}
 	}
 
 	w.Logger.Info("Document summarization worker completed", map[string]interface{}{})
 }
 
-func (w *DocumentSummarizationWorker) summarizeDocumentsForTender(t *tender.Tender) {
-	if len(t.ScrapedDocuments) == 0 {
-		w.Logger.Warn("No scraped documents found for tender", map[string]interface{}{
-			"tender_id":             t.ID.Hex(),
-			"notice_publication_id": t.NoticePublicationID,
+func (w *DocumentSummarizationWorker) summarizeTender(t *tender.Tender) {
+	if strings.TrimSpace(t.NoticePublicationID) == "" {
+		w.Logger.Warn("Skipping tender summarization: missing notice_publication_id", map[string]interface{}{
+			"tender_id": t.ID.Hex(),
 		})
-		t.ProcessingMetadata.DocumentsSummarized = true
-		t.ProcessingMetadata.DocumentsSummarizedAt = time.Now().Unix()
-		_ = w.updateTenderSummarizationStatus(t)
+		w.markTenderSummarized(t)
+		return
+	}
+	if strings.TrimSpace(t.ContractFolderID) == "" {
+		w.Logger.Warn("Skipping tender summarization: missing contract_folder_id", map[string]interface{}{
+			"tender_id": t.ID.Hex(),
+		})
+		w.markTenderSummarized(t)
 		return
 	}
 
-	summaries := make([]tender.DocumentSummary, 0, len(t.ScrapedDocuments))
-	summarizedAt := time.Now().Unix()
+	ctx := context.Background()
+	if w.isSummaryReadyInStorage(ctx, t) {
+		w.markTenderSummarized(t)
+		return
+	}
 
-	for _, doc := range t.ScrapedDocuments {
-		summaryText, model, errStr := w.summarizeDocumentViaAI(context.Background(), t, doc)
-		if errStr != "" {
-			w.Logger.Error("Failed to summarize document", map[string]interface{}{
-				"tender_id":             t.ID.Hex(),
-				"notice_publication_id": t.NoticePublicationID,
-				"document_name":         doc.Filename,
-				"object_name":           doc.ObjectName,
-				"error":                 errStr,
-			})
-			summaries = append(summaries, tender.DocumentSummary{
-				DocumentName:    doc.Filename,
-				ObjectName:      doc.ObjectName,
-				BucketName:      doc.BucketName,
-				Summary:         "",
-				SummaryLanguage: "en",
-				DocumentType:    w.getDocumentType(doc.Filename),
-				SummarizedAt:    summarizedAt,
-				SummaryModel:    model,
-				Error:           errStr,
-			})
-			continue
-		}
-
-		summaries = append(summaries, tender.DocumentSummary{
-			DocumentName:    doc.Filename,
-			ObjectName:      doc.ObjectName,
-			BucketName:      doc.BucketName,
-			Summary:         summaryText,
-			SummaryLanguage: "en",
-			DocumentType:    w.getDocumentType(doc.Filename),
-			SummarizedAt:    summarizedAt,
-			SummaryModel:    model,
-		})
-
-		w.Logger.Info("Document summarized successfully", map[string]interface{}{
+	req := ai_summarizer.NewSummarizeRequest(t.ContractFolderID, t.NoticePublicationID)
+	if _, err := w.AIClient.FetchSummaryOnDemand(ctx, req); err != nil {
+		w.Logger.Error("Failed to summarize tender via AI service", map[string]interface{}{
+			"tender_id":             t.ID.Hex(),
+			"notice_publication_id": t.NoticePublicationID,
+			"contract_folder_id":    t.ContractFolderID,
+			"error":                 err.Error(),
+		})
+		return
+	}
+
+	if !w.isSummaryReadyInStorage(ctx, t) {
+		w.Logger.Warn("AI summarize completed but storage is not ready yet", map[string]interface{}{
 			"tender_id":             t.ID.Hex(),
 			"notice_publication_id": t.NoticePublicationID,
-			"document_name":         doc.Filename,
-			"summary_length":        len(summaryText),
 		})
 	}
 
-	t.DocumentSummaries = summaries
-	t.ProcessingMetadata.DocumentsSummarized = true
-	t.ProcessingMetadata.DocumentsSummarizedAt = summarizedAt
+	w.markTenderSummarized(t)
+}
 
-	if err := w.updateTenderSummarizationStatus(t); err != nil {
+func (w *DocumentSummarizationWorker) isSummaryReadyInStorage(ctx context.Context, t *tender.Tender) bool {
+	if w.AIStorage == nil {
+		return false
+	}
+	ready, err := w.AIStorage.IsSummaryReady(ctx, t.ContractFolderID, t.NoticePublicationID)
+	if err != nil {
+		w.Logger.Debug("Could not check summarization status in storage", map[string]interface{}{
+			"tender_id":             t.ID.Hex(),
+			"notice_publication_id": t.NoticePublicationID,
+			"error":                 err.Error(),
+		})
+		return false
+	}
+	return ready
+}
+
+func (w *DocumentSummarizationWorker) markTenderSummarized(t *tender.Tender) {
+	now := time.Now().Unix()
+	t.ProcessingMetadata.DocumentsSummarized = true
+	t.ProcessingMetadata.DocumentsSummarizedAt = now
+	t.UpdatedAt = now
+
+	if err := w.TenderRepo.Update(context.Background(), t); err != nil {
 		w.Logger.Error("Failed to update tender summarization status", map[string]interface{}{
 			"tender_id":             t.ID.Hex(),
 			"notice_publication_id": t.NoticePublicationID,
@@ -152,118 +166,8 @@ func (w *DocumentSummarizationWorker) summarizeDocumentsForTender(t *tender.Tend
 		return
 	}
 
-	w.Logger.Info("Document summarization completed for tender", map[string]interface{}{
+	w.Logger.Info("Tender marked as summarized", map[string]interface{}{
 		"tender_id":             t.ID.Hex(),
 		"notice_publication_id": t.NoticePublicationID,
-		"summarized_count":      len(summaries),
-		"total_documents":       len(t.ScrapedDocuments),
 	})
 }
-
-func (w *DocumentSummarizationWorker) summarizeDocumentViaAI(ctx context.Context, t *tender.Tender, doc tender.ScrapedDocument) (summaryText, model, errStr string) {
-	docURL, err := w.MinioSDK.GetPresignedURL(ctx, doc.BucketName, doc.ObjectName, presignedDocumentURLSeconds)
-	if err != nil {
-		return "", "", err.Error()
-	}
-
-	if strings.TrimSpace(t.NoticePublicationID) == "" {
-		return "", "", "tender has no notice_publication_id for AI summarize request"
-	}
-	if strings.TrimSpace(t.ContractFolderID) == "" {
-		return "", "", "tender has no contract_folder_id for AI summarize request"
-	}
-
-	req := ai_summarizer.NewSummarizeRequest(ai_summarizer.SummarizeRequestInput{
-		ContractFolderID:    t.ContractFolderID,
-		NoticePublicationID: t.NoticePublicationID,
-		DocumentURL:         docURL,
-		Title:               t.Title,
-		Description:         t.Description,
-		Country:             t.CountryCode,
-		Currency:            t.Currency,
-		SubmissionDeadline:  t.SubmissionDeadline,
-		EstimatedValue:      t.EstimatedValue,
-		AuthorityName:       authorityNameFromTender(t),
-	})
-
-	resp, err := w.AIClient.FetchSummaryOnDemand(ctx, req)
-	if err != nil {
-		return "", "", err.Error()
-	}
-	if resp == nil {
-		return "", "", "AI summarize returned nil response"
-	}
-
-	summaryText, model = pickDocumentSummary(resp.Summaries, doc.Filename)
-	if summaryText == "" && len(resp.Summaries) > 0 {
-		// Fall back to first summary if filename did not match
-		summaryText = resp.Summaries[0].Summary
-		model = resp.Summaries[0].SummaryModel
-	}
-	if summaryText == "" {
-		return "", model, "AI summarize returned no summary text for document"
-	}
-	return summaryText, model, ""
-}
-
-func authorityNameFromTender(t *tender.Tender) string {
-	if t == nil || t.BuyerOrganization == nil {
-		return ""
-	}
-	return strings.TrimSpace(t.BuyerOrganization.Name)
-}
-
-func pickDocumentSummary(items []ai_summarizer.DocumentSummary, filename string) (text, model string) {
-	filename = strings.TrimSpace(filename)
-	for _, s := range items {
-		if strings.EqualFold(strings.TrimSpace(s.DocumentName), filename) {
-			return strings.TrimSpace(s.Summary), strings.TrimSpace(s.SummaryModel)
-		}
-	}
-	for _, s := range items {
-		if strings.TrimSpace(s.Summary) != "" {
-			return strings.TrimSpace(s.Summary), strings.TrimSpace(s.SummaryModel)
-		}
-	}
-	return "", ""
-}
-
-func (w *DocumentSummarizationWorker) getDocumentType(filename string) string {
-	ext := strings.ToLower(filepath.Ext(filename))
-	switch ext {
-	case ".xlsx":
-		return "xlsx"
-	case ".pdf":
-		return "pdf"
-	case ".docx":
-		return "docx"
-	case ".doc":
-		return "doc"
-	case ".txt":
-		return "txt"
-	case ".html", ".htm":
-		return "html"
-	default:
-		return "unknown"
-	}
-}
-
-func (w *DocumentSummarizationWorker) updateTenderSummarizationStatus(tender *tender.Tender) error {
-	err := w.TenderRepo.Update(context.Background(), tender)
-	if err != nil {
-		w.Logger.Error("Failed to update tender summarization status", map[string]interface{}{
-			"tender_id": tender.ID.Hex(),
-			"error":     err.Error(),
-		})
-		return err
-	}
-
-	w.Logger.Debug("Updated tender summarization status", map[string]interface{}{
-		"tender_id":               tender.ID.Hex(),
-		"documents_summarized":    tender.ProcessingMetadata.DocumentsSummarized,
-		"documents_summarized_at": tender.ProcessingMetadata.DocumentsSummarizedAt,
-		"summary_count":           len(tender.DocumentSummaries),
-	})
-
-	return nil
-}

cmd/worker/workers/translation.go

@@ -2,7 +2,6 @@ package workers
 
 import (
 	"context"
-	"errors"
 	"strings"
 	"tm/internal/tender"
 	"tm/pkg/ai_summarizer"
@@ -10,16 +9,18 @@ import (
 	"tm/pkg/mongo"
 )
 
-// TranslationWorker backfills missing tender translations using the AI pipeline
-// and on-demand translate API. Results are persisted by the AI service in MinIO only.
+// TranslationWorker syncs translation completion from the AI pipeline MinIO storage.
+// Batch translation is triggered via POST /pipeline/translate; this worker only
+// verifies which tenders now have translations available in storage.
 type TranslationWorker struct {
-	Mongo            *mongo.ConnectionManager
-	Logger           logger.Logger
-	TenderRepo       tender.TenderRepository
-	AIClient         *ai_summarizer.Client
-	AIStorage        *ai_summarizer.StorageClient
-	TargetLanguages  []string
-	BatchSize        int
+	Mongo                     *mongo.ConnectionManager
+	Logger                    logger.Logger
+	TenderRepo                tender.TenderRepository
+	AIClient                  *ai_summarizer.Client
+	AIStorage                 *ai_summarizer.StorageClient
+	TranslationSuccessCounter *mongo.Counter
+	TargetLanguages           []string
+	BatchSize                 int
 }
 
 // NewTranslationWorker creates a translation worker for the given target languages.
@@ -29,6 +30,7 @@ func NewTranslationWorker(
 	tenderRepo tender.TenderRepository,
 	aiClient *ai_summarizer.Client,
 	aiStorage *ai_summarizer.StorageClient,
+	translationSuccessCounter *mongo.Counter,
 	targetLanguages []string,
 	batchSize int,
 ) *TranslationWorker {
@@ -53,17 +55,18 @@ func NewTranslationWorker(
 	}
 
 	return &TranslationWorker{
-		Mongo:           mongo,
-		Logger:          logger,
-		TenderRepo:      tenderRepo,
-		AIClient:        aiClient,
-		AIStorage:       aiStorage,
-		TargetLanguages: langs,
-		BatchSize:       batchSize,
+		Mongo:                     mongo,
+		Logger:                    logger,
+		TenderRepo:                tenderRepo,
+		AIClient:                  aiClient,
+		AIStorage:                 aiStorage,
+		TranslationSuccessCounter: translationSuccessCounter,
+		TargetLanguages:           langs,
+		BatchSize:                 batchSize,
 	}
 }
 
-// Run starts backfilling translations for tenders missing target-language content.
+// Run triggers the AI translation pipeline and logs tenders with ready translations in MinIO.
 func (w *TranslationWorker) Run() {
 	if w.AIClient == nil {
 		w.Logger.Warn("Translation worker skipped: AI client is not configured", map[string]interface{}{
@@ -77,35 +80,72 @@ func (w *TranslationWorker) Run() {
 		"batch_size":       w.BatchSize,
 	})
 
+	startCount := w.getSuccessfulTranslationCount(mongo.AITranslationSuccessCounterKeyDailyJob)
+
 	if _, err := w.AIClient.TriggerPipelineTranslate(context.Background(), w.TargetLanguages); err != nil {
-		w.Logger.Warn("Failed to trigger AI pipeline translate (continuing with per-tender sync)", map[string]interface{}{
+		w.Logger.Warn("Failed to trigger AI pipeline translate", map[string]interface{}{
 			"target_languages": w.TargetLanguages,
 			"error":            err.Error(),
 		})
 	}
 
+	readyCount := 0
 	for _, language := range w.TargetLanguages {
-		w.runForLanguage(language)
+		readyCount += w.syncLanguageFromStorage(language)
+	}
+
+	endCount := w.getSuccessfulTranslationCount(mongo.AITranslationSuccessCounterKeyDailyJob)
+	totalCount := w.getSuccessfulTranslationCount(mongo.AITranslationSuccessCounterKey)
+	runCount := endCount - startCount
+	if runCount < 0 {
+		runCount = 0
 	}
 
 	w.Logger.Info("Translation worker completed", map[string]interface{}{
-		"target_languages": w.TargetLanguages,
+		"target_languages":                             w.TargetLanguages,
+		"translations_ready_in_storage":                readyCount,
+		"successful_ai_translation_requests_run":       runCount,
+		"successful_ai_translation_requests_daily_job": endCount,
+		"successful_ai_translation_requests_total":     totalCount,
 	})
 }
 
-func (w *TranslationWorker) runForLanguage(language string) {
+func (w *TranslationWorker) getSuccessfulTranslationCount(key string) int64 {
+	if w.TranslationSuccessCounter == nil {
+		return 0
+	}
+	count, err := w.TranslationSuccessCounter.Get(context.Background(), key)
+	if err != nil {
+		w.Logger.Warn("Failed to read AI translation success counter", map[string]interface{}{
+			"counter_key": key,
+			"error":       err.Error(),
+		})
+		return 0
+	}
+	return count
+}
+
+func (w *TranslationWorker) syncLanguageFromStorage(language string) int {
+	if w.AIStorage == nil {
+		w.Logger.Warn("Translation storage client not configured; skipping MinIO sync", map[string]interface{}{
+			"target_language": language,
+		})
+		return 0
+	}
+
+	readyCount := 0
 	skip := 0
 	for {
 		tenders, totalCount, err := w.TenderRepo.GetUnTranslatedTenders(context.Background(), language, w.BatchSize, skip)
 		if err != nil {
-			w.Logger.Error("Failed to fetch tenders for translation backfill", map[string]interface{}{
+			w.Logger.Error("Failed to fetch tenders for translation sync", map[string]interface{}{
 				"target_language": language,
 				"error":           err.Error(),
 			})
-			return
+			return readyCount
 		}
 
-		w.Logger.Info("Fetched tenders for translation backfill", map[string]interface{}{
+		w.Logger.Info("Fetched tenders for translation sync", map[string]interface{}{
 			"target_language": language,
 			"batch_count":     len(tenders),
 			"total_count":     totalCount,
@@ -113,63 +153,29 @@ func (w *TranslationWorker) runForLanguage(language string) {
 		})
 
 		if len(tenders) == 0 {
-			return
+			return readyCount
 		}
 
 		for _, t := range tenders {
-			w.translateTender(&t, language)
+			if w.hasTranslationInStorage(&t, language) {
+				readyCount++
+				w.Logger.Debug("Translation available in MinIO", map[string]interface{}{
+					"tender_id":       t.ID.Hex(),
+					"notice_id":       t.NoticePublicationID,
+					"target_language": language,
+				})
+			}
 		}
 
 		skip += len(tenders)
 		if int64(skip) >= totalCount {
-			return
+			return readyCount
 		}
 	}
 }
 
-func (w *TranslationWorker) translateTender(t *tender.Tender, language string) {
-	if t.NoticePublicationID == "" {
-		return
-	}
-	if strings.TrimSpace(t.ContractFolderID) == "" {
-		w.Logger.Debug("Skipping tender translation: missing contract_folder_id", map[string]interface{}{
-			"tender_id": t.ID.Hex(),
-		})
-		return
-	}
-	if w.hasTranslationInStorage(t, language) {
-		w.Logger.Debug("Skipping tender translation: already available in MinIO", map[string]interface{}{
-			"tender_id":       t.ID.Hex(),
-			"notice_id":       t.NoticePublicationID,
-			"target_language": language,
-		})
-		return
-	}
-
-	title, description, source, err := w.resolveTranslation(t, language)
-	if err != nil {
-		w.Logger.Error("Failed to translate tender", map[string]interface{}{
-			"tender_id":          t.ID.Hex(),
-			"notice_id":          t.NoticePublicationID,
-			"target_language":    language,
-			"translation_source": source,
-			"translation_error":  err.Error(),
-		})
-		return
-	}
-
-	w.Logger.Info("Tender translated successfully", map[string]interface{}{
-		"tender_id":          t.ID.Hex(),
-		"notice_id":          t.NoticePublicationID,
-		"target_language":    language,
-		"translation_source": source,
-		"title_len":          len(title),
-		"description_len":    len(description),
-	})
-}
-
 func (w *TranslationWorker) hasTranslationInStorage(t *tender.Tender, language string) bool {
-	if w.AIStorage == nil {
+	if strings.TrimSpace(t.ContractFolderID) == "" || t.NoticePublicationID == "" {
 		return false
 	}
 	stored, err := w.AIStorage.GetTranslationFromStorage(
@@ -180,39 +186,3 @@ func (w *TranslationWorker) hasTranslationInStorage(t *tender.Tender, language s
 	)
 	return err == nil && strings.TrimSpace(stored.Title) != ""
 }
-
-func (w *TranslationWorker) resolveTranslation(t *tender.Tender, language string) (title, description, source string, err error) {
-	if w.AIStorage != nil {
-		stored, storageErr := w.AIStorage.GetTranslationFromStorage(
-			context.Background(),
-			t.ContractFolderID,
-			t.NoticePublicationID,
-			language,
-		)
-		if storageErr == nil {
-			return stored.Title, stored.Description, "storage", nil
-		}
-		if !errors.Is(storageErr, ai_summarizer.ErrTranslationNotReady) &&
-			!errors.Is(storageErr, ai_summarizer.ErrObjectNotFound) &&
-			!errors.Is(storageErr, ai_summarizer.ErrNoTranslation) {
-			w.Logger.Debug("Translation not available from storage yet", map[string]interface{}{
-				"tender_id":       t.ID.Hex(),
-				"notice_id":       t.NoticePublicationID,
-				"target_language": language,
-				"error":           storageErr.Error(),
-			})
-		}
-	}
-
-	resp, apiErr := w.AIClient.FetchTranslationOnDemand(context.Background(), ai_summarizer.TranslateRequest{
-		ContractFolderID:    t.ContractFolderID,
-		NoticePublicationID: t.NoticePublicationID,
-		Title:               t.Title,
-		Description:         t.Description,
-		Language:            language,
-	})
-	if apiErr != nil {
-		return "", "", "api", apiErr
-	}
-	return resp.TranslatedTitle, resp.TranslatedDescription, "api", nil
-}

internal/customer/errors.go

@@ -1,7 +1,42 @@
 package customer
 
-import "errors"
+import (
+	"errors"
 
-var (
-	ErrCustomerNotFound = errors.New("customer not found")
+	orm "tm/pkg/mongo"
 )
+
+var ErrCustomerNotFound = errors.New("customer not found")
+
+func mapCustomerWriteError(err error) error {
+	if err == nil || !orm.IsDuplicateKeyError(err) {
+		return err
+	}
+	switch {
+	case orm.DuplicateKeyMatchesField(err, "phone"):
+		return errors.New("customer with this phone number already exists")
+	case orm.DuplicateKeyMatchesField(err, "email"):
+		return errors.New("customer with this email already exists")
+	case orm.DuplicateKeyMatchesField(err, "username"):
+		return errors.New("customer with this username already exists")
+	default:
+		return err
+	}
+}
+
+func isCustomerConflictError(err error) bool {
+	if err == nil {
+		return false
+	}
+	switch err.Error() {
+	case "customer with this phone number already exists",
+		"customer with this email already exists",
+		"customer with this username already exists",
+		"company with this name already exists",
+		"company with this registration number already exists",
+		"company with this tax ID already exists":
+		return true
+	default:
+		return false
+	}
+}

internal/customer/handler.go

@@ -53,11 +53,7 @@ func (h *Handler) CreateCustomer(c echo.Context) error {
 
 	customer, err := h.service.Register(c.Request().Context(), form)
 	if err != nil {
-		if err.Error() == "customer with this email already exists" ||
-			err.Error() == "customer with this username already exists" ||
-			err.Error() == "company with this name already exists" ||
-			err.Error() == "company with this registration number already exists" ||
-			err.Error() == "company with this tax ID already exists" {
+		if isCustomerConflictError(err) {
 			return response.Conflict(c, err.Error())
 		}
 		if strings.HasPrefix(err.Error(), "invalid company ID") || err.Error() == "invalid username format" {
@@ -126,11 +122,7 @@ func (h *Handler) UpdateCustomer(c echo.Context) error {
 		if err.Error() == "customer not found" {
 			return response.NotFound(c, "Customer not found")
 		}
-		if err.Error() == "customer with this email already exists" ||
-			err.Error() == "customer with this username already exists" ||
-			err.Error() == "company with this name already exists" ||
-			err.Error() == "company with this registration number already exists" ||
-			err.Error() == "company with this tax ID already exists" {
+		if isCustomerConflictError(err) {
 			return response.Conflict(c, err.Error())
 		}
 		if strings.HasPrefix(err.Error(), "invalid company ID") {

internal/customer/repository.go

@@ -25,6 +25,7 @@ type Repository interface {
 	GetByID(ctx context.Context, id string) (*Customer, error)
 	GetByEmail(ctx context.Context, email string) (*Customer, error)
 	GetByUsername(ctx context.Context, username string) (*Customer, error)
+	GetByPhone(ctx context.Context, phone string) (*Customer, error)
 	GetByIDs(ctx context.Context, ids []string) ([]Customer, error)
 	GetByRole(ctx context.Context, role CustomerRole) ([]Customer, error)
 	GetByCompanies(ctx context.Context, companies []string) ([]Customer, error)
@@ -50,6 +51,8 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Re
 	indexes := []orm.Index{
 		*orm.CreateUniqueIndex("username_idx", bson.D{{Key: "username", Value: 1}}),
 		*orm.CreateTextIndex("search_idx", "email", "username"),
+		*orm.CreateUniqueIndex("phone_idx", bson.D{{Key: "phone", Value: 1}}).
+			WithPartialFilterExpression(bson.M{"phone": bson.M{"$type": "string", "$ne": ""}}),
 	}
 
 	// Create indexes
@@ -166,6 +169,24 @@ func (r *customerRepository) GetByEmail(ctx context.Context, email string) (*Cus
 	return customer, nil
 }
 
+// GetByPhone retrieves a customer by phone number
+func (r *customerRepository) GetByPhone(ctx context.Context, phone string) (*Customer, error) {
+	filter := bson.M{"phone": phone}
+	customer, err := r.ormRepo.FindOne(ctx, filter)
+	if err != nil {
+		if errors.Is(err, orm.ErrDocumentNotFound) {
+			return nil, ErrCustomerNotFound
+		}
+		r.logger.Error("Failed to get customer by phone", map[string]interface{}{
+			"error": err.Error(),
+			"phone": phone,
+		})
+		return nil, err
+	}
+
+	return customer, nil
+}
+
 // GetByUsername retrieves a customer by username
 func (r *customerRepository) GetByUsername(ctx context.Context, username string) (*Customer, error) {
 	filter := bson.M{"username": username}

internal/customer/service.go

@@ -95,6 +95,13 @@ func (s *customerService) Register(ctx context.Context, form *CreateCustomerForm
 		return nil, errors.New("customer with this username already exists")
 	}
 
+	if form.Phone != nil && *form.Phone != "" {
+		existingCustomer, _ = s.repository.GetByPhone(ctx, *form.Phone)
+		if existingCustomer != nil {
+			return nil, errors.New("customer with this phone number already exists")
+		}
+	}
+
 	// Check validator
 	if !s.validator.IsValidUsername(form.Username) {
 		return nil, errors.New("invalid username format")
@@ -149,6 +156,9 @@ func (s *customerService) Register(ctx context.Context, form *CreateCustomerForm
 	// Save to database
 	err = s.repository.Register(ctx, customer)
 	if err != nil {
+		if mapped := mapCustomerWriteError(err); mapped != err {
+			return nil, mapped
+		}
 		s.logger.Error("Failed to create customer", map[string]interface{}{
 			"error":    err.Error(),
 			"email":    form.Email,
@@ -373,6 +383,12 @@ func (s *customerService) Update(ctx context.Context, id string, form *UpdateCus
 	}
 
 	if form.Phone != nil && (customer.Phone == nil || *form.Phone != *customer.Phone) {
+		if *form.Phone != "" {
+			existingCustomer, _ := s.repository.GetByPhone(ctx, *form.Phone)
+			if existingCustomer != nil && existingCustomer.ID.Hex() != id {
+				return nil, errors.New("customer with this phone number already exists")
+			}
+		}
 		customer.Phone = form.Phone
 		infoChanged = true
 	}
@@ -411,6 +427,9 @@ func (s *customerService) Update(ctx context.Context, id string, form *UpdateCus
 	// Update in database
 	err = s.repository.Update(ctx, customer)
 	if err != nil {
+		if mapped := mapCustomerWriteError(err); mapped != err {
+			return nil, mapped
+		}
 		s.logger.Error("Failed to update customer", map[string]interface{}{
 			"error":       err.Error(),
 			"customer_id": id,

internal/dashboard/entity.go

@@ -85,3 +85,24 @@ type RecentItem struct {
 	CreatedAt       int64  `json:"created_at"`
 	PublicationDate int64  `json:"publication_date"`
 }
+
+// StatisticsReportResponse powers the statistics and charts report.
+type StatisticsReportResponse struct {
+	Days        int                      `json:"days"`
+	GeneratedAt int64                    `json:"generated_at"`
+	Daily       StatisticsDailySeries    `json:"daily"`
+	Totals      StatisticsLifetimeTotals `json:"totals"`
+}
+
+// StatisticsDailySeries contains per-day chart series.
+type StatisticsDailySeries struct {
+	ScrapedTED       []TrendPoint `json:"scraped_ted"`
+	ScrapedDocuments []TrendPoint `json:"scraped_documents"`
+	TranslatedNotices []TrendPoint `json:"translated_notices"`
+}
+
+// StatisticsLifetimeTotals contains all-time totals.
+type StatisticsLifetimeTotals struct {
+	ScrapedDocuments  int64 `json:"scraped_documents"`
+	TranslatedTenders int64 `json:"translated_tenders"`
+}

internal/dashboard/form.go

@@ -27,3 +27,8 @@ type RecentQuery struct {
 	Limit  int    `query:"limit"`
 	Cursor string `query:"cursor"`
 }
+
+// StatisticsQuery binds query params for GET /dashboard/statistics.
+type StatisticsQuery struct {
+	Days int `query:"days"`
+}

internal/dashboard/handler.go

@@ -161,6 +161,30 @@ func (h *Handler) Recent(c echo.Context) error {
 	return response.Success(c, out, "Recent tenders retrieved successfully")
 }
 
+// Statistics returns scraping and translation statistics for charts.
+// @Summary Dashboard statistics report
+// @Description Daily and lifetime statistics for TED scraping, document scraping, and AI translation
+// @Tags Admin-Dashboard
+// @Produce json
+// @Param days query int false "Days back for daily series (default 14, max 90)"
+// @Success 200 {object} response.APIResponse{data=StatisticsReportResponse}
+// @Failure 500 {object} response.APIResponse
+// @Router /admin/v1/dashboard/statistics [get]
+// @Security BearerAuth
+func (h *Handler) Statistics(c echo.Context) error {
+	query := StatisticsQuery{
+		Days: parseIntQuery(c, "days", 0),
+	}
+
+	out, err := h.service.Statistics(c.Request().Context(), query)
+	if err != nil {
+		return response.InternalServerError(c, "Failed to load dashboard statistics")
+	}
+
+	setPrivateCache(c, 60)
+	return response.Success(c, out, "Dashboard statistics retrieved successfully")
+}
+
 func parseIntQuery(c echo.Context, name string, fallback int) int {
 	raw := c.QueryParam(name)
 	if raw == "" {

internal/dashboard/repository.go

@@ -23,18 +23,23 @@ type Repository interface {
 	NoticeTypes(ctx context.Context) (*NoticeTypesResponse, error)
 	ClosingSoon(ctx context.Context, limit int, windowSec int64) ([]ClosingSoonItem, error)
 	Recent(ctx context.Context, limit int, cursor string) ([]RecentItem, *string, error)
+	Statistics(ctx context.Context, days int, startUnix int64) (*StatisticsReportResponse, error)
 }
 
 type repository struct {
-	ormRepo orm.Repository[tender.Tender]
-	logger  logger.Logger
+	ormRepo        orm.Repository[tender.Tender]
+	mongoManager   *orm.ConnectionManager
+	metricsCounter *orm.Counter
+	logger         logger.Logger
 }
 
 // NewRepository creates a dashboard repository backed by the tenders collection.
 func NewRepository(mongoManager *orm.ConnectionManager, log logger.Logger) Repository {
 	return &repository{
-		ormRepo: orm.NewRepository[tender.Tender](mongoManager.GetCollection(tenderCollectionName()), log),
-		logger:  log,
+		ormRepo:        orm.NewRepository[tender.Tender](mongoManager.GetCollection(tenderCollectionName()), log),
+		mongoManager:   mongoManager,
+		metricsCounter: orm.NewCounter(mongoManager),
+		logger:         log,
 	}
 }
 

internal/dashboard/service.go

@@ -25,6 +25,7 @@ type Service interface {
 	NoticeTypes(ctx context.Context) (*NoticeTypesResponse, error)
 	ClosingSoon(ctx context.Context, query ClosingSoonQuery) (*ClosingSoonResponse, error)
 	Recent(ctx context.Context, query RecentQuery) (*RecentResponse, error)
+	Statistics(ctx context.Context, query StatisticsQuery) (*StatisticsReportResponse, error)
 }
 
 type service struct {
@@ -157,6 +158,25 @@ func (s *service) Recent(ctx context.Context, query RecentQuery) (*RecentRespons
 	}, nil
 }
 
+func (s *service) Statistics(ctx context.Context, query StatisticsQuery) (*StatisticsReportResponse, error) {
+	days := trendDays(query.Days)
+	startUnix := trendStartUnix(days)
+
+	s.logger.Info("Fetching dashboard statistics report", map[string]interface{}{
+		"days": days,
+	})
+
+	out, err := s.repo.Statistics(ctx, days, startUnix)
+	if err != nil {
+		s.logger.Error("Failed to fetch dashboard statistics report", map[string]interface{}{
+			"error": err.Error(),
+		})
+		return nil, fmt.Errorf("dashboard statistics: %w", err)
+	}
+
+	return out, nil
+}
+
 func closingWindowHours(hours int) int {
 	if hours <= 0 {
 		return defaultClosingWindowHours

internal/dashboard/statistics_repository.go

@@ -0,0 +1,189 @@
+package dashboard
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"tm/internal/notice"
+	orm "tm/pkg/mongo"
+
+	"go.mongodb.org/mongo-driver/v2/bson"
+	mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
+)
+
+const noticesCollectionName = "notices"
+
+func (r *repository) Statistics(ctx context.Context, days int, startUnix int64) (*StatisticsReportResponse, error) {
+	now := time.Now().UTC()
+	endDay := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, time.UTC)
+	startDay := endDay.AddDate(0, 0, -(days - 1))
+
+	scrapedTED, err := r.dailyCountByTimestamp(ctx, noticesCollectionName, bson.M{
+		"source": notice.TenderSourceTEDScraper,
+		"processing_metadata.scraped_at": bson.M{"$gte": startUnix, "$gt": 0},
+	}, "processing_metadata.scraped_at")
+	if err != nil {
+		return nil, fmt.Errorf("scraped ted per day: %w", err)
+	}
+
+	scrapedDocuments, err := r.scrapedDocumentsPerDay(ctx, startUnix)
+	if err != nil {
+		return nil, fmt.Errorf("scraped documents per day: %w", err)
+	}
+
+	translatedNotices, err := r.translatedNoticesPerDay(ctx, startDay, endDay)
+	if err != nil {
+		return nil, fmt.Errorf("translated notices per day: %w", err)
+	}
+
+	totalDocuments, err := r.totalScrapedDocuments(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("total scraped documents: %w", err)
+	}
+
+	totalTranslated, err := r.metricsCounter.Get(ctx, orm.AITranslationSuccessCounterKey)
+	if err != nil {
+		return nil, fmt.Errorf("total translated tenders: %w", err)
+	}
+
+	return &StatisticsReportResponse{
+		Days:        days,
+		GeneratedAt: now.Unix(),
+		Daily: StatisticsDailySeries{
+			ScrapedTED:        fillTrendSeries(days, scrapedTED),
+			ScrapedDocuments:  fillTrendSeries(days, scrapedDocuments),
+			TranslatedNotices: fillTrendSeries(days, translatedNotices),
+		},
+		Totals: StatisticsLifetimeTotals{
+			ScrapedDocuments:  totalDocuments,
+			TranslatedTenders: totalTranslated,
+		},
+	}, nil
+}
+
+func (r *repository) dailyCountByTimestamp(ctx context.Context, collection string, match bson.M, timestampField string) (map[string]int64, error) {
+	pipeline := mongodriver.Pipeline{
+		{{Key: "$match", Value: match}},
+		{{Key: "$addFields", Value: bson.M{
+			"metric_ts": normalizeTimestampExpr(timestampField),
+		}}},
+		{{Key: "$group", Value: bson.M{
+			"_id": bson.M{
+				"$dateToString": bson.M{
+					"format":   "%Y-%m-%d",
+					"date":     bson.M{"$toDate": bson.M{"$multiply": bson.A{"$metric_ts", 1000}}},
+					"timezone": "UTC",
+				},
+			},
+			"count": bson.M{"$sum": 1},
+		}}},
+	}
+
+	cursor, err := r.mongoManager.GetCollection(collection).Aggregate(ctx, pipeline)
+	if err != nil {
+		return nil, err
+	}
+	defer cursor.Close(ctx)
+
+	return decodeDailyCounts(ctx, cursor)
+}
+
+func (r *repository) scrapedDocumentsPerDay(ctx context.Context, startUnix int64) (map[string]int64, error) {
+	pipeline := mongodriver.Pipeline{
+		{{Key: "$match", Value: bson.M{
+			"processing_metadata.documents_scraped": true,
+			"scraped_documents":                   bson.M{"$exists": true, "$ne": bson.A{}},
+		}}},
+		{{Key: "$unwind", Value: "$scraped_documents"}},
+		{{Key: "$addFields", Value: bson.M{
+			"metric_ts": bson.M{
+				"$cond": bson.A{
+					bson.M{"$gt": bson.A{"$scraped_documents.scraped_at", 0}},
+					"$scraped_documents.scraped_at",
+					"$processing_metadata.documents_scraped_at",
+				},
+			},
+		}}},
+		{{Key: "$match", Value: bson.M{
+			"metric_ts": bson.M{"$gte": startUnix, "$gt": 0},
+		}}},
+		{{Key: "$addFields", Value: bson.M{
+			"metric_ts": normalizeTimestampExpr("metric_ts"),
+		}}},
+		{{Key: "$group", Value: bson.M{
+			"_id": bson.M{
+				"$dateToString": bson.M{
+					"format":   "%Y-%m-%d",
+					"date":     bson.M{"$toDate": bson.M{"$multiply": bson.A{"$metric_ts", 1000}}},
+					"timezone": "UTC",
+				},
+			},
+			"count": bson.M{"$sum": 1},
+		}}},
+	}
+
+	cursor, err := r.mongoManager.GetCollection(tenderCollectionName()).Aggregate(ctx, pipeline)
+	if err != nil {
+		return nil, err
+	}
+	defer cursor.Close(ctx)
+
+	return decodeDailyCounts(ctx, cursor)
+}
+
+func (r *repository) translatedNoticesPerDay(ctx context.Context, startDay, endDay time.Time) (map[string]int64, error) {
+	return r.metricsCounter.GetDailyCounts(ctx, startDay, endDay)
+}
+
+func (r *repository) totalScrapedDocuments(ctx context.Context) (int64, error) {
+	pipeline := mongodriver.Pipeline{
+		{{Key: "$match", Value: bson.M{
+			"processing_metadata.documents_scraped": true,
+			"scraped_documents":                   bson.M{"$exists": true, "$ne": bson.A{}},
+		}}},
+		{{Key: "$project", Value: bson.M{
+			"doc_count": bson.M{"$size": "$scraped_documents"},
+		}}},
+		{{Key: "$group", Value: bson.M{
+			"_id":   nil,
+			"total": bson.M{"$sum": "$doc_count"},
+		}}},
+	}
+
+	cursor, err := r.mongoManager.GetCollection(tenderCollectionName()).Aggregate(ctx, pipeline)
+	if err != nil {
+		return 0, err
+	}
+	defer cursor.Close(ctx)
+
+	if !cursor.Next(ctx) {
+		return 0, nil
+	}
+
+	var row bson.M
+	if err := cursor.Decode(&row); err != nil {
+		return 0, err
+	}
+
+	return aggregateCount(row, "total"), nil
+}
+
+func decodeDailyCounts(ctx context.Context, cursor *mongodriver.Cursor) (map[string]int64, error) {
+	counts := make(map[string]int64)
+	for cursor.Next(ctx) {
+		var row bson.M
+		if err := cursor.Decode(&row); err != nil {
+			return nil, err
+		}
+		date, ok := row["_id"].(string)
+		if !ok || date == "" {
+			continue
+		}
+		counts[date] = aggregateCount(row, "count")
+	}
+	if err := cursor.Err(); err != nil {
+		return nil, err
+	}
+	return counts, nil
+}

internal/tender/errors.go

@@ -0,0 +1,13 @@
+package tender
+
+import "errors"
+
+// Sentinel errors for AI trigger operations and HTTP mapping.
+var (
+	ErrEmptyTenderID                 = errors.New("tender: id is required")
+	ErrInvalidTenderID               = errors.New("tender: invalid id")
+	ErrTenderNotFound                = errors.New("tender: not found")
+	ErrTenderMissingNoticeID         = errors.New("tender: missing notice publication id")
+	ErrTenderMissingContractFolderID = errors.New("tender: missing contract folder id")
+	ErrAINotConfigured               = errors.New("tender: ai service not configured")
+)

internal/tender/form.go

@@ -573,3 +573,23 @@ type AIDocumentSummary struct {
 	SummaryModel string `json:"summary_model"`
 	Error        string `json:"error,omitempty"`
 }
+
+// AIAnalyzeResponse represents the response for on-demand agentic analysis.
+type AIAnalyzeResponse struct {
+	ContractFolderID    string              `json:"contract_folder_id"`
+	NoticePublicationID string              `json:"notice_publication_id"`
+	Summary             string              `json:"summary"`
+	Documents           []AIAnalyzeDocument `json:"documents"`
+	ToolCallsLog        []string            `json:"tool_calls_log"`
+	Iterations          int                 `json:"iterations"`
+	Model               string              `json:"model"`
+}
+
+// AIAnalyzeDocument describes one document referenced during analysis.
+type AIAnalyzeDocument struct {
+	Filename     string `json:"filename"`
+	SizeBytes    int64  `json:"size_bytes"`
+	TimesRead    int    `json:"times_read"`
+	TimesQueried int    `json:"times_queried"`
+	Description  string `json:"description"`
+}

internal/tender/handler.go

@@ -398,7 +398,7 @@ func (h *TenderHandler) GetPublicTenderDetails(c echo.Context) error {
 
 // GetDocumentSummaries retrieves document summaries for a tender
 // @Summary Get document summaries for a tender
-// @Description Retrieve AI-generated summaries of documents for a specific tender
+// @Description Retrieve per-document AI summaries from the AI service MinIO storage for a specific tender
 // @Tags Tenders
 // @Produce json
 // @Param id path string true "Tender ID"
@@ -589,12 +589,58 @@ func (h *TenderHandler) TriggerAISummarize(c echo.Context) error {
 			"tender_id": id,
 			"error":     err.Error(),
 		})
-		return response.InternalServerError(c, err.Error())
+		return mapAITriggerHTTPError(c, err, "AI summarization")
 	}
 
 	return response.Success(c, result, "AI summarization completed successfully")
 }
 
+// TriggerAIAnalyze triggers on-demand agentic analysis for a tender
+// @Summary Trigger AI analysis
+// @Description Trigger on-demand agentic document analysis for a tender via the external AI service
+// @Tags Admin-Tenders
+// @Produce json
+// @Param id path string true "Tender ID"
+// @Success 200 {object} response.APIResponse{data=AIAnalyzeResponse}
+// @Failure 400 {object} response.APIResponse
+// @Failure 404 {object} response.APIResponse
+// @Failure 500 {object} response.APIResponse
+// @Router /admin/v1/tenders/{id}/ai-analyze [post]
+// @Security BearerAuth
+func (h *TenderHandler) TriggerAIAnalyze(c echo.Context) error {
+	id := c.Param("id")
+	if id == "" {
+		return response.BadRequest(c, "Tender ID is required", "ID parameter cannot be empty")
+	}
+
+	result, err := h.service.TriggerAIAnalyze(c.Request().Context(), id)
+	if err != nil {
+		h.logger.Error("Failed to trigger AI analysis", map[string]interface{}{
+			"tender_id": id,
+			"error":     err.Error(),
+		})
+		return mapAITriggerHTTPError(c, err, "AI analysis")
+	}
+
+	return response.Success(c, result, "AI analysis completed successfully")
+}
+
+// mapAITriggerHTTPError maps AI trigger service errors to safe HTTP responses.
+func mapAITriggerHTTPError(c echo.Context, err error, action string) error {
+	switch {
+	case errors.Is(err, ErrEmptyTenderID), errors.Is(err, ErrInvalidTenderID):
+		return response.BadRequest(c, "Invalid request", "Invalid tender ID")
+	case errors.Is(err, ErrTenderNotFound), errors.Is(err, orm.ErrDocumentNotFound):
+		return response.NotFound(c, "Tender not found")
+	case errors.Is(err, ErrTenderMissingNoticeID), errors.Is(err, ErrTenderMissingContractFolderID):
+		return response.BadRequest(c, "Tender not eligible", "Tender is missing required identifiers for AI processing")
+	case errors.Is(err, ErrAINotConfigured):
+		return response.InternalServerError(c, "AI service is not available")
+	default:
+		return response.InternalServerError(c, action+" failed")
+	}
+}
+
 // TriggerAITranslate triggers on-demand AI translation for a tender
 // @Summary Trigger AI translation
 // @Description Trigger on-demand AI translation for tender title and description (cached in MinIO by the AI service)

internal/tender/repository.go

@@ -31,7 +31,6 @@ type TenderRepository interface {
 	GetTenderCountByClassification(ctx context.Context) (map[string]int64, error)
 	Search(ctx context.Context, form *SearchForm, pagination *response.Pagination) (*SearchPageResult, error)
 	GetExpiredTenders(ctx context.Context, beforeDate int64) ([]Tender, error)
-	GetUnScrapedTenders(ctx context.Context, limit, skip int) ([]Tender, int64, error)
 	GetPendingDocumentScrapeTenders(ctx context.Context, countryCodes []string, minDeadline int64, limit, skip int) ([]Tender, bool, error)
 	GetUnSummarizedTenders(ctx context.Context, limit, skip int) ([]Tender, int64, error)
 	GetUnTranslatedTenders(ctx context.Context, language string, limit, skip int) ([]Tender, int64, error)
@@ -549,36 +548,6 @@ func (r *tenderRepository) GetExpiredTenders(ctx context.Context, beforeDate int
 	return result.Items, nil
 }
 
-// GetUnScrapedTenders retrieves tenders that haven't been processed for document scraping
-func (r *tenderRepository) GetUnScrapedTenders(ctx context.Context, limit, skip int) ([]Tender, int64, error) {
-	filter := bson.M{
-		"$or": []bson.M{
-			{"processing_metadata.documents_scraped": bson.M{"$exists": false}},
-			{"processing_metadata.documents_scraped": false},
-			{"processing_metadata.documents_scraped": nil},
-		},
-	}
-
-	pagination := orm.Pagination{
-		Limit:     limit,
-		Skip:      skip,
-		SortField: "created_at",
-		SortOrder: 1, // Oldest first
-	}
-
-	result, err := r.ormRepo.FindAll(ctx, filter, pagination)
-	if err != nil {
-		r.logger.Error("Failed to get un-scraped tenders", map[string]interface{}{
-			"limit": limit,
-			"skip":  skip,
-			"error": err.Error(),
-		})
-		return nil, 0, err
-	}
-
-	return result.Items, result.TotalCount, nil
-}
-
 // GetPendingDocumentScrapeTenders returns unscraped tenders for the given countries with an active deadline.
 func (r *tenderRepository) GetPendingDocumentScrapeTenders(ctx context.Context, countryCodes []string, minDeadline int64, limit, skip int) ([]Tender, bool, error) {
 	if len(countryCodes) == 0 {

internal/tender/service.go

@@ -15,13 +15,15 @@ import (
 	"tm/internal/customer"
 	"tm/pkg/ai_summarizer"
 	"tm/pkg/logger"
+	orm "tm/pkg/mongo"
 	"tm/pkg/response"
 )
 
-// AISummarizerClient defines the interface for on-demand AI summarization
+// AISummarizerClient defines the interface for on-demand AI operations.
 type AISummarizerClient interface {
 	FetchSummaryOnDemand(ctx context.Context, reqBody ai_summarizer.SummarizeRequest) (*ai_summarizer.SummarizeResponse, error)
 	FetchTranslationOnDemand(ctx context.Context, reqBody ai_summarizer.TranslateRequest) (*ai_summarizer.TranslateResponse, error)
+	FetchAnalyzeOnDemand(ctx context.Context, reqBody ai_summarizer.AnalyzeRequest) (*ai_summarizer.AnalyzeResponse, error)
 	TriggerPipelineTranslate(ctx context.Context, languages []string) (*ai_summarizer.PipelineTranslateResponse, error)
 }
 
@@ -30,6 +32,7 @@ type AISummarizerClient interface {
 // PROC_<contractFolderID>/<noticePublicationID>/ in the shared bucket.
 type AISummarizerStorage interface {
 	GetSummaryFromStorage(ctx context.Context, contractFolderID, noticePublicationID string) (string, error)
+	GetDocumentSummariesFromStorage(ctx context.Context, contractFolderID, noticePublicationID string) ([]ai_summarizer.DocumentSummary, error)
 	GetTranslationFromStorage(ctx context.Context, contractFolderID, noticePublicationID, language string) (ai_summarizer.StoredTranslation, error)
 	ListDocuments(ctx context.Context, contractFolderID, noticePublicationID string) ([]ai_summarizer.StoredDocument, error)
 	DownloadDocument(ctx context.Context, objectName string) (io.ReadCloser, error)
@@ -60,6 +63,7 @@ type Service interface {
 	// AI summarization operations
 	GetAISummary(ctx context.Context, id string) (*AISummaryResponse, error)
 	TriggerAISummarize(ctx context.Context, id string) (*AISummarizeResponse, error)
+	TriggerAIAnalyze(ctx context.Context, id string) (*AIAnalyzeResponse, error)
 	TriggerAITranslate(ctx context.Context, id, language string) (*AITranslateResponse, error)
 	GetAllAISummaries(ctx context.Context) ([]ai_summarizer.TenderSummaryItem, error)
 	// ListTendersWithScrapedDocuments lists tenders that have documents in MinIO. When syncMongoMetadata is true,
@@ -451,14 +455,13 @@ func (s *tenderService) Delete(ctx context.Context, id string) error {
 	return nil
 }
 
-// GetDocumentSummaries retrieves document summaries for a specific tender
+// GetDocumentSummaries retrieves per-document summaries from the AI service MinIO storage.
 func (s *tenderService) GetDocumentSummaries(ctx context.Context, id string) ([]DocumentSummary, error) {
 	if id == "" {
 		return nil, fmt.Errorf("tender ID cannot be empty")
 	}
 
-	// Get tender from repository
-	tender, err := s.repository.GetByID(ctx, id)
+	t, err := s.repository.GetByID(ctx, id)
 	if err != nil {
 		s.logger.Error("Failed to get tender for document summaries", map[string]interface{}{
 			"tender_id": id,
@@ -466,22 +469,60 @@ func (s *tenderService) GetDocumentSummaries(ctx context.Context, id string) ([]
 		})
 		return nil, fmt.Errorf("failed to get tender: %w", err)
 	}
-
-	if tender == nil {
+	if t == nil {
 		return nil, fmt.Errorf("tender not found")
 	}
-
-	// Return document summaries (empty slice if none exist)
-	if tender.DocumentSummaries == nil {
+	if strings.TrimSpace(t.NoticePublicationID) == "" || strings.TrimSpace(t.ContractFolderID) == "" {
+		return []DocumentSummary{}, nil
+	}
+	if s.aiSummarizerStorage == nil {
 		return []DocumentSummary{}, nil
 	}
 
+	stored, err := s.aiSummarizerStorage.GetDocumentSummariesFromStorage(ctx, t.ContractFolderID, t.NoticePublicationID)
+	if errors.Is(err, ai_summarizer.ErrSummaryNotReady) {
+		return []DocumentSummary{}, nil
+	}
+	if errors.Is(err, ai_summarizer.ErrObjectNotFound) {
+		return []DocumentSummary{}, nil
+	}
+	if err != nil {
+		if errors.Is(err, ai_summarizer.ErrMinIOUnavailable) {
+			return nil, fmt.Errorf("failed to get document summaries: MinIO connection unavailable: %w", err)
+		}
+		s.logger.Error("Failed to get document summaries from storage", map[string]interface{}{
+			"tender_id": id,
+			"error":     err.Error(),
+		})
+		return nil, fmt.Errorf("failed to get document summaries: %w", err)
+	}
+
+	summaries := mapAIDocumentSummaries(stored)
 	s.logger.Info("Retrieved document summaries for tender", map[string]interface{}{
 		"tender_id":       id,
-		"summaries_count": len(tender.DocumentSummaries),
+		"summaries_count": len(summaries),
+		"source":          "storage",
 	})
 
-	return tender.DocumentSummaries, nil
+	return summaries, nil
+}
+
+func mapAIDocumentSummaries(items []ai_summarizer.DocumentSummary) []DocumentSummary {
+	if len(items) == 0 {
+		return []DocumentSummary{}
+	}
+	out := make([]DocumentSummary, 0, len(items))
+	for _, item := range items {
+		out = append(out, DocumentSummary{
+			DocumentName:    item.DocumentName,
+			Summary:         item.Summary,
+			SummaryLanguage: "en",
+			DocumentType:    item.DocumentType,
+			SummaryModel:    item.SummaryModel,
+			Error:           item.Error,
+		})
+	}
+	return out
 }
 
 // GetDocuments retrieves scraped document metadata for a tender.
@@ -1147,11 +1188,11 @@ func (s *tenderService) GetAISummary(ctx context.Context, id string) (*AISummary
 // It calls the external AI service HTTP API to generate summaries.
 func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AISummarizeResponse, error) {
 	if id == "" {
-		return nil, fmt.Errorf("tender ID cannot be empty")
+		return nil, ErrEmptyTenderID
 	}
 
 	if s.aiSummarizerClient == nil {
-		return nil, fmt.Errorf("AI summarizer service is not configured")
+		return nil, ErrAINotConfigured
 	}
 
 	// Get tender to build the request
@@ -1161,37 +1202,27 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
 			"tender_id": id,
 			"error":     err.Error(),
 		})
+		if errors.Is(err, orm.ErrDocumentNotFound) {
+			return nil, ErrTenderNotFound
+		}
+		if strings.Contains(err.Error(), "invalid id format") {
+			return nil, ErrInvalidTenderID
+		}
 		return nil, fmt.Errorf("failed to get tender: %w", err)
 	}
 
 	if t == nil {
-		return nil, fmt.Errorf("tender not found")
+		return nil, ErrTenderNotFound
 	}
 
 	if t.NoticePublicationID == "" {
-		return nil, fmt.Errorf("tender has no notice publication ID")
+		return nil, ErrTenderMissingNoticeID
 	}
 	if strings.TrimSpace(t.ContractFolderID) == "" {
-		return nil, fmt.Errorf("tender has no contract folder ID")
+		return nil, ErrTenderMissingContractFolderID
 	}
 
-	authorityName := ""
-	if t.BuyerOrganization != nil {
-		authorityName = t.BuyerOrganization.Name
-	}
-
-	reqBody := ai_summarizer.NewSummarizeRequest(ai_summarizer.SummarizeRequestInput{
-		ContractFolderID:    t.ContractFolderID,
-		NoticePublicationID: t.NoticePublicationID,
-		DocumentURL:         t.DocumentURI,
-		Title:               t.Title,
-		Description:         t.Description,
-		Country:             t.CountryCode,
-		Currency:            t.Currency,
-		SubmissionDeadline:  t.SubmissionDeadline,
-		EstimatedValue:      t.EstimatedValue,
-		AuthorityName:       authorityName,
-	})
+	reqBody := ai_summarizer.NewSummarizeRequest(t.ContractFolderID, t.NoticePublicationID)
 
 	s.logger.Info("Triggering on-demand AI summarization", map[string]interface{}{
 		"tender_id":             id,
@@ -1247,6 +1278,95 @@ func (s *tenderService) TriggerAISummarize(ctx context.Context, id string) (*AIS
 	}, nil
 }
 
+// TriggerAIAnalyze triggers on-demand agentic analysis for a tender.
+func (s *tenderService) TriggerAIAnalyze(ctx context.Context, id string) (*AIAnalyzeResponse, error) {
+	if id == "" {
+		return nil, ErrEmptyTenderID
+	}
+	if s.aiSummarizerClient == nil {
+		return nil, ErrAINotConfigured
+	}
+
+	t, err := s.repository.GetByID(ctx, id)
+	if err != nil {
+		s.logger.Error("Failed to get tender for on-demand analysis", map[string]interface{}{
+			"tender_id": id,
+			"error":     err.Error(),
+		})
+		if errors.Is(err, orm.ErrDocumentNotFound) {
+			return nil, ErrTenderNotFound
+		}
+		if strings.Contains(err.Error(), "invalid id format") {
+			return nil, ErrInvalidTenderID
+		}
+		return nil, fmt.Errorf("failed to get tender: %w", err)
+	}
+	if t == nil {
+		return nil, ErrTenderNotFound
+	}
+	if t.NoticePublicationID == "" {
+		return nil, ErrTenderMissingNoticeID
+	}
+	if strings.TrimSpace(t.ContractFolderID) == "" {
+		return nil, ErrTenderMissingContractFolderID
+	}
+
+	reqBody := ai_summarizer.NewAnalyzeRequest(t.ContractFolderID, t.NoticePublicationID)
+	s.logger.Info("Triggering on-demand AI analysis", map[string]interface{}{
+		"tender_id":             id,
+		"contract_folder_id":    t.ContractFolderID,
+		"notice_publication_id": t.NoticePublicationID,
+	})
+
+	resp, err := s.aiSummarizerClient.FetchAnalyzeOnDemand(ctx, reqBody)
+	if err != nil {
+		s.logger.Error("On-demand AI analysis failed", map[string]interface{}{
+			"tender_id":             id,
+			"contract_folder_id":    t.ContractFolderID,
+			"notice_publication_id": t.NoticePublicationID,
+			"error":                 err.Error(),
+		})
+		return nil, fmt.Errorf("AI analysis request failed: %w", err)
+	}
+
+	documents := make([]AIAnalyzeDocument, 0, len(resp.Documents))
+	for _, doc := range resp.Documents {
+		documents = append(documents, AIAnalyzeDocument{
+			Filename:     doc.Filename,
+			SizeBytes:    doc.SizeBytes,
+			TimesRead:    doc.TimesRead,
+			TimesQueried: doc.TimesQueried,
+			Description:  doc.Description,
+		})
+	}
+
+	noticePublicationID := strings.TrimSpace(resp.NoticePublicationID)
+	if noticePublicationID == "" {
+		noticePublicationID = t.NoticePublicationID
+	}
+	contractFolderID := strings.TrimSpace(resp.ContractFolderID)
+	if contractFolderID == "" {
+		contractFolderID = t.ContractFolderID
+	}
+
+	s.logger.Info("On-demand AI analysis completed", map[string]interface{}{
+		"tender_id":             id,
+		"contract_folder_id":    contractFolderID,
+		"notice_publication_id": noticePublicationID,
+		"iterations":            resp.Iterations,
+	})
+
+	return &AIAnalyzeResponse{
+		ContractFolderID:    contractFolderID,
+		NoticePublicationID: noticePublicationID,
+		Summary:             resp.Summary,
+		Documents:           documents,
+		ToolCallsLog:        resp.ToolCallsLog,
+		Iterations:          resp.Iterations,
+		Model:               resp.Model,
+	}, nil
+}
+
 // GetAllAISummaries retrieves all AI-generated summaries from the MinIO storage.
 // It lists all tender.json files in the bucket and returns summaries for tenders
 // where the AI pipeline has completed processing.
@@ -1476,7 +1596,7 @@ func (s *tenderService) TriggerAITranslate(ctx context.Context, id, language str
 		return nil, fmt.Errorf("tender has no contract folder ID")
 	}
 
-	title, description, err := s.resolveTranslation(ctx, t, targetLanguage)
+	title, description, fromAPI, err := s.resolveTranslation(ctx, t, targetLanguage, ai_summarizer.TranslationRequestSourceManualTrigger)
 	if err != nil {
 		s.logger.Error("AI translation failed", map[string]interface{}{
 			"tender_id": id,
@@ -1487,6 +1607,14 @@ func (s *tenderService) TriggerAITranslate(ctx context.Context, id, language str
 		return nil, fmt.Errorf("AI translation request failed: %w", err)
 	}
 
+	if fromAPI {
+		s.logger.Info("Manual AI translation request succeeded", map[string]interface{}{
+			"tender_id": id,
+			"notice_id": t.NoticePublicationID,
+			"language":  targetLanguage,
+		})
+	}
+
 	return s.buildAITranslateResponse(t, targetLanguage, title, description), nil
 }
 
@@ -1500,13 +1628,14 @@ func (s *tenderService) buildAITranslateResponse(t *Tender, language, title, des
 	}
 }
 
-func (s *tenderService) resolveTranslation(ctx context.Context, t *Tender, language string) (string, string, error) {
+func (s *tenderService) resolveTranslation(ctx context.Context, t *Tender, language, requestSource string) (string, string, bool, error) {
 	if s.aiSummarizerStorage != nil {
-		stored, err := s.aiSummarizerStorage.GetTranslationFromStorage(ctx, t.ContractFolderID, t.NoticePublicationID, language)
-		if err == nil {
-			return stored.Title, stored.Description, nil
+		stored, _, err := s.lookupTranslationForTender(ctx, t, language)
+		if err == nil && strings.TrimSpace(stored.Title) != "" {
+			return stored.Title, stored.Description, false, nil
 		}
-		if !errors.Is(err, ai_summarizer.ErrTranslationNotReady) &&
+		if err != nil &&
+			!errors.Is(err, ai_summarizer.ErrTranslationNotReady) &&
 			!errors.Is(err, ai_summarizer.ErrObjectNotFound) &&
 			!errors.Is(err, ai_summarizer.ErrNoTranslation) {
 			s.logger.Debug("Translation not available from storage", map[string]interface{}{
@@ -1520,14 +1649,26 @@ func (s *tenderService) resolveTranslation(ctx context.Context, t *Tender, langu
 	resp, err := s.aiSummarizerClient.FetchTranslationOnDemand(ctx, ai_summarizer.TranslateRequest{
 		ContractFolderID:    t.ContractFolderID,
 		NoticePublicationID: t.NoticePublicationID,
-		Title:               t.Title,
-		Description:         t.Description,
 		Language:            language,
+		RequestSource:       requestSource,
 	})
 	if err != nil {
-		return "", "", err
+		return "", "", false, err
 	}
-	return resp.TranslatedTitle, resp.TranslatedDescription, nil
+
+	title := strings.TrimSpace(resp.TranslatedTitle)
+	description := resp.TranslatedDescription
+	if s.aiSummarizerStorage != nil {
+		if stored, _, storageErr := s.lookupTranslationForTender(ctx, t, language); storageErr == nil && strings.TrimSpace(stored.Title) != "" {
+			title = stored.Title
+			description = stored.Description
+		}
+	}
+	if title == "" {
+		title = strings.TrimSpace(resp.TranslatedTitle)
+	}
+
+	return title, description, true, nil
 }
 
 func (s *tenderService) pickResponseLanguage(language *string) string {

internal/user/errors.go

@@ -1,7 +1,37 @@
 package user
 
-import "errors"
+import (
+	"errors"
 
-var (
-	ErrUserNotFound = errors.New("user not found")
+	orm "tm/pkg/mongo"
 )
+
+var ErrUserNotFound = errors.New("user not found")
+
+func mapUserWriteError(err error) error {
+	if err == nil || !orm.IsDuplicateKeyError(err) {
+		return err
+	}
+	switch {
+	case orm.DuplicateKeyMatchesField(err, "phone"):
+		return errors.New("phone number already exists")
+	case orm.DuplicateKeyMatchesField(err, "email"):
+		return errors.New("email already exists")
+	case orm.DuplicateKeyMatchesField(err, "username"):
+		return errors.New("username already exists")
+	default:
+		return err
+	}
+}
+
+func isUserConflictError(err error) bool {
+	if err == nil {
+		return false
+	}
+	switch err.Error() {
+	case "phone number already exists", "email already exists", "username already exists":
+		return true
+	default:
+		return false
+	}
+}

internal/user/handler.go

@@ -170,6 +170,9 @@ func (h *Handler) UpdateProfile(c echo.Context) error {
 
 	user, err := h.service.Update(c.Request().Context(), userID, form)
 	if err != nil {
+		if isUserConflictError(err) {
+			return response.Conflict(c, err.Error())
+		}
 		return response.BadRequest(c, err.Error(), "")
 	}
 
@@ -274,6 +277,9 @@ func (h *Handler) CreateUser(c echo.Context) error {
 	// Call service
 	user, err := h.service.Register(c.Request().Context(), form)
 	if err != nil {
+		if isUserConflictError(err) {
+			return response.Conflict(c, err.Error())
+		}
 		return response.BadRequest(c, err.Error(), "")
 	}
 
@@ -386,6 +392,9 @@ func (h *Handler) UpdateUser(c echo.Context) error {
 	// Call service
 	user, err := h.service.Update(c.Request().Context(), idStr, form)
 	if err != nil {
+		if isUserConflictError(err) {
+			return response.Conflict(c, err.Error())
+		}
 		return response.BadRequest(c, err.Error(), "")
 	}
 

internal/user/repository.go

@@ -22,6 +22,7 @@ type Repository interface {
 	UpdatePasswordReset(ctx context.Context, id, hashedPassword, updatedBy string) error
 	GetByEmail(ctx context.Context, email string) (*User, error)
 	GetByUsername(ctx context.Context, username string) (*User, error)
+	GetByPhone(ctx context.Context, phone string) (*User, error)
 	GetByIDs(ctx context.Context, userIDs []string) ([]User, error)
 	Search(ctx context.Context, form *SearchUsersForm, pagination *response.Pagination) (*orm.PaginatedResult[User], error)
 }
@@ -38,6 +39,8 @@ func NewRepository(mongoManager *orm.ConnectionManager, logger logger.Logger) Re
 	// Create indexes using the ORM's index management
 	indexes := []orm.Index{
 		*orm.CreateTextIndex("search_idx", "full_name", "email", "username"),
+		*orm.CreateUniqueIndex("phone_idx", bson.D{{Key: "phone", Value: 1}}).
+			WithPartialFilterExpression(bson.M{"phone": bson.M{"$type": "string", "$ne": ""}}),
 	}
 
 	// Create indexes
@@ -250,6 +253,23 @@ func (r *userRepository) GetByEmail(ctx context.Context, email string) (*User, e
 	return user, nil
 }
 
+// GetByPhone retrieves a user by phone number
+func (r *userRepository) GetByPhone(ctx context.Context, phone string) (*User, error) {
+	user, err := r.ormRepo.FindOne(ctx, bson.M{"phone": phone})
+	if err != nil {
+		if errors.Is(err, orm.ErrDocumentNotFound) {
+			return nil, errors.New("user not found")
+		}
+		r.logger.Error("Failed to get user by phone", map[string]interface{}{
+			"error": err.Error(),
+			"phone": phone,
+		})
+		return nil, err
+	}
+
+	return user, nil
+}
+
 // GetByUsername retrieves a user by username
 func (r *userRepository) GetByUsername(ctx context.Context, username string) (*User, error) {
 	user, err := r.ormRepo.FindOne(ctx, bson.M{"username": username})

internal/user/service.go

@@ -77,6 +77,13 @@ func (s *userService) Register(ctx context.Context, form *CreateUserForm) (*User
 		return nil, errors.New("username already exists")
 	}
 
+	if form.Phone != nil && *form.Phone != "" {
+		existingUser, _ = s.repository.GetByPhone(ctx, *form.Phone)
+		if existingUser != nil {
+			return nil, errors.New("phone number already exists")
+		}
+	}
+
 	// Hash password
 	hashedPassword, err := bcrypt.GenerateFromPassword([]byte(form.Password), bcrypt.DefaultCost)
 	if err != nil {
@@ -110,6 +117,9 @@ func (s *userService) Register(ctx context.Context, form *CreateUserForm) (*User
 	// Save to database
 	err = s.repository.Create(ctx, user)
 	if err != nil {
+		if mapped := mapUserWriteError(err); mapped != err {
+			return nil, mapped
+		}
 		s.logger.Error("Failed to register user", map[string]interface{}{
 			"error":    err.Error(),
 			"email":    form.Email,
@@ -184,7 +194,13 @@ func (s *userService) Update(ctx context.Context, id string, form *UpdateUserFor
 		user.Position = form.Position
 	}
 
-	if form.Phone != nil {
+	if form.Phone != nil && (user.Phone == nil || *form.Phone != *user.Phone) {
+		if *form.Phone != "" {
+			existingUser, _ := s.repository.GetByPhone(ctx, *form.Phone)
+			if existingUser != nil && existingUser.ID.Hex() != id {
+				return nil, errors.New("phone number already exists")
+			}
+		}
 		user.Phone = form.Phone
 	}
 
@@ -195,6 +211,9 @@ func (s *userService) Update(ctx context.Context, id string, form *UpdateUserFor
 	// Update in database
 	err = s.repository.Update(ctx, user)
 	if err != nil {
+		if mapped := mapUserWriteError(err); mapped != err {
+			return nil, mapped
+		}
 		s.logger.Error("Failed to update user", map[string]interface{}{
 			"error":   err.Error(),
 			"user_id": id,
@@ -573,7 +592,13 @@ func (s *userService) UpdateProfile(ctx context.Context, userID string, form *Up
 	if form.Position != nil {
 		user.Position = form.Position
 	}
-	if form.Phone != nil {
+	if form.Phone != nil && (user.Phone == nil || *form.Phone != *user.Phone) {
+		if *form.Phone != "" {
+			existingUser, _ := s.repository.GetByPhone(ctx, *form.Phone)
+			if existingUser != nil && existingUser.ID.Hex() != userID {
+				return errors.New("phone number already exists")
+			}
+		}
 		user.Phone = form.Phone
 	}
 	if form.ProfileImage != nil {
@@ -586,6 +611,9 @@ func (s *userService) UpdateProfile(ctx context.Context, userID string, form *Up
 	// Save to database
 	err = s.repository.Update(ctx, user)
 	if err != nil {
+		if mapped := mapUserWriteError(err); mapped != err {
+			return mapped
+		}
 		s.logger.Error("Failed to update user profile", map[string]interface{}{
 			"error":   err.Error(),
 			"user_id": userID,

pkg/ai_summarizer/client.go

@@ -107,7 +107,7 @@ func (c *Client) FetchTranslationOnDemand(ctx context.Context, reqBody Translate
 			}
 		}
 
-		resp, err := c.doTranslatePost(ctx, url, jsonBody)
+		resp, err := c.doTranslatePost(ctx, url, jsonBody, reqBody.RequestSource)
 		if err != nil {
 			lastErr = err
 			c.logger.Error("AI translate request failed", map[string]interface{}{
@@ -169,6 +169,98 @@ func (c *Client) TriggerPipelineTranslate(ctx context.Context, languages []strin
 	return &result, nil
 }
 
+// FetchAnalyzeOnDemand calls POST /ai/analyze for one tender.
+func (c *Client) FetchAnalyzeOnDemand(ctx context.Context, reqBody AnalyzeRequest) (*AnalyzeResponse, error) {
+	jsonBody, err := json.Marshal(reqBody)
+	if err != nil {
+		return nil, fmt.Errorf("ai_summarizer: failed to marshal analyze request body: %w", err)
+	}
+
+	url := c.config.APIBaseURL + "/ai/analyze"
+	httpResp, bodyBytes, err := c.doRawPost(ctx, url, jsonBody)
+	if err != nil {
+		return nil, err
+	}
+	defer httpResp.Body.Close()
+
+	if httpResp.StatusCode == http.StatusNotFound {
+		return nil, fmt.Errorf("%w: status 404, body: %s", ErrObjectNotFound, string(bodyBytes))
+	}
+	if httpResp.StatusCode >= 400 {
+		return nil, fmt.Errorf("%w: status %d, body: %s", ErrAPINonSuccess, httpResp.StatusCode, string(bodyBytes))
+	}
+
+	var result AnalyzeResponse
+	if err := json.Unmarshal(bodyBytes, &result); err != nil {
+		return nil, fmt.Errorf("ai_summarizer: failed to decode analyze response JSON: %w", err)
+	}
+
+	c.logger.Info("AI analyze request succeeded", map[string]interface{}{
+		"contract_folder_id":    result.ContractFolderID,
+		"notice_publication_id": result.NoticePublicationID,
+		"iterations":            result.Iterations,
+	})
+
+	return &result, nil
+}
+
+// TriggerPipelineSummarize calls POST /pipeline/summarize.
+func (c *Client) TriggerPipelineSummarize(ctx context.Context) (*PipelineActionResponse, error) {
+	return c.triggerPipelineAction(ctx, "/pipeline/summarize", "pipeline summarize")
+}
+
+func (c *Client) triggerPipelineAction(ctx context.Context, path, label string) (*PipelineActionResponse, error) {
+	url := c.config.APIBaseURL + path
+	httpResp, bodyBytes, err := c.doRawPost(ctx, url, nil)
+	if err != nil {
+		return nil, err
+	}
+	defer httpResp.Body.Close()
+
+	if httpResp.StatusCode >= 400 {
+		return nil, fmt.Errorf("%w: status %d, body: %s", ErrAPINonSuccess, httpResp.StatusCode, string(bodyBytes))
+	}
+
+	var result PipelineActionResponse
+	if len(bodyBytes) > 0 {
+		if err := json.Unmarshal(bodyBytes, &result); err != nil {
+			return nil, fmt.Errorf("ai_summarizer: failed to decode %s response: %w", label, err)
+		}
+	}
+
+	c.logger.Info("AI pipeline action triggered", map[string]interface{}{
+		"path":   path,
+		"status": result.Status,
+	})
+
+	return &result, nil
+}
+
+func (c *Client) doRawPost(ctx context.Context, url string, jsonBody []byte) (*http.Response, []byte, error) {
+	var body io.Reader
+	if len(jsonBody) > 0 {
+		body = bytes.NewReader(jsonBody)
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, body)
+	if err != nil {
+		return nil, nil, fmt.Errorf("ai_summarizer: failed to create request: %w", err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+	req.Header.Set("Accept", "application/json")
+
+	httpResp, err := c.httpClient.Do(req)
+	if err != nil {
+		return nil, nil, fmt.Errorf("ai_summarizer: request error: %w", err)
+	}
+
+	bodyBytes, err := io.ReadAll(httpResp.Body)
+	if err != nil {
+		httpResp.Body.Close()
+		return nil, nil, fmt.Errorf("ai_summarizer: failed to read response body: %w", err)
+	}
+	return httpResp, bodyBytes, nil
+}
+
 // doPost performs a single POST request and parses the response.
 func (c *Client) doPost(ctx context.Context, url string, jsonBody []byte) (*SummarizeResponse, error) {
 	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(jsonBody))
@@ -216,7 +308,7 @@ func (c *Client) doPost(ctx context.Context, url string, jsonBody []byte) (*Summ
 }
 
 // doTranslatePost performs a single translation POST request and parses the response.
-func (c *Client) doTranslatePost(ctx context.Context, url string, jsonBody []byte) (*TranslateResponse, error) {
+func (c *Client) doTranslatePost(ctx context.Context, url string, jsonBody []byte, requestSource string) (*TranslateResponse, error) {
 	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(jsonBody))
 	if err != nil {
 		return nil, fmt.Errorf("ai_summarizer: failed to create translation request: %w", err)
@@ -253,6 +345,17 @@ func (c *Client) doTranslatePost(ctx context.Context, url string, jsonBody []byt
 		"language":              result.Language,
 	})
 
+	if c.config.OnSuccessfulTranslation != nil {
+		if err := c.config.OnSuccessfulTranslation(ctx, requestSource); err != nil {
+			c.logger.Warn("Failed to increment AI translation success counter", map[string]interface{}{
+				"notice_publication_id": result.NoticePublicationID,
+				"language":              result.Language,
+				"request_source":        requestSource,
+				"error":                 err.Error(),
+			})
+		}
+	}
+
 	return result, nil
 }
 

pkg/ai_summarizer/config.go

@@ -1,6 +1,7 @@
 package ai_summarizer
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"time"
@@ -15,6 +16,10 @@ type Config struct {
 	APIRetryCount int           // Number of retry attempts for failed API requests
 	APIRetryDelay time.Duration // Delay between retry attempts
 
+	// OnSuccessfulTranslation is called after a successful POST /ai/translate request.
+	// source identifies the caller (daily_job or manual_trigger).
+	OnSuccessfulTranslation func(ctx context.Context, source string) error
+
 	// MinIO storage settings
 	MinioEndpoint  string        // MinIO endpoint, e.g. "ai-host:9000"
 	MinioAccessKey string        // MinIO access key ID

pkg/ai_summarizer/entities.go

@@ -2,58 +2,20 @@ package ai_summarizer
 
 import (
 	"strings"
-	"time"
 )
 
-// SummarizeRequest represents the request payload sent to the AI service
-// POST /ai/summarize endpoint for on-demand summarization.
+// SummarizeRequest is the payload for POST /ai/summarize.
 type SummarizeRequest struct {
-	ContractFolderID    string   `json:"contract_folder_id"`
-	NoticePublicationID string   `json:"notice_publication_id"`
-	DocumentURL         string   `json:"document_url,omitempty"`
-	Title               string   `json:"title,omitempty"`
-	Description         string   `json:"description,omitempty"`
-	Country             string   `json:"country,omitempty"`
-	EstimatedValue      *float64 `json:"estimated_value,omitempty"`
-	Currency            string   `json:"currency,omitempty"`
-	SubmissionDeadline  string   `json:"submission_deadline,omitempty"`
-	AuthorityName       string   `json:"authority_name,omitempty"`
-}
-
-// SummarizeRequestInput collects fields used to build SummarizeRequest.
-type SummarizeRequestInput struct {
-	ContractFolderID    string
-	NoticePublicationID string
-	DocumentURL         string
-	Title               string
-	Description         string
-	Country             string
-	Currency            string
-	SubmissionDeadline  int64 // Unix seconds; omitted from request when zero
-	EstimatedValue      float64
-	AuthorityName       string
+	ContractFolderID    string `json:"contract_folder_id"`
+	NoticePublicationID string `json:"notice_publication_id"`
 }
 
 // NewSummarizeRequest builds a SummarizeRequest for POST /ai/summarize.
-func NewSummarizeRequest(in SummarizeRequestInput) SummarizeRequest {
-	req := SummarizeRequest{
-		ContractFolderID:    strings.TrimSpace(in.ContractFolderID),
-		NoticePublicationID: strings.TrimSpace(in.NoticePublicationID),
-		DocumentURL:         strings.TrimSpace(in.DocumentURL),
-		Title:               in.Title,
-		Description:         in.Description,
-		Country:             in.Country,
-		Currency:            in.Currency,
-		AuthorityName:       strings.TrimSpace(in.AuthorityName),
+func NewSummarizeRequest(contractFolderID, noticePublicationID string) SummarizeRequest {
+	return SummarizeRequest{
+		ContractFolderID:    strings.TrimSpace(contractFolderID),
+		NoticePublicationID: strings.TrimSpace(noticePublicationID),
 	}
-	if in.EstimatedValue > 0 {
-		v := in.EstimatedValue
-		req.EstimatedValue = &v
-	}
-	if in.SubmissionDeadline > 0 {
-		req.SubmissionDeadline = time.Unix(in.SubmissionDeadline, 0).UTC().Format(time.RFC3339)
-	}
-	return req
 }
 
 // SummarizeResponse represents the response from the AI summarization API.
@@ -63,6 +25,8 @@ type SummarizeResponse struct {
 	NoticeID            string            `json:"notice_id"` // legacy field name
 	Summaries           []DocumentSummary `json:"summaries"`
 	OverallSummary      *string           `json:"overall_summary"`
+	RollupS             float64           `json:"rollup_s"`
+	TotalS              float64           `json:"total_s"`
 }
 
 // ResolvedNoticePublicationID returns the notice publication ID from the response.
@@ -82,7 +46,7 @@ type DocumentSummary struct {
 	DocumentType string `json:"document_type"`
 	Summary      string `json:"summary"`
 	SummaryModel string `json:"summary_model"`
-	Error        string `json:"error"` // empty string when no error
+	Error        string `json:"error"` // empty when no error
 }
 
 // ProcedureDocumentsSummary reports scraped documents stored under one procedure folder in MinIO.
@@ -101,16 +65,19 @@ type StoredDocument struct {
 	DocumentType string `json:"document_type,omitempty"`
 }
 
-// TranslateRequest represents the request payload sent to the AI service
-// POST /ai/translate endpoint.
+// TranslateRequest is the payload for POST /ai/translate.
 type TranslateRequest struct {
 	ContractFolderID    string `json:"contract_folder_id"`
 	NoticePublicationID string `json:"notice_publication_id"`
-	Title               string `json:"title"`
-	Description         string `json:"description"`
 	Language            string `json:"language"`
+	RequestSource       string `json:"-"` // internal: daily_job | manual_trigger
 }
 
+const (
+	TranslationRequestSourceDailyJob      = "daily_job"
+	TranslationRequestSourceManualTrigger = "manual_trigger"
+)
+
 // TranslateResponse represents the translation payload from POST /ai/translate.
 type TranslateResponse struct {
 	ContractFolderID      string `json:"contract_folder_id"`
@@ -120,6 +87,41 @@ type TranslateResponse struct {
 	TranslatedDescription string `json:"translated_description"`
 }
 
+// AnalyzeRequest is the payload for POST /ai/analyze.
+type AnalyzeRequest struct {
+	ContractFolderID    string `json:"contract_folder_id"`
+	NoticePublicationID string `json:"notice_publication_id"`
+}
+
+// NewAnalyzeRequest builds an AnalyzeRequest for POST /ai/analyze.
+func NewAnalyzeRequest(contractFolderID, noticePublicationID string) AnalyzeRequest {
+	return AnalyzeRequest{
+		ContractFolderID:    strings.TrimSpace(contractFolderID),
+		NoticePublicationID: strings.TrimSpace(noticePublicationID),
+	}
+}
+
+// AnalyzeDocumentStats describes one document touched during agentic analysis.
+type AnalyzeDocumentStats struct {
+	Filename     string `json:"filename"`
+	SizeBytes    int64  `json:"size_bytes"`
+	TimesRead    int    `json:"times_read"`
+	TimesQueried int    `json:"times_queried"`
+	Description  string `json:"description"`
+}
+
+// AnalyzeResponse is the payload from POST /ai/analyze.
+type AnalyzeResponse struct {
+	ContractFolderID    string                 `json:"contract_folder_id"`
+	NoticePublicationID string                 `json:"notice_publication_id"`
+	Summary             string                 `json:"summary"`
+	Documents           []AnalyzeDocumentStats `json:"documents"`
+	Scratchpad          map[string]interface{} `json:"scratchpad"`
+	ToolCallsLog        []string               `json:"tool_calls_log"`
+	Iterations          int                    `json:"iterations"`
+	Model               string                 `json:"model"`
+}
+
 // PipelineTranslateRequest represents the request payload for POST /pipeline/translate.
 type PipelineTranslateRequest struct {
 	Languages []string `json:"languages"`
@@ -131,6 +133,11 @@ type PipelineTranslateResponse struct {
 	Languages []string `json:"languages"`
 }
 
+// PipelineActionResponse is returned by fire-and-forget pipeline endpoints.
+type PipelineActionResponse struct {
+	Status string `json:"status"`
+}
+
 // StoredTranslation is one language entry inside tender.json translations map.
 type StoredTranslation struct {
 	Title       string `json:"title"`
@@ -167,6 +174,7 @@ type TenderJSON struct {
 	Summary                string                       `json:"summary"`
 	OverallSummary         *string                      `json:"overall_summary"`
 	OverallSummaryCamel    *string                      `json:"overallSummary"`
+	Summaries              []DocumentSummary            `json:"summaries"`
 	SummarizeStatus        SummarizeStatus              `json:"summarize_status"`
 	SummarizeStatusCamel   SummarizeStatus              `json:"summarizeStatus"`
 	Translations           map[string]StoredTranslation `json:"translations"`
@@ -219,23 +227,31 @@ func (t *TenderJSON) translationsDone() []string {
 	return t.TranslationStatusCamel.languagesDone()
 }
 
-// translationForLanguage returns stored translation text when the pipeline marked
-// the language done and non-empty title exists.
+// storedTranslationText returns translation text from the translations map when
+// title is non-empty. It does not require translation_status.done — on-demand
+// writes may persist translations before updating status.
+func (t *TenderJSON) storedTranslationText(language string) (StoredTranslation, bool) {
+	if t == nil || t.Translations == nil {
+		return StoredTranslation{}, false
+	}
+	lang := strings.ToLower(strings.TrimSpace(language))
+	for key, entry := range t.Translations {
+		if strings.ToLower(strings.TrimSpace(key)) != lang {
+			continue
+		}
+		if strings.TrimSpace(entry.Title) == "" {
+			continue
+		}
+		return entry, true
+	}
+	return StoredTranslation{}, false
+}
+
+// translationForLanguage returns stored translation when the pipeline marked the
+// language done and non-empty title exists.
 func (t *TenderJSON) translationForLanguage(language string) (StoredTranslation, bool) {
 	if t == nil || !languageInDone(t.translationsDone(), language) {
 		return StoredTranslation{}, false
 	}
-	lang := strings.ToLower(strings.TrimSpace(language))
-	if t.Translations != nil {
-		for key, entry := range t.Translations {
-			if strings.ToLower(strings.TrimSpace(key)) != lang {
-				continue
-			}
-			if strings.TrimSpace(entry.Title) == "" {
-				continue
-			}
-			return entry, true
-		}
-	}
-	return StoredTranslation{}, false
+	return t.storedTranslationText(language)
 }

pkg/ai_summarizer/storage.go

@@ -326,8 +326,9 @@ func (s *StorageClient) GetSummaryFromStorage(ctx context.Context, contractFolde
 	return s.getSummaryFromObjectKey(ctx, prefix+"tender.json", contractFolderID, noticePublicationID)
 }
 
-// GetTranslationFromStorage reads tender.json and returns the translation for a language
-// when translation_status.done includes that language.
+// GetTranslationFromStorage reads tender.json and returns the translation for a language.
+// It prefers entries marked done in translation_status; if missing, it falls back to
+// translations[<lang>] when title is non-empty (on-demand API may lag status updates).
 func (s *StorageClient) GetTranslationFromStorage(ctx context.Context, contractFolderID, noticePublicationID, language string) (StoredTranslation, error) {
 	if strings.TrimSpace(contractFolderID) == "" {
 		return StoredTranslation{}, fmt.Errorf("ai_summarizer: contractFolderID is required")
@@ -353,14 +354,16 @@ func (s *StorageClient) getTranslationFromObjectKey(ctx context.Context, objectK
 		return StoredTranslation{}, err
 	}
 
+	if entry, ok := tenderJSON.translationForLanguage(language); ok {
+		return entry, nil
+	}
+	if entry, ok := tenderJSON.storedTranslationText(language); ok {
+		return entry, nil
+	}
 	if !languageInDone(tenderJSON.translationsDone(), language) {
 		return StoredTranslation{}, ErrTranslationNotReady
 	}
-	entry, ok := tenderJSON.translationForLanguage(language)
-	if !ok {
-		return StoredTranslation{}, ErrNoTranslation
-	}
-	return entry, nil
+	return StoredTranslation{}, ErrNoTranslation
 }
 
 // getSummaryFromObjectKey reads tender.json at objectKey and returns summary text when ready.
@@ -432,6 +435,39 @@ func (s *StorageClient) getSummaryFromObjectKey(ctx context.Context, objectKey,
 	return summaryText, nil
 }
 
+// GetDocumentSummariesFromStorage reads per-document summaries from tender.json
+// when summarize_status.done is true.
+func (s *StorageClient) GetDocumentSummariesFromStorage(ctx context.Context, contractFolderID, noticePublicationID string) ([]DocumentSummary, error) {
+	if strings.TrimSpace(contractFolderID) == "" {
+		return nil, fmt.Errorf("ai_summarizer: contractFolderID is required")
+	}
+	if strings.TrimSpace(noticePublicationID) == "" {
+		return nil, fmt.Errorf("ai_summarizer: noticePublicationID is required")
+	}
+
+	prefix, err := s.resolveNoticeStoragePrefix(ctx, contractFolderID, noticePublicationID)
+	if err != nil {
+		return nil, err
+	}
+
+	tenderJSON, err := s.readTenderJSONAtKey(ctx, prefix+"tender.json")
+	if err != nil {
+		return nil, err
+	}
+
+	done, _ := tenderJSON.resolved()
+	if !done {
+		return nil, ErrSummaryNotReady
+	}
+	if len(tenderJSON.Summaries) == 0 {
+		return []DocumentSummary{}, nil
+	}
+
+	out := make([]DocumentSummary, len(tenderJSON.Summaries))
+	copy(out, tenderJSON.Summaries)
+	return out, nil
+}
+
 // IsSummaryReady is a convenience method that checks whether the tender.json
 // exists and the summarize_status.done flag is true, without returning the
 // actual summary text.

pkg/mongo/counter.go

@@ -0,0 +1,130 @@
+package mongo
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"time"
+
+	"go.mongodb.org/mongo-driver/v2/bson"
+	mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
+	"go.mongodb.org/mongo-driver/v2/mongo/options"
+)
+
+const (
+	metricsCountersCollection                   = "metrics_counters"
+	AITranslationSuccessCounterKey              = "ai_translation_successful_requests"
+	AITranslationSuccessCounterKeyDailyJob      = "ai_translation_successful_requests_daily_job"
+	AITranslationSuccessCounterKeyManualTrigger = "ai_translation_successful_requests_manual_trigger"
+)
+
+// Counter provides atomic increment/read operations for named metrics stored in MongoDB.
+type Counter struct {
+	mongo *ConnectionManager
+}
+
+// NewCounter creates a counter backed by the metrics_counters collection.
+func NewCounter(mongo *ConnectionManager) *Counter {
+	return &Counter{mongo: mongo}
+}
+
+// Increment atomically increments the counter identified by key and returns the new value.
+func (c *Counter) Increment(ctx context.Context, key string) (int64, error) {
+	if key == "" {
+		return 0, fmt.Errorf("mongo counter: key is required")
+	}
+
+	var res struct {
+		Count int64 `bson:"count"`
+	}
+	err := c.mongo.GetCollection(metricsCountersCollection).FindOneAndUpdate(
+		ctx,
+		bson.M{"_id": key},
+		bson.M{
+			"$inc":         bson.M{"count": 1},
+			"$set":         bson.M{"updated_at": time.Now().Unix()},
+			"$setOnInsert": bson.M{"_id": key},
+		},
+		options.FindOneAndUpdate().SetUpsert(true).SetReturnDocument(options.After),
+	).Decode(&res)
+	if err != nil {
+		return 0, fmt.Errorf("mongo counter increment %q: %w", key, err)
+	}
+
+	return res.Count, nil
+}
+
+// Get returns the current value of the counter identified by key (0 when missing).
+func (c *Counter) Get(ctx context.Context, key string) (int64, error) {
+	if key == "" {
+		return 0, fmt.Errorf("mongo counter: key is required")
+	}
+
+	var doc struct {
+		Count int64 `bson:"count"`
+	}
+	err := c.mongo.GetCollection(metricsCountersCollection).FindOne(ctx, bson.M{"_id": key}).Decode(&doc)
+	if err != nil {
+		if errors.Is(err, mongodriver.ErrNoDocuments) {
+			return 0, nil
+		}
+		return 0, fmt.Errorf("mongo counter get %q: %w", key, err)
+	}
+
+	return doc.Count, nil
+}
+
+// AITranslationSuccessCallback returns a callback that increments successful AI translation counters.
+func AITranslationSuccessCallback(counter *Counter) func(ctx context.Context, source string) error {
+	return func(ctx context.Context, source string) error {
+		if _, err := counter.Increment(ctx, AITranslationSuccessCounterKey); err != nil {
+			return err
+		}
+		if _, err := counter.Increment(ctx, AITranslationSuccessDailyCounterKey(time.Now().UTC())); err != nil {
+			return err
+		}
+		if key := AITranslationSuccessCounterKeyForSource(source); key != "" {
+			_, err := counter.Increment(ctx, key)
+			return err
+		}
+		return nil
+	}
+}
+
+// AITranslationSuccessDailyCounterKey returns the metrics key for successful translations on a given UTC day.
+func AITranslationSuccessDailyCounterKey(day time.Time) string {
+	return AITranslationSuccessCounterKey + "_day:" + day.UTC().Format("2006-01-02")
+}
+
+// GetDailyCounts returns counter values for each UTC day from start through end inclusive.
+func (c *Counter) GetDailyCounts(ctx context.Context, startDay, endDay time.Time) (map[string]int64, error) {
+	startDay = time.Date(startDay.Year(), startDay.Month(), startDay.Day(), 0, 0, 0, 0, time.UTC)
+	endDay = time.Date(endDay.Year(), endDay.Month(), endDay.Day(), 0, 0, 0, 0, time.UTC)
+	if endDay.Before(startDay) {
+		return map[string]int64{}, nil
+	}
+
+	counts := make(map[string]int64)
+	for day := startDay; !day.After(endDay); day = day.AddDate(0, 0, 1) {
+		date := day.Format("2006-01-02")
+		value, err := c.Get(ctx, AITranslationSuccessDailyCounterKey(day))
+		if err != nil {
+			return nil, err
+		}
+		counts[date] = value
+	}
+
+	return counts, nil
+}
+
+// AITranslationSuccessCounterKeyForSource maps a request source to its metrics counter key.
+func AITranslationSuccessCounterKeyForSource(source string) string {
+	switch source {
+	case "daily_job":
+		return AITranslationSuccessCounterKeyDailyJob
+	case "manual_trigger":
+		return AITranslationSuccessCounterKeyManualTrigger
+	default:
+		return ""
+	}
+}

pkg/mongo/duplicate.go

@@ -0,0 +1,192 @@
+package mongo
+
+import (
+	"errors"
+	"regexp"
+	"strings"
+
+	mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
+)
+
+var dupKeyFieldNamePattern = regexp.MustCompile(`(?:^|[,{]\s*)([a-zA-Z_][a-zA-Z0-9_]*)\s*:`)
+
+// IsDuplicateKeyError reports whether err is a MongoDB E11000 duplicate key error.
+func IsDuplicateKeyError(err error) bool {
+	if err == nil {
+		return false
+	}
+
+	if len(duplicateKeyMessages(err)) > 0 {
+		return true
+	}
+
+	if hasStructuredWriteException(err) {
+		return false
+	}
+
+	msg := strings.ToLower(err.Error())
+	return strings.Contains(msg, "e11000") || strings.Contains(msg, "duplicate key")
+}
+
+// DuplicateKeyMatchesField reports whether a duplicate key error involves the given field
+// by parsing WriteException messages (index name and dup key document keys), not broad
+// substring matching on the full error text.
+func DuplicateKeyMatchesField(err error, field string) bool {
+	if err == nil || field == "" {
+		return false
+	}
+
+	field = strings.ToLower(field)
+	msgs := duplicateKeyMessages(err)
+	if len(msgs) == 0 {
+		if hasStructuredWriteException(err) {
+			return false
+		}
+		if !IsDuplicateKeyError(err) {
+			return false
+		}
+		msgs = []string{err.Error()}
+	}
+	for _, msg := range msgs {
+		if duplicateKeyMessageMatchesField(msg, field) {
+			return true
+		}
+	}
+	return false
+}
+
+func duplicateKeyMessages(err error) []string {
+	var msgs []string
+
+	var writeException mongodriver.WriteException
+	if errors.As(err, &writeException) {
+		for _, writeErr := range writeException.WriteErrors {
+			if writeErr.Code == 11000 {
+				msgs = append(msgs, writeErr.Message)
+			}
+		}
+	}
+
+	var bulkWriteException mongodriver.BulkWriteException
+	if errors.As(err, &bulkWriteException) {
+		for _, writeErr := range bulkWriteException.WriteErrors {
+			if writeErr.Code == 11000 {
+				msgs = append(msgs, writeErr.Message)
+			}
+		}
+	}
+
+	return msgs
+}
+
+func hasStructuredWriteException(err error) bool {
+	var writeException mongodriver.WriteException
+	if errors.As(err, &writeException) {
+		return true
+	}
+
+	var bulkWriteException mongodriver.BulkWriteException
+	return errors.As(err, &bulkWriteException)
+}
+
+func duplicateKeyMessageMatchesField(message, field string) bool {
+	message = strings.ToLower(message)
+
+	if indexName := extractDuplicateIndexName(message); indexName != "" {
+		if indexNameMatchesField(indexName, field) {
+			return true
+		}
+	}
+
+	for _, key := range extractDuplicateKeyFields(message) {
+		if key == field {
+			return true
+		}
+	}
+
+	return false
+}
+
+func extractDuplicateIndexName(message string) string {
+	const prefix = "index: "
+	start := strings.Index(message, prefix)
+	if start < 0 {
+		return ""
+	}
+
+	rest := strings.TrimSpace(message[start+len(prefix):])
+	if rest == "" {
+		return ""
+	}
+
+	end := strings.IndexByte(rest, ' ')
+	if end >= 0 {
+		return rest[:end]
+	}
+
+	return rest
+}
+
+func indexNameMatchesField(indexName, field string) bool {
+	switch indexName {
+	case field, field + "_idx", field + "_1":
+		return true
+	default:
+		return false
+	}
+}
+
+func extractDuplicateKeyFields(message string) []string {
+	const prefix = "dup key:"
+	start := strings.Index(message, prefix)
+	if start < 0 {
+		return nil
+	}
+
+	document := strings.TrimSpace(message[start+len(prefix):])
+	if !strings.HasPrefix(document, "{") {
+		return nil
+	}
+
+	depth := 0
+	end := -1
+	for i, char := range document {
+		switch char {
+		case '{':
+			depth++
+		case '}':
+			depth--
+			if depth == 0 {
+				end = i
+			}
+		}
+		if end >= 0 {
+			break
+		}
+	}
+	if end <= 0 {
+		return nil
+	}
+
+	inner := strings.TrimSpace(document[1:end])
+	matches := dupKeyFieldNamePattern.FindAllStringSubmatch(inner, -1)
+	if len(matches) == 0 {
+		return nil
+	}
+
+	fields := make([]string, 0, len(matches))
+	seen := make(map[string]struct{}, len(matches))
+	for _, match := range matches {
+		if len(match) < 2 {
+			continue
+		}
+		key := strings.ToLower(match[1])
+		if _, ok := seen[key]; ok {
+			continue
+		}
+		seen[key] = struct{}{}
+		fields = append(fields, key)
+	}
+
+	return fields
+}

pkg/mongo/duplicate_test.go

@@ -0,0 +1,303 @@
+package mongo
+
+import (
+	"errors"
+	"fmt"
+	"testing"
+
+	mongodriver "go.mongodb.org/mongo-driver/v2/mongo"
+)
+
+func TestIsDuplicateKeyError(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		err  error
+		want bool
+	}{
+		{
+			name: "nil error",
+			err:  nil,
+			want: false,
+		},
+		{
+			name: "unrelated error",
+			err:  errors.New("connection refused"),
+			want: false,
+		},
+		{
+			name: "write exception duplicate key",
+			err: writeExceptionWithMessage(
+				11000,
+				`E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "+1234567890" }`,
+			),
+			want: true,
+		},
+		{
+			name: "write exception non duplicate code",
+			err: writeExceptionWithMessage(
+				121,
+				`Document failed validation`,
+			),
+			want: false,
+		},
+		{
+			name: "wrapped write exception duplicate key",
+			err: fmt.Errorf(
+				"failed to create user: %w",
+				writeExceptionWithMessage(
+					11000,
+					`E11000 duplicate key error collection: tm.users index: username_idx dup key: { username: "admin" }`,
+				),
+			),
+			want: true,
+		},
+		{
+			name: "bulk write exception duplicate key",
+			err: bulkWriteExceptionWithMessage(
+				11000,
+				`E11000 duplicate key error collection: tm.customers index: phone_idx dup key: { phone: "+1234567890" }`,
+			),
+			want: true,
+		},
+		{
+			name: "string fallback duplicate key",
+			err:  errors.New(`write exception: write errors: [E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "+1234567890" }] `),
+			want: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			if got := IsDuplicateKeyError(tt.err); got != tt.want {
+				t.Fatalf("IsDuplicateKeyError() = %v, want %v", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestDuplicateKeyMatchesField(t *testing.T) {
+	t.Parallel()
+
+	const phoneMsg = `E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "+1234567890" }`
+	const usernameMsg = `E11000 duplicate key error collection: tm.users index: username_idx dup key: { username: "admin" }`
+	const emailMsg = `E11000 duplicate key error collection: tm.customers index: email_1 dup key: { email: "user@example.com" }`
+	const phoneValueContainsNameMsg = `E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "username-like-value" }`
+
+	tests := []struct {
+		name  string
+		err   error
+		field string
+		want  bool
+	}{
+		{
+			name:  "nil error",
+			err:   nil,
+			field: "phone",
+			want:  false,
+		},
+		{
+			name:  "empty field",
+			err:   writeExceptionWithMessage(11000, phoneMsg),
+			field: "",
+			want:  false,
+		},
+		{
+			name:  "phone duplicate matches phone field",
+			err:   writeExceptionWithMessage(11000, phoneMsg),
+			field: "phone",
+			want:  true,
+		},
+		{
+			name:  "phone duplicate does not match email field",
+			err:   writeExceptionWithMessage(11000, phoneMsg),
+			field: "email",
+			want:  false,
+		},
+		{
+			name:  "username duplicate matches username field",
+			err:   writeExceptionWithMessage(11000, usernameMsg),
+			field: "username",
+			want:  true,
+		},
+		{
+			name:  "username duplicate does not match name substring",
+			err:   writeExceptionWithMessage(11000, usernameMsg),
+			field: "name",
+			want:  false,
+		},
+		{
+			name:  "email duplicate matches email field via default index suffix",
+			err:   writeExceptionWithMessage(11000, emailMsg),
+			field: "email",
+			want:  true,
+		},
+		{
+			name:  "dup key value substring does not match unrelated field",
+			err:   writeExceptionWithMessage(11000, phoneValueContainsNameMsg),
+			field: "name",
+			want:  false,
+		},
+		{
+			name:  "wrapped duplicate key matches field",
+			err: fmt.Errorf(
+				"failed to register customer: %w",
+				writeExceptionWithMessage(11000, phoneMsg),
+			),
+			field: "phone",
+			want:  true,
+		},
+		{
+			name:  "non duplicate write error does not match",
+			err:   writeExceptionWithMessage(121, phoneMsg),
+			field: "phone",
+			want:  false,
+		},
+		{
+			name:  "unrelated error does not match",
+			err:   errors.New("timeout"),
+			field: "phone",
+			want:  false,
+		},
+		{
+			name:  "string fallback duplicate key matches field",
+			err:   errors.New(`write exception: write errors: [E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "+1234567890" }] `),
+			field: "phone",
+			want:  true,
+		},
+		{
+			name:  "string fallback duplicate key does not match unrelated field",
+			err:   errors.New(`write exception: write errors: [E11000 duplicate key error collection: tm.users index: username_idx dup key: { username: "admin" }] `),
+			field: "name",
+			want:  false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			if got := DuplicateKeyMatchesField(tt.err, tt.field); got != tt.want {
+				t.Fatalf("DuplicateKeyMatchesField(%q) = %v, want %v", tt.field, got, tt.want)
+			}
+		})
+	}
+}
+
+func TestExtractDuplicateIndexName(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		message string
+		want    string
+	}{
+		{
+			name:    "phone index",
+			message: `E11000 duplicate key error collection: tm.users index: phone_idx dup key: { phone: "+1234567890" }`,
+			want:    "phone_idx",
+		},
+		{
+			name:    "missing index segment",
+			message: `E11000 duplicate key error collection: tm.users dup key: { phone: "+1234567890" }`,
+			want:    "",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			if got := extractDuplicateIndexName(tt.message); got != tt.want {
+				t.Fatalf("extractDuplicateIndexName() = %q, want %q", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestExtractDuplicateKeyFields(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name    string
+		message string
+		want    []string
+	}{
+		{
+			name:    "single field",
+			message: `index: phone_idx dup key: { phone: "+1234567890" }`,
+			want:    []string{"phone"},
+		},
+		{
+			name:    "compound dup key",
+			message: `index: user_company_idx dup key: { user_id: ObjectId('507f1f77bcf86cd799439011'), company_id: ObjectId('507f1f77bcf86cd799439012') }`,
+			want:    []string{"user_id", "company_id"},
+		},
+		{
+			name:    "value containing colon does not add false field",
+			message: `index: phone_idx dup key: { phone: "office:123" }`,
+			want:    []string{"phone"},
+		},
+		{
+			name:    "missing dup key segment",
+			message: `index: phone_idx`,
+			want:    nil,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			got := extractDuplicateKeyFields(tt.message)
+			if len(got) != len(tt.want) {
+				t.Fatalf("extractDuplicateKeyFields() = %v, want %v", got, tt.want)
+			}
+			for i := range got {
+				if got[i] != tt.want[i] {
+					t.Fatalf("extractDuplicateKeyFields() = %v, want %v", got, tt.want)
+				}
+			}
+		})
+	}
+}
+
+func TestIndexNameMatchesField(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		indexName string
+		field     string
+		want      bool
+	}{
+		{indexName: "phone_idx", field: "phone", want: true},
+		{indexName: "phone_1", field: "phone", want: true},
+		{indexName: "phone", field: "phone", want: true},
+		{indexName: "username_idx", field: "name", want: false},
+		{indexName: "email_idx", field: "mail", want: false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.indexName+"_"+tt.field, func(t *testing.T) {
+			t.Parallel()
+			if got := indexNameMatchesField(tt.indexName, tt.field); got != tt.want {
+				t.Fatalf("indexNameMatchesField(%q, %q) = %v, want %v", tt.indexName, tt.field, got, tt.want)
+			}
+		})
+	}
+}
+
+func writeExceptionWithMessage(code int, message string) error {
+	return mongodriver.WriteException{
+		WriteErrors: []mongodriver.WriteError{
+			{Code: code, Message: message},
+		},
+	}
+}
+
+func bulkWriteExceptionWithMessage(code int, message string) error {
+	return mongodriver.BulkWriteException{
+		WriteErrors: []mongodriver.BulkWriteError{
+			{WriteError: mongodriver.WriteError{Code: code, Message: message}},
+		},
+	}
+}

pkg/queue/config.go

@@ -1,53 +0,0 @@
-package queue
-
-import (
-	"time"
-)
-
-/*
-Queue Configuration and Constants
-
-This package provides a Redis-based job queue system for managing
-document scraping tasks in a distributed manner.
-*/
-
-const (
-	// Queue names
-	DocumentScraperQueue = "scraper:document:queue"
-	DocumentScraperDLQ   = "scraper:document:dlq" // Dead Letter Queue
-
-	// Job status keys
-	JobStatusPrefix = "job:"
-	QueueMetricsKey = "queue:metrics"
-
-	// Default timeouts
-	DefaultJobTimeout    = 5 * time.Minute
-	DefaultMaxRetries    = 3
-	DefaultRetryBackoff  = 10 * time.Second
-	MaxRetryBackoff      = 5 * time.Minute  // Cap retry backoff at 5 minutes
-	InProgressExpiration = 10 * time.Minute // Job must complete within this time
-)
-
-// QueueConfig represents configuration for the queue service
-type QueueConfig struct {
-	RedisURL       string
-	MaxRetries     int
-	RetryBackoff   time.Duration
-	JobTimeout     time.Duration
-	PoolSize       int
-	IsDLQEnabled   bool
-	MaxQueueLength int // 0 means unlimited
-}
-
-// DefaultConfig returns default queue configuration
-func DefaultConfig() QueueConfig {
-	return QueueConfig{
-		RedisURL:       "redis://localhost:6379",
-		MaxRetries:     DefaultMaxRetries,
-		RetryBackoff:   DefaultRetryBackoff,
-		JobTimeout:     DefaultJobTimeout,
-		PoolSize:       10,
-		IsDLQEnabled:   true,
-		MaxQueueLength: 0, // Unlimited by default
-	}
-}

pkg/queue/models.go

@@ -1,85 +0,0 @@
-package queue
-
-import (
-	"encoding/json"
-	"time"
-)
-
-// JobStatus represents the status of a queue job
-type JobStatus string
-
-const (
-	StatusPending    JobStatus = "pending"
-	StatusInProgress JobStatus = "in_progress"
-	StatusCompleted  JobStatus = "completed"
-	StatusFailed     JobStatus = "failed"
-	StatusRetrying   JobStatus = "retrying"
-)
-
-// DocumentScraperJob represents a document scraping task in the queue
-type DocumentScraperJob struct {
-	ID                 string `json:"id"`                  // Unique job ID (typically tender ID)
-	NoticeID           string `json:"notice_id"`           // Notice publication ID
-	NoticeURL          string `json:"notice_url"`          // Document URL to scrape
-	TenderID           string `json:"tender_id,omitempty"` // Associated tender ID from MongoDB
-	CreatedAt          int64  `json:"created_at"`
-	UpdatedAt          int64  `json:"updated_at"`
-	Attempts           int    `json:"attempts"`
-	MaxRetries         int    `json:"max_retries"`
-	Status             string `json:"status"`
-	Error              string `json:"error,omitempty"`
-	NextRetryTime      int64  `json:"next_retry_time,omitempty"`
-	CompletedAt        int64  `json:"completed_at,omitempty"`
-	ProcessingStartedAt int64 `json:"processing_started_at,omitempty"`
-}
-
-// NewDocumentScraperJob creates a new document scraper job
-func NewDocumentScraperJob(noticeID, noticeURL, tenderID string, maxRetries int) *DocumentScraperJob {
-	now := time.Now().Unix()
-	return &DocumentScraperJob{
-		ID:         noticeID,
-		NoticeID:   noticeID,
-		NoticeURL:  noticeURL,
-		TenderID:   tenderID,
-		CreatedAt:  now,
-		UpdatedAt:  now,
-		Attempts:   0,
-		MaxRetries: maxRetries,
-		Status:     string(StatusPending),
-	}
-}
-
-// ToJSON marshals the job to JSON
-func (j *DocumentScraperJob) ToJSON() (string, error) {
-	data, err := json.Marshal(j)
-	return string(data), err
-}
-
-// FromJSON unmarshals JSON to DocumentScraperJob
-func (j *DocumentScraperJob) FromJSON(data string) error {
-	return json.Unmarshal([]byte(data), j)
-}
-
-// QueueMetrics represents queue performance metrics
-type QueueMetrics struct {
-	TotalJobs       int64         `json:"total_jobs"`
-	PendingJobs     int64         `json:"pending_jobs"`
-	InProgressJobs  int64         `json:"in_progress_jobs"`
-	CompletedJobs   int64         `json:"completed_jobs"`
-	FailedJobs      int64         `json:"failed_jobs"`
-	AverageWaitTime time.Duration `json:"average_wait_time"`
-	AverageProcessTime time.Duration `json:"average_process_time"`
-	UpdatedAt       int64         `json:"updated_at"`
-}
-
-// JobResult represents the result of a completed job
-type JobResult struct {
-	JobID         string                 `json:"job_id"`
-	Status        string                 `json:"status"`
-	Success       bool                   `json:"success"`
-	Error         string                 `json:"error,omitempty"`
-	CompletedAt   int64                  `json:"completed_at"`
-	Duration      int64                  `json:"duration_ms"`
-	UploadedCount int                    `json:"uploaded_count"`
-	Details       map[string]interface{} `json:"details,omitempty"`
-}

pkg/queue/queue.go

@@ -1,357 +0,0 @@
-package queue
-
-import (
-	"context"
-	"fmt"
-	"time"
-	"tm/pkg/logger"
-
-	"github.com/redis/go-redis/v9"
-)
-
-// Queue interface defines the queue operations
-type Queue interface {
-	// Job operations
-	Enqueue(ctx context.Context, job *DocumentScraperJob) error
-	Dequeue(ctx context.Context) (*DocumentScraperJob, error)
-	AckJob(ctx context.Context, jobID string) error
-	NackJob(ctx context.Context, jobID string, err error) error
-	GetJobStatus(ctx context.Context, jobID string) (JobStatus, error)
-
-	// Queue inspection
-	GetMetrics(ctx context.Context) (*QueueMetrics, error)
-	GetQueueSize(ctx context.Context) (int64, error)
-	GetDLQSize(ctx context.Context) (int64, error)
-	GetInProgressJobs(ctx context.Context) ([]DocumentScraperJob, error)
-
-	// Cleanup operations
-	PurgeQueue(ctx context.Context) error
-	PurgeDLQ(ctx context.Context) error
-	Close() error
-}
-
-// RedisQueue implements Queue interface using Redis
-type RedisQueue struct {
-	client *redis.Client
-	config QueueConfig
-	logger logger.Logger
-}
-
-// NewRedisQueue creates a new Redis-based queue
-func NewRedisQueue(config QueueConfig, logger logger.Logger) (Queue, error) {
-	opt, err := redis.ParseURL(config.RedisURL)
-	if err != nil {
-		logger.Error("Failed to parse Redis URL", map[string]interface{}{
-			"url":   config.RedisURL,
-			"error": err.Error(),
-		})
-		return nil, fmt.Errorf("failed to parse redis url: %w", err)
-	}
-
-	opt.PoolSize = config.PoolSize
-	client := redis.NewClient(opt)
-
-	// Test connection
-	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
-	defer cancel()
-
-	if err := client.Ping(ctx).Err(); err != nil {
-		logger.Error("Failed to connect to Redis", map[string]interface{}{
-			"error": err.Error(),
-		})
-		return nil, fmt.Errorf("failed to connect to redis: %w", err)
-	}
-
-	logger.Info("Redis queue initialized successfully", map[string]interface{}{
-		"pool_size": config.PoolSize,
-	})
-
-	return &RedisQueue{
-		client: client,
-		config: config,
-		logger: logger,
-	}, nil
-}
-
-// Enqueue adds a job to the queue
-func (q *RedisQueue) Enqueue(ctx context.Context, job *DocumentScraperJob) error {
-	if q.config.MaxQueueLength > 0 {
-		size, err := q.GetQueueSize(ctx)
-		if err != nil {
-			q.logger.Warn("Failed to check queue size", map[string]interface{}{
-				"error": err.Error(),
-			})
-		} else if size >= int64(q.config.MaxQueueLength) {
-			return fmt.Errorf("queue is full: %d/%d", size, q.config.MaxQueueLength)
-		}
-	}
-
-	job.CreatedAt = time.Now().Unix()
-	job.UpdatedAt = time.Now().Unix()
-	job.Status = string(StatusPending)
-
-	jsonData, err := job.ToJSON()
-	if err != nil {
-		q.logger.Error("Failed to marshal job", map[string]interface{}{
-			"job_id": job.ID,
-			"error":  err.Error(),
-		})
-		return err
-	}
-
-	// Push to queue
-	if err := q.client.RPush(ctx, DocumentScraperQueue, jsonData).Err(); err != nil {
-		q.logger.Error("Failed to enqueue job", map[string]interface{}{
-			"job_id": job.ID,
-			"error":  err.Error(),
-		})
-		return err
-	}
-
-	// Store job metadata with TTL
-	jobKey := fmt.Sprintf("%s%s", JobStatusPrefix, job.ID)
-	if err := q.client.Set(ctx, jobKey, string(StatusPending), 24*time.Hour).Err(); err != nil {
-		q.logger.Warn("Failed to set job status", map[string]interface{}{
-			"job_id": job.ID,
-			"error":  err.Error(),
-		})
-	}
-
-	q.logger.Info("Job enqueued successfully", map[string]interface{}{
-		"job_id":    job.ID,
-		"notice_id": job.NoticeID,
-	})
-
-	return nil
-}
-
-// Dequeue retrieves and locks the next job from the queue
-func (q *RedisQueue) Dequeue(ctx context.Context) (*DocumentScraperJob, error) {
-	// Use BLPOP to get next job with timeout
-	result, err := q.client.BLPop(ctx, 30*time.Second, DocumentScraperQueue).Result()
-	if err != nil {
-		if err == redis.Nil {
-			return nil, nil // Queue is empty
-		}
-		q.logger.Error("Failed to dequeue job", map[string]interface{}{
-			"error": err.Error(),
-		})
-		return nil, err
-	}
-
-	if len(result) < 2 {
-		return nil, fmt.Errorf("invalid dequeue result")
-	}
-
-	jobData := result[1]
-
-	// Parse job
-	var job DocumentScraperJob
-	if err := job.FromJSON(jobData); err != nil {
-		q.logger.Error("Failed to parse job", map[string]interface{}{
-			"error": err.Error(),
-			"data":  jobData,
-		})
-		return nil, err
-	}
-
-	// Mark as in progress with TTL
-	job.Status = string(StatusInProgress)
-	job.Attempts++
-	job.ProcessingStartedAt = time.Now().Unix()
-	job.UpdatedAt = time.Now().Unix()
-
-	jobKey := fmt.Sprintf("%s%s", JobStatusPrefix, job.ID)
-	if err := q.client.Set(ctx, jobKey, string(StatusInProgress), InProgressExpiration).Err(); err != nil {
-		q.logger.Warn("Failed to update job status", map[string]interface{}{
-			"job_id": job.ID,
-			"error":  err.Error(),
-		})
-	}
-
-	q.logger.Info("Job dequeued successfully", map[string]interface{}{
-		"job_id":    job.ID,
-		"attempt":   job.Attempts,
-		"max_retry": job.MaxRetries,
-	})
-
-	return &job, nil
-}
-
-// AckJob marks a job as completed
-func (q *RedisQueue) AckJob(ctx context.Context, jobID string) error {
-	now := time.Now().Unix()
-	jobKey := fmt.Sprintf("%s%s", JobStatusPrefix, jobID)
-
-	// Update status with TTL for record keeping
-	if err := q.client.Set(ctx, jobKey, string(StatusCompleted), 48*time.Hour).Err(); err != nil {
-		q.logger.Warn("Failed to ack job", map[string]interface{}{
-			"job_id": jobID,
-			"error":  err.Error(),
-		})
-	}
-
-	// Store completion timestamp
-	completedKey := fmt.Sprintf("%s:completed:%s", DocumentScraperQueue, jobID)
-	if err := q.client.Set(ctx, completedKey, now, 48*time.Hour).Err(); err != nil {
-		q.logger.Warn("Failed to store completion timestamp", map[string]interface{}{
-			"job_id": jobID,
-			"error":  err.Error(),
-		})
-	}
-
-	q.logger.Info("Job acknowledged", map[string]interface{}{
-		"job_id": jobID,
-	})
-
-	return nil
-}
-
-// NackJob marks a job as failed or for retry
-func (q *RedisQueue) NackJob(ctx context.Context, jobID string, jobErr error) error {
-	jobKey := fmt.Sprintf("%s%s", JobStatusPrefix, jobID)
-
-	// Get the job from Redis to check retry attempts
-	_, err := q.client.Get(ctx, jobKey).Result()
-	if err != nil && err != redis.Nil {
-		q.logger.Warn("Failed to get job for nack", map[string]interface{}{
-			"job_id": jobID,
-			"error":  err.Error(),
-		})
-	}
-
-	// Check if we should retry by trying to get the job back from processing
-	// For now, mark as failed
-	status := StatusFailed
-	duration := 48 * time.Hour
-
-	if err := q.client.Set(ctx, jobKey, string(status), duration).Err(); err != nil {
-		q.logger.Warn("Failed to update job status on nack", map[string]interface{}{
-			"job_id": jobID,
-			"error":  err.Error(),
-		})
-	}
-
-	// Store error message
-	errorKey := fmt.Sprintf("%s:error:%s", DocumentScraperQueue, jobID)
-	if err := q.client.Set(ctx, errorKey, jobErr.Error(), duration).Err(); err != nil {
-		q.logger.Warn("Failed to store error message", map[string]interface{}{
-			"job_id": jobID,
-			"error":  err.Error(),
-		})
-	}
-
-	// Optionally move to DLQ
-	if q.config.IsDLQEnabled {
-		jobData, _ := q.client.Get(ctx, JobStatusPrefix+jobID).Result()
-		if jobData != "" {
-			if err := q.client.RPush(ctx, DocumentScraperDLQ, jobData).Err(); err != nil {
-				q.logger.Warn("Failed to add job to DLQ", map[string]interface{}{
-					"job_id": jobID,
-					"error":  err.Error(),
-				})
-			}
-		}
-	}
-
-	q.logger.Info("Job nacked", map[string]interface{}{
-		"job_id": jobID,
-		"error":  jobErr.Error(),
-	})
-
-	return nil
-}
-
-// GetJobStatus retrieves the status of a job
-func (q *RedisQueue) GetJobStatus(ctx context.Context, jobID string) (JobStatus, error) {
-	jobKey := fmt.Sprintf("%s%s", JobStatusPrefix, jobID)
-	status, err := q.client.Get(ctx, jobKey).Result()
-	if err != nil {
-		if err == redis.Nil {
-			return StatusPending, nil
-		}
-		return StatusPending, err
-	}
-	return JobStatus(status), nil
-}
-
-// GetMetrics returns queue metrics
-func (q *RedisQueue) GetMetrics(ctx context.Context) (*QueueMetrics, error) {
-	metrics := &QueueMetrics{
-		UpdatedAt: time.Now().Unix(),
-	}
-
-	// Get queue size
-	size, err := q.GetQueueSize(ctx)
-	if err != nil {
-		q.logger.Warn("Failed to get queue size for metrics", map[string]interface{}{
-			"error": err.Error(),
-		})
-	}
-	metrics.PendingJobs = size
-
-	// Get DLQ size
-	dlqSize, err := q.GetDLQSize(ctx)
-	if err != nil {
-		q.logger.Warn("Failed to get DLQ size for metrics", map[string]interface{}{
-			"error": err.Error(),
-		})
-	}
-	metrics.FailedJobs = dlqSize
-
-	return metrics, nil
-}
-
-// GetQueueSize returns the number of items in the queue
-func (q *RedisQueue) GetQueueSize(ctx context.Context) (int64, error) {
-	size, err := q.client.LLen(ctx, DocumentScraperQueue).Result()
-	if err != nil {
-		return 0, err
-	}
-	return size, nil
-}
-
-// GetDLQSize returns the number of items in the dead letter queue
-func (q *RedisQueue) GetDLQSize(ctx context.Context) (int64, error) {
-	size, err := q.client.LLen(ctx, DocumentScraperDLQ).Result()
-	if err != nil {
-		return 0, err
-	}
-	return size, nil
-}
-
-// GetInProgressJobs returns jobs currently being processed
-func (q *RedisQueue) GetInProgressJobs(ctx context.Context) ([]DocumentScraperJob, error) {
-	// This is a simplified implementation
-	// In production, you'd track this separately
-	return nil, nil
-}
-
-// PurgeQueue removes all jobs from the queue
-func (q *RedisQueue) PurgeQueue(ctx context.Context) error {
-	if err := q.client.Del(ctx, DocumentScraperQueue).Err(); err != nil {
-		q.logger.Error("Failed to purge queue", map[string]interface{}{
-			"error": err.Error(),
-		})
-		return err
-	}
-	q.logger.Info("Queue purged successfully", map[string]interface{}{})
-	return nil
-}
-
-// PurgeDLQ removes all jobs from the dead letter queue
-func (q *RedisQueue) PurgeDLQ(ctx context.Context) error {
-	if err := q.client.Del(ctx, DocumentScraperDLQ).Err(); err != nil {
-		q.logger.Error("Failed to purge DLQ", map[string]interface{}{
-			"error": err.Error(),
-		})
-		return err
-	}
-	q.logger.Info("DLQ purged successfully", map[string]interface{}{})
-	return nil
-}
-
-// Close closes the Redis connection
-func (q *RedisQueue) Close() error {
-	return q.client.Close()
-}