# Auto-Match Automation Plan

**Version:** 1.0  
**Created:** November 2025  
**Status:** Planning Phase

---

## Executive Summary

This plan outlines the implementation of automated face matching in the Auto-Match feature. Currently, users must manually select faces on the right panel to match with the person on the left panel. This plan automates that process by automatically identifying faces that meet specific criteria:

1. **Reference face (left panel)** must be frontal (not portrait)
2. **Match faces (right panel)** must be frontal (not portrait)
3. **Match similarity** must be high (configurable threshold, default ≥70%)

When these conditions are met, faces are automatically identified to the person without user intervention.

---

## Current State Analysis

### Current Auto-Match Workflow

**Frontend Flow:**
1. User navigates to Auto-Match tab
2. Auto-match starts automatically (calls `POST /api/v1/faces/auto-match`)
3. Results display:
   - **Left Panel:** Identified person with reference face
   - **Right Panel:** Unidentified faces that match the reference face
4. User manually selects faces via checkboxes
5. User clicks "Save changes" button
6. Frontend calls `POST /api/v1/people/{person_id}/accept-matches` with selected face IDs

**Backend Flow:**
1. `POST /api/v1/faces/auto-match` endpoint:
   - Gets all identified people (one reference face per person)
   - For each person, finds similar unidentified faces using cosine similarity
   - Filters by tolerance (default 0.6, maps to confidence ≥40%)
   - Returns matches with similarity percentage (0-100%)

2. `POST /api/v1/people/{person_id}/accept-matches` endpoint:
   - Accepts list of face IDs
   - Sets `person_id` on each face
   - Creates `person_encodings` records
   - Commits changes

### Data Available

**Face Model Fields:**
- `id`: Face ID
- `person_id`: Person ID (NULL for unidentified)
- `photo_id`: Photo ID
- `encoding`: Face encoding (512-dimensional vector)
- `location`: Face bounding box (JSON)
- `confidence`: Detection confidence
- `quality_score`: Quality score (0.0-1.0)
- `pose_mode`: Pose classification (default: 'frontal')
- `yaw_angle`: Yaw angle in degrees (optional)
- `pitch_angle`: Pitch angle in degrees (optional)
- `roll_angle`: Roll angle in degrees (optional)

**AutoMatchFaceItem Schema:**
```python
{
    "id": int,
    "photo_id": int,
    "photo_filename": str,
    "location": str,
    "quality_score": float,
    "similarity": float,  # 0-100 percentage
    "distance": float
}
```

**AutoMatchPersonItem Schema:**
```python
{
    "person_id": int,
    "person_name": str,
    "reference_face_id": int,
    "reference_photo_id": int,
    "reference_photo_filename": str,
    "reference_location": str,
    "face_count": int,
    "matches": [AutoMatchFaceItem],
    "total_matches": int
}
```

### Limitations

1. **No pose_mode in API response:** Current API doesn't return `pose_mode` for faces
2. **No automatic acceptance:** All matches require manual selection
3. **No filtering by pose:** Cannot filter out profile/portrait faces
4. **No high-confidence threshold:** No automatic filtering by match confidence

---

## Requirements

### Functional Requirements

1. **Automated Face Matching:**
   - When "Start Auto-Match" button is clicked, automatically identify faces that meet criteria
   - Only process persons whose reference face is frontal (`pose_mode == 'frontal'`)
   - Only match with unidentified faces that are frontal (`pose_mode == 'frontal'`)
   - Only match faces with similarity ≥70% (configurable threshold)
   - Automatically call accept-matches API for qualifying faces

2. **Pose-Based Filtering:**
   - Check reference face `pose_mode` field
   - Check match face `pose_mode` field
   - Skip matching if reference face is not frontal
   - Skip matching if match face is not frontal

3. **Similarity Threshold:**
   - Configurable threshold (default: 70%)
   - Only auto-match faces with similarity ≥ threshold
   - Threshold should be adjustable via UI (optional for Phase 1)

4. **User Feedback:**
   - Show progress/status during auto-matching
   - Display summary: how many faces were auto-matched vs. skipped
   - Show which persons were processed and which were skipped (due to non-frontal reference)

5. **Backward Compatibility:**
   - Keep existing manual selection workflow
   - Auto-match should be opt-in (button click)
   - Manual selection should still work after auto-match

### Technical Requirements

1. **API Enhancements:**
   - Add `pose_mode` to `AutoMatchFaceItem` response
   - Add `pose_mode` to `AutoMatchPersonItem` reference face info
   - Add optional `auto_accept` parameter to auto-match endpoint
   - Add `auto_accept_threshold` parameter (default: 70.0)

2. **Backend Logic:**
   - Filter reference faces by `pose_mode == 'frontal'` in auto-match query
   - Filter match faces by `pose_mode == 'frontal'` in similarity search
   - Filter matches by similarity threshold
   - Automatically call `accept_auto_match_matches` for qualifying faces

3. **Frontend Changes:**
   - Update "Start Auto-Match" button to trigger auto-acceptance
   - Add loading state during auto-matching
   - Display results summary (auto-matched count, skipped count)
   - Update UI to show pose_mode for faces (optional)

---

## Technical Approach

### Architecture Overview

```
┌─────────────────────────────────────────────────────────┐
│              Auto-Match Automation Flow                   │
├─────────────────────────────────────────────────────────┤
│                                                           │
│  1. User clicks "Start Auto-Match" button                │
│     └─> Frontend: POST /api/v1/faces/auto-match         │
│         with auto_accept=true, threshold=70              │
│                                                           │
│  2. Backend: Find all identified people                  │
│     └─> Filter: Only persons with frontal reference face │
│         (pose_mode == 'frontal')                         │
│                                                           │
│  3. For each person (with frontal reference):            │
│     └─> Find similar unidentified faces                  │
│         └─> Filter: Only frontal faces                  │
│             (pose_mode == 'frontal')                      │
│         └─> Filter: Similarity ≥ threshold (70%)        │
│         └─> Auto-accept: Call accept_auto_match_matches │
│                                                           │
│  4. Return summary:                                      │
│     └─> Total persons processed                          │
│     └─> Total faces auto-matched                         │
│     └─> Persons skipped (non-frontal reference)          │
│     └─> Matches found but not auto-matched               │
│                                                           │
└─────────────────────────────────────────────────────────┘
```

### Phase 1: Backend API Enhancements

#### Step 1.1: Update Auto-Match Request Schema

**File: `src/web/schemas/faces.py`**

Add optional parameters to `AutoMatchRequest`:
```python
class AutoMatchRequest(BaseModel):
    tolerance: float
    auto_accept: bool = False  # New: Enable auto-acceptance
    auto_accept_threshold: float = 70.0  # New: Similarity threshold (0-100)
```

#### Step 1.2: Update Auto-Match Response Schema

**File: `src/web/schemas/faces.py`**

Add `pose_mode` to `AutoMatchFaceItem`:
```python
class AutoMatchFaceItem(BaseModel):
    id: int
    photo_id: int
    photo_filename: str
    location: str
    quality_score: float
    similarity: float  # Confidence percentage (0-100)
    distance: float
    pose_mode: str = "frontal"  # New: Pose classification
```

Add `pose_mode` to `AutoMatchPersonItem` reference face:
```python
class AutoMatchPersonItem(BaseModel):
    person_id: int
    person_name: str
    reference_face_id: int
    reference_photo_id: int
    reference_photo_filename: str
    reference_location: str
    reference_pose_mode: str = "frontal"  # New: Reference face pose
    face_count: int
    matches: list[AutoMatchFaceItem]
    total_matches: int
```

Add summary fields to `AutoMatchResponse`:
```python
class AutoMatchResponse(BaseModel):
    people: list[AutoMatchPersonItem]
    total_people: int
    total_matches: int
    # New: Auto-accept summary
    auto_accepted: bool = False
    auto_accepted_faces: int = 0
    skipped_persons: int = 0  # Persons with non-frontal reference
    skipped_matches: int = 0  # Matches that didn't meet criteria
```

#### Step 1.3: Update Auto-Match Service Function

**File: `src/web/services/face_service.py`**

Modify `find_auto_match_matches` to:
1. Filter reference faces by `pose_mode == 'frontal'`
2. Include `pose_mode` in results

```python
def find_auto_match_matches(
    db: Session,
    tolerance: float = None,
    filter_frontal_only: bool = True,  # New parameter
) -> List[Tuple[int, int, Face, List[Tuple[Face, float, float]]]]:
    """Find auto-match matches for all identified people.
    
    Args:
        tolerance: Similarity tolerance (default: 0.6)
        filter_frontal_only: Only include persons with frontal reference face
    """
    # ... existing code ...
    
    # Filter reference faces by pose_mode if requested
    if filter_frontal_only:
        identified_faces = [
            f for f in identified_faces 
            if f.pose_mode == 'frontal'
        ]
    
    # ... rest of existing code ...
```

#### Step 1.4: Update Auto-Match API Endpoint

**File: `src/web/api/faces.py`**

Modify `auto_match_faces` endpoint to:
1. Accept `auto_accept` and `auto_accept_threshold` parameters
2. Filter matches by pose_mode and similarity threshold
3. Auto-accept qualifying faces
4. Return summary statistics

```python
@router.post("/auto-match", response_model=AutoMatchResponse)
def auto_match_faces(
    request: AutoMatchRequest,
    db: Session = Depends(get_db),
) -> AutoMatchResponse:
    """Start auto-match process with optional auto-acceptance.
    
    If auto_accept=True:
    - Only processes persons with frontal reference faces
    - Only matches with frontal unidentified faces
    - Only auto-accepts matches with similarity ≥ threshold
    """
    from src.web.db.models import Person, Photo, Face
    
    # Find matches for all identified people
    # Filter by frontal reference faces if auto_accept enabled
    matches_data = find_auto_match_matches(
        db, 
        tolerance=request.tolerance,
        filter_frontal_only=request.auto_accept
    )
    
    # Track statistics
    auto_accepted_faces = 0
    skipped_persons = 0
    skipped_matches = 0
    
    # If auto_accept enabled, process matches automatically
    if request.auto_accept:
        for person_id, reference_face_id, reference_face, similar_faces in matches_data:
            # Filter matches by criteria:
            # 1. Match face must be frontal
            # 2. Similarity must be ≥ threshold
            
            qualifying_faces = []
            for face, distance, confidence_pct in similar_faces:
                # Check pose_mode
                if face.pose_mode != 'frontal':
                    skipped_matches += 1
                    continue
                
                # Check similarity threshold
                if confidence_pct < request.auto_accept_threshold:
                    skipped_matches += 1
                    continue
                
                qualifying_faces.append(face.id)
            
            # Auto-accept qualifying faces
            if qualifying_faces:
                try:
                    identified_count, updated_count = accept_auto_match_matches(
                        db, person_id, qualifying_faces
                    )
                    auto_accepted_faces += identified_count
                except Exception as e:
                    print(f"Error auto-accepting matches for person {person_id}: {e}")
    
    # Build response (existing logic)
    people_items = []
    total_matches = 0
    
    for person_id, reference_face_id, reference_face, similar_faces in matches_data:
        # ... existing person building logic ...
        
        # Add pose_mode to reference face info
        reference_pose_mode = reference_face.pose_mode or 'frontal'
        
        # Build matches list with pose_mode
        match_items = []
        for face, distance, confidence_pct in similar_faces:
            match_photo = db.query(Photo).filter(Photo.id == face.photo_id).first()
            if not match_photo:
                continue
            
            match_items.append(
                AutoMatchFaceItem(
                    id=face.id,
                    photo_id=face.photo_id,
                    photo_filename=match_photo.filename,
                    location=face.location,
                    quality_score=float(face.quality_score),
                    similarity=confidence_pct,
                    distance=distance,
                    pose_mode=face.pose_mode or 'frontal',  # New
                )
            )
        
        if match_items:
            people_items.append(
                AutoMatchPersonItem(
                    person_id=person_id,
                    person_name=person_name,
                    reference_face_id=reference_face_id,
                    reference_photo_id=reference_face.photo_id,
                    reference_photo_filename=reference_photo.filename,
                    reference_location=reference_face.location,
                    reference_pose_mode=reference_pose_mode,  # New
                    face_count=face_count,
                    matches=match_items,
                    total_matches=len(match_items),
                )
            )
            total_matches += len(match_items)
    
    return AutoMatchResponse(
        people=people_items,
        total_people=len(people_items),
        total_matches=total_matches,
        auto_accepted=request.auto_accept,
        auto_accepted_faces=auto_accepted_faces,
        skipped_persons=skipped_persons,
        skipped_matches=skipped_matches,
    )
```

#### Step 1.5: Update Similar Face Search to Include Pose Mode

**File: `src/web/services/face_service.py`**

Ensure `find_similar_faces` returns faces with pose_mode information:
```python
def find_similar_faces(
    db: Session,
    reference_face_id: int,
    limit: int = 100,
    tolerance: float = None,
    filter_frontal_only: bool = False,  # New parameter
) -> List[Tuple[Face, float, float]]:
    """Find similar faces to reference face.
    
    Args:
        filter_frontal_only: Only return frontal faces
    """
    # ... existing similarity search ...
    
    # Filter by pose_mode if requested
    if filter_frontal_only:
        similar_faces = [
            (face, distance, confidence_pct)
            for face, distance, confidence_pct in similar_faces
            if face.pose_mode == 'frontal'
        ]
    
    return similar_faces
```

### Phase 2: Frontend Updates

#### Step 2.1: Update AutoMatch API Interface

**File: `frontend/src/api/faces.ts`**

Update `AutoMatchRequest` interface:
```typescript
export interface AutoMatchRequest {
  tolerance: number
  auto_accept?: boolean  // New: Enable auto-acceptance
  auto_accept_threshold?: number  // New: Similarity threshold (0-100)
}
```

Update `AutoMatchFaceItem` interface:
```typescript
export interface AutoMatchFaceItem {
  id: number
  photo_id: number
  photo_filename: string
  location: string
  quality_score: number
  similarity: number  // Confidence percentage (0-100)
  distance: number
  pose_mode?: string  // New: Pose classification
}
```

Update `AutoMatchPersonItem` interface:
```typescript
export interface AutoMatchPersonItem {
  person_id: number
  person_name: string
  reference_face_id: number
  reference_photo_id: number
  reference_photo_filename: string
  reference_location: string
  reference_pose_mode?: string  // New: Reference face pose
  face_count: number
  matches: AutoMatchFaceItem[]
  total_matches: number
}
```

Update `AutoMatchResponse` interface:
```typescript
export interface AutoMatchResponse {
  people: AutoMatchPersonItem[]
  total_people: number
  total_matches: number
  auto_accepted?: boolean  // New
  auto_accepted_faces?: number  // New
  skipped_persons?: number  // New
  skipped_matches?: number  // New
}
```

#### Step 2.2: Update AutoMatch Component

**File: `frontend/src/pages/AutoMatch.tsx`**

1. Add state for auto-accept threshold:
```typescript
const [autoAcceptThreshold, setAutoAcceptThreshold] = useState(70)
```

2. Update `startAutoMatch` function to use auto-accept:
```typescript
const startAutoMatch = async () => {
  if (tolerance < 0 || tolerance > 1) {
    alert('Please enter a valid tolerance value between 0.0 and 1.0.')
    return
  }

  setBusy(true)
  try {
    const response = await facesApi.autoMatch({ 
      tolerance,
      auto_accept: true,  // Enable auto-acceptance
      auto_accept_threshold: autoAcceptThreshold
    })
    
    // Show summary
    if (response.auto_accepted) {
      const summary = [
        `✅ Auto-matched ${response.auto_accepted_faces || 0} faces`,
        response.skipped_persons ? `⚠️ Skipped ${response.skipped_persons} persons (non-frontal reference)` : '',
        response.skipped_matches ? `ℹ️ Skipped ${response.skipped_matches} matches (didn't meet criteria)` : ''
      ].filter(Boolean).join('\n')
      
      alert(summary)
    }
    
    if (response.people.length === 0) {
      alert('🔍 No similar faces found for auto-identification')
      setBusy(false)
      return
    }

    setPeople(response.people)
    // ... rest of existing code ...
  } catch (error) {
    console.error('Auto-match failed:', error)
    alert('Failed to start auto-match. Please try again.')
  } finally {
    setBusy(false)
  }
}
```

3. Add threshold input field (optional):
```tsx
<div className="flex items-center gap-2">
  <label className="text-sm font-medium text-gray-700">Auto-Accept Threshold:</label>
  <input
    type="number"
    min="0"
    max="100"
    step="5"
    value={autoAcceptThreshold}
    onChange={(e) => setAutoAcceptThreshold(parseInt(e.target.value) || 70)}
    disabled={isActive}
    className="w-20 px-2 py-1 border border-gray-300 rounded text-sm"
  />
  <span className="text-xs text-gray-500">% (min similarity)</span>
</div>
```

4. Update UI to show pose_mode for faces (optional visual indicator):
```tsx
{/* Show pose mode badge if not frontal */}
{match.pose_mode && match.pose_mode !== 'frontal' && (
  <span className="px-2 py-1 rounded text-xs bg-orange-100 text-orange-800">
    {match.pose_mode}
  </span>
)}
```

### Phase 3: Testing & Validation

#### Step 3.1: Unit Tests

**File: `tests/test_auto_match_automation.py`**

1. Test auto-match with auto_accept enabled
2. Test filtering by frontal faces only
3. Test similarity threshold filtering
4. Test skipping non-frontal reference faces
5. Test summary statistics

#### Step 3.2: Integration Tests

1. Test full auto-match flow with auto-acceptance
2. Test with mixed frontal/non-frontal faces
3. Test with various similarity thresholds
4. Verify database updates (person_id, person_encodings)

#### Step 3.3: Manual Testing

1. Create test data with frontal and non-frontal faces
2. Run auto-match with auto_accept enabled
3. Verify only frontal faces are matched
4. Verify only high-similarity faces are auto-accepted
5. Verify summary statistics are correct

---

## Implementation Plan

### Phase 1: Backend (2-3 days)

1. ✅ Update schemas (AutoMatchRequest, AutoMatchResponse, etc.)
2. ✅ Update `find_auto_match_matches` to filter by pose_mode
3. ✅ Update auto-match API endpoint with auto-accept logic
4. ✅ Update `find_similar_faces` to filter by pose_mode
5. ✅ Add unit tests for filtering logic

### Phase 2: Frontend (1-2 days)

1. ✅ Update API interfaces
2. ✅ Update AutoMatch component to use auto-accept
3. ✅ Add threshold input field (optional)
4. ✅ Add summary display
5. ✅ Add pose_mode indicators (optional)

### Phase 3: Testing (1-2 days)

1. ✅ Write unit tests
2. ✅ Write integration tests
3. ✅ Manual testing with real data
4. ✅ Performance testing

**Total Estimated Time: 4-7 days**

---

## Configuration

### Default Values

- **Auto-accept threshold:** 70% similarity
- **Pose mode filter:** Only frontal faces (`pose_mode == 'frontal'`)
- **Tolerance:** 0.6 (existing default)

### Future Enhancements (Not in Phase 1)

1. **Configurable pose mode filter:** Allow users to specify which pose modes to include
2. **Per-person threshold:** Different thresholds for different people
3. **Batch processing:** Process multiple persons in parallel
4. **Progress tracking:** Real-time progress updates during auto-matching
5. **Undo functionality:** Ability to undo auto-accepted matches
6. **Logging/Audit trail:** Track which faces were auto-matched and when

---

## Edge Cases & Error Handling

1. **No frontal reference faces:** Skip person, show in summary
2. **No frontal match faces:** Skip matches, show in summary
3. **Database errors:** Rollback transaction, show error message
4. **Missing pose_mode data:** Default to 'frontal' (backward compatibility)
5. **Concurrent updates:** Use database transactions to prevent conflicts

---

## Success Criteria

1. ✅ Auto-match automatically identifies faces meeting criteria
2. ✅ Only frontal faces (reference and matches) are processed
3. ✅ Only high-similarity matches (≥70%) are auto-accepted
4. ✅ Summary statistics are accurate
5. ✅ Manual selection still works after auto-match
6. ✅ No breaking changes to existing API

---

## Notes

- This plan assumes portrait detection is already implemented (which it is)
- The `pose_mode` field should already exist in the database
- Backward compatibility is maintained: existing API calls without `auto_accept` work as before
- The "Start Auto-Match" button will trigger auto-acceptance when clicked
- Users can still manually select/deselect faces after auto-matching

