# Analysis: Identifying People in Videos

**Date:** December 2024  
**Status:** Analysis Only (No Implementation)  
**Feature:** Direct person identification in videos without face detection

---

## Executive Summary

This document analyzes how to implement the ability to identify people directly in videos within the "Identify People" tab. Unlike photos where people are identified through detected faces, videos will allow direct person-to-video associations without requiring face detection or frame extraction.

**Key Requirements:**
- List videos with filtering capabilities
- Each video can have multiple people identified
- Can add more people to a video even after some are already identified
- Located in "Identify People" tab under "Identify People in Videos" sub-tab

---

## Current System Architecture

### Database Schema

**Current Relationships:**
- `Photo` (includes videos via `media_type='video'`) → `Face` → `Person`
- People are linked to photos **only** through faces
- No direct Photo-Person relationship exists

**Relevant Models:**
```python
class Photo:
    id: int
    path: str
    media_type: str  # "image" or "video"
    # ... other fields
    
class Face:
    id: int
    photo_id: int  # FK to Photo
    person_id: int  # FK to Person (nullable)
    # ... encoding, location, etc.
    
class Person:
    id: int
    first_name: str
    last_name: str
    # ... other fields
```

**Current State:**
- Videos are stored in `photos` table with `media_type='video'`
- Videos are marked as `processed=True` but face processing is skipped
- No faces exist for videos currently
- People cannot be linked to videos through the existing Face model

### Frontend Structure

**Identify Page (`frontend/src/pages/Identify.tsx`):**
- Has two tabs: "Identify Faces" and "Identify People in Videos"
- Videos tab currently shows placeholder: "This functionality will be available in a future update"
- Faces tab has full functionality for identifying people through faces

### API Endpoints

**Existing Photo/Video Endpoints:**
- `GET /api/v1/photos` - Search photos/videos with `media_type` filter
- Supports filtering by `media_type='video'` to get videos
- Returns `PhotoSearchResult` with video metadata

**No Existing Endpoints For:**
- Listing videos specifically for person identification
- Getting people associated with a video
- Identifying people in videos
- Managing video-person relationships

---

## Proposed Solution

### 1. Database Schema Changes

#### Option A: New PhotoPersonLinkage Table (Recommended)

Create a new junction table to link people directly to photos/videos:

```python
class PhotoPersonLinkage(Base):
    """Direct linkage between Photo/Video and Person (without faces)."""
    
    __tablename__ = "photo_person_linkage"
    
    id = Column(Integer, primary_key=True, autoincrement=True)
    photo_id = Column(Integer, ForeignKey("photos.id"), nullable=False, index=True)
    person_id = Column(Integer, ForeignKey("people.id"), nullable=False, index=True)
    identified_by_user_id = Column(Integer, ForeignKey("users.id"), nullable=True, index=True)
    created_date = Column(DateTime, default=datetime.utcnow, nullable=False)
    
    photo = relationship("Photo", back_populates="direct_people")
    person = relationship("Person", back_populates="direct_photos")
    
    __table_args__ = (
        UniqueConstraint("photo_id", "person_id", name="uq_photo_person"),
        Index("idx_photo_person_photo", "photo_id"),
        Index("idx_photo_person_person", "person_id"),
    )
```

**Update Photo Model:**
```python
class Photo(Base):
    # ... existing fields ...
    direct_people = relationship("PhotoPersonLinkage", back_populates="photo", cascade="all, delete-orphan")
```

**Update Person Model:**
```python
class Person(Base):
    # ... existing fields ...
    direct_photos = relationship("PhotoPersonLinkage", back_populates="person", cascade="all, delete-orphan")
```

**Pros:**
- ✅ Clean separation: Face-based identification vs. direct identification
- ✅ Supports both photos and videos (unified approach)
- ✅ Can track who identified the person (`identified_by_user_id`)
- ✅ Prevents duplicate person-video associations
- ✅ Similar pattern to `PhotoTagLinkage` (consistent architecture)

**Cons:**
- ⚠️ Requires database migration
- ⚠️ Need to update queries that get "people in photo" to check both Face and PhotoPersonLinkage

#### Option B: Use Face Model with Dummy Faces (Not Recommended)

Create "virtual" faces for videos without encodings.

**Cons:**
- ❌ Misleading data model (faces without actual face data)
- ❌ Breaks assumptions about Face model (encoding required)
- ❌ Confusing for queries and logic
- ❌ Not semantically correct

**Recommendation:** Option A (PhotoPersonLinkage table)

### 2. API Endpoints

#### 2.1 List Videos for Identification

**Endpoint:** `GET /api/v1/videos`

**Query Parameters:**
- `page`: int (default: 1)
- `page_size`: int (default: 50, max: 200)
- `folder_path`: Optional[str] - Filter by folder
- `date_from`: Optional[str] - Filter by date taken (from)
- `date_to`: Optional[str] - Filter by date taken (to)
- `has_people`: Optional[bool] - Filter videos with/without identified people
- `person_name`: Optional[str] - Filter videos containing specific person
- `sort_by`: str (default: "filename") - "filename", "date_taken", "date_added"
- `sort_dir`: str (default: "asc") - "asc" or "desc"

**Response:**
```python
class VideoListItem(BaseModel):
    id: int
    filename: str
    path: str
    date_taken: Optional[date]
    date_added: date
    identified_people: List[PersonInfo]  # People identified in this video
    identified_people_count: int

class ListVideosResponse(BaseModel):
    items: List[VideoListItem]
    page: int
    page_size: int
    total: int
```

#### 2.2 Get People in a Video

**Endpoint:** `GET /api/v1/videos/{video_id}/people`

**Response:**
```python
class VideoPersonInfo(BaseModel):
    person_id: int
    first_name: str
    last_name: str
    middle_name: Optional[str]
    maiden_name: Optional[str]
    date_of_birth: Optional[date]
    identified_by: Optional[str]  # Username
    identified_date: datetime

class VideoPeopleResponse(BaseModel):
    video_id: int
    people: List[VideoPersonInfo]
```

#### 2.3 Identify People in Video

**Endpoint:** `POST /api/v1/videos/{video_id}/identify`

**Request:**
```python
class IdentifyVideoRequest(BaseModel):
    person_id: Optional[int] = None  # Use existing person
    first_name: Optional[str] = None  # Create new person
    last_name: Optional[str] = None
    middle_name: Optional[str] = None
    maiden_name: Optional[str] = None
    date_of_birth: Optional[date] = None
```

**Response:**
```python
class IdentifyVideoResponse(BaseModel):
    video_id: int
    person_id: int
    created_person: bool
    message: str
```

**Behavior:**
- If `person_id` provided: Link existing person to video
- If person info provided: Create new person and link to video
- If person already linked: Return success (idempotent)
- Track `identified_by_user_id` for audit

#### 2.4 Remove Person from Video

**Endpoint:** `DELETE /api/v1/videos/{video_id}/people/{person_id}`

**Response:**
```python
class RemoveVideoPersonResponse(BaseModel):
    video_id: int
    person_id: int
    removed: bool
    message: str
```

### 3. Service Layer Functions

#### 3.1 Video Service Functions

**File:** `src/web/services/video_service.py` (new file)

```python
def list_videos_for_identification(
    db: Session,
    folder_path: Optional[str] = None,
    date_from: Optional[date] = None,
    date_to: Optional[date] = None,
    has_people: Optional[bool] = None,
    person_name: Optional[str] = None,
    sort_by: str = "filename",
    sort_dir: str = "asc",
    page: int = 1,
    page_size: int = 50,
) -> Tuple[List[Photo], int]:
    """List videos for person identification."""
    # Query videos (media_type='video')
    # Apply filters
    # Join with PhotoPersonLinkage to get people count
    # Return paginated results

def get_video_people(
    db: Session,
    video_id: int,
) -> List[Tuple[Person, PhotoPersonLinkage]]:
    """Get all people identified in a video."""
    # Query PhotoPersonLinkage for video_id
    # Join with Person
    # Return list with identification metadata

def identify_person_in_video(
    db: Session,
    video_id: int,
    person_id: Optional[int] = None,
    person_data: Optional[dict] = None,
    user_id: Optional[int] = None,
) -> Tuple[Person, bool]:
    """Identify a person in a video.
    
    Returns:
        (Person, created_person: bool)
    """
    # Validate video exists and is actually a video
    # Get or create person
    # Create PhotoPersonLinkage if doesn't exist
    # Return person and created flag

def remove_person_from_video(
    db: Session,
    video_id: int,
    person_id: int,
) -> bool:
    """Remove person identification from video."""
    # Delete PhotoPersonLinkage
    # Return success
```

#### 3.2 Update Existing Search Functions

**File:** `src/web/services/search_service.py`

Update `get_photo_person()` to check both:
1. Face-based identification (existing)
2. Direct PhotoPersonLinkage (new)

```python
def get_photo_people(db: Session, photo_id: int) -> List[Person]:
    """Get all people in a photo/video (both face-based and direct)."""
    people = []
    
    # Get people through faces
    face_people = (
        db.query(Person)
        .join(Face, Person.id == Face.person_id)
        .filter(Face.photo_id == photo_id)
        .distinct()
        .all()
    )
    people.extend(face_people)
    
    # Get people through direct linkage
    direct_people = (
        db.query(Person)
        .join(PhotoPersonLinkage, Person.id == PhotoPersonLinkage.person_id)
        .filter(PhotoPersonLinkage.photo_id == photo_id)
        .distinct()
        .all()
    )
    people.extend(direct_people)
    
    # Remove duplicates
    seen_ids = set()
    unique_people = []
    for person in people:
        if person.id not in seen_ids:
            seen_ids.add(person.id)
            unique_people.append(person)
    
    return unique_people
```

### 4. Frontend Implementation

#### 4.1 Video List Component

**Location:** `frontend/src/pages/Identify.tsx` (videos tab)

**Features:**
- Video grid/list view with thumbnails
- Filter panel:
  - Folder path
  - Date range (date taken)
  - Has people / No people
  - Person name search
- Sort options: filename, date taken, date added
- Pagination
- Each video shows:
  - Thumbnail (video poster/first frame)
  - Filename
  - Date taken
  - List of identified people (badges/chips)
  - "Identify People" button

#### 4.2 Video Detail / Identification Panel

**When video is selected:**

**Left Panel:**
- Video player (HTML5 `<video>` element)
- Video metadata (filename, path, date taken, etc.)

**Right Panel:**
- List of currently identified people
  - Person name
  - Remove button
- "Add Person" section:
  - Search existing people (autocomplete)
  - Or create new person form:
    - First name, last name (required)
    - Middle name, maiden name (optional)
    - Date of birth (optional)
  - "Add" button

**UI Flow:**
1. User selects video from list
2. Video loads in player
3. Right panel shows identified people
4. User can add more people or remove existing ones
5. Changes saved immediately (or with "Save" button)

#### 4.3 API Client Functions

**File:** `frontend/src/api/videos.ts` (new file)

```typescript
export interface VideoListItem {
  id: number
  filename: string
  path: string
  date_taken: string | null
  date_added: string
  identified_people: PersonInfo[]
  identified_people_count: number
}

export interface ListVideosResponse {
  items: VideoListItem[]
  page: number
  page_size: number
  total: number
}

export interface IdentifyVideoRequest {
  person_id?: number
  first_name?: string
  last_name?: string
  middle_name?: string
  maiden_name?: string
  date_of_birth?: string
}

export const videosApi = {
  listVideos: async (params: {
    page?: number
    page_size?: number
    folder_path?: string
    date_from?: string
    date_to?: string
    has_people?: boolean
    person_name?: string
    sort_by?: string
    sort_dir?: string
  }): Promise<ListVideosResponse> => {
    // Implementation
  },
  
  getVideoPeople: async (videoId: number): Promise<VideoPeopleResponse> => {
    // Implementation
  },
  
  identifyPerson: async (
    videoId: number,
    request: IdentifyVideoRequest
  ): Promise<IdentifyVideoResponse> => {
    // Implementation
  },
  
  removePerson: async (
    videoId: number,
    personId: number
  ): Promise<RemoveVideoPersonResponse> => {
    // Implementation
  },
}
```

### 5. Database Migration

**Migration Script:** Add `photo_person_linkage` table

```python
def add_photo_person_linkage_table(db: Session):
    """Add photo_person_linkage table for direct person-video associations."""
    db.execute(text("""
        CREATE TABLE IF NOT EXISTS photo_person_linkage (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            photo_id INTEGER NOT NULL,
            person_id INTEGER NOT NULL,
            identified_by_user_id INTEGER,
            created_date DATETIME DEFAULT CURRENT_TIMESTAMP,
            FOREIGN KEY (photo_id) REFERENCES photos (id) ON DELETE CASCADE,
            FOREIGN KEY (person_id) REFERENCES people (id) ON DELETE CASCADE,
            FOREIGN KEY (identified_by_user_id) REFERENCES users (id),
            UNIQUE(photo_id, person_id)
        )
    """))
    
    db.execute(text("""
        CREATE INDEX IF NOT EXISTS idx_photo_person_photo 
        ON photo_person_linkage(photo_id)
    """))
    
    db.execute(text("""
        CREATE INDEX IF NOT EXISTS idx_photo_person_person 
        ON photo_person_linkage(person_id)
    """))
    
    db.execute(text("""
        CREATE INDEX IF NOT EXISTS idx_photo_person_user 
        ON photo_person_linkage(identified_by_user_id)
    """))
    
    db.commit()
```

---

## Implementation Considerations

### 1. Data Consistency

**Question:** Should we allow both face-based and direct identification for the same person-video pair?

**Recommendation:** Yes, allow both. They serve different purposes:
- Face-based: Person identified through detected faces in video frames
- Direct: Person identified manually without face detection

**Implementation:** Check for duplicates when querying, but allow both to exist.

### 2. Search Integration

**Update search functions to include direct linkages:**
- `search_photos_by_name()` - Should find videos with direct person linkages
- `get_photo_person()` - Should check both Face and PhotoPersonLinkage
- Photo viewer - Should show people from both sources

### 3. Video Thumbnails

**Current State:** Videos don't have thumbnails generated yet.

**Options:**
1. Generate thumbnail on-demand when needed
2. Generate thumbnail during video import
3. Use first frame as thumbnail (extract on-demand)

**Recommendation:** Extract first frame on-demand for now (simpler, no storage overhead). Can optimize later with caching.

### 4. Performance

**Considerations:**
- Video list queries with people counts (JOIN with PhotoPersonLinkage)
- Pagination for large video libraries
- Video thumbnail generation (can be slow for large videos)

**Optimizations:**
- Index on `photo_id` and `person_id` in PhotoPersonLinkage
- Consider caching video thumbnails
- Use efficient queries (avoid N+1 problems)

### 5. User Experience

**Key UX Points:**
- Clear indication of which people are identified (badges/chips)
- Easy to add multiple people quickly
- Search/filter to find specific videos
- Visual feedback when person is added/removed
- Handle edge cases (person already added, etc.)

---

## Implementation Phases

### Phase 1: Database & Models
1. Create PhotoPersonLinkage model
2. Add migration script
3. Update Photo and Person models with relationships
4. Test database changes

### Phase 2: Backend API
1. Create video service functions
2. Create API endpoints for videos
3. Update search functions to include direct linkages
4. Add API tests

### Phase 3: Frontend API Client
1. Create `videos.ts` API client
2. Add TypeScript interfaces
3. Test API integration

### Phase 4: Frontend UI
1. Implement video list component
2. Implement video detail/identification panel
3. Add filters and sorting
4. Add video player integration
5. Test user flows

### Phase 5: Integration & Polish
1. Update search to include direct linkages
2. Update photo viewer to show direct linkages
3. Add error handling
4. Performance optimization
5. Documentation

---

## Open Questions

1. **Should we prevent duplicate person-video associations?**
   - **Answer:** Yes, use UNIQUE constraint on (photo_id, person_id)

2. **Should direct linkages work for photos too, or only videos?**
   - **Answer:** Could work for both, but focus on videos first. Photos already have face-based identification.

3. **How to handle video thumbnails?**
   - **Answer:** Extract first frame on-demand initially, cache later if needed

4. **Should we show both face-based and direct identifications together?**
   - **Answer:** Yes, merge them in queries and display

5. **What happens if a video is deleted?**
   - **Answer:** CASCADE delete removes PhotoPersonLinkage records (handled by FK constraint)

---

## Summary

**Key Design Decisions:**
1. ✅ New `PhotoPersonLinkage` table for direct person-video associations
2. ✅ Separate from face-based identification (both can coexist)
3. ✅ Unified API endpoints under `/api/v1/videos`
4. ✅ Frontend in existing Identify page videos tab
5. ✅ Support filtering, sorting, and pagination
6. ✅ Allow adding multiple people to same video
7. ✅ Track who identified each person (audit trail)

**Next Steps:**
1. Review and approve this analysis
2. Create database migration
3. Implement backend API
4. Implement frontend UI
5. Test and iterate

---

**Document Version:** 1.0  
**Last Updated:** December 2024  
**Author:** AI Assistant (Cursor)







