# Analysis: Adding Video Support to PunimTag

**Date:** December 2024  
**Status:** Partial Implementation (Scanning Only)  
**Current Phase:** Video scanning implemented, face processing skipped

---

## Executive Summary

This document provides a comprehensive analysis of adding video support to PunimTag, a photo management application with facial recognition capabilities. The analysis covers technical feasibility, storage implications, performance considerations, database schema changes, and implementation strategy.

**Current Implementation Status:**
- ✅ Video scanning and import implemented
- ✅ Videos are detected and imported into database
- ✅ Videos are automatically marked as processed (skipped for face detection)
- ⏸️ Video frame extraction (not implemented)
- ⏸️ Face detection on video frames (not implemented)
- ⏸️ Video playback in frontend (not implemented)

---

## Current System State

### Supported Formats

**Images:**
- `.jpg`, `.jpeg`, `.png`, `.bmp`, `.tiff`, `.tif`

**Videos (New):**
- `.mp4`, `.mov`, `.avi`, `.mkv`, `.webm`, `.m4v`, `.flv`, `.wmv`, `.mpg`, `.mpeg`

### Current Architecture

- **Database:** SQLite/PostgreSQL with `Photo` model
- **Face Processing:** DeepFace (RetinaFace detector + ArcFace model)
- **Storage:** Files stored by path reference (not in database)
- **Frontend:** React with image display using `<img>` tags
- **Processing:** Background jobs using RQ (Redis Queue)

---

## Technical Feasibility

### ✅ Feasible with Moderate Changes

DeepFace can process video frames, but requires:
1. **Video frame extraction** (OpenCV/ffmpeg)
2. **Frame sampling strategy** (time-based or scene-change detection)
3. **Database schema updates** (add `media_type` field)
4. **Frontend video player component** (HTML5 `<video>` element)
5. **Processing pipeline modifications** (handle video frames)

### DeepFace Video Capabilities

DeepFace processes individual frames (not full videos):
- Must extract frames first using OpenCV or ffmpeg
- Each frame is processed like a static image
- No built-in video file support

**Implementation Approach:**
```python
# Pseudo-code for video processing
import cv2
from deepface import DeepFace

cap = cv2.VideoCapture(video_path)
frame_number = 0
fps = cap.get(cv2.CAP_PROP_FPS)

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    
    # Process every Nth frame (e.g., every 1 second)
    if frame_number % int(fps) == 0:
        # Save frame temporarily
        frame_path = f"/tmp/frame_{frame_number}.jpg"
        cv2.imwrite(frame_path, frame)
        
        # Process with DeepFace
        results = DeepFace.represent(
            img_path=frame_path,
            model_name="ArcFace",
            detector_backend="retinaface"
        )
        
        # Store faces with frame_number
        # ...
    
    frame_number += 1
```

---

## Storage Implications

### File Size Impact

| Media Type | Typical Size | Storage Impact |
|------------|--------------|----------------|
| Photo (JPEG) | 2-10 MB | Baseline |
| Video (1080p, 1 min) | 50-150 MB | **5-75x larger** |
| Video (4K, 1 min) | 200-500 MB | **20-250x larger** |
| Video (1080p, 10 min) | 500 MB - 1.5 GB | **50-750x larger** |

### Storage Considerations

#### 1. Video File Storage
- Videos are **much larger** than photos
- A 1-hour 1080p video can be **3-9 GB**
- Large video libraries will require **significant storage capacity**

#### 2. Frame Extraction Storage

**Option A: Extract frames to disk (temporary or permanent)**
- ~1-5 MB per extracted frame (JPEG)
- For a 10-minute video at 1 fps: ~600 frames = **600 MB - 3 GB**
- Permanent storage would be massive

**Option B: Extract frames on-demand (no permanent storage)**
- Minimal storage impact
- But repeated processing overhead if re-processing needed

**Recommendation:** Extract frames on-demand during processing (don't store permanently)

#### 3. Thumbnail Generation
- Video thumbnails: ~50-200 KB each
- Much smaller than full frames
- **Recommendation:** Generate and cache video thumbnails

#### 4. Storage Recommendations

- ✅ Store original videos by path reference (like photos)
- ✅ Extract frames on-demand during processing (don't store permanently)
- ✅ Generate and cache video thumbnails (small storage impact)
- ⚠️ Consider video compression/transcoding for storage optimization

---

## Performance Implications

### Processing Performance

#### 1. Video Decoding Overhead
- Video decoding is **CPU/GPU intensive**
- **10x-100x slower** than loading a single image
- Example: Processing 1 video frame ≈ 0.1-1 second vs. 0.01-0.1s for a photo

#### 2. Frame Extraction Strategy

**Process every frame:**
- Very slow (e.g., 30 fps = 1800 frames/min)
- **Not recommended** for most use cases

**Process every Nth frame:**
- Faster (e.g., 1 fps = 60 frames/min)
- **Recommended:** 0.5-2 fps for face detection
- Balance between speed and coverage

**Scene-change detection:**
- Smart but adds complexity
- Only process frames with scene changes

#### 3. Face Detection on Video Frames
- DeepFace processing time per frame: **~0.5-2 seconds**
- For a 10-minute video at 1 fps: 600 frames × 1s = **~10 minutes processing**
- For a 1-hour video: **~1 hour processing time**

#### 4. Batch Processing Impact
- Current system processes photos sequentially
- Videos will **significantly increase processing time**
- **Consider:**
  - Parallel processing (multiple workers)
  - Background job queue (already using RQ)
  - Progress tracking per video

### Database Performance

#### 1. Increased Record Count
- Each video frame with faces = multiple `Face` records
- A 10-minute video could generate **50-200 face records**
- Index performance should remain good (existing indices work)

#### 2. Query Performance
- Current queries filter by `photo_id` (indexed)
- No significant impact expected
- Consider adding `media_type` index if separating photos/videos

### Frontend Performance

#### 1. Video Playback
- Browser video players handle large files well
- Streaming/range requests needed for large videos
- Thumbnail generation critical for grid views

#### 2. Network Bandwidth
- Serving full videos requires **high bandwidth**
- **Consider:**
  - Video transcoding (multiple quality levels)
  - Progressive loading
  - CDN for large deployments

---

## Database Schema Changes

### Option 1: Extend Photo Model (✅ Implemented)

Add a `media_type` field to distinguish photos from videos:

```python
class Photo(Base):
    # ... existing fields ...
    media_type = Column(Text, default="image", nullable=False)  # "image" or "video"
    duration = Column(Integer, nullable=True)  # Video duration in seconds (future)
    frame_count = Column(Integer, nullable=True)  # Total frames in video (future)
    video_codec = Column(Text, nullable=True)  # e.g., "h264", "hevc" (future)
    video_resolution = Column(Text, nullable=True)  # e.g., "1920x1080" (future)
```

**Pros:**
- ✅ Minimal schema changes
- ✅ Reuses existing relationships (faces, tags, etc.)
- ✅ Unified query interface

**Cons:**
- Model name "Photo" becomes slightly misleading
- Some fields only apply to videos

**Status:** ✅ Implemented with `media_type` field

### Option 2: Separate Video Model (Not Implemented)

Create a new `Video` model with similar structure:

```python
class Video(Base):
    # Similar to Photo but video-specific
    # Separate relationships for faces, tags, etc.
```

**Pros:**
- Clear separation of concerns
- Video-specific fields without clutter

**Cons:**
- Duplicate code/logic
- More complex queries (union for mixed results)
- More refactoring needed

**Recommendation:** Option 1 (extend Photo model) for simplicity and unified queries

---

## Processing Pipeline Changes

### Current Flow (Photos)
```
1. Scan folder → find image files
2. Import to database (hash, metadata)
3. Process faces (DeepFace on full image)
4. Store face encodings
```

### Proposed Flow (Videos - Future)
```
1. Scan folder → find video files
2. Import to database (hash, metadata, duration, codec)
3. Extract frames (OpenCV/ffmpeg)
   - Sample at 0.5-2 fps
   - Or use scene-change detection
4. Process faces on each frame (DeepFace)
5. Store face encodings with frame timestamp
6. Generate video thumbnail
```

### Required Changes (Future Implementation)

#### 1. Video Frame Extraction
- Use OpenCV (`cv2.VideoCapture`) or ffmpeg-python
- Extract frames at configurable intervals
- Handle various video codecs/formats

#### 2. Face Processing Modifications
- Current: `process_photo_faces()` processes one image
- New: `process_video_faces()` processes multiple frames
- Track frame number/timestamp for each face

#### 3. Face Model Updates
- Add `frame_number` or `timestamp` to `Face` model
- Link faces to specific moments in video

#### 4. Thumbnail Generation
- Extract keyframe or generate from first frame
- Store thumbnail path or generate on-demand

---

## Frontend Changes (Future)

### Display Components

#### 1. Photo Grid View
- Currently: `<img src={photoUrl} />`
- Need: Conditional rendering
  - Photos: `<img>`
  - Videos: `<video>` with poster/thumbnail

#### 2. Photo Viewer Component
- Currently: Image viewer with zoom/pan
- Need: Video player for videos
  - Play/pause controls
  - Seek/scrub
  - Show detected faces at specific timestamps

#### 3. Video Player Requirements
- HTML5 `<video>` element
- Controls for playback
- Face indicators at specific timestamps
- Thumbnail generation for grid views

### API Changes (Future)

#### 1. Video Serving Endpoint
- Similar to `/api/v1/photos/{id}/image`
- `/api/v1/photos/{id}/video` for full video
- Range request support for streaming

#### 2. Thumbnail Endpoint
- `/api/v1/photos/{id}/thumbnail` for video thumbnails

#### 3. Frame Extraction Endpoint (Optional)
- `/api/v1/photos/{id}/frames/{frame_number}` for specific frames

---

## Implementation Strategy

### Phase 1: Foundation ✅ COMPLETE

1. ✅ Add `media_type` to Photo model
2. ✅ Extend `SUPPORTED_IMAGE_FORMATS` to include video formats
3. ✅ Update scanning to detect video files
4. ✅ Import videos to database (metadata only, no processing yet)
5. ✅ Mark videos as processed (skip face processing)

**Status:** ✅ Complete

### Phase 2: Frame Extraction (Future)

1. Implement video frame extraction service
2. Add frame sampling configuration
3. Store frame metadata (optional)

### Phase 3: Face Processing (Future)

1. Extend face processing to handle video frames
2. Add `frame_number`/`timestamp` to Face model
3. Process videos in background jobs

### Phase 4: Frontend (Future)

1. Add video player component
2. Update grid view for video thumbnails
3. Update photo viewer for video playback
4. Show face timestamps in video player

### Phase 5: Optimization (Future)

1. Implement thumbnail caching
2. Add video transcoding (optional)
3. Optimize frame extraction performance
4. Add progress tracking for video processing

---

## Key Considerations Summary

| Aspect | Impact | Mitigation |
|--------|--------|------------|
| **Storage** | **High** (10-1000x larger files) | Store by reference, extract frames on-demand, cache thumbnails |
| **Processing Time** | **High** (10-100x slower) | Background jobs, frame sampling, parallel processing |
| **Database** | **Low** (similar structure) | Extend Photo model, add video-specific fields |
| **Frontend** | **Medium** (new components) | Conditional rendering, video player component |
| **Network** | **High** (large file transfers) | Streaming, range requests, thumbnails |

---

## Current Implementation Details

### Files Modified

1. **`src/web/config.py`**
   - Added `SUPPORTED_VIDEO_FORMATS` constant

2. **`src/web/db/models.py`**
   - Added `media_type` field to `Photo` model (defaults to "image")

3. **`src/web/services/photo_service.py`**
   - Updated `find_photos_in_folder()` to detect videos
   - Added `extract_video_date()` function
   - Updated `import_photo_from_path()` to detect and set media type
   - Videos automatically marked as `processed=True`

4. **`src/web/services/face_service.py`**
   - Updated `process_photo_faces()` to skip videos
   - Updated `process_unprocessed_photos()` to filter out videos

5. **`src/web/app.py`**
   - Added `ensure_photo_media_type_column()` migration function

### Behavior

- ✅ Videos are scanned and imported into the database
- ✅ Videos are marked as `processed=True` so they don't appear in unprocessed photo lists
- ✅ Videos are skipped during face processing
- ✅ Video dates are extracted from metadata (if available) or file modification time
- ✅ All existing photo functionality remains unchanged

### Notes

- The `media_type` field is new. Existing databases will automatically get the column added on next server startup.
- `ffprobe` (from ffmpeg) is optional for video date extraction. If not available, the system falls back to file modification time.
- Videos are stored in the same `photos` table with `media_type='video'` for unified querying.

---

## Conclusion

Adding video support is **technically feasible** but requires significant consideration of:

- **Storage capacity** (videos are much larger)
- **Processing time** (much slower than photos)
- **Frontend changes** (video player components)

The current implementation provides the foundation for video scanning. Future phases can add frame extraction and face processing capabilities as needed.

The architecture can accommodate videos with moderate changes. The main challenges are **storage capacity** and **processing time** for large video libraries.

---

## Future Work

### Immediate Next Steps (If Implementing Full Video Support)

1. **Frame Extraction Service**
   - Implement OpenCV-based frame extraction
   - Add configurable frame sampling rate
   - Handle various video codecs

2. **Video Face Processing**
   - Extend `process_photo_faces()` to handle video frames
   - Add frame number tracking to Face model
   - Implement batch frame processing

3. **Frontend Video Player**
   - Create video player component
   - Add thumbnail generation
   - Implement face timestamp indicators

4. **Performance Optimization**
   - Implement parallel video processing
   - Add video transcoding for web delivery
   - Optimize thumbnail caching

---

**Document Version:** 1.0  
**Last Updated:** December 2024  
**Author:** AI Assistant (Cursor)







