chore: untrack internal notes and Vite cache, ignore logs
This commit is contained in:
@@ -95,3 +95,8 @@ e2e/test-results/
|
||||
e2e/blob-report/
|
||||
e2e/.auth/
|
||||
e2e/.env
|
||||
|
||||
# Internal notes and caches
|
||||
.notes/
|
||||
logs/
|
||||
.vite/
|
||||
|
||||
@@ -1,805 +0,0 @@
|
||||
# Migration Plan: Replace face_recognition with DeepFace in PunimTag
|
||||
|
||||
**Version:** 1.0
|
||||
**Created:** October 15, 2025
|
||||
**Status:** Planning Phase
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This plan outlines the complete migration from `face_recognition` library to `DeepFace` library for the PunimTag photo tagging application. Based on testing in `test_deepface_gui.py`, DeepFace provides superior accuracy using the ArcFace model with configurable detector backends.
|
||||
|
||||
**Key Changes:**
|
||||
- Face encoding dimensions: 128 → 512 (ArcFace model)
|
||||
- Detection method: HOG/CNN → RetinaFace/MTCNN/OpenCV/SSD (configurable)
|
||||
- Similarity metric: Euclidean distance → Cosine similarity
|
||||
- Face location format: (top, right, bottom, left) → {x, y, w, h}
|
||||
- No backward compatibility - fresh start with new database
|
||||
|
||||
---
|
||||
|
||||
## PHASE 1: Database Schema Updates
|
||||
|
||||
### Step 1.1: Update Database Schema
|
||||
**File:** `src/core/database.py`
|
||||
|
||||
**Actions:**
|
||||
1. **Modify `faces` table** to add DeepFace-specific columns:
|
||||
```sql
|
||||
ALTER TABLE faces ADD COLUMN detector_backend TEXT DEFAULT 'retinaface';
|
||||
ALTER TABLE faces ADD COLUMN model_name TEXT DEFAULT 'ArcFace';
|
||||
ALTER TABLE faces ADD COLUMN face_confidence REAL DEFAULT 0.0;
|
||||
```
|
||||
|
||||
2. **Update `person_encodings` table** similarly:
|
||||
```sql
|
||||
ALTER TABLE person_encodings ADD COLUMN detector_backend TEXT DEFAULT 'retinaface';
|
||||
ALTER TABLE person_encodings ADD COLUMN model_name TEXT DEFAULT 'ArcFace';
|
||||
```
|
||||
|
||||
3. **Update `init_database()` method** in `DatabaseManager` class:
|
||||
- Add new columns to CREATE TABLE statements for `faces` and `person_encodings`
|
||||
- Add indices for new columns if needed
|
||||
|
||||
**Expected encoding size change:**
|
||||
- face_recognition: 128 floats × 8 bytes = 1,024 bytes per encoding
|
||||
- DeepFace ArcFace: 512 floats × 8 bytes = 4,096 bytes per encoding
|
||||
|
||||
### Step 1.2: Update Database Methods
|
||||
**File:** `src/core/database.py`
|
||||
|
||||
**Actions:**
|
||||
1. **Modify `add_face()` method signature:**
|
||||
```python
|
||||
def add_face(self, photo_id: int, encoding: bytes, location: str,
|
||||
confidence: float = 0.0, quality_score: float = 0.0,
|
||||
person_id: Optional[int] = None,
|
||||
detector_backend: str = 'retinaface',
|
||||
model_name: str = 'ArcFace',
|
||||
face_confidence: float = 0.0) -> int:
|
||||
```
|
||||
|
||||
2. **Modify `add_person_encoding()` method signature:**
|
||||
```python
|
||||
def add_person_encoding(self, person_id: int, face_id: int, encoding: bytes,
|
||||
quality_score: float,
|
||||
detector_backend: str = 'retinaface',
|
||||
model_name: str = 'ArcFace'):
|
||||
```
|
||||
|
||||
3. **Update all database queries** that insert/update faces to include new fields
|
||||
|
||||
---
|
||||
|
||||
## PHASE 2: Configuration Updates
|
||||
|
||||
### Step 2.1: Update Configuration Constants
|
||||
**File:** `src/core/config.py`
|
||||
|
||||
**Actions:**
|
||||
1. **Replace face_recognition settings:**
|
||||
```python
|
||||
# OLD - Remove these:
|
||||
# DEFAULT_FACE_DETECTION_MODEL = "hog"
|
||||
# DEFAULT_FACE_TOLERANCE = 0.6
|
||||
|
||||
# NEW - Add these:
|
||||
DEEPFACE_DETECTOR_BACKEND = "retinaface" # Options: retinaface, mtcnn, opencv, ssd
|
||||
DEEPFACE_MODEL_NAME = "ArcFace" # Best accuracy model
|
||||
DEEPFACE_DISTANCE_METRIC = "cosine" # For similarity calculation
|
||||
DEEPFACE_ENFORCE_DETECTION = False # Don't fail if no faces found
|
||||
DEEPFACE_ALIGN_FACES = True # Face alignment for better accuracy
|
||||
|
||||
# Tolerance/threshold adjustments for DeepFace
|
||||
DEFAULT_FACE_TOLERANCE = 0.4 # Lower for DeepFace (was 0.6 for face_recognition)
|
||||
DEEPFACE_SIMILARITY_THRESHOLD = 60 # Minimum similarity percentage (0-100)
|
||||
|
||||
# Environment settings for TensorFlow (DeepFace uses TensorFlow backend)
|
||||
import os
|
||||
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # Suppress TensorFlow warnings
|
||||
```
|
||||
|
||||
2. **Add detector backend options:**
|
||||
```python
|
||||
DEEPFACE_DETECTOR_OPTIONS = ["retinaface", "mtcnn", "opencv", "ssd"]
|
||||
DEEPFACE_MODEL_OPTIONS = ["ArcFace", "Facenet", "Facenet512", "VGG-Face"]
|
||||
```
|
||||
|
||||
### Step 2.2: Add TensorFlow Suppression
|
||||
**File:** Add to all main entry points (dashboard_gui.py, photo_tagger.py, etc.)
|
||||
|
||||
**Actions:**
|
||||
```python
|
||||
# At the top of file, before other imports
|
||||
import os
|
||||
import warnings
|
||||
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
# Then import DeepFace
|
||||
from deepface import DeepFace
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 3: Face Processing Core Migration
|
||||
|
||||
### Step 3.1: Replace Face Detection and Encoding
|
||||
**File:** `src/core/face_processing.py` → `FaceProcessor` class
|
||||
|
||||
**Method:** `process_faces()`
|
||||
|
||||
**Current Implementation (lines 59-144):**
|
||||
```python
|
||||
# OLD CODE:
|
||||
image = face_recognition.load_image_file(photo_path)
|
||||
face_locations = face_recognition.face_locations(image, model=model)
|
||||
face_encodings = face_recognition.face_encodings(image, face_locations)
|
||||
```
|
||||
|
||||
**New Implementation:**
|
||||
```python
|
||||
# NEW CODE:
|
||||
try:
|
||||
# Use DeepFace.represent() to get face detection and encodings
|
||||
results = DeepFace.represent(
|
||||
img_path=photo_path,
|
||||
model_name=DEEPFACE_MODEL_NAME, # 'ArcFace'
|
||||
detector_backend=DEEPFACE_DETECTOR_BACKEND, # 'retinaface'
|
||||
enforce_detection=DEEPFACE_ENFORCE_DETECTION, # False
|
||||
align=DEEPFACE_ALIGN_FACES # True
|
||||
)
|
||||
|
||||
if not results:
|
||||
if self.verbose >= 1:
|
||||
print(f" 👤 No faces found")
|
||||
# Mark as processed even with no faces
|
||||
self.db.mark_photo_processed(photo_id)
|
||||
continue
|
||||
|
||||
if self.verbose >= 1:
|
||||
print(f" 👤 Found {len(results)} faces")
|
||||
|
||||
# Process each detected face
|
||||
for i, result in enumerate(results):
|
||||
# Extract face region info from DeepFace result
|
||||
facial_area = result.get('facial_area', {})
|
||||
face_confidence = result.get('face_confidence', 0.0)
|
||||
embedding = np.array(result['embedding'])
|
||||
|
||||
# Convert DeepFace facial_area {x, y, w, h} to our location format
|
||||
# Store as dict for consistency
|
||||
location = {
|
||||
'x': facial_area.get('x', 0),
|
||||
'y': facial_area.get('y', 0),
|
||||
'w': facial_area.get('w', 0),
|
||||
'h': facial_area.get('h', 0)
|
||||
}
|
||||
|
||||
# Calculate face quality score (reuse existing method)
|
||||
# Convert facial_area to (top, right, bottom, left) for quality calculation
|
||||
face_location_tuple = (
|
||||
facial_area.get('y', 0), # top
|
||||
facial_area.get('x', 0) + facial_area.get('w', 0), # right
|
||||
facial_area.get('y', 0) + facial_area.get('h', 0), # bottom
|
||||
facial_area.get('x', 0) # left
|
||||
)
|
||||
|
||||
# Load image for quality calculation
|
||||
image = Image.open(photo_path)
|
||||
image_np = np.array(image)
|
||||
quality_score = self._calculate_face_quality_score(image_np, face_location_tuple)
|
||||
|
||||
# Store in database with new format
|
||||
self.db.add_face(
|
||||
photo_id=photo_id,
|
||||
encoding=embedding.tobytes(),
|
||||
location=str(location), # Store as string representation of dict
|
||||
confidence=0.0, # Legacy field, keep for compatibility
|
||||
quality_score=quality_score,
|
||||
person_id=None,
|
||||
detector_backend=DEEPFACE_DETECTOR_BACKEND,
|
||||
model_name=DEEPFACE_MODEL_NAME,
|
||||
face_confidence=face_confidence
|
||||
)
|
||||
|
||||
if self.verbose >= 3:
|
||||
print(f" Face {i+1}: {location} (quality: {quality_score:.2f}, confidence: {face_confidence:.2f})")
|
||||
|
||||
# Mark as processed
|
||||
self.db.mark_photo_processed(photo_id)
|
||||
processed_count += 1
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error processing {filename}: {e}")
|
||||
self.db.mark_photo_processed(photo_id)
|
||||
```
|
||||
|
||||
### Step 3.2: Update Face Location Handling
|
||||
**File:** `src/core/face_processing.py`
|
||||
|
||||
**Method:** `_extract_face_crop()` (appears twice, lines 212-271 and 541-600)
|
||||
|
||||
**Current Implementation:**
|
||||
```python
|
||||
# OLD: face_recognition format (top, right, bottom, left)
|
||||
top, right, bottom, left = location
|
||||
```
|
||||
|
||||
**New Implementation:**
|
||||
```python
|
||||
# NEW: DeepFace format {x, y, w, h}
|
||||
# Parse location from string if needed
|
||||
if isinstance(location, str):
|
||||
location = eval(location) # Convert string to dict
|
||||
|
||||
# Handle both formats for compatibility during migration
|
||||
if isinstance(location, dict):
|
||||
# DeepFace format
|
||||
left = location.get('x', 0)
|
||||
top = location.get('y', 0)
|
||||
width = location.get('w', 0)
|
||||
height = location.get('h', 0)
|
||||
right = left + width
|
||||
bottom = top + height
|
||||
else:
|
||||
# Legacy face_recognition format (top, right, bottom, left)
|
||||
top, right, bottom, left = location
|
||||
|
||||
# Rest of the method remains the same
|
||||
face_width = right - left
|
||||
face_height = bottom - top
|
||||
# ... etc
|
||||
```
|
||||
|
||||
### Step 3.3: Replace Similarity Calculation
|
||||
**File:** `src/core/face_processing.py`
|
||||
|
||||
**Method:** `find_similar_faces()` and helper methods
|
||||
|
||||
**Current Implementation (line 457):**
|
||||
```python
|
||||
# OLD:
|
||||
distance = face_recognition.face_distance([target_encoding], other_enc)[0]
|
||||
```
|
||||
|
||||
**New Implementation:**
|
||||
```python
|
||||
# NEW: Use cosine similarity (same as test_deepface_gui.py)
|
||||
def _calculate_cosine_similarity(self, encoding1: np.ndarray, encoding2: np.ndarray) -> float:
|
||||
"""Calculate cosine similarity between two face encodings"""
|
||||
try:
|
||||
# Ensure encodings are numpy arrays
|
||||
enc1 = np.array(encoding1).flatten()
|
||||
enc2 = np.array(encoding2).flatten()
|
||||
|
||||
# Check if encodings have the same length
|
||||
if len(enc1) != len(enc2):
|
||||
print(f"Warning: Encoding length mismatch: {len(enc1)} vs {len(enc2)}")
|
||||
return 0.0
|
||||
|
||||
# Normalize encodings
|
||||
enc1_norm = enc1 / (np.linalg.norm(enc1) + 1e-8)
|
||||
enc2_norm = enc2 / (np.linalg.norm(enc2) + 1e-8)
|
||||
|
||||
# Calculate cosine similarity
|
||||
cosine_sim = np.dot(enc1_norm, enc2_norm)
|
||||
|
||||
# Clamp to valid range [-1, 1]
|
||||
cosine_sim = np.clip(cosine_sim, -1.0, 1.0)
|
||||
|
||||
# Convert to distance (0 = identical, 2 = opposite)
|
||||
# For consistency with face_recognition's distance metric
|
||||
distance = 1.0 - cosine_sim # Range [0, 2], where 0 is perfect match
|
||||
|
||||
return distance
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error calculating similarity: {e}")
|
||||
return 2.0 # Maximum distance on error
|
||||
|
||||
# Replace in find_similar_faces():
|
||||
distance = self._calculate_cosine_similarity(target_encoding, other_enc)
|
||||
```
|
||||
|
||||
**Update adaptive tolerance calculation (line 333-351):**
|
||||
```python
|
||||
def _calculate_adaptive_tolerance(self, base_tolerance: float, face_quality: float, match_confidence: float = None) -> float:
|
||||
"""Calculate adaptive tolerance based on face quality and match confidence
|
||||
|
||||
Note: For DeepFace, tolerance values are generally lower than face_recognition
|
||||
"""
|
||||
# Start with base tolerance (e.g., 0.4 instead of 0.6)
|
||||
tolerance = base_tolerance
|
||||
|
||||
# Adjust based on face quality
|
||||
quality_factor = 0.9 + (face_quality * 0.2) # Range: 0.9 to 1.1
|
||||
tolerance *= quality_factor
|
||||
|
||||
# Adjust based on match confidence if provided
|
||||
if match_confidence is not None:
|
||||
confidence_factor = 0.95 + (match_confidence * 0.1)
|
||||
tolerance *= confidence_factor
|
||||
|
||||
# Ensure tolerance stays within reasonable bounds for DeepFace
|
||||
return max(0.2, min(0.6, tolerance)) # Lower range for DeepFace
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 4: GUI Integration Updates
|
||||
|
||||
### Step 4.1: Add DeepFace Settings to Dashboard
|
||||
**File:** `src/gui/dashboard_gui.py`
|
||||
|
||||
**Actions:**
|
||||
1. **Add detector selection to menu or settings:**
|
||||
```python
|
||||
# Add to settings menu or control panel
|
||||
detector_frame = ttk.Frame(settings_panel)
|
||||
detector_frame.pack(fill=tk.X, padx=5, pady=5)
|
||||
|
||||
ttk.Label(detector_frame, text="Face Detector:").pack(side=tk.LEFT, padx=5)
|
||||
self.detector_var = tk.StringVar(value=DEEPFACE_DETECTOR_BACKEND)
|
||||
detector_combo = ttk.Combobox(detector_frame, textvariable=self.detector_var,
|
||||
values=DEEPFACE_DETECTOR_OPTIONS,
|
||||
state="readonly", width=12)
|
||||
detector_combo.pack(side=tk.LEFT, padx=5)
|
||||
|
||||
ttk.Label(detector_frame, text="Model:").pack(side=tk.LEFT, padx=5)
|
||||
self.model_var = tk.StringVar(value=DEEPFACE_MODEL_NAME)
|
||||
model_combo = ttk.Combobox(detector_frame, textvariable=self.model_var,
|
||||
values=DEEPFACE_MODEL_OPTIONS,
|
||||
state="readonly", width=12)
|
||||
model_combo.pack(side=tk.LEFT, padx=5)
|
||||
```
|
||||
|
||||
2. **Update process_faces calls to use selected settings:**
|
||||
```python
|
||||
# When calling face processor
|
||||
detector = self.detector_var.get()
|
||||
model = self.model_var.get()
|
||||
|
||||
# Pass to FaceProcessor (need to update FaceProcessor.__init__ to accept these)
|
||||
self.face_processor = FaceProcessor(
|
||||
db_manager=self.db,
|
||||
verbose=self.verbose,
|
||||
detector_backend=detector,
|
||||
model_name=model
|
||||
)
|
||||
```
|
||||
|
||||
### Step 4.2: Update Face Processor Initialization
|
||||
**File:** `src/core/face_processing.py`
|
||||
|
||||
**Class:** `FaceProcessor.__init__()`
|
||||
|
||||
**Actions:**
|
||||
```python
|
||||
def __init__(self, db_manager: DatabaseManager, verbose: int = 0,
|
||||
detector_backend: str = None, model_name: str = None):
|
||||
"""Initialize face processor with DeepFace settings"""
|
||||
self.db = db_manager
|
||||
self.verbose = verbose
|
||||
self.detector_backend = detector_backend or DEEPFACE_DETECTOR_BACKEND
|
||||
self.model_name = model_name or DEEPFACE_MODEL_NAME
|
||||
self._face_encoding_cache = {}
|
||||
self._image_cache = {}
|
||||
```
|
||||
|
||||
### Step 4.3: Update GUI Display Methods
|
||||
**File:** Multiple panel files (identify_panel.py, auto_match_panel.py, modify_panel.py)
|
||||
|
||||
**Actions:**
|
||||
1. **Update all face thumbnail extraction** to handle new location format
|
||||
2. **Update confidence display** to use cosine similarity percentages
|
||||
3. **Update any hardcoded face_recognition references**
|
||||
|
||||
**Example for identify_panel.py:**
|
||||
```python
|
||||
# In display methods, convert distance to percentage
|
||||
confidence_pct = (1 - distance) * 100 # Already done correctly
|
||||
# But ensure distance calculation uses cosine similarity
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 5: Dependencies and Installation
|
||||
|
||||
### Step 5.1: Update requirements.txt
|
||||
**File:** `requirements.txt`
|
||||
|
||||
**Actions:**
|
||||
```python
|
||||
# REMOVE these:
|
||||
# face-recognition==1.3.0
|
||||
# face-recognition-models==0.3.0
|
||||
# dlib>=20.0.0
|
||||
|
||||
# ADD these:
|
||||
deepface>=0.0.79
|
||||
tensorflow>=2.13.0 # Required by DeepFace
|
||||
opencv-python>=4.8.0 # Required by DeepFace
|
||||
retina-face>=0.0.13 # For RetinaFace detector (best accuracy)
|
||||
|
||||
# KEEP these:
|
||||
numpy>=1.21.0
|
||||
pillow>=8.0.0
|
||||
click>=8.0.0
|
||||
setuptools>=40.0.0
|
||||
```
|
||||
|
||||
### Step 5.2: Create Migration Script
|
||||
**File:** `scripts/migrate_to_deepface.py` (new file)
|
||||
|
||||
**Purpose:** Drop all tables and reinitialize database for fresh start
|
||||
|
||||
**Actions:**
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration script to prepare database for DeepFace
|
||||
Drops all existing tables and recreates with new schema
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add parent directory to path
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
from src.core.database import DatabaseManager
|
||||
from src.core.config import DEFAULT_DB_PATH
|
||||
|
||||
def migrate_database():
|
||||
"""Drop all tables and reinitialize with DeepFace schema"""
|
||||
print("⚠️ WARNING: This will delete all existing data!")
|
||||
response = input("Type 'DELETE ALL DATA' to confirm: ")
|
||||
|
||||
if response != "DELETE ALL DATA":
|
||||
print("Migration cancelled.")
|
||||
return
|
||||
|
||||
print("\n🗑️ Dropping all existing tables...")
|
||||
|
||||
# Connect directly to database
|
||||
conn = sqlite3.connect(DEFAULT_DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Drop all tables
|
||||
tables = ['phototaglinkage', 'person_encodings', 'faces', 'tags', 'people', 'photos']
|
||||
for table in tables:
|
||||
cursor.execute(f'DROP TABLE IF EXISTS {table}')
|
||||
print(f" Dropped table: {table}")
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print("\n✅ All tables dropped successfully")
|
||||
print("\n🔄 Reinitializing database with DeepFace schema...")
|
||||
|
||||
# Reinitialize with new schema
|
||||
db = DatabaseManager(DEFAULT_DB_PATH, verbose=1)
|
||||
|
||||
print("\n✅ Database migration complete!")
|
||||
print("\nNext steps:")
|
||||
print("1. Add photos using the dashboard")
|
||||
print("2. Process faces with DeepFace")
|
||||
print("3. Identify people")
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 6: Testing and Validation
|
||||
|
||||
### Step 6.1: Create Test Suite
|
||||
**File:** `tests/test_deepface_integration.py` (new file)
|
||||
|
||||
**Actions:**
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test DeepFace integration in PunimTag
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Suppress TensorFlow warnings
|
||||
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
|
||||
import warnings
|
||||
warnings.filterwarnings('ignore')
|
||||
|
||||
from src.core.database import DatabaseManager
|
||||
from src.core.face_processing import FaceProcessor
|
||||
from src.core.config import DEEPFACE_DETECTOR_BACKEND, DEEPFACE_MODEL_NAME
|
||||
|
||||
def test_face_detection():
|
||||
"""Test face detection with DeepFace"""
|
||||
print("🧪 Testing DeepFace face detection...")
|
||||
|
||||
db = DatabaseManager(":memory:", verbose=0) # In-memory database for testing
|
||||
processor = FaceProcessor(db, verbose=1)
|
||||
|
||||
# Test with a sample image
|
||||
test_image = "demo_photos/2019-11-22_0011.jpg"
|
||||
if not os.path.exists(test_image):
|
||||
print(f"❌ Test image not found: {test_image}")
|
||||
return False
|
||||
|
||||
# Add photo to database
|
||||
photo_id = db.add_photo(test_image, Path(test_image).name, None)
|
||||
|
||||
# Process faces
|
||||
count = processor.process_faces(limit=1)
|
||||
|
||||
# Verify results
|
||||
stats = db.get_statistics()
|
||||
print(f"✅ Processed {count} photos, found {stats['total_faces']} faces")
|
||||
|
||||
return stats['total_faces'] > 0
|
||||
|
||||
def test_face_matching():
|
||||
"""Test face matching with DeepFace"""
|
||||
print("\n🧪 Testing DeepFace face matching...")
|
||||
|
||||
db = DatabaseManager(":memory:", verbose=0)
|
||||
processor = FaceProcessor(db, verbose=1)
|
||||
|
||||
# Test with multiple images
|
||||
test_images = [
|
||||
"demo_photos/2019-11-22_0011.jpg",
|
||||
"demo_photos/2019-11-22_0012.jpg"
|
||||
]
|
||||
|
||||
for img in test_images:
|
||||
if os.path.exists(img):
|
||||
photo_id = db.add_photo(img, Path(img).name, None)
|
||||
|
||||
# Process all faces
|
||||
processor.process_faces(limit=10)
|
||||
|
||||
# Find similar faces
|
||||
faces = db.get_all_face_encodings()
|
||||
if len(faces) >= 2:
|
||||
matches = processor.find_similar_faces(faces[0][0])
|
||||
print(f"✅ Found {len(matches)} similar faces")
|
||||
return len(matches) >= 0
|
||||
|
||||
return False
|
||||
|
||||
def run_all_tests():
|
||||
"""Run all tests"""
|
||||
print("=" * 60)
|
||||
print("DeepFace Integration Test Suite")
|
||||
print("=" * 60)
|
||||
|
||||
tests = [
|
||||
test_face_detection,
|
||||
test_face_matching
|
||||
]
|
||||
|
||||
results = []
|
||||
for test in tests:
|
||||
try:
|
||||
result = test()
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
print(f"❌ Test failed with error: {e}")
|
||||
results.append(False)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print(f"Tests passed: {sum(results)}/{len(results)}")
|
||||
print("=" * 60)
|
||||
|
||||
return all(results)
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = run_all_tests()
|
||||
sys.exit(0 if success else 1)
|
||||
```
|
||||
|
||||
### Step 6.2: Validation Checklist
|
||||
1. **Face Detection:**
|
||||
- [ ] DeepFace successfully detects faces in test images
|
||||
- [ ] Face locations are correctly stored in new format
|
||||
- [ ] Face encodings are 512-dimensional (ArcFace)
|
||||
- [ ] Multiple detector backends work (retinaface, mtcnn, etc.)
|
||||
|
||||
2. **Face Matching:**
|
||||
- [ ] Similar faces are correctly identified
|
||||
- [ ] Cosine similarity produces reasonable confidence scores
|
||||
- [ ] Adaptive tolerance works with new metric
|
||||
- [ ] No false positives at default threshold
|
||||
|
||||
3. **GUI Integration:**
|
||||
- [ ] All panels display faces correctly
|
||||
- [ ] Face thumbnails extract properly with new location format
|
||||
- [ ] Confidence scores display correctly
|
||||
- [ ] Detector/model selection works in settings
|
||||
|
||||
4. **Database:**
|
||||
- [ ] New columns are created correctly
|
||||
- [ ] Encodings are stored as 4096-byte BLOBs
|
||||
- [ ] Queries work with new schema
|
||||
- [ ] Indices improve performance
|
||||
|
||||
---
|
||||
|
||||
## PHASE 7: Implementation Order
|
||||
|
||||
**Execute in this order to minimize issues:**
|
||||
|
||||
1. **Day 1: Database & Configuration**
|
||||
- Update `requirements.txt`
|
||||
- Install DeepFace: `pip install deepface tensorflow opencv-python retina-face`
|
||||
- Update `src/core/config.py` with DeepFace settings
|
||||
- Update `src/core/database.py` schema and methods
|
||||
- Create and run `scripts/migrate_to_deepface.py`
|
||||
|
||||
2. **Day 2: Core Face Processing**
|
||||
- Update `src/core/face_processing.py` `process_faces()` method
|
||||
- Update `_extract_face_crop()` to handle new location format
|
||||
- Implement `_calculate_cosine_similarity()` method
|
||||
- Update `find_similar_faces()` to use new similarity
|
||||
- Update `_calculate_adaptive_tolerance()` for DeepFace ranges
|
||||
|
||||
3. **Day 3: GUI Updates**
|
||||
- Add detector/model selection to dashboard
|
||||
- Update `FaceProcessor.__init__()` to accept settings
|
||||
- Test face processing with GUI
|
||||
|
||||
4. **Day 4: Panel Updates**
|
||||
- Update `src/gui/identify_panel.py` for new format
|
||||
- Update `src/gui/auto_match_panel.py` for new format
|
||||
- Update `src/gui/modify_panel.py` for new format
|
||||
- Verify all face displays work correctly
|
||||
|
||||
5. **Day 5: Testing & Refinement**
|
||||
- Create `tests/test_deepface_integration.py`
|
||||
- Run all tests and fix issues
|
||||
- Process test photos from `demo_photos/testdeepface/`
|
||||
- Validate matching accuracy
|
||||
- Adjust thresholds if needed
|
||||
|
||||
6. **Day 6: Documentation & Cleanup**
|
||||
- Update README.md with DeepFace information
|
||||
- Document detector backend options
|
||||
- Document model options and trade-offs
|
||||
- Remove old face_recognition references
|
||||
- Final testing
|
||||
|
||||
---
|
||||
|
||||
## PHASE 8: Key Differences and Gotchas
|
||||
|
||||
### Encoding Size Change
|
||||
- **face_recognition:** 128 floats = 1,024 bytes
|
||||
- **DeepFace ArcFace:** 512 floats = 4,096 bytes
|
||||
- **Impact:** Database size will be ~4x larger for encodings
|
||||
- **Action:** Ensure sufficient disk space
|
||||
|
||||
### Location Format Change
|
||||
- **face_recognition:** tuple `(top, right, bottom, left)`
|
||||
- **DeepFace:** dict `{'x': x, 'y': y, 'w': w, 'h': h}`
|
||||
- **Impact:** All location parsing code must be updated
|
||||
- **Action:** Create helper function to convert between formats
|
||||
|
||||
### Tolerance/Threshold Adjustments
|
||||
- **face_recognition:** Default 0.6 works well
|
||||
- **DeepFace:** Lower tolerance needed (0.4 recommended)
|
||||
- **Impact:** Matching sensitivity changes
|
||||
- **Action:** Test and adjust `DEFAULT_FACE_TOLERANCE` in config
|
||||
|
||||
### Performance Considerations
|
||||
- **DeepFace:** Slower than face_recognition (uses deep learning)
|
||||
- **Mitigation:** Use GPU if available, cache results, process in batches
|
||||
- **Action:** Add progress indicators, allow cancellation
|
||||
|
||||
### Dependencies
|
||||
- **DeepFace requires:** TensorFlow, OpenCV, specific detectors
|
||||
- **Size:** ~500MB+ of additional packages and models
|
||||
- **Action:** Warn users about download size during installation
|
||||
|
||||
---
|
||||
|
||||
## PHASE 9: Rollback Plan (If Needed)
|
||||
|
||||
Since we're starting fresh (no backward compatibility), rollback is simple:
|
||||
|
||||
1. **Restore database:**
|
||||
```bash
|
||||
rm data/photos.db
|
||||
cp data/photos.db.backup data/photos.db # If backup exists
|
||||
```
|
||||
|
||||
2. **Restore code:**
|
||||
```bash
|
||||
git checkout HEAD -- src/core/face_processing.py src/core/database.py src/core/config.py requirements.txt
|
||||
```
|
||||
|
||||
3. **Reinstall dependencies:**
|
||||
```bash
|
||||
pip uninstall deepface tensorflow
|
||||
pip install face-recognition
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PHASE 10: Success Criteria
|
||||
|
||||
Migration is complete when:
|
||||
|
||||
- [ ] All face_recognition imports removed
|
||||
- [ ] DeepFace successfully detects faces in test images
|
||||
- [ ] Face matching produces accurate results
|
||||
- [ ] All GUI panels work with new format
|
||||
- [ ] Database stores DeepFace encodings correctly
|
||||
- [ ] Test suite passes all tests
|
||||
- [ ] Documentation updated
|
||||
- [ ] No regression in core functionality
|
||||
- [ ] Performance is acceptable (may be slower but accurate)
|
||||
|
||||
---
|
||||
|
||||
## Additional Notes for Agent
|
||||
|
||||
1. **Import Order Matters:**
|
||||
```python
|
||||
# ALWAYS import in this order:
|
||||
import os
|
||||
import warnings
|
||||
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
|
||||
warnings.filterwarnings('ignore')
|
||||
# THEN import DeepFace
|
||||
from deepface import DeepFace
|
||||
```
|
||||
|
||||
2. **Error Handling:**
|
||||
- DeepFace can throw various TensorFlow errors
|
||||
- Always wrap DeepFace calls in try-except
|
||||
- Use `enforce_detection=False` to avoid crashes on no-face images
|
||||
|
||||
3. **Model Downloads:**
|
||||
- First run will download models (~100MB+)
|
||||
- Store in `~/.deepface/weights/`
|
||||
- Plan for initial download time
|
||||
|
||||
4. **Testing Strategy:**
|
||||
- Use `demo_photos/testdeepface/` for initial testing
|
||||
- These were already tested in `test_deepface_gui.py`
|
||||
- Known good results for comparison
|
||||
|
||||
5. **Code Quality:**
|
||||
- Maintain existing code style
|
||||
- Keep verbose levels consistent
|
||||
- Preserve all existing functionality
|
||||
- Add comments explaining DeepFace-specific code
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify (Summary)
|
||||
|
||||
1. **`requirements.txt`** - Update dependencies
|
||||
2. **`src/core/config.py`** - Add DeepFace configuration
|
||||
3. **`src/core/database.py`** - Update schema and methods
|
||||
4. **`src/core/face_processing.py`** - Replace all face_recognition code
|
||||
5. **`src/gui/dashboard_gui.py`** - Add detector/model selection
|
||||
6. **`src/gui/identify_panel.py`** - Update for new formats
|
||||
7. **`src/gui/auto_match_panel.py`** - Update for new formats
|
||||
8. **`src/gui/modify_panel.py`** - Update for new formats
|
||||
9. **`scripts/migrate_to_deepface.py`** - New migration script
|
||||
10. **`tests/test_deepface_integration.py`** - New test suite
|
||||
|
||||
---
|
||||
|
||||
**END OF MIGRATION PLAN**
|
||||
|
||||
This plan provides a complete, step-by-step guide for migrating from face_recognition to DeepFace. Execute phases in order, test thoroughly, and refer to `tests/test_deepface_gui.py` for working DeepFace implementation examples.
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
# Directory Structure
|
||||
|
||||
## Overview
|
||||
```
|
||||
punimtag/
|
||||
├── .notes/ # Project notes and planning
|
||||
│ ├── project_overview.md # High-level project info
|
||||
│ ├── task_list.md # Task tracking
|
||||
│ ├── directory_structure.md # This file
|
||||
│ └── meeting_notes.md # Meeting records
|
||||
│
|
||||
├── src/ # Source code
|
||||
│ ├── __init__.py
|
||||
│ ├── photo_tagger.py # CLI entry point
|
||||
│ ├── setup.py # Package setup
|
||||
│ │
|
||||
│ ├── core/ # Business logic
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── config.py # Configuration
|
||||
│ │ ├── database.py # Database manager
|
||||
│ │ ├── face_processing.py # Face recognition
|
||||
│ │ ├── photo_management.py # Photo operations
|
||||
│ │ ├── tag_management.py # Tag operations
|
||||
│ │ └── search_stats.py # Search & analytics
|
||||
│ │
|
||||
│ ├── gui/ # GUI components
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── dashboard_gui.py # Main dashboard
|
||||
│ │ ├── gui_core.py # Common utilities
|
||||
│ │ ├── identify_panel.py # Identification UI
|
||||
│ │ ├── auto_match_panel.py # Auto-matching UI
|
||||
│ │ ├── modify_panel.py # Person editing UI
|
||||
│ │ └── tag_manager_panel.py # Tag management UI
|
||||
│ │
|
||||
│ └── utils/ # Utility functions
|
||||
│ ├── __init__.py
|
||||
│ └── path_utils.py # Path operations
|
||||
│
|
||||
├── tests/ # Test suite
|
||||
│ ├── __init__.py
|
||||
│ ├── test_deepface_gui.py # DeepFace testing
|
||||
│ ├── test_face_recognition.py # Face rec tests
|
||||
│ ├── test_simple_gui.py # GUI tests
|
||||
│ ├── test_thumbnail_sizes.py # UI tests
|
||||
│ ├── debug_face_detection.py # Debug tools
|
||||
│ └── show_large_thumbnails.py # Debug tools
|
||||
│
|
||||
├── docs/ # Documentation
|
||||
│ ├── README.md # Main documentation
|
||||
│ ├── ARCHITECTURE.md # System architecture
|
||||
│ ├── DEMO.md # Demo guide
|
||||
│ └── README_UNIFIED_DASHBOARD.md
|
||||
│
|
||||
├── data/ # Application data
|
||||
│ └── photos.db # SQLite database
|
||||
│
|
||||
├── demo_photos/ # Sample photos for testing
|
||||
│ ├── events/
|
||||
│ ├── more_photos/
|
||||
│ └── testdeepface/
|
||||
│
|
||||
├── scripts/ # Utility scripts
|
||||
│ └── drop_all_tables.py # Database utilities
|
||||
│
|
||||
├── archive/ # Legacy/backup files
|
||||
│ ├── *_backup.py # Old versions
|
||||
│ └── *_gui.py # Legacy GUIs
|
||||
│
|
||||
├── logs/ # Application logs
|
||||
│
|
||||
├── venv/ # Virtual environment
|
||||
│
|
||||
├── .git/ # Git repository
|
||||
├── .gitignore # Git ignore rules
|
||||
├── .cursorrules # Cursor AI rules
|
||||
├── .cursorignore # Cursor ignore rules
|
||||
├── requirements.txt # Python dependencies
|
||||
├── gui_config.json # GUI preferences
|
||||
├── demo.sh # Demo script
|
||||
└── run_deepface_gui.sh # Run script
|
||||
```
|
||||
|
||||
## Import Path Examples
|
||||
|
||||
### From core modules:
|
||||
```python
|
||||
from src.core.database import DatabaseManager
|
||||
from src.core.face_processing import FaceProcessor
|
||||
from src.core.config import DEFAULT_DB_PATH
|
||||
```
|
||||
|
||||
### From GUI modules:
|
||||
```python
|
||||
from src.gui.dashboard_gui import DashboardGUI
|
||||
from src.gui.gui_core import GUICore
|
||||
```
|
||||
|
||||
### From utils:
|
||||
```python
|
||||
from src.utils.path_utils import normalize_path
|
||||
```
|
||||
|
||||
## Entry Points
|
||||
|
||||
### GUI Application
|
||||
```bash
|
||||
python src/gui/dashboard_gui.py
|
||||
```
|
||||
|
||||
### CLI Application
|
||||
```bash
|
||||
python src/photo_tagger.py
|
||||
```
|
||||
|
||||
### Tests
|
||||
```bash
|
||||
python -m pytest tests/
|
||||
|
||||
```
|
||||
|
||||
## Notes
|
||||
- All source code in `src/` directory
|
||||
- Tests separate from source code
|
||||
- Documentation in `docs/`
|
||||
- Project notes in `.notes/`
|
||||
- Legacy code archived in `archive/`
|
||||
|
||||
@@ -1,73 +0,0 @@
|
||||
# Meeting Notes
|
||||
|
||||
## 2025-10-15: Project Restructuring
|
||||
|
||||
### Attendees
|
||||
- Development Team
|
||||
|
||||
### Discussion
|
||||
- Agreed to restructure project for better organization
|
||||
- Adopted standard Python project layout
|
||||
- Separated concerns: core, gui, utils, tests
|
||||
- Created .notes directory for project management
|
||||
|
||||
### Decisions
|
||||
1. Move all business logic to `src/core/`
|
||||
2. Move all GUI components to `src/gui/`
|
||||
3. Move utilities to `src/utils/`
|
||||
4. Consolidate tests in `tests/`
|
||||
5. Move documentation to `docs/`
|
||||
6. Archive legacy code instead of deleting
|
||||
|
||||
### Action Items
|
||||
- [x] Create new directory structure
|
||||
- [x] Move files to appropriate locations
|
||||
- [x] Create __init__.py files for packages
|
||||
- [x] Create project notes
|
||||
- [ ] Update import statements
|
||||
- [ ] Test all functionality
|
||||
- [ ] Update documentation
|
||||
|
||||
---
|
||||
|
||||
## 2025-10-15: DeepFace Migration Planning
|
||||
|
||||
### Attendees
|
||||
- Development Team
|
||||
|
||||
### Discussion
|
||||
- Analyzed test_deepface_gui.py results
|
||||
- DeepFace shows better accuracy than face_recognition
|
||||
- ArcFace model recommended for best results
|
||||
- RetinaFace detector provides best face detection
|
||||
|
||||
### Decisions
|
||||
1. Migrate from face_recognition to DeepFace
|
||||
2. Use ArcFace model (512-dim encodings)
|
||||
3. Use RetinaFace detector as default
|
||||
4. Support multiple detector backends
|
||||
5. No backward compatibility - fresh start
|
||||
|
||||
### Action Items
|
||||
- [x] Document migration plan
|
||||
- [x] Create architecture document
|
||||
- [ ] Update database schema
|
||||
- [ ] Implement DeepFace integration
|
||||
- [ ] Create migration script
|
||||
- [ ] Test with demo photos
|
||||
|
||||
### Technical Notes
|
||||
- Encoding size: 128 → 512 dimensions
|
||||
- Similarity metric: Euclidean → Cosine
|
||||
- Location format: tuple → dict
|
||||
- Tolerance adjustment: 0.6 → 0.4
|
||||
|
||||
---
|
||||
|
||||
## Future Topics
|
||||
- Web interface design
|
||||
- Cloud storage integration
|
||||
- Performance optimization
|
||||
- Multi-user support
|
||||
- Mobile app development
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
# Phase 1 Quick Start Guide
|
||||
|
||||
## What Was Done
|
||||
|
||||
Phase 1: Database Schema Updates ✅ COMPLETE
|
||||
|
||||
All database tables and methods updated to support DeepFace:
|
||||
- New columns for detector backend and model name
|
||||
- Support for 512-dimensional encodings (ArcFace)
|
||||
- Enhanced face confidence tracking
|
||||
- Migration script ready to use
|
||||
|
||||
## Quick Commands
|
||||
|
||||
### Run Tests
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 tests/test_phase1_schema.py
|
||||
```
|
||||
|
||||
### Migrate Existing Database (⚠️ DELETES ALL DATA)
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 scripts/migrate_to_deepface.py
|
||||
```
|
||||
|
||||
### Install New Dependencies (for Phase 2+)
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. **requirements.txt** - DeepFace dependencies
|
||||
2. **src/core/config.py** - DeepFace configuration
|
||||
3. **src/core/database.py** - Schema + method updates
|
||||
|
||||
## Files Created
|
||||
|
||||
1. **scripts/migrate_to_deepface.py** - Migration script
|
||||
2. **tests/test_phase1_schema.py** - Test suite
|
||||
3. **PHASE1_COMPLETE.md** - Full documentation
|
||||
|
||||
## Next Steps
|
||||
|
||||
Ready to proceed to **Phase 2** or **Phase 3**:
|
||||
|
||||
### Phase 2: Configuration Updates
|
||||
- Add TensorFlow suppression to entry points
|
||||
- Update GUI with detector/model selection
|
||||
|
||||
### Phase 3: Core Face Processing
|
||||
- Replace face_recognition with DeepFace
|
||||
- Update process_faces() method
|
||||
- Implement cosine similarity
|
||||
|
||||
## Quick Verification
|
||||
|
||||
```bash
|
||||
# Check schema has new columns
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 -c "
|
||||
from src.core.database import DatabaseManager
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(suffix='.db') as tmp:
|
||||
db = DatabaseManager(tmp.name, verbose=0)
|
||||
print('✅ Database initialized with DeepFace schema')
|
||||
"
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
```
|
||||
Tests passed: 4/4
|
||||
✅ PASS: Schema Columns
|
||||
✅ PASS: add_face() Method
|
||||
✅ PASS: add_person_encoding() Method
|
||||
✅ PASS: Config Constants
|
||||
```
|
||||
|
||||
All systems ready for DeepFace implementation!
|
||||
|
||||
|
||||
@@ -1,132 +0,0 @@
|
||||
# Phase 2 Quick Start Guide
|
||||
|
||||
## What Was Done
|
||||
|
||||
Phase 2: Configuration Updates ✅ COMPLETE
|
||||
|
||||
Added GUI controls for DeepFace settings and updated FaceProcessor to accept them.
|
||||
|
||||
## Quick Commands
|
||||
|
||||
### Run Tests
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 tests/test_phase2_config.py
|
||||
```
|
||||
|
||||
Expected: **5/5 tests passing**
|
||||
|
||||
### Test GUI (Visual Check)
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 run_dashboard.py
|
||||
```
|
||||
|
||||
Then:
|
||||
1. Click "🔍 Process" button
|
||||
2. Look for "DeepFace Settings" section
|
||||
3. Verify two dropdowns:
|
||||
- Face Detector: [retinaface ▼]
|
||||
- Recognition Model: [ArcFace ▼]
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. **run_dashboard.py** - Callback passes detector/model
|
||||
2. **src/gui/dashboard_gui.py** - GUI controls added
|
||||
3. **src/photo_tagger.py** - TF suppression
|
||||
4. **src/core/face_processing.py** - Accepts detector/model params
|
||||
|
||||
## Files Created
|
||||
|
||||
1. **tests/test_phase2_config.py** - 5 tests
|
||||
2. **PHASE2_COMPLETE.md** - Full documentation
|
||||
|
||||
## What Changed
|
||||
|
||||
### Before Phase 2:
|
||||
```python
|
||||
processor = FaceProcessor(db_manager)
|
||||
```
|
||||
|
||||
### After Phase 2:
|
||||
```python
|
||||
processor = FaceProcessor(db_manager,
|
||||
detector_backend='retinaface',
|
||||
model_name='ArcFace')
|
||||
```
|
||||
|
||||
### GUI Process Panel:
|
||||
Now includes DeepFace Settings section with:
|
||||
- Detector selection dropdown (4 options)
|
||||
- Model selection dropdown (4 options)
|
||||
- Help text for each option
|
||||
|
||||
## Test Results
|
||||
|
||||
```
|
||||
✅ PASS: TensorFlow Suppression
|
||||
✅ PASS: FaceProcessor Initialization
|
||||
✅ PASS: Config Imports
|
||||
✅ PASS: Entry Point Imports
|
||||
✅ PASS: GUI Config Constants
|
||||
|
||||
Tests passed: 5/5
|
||||
```
|
||||
|
||||
## Available Options
|
||||
|
||||
### Detectors:
|
||||
- retinaface (default, best accuracy)
|
||||
- mtcnn
|
||||
- opencv
|
||||
- ssd
|
||||
|
||||
### Models:
|
||||
- ArcFace (default, 512-dim, best accuracy)
|
||||
- Facenet (128-dim)
|
||||
- Facenet512 (512-dim)
|
||||
- VGG-Face (2622-dim)
|
||||
|
||||
## Important Note
|
||||
|
||||
⚠️ **Phase 2 adds UI/config only**
|
||||
|
||||
The GUI captures settings, but actual DeepFace processing happens in **Phase 3**. Currently still using face_recognition for processing.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Ready for **Phase 3: Core Face Processing**
|
||||
- Replace face_recognition with DeepFace
|
||||
- Implement actual detector/model usage
|
||||
- Update face location handling
|
||||
- Implement cosine similarity
|
||||
|
||||
## Quick Verification
|
||||
|
||||
```bash
|
||||
# Test FaceProcessor accepts params
|
||||
cd /home/ladmin/Code/punimtag
|
||||
source venv/bin/activate
|
||||
python3 -c "
|
||||
from src.core.database import DatabaseManager
|
||||
from src.core.face_processing import FaceProcessor
|
||||
db = DatabaseManager(':memory:', verbose=0)
|
||||
p = FaceProcessor(db, verbose=0, detector_backend='mtcnn', model_name='Facenet')
|
||||
print(f'Detector: {p.detector_backend}')
|
||||
print(f'Model: {p.model_name}')
|
||||
print('✅ Phase 2 working!')
|
||||
"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Detector: mtcnn
|
||||
Model: Facenet
|
||||
✅ Phase 2 working!
|
||||
```
|
||||
|
||||
All systems ready for Phase 3!
|
||||
|
||||
|
||||
@@ -1,43 +0,0 @@
|
||||
# PunimTag - Project Overview
|
||||
|
||||
## Mission Statement
|
||||
PunimTag is a desktop photo management application that leverages facial recognition AI to help users organize, tag, and search their photo collections efficiently.
|
||||
|
||||
## Core Capabilities
|
||||
- Automated face detection and recognition
|
||||
- Person identification and management
|
||||
- Custom tagging system
|
||||
- Advanced search functionality
|
||||
- Batch processing
|
||||
|
||||
## Current Status
|
||||
- **Version**: 1.0 (Development)
|
||||
- **Stage**: Active Development
|
||||
- **Next Major Feature**: DeepFace Migration
|
||||
|
||||
## Key Technologies
|
||||
- Python 3.12+
|
||||
- Tkinter (GUI)
|
||||
- SQLite (Database)
|
||||
- face_recognition (Current - to be replaced)
|
||||
- DeepFace (Planned migration)
|
||||
|
||||
## Project Goals
|
||||
1. Make photo organization effortless
|
||||
2. Provide accurate face recognition
|
||||
3. Enable powerful search capabilities
|
||||
4. Maintain user privacy (local-only by default)
|
||||
5. Scale to large photo collections (50K+ photos)
|
||||
|
||||
## Success Metrics
|
||||
- Face recognition accuracy > 95%
|
||||
- Process 1000+ photos per hour
|
||||
- Search response time < 1 second
|
||||
- Zero data loss
|
||||
- User-friendly interface
|
||||
|
||||
## Links
|
||||
- Architecture: `docs/ARCHITECTURE.md`
|
||||
- Main README: `docs/README.md`
|
||||
- Demo Guide: `docs/DEMO.md`
|
||||
|
||||
@@ -1,342 +0,0 @@
|
||||
# Project Restructure Migration Guide
|
||||
|
||||
## Overview
|
||||
The project has been restructured to follow Python best practices with a clean separation of concerns.
|
||||
|
||||
---
|
||||
|
||||
## Directory Changes
|
||||
|
||||
### Before → After
|
||||
|
||||
```
|
||||
Root Directory Files → Organized Structure
|
||||
```
|
||||
|
||||
| Old Location | New Location | Type |
|
||||
|-------------|--------------|------|
|
||||
| `config.py` | `src/core/config.py` | Core |
|
||||
| `database.py` | `src/core/database.py` | Core |
|
||||
| `face_processing.py` | `src/core/face_processing.py` | Core |
|
||||
| `photo_management.py` | `src/core/photo_management.py` | Core |
|
||||
| `tag_management.py` | `src/core/tag_management.py` | Core |
|
||||
| `search_stats.py` | `src/core/search_stats.py` | Core |
|
||||
| `dashboard_gui.py` | `src/gui/dashboard_gui.py` | GUI |
|
||||
| `gui_core.py` | `src/gui/gui_core.py` | GUI |
|
||||
| `identify_panel.py` | `src/gui/identify_panel.py` | GUI |
|
||||
| `auto_match_panel.py` | `src/gui/auto_match_panel.py` | GUI |
|
||||
| `modify_panel.py` | `src/gui/modify_panel.py` | GUI |
|
||||
| `tag_manager_panel.py` | `src/gui/tag_manager_panel.py` | GUI |
|
||||
| `path_utils.py` | `src/utils/path_utils.py` | Utils |
|
||||
| `photo_tagger.py` | `src/photo_tagger.py` | Entry |
|
||||
| `test_*.py` | `tests/test_*.py` | Tests |
|
||||
| `README.md` | `docs/README.md` | Docs |
|
||||
| `ARCHITECTURE.md` | `docs/ARCHITECTURE.md` | Docs |
|
||||
|
||||
---
|
||||
|
||||
## Import Path Changes
|
||||
|
||||
### Core Modules
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
from config import DEFAULT_DB_PATH
|
||||
from database import DatabaseManager
|
||||
from face_processing import FaceProcessor
|
||||
from photo_management import PhotoManager
|
||||
from tag_management import TagManager
|
||||
from search_stats import SearchStats
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
from src.core.config import DEFAULT_DB_PATH
|
||||
from src.core.database import DatabaseManager
|
||||
from src.core.face_processing import FaceProcessor
|
||||
from src.core.photo_management import PhotoManager
|
||||
from src.core.tag_management import TagManager
|
||||
from src.core.search_stats import SearchStats
|
||||
```
|
||||
|
||||
### GUI Modules
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
from gui_core import GUICore
|
||||
from identify_panel import IdentifyPanel
|
||||
from auto_match_panel import AutoMatchPanel
|
||||
from modify_panel import ModifyPanel
|
||||
from tag_manager_panel import TagManagerPanel
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
from src.gui.gui_core import GUICore
|
||||
from src.gui.identify_panel import IdentifyPanel
|
||||
from src.gui.auto_match_panel import AutoMatchPanel
|
||||
from src.gui.modify_panel import ModifyPanel
|
||||
from src.gui.tag_manager_panel import TagManagerPanel
|
||||
```
|
||||
|
||||
### Utility Modules
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
from path_utils import normalize_path, validate_path_exists
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
from src.utils.path_utils import normalize_path, validate_path_exists
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Files Requiring Import Updates
|
||||
|
||||
### Priority 1 - Core Files
|
||||
- [ ] `src/core/face_processing.py`
|
||||
- [ ] `src/core/photo_management.py`
|
||||
- [ ] `src/core/tag_management.py`
|
||||
- [ ] `src/core/search_stats.py`
|
||||
- [ ] `src/core/database.py`
|
||||
|
||||
### Priority 2 - GUI Files
|
||||
- [ ] `src/gui/dashboard_gui.py`
|
||||
- [ ] `src/gui/identify_panel.py`
|
||||
- [ ] `src/gui/auto_match_panel.py`
|
||||
- [ ] `src/gui/modify_panel.py`
|
||||
- [ ] `src/gui/tag_manager_panel.py`
|
||||
- [ ] `src/gui/gui_core.py`
|
||||
|
||||
### Priority 3 - Entry Points
|
||||
- [ ] `src/photo_tagger.py`
|
||||
- [ ] `src/setup.py`
|
||||
|
||||
### Priority 4 - Tests
|
||||
- [ ] `tests/test_deepface_gui.py`
|
||||
- [ ] `tests/test_face_recognition.py`
|
||||
- [ ] `tests/test_simple_gui.py`
|
||||
|
||||
---
|
||||
|
||||
## Search & Replace Patterns
|
||||
|
||||
Use these patterns to update imports systematically:
|
||||
|
||||
### Pattern 1: Core imports
|
||||
```bash
|
||||
# Find
|
||||
from config import
|
||||
from database import
|
||||
from face_processing import
|
||||
from photo_management import
|
||||
from tag_management import
|
||||
from search_stats import
|
||||
|
||||
# Replace with
|
||||
from src.core.config import
|
||||
from src.core.database import
|
||||
from src.core.face_processing import
|
||||
from src.core.photo_management import
|
||||
from src.core.tag_management import
|
||||
from src.core.search_stats import
|
||||
```
|
||||
|
||||
### Pattern 2: GUI imports
|
||||
```bash
|
||||
# Find
|
||||
from gui_core import
|
||||
from identify_panel import
|
||||
from auto_match_panel import
|
||||
from modify_panel import
|
||||
from tag_manager_panel import
|
||||
|
||||
# Replace with
|
||||
from src.gui.gui_core import
|
||||
from src.gui.identify_panel import
|
||||
from src.gui.auto_match_panel import
|
||||
from src.gui.modify_panel import
|
||||
from src.gui.tag_manager_panel import
|
||||
```
|
||||
|
||||
### Pattern 3: Utils imports
|
||||
```bash
|
||||
# Find
|
||||
from path_utils import
|
||||
|
||||
# Replace with
|
||||
from src.utils.path_utils import
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Running the Application After Restructure
|
||||
|
||||
### GUI Dashboard
|
||||
```bash
|
||||
# Old
|
||||
python dashboard_gui.py
|
||||
|
||||
# New
|
||||
python src/gui/dashboard_gui.py
|
||||
# OR
|
||||
python -m src.gui.dashboard_gui
|
||||
```
|
||||
|
||||
### CLI Tool
|
||||
```bash
|
||||
# Old
|
||||
python photo_tagger.py
|
||||
|
||||
# New
|
||||
python src/photo_tagger.py
|
||||
# OR
|
||||
python -m src.photo_tagger
|
||||
```
|
||||
|
||||
### Tests
|
||||
```bash
|
||||
# Old
|
||||
python test_deepface_gui.py
|
||||
|
||||
# New
|
||||
python tests/test_deepface_gui.py
|
||||
# OR
|
||||
python -m pytest tests/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Steps
|
||||
|
||||
### 1. Check Import Errors
|
||||
```bash
|
||||
cd /home/ladmin/Code/punimtag
|
||||
python -c "from src.core import DatabaseManager; print('Core imports OK')"
|
||||
python -c "from src.gui import GUICore; print('GUI imports OK')"
|
||||
python -c "from src.utils import normalize_path; print('Utils imports OK')"
|
||||
```
|
||||
|
||||
### 2. Test Each Module
|
||||
```bash
|
||||
# Test core modules
|
||||
python -c "from src.core.database import DatabaseManager; db = DatabaseManager(':memory:'); print('Database OK')"
|
||||
|
||||
# Test GUI modules (may need display)
|
||||
python -c "from src.gui.gui_core import GUICore; print('GUI Core OK')"
|
||||
```
|
||||
|
||||
### 3. Run Application
|
||||
```bash
|
||||
# Try to launch dashboard
|
||||
python src/gui/dashboard_gui.py
|
||||
```
|
||||
|
||||
### 4. Run Tests
|
||||
```bash
|
||||
# Run test suite
|
||||
python -m pytest tests/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Issue 1: ModuleNotFoundError
|
||||
```
|
||||
ModuleNotFoundError: No module named 'config'
|
||||
```
|
||||
|
||||
**Solution:** Update import from `from config import` to `from src.core.config import`
|
||||
|
||||
### Issue 2: Relative Import Error
|
||||
```
|
||||
ImportError: attempted relative import with no known parent package
|
||||
```
|
||||
|
||||
**Solution:** Use absolute imports with `src.` prefix
|
||||
|
||||
### Issue 3: Circular Import
|
||||
```
|
||||
ImportError: cannot import name 'X' from partially initialized module
|
||||
```
|
||||
|
||||
**Solution:** Check for circular dependencies, may need to refactor
|
||||
|
||||
### Issue 4: sys.path Issues
|
||||
If imports still fail, add to top of file:
|
||||
```python
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add project root to path
|
||||
project_root = Path(__file__).parent.parent # Adjust based on file location
|
||||
sys.path.insert(0, str(project_root))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise, you can rollback:
|
||||
|
||||
```bash
|
||||
# Revert git changes
|
||||
git checkout HEAD -- .
|
||||
|
||||
# Or manually move files back
|
||||
mv src/core/*.py .
|
||||
mv src/gui/*.py .
|
||||
mv src/utils/*.py .
|
||||
mv tests/*.py .
|
||||
mv docs/*.md .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Benefits of New Structure
|
||||
|
||||
✅ **Better Organization**: Clear separation of concerns
|
||||
✅ **Easier Navigation**: Files grouped by function
|
||||
✅ **Professional**: Follows Python community standards
|
||||
✅ **Scalable**: Easy to add new modules
|
||||
✅ **Testable**: Tests separate from source
|
||||
✅ **Maintainable**: Clear dependencies
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Directory structure created
|
||||
2. ✅ Files moved to new locations
|
||||
3. ✅ __init__.py files created
|
||||
4. ✅ Documentation updated
|
||||
5. ⏳ Update import statements (NEXT)
|
||||
6. ⏳ Test all functionality
|
||||
7. ⏳ Update scripts and launchers
|
||||
8. ⏳ Commit changes
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
After updating imports, verify:
|
||||
|
||||
- [ ] Dashboard GUI launches
|
||||
- [ ] Can scan for photos
|
||||
- [ ] Face processing works
|
||||
- [ ] Face identification works
|
||||
- [ ] Auto-matching works
|
||||
- [ ] Tag management works
|
||||
- [ ] Search functionality works
|
||||
- [ ] Database operations work
|
||||
- [ ] All tests pass
|
||||
|
||||
---
|
||||
|
||||
**Status**: Files moved, imports need updating
|
||||
**Last Updated**: 2025-10-15
|
||||
**Next Action**: Update import statements in all files
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
# Task List
|
||||
|
||||
> **SoT for current work:** open [Gitea issues](https://git.levkin.ca/ilia/punimtag/issues) and [docs/FIXES_2026-07-09.md](../docs/FIXES_2026-07-09.md).
|
||||
> This file is historical; do not treat unchecked DeepFace items as the active backlog.
|
||||
|
||||
## Active (2026-07)
|
||||
|
||||
Blocked on client only:
|
||||
|
||||
- **#44** JRCC watermark — need SVG/PNG + placement
|
||||
- **#43** Tag without account — need product rules
|
||||
|
||||
Ops (no ticket): production Next.js viewer on LXC 9101 (`next start`); keep DB host `.179`
|
||||
|
||||
## Historical / backlog ideas
|
||||
|
||||
### Performance
|
||||
- [ ] Pre-generate thumbnails at Process time
|
||||
- [ ] Lazy-load face metadata on home page SSR
|
||||
- [ ] Run viewer with `next start` (not `next dev`)
|
||||
|
||||
### Features
|
||||
- [ ] Guest “suggest name” (#43) after product rules
|
||||
- [ ] JRCC watermark swap (#44) when asset arrives
|
||||
|
||||
## Completed (selected)
|
||||
|
||||
- [x] 2026-07-09 open-issue triage (#45/#46/#47 + #21/#26/#30/#31/#33) — PRs #51/#52
|
||||
- [x] Shared Postgres retarget `.181` → `.179` (ansible #81)
|
||||
- [x] Kuma + Beszel for PunimTag (ansible #82)
|
||||
- [x] Docs Postgres `.179` + next-steps (punimtag #53)
|
||||
- [x] Grid thumbnail + disk cache
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"hash": "ffc8c959",
|
||||
"configHash": "afa43417",
|
||||
"lockfileHash": "e3b0c442",
|
||||
"browserHash": "1131262f",
|
||||
"optimized": {},
|
||||
"chunks": {}
|
||||
}
|
||||
@@ -1,3 +0,0 @@
|
||||
{
|
||||
"type": "module"
|
||||
}
|
||||
Reference in New Issue
Block a user