Update project documentation and structure; enhance README, finalize project reorganization, and improve testing standards.
This commit is contained in:
@@ -1,220 +0,0 @@
|
||||
# PunimTag Backend Development Status
|
||||
|
||||
## ✅ Completed Features
|
||||
|
||||
### 1. Configuration System (`config.py`)
|
||||
|
||||
- **Jewish Organization Specific Settings**: Pre-configured with Jewish holidays, events, and locations
|
||||
- **Face Recognition Configuration**: Adjustable thresholds, clustering parameters
|
||||
- **Auto-tagging Settings**: Toggle-able features with confidence thresholds
|
||||
- **Processing Configuration**: Batch sizes, worker settings, file format support
|
||||
- **Persistent Settings**: JSON-based configuration file with load/save functionality
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- 30+ predefined Jewish event tags (shabbat, wedding, bar_mitzvah, chanukah, etc.)
|
||||
- 15+ location tags (synagogue, sanctuary, sukkah, israel, etc.)
|
||||
- Configurable face recognition thresholds
|
||||
- Auto-tagging enable/disable controls
|
||||
|
||||
### 2. Enhanced Face Recognition (`punimtag.py` + `punimtag_simple.py`)
|
||||
|
||||
- **Face Quality Scoring**: Evaluates face size and encoding variance
|
||||
- **Advanced Face Clustering**: DBSCAN-based clustering for grouping unknown faces
|
||||
- **Confidence-based Recognition**: Automatic vs manual identification based on thresholds
|
||||
- **Multiple Face Angles**: Support for storing multiple encodings per person
|
||||
|
||||
**Key Features:**
|
||||
|
||||
- Face quality assessment for better training data
|
||||
- Cluster unknown faces by similarity
|
||||
- Sort by most frequently photographed people
|
||||
- Face verification tools for double-checking identifications
|
||||
|
||||
### 3. Comprehensive Database Schema
|
||||
|
||||
- **Images Table**: Full metadata (GPS, camera info, dimensions, EXIF data)
|
||||
- **People Table**: Named individuals with creation timestamps
|
||||
- **Faces Table**: Precise face locations, encodings, confidence scores
|
||||
- **Tags Table**: Categorized tagging system
|
||||
- **Image-Tags Relationship**: Many-to-many tagging support
|
||||
|
||||
**Performance Optimizations:**
|
||||
|
||||
- Database indexes on key relationships
|
||||
- Efficient foreign key constraints
|
||||
- Optimized query structures
|
||||
|
||||
### 4. Enhanced EXIF Metadata Extraction
|
||||
|
||||
- **GPS Coordinates**: Latitude/longitude extraction with hemisphere handling
|
||||
- **Camera Information**: Make, model, settings
|
||||
- **Date/Time**: Photo taken timestamp
|
||||
- **Error Handling**: Graceful fallbacks for missing data (defaults to "N/A")
|
||||
|
||||
### 5. Advanced Search Capabilities
|
||||
|
||||
- **Multi-criteria Search**: People + tags + dates + location + camera
|
||||
- **Complex Queries**: Support for min_people requirements
|
||||
- **Geographic Filtering**: Bounding box searches with GPS coordinates
|
||||
- **Date Range Filtering**: From/to date searches
|
||||
- **Result Limiting**: Pagination support
|
||||
|
||||
### 6. Batch Processing for Large Collections
|
||||
|
||||
- **Configurable Batch Sizes**: Process 5-10k images efficiently
|
||||
- **Skip Processed Images**: Incremental processing for new photos
|
||||
- **Progress Tracking**: Real-time status updates
|
||||
- **Error Handling**: Continue processing despite individual failures
|
||||
|
||||
### 7. Face Management Tools
|
||||
|
||||
- **Cluster Assignment**: Assign entire face clusters to people
|
||||
- **Face Verification**: Review all faces assigned to a person
|
||||
- **Incorrect Assignment Removal**: Fix misidentifications
|
||||
- **Most Common Faces**: Sort by frequency (most photographed people)
|
||||
|
||||
### 8. Jewish Organization Tag Categories
|
||||
|
||||
```
|
||||
Event Tags: shabbat, wedding, bar_mitzvah, bat_mitzvah, brit_milah,
|
||||
baby_naming, shiva, yahrzeit, rosh_hashanah, yom_kippur,
|
||||
sukkot, chanukah, purim, passover, etc.
|
||||
|
||||
Location Tags: synagogue, sanctuary, social_hall, classroom, library,
|
||||
kitchen, sukkah, israel, jerusalem, etc.
|
||||
|
||||
Activity Tags: praying, studying, celebrating, socializing, ceremony,
|
||||
performance, eating, etc.
|
||||
```
|
||||
|
||||
## 🧪 Testing Status
|
||||
|
||||
### Core Functionality Tests ✅
|
||||
|
||||
- ✅ Database creation and schema validation
|
||||
- ✅ Configuration system load/save
|
||||
- ✅ People and tag management
|
||||
- ✅ Basic search functionality
|
||||
- ✅ EXIF metadata extraction
|
||||
- ✅ Face encoding storage/retrieval
|
||||
|
||||
### Simplified Backend (`punimtag_simple.py`) ✅
|
||||
|
||||
- ✅ Working without sklearn dependencies
|
||||
- ✅ Core face recognition functionality
|
||||
- ✅ Database operations validated
|
||||
- ✅ Tag and people management working
|
||||
- ✅ Search queries functional
|
||||
|
||||
### Performance Tests 📋 (Ready for testing)
|
||||
|
||||
- **Created but not run**: 1000+ face clustering test
|
||||
- **Created but not run**: Large dataset search performance
|
||||
- **Created but not run**: Batch processing with 5-10k images
|
||||
|
||||
## 🔧 Technical Implementation
|
||||
|
||||
### Dependencies Status
|
||||
|
||||
| Package | Status | Purpose |
|
||||
| ---------------- | ----------- | ------------------------------- |
|
||||
| face_recognition | ✅ Working | Core face detection/recognition |
|
||||
| numpy | ✅ Working | Array operations |
|
||||
| Pillow | ✅ Working | Image processing and EXIF |
|
||||
| sqlite3 | ✅ Working | Database operations |
|
||||
| scikit-learn | ⚠️ Optional | Advanced clustering (DBSCAN) |
|
||||
| opencv-python | ⚠️ Optional | GUI face viewer |
|
||||
|
||||
### Performance Optimizations Implemented
|
||||
|
||||
1. **Database Indexes**: On faces(person_id), faces(image_id), image_tags
|
||||
2. **Batch Processing**: Configurable batch sizes (default: 100)
|
||||
3. **Incremental Processing**: Skip already processed images
|
||||
4. **Efficient Queries**: Optimized JOIN operations for search
|
||||
5. **Memory Management**: Process images one at a time
|
||||
|
||||
### Error Handling
|
||||
|
||||
- ✅ Graceful EXIF extraction failures
|
||||
- ✅ Missing file handling
|
||||
- ✅ Database constraint violations
|
||||
- ✅ Face detection errors
|
||||
- ✅ Configuration file corruption
|
||||
|
||||
## 📊 Current Database Schema
|
||||
|
||||
```sql
|
||||
-- Core tables with relationships
|
||||
images (id, path, filename, date_taken, latitude, longitude, camera_make, ...)
|
||||
people (id, name, created_at)
|
||||
faces (id, image_id, person_id, top, right, bottom, left, encoding, confidence, ...)
|
||||
tags (id, name, category, created_at)
|
||||
image_tags (image_id, tag_id, created_at)
|
||||
|
||||
-- Indexes for performance
|
||||
idx_faces_person, idx_faces_image, idx_image_tags_image, idx_image_tags_tag
|
||||
```
|
||||
|
||||
## 🎯 Backend Readiness Assessment
|
||||
|
||||
### ✅ Ready for GUI Development
|
||||
|
||||
The backend is **production-ready** for GUI development with the following capabilities:
|
||||
|
||||
1. **Face Recognition Pipeline**: Complete face detection → encoding → identification
|
||||
2. **Database Operations**: All CRUD operations for images, people, faces, tags
|
||||
3. **Search Engine**: Complex multi-criteria search functionality
|
||||
4. **Jewish Org Features**: Pre-configured with relevant tags and categories
|
||||
5. **Configuration System**: User-configurable settings
|
||||
6. **Performance**: Optimized for 5-10k image collections
|
||||
|
||||
### 🔄 Next Steps for GUI
|
||||
|
||||
1. **Face Clustering Interface**: Visual display of clustered unknown faces
|
||||
2. **Interactive Identification**: Click-to-identify unknown faces
|
||||
3. **Search Interface**: Form-based search with filters
|
||||
4. **Tag Management**: Visual tag assignment and management
|
||||
5. **Statistics Dashboard**: Charts and graphs of collection data
|
||||
6. **Face Verification**: Review and correct face assignments
|
||||
|
||||
### 📋 Optional Enhancements (Post-GUI)
|
||||
|
||||
- [ ] Hebrew calendar integration for automatic holiday tagging
|
||||
- [ ] Advanced clustering with scikit-learn when available
|
||||
- [ ] Thumbnail generation for faster GUI loading
|
||||
- [ ] Export functionality (albums, tagged collections)
|
||||
- [ ] Import from other photo management systems
|
||||
|
||||
## 🚀 Deployment Notes
|
||||
|
||||
### For Production Use:
|
||||
|
||||
1. **Install Core Dependencies**: `pip install face_recognition pillow numpy`
|
||||
2. **Optional GUI Dependencies**: `pip install opencv-python scikit-learn`
|
||||
3. **Create Configuration**: Run `python config.py` to generate default config
|
||||
4. **Initialize Database**: Run `python punimtag_simple.py` to create tables
|
||||
5. **Add Photos**: Place images in `photos/` directory
|
||||
6. **Process Images**: Run the main processing script
|
||||
|
||||
### Performance Recommendations:
|
||||
|
||||
- **For 1k-5k images**: Use default batch size (100)
|
||||
- **For 5k-10k images**: Increase batch size to 200-500
|
||||
- **For 10k+ images**: Consider database optimization and larger batches
|
||||
|
||||
## 🏁 Conclusion
|
||||
|
||||
**The PunimTag backend is fully functional and ready for GUI development.**
|
||||
|
||||
All core requirements have been implemented:
|
||||
|
||||
- ✅ Face recognition with identification
|
||||
- ✅ Complex search capabilities
|
||||
- ✅ Jewish organization specific features
|
||||
- ✅ Comprehensive tagging system
|
||||
- ✅ CRUD interface for all entities
|
||||
- ✅ Performance optimizations for large collections
|
||||
- ✅ Configuration system with auto-tagging controls
|
||||
|
||||
The system is tested, documented, and ready to support a GUI interface that will provide all the functionality requested in the original requirements.
|
||||
-194
@@ -1,194 +0,0 @@
|
||||
# PunimTag - Future Enhancement Ideas
|
||||
|
||||
## 🎯 Core Improvements
|
||||
|
||||
### 1. Enhanced Face Recognition
|
||||
|
||||
- **Multi-angle face training**: Store multiple angles of the same person for better recognition
|
||||
- **Face quality scoring**: Rate face image quality and use only high-quality samples for training
|
||||
- **Age progression handling**: Account for aging when matching faces across time periods
|
||||
- **Expression normalization**: Better handle different facial expressions
|
||||
- **Confidence thresholds**: User-configurable confidence levels for automatic vs manual identification
|
||||
|
||||
### 2. Performance Optimizations
|
||||
|
||||
- **Incremental processing**: Only process new/modified images
|
||||
- **Parallel processing**: Use multiprocessing for faster batch operations
|
||||
- **Face encoding cache**: Cache encodings to avoid recomputation
|
||||
- **Thumbnail generation**: Create and store thumbnails for faster UI display
|
||||
- **Database indexing**: Optimize queries with better indexes and query plans
|
||||
|
||||
### 3. Advanced Tagging
|
||||
|
||||
- **AI-powered auto-tagging**:
|
||||
- Scene detection (beach, mountain, city, etc.)
|
||||
- Object detection (cars, pets, food, etc.)
|
||||
- Activity recognition (eating, sports, working)
|
||||
- Emotion detection (happy, sad, surprised)
|
||||
- Indoor/outdoor classification
|
||||
- **Tag hierarchies**: Parent-child tag relationships (e.g., "vacation" → "beach vacation")
|
||||
- **Smart tag suggestions**: Based on similar images and user patterns
|
||||
- **Batch tag operations**: Apply/remove tags from multiple images efficiently
|
||||
|
||||
## 🌐 Web Interface
|
||||
|
||||
### 1. Modern Web UI
|
||||
|
||||
- **React/Vue.js frontend** with responsive design
|
||||
- **Gallery view** with filtering and sorting
|
||||
- **Face clustering visualization**: Interactive graph showing face relationships
|
||||
- **Drag-and-drop uploads**: Easy image addition
|
||||
- **Real-time updates**: WebSocket for live processing status
|
||||
|
||||
### 2. Features
|
||||
|
||||
- **Interactive face identification**: Click faces to identify them
|
||||
- **Tag cloud**: Visual representation of tag frequency
|
||||
- **Timeline view**: Browse photos chronologically
|
||||
- **Map view**: Show photos on a map using GPS data
|
||||
- **Slideshow mode**: With face and tag filters
|
||||
|
||||
## 🔗 Integrations
|
||||
|
||||
### 1. Cloud Storage
|
||||
|
||||
- **Google Photos sync**: Import/export with Google Photos
|
||||
- **iCloud integration**: Sync with Apple Photos
|
||||
- **Dropbox/OneDrive**: Monitor folders for new images
|
||||
- **S3 compatibility**: Store images in cloud storage
|
||||
|
||||
### 2. Social Media
|
||||
|
||||
- **Facebook integration**: Import tagged faces (with permission)
|
||||
- **Instagram import**: Bring in photos with hashtags as tags
|
||||
- **Privacy-aware sharing**: Share photos only with people in them
|
||||
|
||||
## 🛡️ Privacy & Security
|
||||
|
||||
### 1. Privacy Features
|
||||
|
||||
- **Face anonymization**: Blur unidentified faces on export
|
||||
- **Consent management**: Track consent for face recognition
|
||||
- **GDPR compliance**: Right to be forgotten, data export
|
||||
- **Encryption**: Client-side encryption option
|
||||
- **Access controls**: User/group permissions
|
||||
|
||||
### 2. Backup & Recovery
|
||||
|
||||
- **Automated backups**: Scheduled database and image backups
|
||||
- **Version control**: Track changes to face identifications
|
||||
- **Disaster recovery**: Restore from backups easily
|
||||
- **Export formats**: Multiple export options (JSON, CSV, etc.)
|
||||
|
||||
## 🤖 AI Enhancements
|
||||
|
||||
### 1. Advanced ML Features
|
||||
|
||||
- **Face clustering improvements**: Use deep learning for better grouping
|
||||
- **Duplicate detection**: Find and manage similar photos
|
||||
- **Photo quality assessment**: Identify blurry/poor quality images
|
||||
- **Automatic album creation**: Group photos by events
|
||||
- **Style transfer**: Apply artistic filters based on tags
|
||||
|
||||
### 2. Natural Language Processing
|
||||
|
||||
- **Natural language search**: "Show me beach photos with John from last summer"
|
||||
- **Voice commands**: Control the app with voice
|
||||
- **Caption generation**: Auto-generate photo descriptions
|
||||
- **Story creation**: Generate photo stories/albums automatically
|
||||
|
||||
## 🔧 Developer Features
|
||||
|
||||
### 1. API & Extensions
|
||||
|
||||
- **RESTful API**: Full API for third-party integration
|
||||
- **GraphQL endpoint**: Flexible data querying
|
||||
- **Plugin system**: Allow custom extensions
|
||||
- **Webhook support**: Notify external systems of changes
|
||||
- **SDK development**: Python/JavaScript SDKs
|
||||
|
||||
### 2. Advanced Tools
|
||||
|
||||
- **Batch processing CLI**: Command-line tools for power users
|
||||
- **Migration tools**: Import from other photo management systems
|
||||
- **Analytics dashboard**: Usage statistics and insights
|
||||
- **Performance monitoring**: Track system performance
|
||||
|
||||
## 📊 Analytics & Insights
|
||||
|
||||
### 1. Photo Statistics
|
||||
|
||||
- **Face frequency**: Most photographed people
|
||||
- **Tag analytics**: Most used tags over time
|
||||
- **Location heatmap**: Where most photos are taken
|
||||
- **Time patterns**: When photos are typically taken
|
||||
- **Relationship graphs**: Visualize people connections
|
||||
|
||||
### 2. Personal Insights
|
||||
|
||||
- **Year in review**: Automated yearly summaries
|
||||
- **Memory reminders**: "On this day" features
|
||||
- **Growth tracking**: Watch children grow over time
|
||||
- **Event detection**: Automatically identify special events
|
||||
|
||||
## 🎨 Creative Features
|
||||
|
||||
### 1. Photo Enhancement
|
||||
|
||||
- **Automatic enhancement**: AI-powered photo improvement
|
||||
- **Red-eye removal**: Automatic detection and correction
|
||||
- **Background replacement**: Change photo backgrounds
|
||||
- **Face beautification**: Optional beauty filters
|
||||
|
||||
### 2. Creative Tools
|
||||
|
||||
- **Collage generation**: Auto-create collages by tags/people
|
||||
- **Photo books**: Design and export photo books
|
||||
- **Video generation**: Create videos from photo sets
|
||||
- **AR features**: View photos in augmented reality
|
||||
|
||||
## 🔮 Future Technologies
|
||||
|
||||
### 1. Emerging Tech
|
||||
|
||||
- **Blockchain**: Decentralized photo ownership proof
|
||||
- **IPFS storage**: Distributed photo storage
|
||||
- **Edge AI**: On-device processing for privacy
|
||||
- **5G optimization**: Fast mobile sync and processing
|
||||
|
||||
### 2. Experimental Features
|
||||
|
||||
- **3D face modeling**: Create 3D models from multiple photos
|
||||
- **Time-lapse generation**: Show aging/changes over time
|
||||
- **DeepFake detection**: Identify manipulated images
|
||||
- **Holographic displays**: Future display technology support
|
||||
|
||||
## 📋 Implementation Priority
|
||||
|
||||
### Phase 1 (Next 3 months)
|
||||
|
||||
1. Web UI basic implementation
|
||||
2. Performance optimizations
|
||||
3. Better error handling
|
||||
4. Basic auto-tagging
|
||||
|
||||
### Phase 2 (6 months)
|
||||
|
||||
1. Mobile PWA
|
||||
2. Cloud storage integration
|
||||
3. Advanced search
|
||||
4. API development
|
||||
|
||||
### Phase 3 (1 year)
|
||||
|
||||
1. AI enhancements
|
||||
2. Social integrations
|
||||
3. Analytics dashboard
|
||||
4. Plugin system
|
||||
|
||||
### Long-term (2+ years)
|
||||
|
||||
1. Native mobile apps
|
||||
2. Blockchain integration
|
||||
3. AR/VR features
|
||||
4. Advanced AI features
|
||||
@@ -1,283 +0,0 @@
|
||||
# PunimTag Testing Guide
|
||||
|
||||
## 🧪 Testing with Real Images
|
||||
|
||||
### Step 1: Prepare Your Test Images
|
||||
|
||||
1. **Create/Use Photos Directory**:
|
||||
|
||||
```bash
|
||||
mkdir -p photos
|
||||
```
|
||||
|
||||
2. **Add Test Images**:
|
||||
|
||||
- Copy 10-20 photos with faces to the `photos/` directory
|
||||
- Supported formats: `.jpg`, `.jpeg`, `.png`, `.bmp`, `.tiff`, `.gif`
|
||||
- For best results, use photos with clear, front-facing faces
|
||||
- Include photos with the same people for face recognition testing
|
||||
|
||||
3. **Organize by Subdirectories** (optional):
|
||||
```
|
||||
photos/
|
||||
├── events/
|
||||
│ ├── wedding_2023/
|
||||
│ └── bar_mitzvah/
|
||||
├── family/
|
||||
└── synagogue/
|
||||
```
|
||||
|
||||
### Step 2: Process Images
|
||||
|
||||
```bash
|
||||
# Process all images in photos directory
|
||||
python punimtag_simple.py
|
||||
```
|
||||
|
||||
This will:
|
||||
|
||||
- Scan all images in `photos/` directory (including subdirectories)
|
||||
- Extract EXIF metadata (GPS, camera info, dates)
|
||||
- Detect all faces and create encodings
|
||||
- Store everything in `punimtag_simple.db`
|
||||
|
||||
### Step 3: Inspect Results
|
||||
|
||||
```bash
|
||||
# Check what was processed
|
||||
python db_manager.py
|
||||
# Choose option 1 to inspect database
|
||||
```
|
||||
|
||||
### Step 4: Identify People (Interactive)
|
||||
|
||||
```bash
|
||||
# Use the CLI face identifier
|
||||
python interactive_identifier.py
|
||||
```
|
||||
|
||||
This will show you unidentified faces and let you name them.
|
||||
|
||||
### Step 5: Add Tags
|
||||
|
||||
```bash
|
||||
# Use the tag manager
|
||||
python tag_manager.py
|
||||
```
|
||||
|
||||
Add Jewish organization specific tags like:
|
||||
|
||||
- Events: `shabbat`, `wedding`, `bar_mitzvah`, `chanukah`
|
||||
- Locations: `synagogue`, `home`, `israel`
|
||||
- Activities: `praying`, `celebrating`, `studying`
|
||||
|
||||
## 🧹 Database Management
|
||||
|
||||
### Clean Database (Keep Schema)
|
||||
|
||||
```bash
|
||||
python db_manager.py
|
||||
# Choose option 2
|
||||
```
|
||||
|
||||
- Removes all data but keeps tables
|
||||
- Creates automatic backup first
|
||||
|
||||
### Delete Database Completely
|
||||
|
||||
```bash
|
||||
python db_manager.py
|
||||
# Choose option 3
|
||||
```
|
||||
|
||||
- Deletes entire database file
|
||||
- Creates automatic backup first
|
||||
|
||||
### Inspect Database
|
||||
|
||||
```bash
|
||||
python db_manager.py
|
||||
# Choose option 1
|
||||
```
|
||||
|
||||
Shows:
|
||||
|
||||
- Image/face/people counts
|
||||
- Top people by frequency
|
||||
- Most used tags
|
||||
- Database file size
|
||||
|
||||
## 🔍 Testing Search Functionality
|
||||
|
||||
### Basic Search Test
|
||||
|
||||
```python
|
||||
from punimtag_simple import SimplePunimTag
|
||||
|
||||
tagger = SimplePunimTag()
|
||||
|
||||
# Search by person
|
||||
results = tagger.simple_search(people=["Rabbi Cohen"])
|
||||
print(f"Found {len(results)} images with Rabbi Cohen")
|
||||
|
||||
# Search by tag
|
||||
results = tagger.simple_search(tags=["wedding"])
|
||||
print(f"Found {len(results)} wedding images")
|
||||
|
||||
# Combined search
|
||||
results = tagger.simple_search(
|
||||
people=["Sarah Goldberg"],
|
||||
tags=["shabbat"]
|
||||
)
|
||||
print(f"Found {len(results)} images of Sarah at Shabbat")
|
||||
|
||||
tagger.close()
|
||||
```
|
||||
|
||||
## 📊 Performance Testing
|
||||
|
||||
### Test with Different Collection Sizes
|
||||
|
||||
1. **Small Collection (10-50 images)**:
|
||||
|
||||
- Process time: ~1-5 minutes
|
||||
- Good for initial testing
|
||||
|
||||
2. **Medium Collection (100-500 images)**:
|
||||
|
||||
- Process time: ~10-30 minutes
|
||||
- Test face recognition accuracy
|
||||
|
||||
3. **Large Collection (1000+ images)**:
|
||||
- Process time: 1+ hours
|
||||
- Test batch processing and performance
|
||||
|
||||
### Monitor Performance
|
||||
|
||||
```python
|
||||
import time
|
||||
from punimtag_simple import SimplePunimTag
|
||||
|
||||
start_time = time.time()
|
||||
tagger = SimplePunimTag()
|
||||
processed = tagger.process_directory()
|
||||
end_time = time.time()
|
||||
|
||||
print(f"Processed {processed} images in {end_time - start_time:.2f} seconds")
|
||||
tagger.close()
|
||||
```
|
||||
|
||||
## 🎯 Testing Specific Features
|
||||
|
||||
### 1. Face Recognition Accuracy
|
||||
|
||||
1. Process images with same people
|
||||
2. Identify some faces manually
|
||||
3. Process new images with same people
|
||||
4. Check if they're automatically recognized
|
||||
|
||||
### 2. Jewish Organization Tags
|
||||
|
||||
```python
|
||||
from punimtag_simple import SimplePunimTag
|
||||
from config import get_config
|
||||
|
||||
config = get_config()
|
||||
event_tags = config.get_tag_suggestions('event')
|
||||
print("Available Jewish event tags:", event_tags[:10])
|
||||
```
|
||||
|
||||
### 3. EXIF Metadata Extraction
|
||||
|
||||
```python
|
||||
from punimtag_simple import SimplePunimTag
|
||||
|
||||
tagger = SimplePunimTag()
|
||||
metadata = tagger.extract_metadata("photos/your_image.jpg")
|
||||
print("Extracted metadata:", metadata)
|
||||
tagger.close()
|
||||
```
|
||||
|
||||
### 4. GPS Location Data
|
||||
|
||||
- Use photos taken with smartphones (usually have GPS)
|
||||
- Check if latitude/longitude are extracted
|
||||
- Test location-based searches
|
||||
|
||||
## 🐛 Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **"No faces detected"**:
|
||||
|
||||
- Check image quality
|
||||
- Ensure faces are clearly visible
|
||||
- Try different lighting conditions
|
||||
|
||||
2. **"EXIF data missing"**:
|
||||
|
||||
- Some images don't have EXIF data
|
||||
- System will default to "N/A"
|
||||
- This is normal behavior
|
||||
|
||||
3. **"Face recognition not working"**:
|
||||
|
||||
- Need multiple photos of same person
|
||||
- Faces should be front-facing and clear
|
||||
- Check confidence threshold in config
|
||||
|
||||
4. **"Processing is slow"**:
|
||||
- Normal for large collections
|
||||
- Adjust batch size in config
|
||||
- Consider using smaller test set first
|
||||
|
||||
### Debug Mode
|
||||
|
||||
```python
|
||||
# Add debug logging to see what's happening
|
||||
import logging
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
|
||||
from punimtag_simple import SimplePunimTag
|
||||
tagger = SimplePunimTag()
|
||||
# ... rest of your code
|
||||
```
|
||||
|
||||
## ✅ Validation Checklist
|
||||
|
||||
Before moving to GUI development, validate:
|
||||
|
||||
- [ ] Images are processing without errors
|
||||
- [ ] Faces are being detected correctly
|
||||
- [ ] EXIF metadata is being extracted
|
||||
- [ ] People can be identified and assigned
|
||||
- [ ] Tags can be added and searched
|
||||
- [ ] Database operations work smoothly
|
||||
- [ ] Search functionality returns expected results
|
||||
- [ ] Performance is acceptable for your collection size
|
||||
|
||||
## 🔄 Reset for Fresh Testing
|
||||
|
||||
```bash
|
||||
# Clean everything and start fresh
|
||||
python db_manager.py # Choose option 2 to clean
|
||||
rm -f punimtag_config.json # Reset config
|
||||
python config.py # Regenerate default config
|
||||
```
|
||||
|
||||
## 📝 Next Steps After Testing
|
||||
|
||||
Once testing is successful:
|
||||
|
||||
1. **GUI Development**: Create visual interface
|
||||
2. **Advanced Features**: Add clustering, verification tools
|
||||
3. **Performance Optimization**: Fine-tune for your specific needs
|
||||
|
||||
## 💡 Testing Tips
|
||||
|
||||
1. **Start Small**: Test with 10-20 images first
|
||||
2. **Use Clear Photos**: Better face detection results
|
||||
3. **Same People**: Include multiple photos of same people
|
||||
4. **Variety**: Test different scenarios (indoor/outdoor, events, etc.)
|
||||
5. **Monitor Progress**: Watch console output during processing
|
||||
6. **Backup Often**: Use database manager to create backups
|
||||
@@ -1,335 +0,0 @@
|
||||
# PunimTag API Standards
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the standards for designing and implementing API endpoints in PunimTag.
|
||||
|
||||
## Response Format
|
||||
|
||||
### Success Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
// Response data here
|
||||
},
|
||||
"message": "Optional success message"
|
||||
}
|
||||
```
|
||||
|
||||
### Error Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": false,
|
||||
"error": "Descriptive error message",
|
||||
"code": "ERROR_CODE_OPTIONAL"
|
||||
}
|
||||
```
|
||||
|
||||
### Paginated Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"data": {
|
||||
"items": [...],
|
||||
"pagination": {
|
||||
"page": 1,
|
||||
"per_page": 20,
|
||||
"total": 150,
|
||||
"pages": 8
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## HTTP Status Codes
|
||||
|
||||
### Success Codes
|
||||
|
||||
- **200 OK**: Request successful
|
||||
- **201 Created**: Resource created successfully
|
||||
- **204 No Content**: Request successful, no content to return
|
||||
|
||||
### Client Error Codes
|
||||
|
||||
- **400 Bad Request**: Invalid request data
|
||||
- **401 Unauthorized**: Authentication required
|
||||
- **403 Forbidden**: Access denied
|
||||
- **404 Not Found**: Resource not found
|
||||
- **409 Conflict**: Resource conflict
|
||||
- **422 Unprocessable Entity**: Validation error
|
||||
|
||||
### Server Error Codes
|
||||
|
||||
- **500 Internal Server Error**: Server error
|
||||
- **503 Service Unavailable**: Service temporarily unavailable
|
||||
|
||||
## Endpoint Naming Conventions
|
||||
|
||||
### RESTful Patterns
|
||||
|
||||
- **GET /photos**: List photos
|
||||
- **GET /photos/{id}**: Get specific photo
|
||||
- **POST /photos**: Create new photo
|
||||
- **PUT /photos/{id}**: Update photo
|
||||
- **DELETE /photos/{id}**: Delete photo
|
||||
|
||||
### Custom Actions
|
||||
|
||||
- **POST /photos/{id}/identify**: Identify faces in photo
|
||||
- **POST /photos/{id}/duplicates**: Find duplicates
|
||||
- **GET /photos/{id}/faces**: Get faces in photo
|
||||
|
||||
## Request Parameters
|
||||
|
||||
### Query Parameters
|
||||
|
||||
```python
|
||||
# Standard pagination
|
||||
page = request.args.get('page', 1, type=int)
|
||||
per_page = request.args.get('per_page', 20, type=int)
|
||||
|
||||
# Filtering
|
||||
filter_name = request.args.get('filter', '')
|
||||
sort_by = request.args.get('sort', 'date_taken')
|
||||
sort_order = request.args.get('order', 'desc')
|
||||
```
|
||||
|
||||
### JSON Body Parameters
|
||||
|
||||
```python
|
||||
# Validate required fields
|
||||
data = request.get_json()
|
||||
if not data:
|
||||
return jsonify({'success': False, 'error': 'No JSON data provided'}), 400
|
||||
|
||||
required_fields = ['name', 'email']
|
||||
for field in required_fields:
|
||||
if field not in data:
|
||||
return jsonify({'success': False, 'error': f'Missing required field: {field}'}), 400
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Standard Error Handler
|
||||
|
||||
```python
|
||||
@app.errorhandler(404)
|
||||
def not_found(error):
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': 'Resource not found',
|
||||
'code': 'NOT_FOUND'
|
||||
}), 404
|
||||
|
||||
@app.errorhandler(500)
|
||||
def internal_error(error):
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': 'Internal server error',
|
||||
'code': 'INTERNAL_ERROR'
|
||||
}), 500
|
||||
```
|
||||
|
||||
### Validation Errors
|
||||
|
||||
```python
|
||||
def validate_photo_data(data):
|
||||
errors = []
|
||||
|
||||
if 'filename' not in data:
|
||||
errors.append('filename is required')
|
||||
|
||||
if 'path' in data and not os.path.exists(data['path']):
|
||||
errors.append('file path does not exist')
|
||||
|
||||
return errors
|
||||
|
||||
# Usage in endpoint
|
||||
errors = validate_photo_data(data)
|
||||
if errors:
|
||||
return jsonify({
|
||||
'success': False,
|
||||
'error': 'Validation failed',
|
||||
'details': errors
|
||||
}), 422
|
||||
```
|
||||
|
||||
## Database Operations
|
||||
|
||||
### Connection Management
|
||||
|
||||
```python
|
||||
def get_db_connection():
|
||||
conn = sqlite3.connect('punimtag_simple.db')
|
||||
conn.row_factory = sqlite3.Row # Enable dict-like access
|
||||
return conn
|
||||
|
||||
# Usage in endpoint
|
||||
try:
|
||||
conn = get_db_connection()
|
||||
cursor = conn.cursor()
|
||||
# Database operations
|
||||
conn.commit()
|
||||
except Exception as e:
|
||||
conn.rollback()
|
||||
return jsonify({'success': False, 'error': str(e)}), 500
|
||||
finally:
|
||||
conn.close()
|
||||
```
|
||||
|
||||
### Parameterized Queries
|
||||
|
||||
```python
|
||||
# Always use parameterized queries to prevent SQL injection
|
||||
cursor.execute('SELECT * FROM images WHERE id = ?', (image_id,))
|
||||
cursor.execute('INSERT INTO photos (name, path) VALUES (?, ?)', (name, path))
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
### Basic Rate Limiting
|
||||
|
||||
```python
|
||||
from functools import wraps
|
||||
import time
|
||||
|
||||
def rate_limit(requests_per_minute=60):
|
||||
def decorator(f):
|
||||
@wraps(f)
|
||||
def wrapped(*args, **kwargs):
|
||||
# Implement rate limiting logic here
|
||||
return f(*args, **kwargs)
|
||||
return wrapped
|
||||
return decorator
|
||||
|
||||
# Usage
|
||||
@app.route('/api/photos')
|
||||
@rate_limit(requests_per_minute=30)
|
||||
def get_photos():
|
||||
# Endpoint implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## Caching
|
||||
|
||||
### Response Caching
|
||||
|
||||
```python
|
||||
from functools import wraps
|
||||
import hashlib
|
||||
import json
|
||||
|
||||
def cache_response(ttl_seconds=300):
|
||||
def decorator(f):
|
||||
@wraps(f)
|
||||
def wrapped(*args, **kwargs):
|
||||
# Implement caching logic here
|
||||
return f(*args, **kwargs)
|
||||
return wrapped
|
||||
return decorator
|
||||
|
||||
# Usage
|
||||
@app.route('/api/photos')
|
||||
@cache_response(ttl_seconds=60)
|
||||
def get_photos():
|
||||
# Endpoint implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## Logging
|
||||
|
||||
### Request Logging
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@app.before_request
|
||||
def log_request():
|
||||
logger.info(f'{request.method} {request.path} - {request.remote_addr}')
|
||||
|
||||
@app.after_request
|
||||
def log_response(response):
|
||||
logger.info(f'Response: {response.status_code}')
|
||||
return response
|
||||
```
|
||||
|
||||
## Security
|
||||
|
||||
### Input Sanitization
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def sanitize_filename(filename):
|
||||
# Remove dangerous characters
|
||||
filename = re.sub(r'[<>:"/\\|?*]', '', filename)
|
||||
# Limit length
|
||||
return filename[:255]
|
||||
|
||||
def validate_file_type(filename):
|
||||
allowed_extensions = {'.jpg', '.jpeg', '.png', '.gif', '.bmp'}
|
||||
ext = os.path.splitext(filename)[1].lower()
|
||||
return ext in allowed_extensions
|
||||
```
|
||||
|
||||
### CORS Headers
|
||||
|
||||
```python
|
||||
@app.after_request
|
||||
def add_cors_headers(response):
|
||||
response.headers['Access-Control-Allow-Origin'] = '*'
|
||||
response.headers['Access-Control-Allow-Methods'] = 'GET, POST, PUT, DELETE, OPTIONS'
|
||||
response.headers['Access-Control-Allow-Headers'] = 'Content-Type'
|
||||
return response
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Endpoint Testing
|
||||
|
||||
```python
|
||||
def test_get_photos():
|
||||
response = app.test_client().get('/api/photos')
|
||||
assert response.status_code == 200
|
||||
data = json.loads(response.data)
|
||||
assert data['success'] == True
|
||||
assert 'data' in data
|
||||
|
||||
def test_create_photo():
|
||||
response = app.test_client().post('/api/photos',
|
||||
json={'filename': 'test.jpg', 'path': '/test/path'})
|
||||
assert response.status_code == 201
|
||||
data = json.loads(response.data)
|
||||
assert data['success'] == True
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
### Endpoint Documentation
|
||||
|
||||
```python
|
||||
@app.route('/api/photos', methods=['GET'])
|
||||
def get_photos():
|
||||
"""
|
||||
Get a list of photos with optional filtering and pagination.
|
||||
|
||||
Query Parameters:
|
||||
page (int): Page number (default: 1)
|
||||
per_page (int): Items per page (default: 20)
|
||||
filter (str): Filter by name or tags
|
||||
sort (str): Sort field (default: date_taken)
|
||||
order (str): Sort order (asc/desc, default: desc)
|
||||
|
||||
Returns:
|
||||
JSON response with photos and pagination info
|
||||
"""
|
||||
# Implementation
|
||||
pass
|
||||
```
|
||||
@@ -1,725 +0,0 @@
|
||||
# PunimTag Code Conventions
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the coding standards and conventions for PunimTag development.
|
||||
|
||||
## Python Conventions
|
||||
|
||||
### Code Style
|
||||
|
||||
Follow PEP 8 with these specific guidelines:
|
||||
|
||||
```python
|
||||
# Imports
|
||||
import os
|
||||
import sys
|
||||
from typing import List, Dict, Optional
|
||||
from flask import Flask, request, jsonify
|
||||
|
||||
# Constants
|
||||
MAX_FILE_SIZE = 10 * 1024 * 1024 # 10MB
|
||||
ALLOWED_EXTENSIONS = {'.jpg', '.jpeg', '.png', '.gif'}
|
||||
|
||||
# Functions
|
||||
def process_image(image_path: str, max_size: int = MAX_FILE_SIZE) -> Dict[str, any]:
|
||||
"""
|
||||
Process an image file and extract metadata.
|
||||
|
||||
Args:
|
||||
image_path: Path to the image file
|
||||
max_size: Maximum file size in bytes
|
||||
|
||||
Returns:
|
||||
Dictionary containing image metadata
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If image file doesn't exist
|
||||
ValueError: If file size exceeds limit
|
||||
"""
|
||||
if not os.path.exists(image_path):
|
||||
raise FileNotFoundError(f"Image file not found: {image_path}")
|
||||
|
||||
file_size = os.path.getsize(image_path)
|
||||
if file_size > max_size:
|
||||
raise ValueError(f"File size {file_size} exceeds limit {max_size}")
|
||||
|
||||
# Process the image
|
||||
metadata = extract_metadata(image_path)
|
||||
return metadata
|
||||
|
||||
# Classes
|
||||
class ImageProcessor:
|
||||
"""Handles image processing operations."""
|
||||
|
||||
def __init__(self, config: Dict[str, any]):
|
||||
"""
|
||||
Initialize the image processor.
|
||||
|
||||
Args:
|
||||
config: Configuration dictionary
|
||||
"""
|
||||
self.config = config
|
||||
self.supported_formats = config.get('supported_formats', ALLOWED_EXTENSIONS)
|
||||
|
||||
def process_batch(self, image_paths: List[str]) -> List[Dict[str, any]]:
|
||||
"""
|
||||
Process multiple images in batch.
|
||||
|
||||
Args:
|
||||
image_paths: List of image file paths
|
||||
|
||||
Returns:
|
||||
List of processed image metadata
|
||||
"""
|
||||
results = []
|
||||
for path in image_paths:
|
||||
try:
|
||||
result = self.process_single(path)
|
||||
results.append(result)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to process {path}: {e}")
|
||||
results.append({'error': str(e), 'path': path})
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
#### Variables and Functions
|
||||
|
||||
```python
|
||||
# Use snake_case for variables and functions
|
||||
user_name = "john_doe"
|
||||
photo_count = 150
|
||||
max_file_size = 10 * 1024 * 1024
|
||||
|
||||
def get_user_photos(user_id: int) -> List[Dict]:
|
||||
"""Get photos for a specific user."""
|
||||
pass
|
||||
|
||||
def calculate_face_similarity(face1: List[float], face2: List[float]) -> float:
|
||||
"""Calculate similarity between two face encodings."""
|
||||
pass
|
||||
```
|
||||
|
||||
#### Classes
|
||||
|
||||
```python
|
||||
# Use PascalCase for classes
|
||||
class PhotoManager:
|
||||
"""Manages photo operations."""
|
||||
pass
|
||||
|
||||
class FaceRecognitionEngine:
|
||||
"""Handles face recognition operations."""
|
||||
pass
|
||||
```
|
||||
|
||||
#### Constants
|
||||
|
||||
```python
|
||||
# Use UPPER_CASE for constants
|
||||
DATABASE_PATH = "punimtag_simple.db"
|
||||
MAX_THUMBNAIL_SIZE = (200, 200)
|
||||
DEFAULT_PAGE_SIZE = 20
|
||||
```
|
||||
|
||||
### Type Hints
|
||||
|
||||
```python
|
||||
from typing import List, Dict, Optional, Union, Tuple
|
||||
|
||||
def get_photos(
|
||||
user_id: int,
|
||||
page: int = 1,
|
||||
per_page: int = DEFAULT_PAGE_SIZE,
|
||||
filters: Optional[Dict[str, any]] = None
|
||||
) -> Dict[str, Union[List[Dict], int]]:
|
||||
"""
|
||||
Get photos with pagination and filtering.
|
||||
|
||||
Returns:
|
||||
Dictionary with 'photos' list and 'total' count
|
||||
"""
|
||||
pass
|
||||
|
||||
def process_face_encodings(
|
||||
encodings: List[List[float]]
|
||||
) -> Tuple[List[float], float]:
|
||||
"""
|
||||
Process face encodings and return average encoding and confidence.
|
||||
|
||||
Returns:
|
||||
Tuple of (average_encoding, confidence_score)
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```python
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
def safe_operation(func):
|
||||
"""Decorator for safe operation execution."""
|
||||
def wrapper(*args, **kwargs):
|
||||
try:
|
||||
return func(*args, **kwargs)
|
||||
except Exception as e:
|
||||
logger.error(f"Error in {func.__name__}: {e}")
|
||||
return None
|
||||
return wrapper
|
||||
|
||||
@safe_operation
|
||||
def load_image_safely(image_path: str) -> Optional[PIL.Image.Image]:
|
||||
"""Load image with error handling."""
|
||||
return PIL.Image.open(image_path)
|
||||
|
||||
def process_user_request(user_data: Dict) -> Dict[str, any]:
|
||||
"""Process user request with comprehensive error handling."""
|
||||
try:
|
||||
# Validate input
|
||||
if not user_data.get('user_id'):
|
||||
return {'success': False, 'error': 'Missing user_id'}
|
||||
|
||||
# Process request
|
||||
result = perform_operation(user_data)
|
||||
return {'success': True, 'data': result}
|
||||
|
||||
except ValueError as e:
|
||||
logger.warning(f"Validation error: {e}")
|
||||
return {'success': False, 'error': str(e)}
|
||||
except FileNotFoundError as e:
|
||||
logger.error(f"File not found: {e}")
|
||||
return {'success': False, 'error': 'File not found'}
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error: {e}")
|
||||
return {'success': False, 'error': 'Internal server error'}
|
||||
```
|
||||
|
||||
## JavaScript Conventions
|
||||
|
||||
### Code Style
|
||||
|
||||
Follow ESLint with these specific guidelines:
|
||||
|
||||
```javascript
|
||||
// Constants
|
||||
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10MB
|
||||
const ALLOWED_EXTENSIONS = [".jpg", ".jpeg", ".png", ".gif"];
|
||||
|
||||
// Functions
|
||||
function processImage(imagePath, maxSize = MAX_FILE_SIZE) {
|
||||
/**
|
||||
* Process an image file and extract metadata.
|
||||
* @param {string} imagePath - Path to the image file
|
||||
* @param {number} maxSize - Maximum file size in bytes
|
||||
* @returns {Promise<Object>} Image metadata
|
||||
*/
|
||||
return new Promise((resolve, reject) => {
|
||||
if (!imagePath) {
|
||||
reject(new Error("Image path is required"));
|
||||
return;
|
||||
}
|
||||
|
||||
// Process the image
|
||||
resolve(extractMetadata(imagePath));
|
||||
});
|
||||
}
|
||||
|
||||
// Classes
|
||||
class ImageProcessor {
|
||||
/**
|
||||
* Handles image processing operations.
|
||||
* @param {Object} config - Configuration object
|
||||
*/
|
||||
constructor(config) {
|
||||
this.config = config;
|
||||
this.supportedFormats = config.supportedFormats || ALLOWED_EXTENSIONS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Process multiple images in batch.
|
||||
* @param {string[]} imagePaths - Array of image file paths
|
||||
* @returns {Promise<Object[]>} Array of processed image metadata
|
||||
*/
|
||||
async processBatch(imagePaths) {
|
||||
const results = [];
|
||||
|
||||
for (const path of imagePaths) {
|
||||
try {
|
||||
const result = await this.processSingle(path);
|
||||
results.push(result);
|
||||
} catch (error) {
|
||||
console.error(`Failed to process ${path}:`, error);
|
||||
results.push({ error: error.message, path });
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
#### Variables and Functions
|
||||
|
||||
```javascript
|
||||
// Use camelCase for variables and functions
|
||||
const userName = "johnDoe";
|
||||
const photoCount = 150;
|
||||
const maxFileSize = 10 * 1024 * 1024;
|
||||
|
||||
function getUserPhotos(userId) {
|
||||
// Get photos for a specific user
|
||||
}
|
||||
|
||||
function calculateFaceSimilarity(face1, face2) {
|
||||
// Calculate similarity between two face encodings
|
||||
}
|
||||
```
|
||||
|
||||
#### Classes
|
||||
|
||||
```javascript
|
||||
// Use PascalCase for classes
|
||||
class PhotoManager {
|
||||
// Manages photo operations
|
||||
}
|
||||
|
||||
class FaceRecognitionEngine {
|
||||
// Handles face recognition operations
|
||||
}
|
||||
```
|
||||
|
||||
#### Constants
|
||||
|
||||
```javascript
|
||||
// Use UPPER_SNAKE_CASE for constants
|
||||
const DATABASE_PATH = "punimtag_simple.db";
|
||||
const MAX_THUMBNAIL_SIZE = { width: 200, height: 200 };
|
||||
const DEFAULT_PAGE_SIZE = 20;
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```javascript
|
||||
// Async/await with try-catch
|
||||
async function processUserRequest(userData) {
|
||||
try {
|
||||
// Validate input
|
||||
if (!userData.userId) {
|
||||
return { success: false, error: "Missing userId" };
|
||||
}
|
||||
|
||||
// Process request
|
||||
const result = await performOperation(userData);
|
||||
return { success: true, data: result };
|
||||
} catch (error) {
|
||||
console.error("Error processing request:", error);
|
||||
return { success: false, error: "Internal server error" };
|
||||
}
|
||||
}
|
||||
|
||||
// Promise-based error handling
|
||||
function loadImageSafely(imagePath) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (!imagePath) {
|
||||
reject(new Error("Image path is required"));
|
||||
return;
|
||||
}
|
||||
|
||||
// Load image logic
|
||||
resolve(imageData);
|
||||
}).catch((error) => {
|
||||
console.error("Error loading image:", error);
|
||||
return null;
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
## Database Conventions
|
||||
|
||||
### Table Naming
|
||||
|
||||
```sql
|
||||
-- Use snake_case for table names
|
||||
CREATE TABLE user_profiles (
|
||||
id INTEGER PRIMARY KEY,
|
||||
user_name TEXT NOT NULL,
|
||||
email_address TEXT UNIQUE,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE photo_metadata (
|
||||
id INTEGER PRIMARY KEY,
|
||||
image_id INTEGER REFERENCES images(id),
|
||||
exif_data TEXT,
|
||||
gps_coordinates TEXT,
|
||||
processing_status TEXT DEFAULT 'pending'
|
||||
);
|
||||
```
|
||||
|
||||
### Column Naming
|
||||
|
||||
```sql
|
||||
-- Use snake_case for column names
|
||||
CREATE TABLE images (
|
||||
id INTEGER PRIMARY KEY,
|
||||
file_name TEXT NOT NULL,
|
||||
file_path TEXT NOT NULL,
|
||||
file_size INTEGER,
|
||||
date_taken TIMESTAMP,
|
||||
upload_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
is_processed BOOLEAN DEFAULT FALSE
|
||||
);
|
||||
```
|
||||
|
||||
### Index Naming
|
||||
|
||||
```sql
|
||||
-- Use descriptive names for indexes
|
||||
CREATE INDEX idx_images_date_taken ON images(date_taken);
|
||||
CREATE INDEX idx_faces_person_id ON faces(person_id);
|
||||
CREATE INDEX idx_photos_user_id_date ON photos(user_id, date_taken);
|
||||
```
|
||||
|
||||
## File Organization
|
||||
|
||||
### Python Files
|
||||
|
||||
```python
|
||||
# File: src/backend/photo_manager.py
|
||||
"""
|
||||
Photo management module.
|
||||
|
||||
This module handles all photo-related operations including
|
||||
upload, processing, and metadata extraction.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from typing import List, Dict, Optional
|
||||
from PIL import Image
|
||||
|
||||
# Constants
|
||||
MAX_FILE_SIZE = 10 * 1024 * 1024
|
||||
ALLOWED_EXTENSIONS = {'.jpg', '.jpeg', '.png', '.gif'}
|
||||
|
||||
# Logging
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
class PhotoManager:
|
||||
"""Manages photo operations."""
|
||||
|
||||
def __init__(self, config: Dict[str, any]):
|
||||
self.config = config
|
||||
self.storage_path = config.get('storage_path', './photos')
|
||||
|
||||
def process_photo(self, photo_path: str) -> Dict[str, any]:
|
||||
"""Process a single photo."""
|
||||
# Implementation
|
||||
pass
|
||||
|
||||
# Main execution (if applicable)
|
||||
if __name__ == "__main__":
|
||||
# Test or standalone execution
|
||||
pass
|
||||
```
|
||||
|
||||
### JavaScript Files
|
||||
|
||||
```javascript
|
||||
// File: src/frontend/photoManager.js
|
||||
/**
|
||||
* Photo management module.
|
||||
*
|
||||
* This module handles all photo-related operations including
|
||||
* upload, processing, and metadata extraction.
|
||||
*/
|
||||
|
||||
// Constants
|
||||
const MAX_FILE_SIZE = 10 * 1024 * 1024;
|
||||
const ALLOWED_EXTENSIONS = [".jpg", ".jpeg", ".png", ".gif"];
|
||||
|
||||
// Logging
|
||||
const logger = {
|
||||
info: (msg) => console.log(`[INFO] ${msg}`),
|
||||
error: (msg) => console.error(`[ERROR] ${msg}`),
|
||||
warn: (msg) => console.warn(`[WARN] ${msg}`),
|
||||
};
|
||||
|
||||
class PhotoManager {
|
||||
/**
|
||||
* Manages photo operations.
|
||||
* @param {Object} config - Configuration object
|
||||
*/
|
||||
constructor(config) {
|
||||
this.config = config;
|
||||
this.storagePath = config.storagePath || "./photos";
|
||||
}
|
||||
|
||||
/**
|
||||
* Process a single photo.
|
||||
* @param {string} photoPath - Path to the photo
|
||||
* @returns {Promise<Object>} Processing result
|
||||
*/
|
||||
async processPhoto(photoPath) {
|
||||
// Implementation
|
||||
}
|
||||
}
|
||||
|
||||
// Export for module systems
|
||||
if (typeof module !== "undefined" && module.exports) {
|
||||
module.exports = PhotoManager;
|
||||
}
|
||||
```
|
||||
|
||||
## Documentation Standards
|
||||
|
||||
### Function Documentation
|
||||
|
||||
```python
|
||||
def extract_face_features(image_path: str, face_coordinates: Tuple[int, int, int, int]) -> List[float]:
|
||||
"""
|
||||
Extract face features from an image region.
|
||||
|
||||
This function takes an image and face coordinates, then extracts
|
||||
128-dimensional feature vectors using dlib's face recognition model.
|
||||
|
||||
Args:
|
||||
image_path: Path to the source image file
|
||||
face_coordinates: Tuple of (left, top, right, bottom) coordinates
|
||||
|
||||
Returns:
|
||||
List of 128 float values representing face features
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If image file doesn't exist
|
||||
ValueError: If face coordinates are invalid
|
||||
RuntimeError: If face recognition model fails
|
||||
|
||||
Example:
|
||||
>>> coords = (100, 100, 200, 200)
|
||||
>>> features = extract_face_features("photo.jpg", coords)
|
||||
>>> len(features)
|
||||
128
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
### Class Documentation
|
||||
|
||||
```python
|
||||
class FaceRecognitionEngine:
|
||||
"""
|
||||
Engine for face recognition operations.
|
||||
|
||||
This class provides methods for detecting faces in images,
|
||||
extracting face features, and comparing face similarities.
|
||||
|
||||
Attributes:
|
||||
model_path (str): Path to the face recognition model
|
||||
confidence_threshold (float): Minimum confidence for face detection
|
||||
max_faces (int): Maximum number of faces to detect per image
|
||||
|
||||
Example:
|
||||
>>> engine = FaceRecognitionEngine()
|
||||
>>> faces = engine.detect_faces("group_photo.jpg")
|
||||
>>> print(f"Found {len(faces)} faces")
|
||||
"""
|
||||
|
||||
def __init__(self, model_path: str = None, confidence_threshold: float = 0.6):
|
||||
"""
|
||||
Initialize the face recognition engine.
|
||||
|
||||
Args:
|
||||
model_path: Path to the face recognition model file
|
||||
confidence_threshold: Minimum confidence for face detection
|
||||
"""
|
||||
pass
|
||||
```
|
||||
|
||||
## Testing Conventions
|
||||
|
||||
### Test File Structure
|
||||
|
||||
```python
|
||||
# File: tests/unit/test_photo_manager.py
|
||||
"""
|
||||
Unit tests for PhotoManager class.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from src.backend.photo_manager import PhotoManager
|
||||
|
||||
class TestPhotoManager:
|
||||
"""Test cases for PhotoManager class."""
|
||||
|
||||
@pytest.fixture
|
||||
def photo_manager(self):
|
||||
"""Create a PhotoManager instance for testing."""
|
||||
config = {'storage_path': '/test/path'}
|
||||
return PhotoManager(config)
|
||||
|
||||
def test_process_photo_with_valid_file(self, photo_manager):
|
||||
"""Test processing a valid photo file."""
|
||||
# Test implementation
|
||||
pass
|
||||
|
||||
def test_process_photo_with_invalid_file(self, photo_manager):
|
||||
"""Test processing an invalid photo file."""
|
||||
# Test implementation
|
||||
pass
|
||||
```
|
||||
|
||||
## Git Conventions
|
||||
|
||||
### Commit Messages
|
||||
|
||||
```
|
||||
feat: add face recognition feature
|
||||
fix: resolve duplicate photo detection issue
|
||||
docs: update API documentation
|
||||
test: add unit tests for photo processing
|
||||
refactor: improve error handling in face detection
|
||||
style: format code according to PEP 8
|
||||
perf: optimize thumbnail generation
|
||||
chore: update dependencies
|
||||
```
|
||||
|
||||
### Branch Naming
|
||||
|
||||
```
|
||||
feature/face-recognition
|
||||
bugfix/duplicate-detection
|
||||
hotfix/security-vulnerability
|
||||
docs/api-documentation
|
||||
test/photo-processing
|
||||
refactor/error-handling
|
||||
```
|
||||
|
||||
## Performance Guidelines
|
||||
|
||||
### Python Performance
|
||||
|
||||
```python
|
||||
# Use list comprehensions instead of loops when appropriate
|
||||
# Good
|
||||
squares = [x**2 for x in range(1000)]
|
||||
|
||||
# Avoid
|
||||
squares = []
|
||||
for x in range(1000):
|
||||
squares.append(x**2)
|
||||
|
||||
# Use generators for large datasets
|
||||
def process_large_dataset(file_path):
|
||||
"""Process large dataset using generator."""
|
||||
with open(file_path, 'r') as file:
|
||||
for line in file:
|
||||
yield process_line(line)
|
||||
|
||||
# Use appropriate data structures
|
||||
from collections import defaultdict, Counter
|
||||
|
||||
# Use defaultdict for counting
|
||||
word_count = defaultdict(int)
|
||||
for word in words:
|
||||
word_count[word] += 1
|
||||
|
||||
# Use Counter for frequency analysis
|
||||
word_freq = Counter(words)
|
||||
```
|
||||
|
||||
### JavaScript Performance
|
||||
|
||||
```javascript
|
||||
// Use appropriate array methods
|
||||
// Good
|
||||
const squares = Array.from({ length: 1000 }, (_, i) => i ** 2);
|
||||
|
||||
// Avoid
|
||||
const squares = [];
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
squares.push(i ** 2);
|
||||
}
|
||||
|
||||
// Use async/await for I/O operations
|
||||
async function processImages(imagePaths) {
|
||||
const results = await Promise.all(
|
||||
imagePaths.map((path) => processImage(path))
|
||||
);
|
||||
return results;
|
||||
}
|
||||
|
||||
// Use appropriate data structures
|
||||
const wordCount = new Map();
|
||||
words.forEach((word) => {
|
||||
wordCount.set(word, (wordCount.get(word) || 0) + 1);
|
||||
});
|
||||
```
|
||||
|
||||
## Security Guidelines
|
||||
|
||||
### Input Validation
|
||||
|
||||
```python
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def validate_filename(filename: str) -> bool:
|
||||
"""Validate filename for security."""
|
||||
# Check for dangerous characters
|
||||
dangerous_chars = r'[<>:"/\\|?*]'
|
||||
if re.search(dangerous_chars, filename):
|
||||
return False
|
||||
|
||||
# Check for path traversal
|
||||
if '..' in filename or filename.startswith('/'):
|
||||
return False
|
||||
|
||||
# Check length
|
||||
if len(filename) > 255:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def sanitize_user_input(user_input: str) -> str:
|
||||
"""Sanitize user input to prevent injection attacks."""
|
||||
# Remove HTML tags
|
||||
import html
|
||||
sanitized = html.escape(user_input)
|
||||
|
||||
# Remove SQL injection patterns
|
||||
sql_patterns = [';', '--', '/*', '*/', 'union', 'select', 'drop']
|
||||
for pattern in sql_patterns:
|
||||
sanitized = sanitized.replace(pattern.lower(), '')
|
||||
|
||||
return sanitized
|
||||
```
|
||||
|
||||
### Database Security
|
||||
|
||||
```python
|
||||
# Always use parameterized queries
|
||||
def get_user_photos(user_id: int):
|
||||
"""Get photos for a user using parameterized query."""
|
||||
cursor.execute(
|
||||
'SELECT * FROM photos WHERE user_id = ?',
|
||||
(user_id,)
|
||||
)
|
||||
return cursor.fetchall()
|
||||
|
||||
# Never use string formatting for SQL
|
||||
# BAD - vulnerable to SQL injection
|
||||
def bad_get_user_photos(user_id: int):
|
||||
cursor.execute(f'SELECT * FROM photos WHERE user_id = {user_id}')
|
||||
return cursor.fetchall()
|
||||
```
|
||||
@@ -1,69 +0,0 @@
|
||||
# PunimTag Product Vision
|
||||
|
||||
## Overview
|
||||
|
||||
PunimTag is an intelligent photo management system that uses face recognition to automatically organize, tag, and manage personal photo collections.
|
||||
|
||||
## Core Value Proposition
|
||||
|
||||
- **Automatic Face Recognition**: Identify and tag people in photos without manual effort
|
||||
- **Smart Organization**: Group photos by people, events, and locations
|
||||
- **Duplicate Detection**: Find and manage duplicate photos automatically
|
||||
- **Intuitive Interface**: Web-based GUI that's easy to use for non-technical users
|
||||
- **Privacy-First**: Local processing, no cloud dependencies
|
||||
|
||||
## Target Users
|
||||
|
||||
- **Primary**: Individuals with large photo collections (families, photographers, content creators)
|
||||
- **Secondary**: Small businesses needing photo organization (real estate, events, etc.)
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Photo Management
|
||||
|
||||
- Upload and organize photos by date, location, and content
|
||||
- Automatic metadata extraction (EXIF data, GPS coordinates)
|
||||
- Batch operations for efficiency
|
||||
|
||||
### 2. Face Recognition & Tagging
|
||||
|
||||
- Automatic face detection in photos
|
||||
- Face identification and naming
|
||||
- Group photos by people
|
||||
- Handle multiple faces per photo
|
||||
|
||||
### 3. Duplicate Management
|
||||
|
||||
- Find duplicate photos automatically
|
||||
- Visual comparison tools
|
||||
- Bulk removal options
|
||||
- Keep best quality versions
|
||||
|
||||
### 4. Search & Discovery
|
||||
|
||||
- Search by person name
|
||||
- Filter by date ranges
|
||||
- Tag-based filtering
|
||||
- Similar face suggestions
|
||||
|
||||
### 5. User Experience
|
||||
|
||||
- Progressive loading for large collections
|
||||
- Responsive web interface
|
||||
- Custom dialogs (no browser alerts)
|
||||
- Real-time notifications
|
||||
|
||||
## Success Metrics
|
||||
|
||||
- **User Engagement**: Time spent organizing photos
|
||||
- **Accuracy**: Face recognition precision
|
||||
- **Performance**: Load times for large collections
|
||||
- **Usability**: User satisfaction and ease of use
|
||||
|
||||
## Future Roadmap
|
||||
|
||||
- Cloud sync capabilities
|
||||
- Mobile app companion
|
||||
- Advanced AI features (emotion detection, age progression)
|
||||
- Social sharing features
|
||||
- Integration with existing photo services
|
||||
@@ -1,109 +0,0 @@
|
||||
# PunimTag Project Structure
|
||||
|
||||
## Directory Organization
|
||||
|
||||
```
|
||||
PunimTag/
|
||||
├── src/ # Main application source code
|
||||
│ ├── backend/ # Flask backend and API
|
||||
│ ├── frontend/ # JavaScript and UI components
|
||||
│ └── utils/ # Utility functions and helpers
|
||||
├── docs/ # Documentation and steering documents
|
||||
├── tests/ # All test files and test utilities
|
||||
├── data/ # Database files and user data
|
||||
├── assets/ # Static assets (images, CSS, etc.)
|
||||
├── config/ # Configuration files
|
||||
└── scripts/ # Build and deployment scripts
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### Backend (Flask)
|
||||
|
||||
- **Main Application**: `simple_web_gui.py` - Primary Flask app
|
||||
- **Database Management**: `db_manager.py` - Database operations
|
||||
- **Face Recognition**: `visual_identifier.py` - Face detection and recognition
|
||||
- **Configuration**: `config.py` - App configuration
|
||||
|
||||
### Frontend (JavaScript)
|
||||
|
||||
- **UI Components**: Embedded in Flask templates
|
||||
- **Progressive Loading**: Handles large photo collections
|
||||
- **Custom Dialogs**: Replaces browser alerts
|
||||
- **Face Management**: Face identification and tagging interface
|
||||
|
||||
### Data Layer
|
||||
|
||||
- **SQLite Database**: `punimtag_simple.db` - Main database
|
||||
- **Image Storage**: `photos/` directory
|
||||
- **Thumbnails**: Generated on-demand
|
||||
- **Face Encodings**: Stored as binary data
|
||||
|
||||
## Architecture Principles
|
||||
|
||||
### 1. Separation of Concerns
|
||||
|
||||
- **Backend**: Business logic, data processing, API endpoints
|
||||
- **Frontend**: User interface, interactions, state management
|
||||
- **Data**: Persistent storage, caching, optimization
|
||||
|
||||
### 2. Progressive Enhancement
|
||||
|
||||
- **Core Functionality**: Works without JavaScript
|
||||
- **Enhanced Features**: Progressive loading, real-time updates
|
||||
- **Fallbacks**: Graceful degradation for older browsers
|
||||
|
||||
### 3. Performance Optimization
|
||||
|
||||
- **Lazy Loading**: Images and data loaded on demand
|
||||
- **Caching**: Thumbnails and frequently accessed data
|
||||
- **Batch Operations**: Efficient bulk processing
|
||||
|
||||
### 4. User Experience
|
||||
|
||||
- **Responsive Design**: Works on all screen sizes
|
||||
- **Accessibility**: Keyboard navigation, screen reader support
|
||||
- **Error Handling**: Graceful error recovery and user feedback
|
||||
|
||||
## File Naming Conventions
|
||||
|
||||
### Python Files
|
||||
|
||||
- **snake_case** for file names and functions
|
||||
- **PascalCase** for classes
|
||||
- **UPPER_CASE** for constants
|
||||
|
||||
### JavaScript Files
|
||||
|
||||
- **camelCase** for functions and variables
|
||||
- **PascalCase** for classes and components
|
||||
- **kebab-case** for CSS classes
|
||||
|
||||
### Database
|
||||
|
||||
- **snake_case** for table and column names
|
||||
- **Descriptive names** that clearly indicate purpose
|
||||
|
||||
## Dependencies
|
||||
|
||||
### Backend Dependencies
|
||||
|
||||
- **Flask**: Web framework
|
||||
- **SQLite**: Database
|
||||
- **dlib**: Face recognition
|
||||
- **Pillow**: Image processing
|
||||
- **numpy**: Numerical operations
|
||||
|
||||
### Frontend Dependencies
|
||||
|
||||
- **Vanilla JavaScript**: No external frameworks
|
||||
- **CSS Grid/Flexbox**: Layout system
|
||||
- **Fetch API**: HTTP requests
|
||||
- **Intersection Observer**: Progressive loading
|
||||
|
||||
## Configuration Management
|
||||
|
||||
- **Environment Variables**: For sensitive data
|
||||
- **JSON Config Files**: For application settings
|
||||
- **Database Migrations**: For schema changes
|
||||
- **Feature Flags**: For experimental features
|
||||
-136
@@ -1,136 +0,0 @@
|
||||
# PunimTag Technical Architecture
|
||||
|
||||
## Technology Stack
|
||||
|
||||
### Backend
|
||||
|
||||
- **Framework**: Flask (Python web framework)
|
||||
- **Database**: SQLite (lightweight, file-based)
|
||||
- **Face Recognition**: dlib (C++ library with Python bindings)
|
||||
- **Image Processing**: Pillow (PIL fork)
|
||||
- **Data Processing**: NumPy (numerical operations)
|
||||
|
||||
### Frontend
|
||||
|
||||
- **Language**: Vanilla JavaScript (ES6+)
|
||||
- **Styling**: CSS3 with Grid/Flexbox
|
||||
- **HTTP Client**: Fetch API
|
||||
- **Progressive Loading**: Intersection Observer API
|
||||
- **No Frameworks**: Pure JavaScript for simplicity
|
||||
|
||||
### Development Tools
|
||||
|
||||
- **Version Control**: Git
|
||||
- **Package Management**: pip (Python), npm (optional for frontend tools)
|
||||
- **Testing**: pytest (Python), Jest (JavaScript)
|
||||
- **Code Quality**: flake8, black (Python), ESLint (JavaScript)
|
||||
|
||||
## Core Technologies
|
||||
|
||||
### Face Recognition Pipeline
|
||||
|
||||
1. **Image Loading**: Pillow for image processing
|
||||
2. **Face Detection**: dlib's CNN face detector
|
||||
3. **Feature Extraction**: dlib's 128-dimensional face encodings
|
||||
4. **Similarity Matching**: Euclidean distance calculation
|
||||
5. **Storage**: Binary encoding storage in SQLite
|
||||
|
||||
### Database Schema
|
||||
|
||||
```sql
|
||||
-- Core tables
|
||||
images (id, filename, path, date_taken, metadata)
|
||||
faces (id, image_id, person_id, encoding, coordinates, confidence)
|
||||
people (id, name, created_date)
|
||||
tags (id, name)
|
||||
image_tags (image_id, tag_id)
|
||||
|
||||
-- Supporting tables
|
||||
face_encodings (id, face_id, encoding_data)
|
||||
photo_metadata (image_id, exif_data, gps_data)
|
||||
```
|
||||
|
||||
### API Design
|
||||
|
||||
- **RESTful Endpoints**: Standard HTTP methods (GET, POST, DELETE)
|
||||
- **JSON Responses**: Consistent response format
|
||||
- **Error Handling**: HTTP status codes with descriptive messages
|
||||
- **Pagination**: Offset-based for large datasets
|
||||
|
||||
## Performance Considerations4
|
||||
|
||||
### Image Processing
|
||||
|
||||
- **Thumbnail Generation**: On-demand with caching
|
||||
- **Face Detection**: Optimized for speed vs accuracy
|
||||
- **Batch Processing**: Efficient handling of large photo sets
|
||||
- **Memory Management**: Streaming for large images
|
||||
|
||||
### Database Optimization
|
||||
|
||||
- **Indexing**: Strategic indexes on frequently queried columns
|
||||
- **Connection Pooling**: Efficient database connections
|
||||
- **Query Optimization**: Minimize N+1 query problems
|
||||
- **Data Archiving**: Move old data to separate tables
|
||||
|
||||
### Frontend Performance
|
||||
|
||||
- **Progressive Loading**: Load data in chunks
|
||||
- **Image Lazy Loading**: Load images as they become visible
|
||||
- **Caching**: Browser caching for static assets
|
||||
- **Debouncing**: Prevent excessive API calls
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Data Protection
|
||||
|
||||
- **Local Storage**: No cloud dependencies
|
||||
- **Input Validation**: Sanitize all user inputs
|
||||
- **SQL Injection Prevention**: Parameterized queries
|
||||
- **File Upload Security**: Validate file types and sizes
|
||||
|
||||
### Privacy
|
||||
|
||||
- **Face Data**: Stored locally, not shared
|
||||
- **Metadata**: User controls what's stored
|
||||
- **Access Control**: Local access only
|
||||
- **Data Export**: User can export/delete their data
|
||||
|
||||
## Scalability
|
||||
|
||||
### Current Limitations
|
||||
|
||||
- **Single User**: Designed for personal use
|
||||
- **Local Storage**: Limited by disk space
|
||||
- **Processing Power**: CPU-intensive face recognition
|
||||
- **Memory**: Large photo collections require significant RAM
|
||||
|
||||
### Future Scalability
|
||||
|
||||
- **Multi-User Support**: Database schema supports multiple users
|
||||
- **Cloud Integration**: Optional cloud storage and processing
|
||||
- **Distributed Processing**: GPU acceleration for face recognition
|
||||
- **Microservices**: Separate services for different functions
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Code Organization
|
||||
|
||||
- **Modular Design**: Separate concerns into modules
|
||||
- **Configuration Management**: Environment-based settings
|
||||
- **Error Handling**: Comprehensive error catching and logging
|
||||
- **Documentation**: Inline code documentation
|
||||
|
||||
### Testing Strategy
|
||||
|
||||
- **Unit Tests**: Test individual functions and classes
|
||||
- **Integration Tests**: Test API endpoints and database operations
|
||||
- **End-to-End Tests**: Test complete user workflows
|
||||
- **Performance Tests**: Test with large datasets
|
||||
|
||||
### Deployment
|
||||
|
||||
- **Local Development**: Flask development server
|
||||
- **Production**: WSGI server (Gunicorn) with reverse proxy
|
||||
- **Containerization**: Docker for consistent environments
|
||||
- **Monitoring**: Logging and health checks
|
||||
@@ -1,531 +0,0 @@
|
||||
# PunimTag Testing Standards
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the standards for writing and organizing tests in PunimTag.
|
||||
|
||||
## Test Organization
|
||||
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
├── unit/ # Unit tests for individual functions
|
||||
├── integration/ # Integration tests for API endpoints
|
||||
├── e2e/ # End-to-end tests for complete workflows
|
||||
├── fixtures/ # Test data and fixtures
|
||||
├── utils/ # Test utilities and helpers
|
||||
└── conftest.py # pytest configuration and shared fixtures
|
||||
```
|
||||
|
||||
### Test File Naming
|
||||
|
||||
- **Unit Tests**: `test_<module_name>.py`
|
||||
- **Integration Tests**: `test_<feature>_integration.py`
|
||||
- **E2E Tests**: `test_<workflow>_e2e.py`
|
||||
- **Test Utilities**: `test_<utility_name>.py`
|
||||
|
||||
## Test Categories
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Test individual functions and classes in isolation.
|
||||
|
||||
```python
|
||||
# tests/unit/test_face_recognition.py
|
||||
import pytest
|
||||
from src.utils.face_recognition import detect_faces, encode_face
|
||||
|
||||
def test_detect_faces_with_valid_image():
|
||||
"""Test face detection with a valid image."""
|
||||
image_path = "tests/fixtures/valid_face.jpg"
|
||||
faces = detect_faces(image_path)
|
||||
|
||||
assert len(faces) > 0
|
||||
assert all(hasattr(face, 'left') for face in faces)
|
||||
assert all(hasattr(face, 'top') for face in faces)
|
||||
|
||||
def test_detect_faces_with_no_faces():
|
||||
"""Test face detection with an image containing no faces."""
|
||||
image_path = "tests/fixtures/no_faces.jpg"
|
||||
faces = detect_faces(image_path)
|
||||
|
||||
assert len(faces) == 0
|
||||
|
||||
def test_encode_face_with_valid_face():
|
||||
"""Test face encoding with a valid face."""
|
||||
face_image = load_test_face_image()
|
||||
encoding = encode_face(face_image)
|
||||
|
||||
assert len(encoding) == 128
|
||||
assert all(isinstance(x, float) for x in encoding)
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Test API endpoints and database interactions.
|
||||
|
||||
```python
|
||||
# tests/integration/test_photo_api.py
|
||||
import pytest
|
||||
from src.app import app
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
app.config['TESTING'] = True
|
||||
app.config['DATABASE'] = 'test.db'
|
||||
|
||||
with app.test_client() as client:
|
||||
yield client
|
||||
|
||||
def test_get_photos_endpoint(client):
|
||||
"""Test the GET /photos endpoint."""
|
||||
response = client.get('/photos')
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.get_json()
|
||||
assert data['success'] == True
|
||||
assert 'photos' in data
|
||||
|
||||
def test_create_photo_endpoint(client):
|
||||
"""Test the POST /photos endpoint."""
|
||||
photo_data = {
|
||||
'filename': 'test.jpg',
|
||||
'path': '/test/path/test.jpg'
|
||||
}
|
||||
|
||||
response = client.post('/photos', json=photo_data)
|
||||
|
||||
assert response.status_code == 201
|
||||
data = response.get_json()
|
||||
assert data['success'] == True
|
||||
assert 'photo_id' in data
|
||||
|
||||
def test_get_photo_not_found(client):
|
||||
"""Test getting a non-existent photo."""
|
||||
response = client.get('/photos/99999')
|
||||
|
||||
assert response.status_code == 404
|
||||
data = response.get_json()
|
||||
assert data['success'] == False
|
||||
assert 'error' in data
|
||||
```
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
Test complete user workflows.
|
||||
|
||||
```python
|
||||
# tests/e2e/test_photo_workflow.py
|
||||
import pytest
|
||||
from selenium import webdriver
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.ui import WebDriverWait
|
||||
from selenium.webdriver.support import expected_conditions as EC
|
||||
|
||||
@pytest.fixture
|
||||
def driver():
|
||||
"""Create a web driver for E2E tests."""
|
||||
driver = webdriver.Chrome()
|
||||
driver.implicitly_wait(10)
|
||||
yield driver
|
||||
driver.quit()
|
||||
|
||||
def test_upload_and_identify_photo(driver):
|
||||
"""Test the complete workflow of uploading and identifying a photo."""
|
||||
# Navigate to the app
|
||||
driver.get("http://localhost:5000")
|
||||
|
||||
# Upload a photo
|
||||
file_input = driver.find_element(By.ID, "photo-upload")
|
||||
file_input.send_keys("tests/fixtures/test_photo.jpg")
|
||||
|
||||
# Wait for upload to complete
|
||||
WebDriverWait(driver, 30).until(
|
||||
EC.presence_of_element_located((By.CLASS_NAME, "photo-card"))
|
||||
)
|
||||
|
||||
# Click on the photo to open details
|
||||
photo_card = driver.find_element(By.CLASS_NAME, "photo-card")
|
||||
photo_card.click()
|
||||
|
||||
# Wait for photo details to load
|
||||
WebDriverWait(driver, 10).until(
|
||||
EC.presence_of_element_located((By.ID, "photoDetails"))
|
||||
)
|
||||
|
||||
# Verify faces are detected
|
||||
faces = driver.find_elements(By.CLASS_NAME, "face-item")
|
||||
assert len(faces) > 0
|
||||
|
||||
# Identify a face
|
||||
face_input = driver.find_element(By.CLASS_NAME, "face-name-input")
|
||||
face_input.send_keys("Test Person")
|
||||
|
||||
identify_button = driver.find_element(By.CLASS_NAME, "identify-face-btn")
|
||||
identify_button.click()
|
||||
|
||||
# Verify identification
|
||||
WebDriverWait(driver, 10).until(
|
||||
EC.text_to_be_present_in_element((By.CLASS_NAME, "face-name"), "Test Person")
|
||||
)
|
||||
```
|
||||
|
||||
## Test Fixtures
|
||||
|
||||
### Database Fixtures
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import pytest
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
@pytest.fixture
|
||||
def test_db():
|
||||
"""Create a temporary test database."""
|
||||
db_fd, db_path = tempfile.mkstemp()
|
||||
|
||||
# Create test database schema
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
cursor.execute('''
|
||||
CREATE TABLE images (
|
||||
id INTEGER PRIMARY KEY,
|
||||
filename TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
date_taken TEXT
|
||||
)
|
||||
''')
|
||||
|
||||
cursor.execute('''
|
||||
CREATE TABLE faces (
|
||||
id INTEGER PRIMARY KEY,
|
||||
image_id INTEGER,
|
||||
person_id INTEGER,
|
||||
encoding BLOB,
|
||||
left INTEGER,
|
||||
top INTEGER,
|
||||
right INTEGER,
|
||||
bottom INTEGER
|
||||
)
|
||||
''')
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
yield db_path
|
||||
|
||||
# Cleanup
|
||||
os.close(db_fd)
|
||||
os.unlink(db_path)
|
||||
|
||||
@pytest.fixture
|
||||
def sample_photos(test_db):
|
||||
"""Add sample photos to the test database."""
|
||||
conn = sqlite3.connect(test_db)
|
||||
cursor = conn.cursor()
|
||||
|
||||
photos = [
|
||||
('photo1.jpg', '/test/path/photo1.jpg', '2023-01-01'),
|
||||
('photo2.jpg', '/test/path/photo2.jpg', '2023-01-02'),
|
||||
('photo3.jpg', '/test/path/photo3.jpg', '2023-01-03')
|
||||
]
|
||||
|
||||
cursor.executemany(
|
||||
'INSERT INTO images (filename, path, date_taken) VALUES (?, ?, ?)',
|
||||
photos
|
||||
)
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
return photos
|
||||
```
|
||||
|
||||
### Mock Fixtures
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
@pytest.fixture
|
||||
def mock_face_recognition():
|
||||
"""Mock face recognition functions."""
|
||||
with patch('src.utils.face_recognition.detect_faces') as mock_detect:
|
||||
with patch('src.utils.face_recognition.encode_face') as mock_encode:
|
||||
mock_detect.return_value = [
|
||||
Mock(left=100, top=100, right=200, bottom=200)
|
||||
]
|
||||
mock_encode.return_value = [0.1] * 128
|
||||
|
||||
yield {
|
||||
'detect': mock_detect,
|
||||
'encode': mock_encode
|
||||
}
|
||||
|
||||
@pytest.fixture
|
||||
def mock_file_system():
|
||||
"""Mock file system operations."""
|
||||
with patch('os.path.exists') as mock_exists:
|
||||
with patch('os.path.getsize') as mock_size:
|
||||
mock_exists.return_value = True
|
||||
mock_size.return_value = 1024 * 1024 # 1MB
|
||||
|
||||
yield {
|
||||
'exists': mock_exists,
|
||||
'size': mock_size
|
||||
}
|
||||
```
|
||||
|
||||
## Test Data Management
|
||||
|
||||
### Test Images
|
||||
|
||||
```python
|
||||
# tests/fixtures/test_images.py
|
||||
import os
|
||||
from PIL import Image
|
||||
import numpy as np
|
||||
|
||||
def create_test_image(width=100, height=100, filename="test.jpg"):
|
||||
"""Create a test image for testing."""
|
||||
# Create a simple test image
|
||||
image = Image.new('RGB', (width, height), color='red')
|
||||
|
||||
# Add a simple face-like pattern
|
||||
pixels = np.array(image)
|
||||
# Draw a simple face outline
|
||||
pixels[30:70, 40:60] = [255, 255, 255] # White face
|
||||
pixels[40:50, 45:55] = [0, 0, 0] # Black eyes
|
||||
|
||||
test_image = Image.fromarray(pixels)
|
||||
test_path = f"tests/fixtures/{filename}"
|
||||
test_image.save(test_path)
|
||||
|
||||
return test_path
|
||||
|
||||
def cleanup_test_images():
|
||||
"""Clean up test images."""
|
||||
fixture_dir = "tests/fixtures"
|
||||
for file in os.listdir(fixture_dir):
|
||||
if file.endswith(('.jpg', '.png', '.jpeg')):
|
||||
os.remove(os.path.join(fixture_dir, file))
|
||||
```
|
||||
|
||||
## Performance Testing
|
||||
|
||||
### Load Testing
|
||||
|
||||
```python
|
||||
# tests/performance/test_load.py
|
||||
import pytest
|
||||
import time
|
||||
import concurrent.futures
|
||||
from src.app import app
|
||||
|
||||
def test_concurrent_photo_requests():
|
||||
"""Test handling multiple concurrent photo requests."""
|
||||
client = app.test_client()
|
||||
|
||||
def make_request():
|
||||
return client.get('/photos?page=1&per_page=20')
|
||||
|
||||
# Make 10 concurrent requests
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(make_request) for _ in range(10)]
|
||||
responses = [future.result() for future in futures]
|
||||
|
||||
# All requests should succeed
|
||||
for response in responses:
|
||||
assert response.status_code == 200
|
||||
|
||||
# Check response times
|
||||
start_time = time.time()
|
||||
for _ in range(5):
|
||||
client.get('/photos?page=1&per_page=20')
|
||||
end_time = time.time()
|
||||
|
||||
avg_time = (end_time - start_time) / 5
|
||||
assert avg_time < 1.0 # Should respond within 1 second
|
||||
|
||||
def test_large_photo_collection():
|
||||
"""Test performance with a large photo collection."""
|
||||
# This would require setting up a large test dataset
|
||||
pass
|
||||
```
|
||||
|
||||
## Test Configuration
|
||||
|
||||
### pytest Configuration
|
||||
|
||||
```ini
|
||||
# pytest.ini
|
||||
[tool:pytest]
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
addopts =
|
||||
-v
|
||||
--tb=short
|
||||
--strict-markers
|
||||
--disable-warnings
|
||||
markers =
|
||||
unit: Unit tests
|
||||
integration: Integration tests
|
||||
e2e: End-to-end tests
|
||||
slow: Slow running tests
|
||||
performance: Performance tests
|
||||
```
|
||||
|
||||
### Test Environment Variables
|
||||
|
||||
```python
|
||||
# tests/conftest.py
|
||||
import os
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def test_environment():
|
||||
"""Set up test environment variables."""
|
||||
os.environ['TESTING'] = 'true'
|
||||
os.environ['DATABASE_PATH'] = 'test.db'
|
||||
os.environ['PHOTOS_DIR'] = 'tests/fixtures/photos'
|
||||
|
||||
yield
|
||||
|
||||
# Cleanup
|
||||
if 'TESTING' in os.environ:
|
||||
del os.environ['TESTING']
|
||||
```
|
||||
|
||||
## Code Coverage
|
||||
|
||||
### Coverage Configuration
|
||||
|
||||
```ini
|
||||
# .coveragerc
|
||||
[run]
|
||||
source = src
|
||||
omit =
|
||||
*/tests/*
|
||||
*/venv/*
|
||||
*/__pycache__/*
|
||||
*/migrations/*
|
||||
|
||||
[report]
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
def __repr__
|
||||
raise AssertionError
|
||||
raise NotImplementedError
|
||||
if 0:
|
||||
if __name__ == .__main__.:
|
||||
```
|
||||
|
||||
### Coverage Testing
|
||||
|
||||
```python
|
||||
# tests/test_coverage.py
|
||||
import pytest
|
||||
import coverage
|
||||
|
||||
def test_code_coverage():
|
||||
"""Ensure code coverage meets minimum requirements."""
|
||||
cov = coverage.Coverage()
|
||||
cov.start()
|
||||
|
||||
# Run the application
|
||||
from src.app import app
|
||||
client = app.test_client()
|
||||
client.get('/photos')
|
||||
|
||||
cov.stop()
|
||||
cov.save()
|
||||
|
||||
# Generate coverage report
|
||||
cov.report()
|
||||
|
||||
# Check coverage percentage
|
||||
total_coverage = cov.report()
|
||||
assert total_coverage >= 80.0 # Minimum 80% coverage
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
### GitHub Actions
|
||||
|
||||
```yaml
|
||||
# .github/workflows/test.yml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: 3.9
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -r requirements.txt
|
||||
pip install pytest pytest-cov
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
pytest tests/ --cov=src --cov-report=xml
|
||||
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v1
|
||||
with:
|
||||
file: ./coverage.xml
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Test Naming
|
||||
|
||||
- Use descriptive test names that explain what is being tested
|
||||
- Follow the pattern: `test_<function>_<scenario>_<expected_result>`
|
||||
- Example: `test_detect_faces_with_multiple_faces_returns_correct_count`
|
||||
|
||||
### Test Independence
|
||||
|
||||
- Each test should be independent and not rely on other tests
|
||||
- Use fixtures to set up test data
|
||||
- Clean up after each test
|
||||
|
||||
### Test Data
|
||||
|
||||
- Use realistic but minimal test data
|
||||
- Create helper functions for generating test data
|
||||
- Keep test data in fixtures directory
|
||||
|
||||
### Error Testing
|
||||
|
||||
- Test both success and failure scenarios
|
||||
- Test edge cases and boundary conditions
|
||||
- Test error handling and recovery
|
||||
|
||||
### Performance
|
||||
|
||||
- Keep tests fast and efficient
|
||||
- Use mocking for slow operations
|
||||
- Separate slow tests with `@pytest.mark.slow`
|
||||
|
||||
### Documentation
|
||||
|
||||
- Document complex test scenarios
|
||||
- Explain the purpose of each test
|
||||
- Keep test code readable and maintainable
|
||||
Reference in New Issue
Block a user