ilia/punimtag
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

migration to web

Browse Source
This commit is contained in:
tanyar09
2025-10-31 12:10:44 -04:00
parent d6b1e85998
commit 94385e3dcc
3560 changed files with 575829 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+206 -169
View File
@@ -1,34 +1,33 @@
# PunimTag
# PunimTag Web
**Photo Management and Facial Recognition System**
**Modern Photo Management and Facial Recognition System**
A powerful desktop application for organizing and tagging photos using **state-of-the-art DeepFace AI** with ArcFace recognition model.
A fast, simple, and modern web application for organizing and tagging photos using state-of-the-art DeepFace AI with ArcFace recognition model.
---
## 🎯 Features
- **🌐 Web-Based**: Modern React frontend with FastAPI backend
- **🔥 DeepFace AI**: State-of-the-art face detection with RetinaFace and ArcFace models
- **🎯 Superior Accuracy**: 512-dimensional embeddings (4x more detailed than face_recognition)
- **⚙️ Multiple Detectors**: Choose from RetinaFace, MTCNN, OpenCV, or SSD detectors
- **🎨 Flexible Models**: Select ArcFace, Facenet, Facenet512, or VGG-Face recognition models
- **📊 Rich Metadata**: Face confidence scores, quality metrics, detector/model info displayed in GUI
- **👤 Person Identification**: Identify and tag people across your photo collection
- **🤖 Smart Auto-Matching**: Intelligent face matching with quality scoring and cosine similarity
- **🔍 Advanced Search**: Search by people, dates, tags, and folders
- **🎚️ Quality Filtering**: Filter faces by quality score in Identify panel (0-100%)
- **🏷️ Tag Management**: Organize photos with hierarchical tags
- **⚡ Batch Processing**: Process thousands of photos efficiently
- **🔒 Privacy-First**: All data stored locally, no cloud dependencies
- **✅ Production Ready**: Complete migration with 20/20 tests passing
---
## 🚀 Quick Start
### Prerequisites
- Python 3.12 or higher
- pip package manager
- Node.js 18+ and npm
- Virtual environment (recommended)
### Installation
@@ -42,46 +41,57 @@ cd punimtag
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
# Install Python dependencies
pip install -r requirements.txt
# Install frontend dependencies
cd frontend
npm install
cd ..
```
### Database Setup
```bash
# Generate and run initial migration
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
alembic revision --autogenerate -m "Initial schema"
alembic upgrade head
```
This creates the SQLite database at `data/punimtag.db` (default). For PostgreSQL, set the `DATABASE_URL` environment variable.
### Running the Application
#### GUI Dashboard (Recommended)
**Terminal 1 - Backend API:**
```bash
python run_dashboard.py
cd /home/ladmin/Code/punimtag
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
uvicorn src.web.app:app --host 127.0.0.1 --port 8000
```
Or:
**Terminal 2 - Frontend:**
```bash
python src/gui/dashboard_gui.py
cd /home/ladmin/Code/punimtag/frontend
npm run dev
```
#### CLI Interface
```bash
python src/photo_tagger.py --help
```
Then open your browser to **http://localhost:3000**
### First-Time Setup
If you have an existing database from before the DeepFace migration, you need to migrate:
```bash
# IMPORTANT: This will delete all existing data!
python scripts/migrate_to_deepface.py
```
Then re-add your photos and process them with DeepFace.
**Default Login:**
- Username: `admin`
- Password: `admin`
---
## 📖 Documentation
- **[Architecture](docs/ARCHITECTURE.md)**: System design and technical details
- **[Demo Guide](docs/DEMO.md)**: Step-by-step tutorial
- **[Dashboard Guide](docs/README_UNIFIED_DASHBOARD.md)**: GUI reference
- **[Contributing](CONTRIBUTING.md)**: How to contribute
- **[Web Migration Plan](docs/WEBSITE_MIGRATION_PLAN.md)**: Detailed migration roadmap
- **[Phase 1 Status](docs/PHASE1_FOUNDATION_STATUS.md)**: Phase 1 implementation status
- **[Phase 1 Checklist](docs/PHASE1_CHECKLIST.md)**: Complete Phase 1 checklist
---
@@ -90,162 +100,208 @@ Then re-add your photos and process them with DeepFace.
```
punimtag/
├── src/ # Source code
│ ├── core/ # Business logic
│ ├── gui/ # GUI components
│ └── utils/ # Utilities
├── tests/ # Test suite
├── docs/ # Documentation
├── .notes/ # Project notes
└── data/ # Application data
│ ├── web/ # Web backend
│ │ ├── api/ # API routers
│ │ ├── db/ # Database models and session
│ │ ├── schemas/ # Pydantic models
│ │ └── services/ # Business logic services
│ └── core/ # Legacy desktop business logic
├── frontend/ # React frontend
│ ├── src/
│ │ ├── api/ # API client
│ │ ├── components/ # React components
│ │ ├── context/ # React contexts (Auth)
│ │ ├── hooks/ # Custom hooks
│ │ └── pages/ # Page components
│ └── package.json
├── tests/ # Test suite
├── docs/ # Documentation
├── data/ # Application data (database, images)
├── alembic/ # Database migrations
└── deploy/ # Docker deployment configs
```
See [Directory Structure](.notes/directory_structure.md) for details.
---
## 🎮 Usage
## 📊 Current Status
### 1. Import Photos
```bash
# Add photos from a folder
python src/photo_tagger.py scan /path/to/photos
```
### Phase 1: Foundations ✅ **COMPLETE**
### 2. Process Faces
Open the dashboard and click "Process Photos" to detect faces.
**Backend:**
- ✅ FastAPI application with CORS middleware
- ✅ Health, version, and metrics endpoints
- ✅ JWT authentication (login, refresh, user info)
- ✅ Job management endpoints (RQ/Redis integration)
- ✅ API routers for photos, faces, people, tags (placeholders)
- ✅ SQLAlchemy models for all entities
- ✅ Alembic migrations configured and applied
- ✅ Database initialized (SQLite default, PostgreSQL supported)
### 3. Identify People
Use the "Identify" panel to tag faces with names:
- **Quality Filter**: Adjust the quality slider (0-100%) to filter out low-quality faces
- **Unique Faces**: Enable to hide duplicate faces using cosine similarity
- **Date Filters**: Filter faces by date range
- **Navigation**: Browse through unidentified faces with prev/next buttons
- **Photo Viewer**: Click the photo icon to view the full source image
**Frontend:**
- ✅ React + Vite + TypeScript setup
- ✅ Tailwind CSS configured
- ✅ Authentication flow with login page
- ✅ Protected routes with auth context
- ✅ Navigation layout (left sidebar + top bar)
- ✅ All page routes (Dashboard, Scan, Process, Search, Identify, Auto-Match, Tags, Settings)
### 4. Search
Use the "Search" panel to find photos by people, dates, or tags.
**Database:**
- ✅ All tables created: `photos`, `faces`, `people`, `person_embeddings`, `tags`, `photo_tags`
- ✅ Indices configured for performance
- ✅ SQLite database at `data/punimtag.db`
### Next: Phase 2 - Processing & Identify
- Photo import (folder scan and upload)
- Face detection and processing pipeline
- Identify workflow UI
- Auto-match engine
- Scan and Process tab implementations
---
## 🔧 Configuration
### GUI Configuration (Recommended)
Use the dashboard to configure DeepFace settings:
1. Open the dashboard: `python run_dashboard.py`
2. Click "🔍 Process"
3. Select your preferred:
- **Face Detector**: RetinaFace (best), MTCNN, OpenCV, or SSD
- **Recognition Model**: ArcFace (best), Facenet, Facenet512, or VGG-Face
### Database
### Manual Configuration
Edit `src/core/config.py` to customize:
- `DEEPFACE_DETECTOR_BACKEND` - Face detection model (default: `retinaface`)
- `DEEPFACE_MODEL_NAME` - Recognition model (default: `ArcFace`)
- `DEFAULT_FACE_TOLERANCE` - Similarity tolerance (default: `0.6` for DeepFace)
- `DEEPFACE_SIMILARITY_THRESHOLD` - Minimum similarity percentage (default: `60`)
- `MIN_FACE_QUALITY` - Minimum face quality score (default: `0.3`)
- Batch sizes and other processing thresholds
**SQLite (Default for Development):**
```bash
# Default location: data/punimtag.db
# No configuration needed
```
**PostgreSQL (Production):**
```bash
export DATABASE_URL=postgresql+psycopg2://user:password@host:port/database
```
### Environment Variables
```bash
# Database (optional, defaults to SQLite)
DATABASE_URL=sqlite:///data/punimtag.db
# JWT Secrets (change in production!)
SECRET_KEY=your-secret-key-here
# Single-user credentials (change in production!)
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin
```
---
## 🧪 Testing
```bash
# Run all migration tests (20 tests total)
python tests/test_phase1_schema.py # Phase 1: Database schema (5 tests)
python tests/test_phase2_config.py # Phase 2: Configuration (5 tests)
python tests/test_phase3_deepface.py # Phase 3: Core processing (5 tests)
python tests/test_phase4_gui.py # Phase 4: GUI integration (5 tests)
python tests/test_deepface_integration.py # Phase 6: Integration tests (5 tests)
# Backend tests (to be implemented)
cd /home/ladmin/Code/punimtag
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
pytest tests/
# Run DeepFace GUI test (working example)
python tests/test_deepface_gui.py
# All tests should pass ✅ (20/20 passing)
# Frontend tests (to be implemented)
cd frontend
npm test
```
---
## 🗺️ Roadmap
### Current (v1.1 - DeepFace Edition) ✅
- ✅ Complete DeepFace migration (all 6 phases)
- ✅ Unified dashboard interface
- ✅ ArcFace recognition model (512-dim embeddings)
- ✅ RetinaFace detection (state-of-the-art)
- ✅ Multiple detector/model options (GUI selectable)
- ✅ Cosine similarity matching
- ✅ Face confidence scores and quality metrics
- ✅ Quality filtering in Identify panel (adjustable 0-100%)
- ✅ Unique faces detection (cosine similarity-based deduplication)
- ✅ Enhanced thumbnail display (100x100px)
- ✅ External system photo viewer integration
- ✅ Improved auto-match save responsiveness
- ✅ Metadata display (detector/model info in GUI)
- ✅ Enhanced accuracy and reliability
- ✅ Comprehensive test coverage (20/20 tests passing)
### ✅ Phase 1: Foundations (Complete)
- FastAPI backend scaffold
- React frontend scaffold
- Authentication system
- Database setup
- Basic API endpoints
### Next (v1.2)
- 📋 GPU acceleration for faster processing
- 📋 Performance optimization
- 📋 Enhanced GUI features
- 📋 Batch processing improvements
### 🔄 Phase 2: Processing & Identify (In Progress)
- Photo import (scan/upload)
- DeepFace processing pipeline
- Identify workflow UI
- Auto-match engine
- Scan and Process tabs
### Future (v2.0+)
- Web interface
- Cloud storage integration
- Mobile app
- Video face detection
- Face clustering (unsupervised)
- Age estimation
- Emotion detection
### 📋 Phase 3: Search & Tags
- Search endpoints with filters
- Tag management UI
- Virtualized photo grid
- Advanced filtering
### 🎨 Phase 4: Polish & Release
- Performance optimization
- Accessibility improvements
- Production deployment
- Documentation
---
## 🤝 Contributing
## 🏗️ Architecture
We welcome contributions! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
**Backend:**
- **Framework**: FastAPI (Python 3.12+)
- **Database**: SQLite (dev), PostgreSQL (production)
- **ORM**: SQLAlchemy 2.0
- **Migrations**: Alembic
- **Jobs**: Redis + RQ
- **Auth**: JWT (python-jose)
### Quick Contribution Guide
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request
**Frontend:**
- **Framework**: React 18 + TypeScript
- **Build Tool**: Vite
- **Styling**: Tailwind CSS
- **State**: React Query + Context API
- **Routing**: React Router
**Deployment:**
- Docker Compose for local development
- Containerized services for production
---
## 📊 Current Status
## 📦 Dependencies
- **Version**: 1.1 (DeepFace Edition)
- **Face Detection**: DeepFace with RetinaFace (state-of-the-art)
- **Recognition Model**: ArcFace (512-dimensional embeddings)
- **Database**: SQLite with DeepFace schema and metadata columns
- **GUI**: Tkinter with model selection and metadata display
- **Platform**: Cross-platform (Linux, Windows, macOS)
- **Migration Status**: ✅ Complete (all 6 phases done, 20/20 tests passing)
- **Test Coverage**: 100% (20 tests across 6 phases)
- **Production Ready**: Yes ✅
**Backend:**
- `fastapi==0.115.0`
- `uvicorn[standard]==0.30.6`
- `pydantic==2.9.1`
- `SQLAlchemy==2.0.36`
- `alembic==1.13.2`
- `python-jose[cryptography]==3.3.0`
- `redis==5.0.8`
- `rq==1.16.2`
- `deepface>=0.0.79`
- `tensorflow>=2.13.0`
**Frontend:**
- `react==18.2.0`
- `react-router-dom==6.20.0`
- `@tanstack/react-query==5.8.4`
- `axios==1.6.2`
- `tailwindcss==3.3.5`
---
## 🔒 Security
- JWT-based authentication with refresh tokens
- Password hashing (to be implemented in production)
- CORS configured for development (restrict in production)
- SQL injection prevention via SQLAlchemy ORM
- Input validation via Pydantic schemas
**⚠️ Note**: Default credentials (`admin`/`admin`) are for development only. Change in production!
---
## 🐛 Known Limitations
- Processing ~2-3x slower than old face_recognition (but much more accurate!)
- Large databases (>50K photos) may experience slowdown
- No GPU acceleration yet (CPU-only processing)
- First run downloads models (~100MB+)
- Existing databases require migration (data will be lost)
See [Task List](.notes/task_list.md) for all tracked issues.
## 📦 Model Downloads
On first run, DeepFace will download required models:
- ArcFace model (~100MB)
- RetinaFace detector (~1.5MB)
- Models stored in `~/.deepface/weights/`
- Requires internet connection for first run only
- Single-user mode only (multi-user support planned)
- SQLite for development (PostgreSQL recommended for production)
- No password hashing yet (plain text comparison - fix before production)
- GPU acceleration not yet implemented
- Large databases (>50K photos) may require optimization
---
@@ -266,39 +322,20 @@ PunimTag Development Team
- **DeepFace** library by Sefik Ilkin Serengil - Modern face recognition framework
- **ArcFace** - Additive Angular Margin Loss for Deep Face Recognition
- **RetinaFace** - State-of-the-art face detection
- TensorFlow, OpenCV, NumPy, and Pillow teams
- All contributors and users
## 📚 Technical Details
### Face Recognition Technology
- **Detection**: RetinaFace (default), MTCNN, OpenCV, or SSD
- **Model**: ArcFace (512-dim), Facenet (128-dim), Facenet512 (512-dim), or VGG-Face (2622-dim)
- **Similarity**: Cosine similarity (industry standard for deep learning embeddings)
- **Accuracy**: Significantly improved over previous face_recognition library
### Migration Documentation
- [Phase 1: Database Schema](PHASE1_COMPLETE.md) - Database updates with DeepFace columns
- [Phase 2: Configuration](PHASE2_COMPLETE.md) - Configuration settings for DeepFace
- [Phase 3: Core Processing](PHASE3_COMPLETE.md) - Face processing with DeepFace
- [Phase 4: GUI Integration](PHASE4_COMPLETE.md) - GUI updates and metadata display
- [Phase 5 & 6: Dependencies and Testing](PHASE5_AND_6_COMPLETE.md) - Final validation
- [Complete Migration Summary](DEEPFACE_MIGRATION_COMPLETE_SUMMARY.md) - Full overview
- [Original Migration Plan](.notes/deepface_migration_plan.md) - Detailed plan
- TensorFlow, React, FastAPI, and all open-source contributors
---
## 📧 Contact
## 📧 Support
[Add contact information]
---
## ⭐ Star History
If you find this project useful, please consider giving it a star!
For questions or issues:
1. Check documentation in `docs/`
2. See [Phase 1 Checklist](docs/PHASE1_CHECKLIST.md) for implementation status
3. Review [Migration Plan](docs/WEBSITE_MIGRATION_PLAN.md) for roadmap
---
**Made with ❤️ for photo enthusiasts**
*For the desktop version, see [README_DESKTOP.md](README_DESKTOP.md)*