Some checks failed
CI / skip-ci-check (push) Successful in 1m29s
CI / skip-ci-check (pull_request) Successful in 1m29s
CI / python-lint (push) Has been cancelled
CI / test-backend (push) Has been cancelled
CI / build (push) Has been cancelled
CI / secret-scanning (push) Has been cancelled
CI / dependency-scan (push) Has been cancelled
CI / sast-scan (push) Has been cancelled
CI / workflow-summary (push) Has been cancelled
CI / lint-and-type-check (push) Has been cancelled
CI / lint-and-type-check (pull_request) Successful in 2m6s
CI / python-lint (pull_request) Successful in 1m47s
CI / test-backend (pull_request) Successful in 3m13s
CI / build (pull_request) Successful in 2m25s
CI / secret-scanning (pull_request) Successful in 1m42s
CI / dependency-scan (pull_request) Successful in 1m33s
CI / sast-scan (pull_request) Successful in 2m42s
CI / workflow-summary (pull_request) Successful in 1m27s
This commit replaces hardcoded sensitive information, such as database passwords and secret keys, in the README and deployment documentation with placeholders and instructions to use environment variables. This change enhances security by preventing exposure of sensitive data in the codebase. Additionally, it updates the database session management to raise an error if the DATABASE_URL environment variable is not set, ensuring proper configuration for development environments.
899 lines
28 KiB
Markdown
899 lines
28 KiB
Markdown
# PunimTag Web
|
|
|
|
**Modern Photo Management and Facial Recognition System**
|
|
|
|
A fast, simple, and modern web application for organizing and tagging photos using state-of-the-art DeepFace AI with ArcFace recognition model.
|
|
|
|
**Monorepo Structure:** This project contains both the admin interface (React) and viewer interface (Next.js) in a unified repository for easier maintenance and setup.
|
|
|
|
---
|
|
|
|
## 🎯 Features
|
|
|
|
- **🌐 Web-Based**: Modern React frontend with FastAPI backend
|
|
- **🔥 DeepFace AI**: State-of-the-art face detection with RetinaFace and ArcFace models
|
|
- **🎯 Superior Accuracy**: 512-dimensional embeddings (4x more detailed than face_recognition)
|
|
- **⚙️ Multiple Detectors**: Choose from RetinaFace, MTCNN, OpenCV, or SSD detectors
|
|
- **🎨 Flexible Models**: Select ArcFace, Facenet, Facenet512, or VGG-Face recognition models
|
|
- **👤 Person Identification**: Identify and tag people across your photo collection
|
|
- **🤖 Smart Auto-Matching**: Intelligent face matching with quality scoring and cosine similarity
|
|
- **📊 Confidence Calibration**: Empirical-based confidence scores for realistic match probabilities
|
|
- **🔍 Advanced Search**: Search by people, dates, tags, and folders
|
|
- **🏷️ Tag Management**: Organize photos with hierarchical tags
|
|
- **⚡ Batch Processing**: Process thousands of photos efficiently
|
|
- **🎯 Unique Faces Filter**: Hide duplicate faces to focus on unique individuals
|
|
- **🔄 Real-time Updates**: Live progress tracking and job status updates
|
|
- **🌐 Network Path Support**: Browse and scan folders on network shares (UNC paths on Windows, mounted shares on Linux)
|
|
- **📁 Native Folder Picker**: Browse button uses native OS folder picker with full absolute path support
|
|
- **🔒 Privacy-First**: All data stored locally, no cloud dependencies
|
|
|
|
---
|
|
|
|
## 🚀 Quick Start
|
|
|
|
### Prerequisites
|
|
|
|
- **Python 3.12 or higher** (with pip)
|
|
- **Node.js 18+ and npm**
|
|
- **PostgreSQL** (required for both development and production)
|
|
- **Redis** (for background job processing)
|
|
- **Python tkinter** (for native folder picker in Scan tab)
|
|
|
|
**Note:** The automated installation script (`./install.sh`) will install PostgreSQL, Redis, and Python tkinter automatically on Ubuntu/Debian systems.
|
|
|
|
### Installation
|
|
|
|
#### Option 1: Automated Installation (Recommended for Linux/Ubuntu/Debian)
|
|
|
|
The automated installation script will install all system dependencies, Python packages, frontend dependencies, and set up databases:
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone <repository-url>
|
|
cd punimtag
|
|
|
|
# Run the installation script
|
|
./install.sh
|
|
```
|
|
|
|
The script will:
|
|
- ✅ Check prerequisites (Python 3.12+, Node.js 18+)
|
|
- ✅ Install system dependencies (PostgreSQL, Redis, Python tkinter) on Ubuntu/Debian
|
|
- ✅ Set up PostgreSQL databases (main + auth)
|
|
- ✅ Create Python virtual environment
|
|
- ✅ Install all Python dependencies
|
|
- ✅ Install all frontend dependencies (admin-frontend and viewer-frontend)
|
|
- ✅ Create `.env` configuration files
|
|
- ✅ Create necessary data directories
|
|
|
|
**Note:** After installation, you'll need to generate Prisma clients for the viewer-frontend:
|
|
```bash
|
|
cd viewer-frontend
|
|
npx prisma generate
|
|
```
|
|
|
|
**Note:** On macOS or other systems, the script will skip system dependency installation. You'll need to install PostgreSQL, Redis, and Python tkinter manually.
|
|
|
|
**Installing tkinter manually:**
|
|
- **Ubuntu/Debian:** `sudo apt install python3-tk`
|
|
- **RHEL/CentOS:** `sudo yum install python3-tkinter`
|
|
- **macOS:** Usually included with Python, but if missing: `brew install python-tk` (if using Homebrew Python)
|
|
- **Windows:** Usually included with Python installation
|
|
|
|
#### Option 2: Manual Installation
|
|
|
|
```bash
|
|
# Clone the repository
|
|
git clone <repository-url>
|
|
cd punimtag
|
|
|
|
# Create and activate virtual environment
|
|
python -m venv venv
|
|
source venv/bin/activate # On Windows: venv\Scripts\activate
|
|
|
|
# Install Python dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Install frontend dependencies
|
|
cd admin-frontend
|
|
npm install
|
|
cd ../viewer-frontend
|
|
npm install
|
|
# Generate Prisma clients for viewer-frontend (after setting up .env)
|
|
npx prisma generate
|
|
cd ..
|
|
```
|
|
|
|
### Database Setup
|
|
|
|
**Database Configuration:**
|
|
The application uses **two separate PostgreSQL databases**:
|
|
1. **Main database** (`punimtag`) - Stores photos, faces, people, tags, and backend user accounts
|
|
- **Required: PostgreSQL**
|
|
2. **Auth database** (`punimtag_auth`) - Stores frontend website user accounts and moderation data
|
|
- **Required: PostgreSQL**
|
|
|
|
Both database connections are configured via the `.env` file.
|
|
|
|
#### Development Database
|
|
|
|
For development, you can use the shared development PostgreSQL server:
|
|
|
|
**Dev PostgreSQL Server:**
|
|
- **Host**: 10.0.10.181
|
|
- **Port**: 5432
|
|
- **User**: ladmin
|
|
- **Password**: [Contact administrator for password]
|
|
|
|
**Development Server:**
|
|
- **Host**: 10.0.10.121
|
|
- **User**: appuser
|
|
- **Password**: [Contact administrator for password]
|
|
|
|
Configure your `.env` file for development:
|
|
```bash
|
|
# Main database (dev)
|
|
DATABASE_URL=postgresql+psycopg2://ladmin:[PASSWORD]@10.0.10.181:5432/punimtag
|
|
|
|
# Auth database (dev)
|
|
DATABASE_URL_AUTH=postgresql+psycopg2://ladmin:[PASSWORD]@10.0.10.181:5432/punimtag_auth
|
|
```
|
|
|
|
**Install PostgreSQL (if not installed):**
|
|
```bash
|
|
# On Ubuntu/Debian:
|
|
sudo apt update && sudo apt install -y postgresql postgresql-contrib
|
|
sudo systemctl start postgresql
|
|
sudo systemctl enable postgresql
|
|
|
|
# Or use the automated setup script:
|
|
./scripts/setup_postgresql.sh
|
|
```
|
|
|
|
**Create Main Database and User:**
|
|
```bash
|
|
sudo -u postgres psql -c "CREATE USER punimtag WITH PASSWORD 'punimtag_password';"
|
|
sudo -u postgres psql -c "CREATE DATABASE punimtag OWNER punimtag;"
|
|
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE punimtag TO punimtag;"
|
|
```
|
|
|
|
**Create Auth Database (for frontend website user accounts):**
|
|
```bash
|
|
sudo -u postgres psql -c "CREATE DATABASE punimtag_auth OWNER punimtag;"
|
|
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE punimtag_auth TO punimtag;"
|
|
```
|
|
|
|
**Note:** The auth database (`punimtag_auth`) stores user accounts for the frontend website, separate from the main application database. Both databases are required for full functionality.
|
|
|
|
**Grant DELETE Permissions on Auth Database Tables:**
|
|
If you encounter permission errors when trying to delete records from the auth database (e.g., when using "Clear database" in the admin panel), grant DELETE permissions:
|
|
|
|
```bash
|
|
# Grant DELETE permission on all auth database tables
|
|
sudo -u postgres psql -d punimtag_auth << 'EOF'
|
|
GRANT DELETE ON TABLE pending_photos TO punimtag;
|
|
GRANT DELETE ON TABLE users TO punimtag;
|
|
GRANT DELETE ON TABLE pending_identifications TO punimtag;
|
|
GRANT DELETE ON TABLE inappropriate_photo_reports TO punimtag;
|
|
EOF
|
|
|
|
# Or grant on a single table:
|
|
sudo -u postgres psql -d punimtag_auth -c "GRANT DELETE ON TABLE pending_photos TO punimtag;"
|
|
```
|
|
|
|
Alternatively, use the automated script (requires sudo password):
|
|
```bash
|
|
./scripts/grant_auth_db_delete_permission.sh
|
|
```
|
|
|
|
**Configuration:**
|
|
The `.env` file in the project root contains database connection strings:
|
|
|
|
**Local Development:**
|
|
```bash
|
|
# Main application database (PostgreSQL - required)
|
|
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
|
|
|
|
# Auth database (PostgreSQL - required for frontend website users)
|
|
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
|
|
```
|
|
|
|
**Development Server:**
|
|
```bash
|
|
# Main database (dev PostgreSQL server)
|
|
DATABASE_URL=postgresql+psycopg2://ladmin:[PASSWORD]@10.0.10.181:5432/punimtag
|
|
|
|
# Auth database (dev PostgreSQL server)
|
|
DATABASE_URL_AUTH=postgresql+psycopg2://ladmin:[PASSWORD]@10.0.10.181:5432/punimtag_auth
|
|
```
|
|
|
|
**Automatic Initialization:**
|
|
The database and all tables are automatically created on first startup. No manual migration is needed!
|
|
|
|
The web application will:
|
|
- Connect to the database using the `.env` configuration
|
|
- Create all required tables with the correct schema on startup
|
|
- Match the desktop version schema exactly for compatibility
|
|
|
|
**Database Schema:**
|
|
The web version uses the **exact same schema** as the desktop version for full compatibility:
|
|
- `photos` - Photo metadata (path, filename, date_taken, processed, media_type)
|
|
- `people` - Person records (first_name, last_name, middle_name, maiden_name, date_of_birth)
|
|
- `faces` - Face detections (encoding, location, quality_score, face_confidence, exif_orientation, excluded)
|
|
- `person_encodings` - Person face encodings for matching
|
|
- `tags` - Tag definitions
|
|
- `phototaglinkage` - Photo-tag relationships (with linkage_type)
|
|
- `users` - Backend user accounts (with password hashing, roles, permissions)
|
|
- `photo_person_linkage` - Direct photo-person associations (for videos)
|
|
- `role_permissions` - Role-based permission matrix
|
|
|
|
**Auth Database Schema:**
|
|
The separate auth database (`punimtag_auth`) stores frontend website user accounts:
|
|
- `users` - Frontend website user accounts (email, password_hash, is_active)
|
|
- `pending_photos` - Photos pending moderation
|
|
- `pending_identifications` - Face identifications pending approval
|
|
- `inappropriate_photo_reports` - Reported photos for review
|
|
|
|
### Running the Application
|
|
|
|
**Prerequisites:**
|
|
- **PostgreSQL** must be installed and running (see Database Setup section above)
|
|
- **Redis** must be installed and running (for background jobs)
|
|
|
|
**Install Redis (if not installed):**
|
|
```bash
|
|
# On Ubuntu/Debian:
|
|
sudo apt update && sudo apt install -y redis-server
|
|
sudo systemctl start redis-server
|
|
sudo systemctl enable redis-server # Auto-start on boot
|
|
|
|
# On macOS with Homebrew:
|
|
brew install redis
|
|
brew services start redis
|
|
1
|
|
# Verify Redis is running:
|
|
redis-cli ping # Should respond with "PONG"
|
|
```
|
|
|
|
**Start Redis (if installed but not running):**
|
|
```bash
|
|
# On Linux:
|
|
sudo systemctl start redis-server
|
|
|
|
# Or run directly:
|
|
redis-server
|
|
```
|
|
|
|
#### Option 1: Using Helper Scripts (Recommended)
|
|
|
|
**Terminal 1 - Backend API + Worker:**
|
|
```bash
|
|
cd punimtag
|
|
./run_api_with_worker.sh
|
|
```
|
|
|
|
This script will:
|
|
- Check if Redis is running (start it if needed)
|
|
- Ensure database schema is up to date
|
|
- Start the RQ worker in the background
|
|
- Start the FastAPI server
|
|
- Handle cleanup on Ctrl+C
|
|
|
|
You should see:
|
|
```
|
|
✅ Database schema ready
|
|
🚀 Starting RQ worker...
|
|
🚀 Starting FastAPI server...
|
|
✅ Server running on http://127.0.0.1:8000
|
|
✅ Worker running (PID: ...)
|
|
✅ API running (PID: ...)
|
|
```
|
|
|
|
**Alternative: Start backend only (without worker):**
|
|
```bash
|
|
cd punimtag
|
|
./start_backend.sh
|
|
```
|
|
|
|
**Stop the backend:**
|
|
```bash
|
|
cd punimtag
|
|
./stop_backend.sh
|
|
```
|
|
|
|
**Terminal 2 - Admin Frontend:**
|
|
```bash
|
|
cd punimtag/admin-frontend
|
|
npm run dev
|
|
```
|
|
|
|
You should see:
|
|
```
|
|
VITE v5.4.21 ready in 811 ms
|
|
➜ Local: http://localhost:3000/
|
|
```
|
|
|
|
**Terminal 3 - Viewer Frontend (Optional):**
|
|
```bash
|
|
cd punimtag/viewer-frontend
|
|
# Generate Prisma clients (only needed once or after schema changes)
|
|
npx prisma generate
|
|
npm run dev
|
|
```
|
|
|
|
You should see:
|
|
```
|
|
▲ Next.js 16.1.1 (Turbopack)
|
|
- Local: http://localhost:3001/
|
|
```
|
|
|
|
#### Option 2: Manual Start
|
|
|
|
**Terminal 1 - Backend API:**
|
|
```bash
|
|
cd punimtag
|
|
source venv/bin/activate
|
|
export PYTHONPATH="$(pwd)"
|
|
python3 -m uvicorn backend.app:app --host 127.0.0.1 --port 8000 --reload
|
|
```
|
|
|
|
**Note:** If you encounter warnings about "Electron/Chromium" when running `uvicorn`, use `python3 -m uvicorn` instead, or use the helper scripts above.
|
|
|
|
**Terminal 2 - Admin Frontend:**
|
|
```bash
|
|
cd punimtag/admin-frontend
|
|
npm run dev
|
|
```
|
|
|
|
**Terminal 3 - Viewer Frontend (Optional):**
|
|
```bash
|
|
cd punimtag/viewer-frontend
|
|
npx prisma generate # Only needed once or after schema changes
|
|
npm run dev
|
|
```
|
|
|
|
#### Access the Applications
|
|
|
|
1. **Admin Interface**: Open your browser to **http://localhost:3000**
|
|
- Login with default credentials:
|
|
- Username: `admin`
|
|
- Password: `admin`
|
|
2. **Viewer Interface** (Optional): Open your browser to **http://localhost:3001**
|
|
- Public photo viewing interface
|
|
- Separate authentication system
|
|
3. **API Documentation**: Available at **http://127.0.0.1:8000/docs**
|
|
|
|
#### Troubleshooting
|
|
|
|
**Port 8000 already in use:**
|
|
```bash
|
|
# Use the stop script
|
|
cd punimtag
|
|
./stop_backend.sh
|
|
|
|
# Or manually find and kill the process
|
|
lsof -i :8000
|
|
kill <PID>
|
|
|
|
# Or use pkill
|
|
pkill -f "uvicorn.*backend.app"
|
|
```
|
|
|
|
**Port 3000 already in use:**
|
|
```bash
|
|
# Find and kill the process using port 3000
|
|
lsof -i :3000
|
|
kill <PID>
|
|
|
|
# Or change the port in admin-frontend/vite.config.ts
|
|
```
|
|
|
|
**Redis not running:**
|
|
```bash
|
|
# Start Redis
|
|
sudo systemctl start redis-server
|
|
# Or
|
|
redis-server
|
|
|
|
# Verify Redis is running
|
|
redis-cli ping # Should respond with "PONG"
|
|
```
|
|
|
|
**Worker module not found error:**
|
|
If you see `ModuleNotFoundError: No module named 'backend'`:
|
|
- Make sure you're using the helper scripts (`./run_api_with_worker.sh` or `./start_backend.sh`)
|
|
- These scripts set PYTHONPATH correctly
|
|
- If running manually, ensure `export PYTHONPATH="$(pwd)"` is set
|
|
|
|
**Python/Cursor interception warnings:**
|
|
If you see warnings about "Electron/Chromium" when running `uvicorn`:
|
|
- Use `python3 -m uvicorn` instead of just `uvicorn`
|
|
- Or use the helper scripts which handle this automatically
|
|
|
|
**Database issues:**
|
|
```bash
|
|
# The database is automatically created on first startup
|
|
# If you need to reset it, delete the database file:
|
|
rm data/punimtag.db
|
|
|
|
# The schema will be recreated on next startup
|
|
```
|
|
|
|
**Browse button returns 503 error or doesn't show folder picker:**
|
|
This indicates that Python tkinter is not available. Install it:
|
|
```bash
|
|
# Ubuntu/Debian:
|
|
sudo apt install python3-tk
|
|
|
|
# RHEL/CentOS:
|
|
sudo yum install python3-tkinter
|
|
|
|
# Verify installation:
|
|
python3 -c "import tkinter; print('tkinter available')"
|
|
```
|
|
|
|
**Note:** If running on a remote server without a display, you may need to set the DISPLAY environment variable or use X11 forwarding:
|
|
```bash
|
|
export DISPLAY=:0
|
|
# Or for X11 forwarding:
|
|
export DISPLAY=localhost:10.0
|
|
```
|
|
|
|
**Viewer frontend shows 0 photos:**
|
|
- Make sure the database has photos (import them via admin frontend)
|
|
- Verify `DATABASE_URL` in `viewer-frontend/.env` points to the correct database
|
|
- Ensure Prisma client is generated: `cd viewer-frontend && npx prisma generate`
|
|
- Check that photos are marked as `processed: true` in the database
|
|
|
|
#### Important Notes
|
|
|
|
- The database and tables are **automatically created on first startup** - no manual setup needed!
|
|
- The RQ worker starts automatically in a background subprocess when the API server starts
|
|
- Make sure Redis is running first, or the worker won't start
|
|
- Worker names are unique to avoid conflicts when restarting
|
|
- Photo uploads are stored in `data/uploads` (configurable via `PHOTO_STORAGE_DIR` env var)
|
|
- **DeepFace models download automatically on first use** (can take 5-10 minutes, ~100MB)
|
|
- First run is slower due to model downloads (subsequent runs are faster)
|
|
|
|
---
|
|
|
|
## 📖 Documentation
|
|
|
|
- **[Architecture](docs/ARCHITECTURE.md)**: System design and technical details
|
|
|
|
|
|
*
|
|
|
|
## 🏗️ Project Structure
|
|
|
|
```
|
|
punimtag/
|
|
├── backend/ # FastAPI backend
|
|
│ ├── api/ # API routers
|
|
│ ├── db/ # Database models and session
|
|
│ ├── schemas/ # Pydantic models
|
|
│ ├── services/ # Business logic services
|
|
│ ├── constants/ # Constants and configuration
|
|
│ ├── utils/ # Utility functions
|
|
│ ├── app.py # FastAPI application
|
|
│ └── worker.py # RQ worker for background jobs
|
|
├── admin-frontend/ # React admin interface
|
|
│ ├── src/
|
|
│ │ ├── api/ # API client
|
|
│ │ ├── components/ # React components
|
|
│ │ ├── context/ # React contexts (Auth)
|
|
│ │ ├── hooks/ # Custom hooks
|
|
│ │ └── pages/ # Page components
|
|
│ └── package.json
|
|
├── viewer-frontend/ # Next.js viewer interface
|
|
│ ├── app/ # Next.js app router
|
|
│ ├── components/ # React components
|
|
│ ├── lib/ # Utilities and database
|
|
│ ├── prisma/ # Prisma schemas
|
|
│ └── package.json
|
|
├── src/ # Legacy desktop code
|
|
│ └── core/ # Legacy desktop business logic
|
|
├── tests/ # Test suite
|
|
├── docs/ # Documentation
|
|
├── data/ # Application data (database, images)
|
|
├── scripts/ # Utility scripts
|
|
├── deploy/ # Docker deployment configs
|
|
└── package.json # Root package.json for monorepo
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Current Status
|
|
|
|
### Foundations
|
|
|
|
**Backend:**
|
|
- ✅ FastAPI application with CORS middleware
|
|
- ✅ Health, version, and metrics endpoints
|
|
- ✅ JWT authentication (login, refresh, user info)
|
|
- ✅ Job management endpoints (RQ/Redis integration)
|
|
- ✅ SQLAlchemy models for all entities
|
|
- ✅ Alembic migrations configured and applied
|
|
- ✅ Database initialized (PostgreSQL required)
|
|
- ✅ RQ worker auto-start (starts automatically with API server)
|
|
- ✅ Pending linkage moderation API for user tag suggestions
|
|
|
|
**Frontend:**
|
|
- ✅ React + Vite + TypeScript setup
|
|
- ✅ Tailwind CSS configured
|
|
- ✅ Authentication flow with login page
|
|
- ✅ Protected routes with auth context
|
|
- ✅ Navigation layout (left sidebar + top bar)
|
|
- ✅ All page routes (Dashboard, Scan, Process, Search, Identify, Auto-Match, Tags, Settings)
|
|
- ✅ User Tagged Photos moderation tab for approving/denying pending tag linkages
|
|
|
|
**Database:**
|
|
- ✅ All tables created automatically on startup: `photos`, `faces`, `people`, `person_encodings`, `tags`, `phototaglinkage`
|
|
- ✅ Schema matches desktop version exactly for full compatibility
|
|
- ✅ Indices configured for performance
|
|
- ✅ PostgreSQL database (required for both development and production)
|
|
- ✅ Separate auth database (PostgreSQL) for frontend user accounts
|
|
|
|
### Image Ingestion & Processing
|
|
|
|
**Backend:**
|
|
- ✅ Photo import service with checksum computation
|
|
- ✅ EXIF date extraction and image metadata
|
|
- ✅ Folder scanning with recursive option
|
|
- ✅ File upload support
|
|
- ✅ Background job processing with RQ
|
|
- ✅ Real-time job progress via SSE (Server-Sent Events)
|
|
- ✅ Duplicate detection (by path and checksum)
|
|
- ✅ Photo storage configuration (`PHOTO_STORAGE_DIR`)
|
|
- ✅ **DeepFace pipeline integration**
|
|
- ✅ **Face detection (RetinaFace, MTCNN, OpenCV, SSD)**
|
|
- ✅ **Face embeddings computation (ArcFace, Facenet, Facenet512, VGG-Face)**
|
|
- ✅ **Face processing service with configurable detectors/models**
|
|
- ✅ **EXIF orientation handling**
|
|
- ✅ **Face quality scoring and validation**
|
|
- ✅ **Batch processing with progress tracking**
|
|
- ✅ **Job cancellation support**
|
|
|
|
**Frontend:**
|
|
- ✅ Scan tab UI with folder selection
|
|
- ✅ **Native folder picker (Browse button)** - Uses tkinter for native OS folder selection
|
|
- ✅ **Network path support** - Handles UNC paths (Windows: `\\server\share\folder`) and mounted network shares (Linux: `/mnt/nfs-share/photos`)
|
|
- ✅ **Full absolute path handling** - Automatically normalizes and validates paths
|
|
- ✅ Drag-and-drop file upload
|
|
- ✅ Recursive scan toggle
|
|
- ✅ Real-time job progress with progress bar
|
|
- ✅ Job status monitoring (SSE integration)
|
|
- ✅ Results display (added/existing counts)
|
|
- ✅ Error handling and user feedback
|
|
- ✅ **Process tab UI with configuration controls**
|
|
- ✅ **Detector/model selection dropdowns**
|
|
- ✅ **Batch size configuration**
|
|
- ✅ **Start/Stop processing controls**
|
|
- ✅ **Processing progress display with photo count**
|
|
- ✅ **Results summary (faces detected, faces stored)**
|
|
- ✅ **Job cancellation support**
|
|
|
|
**Worker:**
|
|
- ✅ RQ worker auto-starts with API server
|
|
- ✅ Unique worker names to avoid conflicts
|
|
- ✅ Graceful shutdown handling
|
|
- ✅ **String-based function paths for reliable serialization**
|
|
|
|
### Identify Workflow & Auto-Match
|
|
|
|
**Backend:**
|
|
- ✅ Identify face endpoints with person creation
|
|
- ✅ Auto-match engine with similarity thresholds
|
|
- ✅ Unidentified faces management and filtering
|
|
- ✅ Person creation and linking
|
|
- ✅ Batch identification support
|
|
- ✅ Similar faces search with cosine similarity
|
|
- ✅ Confidence calibration system (empirical-based)
|
|
- ✅ Face unmatch/removal functionality
|
|
- ✅ Batch similarity calculations
|
|
|
|
**Frontend:**
|
|
- ✅ Identify page UI with face navigation
|
|
- ✅ Person creation and editing
|
|
- ✅ Similar faces panel with confidence display
|
|
- ✅ Auto-Match page with person-centric view
|
|
- ✅ Checkbox selection for batch identification
|
|
- ✅ Confidence percentages with color coding
|
|
- ✅ Unique faces filter (hide duplicates)
|
|
- ✅ Date filtering for faces
|
|
- ✅ Real-time face matching and display
|
|
|
|
### PSearch & Tags
|
|
|
|
**Backend:**
|
|
- ✅ Search endpoints with filters (people, dates, tags, folders)
|
|
- ✅ Tag management endpoints (create, update, delete)
|
|
- ✅ Photo-tag linkage system
|
|
- ✅ Advanced filtering and querying
|
|
- ✅ Photo grid endpoints with pagination
|
|
|
|
**Frontend:**
|
|
- ✅ Search page with advanced filters
|
|
- ✅ Tag management UI
|
|
- ✅ Photo grid with virtualized rendering
|
|
- ✅ Filter by people, dates, tags, and folders
|
|
- ✅ Search results display
|
|
|
|
---
|
|
|
|
## 🔧 Configuration
|
|
|
|
### Database
|
|
|
|
**PostgreSQL (Required):**
|
|
Both databases use PostgreSQL. Configure via the `.env` file:
|
|
```bash
|
|
# Main application database (PostgreSQL - required)
|
|
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
|
|
|
|
# Auth database (PostgreSQL - required for frontend website users)
|
|
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
|
|
```
|
|
|
|
### Environment Variables
|
|
|
|
Configuration is managed via the `.env` file in the project root. A `.env.example` template is provided.
|
|
|
|
**Required Configuration:**
|
|
```bash
|
|
# Main Database (PostgreSQL - required)
|
|
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
|
|
|
|
# Auth Database (PostgreSQL - required for frontend website user accounts)
|
|
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
|
|
|
|
# JWT Secrets (change in production!)
|
|
SECRET_KEY=dev-secret-key-change-in-production
|
|
|
|
# Single-user credentials (change in production!)
|
|
ADMIN_USERNAME=admin
|
|
ADMIN_PASSWORD=admin
|
|
|
|
# Photo storage directory (default: data/uploads)
|
|
PHOTO_STORAGE_DIR=data/uploads
|
|
```
|
|
|
|
**Admin Frontend Configuration:**
|
|
Create a `.env` file in the `admin-frontend/` directory:
|
|
```bash
|
|
# Backend API URL (must be accessible from browsers)
|
|
VITE_API_URL=http://127.0.0.1:8000
|
|
```
|
|
|
|
**Viewer Frontend Configuration:**
|
|
Create a `.env` file in the `viewer-frontend/` directory:
|
|
```bash
|
|
# Main database connection (PostgreSQL - required)
|
|
DATABASE_URL=postgresql://punimtag:punimtag_password@localhost:5432/punimtag
|
|
|
|
# Auth database connection (PostgreSQL - required)
|
|
DATABASE_URL_AUTH=postgresql://punimtag:punimtag_password@localhost:5432/punimtag_auth
|
|
|
|
# Write-capable database connection (optional, falls back to DATABASE_URL if not set)
|
|
DATABASE_URL_WRITE=postgresql://punimtag:punimtag_password@localhost:5432/punimtag
|
|
|
|
# NextAuth configuration
|
|
NEXTAUTH_URL=http://localhost:3001
|
|
NEXTAUTH_SECRET=dev-secret-key-change-in-production
|
|
```
|
|
|
|
**Generate Prisma Clients:**
|
|
After setting up the `.env` file, generate the Prisma clients:
|
|
```bash
|
|
cd viewer-frontend
|
|
npx prisma generate
|
|
```
|
|
|
|
**Important:** The viewer frontend uses **PostgreSQL** for the main database (matching the backend). The Prisma schema is configured for PostgreSQL.
|
|
|
|
**Note:** The viewer frontend uses the same database as the backend by default. For production deployments, you may want to create separate read-only and write users for better security.
|
|
|
|
**Note:** The `.env` file is automatically loaded by the application using `python-dotenv`. Environment variables can also be set directly in your shell if preferred.
|
|
|
|
---
|
|
|
|
|
|
---
|
|
|
|
### 🔄 Phase 5: Polish & Release (In Progress)
|
|
- Performance optimization
|
|
- Accessibility improvements
|
|
- Production deployment
|
|
- Documentation updates
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture
|
|
|
|
**Backend:**
|
|
- **Framework**: FastAPI (Python 3.12+)
|
|
- **Database**: PostgreSQL (required)
|
|
- **ORM**: SQLAlchemy 2.0
|
|
- **Configuration**: Environment variables via `.env` file (python-dotenv)
|
|
- **Jobs**: Redis + RQ
|
|
- **Auth**: JWT (python-jose)
|
|
|
|
**Frontend:**
|
|
- **Framework**: React 18 + TypeScript
|
|
- **Build Tool**: Vite
|
|
- **Styling**: Tailwind CSS
|
|
- **State**: React Query + Context API
|
|
- **Routing**: React Router
|
|
|
|
**Deployment:**
|
|
- Docker Compose for local development
|
|
- Containerized services for production
|
|
|
|
---
|
|
|
|
## 📦 Dependencies
|
|
|
|
**Backend:**
|
|
- `fastapi==0.115.0`
|
|
- `uvicorn[standard]==0.30.6`
|
|
- `pydantic==2.9.1`
|
|
- `SQLAlchemy==2.0.36`
|
|
- `alembic==1.13.2`
|
|
- `python-jose[cryptography]==3.3.0`
|
|
- `redis==5.0.8`
|
|
- `rq==1.16.2`
|
|
- `psycopg2-binary==2.9.9` (PostgreSQL driver)
|
|
- `python-multipart==0.0.9` (file uploads)
|
|
- `python-dotenv==1.0.0` (environment variables)
|
|
- `bcrypt==4.1.2` (password hashing)
|
|
- `deepface>=0.0.79`
|
|
- `tensorflow>=2.13.0`
|
|
- `opencv-python>=4.8.0`
|
|
- `retina-face>=0.0.13`
|
|
- `numpy>=1.21.0`
|
|
- `pillow>=8.0.0`
|
|
|
|
**Frontend:**
|
|
- `react==18.2.0`
|
|
- `react-router-dom==6.20.0`
|
|
- `@tanstack/react-query==5.8.4`
|
|
- `axios==1.6.2`
|
|
- `tailwindcss==3.3.5`
|
|
|
|
---
|
|
|
|
## 🔒 Security
|
|
|
|
- JWT-based authentication with refresh tokens
|
|
- Password hashing with bcrypt
|
|
- CORS configured for development (restrict in production)
|
|
- SQL injection prevention via SQLAlchemy ORM
|
|
- Input validation via Pydantic schemas
|
|
- Separate auth database for frontend website user accounts
|
|
|
|
**⚠️ Note**: Default credentials (`admin`/`admin`) are for development only. Change in production!
|
|
|
|
---
|
|
|
|
## 🐛 Known Limitations
|
|
|
|
- Multi-user support with role-based permissions (single-user mode deprecated)
|
|
- PostgreSQL for both development and production
|
|
- GPU acceleration not yet implemented (CPU-only for now)
|
|
- Large databases (>50K photos) may require optimization
|
|
- DeepFace model downloads on first use (can take 5-10 minutes, ~100MB)
|
|
- Face processing is CPU-intensive (~2-3x slower than face_recognition, but more accurate)
|
|
- First run is slower due to model downloads (subsequent runs are faster)
|
|
|
|
---
|
|
|
|
## 📝 License
|
|
|
|
[Add your license here]
|
|
|
|
---
|
|
|
|
## 👥 Authors
|
|
|
|
PunimTag Development Team
|
|
|
|
---
|
|
|
|
|
|
|
|
## 🙏 Acknowledgments
|
|
|
|
- **DeepFace** library by Sefik Ilkin Serengil - Modern face recognition framework
|
|
- **ArcFace** - Additive Angular Margin Loss for Deep Face Recognition
|
|
- **RetinaFace** - State-of-the-art face detection
|
|
- TensorFlow, React, FastAPI, and all open-source contributors
|
|
|
|
---
|
|
|
|
## 🚀 Deployment
|
|
|
|
### Development Server Deployment
|
|
|
|
The project includes scripts for deploying to the development server.
|
|
|
|
**Development Server:**
|
|
- **Host**: 10.0.10.121
|
|
- **User**: appuser
|
|
- **Password**: [Contact administrator for password]
|
|
|
|
**Development Database:**
|
|
- **Host**: 10.0.10.181
|
|
- **Port**: 5432
|
|
- **User**: ladmin
|
|
- **Password**: [Contact administrator for password]
|
|
|
|
#### Build and Deploy to Dev
|
|
|
|
```bash
|
|
# Build all frontends and prepare for deployment
|
|
npm run deploy:dev
|
|
|
|
# Or build individually
|
|
npm run build:admin
|
|
npm run build:viewer
|
|
```
|
|
|
|
The deployment script will:
|
|
1. Build admin-frontend for production
|
|
2. Build viewer-frontend for production
|
|
3. Prepare deployment package
|
|
4. Copy files to deployment directory (ready for manual transfer)
|
|
|
|
#### Manual Deployment Steps
|
|
|
|
1. **Build the applications:**
|
|
```bash
|
|
npm run deploy:dev
|
|
```
|
|
|
|
2. **Transfer files to server:**
|
|
```bash
|
|
# Transfer backend and built frontends
|
|
scp -r backend admin-frontend/dist viewer-frontend/.next appuser@10.0.10.121:/path/to/deployment
|
|
```
|
|
|
|
3. **Set up environment on server:**
|
|
- Create `.env` file with dev database credentials
|
|
- Install Python dependencies: `pip install -r requirements.txt`
|
|
- Set up systemd services or PM2 for process management
|
|
|
|
4. **Start services:**
|
|
- Backend API (FastAPI)
|
|
- RQ Worker
|
|
- Frontend servers (nginx or similar)
|
|
|
|
See `docs/DEPLOYMENT.md` for detailed deployment instructions.
|
|
|
|
### Production Deployment
|
|
|
|
For production deployment:
|
|
1. Update environment variables with production credentials
|
|
2. Configure PostgreSQL connection strings
|
|
3. Set up Redis for background jobs
|
|
4. Configure reverse proxy (nginx)
|
|
5. Set up SSL certificates
|
|
6. Configure firewall rules
|
|
7. Set up monitoring and logging
|
|
|
|
See `docs/DEPLOYMENT.md` for complete production deployment guide.
|
|
|
|
---
|
|
|
|
## 📧 Support
|
|
|
|
For questions or issues:
|
|
1. Check documentation in `docs/`
|
|
2. Review `docs/DEPLOYMENT.md` for deployment questions
|
|
3. Check `docs/ARCHITECTURE.md` for technical details
|
|
|
|
---
|
|
|
|
**Made with ❤️ for photo enthusiasts**
|
|
|
|
|