ilia/punimtag
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add new analysis documents and update installation scripts for backend integration

Browse Source 
This commit introduces several new analysis documents, including Auto-Match Load Performance Analysis, Folder Picker Analysis, Monorepo Migration Summary, and various performance analysis documents. Additionally, the installation scripts are updated to reflect changes in backend service paths, ensuring proper integration with the new backend structure. These enhancements provide better documentation and streamline the setup process for users.
This commit is contained in:
Tanya
2025-12-30 15:04:32 -05:00
parent 12c62f1deb
commit 68d280e8f5
140 changed files with 5101 additions and 933 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+213 -85
View File
@@ -4,6 +4,8 @@
A fast, simple, and modern web application for organizing and tagging photos using state-of-the-art DeepFace AI with ArcFace recognition model.
**Monorepo Structure:** This project contains both the admin interface (React) and viewer interface (Next.js) in a unified repository for easier maintenance and setup.
---
## 🎯 Features
@@ -57,10 +59,16 @@ The script will:
- ✅ Set up PostgreSQL databases (main + auth)
- ✅ Create Python virtual environment
- ✅ Install all Python dependencies
- ✅ Install all frontend dependencies
- ✅ Install all frontend dependencies (admin-frontend and viewer-frontend)
- ✅ Create `.env` configuration files
- ✅ Create necessary data directories
**Note:** After installation, you'll need to generate Prisma clients for the viewer-frontend:
```bash
cd viewer-frontend
npx prisma generate
```
**Note:** On macOS or other systems, the script will skip system dependency installation. You'll need to install PostgreSQL and Redis manually.
#### Option 2: Manual Installation
@@ -78,17 +86,24 @@ source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
# Install frontend dependencies
cd frontend
cd admin-frontend
npm install
cd ../viewer-frontend
npm install
# Generate Prisma clients for viewer-frontend (after setting up .env)
npx prisma generate
cd ..
```
### Database Setup
**PostgreSQL (Default - Network Database):**
The application is configured to use PostgreSQL by default. The application requires **two separate databases**:
**Database Configuration:**
The application uses **two separate databases**:
1. **Main database** (`punimtag`) - Stores photos, faces, people, tags, and backend user accounts
- **Default: SQLite** at `data/punimtag.db` (for development)
- **Optional: PostgreSQL** (for production)
2. **Auth database** (`punimtag_auth`) - Stores frontend website user accounts and moderation data
- **Required: PostgreSQL** (always uses PostgreSQL)
Both database connections are configured via the `.env` file.
@@ -140,12 +155,12 @@ Alternatively, use the automated script (requires sudo password):
```
**Configuration:**
The `.env` file contains both database connection strings:
The `.env` file in the project root contains database connection strings:
```bash
# Main application database
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
# Main application database (SQLite - default for development)
DATABASE_URL=sqlite:///data/punimtag.db
# Auth database (for frontend website users)
# Auth database (PostgreSQL - always required for frontend website users)
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
```
@@ -153,23 +168,26 @@ DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:543
The database and all tables are automatically created on first startup. No manual migration is needed!
The web application will:
- Connect to PostgreSQL using the `.env` configuration
- Connect to the database using the `.env` configuration
- Create all required tables with the correct schema on startup
- Match the desktop version schema exactly for compatibility
**Manual Setup (Optional):**
If you need to reset the database or create it manually:
**Note:** The main database uses SQLite by default for easier development. For production, you can switch to PostgreSQL by updating `DATABASE_URL` in `.env`.
**SQLite (Default - Local Database):**
The main database uses SQLite by default for development. The `.env` file should contain:
```bash
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
# Recreate all tables from models
python scripts/recreate_tables_web.py
# Main database (SQLite - default for development)
DATABASE_URL=sqlite:///data/punimtag.db
# Or use absolute path:
# DATABASE_URL=file:/home/ladmin/code/punimtag/data/punimtag.db
```
**SQLite (Alternative - Local Database):**
To use SQLite instead of PostgreSQL, comment out or remove the `DATABASE_URL` line in `.env`, or set it to:
**PostgreSQL (Optional - for Production):**
To use PostgreSQL for the main database instead, set:
```bash
DATABASE_URL=sqlite:///data/punimtag.db
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
```
**Database Schema:**
@@ -221,27 +239,46 @@ The separate auth database (`punimtag_auth`) stores frontend website user accoun
redis-server
```
#### Option 1: Manual Start (Recommended for Development)
#### Option 1: Using Helper Scripts (Recommended)
**Terminal 1 - Backend API:**
**Terminal 1 - Backend API + Worker:**
```bash
cd /home/ladmin/Code/punimtag
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
uvicorn src.web.app:app --host 127.0.0.1 --port 8000
cd punimtag
./run_api_with_worker.sh
```
This script will:
- Check if Redis is running (start it if needed)
- Ensure database schema is up to date
- Start the RQ worker in the background
- Start the FastAPI server
- Handle cleanup on Ctrl+C
You should see:
```
✅ Database already initialized (7 tables exist)
✅ RQ worker started in background subprocess (PID: ...)
INFO: Started server process
INFO: Uvicorn running on http://127.0.0.1:8000
✅ Database schema ready
🚀 Starting RQ worker...
🚀 Starting FastAPI server...
✅ Server running on http://127.0.0.1:8000
✅ Worker running (PID: ...)
✅ API running (PID: ...)
```
**Terminal 2 - Frontend:**
**Alternative: Start backend only (without worker):**
```bash
cd /home/ladmin/Code/punimtag/frontend
cd punimtag
./start_backend.sh
```
**Stop the backend:**
```bash
cd punimtag
./stop_backend.sh
```
**Terminal 2 - Admin Frontend:**
```bash
cd punimtag/admin-frontend
npm run dev
```
@@ -251,44 +288,70 @@ VITE v5.4.21 ready in 811 ms
➜ Local: http://localhost:3000/
```
#### Option 2: Using Helper Script (Backend + Worker)
**Terminal 1 - Backend API + Worker:**
**Terminal 3 - Viewer Frontend (Optional):**
```bash
cd /home/ladmin/Code/punimtag
./run_api_with_worker.sh
```
This script will:
- Check if Redis is running (start it if needed)
- Start the RQ worker in the background
- Start the FastAPI server
- Handle cleanup on Ctrl+C
**Terminal 2 - Frontend:**
```bash
cd /home/ladmin/Code/punimtag/frontend
cd punimtag/viewer-frontend
# Generate Prisma clients (only needed once or after schema changes)
npx prisma generate
npm run dev
```
#### Access the Application
You should see:
```
▲ Next.js 16.1.1 (Turbopack)
- Local: http://localhost:3001/
```
1. Open your browser to **http://localhost:3000**
2. Login with default credentials:
- Username: `admin`
- Password: `admin`
3. API documentation available at **http://127.0.0.1:8000/docs**
#### Option 2: Manual Start
**Terminal 1 - Backend API:**
```bash
cd punimtag
source venv/bin/activate
export PYTHONPATH="$(pwd)"
python3 -m uvicorn backend.app:app --host 127.0.0.1 --port 8000 --reload
```
**Note:** If you encounter warnings about "Electron/Chromium" when running `uvicorn`, use `python3 -m uvicorn` instead, or use the helper scripts above.
**Terminal 2 - Admin Frontend:**
```bash
cd punimtag/admin-frontend
npm run dev
```
**Terminal 3 - Viewer Frontend (Optional):**
```bash
cd punimtag/viewer-frontend
npx prisma generate # Only needed once or after schema changes
npm run dev
```
#### Access the Applications
1. **Admin Interface**: Open your browser to **http://localhost:3000**
- Login with default credentials:
- Username: `admin`
- Password: `admin`
2. **Viewer Interface** (Optional): Open your browser to **http://localhost:3001**
- Public photo viewing interface
- Separate authentication system
3. **API Documentation**: Available at **http://127.0.0.1:8000/docs**
#### Troubleshooting
**Port 8000 already in use:**
```bash
# Find and kill the process using port 8000
# Use the stop script
cd punimtag
./stop_backend.sh
# Or manually find and kill the process
lsof -i :8000
kill <PID>
# Or use pkill
pkill -f "uvicorn.*app"
pkill -f "uvicorn.*backend.app"
```
**Port 3000 already in use:**
@@ -297,7 +360,7 @@ pkill -f "uvicorn.*app"
lsof -i :3000
kill <PID>
# Or change the port in frontend/vite.config.ts
# Or change the port in admin-frontend/vite.config.ts
```
**Redis not running:**
@@ -306,17 +369,37 @@ kill <PID>
sudo systemctl start redis-server
# Or
redis-server
# Verify Redis is running
redis-cli ping # Should respond with "PONG"
```
**Worker module not found error:**
If you see `ModuleNotFoundError: No module named 'backend'`:
- Make sure you're using the helper scripts (`./run_api_with_worker.sh` or `./start_backend.sh`)
- These scripts set PYTHONPATH correctly
- If running manually, ensure `export PYTHONPATH="$(pwd)"` is set
**Python/Cursor interception warnings:**
If you see warnings about "Electron/Chromium" when running `uvicorn`:
- Use `python3 -m uvicorn` instead of just `uvicorn`
- Or use the helper scripts which handle this automatically
**Database issues:**
```bash
# Recreate all tables (WARNING: This will delete all data!)
cd /home/ladmin/Code/punimtag
source venv/bin/activate
export PYTHONPATH=/home/ladmin/Code/punimtag
python scripts/recreate_tables_web.py
# The database is automatically created on first startup
# If you need to reset it, delete the database file:
rm data/punimtag.db
# The schema will be recreated on next startup
```
**Viewer frontend shows 0 photos:**
- Make sure the database has photos (import them via admin frontend)
- Verify `DATABASE_URL` in `viewer-frontend/.env` points to the correct database
- Ensure Prisma client is generated: `cd viewer-frontend && npx prisma generate`
- Check that photos are marked as `processed: true` in the database
#### Important Notes
- The database and tables are **automatically created on first startup** - no manual setup needed!
@@ -340,14 +423,16 @@ python scripts/recreate_tables_web.py
```
punimtag/
├── src/ # Source code
│ ├── web/ # Web backend
│ │ ├── api/ # API routers
│ │ ├── db/ # Database models and session
│ │ ├── schemas/ # Pydantic models
│ │ └── services/ # Business logic services
│ └── core/ # Legacy desktop business logic
├── frontend/ # React frontend
├── backend/ # FastAPI backend
│ ├── api/ # API routers
│ ├── db/ # Database models and session
│ ├── schemas/ # Pydantic models
│ ├── services/ # Business logic services
│ ├── constants/ # Constants and configuration
│ ├── utils/ # Utility functions
│ ├── app.py # FastAPI application
│ └── worker.py # RQ worker for background jobs
├── admin-frontend/ # React admin interface
│ ├── src/
│ │ ├── api/ # API client
│ │ ├── components/ # React components
@@ -355,11 +440,20 @@ punimtag/
│ │ ├── hooks/ # Custom hooks
│ │ └── pages/ # Page components
│ └── package.json
├── viewer-frontend/ # Next.js viewer interface
│ ├── app/ # Next.js app router
│ ├── components/ # React components
│ ├── lib/ # Utilities and database
│ ├── prisma/ # Prisma schemas
│ └── package.json
├── src/ # Legacy desktop code
│ └── core/ # Legacy desktop business logic
├── tests/ # Test suite
├── docs/ # Documentation
├── data/ # Application data (database, images)
├── alembic/ # Database migrations
└── deploy/ # Docker deployment configs
├── scripts/ # Utility scripts
├── deploy/ # Docker deployment configs
└── package.json # Root package.json for monorepo
```
---
@@ -392,7 +486,9 @@ punimtag/
- ✅ All tables created automatically on startup: `photos`, `faces`, `people`, `person_encodings`, `tags`, `phototaglinkage`
- ✅ Schema matches desktop version exactly for full compatibility
- ✅ Indices configured for performance
- ✅ SQLite database at `data/punimtag.db` (auto-created if missing)
- ✅ SQLite database at `data/punimtag.db` (auto-created if missing, default for development)
- ✅ PostgreSQL support for production deployments
- ✅ Separate auth database (PostgreSQL) for frontend user accounts
### Image Ingestion & Processing
@@ -482,23 +578,23 @@ punimtag/
### Database
**PostgreSQL (Default - Network Database):**
The application uses PostgreSQL by default, configured via the `.env` file:
**SQLite (Default - Local Database):**
The main database uses SQLite by default for development, configured via the `.env` file:
```bash
# Main application database
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
# Main application database (SQLite - default)
DATABASE_URL=sqlite:///data/punimtag.db
# Auth database (for frontend website users)
# Auth database (PostgreSQL - always required for frontend website users)
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
```
**SQLite (Alternative - Local Database):**
To use SQLite instead, comment out or remove the `DATABASE_URL` line in `.env`, or set:
**PostgreSQL (Optional - for Production):**
To use PostgreSQL for the main database instead:
```bash
DATABASE_URL=sqlite:///data/punimtag.db
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
```
**Note:** When using SQLite, the auth database (`DATABASE_URL_AUTH`) should still be configured as PostgreSQL if you need frontend website user authentication features. The auth database is optional but required for full multi-user functionality.
**Note:** The auth database (`DATABASE_URL_AUTH`) always uses PostgreSQL and is required for frontend website user authentication features.
### Environment Variables
@@ -506,10 +602,10 @@ Configuration is managed via the `.env` file in the project root. A `.env.exampl
**Required Configuration:**
```bash
# Database (PostgreSQL by default)
DATABASE_URL=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag
# Database (SQLite by default for development)
DATABASE_URL=sqlite:///data/punimtag.db
# Auth Database (for frontend website user accounts - separate from main database)
# Auth Database (PostgreSQL - always required for frontend website user accounts)
DATABASE_URL_AUTH=postgresql+psycopg2://punimtag:punimtag_password@localhost:5432/punimtag_auth
# JWT Secrets (change in production!)
@@ -523,13 +619,45 @@ ADMIN_PASSWORD=admin
PHOTO_STORAGE_DIR=data/uploads
```
**Frontend Configuration:**
Create a `.env` file in the `frontend/` directory:
**Admin Frontend Configuration:**
Create a `.env` file in the `admin-frontend/` directory:
```bash
# Backend API URL (must be accessible from browsers)
VITE_API_URL=http://127.0.0.1:8000
```
**Viewer Frontend Configuration:**
Create a `.env` file in the `viewer-frontend/` directory:
```bash
# Main database connection (SQLite - matches backend default)
# Use absolute path for SQLite
DATABASE_URL=file:/home/ladmin/code/punimtag/data/punimtag.db
# Auth database connection (PostgreSQL - always required)
DATABASE_URL_AUTH=postgresql://punimtag:punimtag_password@localhost:5432/punimtag_auth
# Write-capable database connection (optional, falls back to DATABASE_URL if not set)
DATABASE_URL_WRITE=file:/home/ladmin/code/punimtag/data/punimtag.db
# NextAuth configuration
NEXTAUTH_URL=http://localhost:3001
NEXTAUTH_SECRET=dev-secret-key-change-in-production
```
**Generate Prisma Clients:**
After setting up the `.env` file, generate the Prisma clients:
```bash
cd viewer-frontend
npx prisma generate
```
**Important:** The viewer frontend uses **SQLite** for the main database (matching the backend default). The Prisma schema is configured for SQLite. If you change the backend to PostgreSQL, you'll need to:
1. Update `viewer-frontend/prisma/schema.prisma` to use `provider = "postgresql"`
2. Update `DATABASE_URL` in `viewer-frontend/.env` to the PostgreSQL connection string
3. Run `npx prisma generate` again
**Note:** The viewer frontend uses the same database as the backend by default. For production deployments, you may want to create separate read-only and write users for better security.
**Note:** The `.env` file is automatically loaded by the application using `python-dotenv`. Environment variables can also be set directly in your shell if preferred.
---