punimtag/docs/RESOURCE_REQUIREMENTS.md

PunimTag Resource Requirements

How the Software Works & What Resources You Need

This document explains how PunimTag works and what infrastructure resources you need to provision for Development, QA, and Production environments.

---

How the Software Works

System Architecture

PunimTag consists of 5 main components that work together:

┌─────────────────────────────────────────────────────────────┐
│                    USER'S BROWSER                            │
│  ┌──────────────────────────────────────────────────────┐   │
│  │  Frontend Website (React)                            │   │
│  │  - User interface for viewing/searching photos       │   │
│  │  - User authentication                               │   │
│  │  - Photo uploads, face identification, tagging      │   │
│  └──────────────┬──────────────────────────────────────┘   │
└──────────────────┼──────────────────────────────────────────┘
                   │ HTTP/HTTPS
                   ▼
┌─────────────────────────────────────────────────────────────┐
│              APPLICATION SERVER                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │  Backend API (FastAPI - Python)                      │   │
│  │  - Handles all requests from frontend                │   │
│  │  - Manages photos, faces, people, tags               │   │
│  │  - Processes authentication                           │   │
│  │  - Port: 8000                                        │   │
│  └──────────────┬──────────────────────────────────────┘   │
│                 │                                           │
│  ┌──────────────▼──────────────────────────────────────┐   │
│  │  Background Worker (Python)                          │   │
│  │  - Processes photos in background                    │   │
│  │  - Detects faces using AI (DeepFace)                │   │
│  │  - Generates face encodings                          │   │
│  │  - CPU-intensive work                                │   │
│  └──────────────┬──────────────────────────────────────┘   │
└──────────────────┼──────────────────────────────────────────┘
                   │
        ┌──────────┼──────────┬──────────┬──────────┐
        │          │          │          │          │
        ▼          ▼          ▼          ▼          ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│  Main    │ │  Auth    │ │  Redis   │ │  Photo   │ │  Web    │
│ Database │ │ Database │ │  Queue   │ │ Storage  │ │ Server  │
│(Postgres)│ │(Postgres)│ │          │ │  (Disk)  │ │(Nginx)  │
│          │ │          │ │          │ │          │ │         │
│ Photos   │ │ Frontend │ │ Job      │ │ Uploaded │ │ Serves  │
│ Faces    │ │ Users    │ │ Queue    │ │ Photos   │ │ Frontend│
│ People   │ │ Accounts │ │          │ │          │ │ Files   │
│ Tags     │ │          │ │          │ │          │ │         │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘

Component Responsibilities

1. Frontend Website (React)

   • What it does: User interface that runs in browsers
   • Users interact with: View photos, search, identify faces, upload photos, tag photos
   • Technology: React + TypeScript, built with Vite
   • Served by: Web server (Nginx/Apache) in production, or dev server for testing

2. Backend API (FastAPI)

   • What it does: Handles all business logic, processes requests from frontend
   • Responsibilities:
     • Authentication and authorization
     • Photo management (upload, scan, delete)
     • Face detection and processing (queues jobs)
     • Person identification and management
     • Tag management
     • Search functionality
   • Technology: Python 3.12+, FastAPI framework
   • Port: 8000 (configurable)

3. Background Worker

   • What it does: Processes heavy tasks in the background
   • Responsibilities:
     • Face detection in photos (uses AI models)
     • Generating face encodings (512-dimensional vectors)
     • Processing photos uploaded by users
   • Technology: Python 3.12+, uses DeepFace AI library
   • Runs: As a separate process, connects to Redis for job queue

4. Main Database (PostgreSQL)

   • What it stores:
     • Photos metadata (paths, dates, file info)
     • Detected faces (locations, encodings, quality scores)
     • People records (names, dates of birth)
     • Tags and photo-tag relationships
   • Technology: PostgreSQL 12+
   • Size: Grows with number of photos and faces

5. Auth Database (PostgreSQL)

   • What it stores:
     • Frontend website user accounts
     • User-uploaded photos (pending approval)
     • User face identifications (pending approval)
     • User tag suggestions (pending approval)
     • Reported photos
   • Technology: PostgreSQL 12+
   • Size: Small, grows with number of users

6. Redis

   • What it does: Job queue for background processing
   • Stores: Background job status and progress
   • Technology: Redis 5.0+
   • Size: Very small (just job metadata)

7. Photo Storage (File System)

   • What it stores: Actual photo and video files
   • Location: Directory on server or network storage
   • Size: Depends on number and size of photos/videos

8. Web Server (Nginx/Apache)

   • What it does: Serves the frontend website files in production
   • Technology: Nginx or Apache
   • Ports: 80 (HTTP), 443 (HTTPS)

---

Resource Requirements by Environment

Development Environment

Purpose: For development, testing new features, debugging

Components Needed:

• 1 Application Server (can run all services)
• 1 Database Server (can host both databases)
• 1 Redis instance (can be on application server)

Server Specifications:

• CPU: 2-4 cores (sufficient for development)
• RAM: 4-8 GB (enough for development workloads)
• Storage:
  • OS + Software: 20 GB
  • Database: 10-50 GB (depends on test data)
  • Photo storage: 50-200 GB (test photos)
  • Total: 80-270 GB

Network:

• Port 8000 (Backend API)
• Port 3000 (Frontend dev server) OR port 80/443 (if using web server)
• Internal network access sufficient

Software:

• Python 3.12+
• Node.js 18+ (for building frontend)
• PostgreSQL 12+
• Redis 5.0+
• Git

Notes:

• Can run everything on a single server
• Lower performance requirements
• May use SQLite for databases (not recommended but possible)
• Development mode frontend (Vite dev server) is fine

---

QA Environment

Purpose: For quality assurance testing, user acceptance testing, staging

Components Needed:

• 1 Application Server
• 1 Database Server (recommended separate, but can be same as app server)
• 1 Redis instance (can be on application server)

Server Specifications:

• CPU: 4-8 cores (should handle moderate load)
• RAM: 8-16 GB (needs to handle concurrent users and processing)
• Storage:
  • OS + Software: 20 GB
  • Database: 50-200 GB (larger test dataset)
  • Photo storage: 200 GB - 1 TB (realistic photo collection)
  • Total: 270 GB - 1.2 TB

Network:

• Port 8000 (Backend API)
• Port 80/443 (Frontend via web server - production-like)
• Should be accessible to QA team (internal network or VPN)

Software:

• Python 3.12+
• Node.js 18+ (for building frontend)
• PostgreSQL 12+
• Redis 5.0+
• Nginx or Apache (for serving frontend)
• Git

Notes:

• Should mirror production setup as closely as possible
• Use production build of frontend (not dev server)
• Should have realistic data volumes for testing
• Performance should be similar to production

---

Production Environment

Purpose: Live system for end users

Components Needed:

• 1-2 Application Servers (can scale horizontally)
• 1 Database Server (dedicated, high performance)
• 1 Redis Server (can be on database server or separate)
• 1 Web Server (for serving frontend, can be on app server or separate)
• Optional: Load balancer (if multiple app servers)

Server Specifications:

Application Server(s):

• CPU: 8-16 cores (more cores = faster face processing)
• RAM: 16-32 GB (handles concurrent users and background processing)
• Storage:
  • OS + Software: 50 GB
  • Application code: 5 GB
  • Logs: 10-50 GB (with rotation)
  • Total: 65-105 GB per server

Database Server:

• CPU: 8-16 cores (database queries and indexing)
• RAM: 16-64 GB (depends on database size, more RAM = better performance)
• Storage:
  • OS + Software: 50 GB
  • Database data: 100 GB - 5 TB+ (depends on photo collection size)
  • Database backups: 100 GB - 5 TB+ (same as data)
  • Total: 250 GB - 10 TB+
• Storage Type: SSD recommended for better performance
• Backup: Automated daily backups recommended

Photo Storage:

• Storage: 500 GB - 50 TB+ (depends on photo collection)
• Location: Can be on database server, separate storage server, or network storage (NAS)
• Type: Network storage (NAS) recommended for large collections

Redis Server:

• CPU: 2-4 cores (lightweight)
• RAM: 2-4 GB (very small data)
• Storage: 10 GB (minimal)
• Can run on database server or application server

Web Server (if separate):

• CPU: 2-4 cores
• RAM: 2-4 GB
• Storage: 10 GB
• Can run on application server

Network:

• Port 8000 (Backend API) - internal or load balancer
• Port 80/443 (Frontend via web server) - public access
• Port 5432 (Database) - internal only
• Port 6379 (Redis) - internal only
• HTTPS/SSL certificate required

Software:

• Python 3.12+
• Node.js 18+ (for building frontend)
• PostgreSQL 12+ (latest stable recommended)
• Redis 5.0+
• Nginx or Apache (for serving frontend)
• SSL certificate (for HTTPS)
• Monitoring tools (optional but recommended)

High Availability (Optional):

• Multiple application servers behind load balancer
• Database replication (master-slave)
• Redis replication
• Automated backups
• Monitoring and alerting

---

Resource Sizing Guidelines

Database Size Estimation

Main Database:

• Per photo: ~1-5 KB (metadata only, photos stored on disk)
• Per face: ~2-10 KB (encoding + metadata)
• Per person: ~1 KB
• Per tag: ~1 KB

Example:

• 10,000 photos with 50,000 faces = ~250-500 MB
• 1,000 people = ~1 MB
• 100 tags = ~100 KB
• Total: ~250-500 MB for 10K photos

Auth Database:

• Per user: ~1 KB
• Per pending photo: ~1 KB
• Total: Usually < 100 MB

Photo Storage Size Estimation

• Average photo: 2-5 MB
• Average video: 50-200 MB
• 10,000 photos = 20-50 GB
• 100,000 photos = 200-500 GB
• 1,000,000 photos = 2-5 TB

CPU Requirements

Face Processing:

• 1 photo with faces: 2-10 seconds (depends on number of faces)
• CPU-intensive: More cores = faster processing
• Background worker uses CPU heavily during processing

API Server:

• Light CPU usage for most requests
• Higher CPU during photo uploads and searches

RAM Requirements

Application Server:

• Base: 2-4 GB (OS + Python + FastAPI)
• Per concurrent user: ~50-100 MB
• Background worker: 2-4 GB (AI models loaded in memory)
• Recommended: 16-32 GB for production

Database Server:

• Base: 2-4 GB (OS + PostgreSQL)
• Database cache: 25-50% of database size (PostgreSQL uses RAM for caching)
• Recommended: 16-64 GB depending on database size

---

Deployment Scenarios

Scenario 1: Small Deployment (Single Server)

For: Small organizations, < 10,000 photos, < 50 users

• 1 Server: Application + Database + Redis + Web Server
• CPU: 8 cores
• RAM: 16 GB
• Storage: 500 GB - 1 TB
• Cost: Low

Scenario 2: Medium Deployment (Separate Database)

For: Medium organizations, 10,000-100,000 photos, 50-200 users

• 1 Application Server: App + Worker + Web Server
• 1 Database Server: Both databases
• CPU: 8-16 cores each
• RAM: 16-32 GB each
• Storage: 1-5 TB (database), 1-5 TB (photos)
• Cost: Medium

Scenario 3: Large Deployment (Fully Separated)

For: Large organizations, 100,000+ photos, 200+ users

• 2+ Application Servers: Behind load balancer
• 1 Database Server: Main database (high performance)
• 1 Database Server: Auth database (can be smaller)
• 1 Redis Server: Job queue
• 1 Web Server: Frontend (or CDN)
• Storage Server/NAS: Photo storage
• CPU: 16+ cores per server
• RAM: 32-64 GB per server
• Storage: 5-50+ TB
• Cost: High

---

Summary Checklist

Development Environment

• 1 server (4-8 GB RAM, 2-4 cores, 100-300 GB storage)
• PostgreSQL (both databases can be on same server)
• Redis
• Python 3.12+, Node.js 18+

QA Environment

• 1-2 servers (8-16 GB RAM, 4-8 cores, 300 GB - 1 TB storage)
• PostgreSQL (separate server recommended)
• Redis
• Web server (Nginx/Apache)
• Python 3.12+, Node.js 18+

Production Environment

• 1-2 application servers (16-32 GB RAM, 8-16 cores, 100 GB storage)
• 1 database server (16-64 GB RAM, 8-16 cores, 500 GB - 10 TB storage)
• Redis server (2-4 GB RAM, can be on database server)
• Web server (2-4 GB RAM, can be on app server)
• Photo storage (500 GB - 50 TB, can be on database server or separate)
• SSL certificate
• Backup solution
• Monitoring (optional but recommended)

---

Questions? Contact us to discuss your specific requirements and we can help you size the infrastructure appropriately.