outreach/PROJECT.md

# 🏗️ Outreach Engine - Technical Architecture

## 📋 Project Overview

The Outreach Engine is a Node.js-based email automation system designed for professional outreach campaigns to law firms. It features a modular architecture with comprehensive testing, tracking, and safety mechanisms.

## 🏛️ System Architecture

### Core Components

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Main App      │    │   Template      │    │   Database      │
│   (index.js)    │◄──►│   Engine        │◄──►│   (SQLite)      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Rate Limiter  │    │   Attachment    │    │   Tracking      │
│   & Error       │    │   Handler       │    │   Server        │
│   Handler       │    └─────────────────┘    └─────────────────┘
└─────────────────┘
```

### Data Flow

1. **Initialization**: Load config → Initialize DB → Load firm data
2. **Email Processing**: Template rendering → Rate limiting → SMTP sending
3. **Tracking**: Open/click events → Database logging → Analytics
4. **Error Handling**: Retry logic → Fallback mechanisms → Logging

## 📁 Folder Structure

```
outreach-engine/
├── 📄 index.js                    # Main application entry point
├── 📁 config/
│   └── 📄 index.js                # Environment configuration management
├── 📁 lib/                        # Core libraries
│   ├── 📄 attachmentHandler.js    # File attachment processing
│   ├── 📄 database.js             # SQLite database operations
│   ├── 📄 errorHandler.js         # Error handling and retry logic
│   ├── 📄 logger.js               # Winston logging configuration
│   ├── 📄 rateLimiter.js          # Email rate limiting
│   ├── 📄 templateEngine.js       # Handlebars template processing
│   └── 📄 trackingServer.js       # HTTP server for tracking
├── 📁 templates/                  # Email templates (Handlebars)
│   ├── 📄 outreach.html           # Main outreach template
│   ├── 📄 campaign-1-saas.html    # SaaS campaign
│   ├── 📄 campaign-2-data-service.html
│   ├── 📄 campaign-3-license.html
│   ├── 📄 campaign-4-gui.html
│   ├── 📄 general-intro.html
│   └── 📄 employment-layer.html
├── 📁 tests/                      # Test suite
│   ├── 📁 lib/
│   │   ├── 📄 errorHandler.test.js
│   │   ├── 📄 rateLimiter.test.js
│   │   └── 📄 templateEngine.test.js
│   ├── 📄 setup.js                # Test configuration
│   ├── 📄 jest.config.js          # Jest configuration
│   └── 📄 test-campaigns.json     # Test campaign data
├── 📁 scripts/                    # Utility scripts
│   ├── 📄 migrate-to-database.js  # JSON to database migration
│   └── 📄 run-campaigns.js        # Campaign testing script
├── 📁 logs/                       # Application logs
├── 📁 data/                       # Generated data files
├── 📁 coverage/                   # Test coverage reports
├── 📄 firm.json                   # Law firm contact data
├── 📄 tests/test-campaigns.json   # Test campaign data
├── 📄 package.json                # Dependencies and scripts
└── 📄 tests/jest.config.js        # Jest testing configuration
```

## 🔧 Dependencies

### Production Dependencies

| Package      | Version | Purpose                      |
| ------------ | ------- | ---------------------------- |
| `nodemailer` | ^7.0.5  | Email sending via SMTP       |
| `handlebars` | ^4.7.8  | Template engine for emails   |
| `sqlite`     | ^5.1.1  | SQLite database driver       |
| `sqlite3`    | ^5.1.7  | SQLite3 bindings             |
| `winston`    | ^3.17.0 | Structured logging           |
| `dotenv`     | ^17.2.0 | Environment variable loading |
| `delay`      | ^6.0.0  | Promise-based delays         |

### Development Dependencies

| Package         | Version | Purpose               |
| --------------- | ------- | --------------------- |
| `jest`          | ^30.0.4 | Testing framework     |
| `@jest/globals` | ^30.0.4 | Jest global functions |

## 🗄️ Database Schema

### Tables

#### `firms`

```sql
CREATE TABLE firms (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  firm_name TEXT NOT NULL,
  location TEXT,
  website TEXT,
  contact_email TEXT UNIQUE,
  state TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

#### `email_sends`

```sql
CREATE TABLE email_sends (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  campaign_id TEXT,
  firm_id INTEGER,
  recipient_email TEXT,
  subject TEXT,
  status TEXT,
  tracking_id TEXT,
  sent_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  FOREIGN KEY (firm_id) REFERENCES firms (id)
);
```

#### `tracking_events`

```sql
CREATE TABLE tracking_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  tracking_id TEXT,
  event_type TEXT, -- 'open' or 'click'
  ip_address TEXT,
  user_agent TEXT,
  url TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
```

## 🔄 Core Workflows

### 1. Application Startup

```mermaid
graph TD
    A[Start] --> B[Load Environment]
    B --> C[Initialize Database]
    C --> D[Load Firm Data]
    D --> E[Start Tracking Server]
    E --> F[Begin Email Campaign]
```

### 2. Email Sending Process

```mermaid
graph TD
    A[Get Next Firm] --> B[Rate Limit Check]
    B --> C[Render Template]
    C --> D[Add Tracking]
    D --> E[Send via SMTP]
    E --> F[Log Success/Failure]
    F --> G[Wait Delay]
    G --> A
```

### 3. Template Processing

```mermaid
graph TD
    A[Load Template] --> B[Compile Handlebars]
    B --> C[Inject Firm Data]
    C --> D[Add GIF if enabled]
    D --> E[Generate Text Version]
    E --> F[Add Tracking Pixels]
    F --> G[Return Email Content]
```

## 🧪 Testing Architecture

### Test Structure

- **Unit Tests**: `tests/lib/*.test.js` - Test individual modules
- **Integration Tests**: Built into main application
- **Coverage**: Jest generates coverage reports in `coverage/`

### Test Categories

1. **Error Handler Tests**: Retry logic, error classification
2. **Rate Limiter Tests**: Delay calculations, pause logic
3. **Template Engine Tests**: Template rendering, data injection

### Test Commands

```bash
npm test              # Run all tests
npm run test:watch    # Watch mode
npm run test:coverage # Generate coverage report
```

## ⚙️ Configuration System

### Environment Loading

- **Auto-detection**: Based on `NODE_ENV`
- **Files**: `.env.development`, `.env.production`
- **Fallback**: Default values for missing variables

### Configuration Categories

1. **Email Settings**: SMTP, credentials, test mode
2. **Rate Limiting**: Delays, limits, pause intervals
3. **Tracking**: Server port, domain, enable/disable
4. **Logging**: Levels, debug mode, file rotation
5. **Templates**: GIF settings, attachment handling

## 🔒 Security & Safety

### Test Mode Protection

- **Fail-safe validation**: Stops if test recipient not set
- **Double verification**: Checks test mode twice before sending
- **Clear logging**: Every email shows test status
- **No production data**: Test mode never uses real firm emails

### Rate Limiting

- **Configurable delays**: 5+ minutes between emails
- **Hourly limits**: 15 emails/hour default
- **Automatic pauses**: Every 50 emails, pause 30 minutes
- **Randomization**: Adds jitter to prevent detection

## 📊 Monitoring & Analytics

### Logging System

- **Winston**: Structured logging with multiple transports
- **File rotation**: Automatic log file management
- **Debug mode**: Detailed SMTP and template information
- **Error tracking**: Comprehensive error logging

### Tracking Analytics

- **Open tracking**: Invisible pixel tracking
- **Click tracking**: URL wrapping with redirects
- **Database storage**: All events stored in SQLite
- **Real-time server**: HTTP server on port 3000

## 🚀 Performance Considerations

### Optimization Features

- **Database indexing**: Fast queries on email and tracking
- **Template caching**: Handlebars templates compiled once
- **Batch processing**: Configurable batch sizes
- **Memory management**: Proper cleanup of resources

### Scalability

- **Modular design**: Easy to extend with new features
- **Database abstraction**: Can switch to other databases
- **Template system**: Easy to add new email templates
- **Configuration**: Environment-based settings

## 🔧 Development Workflow

### Code Organization

- **Separation of concerns**: Each module has single responsibility
- **Error boundaries**: Comprehensive error handling
- **Async/await**: Modern JavaScript patterns
- **ES6 modules**: Clean import/export structure

### Testing Strategy

- **Unit tests**: Test individual functions
- **Integration tests**: Test module interactions
- **End-to-end**: Test complete email sending flow
- **Coverage goals**: Maintain high test coverage

## 📈 Future Enhancements

### Planned Features

- **A/B testing**: Template variant testing
- **Advanced analytics**: Dashboard for campaign metrics
- **API endpoints**: REST API for external integration
- **Web interface**: Admin dashboard for campaign management

### Technical Debt

- **TypeScript migration**: Add type safety
- **Dependency updates**: Keep packages current
- **Performance optimization**: Database query optimization
- **Security hardening**: Additional security measures

## 🐛 Debugging Guide

### Common Issues

1. **SMTP errors**: Check credentials and 2FA settings
2. **Template errors**: Verify Handlebars syntax
3. **Database errors**: Check SQLite file permissions
4. **Rate limiting**: Adjust delay settings

### Debug Tools

- **Debug mode**: `DEBUG=true npm start`
- **Log files**: Check `logs/` directory
- **Database inspection**: Use SQLite browser
- **Network monitoring**: Check tracking server logs

## 📚 API Reference

### Main Functions

- `sendEmails()`: Main email sending loop
- `sendSingleEmail()`: Individual email sending
- `initializeData()`: Database and data setup
- `processFirm()`: Single firm processing

### Library Functions

- `templateEngine.render()`: Template rendering
- `rateLimiter.check()`: Rate limit validation
- `errorHandler.handle()`: Error processing
- `database.logEmailSend()`: Email logging

---

This document provides comprehensive technical information for development, debugging, and AI assistance. For user-focused information, see `README.md`.