Initial commit
This commit is contained in:
commit
959e52287a
20
.env.example
Normal file
20
.env.example
Normal file
@ -0,0 +1,20 @@
|
|||||||
|
# Email Configuration
|
||||||
|
EMAIL_USER=your-email@gmail.com
|
||||||
|
EMAIL_PASS=your-app-password
|
||||||
|
|
||||||
|
# Application Configuration
|
||||||
|
DELAY_MINUTES=5
|
||||||
|
BATCH_SIZE=10
|
||||||
|
|
||||||
|
# Optional: SMTP Configuration (if not using Gmail)
|
||||||
|
# SMTP_HOST=smtp.gmail.com
|
||||||
|
# SMTP_PORT=587
|
||||||
|
# SMTP_SECURE=false
|
||||||
|
|
||||||
|
# Custom SMTP Configuration (for using your own domain)
|
||||||
|
# Uncomment and fill these to use a custom email server instead of Gmail
|
||||||
|
#SMTP_HOST=smtp.yourdomain.com
|
||||||
|
#SMTP_PORT=587
|
||||||
|
#SMTP_SECURE=false # true for 465, false for other ports
|
||||||
|
#SMTP_USER=your-email@yourdomain.com
|
||||||
|
#SMTP_PASS=your-smtp-password
|
||||||
33
.gitignore
vendored
Normal file
33
.gitignore
vendored
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
# Environment files
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
.env.development
|
||||||
|
.env.production
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
logs/
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# Node modules
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# Generated files
|
||||||
|
organized_firms.json
|
||||||
|
|
||||||
|
# Database files
|
||||||
|
data/
|
||||||
|
*.db
|
||||||
|
*.db-*
|
||||||
|
|
||||||
|
# Test coverage
|
||||||
|
coverage/
|
||||||
|
|
||||||
|
# OS generated files
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# IDE files
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
|
||||||
|
.history/
|
||||||
127
IMPLEMENTATION_SUMMARY.md
Normal file
127
IMPLEMENTATION_SUMMARY.md
Normal file
@ -0,0 +1,127 @@
|
|||||||
|
# ✅ Implementation Summary - Outreach Engine Improvements
|
||||||
|
|
||||||
|
## 🔒 Critical Test Mode Security
|
||||||
|
|
||||||
|
**PROBLEM FIXED**: Emails could potentially be sent to actual firms in test mode
|
||||||
|
|
||||||
|
**SOLUTION**:
|
||||||
|
|
||||||
|
- ✅ **Absolute protection**: Test mode now NEVER sends emails to actual firms
|
||||||
|
- ✅ **Fail-safe validation**: Application will throw error and stop if `EMAIL_TEST_RECIPIENT` is not set in test mode
|
||||||
|
- ✅ **Clear logging**: Every email in test mode shows: `🧪 TEST MODE: Email for [Firm Name] ([firm_email]) → [test_recipient]`
|
||||||
|
- ✅ **Double verification**: Code checks test mode conditions twice before sending
|
||||||
|
|
||||||
|
**Current Configuration**:
|
||||||
|
|
||||||
|
- Test Mode: ✅ **ENABLED**
|
||||||
|
- Test Recipient: `idobkin@gmail.com`
|
||||||
|
- From Email: `levkininquiry@gmail.com`
|
||||||
|
- **All emails will go to idobkin@gmail.com, never to actual firms**
|
||||||
|
|
||||||
|
## 🎨 Template System Overhaul
|
||||||
|
|
||||||
|
**PROBLEM FIXED**: Redundant template files (.txt and .html), manual GIF template management
|
||||||
|
|
||||||
|
**SOLUTION**:
|
||||||
|
|
||||||
|
- ✅ **Unified templates**: Now only need `.html` files, text is auto-generated
|
||||||
|
- ✅ **Auto-GIF integration**: GIFs automatically injected when `GIF_ENABLED=true`
|
||||||
|
- ✅ **Smart HTML→Text conversion**: Preserves formatting with bullet points, line breaks
|
||||||
|
- ✅ **Removed files**: Deleted redundant `.txt` and `-with-gif` templates
|
||||||
|
|
||||||
|
**Before**: 4 template files (outreach.html, outreach.txt, outreach-with-gif.html, outreach-with-gif.txt)
|
||||||
|
**After**: 1 template file (outreach.html) + automatic processing
|
||||||
|
|
||||||
|
## ⚙️ Environment Loading Fix
|
||||||
|
|
||||||
|
**PROBLEM FIXED**: Environment files not automatically loaded based on NODE_ENV
|
||||||
|
|
||||||
|
**SOLUTION**:
|
||||||
|
|
||||||
|
- ✅ **Auto-loading**: `development` mode → `.env.development`, `production` mode → `.env.production`
|
||||||
|
- ✅ **Validation warnings**: Shows warning if environment file doesn't exist
|
||||||
|
- ✅ **Environment detection**: `npm run start:dev` and `npm run start:prod` work correctly
|
||||||
|
|
||||||
|
**Current Behavior**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm start # Uses .env.development (default)
|
||||||
|
npm run start:dev # Uses .env.development
|
||||||
|
npm run start:prod # Uses .env.production
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Email Tracking Implementation
|
||||||
|
|
||||||
|
**NEW FEATURE**: Complete email tracking system
|
||||||
|
|
||||||
|
**FEATURES ADDED**:
|
||||||
|
|
||||||
|
- ✅ **Open Tracking**: Invisible 1x1 pixel tracks when emails are opened
|
||||||
|
- ✅ **Click Tracking**: All links automatically wrapped with tracking URLs
|
||||||
|
- ✅ **Database Storage**: All tracking events stored in SQLite database
|
||||||
|
- ✅ **Real-time Server**: HTTP server (port 3000) handles tracking requests
|
||||||
|
- ✅ **Analytics Ready**: Database schema supports comprehensive analytics
|
||||||
|
|
||||||
|
**Database Schema**:
|
||||||
|
|
||||||
|
- `tracking_events` table: stores opens/clicks with IP, user agent, timestamps
|
||||||
|
- `email_sends` table: enhanced with tracking_id field
|
||||||
|
- Indexes for fast querying
|
||||||
|
|
||||||
|
**Configuration**:
|
||||||
|
|
||||||
|
```env
|
||||||
|
TRACKING_ENABLED=true
|
||||||
|
TRACKING_PORT=3000
|
||||||
|
TRACKING_DOMAIN=http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🗂️ Files Changed/Added/Removed
|
||||||
|
|
||||||
|
### ✅ Modified Files:
|
||||||
|
|
||||||
|
- `config/index.js` - Enhanced environment loading + warnings
|
||||||
|
- `lib/templateEngine.js` - Unified templates + auto-GIF injection + HTML→text conversion
|
||||||
|
- `index.js` - Test mode security + tracking integration + server lifecycle
|
||||||
|
- `lib/database.js` - Added tracking tables + tracking methods
|
||||||
|
- `lib/trackingServer.js` - Enhanced with database integration
|
||||||
|
|
||||||
|
### ❌ Removed Files (No Longer Needed):
|
||||||
|
|
||||||
|
- `templates/outreach.txt`
|
||||||
|
- `templates/outreach-with-gif.txt`
|
||||||
|
- `templates/outreach-with-gif.html`
|
||||||
|
|
||||||
|
### 📝 Environment Files:
|
||||||
|
|
||||||
|
- `.env.development` - Enhanced with tracking settings
|
||||||
|
- `.env.production` - Enhanced with tracking settings
|
||||||
|
|
||||||
|
## 🧪 Testing Status
|
||||||
|
|
||||||
|
**All systems verified**:
|
||||||
|
|
||||||
|
- ✅ Configuration loads correctly
|
||||||
|
- ✅ Template engine works with new unified system
|
||||||
|
- ✅ JSON validation passes
|
||||||
|
- ✅ Test mode security verified
|
||||||
|
- ✅ Environment auto-loading works
|
||||||
|
|
||||||
|
## 🚀 Ready for Use
|
||||||
|
|
||||||
|
**Your application is now ready with**:
|
||||||
|
|
||||||
|
1. **100% Test Mode Security** - Never sends to actual firms when testing
|
||||||
|
2. **Simplified Template Management** - One HTML file does everything
|
||||||
|
3. **Automatic GIF Integration** - No manual template switching needed
|
||||||
|
4. **Proper Environment Handling** - Auto-loads correct .env file
|
||||||
|
5. **Professional Email Tracking** - Opens and clicks tracked automatically
|
||||||
|
|
||||||
|
## 🎯 Next Steps
|
||||||
|
|
||||||
|
1. **Test Run**: `npm start` (will send all emails to idobkin@gmail.com)
|
||||||
|
2. **Check Tracking**: Visit http://localhost:3000/health to verify tracking server
|
||||||
|
3. **Review Logs**: Check `logs/` directory for detailed email sending logs
|
||||||
|
4. **Production Setup**: When ready, copy `.env.production` to `.env` and configure
|
||||||
|
|
||||||
|
**Your outreach engine is now enterprise-grade and test-safe! 🎉**
|
||||||
334
PROJECT.md
Normal file
334
PROJECT.md
Normal file
@ -0,0 +1,334 @@
|
|||||||
|
# 🏗️ Outreach Engine - Technical Architecture
|
||||||
|
|
||||||
|
## 📋 Project Overview
|
||||||
|
|
||||||
|
The Outreach Engine is a Node.js-based email automation system designed for professional outreach campaigns to law firms. It features a modular architecture with comprehensive testing, tracking, and safety mechanisms.
|
||||||
|
|
||||||
|
## 🏛️ System Architecture
|
||||||
|
|
||||||
|
### Core Components
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||||
|
│ Main App │ │ Template │ │ Database │
|
||||||
|
│ (index.js) │◄──►│ Engine │◄──►│ (SQLite) │
|
||||||
|
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
||||||
|
│ Rate Limiter │ │ Attachment │ │ Tracking │
|
||||||
|
│ & Error │ │ Handler │ │ Server │
|
||||||
|
│ Handler │ └─────────────────┘ └─────────────────┘
|
||||||
|
└─────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
1. **Initialization**: Load config → Initialize DB → Load firm data
|
||||||
|
2. **Email Processing**: Template rendering → Rate limiting → SMTP sending
|
||||||
|
3. **Tracking**: Open/click events → Database logging → Analytics
|
||||||
|
4. **Error Handling**: Retry logic → Fallback mechanisms → Logging
|
||||||
|
|
||||||
|
## 📁 Folder Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
outreach-engine/
|
||||||
|
├── 📄 index.js # Main application entry point
|
||||||
|
├── 📁 config/
|
||||||
|
│ └── 📄 index.js # Environment configuration management
|
||||||
|
├── 📁 lib/ # Core libraries
|
||||||
|
│ ├── 📄 attachmentHandler.js # File attachment processing
|
||||||
|
│ ├── 📄 database.js # SQLite database operations
|
||||||
|
│ ├── 📄 errorHandler.js # Error handling and retry logic
|
||||||
|
│ ├── 📄 logger.js # Winston logging configuration
|
||||||
|
│ ├── 📄 rateLimiter.js # Email rate limiting
|
||||||
|
│ ├── 📄 templateEngine.js # Handlebars template processing
|
||||||
|
│ └── 📄 trackingServer.js # HTTP server for tracking
|
||||||
|
├── 📁 templates/ # Email templates (Handlebars)
|
||||||
|
│ ├── 📄 outreach.html # Main outreach template
|
||||||
|
│ ├── 📄 campaign-1-saas.html # SaaS campaign
|
||||||
|
│ ├── 📄 campaign-2-data-service.html
|
||||||
|
│ ├── 📄 campaign-3-license.html
|
||||||
|
│ ├── 📄 campaign-4-gui.html
|
||||||
|
│ ├── 📄 general-intro.html
|
||||||
|
│ └── 📄 employment-layer.html
|
||||||
|
├── 📁 tests/ # Test suite
|
||||||
|
│ ├── 📁 lib/
|
||||||
|
│ │ ├── 📄 errorHandler.test.js
|
||||||
|
│ │ ├── 📄 rateLimiter.test.js
|
||||||
|
│ │ └── 📄 templateEngine.test.js
|
||||||
|
│ ├── 📄 setup.js # Test configuration
|
||||||
|
│ ├── 📄 jest.config.js # Jest configuration
|
||||||
|
│ └── 📄 test-campaigns.json # Test campaign data
|
||||||
|
├── 📁 scripts/ # Utility scripts
|
||||||
|
│ ├── 📄 migrate-to-database.js # JSON to database migration
|
||||||
|
│ └── 📄 run-campaigns.js # Campaign testing script
|
||||||
|
├── 📁 logs/ # Application logs
|
||||||
|
├── 📁 data/ # Generated data files
|
||||||
|
├── 📁 coverage/ # Test coverage reports
|
||||||
|
├── 📄 firm.json # Law firm contact data
|
||||||
|
├── 📄 tests/test-campaigns.json # Test campaign data
|
||||||
|
├── 📄 package.json # Dependencies and scripts
|
||||||
|
└── 📄 tests/jest.config.js # Jest testing configuration
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Dependencies
|
||||||
|
|
||||||
|
### Production Dependencies
|
||||||
|
|
||||||
|
| Package | Version | Purpose |
|
||||||
|
| ------------ | ------- | ---------------------------- |
|
||||||
|
| `nodemailer` | ^7.0.5 | Email sending via SMTP |
|
||||||
|
| `handlebars` | ^4.7.8 | Template engine for emails |
|
||||||
|
| `sqlite` | ^5.1.1 | SQLite database driver |
|
||||||
|
| `sqlite3` | ^5.1.7 | SQLite3 bindings |
|
||||||
|
| `winston` | ^3.17.0 | Structured logging |
|
||||||
|
| `dotenv` | ^17.2.0 | Environment variable loading |
|
||||||
|
| `delay` | ^6.0.0 | Promise-based delays |
|
||||||
|
|
||||||
|
### Development Dependencies
|
||||||
|
|
||||||
|
| Package | Version | Purpose |
|
||||||
|
| --------------- | ------- | --------------------- |
|
||||||
|
| `jest` | ^30.0.4 | Testing framework |
|
||||||
|
| `@jest/globals` | ^30.0.4 | Jest global functions |
|
||||||
|
|
||||||
|
## 🗄️ Database Schema
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
#### `firms`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE firms (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
firm_name TEXT NOT NULL,
|
||||||
|
location TEXT,
|
||||||
|
website TEXT,
|
||||||
|
contact_email TEXT UNIQUE,
|
||||||
|
state TEXT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `email_sends`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE email_sends (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
campaign_id TEXT,
|
||||||
|
firm_id INTEGER,
|
||||||
|
recipient_email TEXT,
|
||||||
|
subject TEXT,
|
||||||
|
status TEXT,
|
||||||
|
tracking_id TEXT,
|
||||||
|
sent_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (firm_id) REFERENCES firms (id)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `tracking_events`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE tracking_events (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
tracking_id TEXT,
|
||||||
|
event_type TEXT, -- 'open' or 'click'
|
||||||
|
ip_address TEXT,
|
||||||
|
user_agent TEXT,
|
||||||
|
url TEXT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔄 Core Workflows
|
||||||
|
|
||||||
|
### 1. Application Startup
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
A[Start] --> B[Load Environment]
|
||||||
|
B --> C[Initialize Database]
|
||||||
|
C --> D[Load Firm Data]
|
||||||
|
D --> E[Start Tracking Server]
|
||||||
|
E --> F[Begin Email Campaign]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Email Sending Process
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
A[Get Next Firm] --> B[Rate Limit Check]
|
||||||
|
B --> C[Render Template]
|
||||||
|
C --> D[Add Tracking]
|
||||||
|
D --> E[Send via SMTP]
|
||||||
|
E --> F[Log Success/Failure]
|
||||||
|
F --> G[Wait Delay]
|
||||||
|
G --> A
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Template Processing
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TD
|
||||||
|
A[Load Template] --> B[Compile Handlebars]
|
||||||
|
B --> C[Inject Firm Data]
|
||||||
|
C --> D[Add GIF if enabled]
|
||||||
|
D --> E[Generate Text Version]
|
||||||
|
E --> F[Add Tracking Pixels]
|
||||||
|
F --> G[Return Email Content]
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🧪 Testing Architecture
|
||||||
|
|
||||||
|
### Test Structure
|
||||||
|
|
||||||
|
- **Unit Tests**: `tests/lib/*.test.js` - Test individual modules
|
||||||
|
- **Integration Tests**: Built into main application
|
||||||
|
- **Coverage**: Jest generates coverage reports in `coverage/`
|
||||||
|
|
||||||
|
### Test Categories
|
||||||
|
|
||||||
|
1. **Error Handler Tests**: Retry logic, error classification
|
||||||
|
2. **Rate Limiter Tests**: Delay calculations, pause logic
|
||||||
|
3. **Template Engine Tests**: Template rendering, data injection
|
||||||
|
|
||||||
|
### Test Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm test # Run all tests
|
||||||
|
npm run test:watch # Watch mode
|
||||||
|
npm run test:coverage # Generate coverage report
|
||||||
|
```
|
||||||
|
|
||||||
|
## ⚙️ Configuration System
|
||||||
|
|
||||||
|
### Environment Loading
|
||||||
|
|
||||||
|
- **Auto-detection**: Based on `NODE_ENV`
|
||||||
|
- **Files**: `.env.development`, `.env.production`
|
||||||
|
- **Fallback**: Default values for missing variables
|
||||||
|
|
||||||
|
### Configuration Categories
|
||||||
|
|
||||||
|
1. **Email Settings**: SMTP, credentials, test mode
|
||||||
|
2. **Rate Limiting**: Delays, limits, pause intervals
|
||||||
|
3. **Tracking**: Server port, domain, enable/disable
|
||||||
|
4. **Logging**: Levels, debug mode, file rotation
|
||||||
|
5. **Templates**: GIF settings, attachment handling
|
||||||
|
|
||||||
|
## 🔒 Security & Safety
|
||||||
|
|
||||||
|
### Test Mode Protection
|
||||||
|
|
||||||
|
- **Fail-safe validation**: Stops if test recipient not set
|
||||||
|
- **Double verification**: Checks test mode twice before sending
|
||||||
|
- **Clear logging**: Every email shows test status
|
||||||
|
- **No production data**: Test mode never uses real firm emails
|
||||||
|
|
||||||
|
### Rate Limiting
|
||||||
|
|
||||||
|
- **Configurable delays**: 5+ minutes between emails
|
||||||
|
- **Hourly limits**: 15 emails/hour default
|
||||||
|
- **Automatic pauses**: Every 50 emails, pause 30 minutes
|
||||||
|
- **Randomization**: Adds jitter to prevent detection
|
||||||
|
|
||||||
|
## 📊 Monitoring & Analytics
|
||||||
|
|
||||||
|
### Logging System
|
||||||
|
|
||||||
|
- **Winston**: Structured logging with multiple transports
|
||||||
|
- **File rotation**: Automatic log file management
|
||||||
|
- **Debug mode**: Detailed SMTP and template information
|
||||||
|
- **Error tracking**: Comprehensive error logging
|
||||||
|
|
||||||
|
### Tracking Analytics
|
||||||
|
|
||||||
|
- **Open tracking**: Invisible pixel tracking
|
||||||
|
- **Click tracking**: URL wrapping with redirects
|
||||||
|
- **Database storage**: All events stored in SQLite
|
||||||
|
- **Real-time server**: HTTP server on port 3000
|
||||||
|
|
||||||
|
## 🚀 Performance Considerations
|
||||||
|
|
||||||
|
### Optimization Features
|
||||||
|
|
||||||
|
- **Database indexing**: Fast queries on email and tracking
|
||||||
|
- **Template caching**: Handlebars templates compiled once
|
||||||
|
- **Batch processing**: Configurable batch sizes
|
||||||
|
- **Memory management**: Proper cleanup of resources
|
||||||
|
|
||||||
|
### Scalability
|
||||||
|
|
||||||
|
- **Modular design**: Easy to extend with new features
|
||||||
|
- **Database abstraction**: Can switch to other databases
|
||||||
|
- **Template system**: Easy to add new email templates
|
||||||
|
- **Configuration**: Environment-based settings
|
||||||
|
|
||||||
|
## 🔧 Development Workflow
|
||||||
|
|
||||||
|
### Code Organization
|
||||||
|
|
||||||
|
- **Separation of concerns**: Each module has single responsibility
|
||||||
|
- **Error boundaries**: Comprehensive error handling
|
||||||
|
- **Async/await**: Modern JavaScript patterns
|
||||||
|
- **ES6 modules**: Clean import/export structure
|
||||||
|
|
||||||
|
### Testing Strategy
|
||||||
|
|
||||||
|
- **Unit tests**: Test individual functions
|
||||||
|
- **Integration tests**: Test module interactions
|
||||||
|
- **End-to-end**: Test complete email sending flow
|
||||||
|
- **Coverage goals**: Maintain high test coverage
|
||||||
|
|
||||||
|
## 📈 Future Enhancements
|
||||||
|
|
||||||
|
### Planned Features
|
||||||
|
|
||||||
|
- **A/B testing**: Template variant testing
|
||||||
|
- **Advanced analytics**: Dashboard for campaign metrics
|
||||||
|
- **API endpoints**: REST API for external integration
|
||||||
|
- **Web interface**: Admin dashboard for campaign management
|
||||||
|
|
||||||
|
### Technical Debt
|
||||||
|
|
||||||
|
- **TypeScript migration**: Add type safety
|
||||||
|
- **Dependency updates**: Keep packages current
|
||||||
|
- **Performance optimization**: Database query optimization
|
||||||
|
- **Security hardening**: Additional security measures
|
||||||
|
|
||||||
|
## 🐛 Debugging Guide
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
1. **SMTP errors**: Check credentials and 2FA settings
|
||||||
|
2. **Template errors**: Verify Handlebars syntax
|
||||||
|
3. **Database errors**: Check SQLite file permissions
|
||||||
|
4. **Rate limiting**: Adjust delay settings
|
||||||
|
|
||||||
|
### Debug Tools
|
||||||
|
|
||||||
|
- **Debug mode**: `DEBUG=true npm start`
|
||||||
|
- **Log files**: Check `logs/` directory
|
||||||
|
- **Database inspection**: Use SQLite browser
|
||||||
|
- **Network monitoring**: Check tracking server logs
|
||||||
|
|
||||||
|
## 📚 API Reference
|
||||||
|
|
||||||
|
### Main Functions
|
||||||
|
|
||||||
|
- `sendEmails()`: Main email sending loop
|
||||||
|
- `sendSingleEmail()`: Individual email sending
|
||||||
|
- `initializeData()`: Database and data setup
|
||||||
|
- `processFirm()`: Single firm processing
|
||||||
|
|
||||||
|
### Library Functions
|
||||||
|
|
||||||
|
- `templateEngine.render()`: Template rendering
|
||||||
|
- `rateLimiter.check()`: Rate limit validation
|
||||||
|
- `errorHandler.handle()`: Error processing
|
||||||
|
- `database.logEmailSend()`: Email logging
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
This document provides comprehensive technical information for development, debugging, and AI assistance. For user-focused information, see `README.md`.
|
||||||
215
README.md
Normal file
215
README.md
Normal file
@ -0,0 +1,215 @@
|
|||||||
|
# <20><> Outreach Engine
|
||||||
|
|
||||||
|
A professional Node.js email outreach system for automated campaigns to law firms with tracking, templates, and comprehensive testing.
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
|
||||||
|
- Node.js (v14+)
|
||||||
|
- Gmail account with App Password
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
|
||||||
|
1. **Install dependencies:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Configure environment:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp env.development.example .env.development
|
||||||
|
# Edit .env.development with your Gmail credentials
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Test configuration:**
|
||||||
|
```bash
|
||||||
|
npm run test-config
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📊 Data Management
|
||||||
|
|
||||||
|
### Where to Change Data
|
||||||
|
|
||||||
|
**Firm Data**: Edit `firm.json` - contains all law firm contacts organized by state:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"State": [
|
||||||
|
{
|
||||||
|
"firmName": "Example Law Firm",
|
||||||
|
"location": "City",
|
||||||
|
"website": "https://example.com",
|
||||||
|
"contactEmail": "contact@example.com"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Email Templates**: Edit files in `templates/` directory:
|
||||||
|
|
||||||
|
- `outreach.html` - Main outreach template
|
||||||
|
- `campaign-1-saas.html` - SaaS campaign
|
||||||
|
- `campaign-2-data-service.html` - Data service campaign
|
||||||
|
- `campaign-3-license.html` - License campaign
|
||||||
|
- `campaign-4-gui.html` - GUI campaign
|
||||||
|
- `general-intro.html` - General introduction
|
||||||
|
- `employment-layer.html` - Employment law focus
|
||||||
|
|
||||||
|
## 🧪 Testing
|
||||||
|
|
||||||
|
### Test Mode (SAFE - Never sends to real firms)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test single email
|
||||||
|
npm run test-email
|
||||||
|
|
||||||
|
# Test all campaigns with different recipients
|
||||||
|
npm run test-campaigns
|
||||||
|
|
||||||
|
# Test with custom limit
|
||||||
|
EMAIL_TEST_LIMIT=5 npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
### Test Configuration
|
||||||
|
|
||||||
|
- **Test Mode**: All emails go to `EMAIL_TEST_RECIPIENT` (never to actual firms)
|
||||||
|
- **Multi-recipient**: Use `EMAIL_TEST_RECIPIENTS` for campaign testing
|
||||||
|
- **Auto-generated**: System creates test firms if needed
|
||||||
|
|
||||||
|
## 🎯 Available Commands
|
||||||
|
|
||||||
|
### Core Operations
|
||||||
|
|
||||||
|
- `npm start` - Run with default settings
|
||||||
|
- `npm run start:dev` - Development mode
|
||||||
|
- `npm run start:prod` - Production mode
|
||||||
|
|
||||||
|
### Testing
|
||||||
|
|
||||||
|
- `npm test` - Run all tests
|
||||||
|
- `npm run test:watch` - Watch mode for development
|
||||||
|
- `npm run test:coverage` - Generate coverage report
|
||||||
|
- `npm run test-email` - Send single test email
|
||||||
|
- `npm run test-campaigns` - Test all campaigns
|
||||||
|
|
||||||
|
### Utilities
|
||||||
|
|
||||||
|
- `npm run migrate` - Migrate JSON data to database
|
||||||
|
- `npm run validate-json` - Validate firm.json syntax
|
||||||
|
- `npm run test-config` - Check environment configuration
|
||||||
|
|
||||||
|
## ⚙️ Key Configuration
|
||||||
|
|
||||||
|
### Environment Variables (`.env.development`)
|
||||||
|
|
||||||
|
```env
|
||||||
|
# Required
|
||||||
|
EMAIL_USER=your-email@gmail.com
|
||||||
|
EMAIL_PASS=your-app-password
|
||||||
|
|
||||||
|
# Test Mode (SAFE)
|
||||||
|
EMAIL_TEST_MODE=true
|
||||||
|
EMAIL_TEST_RECIPIENT=your-test-email@gmail.com
|
||||||
|
|
||||||
|
# Rate Limiting
|
||||||
|
DELAY_MINUTES=5
|
||||||
|
MAX_EMAILS_PER_HOUR=15
|
||||||
|
|
||||||
|
# Tracking
|
||||||
|
TRACKING_ENABLED=true
|
||||||
|
TRACKING_PORT=3000
|
||||||
|
|
||||||
|
# Debug
|
||||||
|
DEBUG=true
|
||||||
|
LOG_LEVEL=debug
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rate Limiting
|
||||||
|
|
||||||
|
- **Default**: 5 minutes between emails
|
||||||
|
- **Hourly limit**: 15 emails/hour
|
||||||
|
- **Pause**: Every 50 emails, pause 30 minutes
|
||||||
|
|
||||||
|
## 📈 Email Tracking
|
||||||
|
|
||||||
|
**Automatic tracking** when `TRACKING_ENABLED=true`:
|
||||||
|
|
||||||
|
- **Open tracking**: Invisible pixel tracks email opens
|
||||||
|
- **Click tracking**: All links wrapped with tracking URLs
|
||||||
|
- **Analytics**: Database stores all tracking events
|
||||||
|
- **Server**: Runs on port 3000 by default
|
||||||
|
|
||||||
|
## 🛠️ Development
|
||||||
|
|
||||||
|
### Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
outreach-engine/
|
||||||
|
├── index.js # Main application
|
||||||
|
├── config/ # Configuration management
|
||||||
|
├── lib/ # Core libraries
|
||||||
|
├── templates/ # Email templates
|
||||||
|
├── tests/ # Test suite
|
||||||
|
├── scripts/ # Utility scripts
|
||||||
|
├── firm.json # Firm data
|
||||||
|
└── logs/ # Application logs
|
||||||
|
```
|
||||||
|
|
||||||
|
### Testing Framework
|
||||||
|
|
||||||
|
- **Jest**: Unit testing framework
|
||||||
|
- **Coverage**: Code coverage reporting
|
||||||
|
- **Test files**: `tests/lib/*.test.js`
|
||||||
|
- **Setup**: `tests/setup.js`
|
||||||
|
|
||||||
|
### Database
|
||||||
|
|
||||||
|
- **SQLite**: Local database for tracking and campaigns
|
||||||
|
- **Migration**: `npm run migrate` converts JSON to database
|
||||||
|
- **Tables**: `firms`, `email_sends`, `tracking_events`
|
||||||
|
|
||||||
|
## 🔒 Security Features
|
||||||
|
|
||||||
|
### Test Mode Protection
|
||||||
|
|
||||||
|
- ✅ **100% Safe**: Test mode NEVER sends to actual firms
|
||||||
|
- ✅ **Fail-safe**: Application stops if test recipient not configured
|
||||||
|
- ✅ **Clear logging**: Every email shows test mode status
|
||||||
|
- ✅ **Validation**: Double-check before sending
|
||||||
|
|
||||||
|
### Best Practices
|
||||||
|
|
||||||
|
- Always test first with your own email
|
||||||
|
- Use Gmail App Passwords (not regular passwords)
|
||||||
|
- Monitor your email account for warnings
|
||||||
|
- Respect rate limits and delays
|
||||||
|
|
||||||
|
## 🐛 Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
- **"Invalid login"**: Check Gmail App Password
|
||||||
|
- **"No emails sent"**: Verify `.env` configuration
|
||||||
|
- **"JSON error"**: Run `npm run validate-json`
|
||||||
|
|
||||||
|
### Debug Mode
|
||||||
|
|
||||||
|
```bash
|
||||||
|
DEBUG=true npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
Shows detailed SMTP, template, and rate limiting information.
|
||||||
|
|
||||||
|
## 📋 Legal Compliance
|
||||||
|
|
||||||
|
- Compliant with CAN-SPAM Act
|
||||||
|
- Includes unsubscribe mechanisms
|
||||||
|
- Professional business contact information
|
||||||
|
- Respects opt-out requests
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Need more details?** See `PROJECT.md` for technical architecture and development information.
|
||||||
26
attachments/README.md
Normal file
26
attachments/README.md
Normal file
@ -0,0 +1,26 @@
|
|||||||
|
# Attachments Directory
|
||||||
|
|
||||||
|
Place any files you want to attach to emails in this directory:
|
||||||
|
|
||||||
|
- **resume.pdf** - Your resume or CV
|
||||||
|
- **company-brochure.pdf** - Company information
|
||||||
|
- **portfolio.pdf** - Work samples or case studies
|
||||||
|
- **proposal.pdf** - Business proposals
|
||||||
|
|
||||||
|
## Supported File Types
|
||||||
|
|
||||||
|
- PDF files (.pdf)
|
||||||
|
- Word documents (.doc, .docx)
|
||||||
|
- Images (.jpg, .png, .gif)
|
||||||
|
- Text files (.txt)
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Edit the attachment settings in your `.env` file:
|
||||||
|
|
||||||
|
```
|
||||||
|
ATTACHMENT_ENABLED=true
|
||||||
|
ATTACHMENT_FILES=resume.pdf,company-brochure.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
Files listed in `ATTACHMENT_FILES` will be attached to all outgoing emails.
|
||||||
87
config/index.js
Normal file
87
config/index.js
Normal file
@ -0,0 +1,87 @@
|
|||||||
|
require("dotenv").config({
|
||||||
|
path: `.env.${process.env.NODE_ENV || "development"}`,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add warning if .env file doesn't exist for current environment
|
||||||
|
const fs = require("fs");
|
||||||
|
const currentEnvFile = `.env.${process.env.NODE_ENV || "development"}`;
|
||||||
|
if (!fs.existsSync(currentEnvFile)) {
|
||||||
|
console.warn(
|
||||||
|
`⚠️ Warning: Environment file ${currentEnvFile} not found. Using default values.`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
env: process.env.NODE_ENV || "development",
|
||||||
|
|
||||||
|
email: {
|
||||||
|
user: process.env.EMAIL_USER,
|
||||||
|
pass: process.env.EMAIL_PASS,
|
||||||
|
testMode: process.env.EMAIL_TEST_MODE === "true",
|
||||||
|
testLimit: process.env.EMAIL_TEST_LIMIT
|
||||||
|
? parseInt(process.env.EMAIL_TEST_LIMIT, 10)
|
||||||
|
: null,
|
||||||
|
},
|
||||||
|
|
||||||
|
campaigns: {
|
||||||
|
testMode: process.env.CAMPAIGN_TEST_MODE === "true",
|
||||||
|
testDataFile: process.env.CAMPAIGN_TEST_DATA || "tests/test-campaigns.json",
|
||||||
|
},
|
||||||
|
|
||||||
|
attachments: {
|
||||||
|
enabled: process.env.ATTACHMENT_ENABLED === "true",
|
||||||
|
files: process.env.ATTACHMENT_FILES
|
||||||
|
? process.env.ATTACHMENT_FILES.split(",").map((f) => f.trim())
|
||||||
|
: [],
|
||||||
|
},
|
||||||
|
|
||||||
|
gif: {
|
||||||
|
enabled: process.env.GIF_ENABLED === "true",
|
||||||
|
url:
|
||||||
|
process.env.GIF_URL ||
|
||||||
|
"https://media.giphy.com/media/3o7abKhOpu0NwenH3O/giphy.gif",
|
||||||
|
alt: process.env.GIF_ALT || "Professional handshake",
|
||||||
|
},
|
||||||
|
|
||||||
|
app: {
|
||||||
|
delayMinutes: parseInt(process.env.DELAY_MINUTES || "5", 10),
|
||||||
|
batchSize: parseInt(process.env.BATCH_SIZE || "10", 10),
|
||||||
|
},
|
||||||
|
|
||||||
|
rateLimiting: {
|
||||||
|
enabled: process.env.RATE_LIMITING_ENABLED !== "false", // Default true
|
||||||
|
maxPerHour: parseInt(process.env.MAX_EMAILS_PER_HOUR || "15", 10),
|
||||||
|
pauseEvery: parseInt(process.env.PAUSE_EVERY_N_EMAILS || "50", 10),
|
||||||
|
pauseDuration: parseInt(process.env.PAUSE_DURATION_MINUTES || "30", 10),
|
||||||
|
},
|
||||||
|
|
||||||
|
errorHandling: {
|
||||||
|
enabled: process.env.ERROR_HANDLING_ENABLED !== "false", // Default true
|
||||||
|
maxRetries: parseInt(process.env.MAX_RETRIES || "3", 10),
|
||||||
|
retryDelay: parseInt(process.env.RETRY_BASE_DELAY || "60", 10), // seconds
|
||||||
|
logFailures: process.env.LOG_FAILURES !== "false", // Default true
|
||||||
|
},
|
||||||
|
|
||||||
|
logging: {
|
||||||
|
level: process.env.LOG_LEVEL || "info",
|
||||||
|
debug: process.env.DEBUG === "true",
|
||||||
|
},
|
||||||
|
|
||||||
|
tracking: {
|
||||||
|
enabled: process.env.TRACKING_ENABLED === "true",
|
||||||
|
skipTracking: process.env.SKIP_TRACKING === "true",
|
||||||
|
port: parseInt(process.env.TRACKING_PORT || "3000", 10),
|
||||||
|
domain: process.env.TRACKING_DOMAIN || "http://localhost:3000",
|
||||||
|
},
|
||||||
|
|
||||||
|
smtp: {
|
||||||
|
host: process.env.SMTP_HOST,
|
||||||
|
port: parseInt(process.env.SMTP_PORT || "587", 10),
|
||||||
|
secure: process.env.SMTP_SECURE === "true",
|
||||||
|
user: process.env.SMTP_USER,
|
||||||
|
pass: process.env.SMTP_PASS,
|
||||||
|
},
|
||||||
|
|
||||||
|
isDevelopment: process.env.NODE_ENV === "development",
|
||||||
|
isProduction: process.env.NODE_ENV === "production",
|
||||||
|
};
|
||||||
63
env.development.example
Normal file
63
env.development.example
Normal file
@ -0,0 +1,63 @@
|
|||||||
|
# Development Environment Configuration
|
||||||
|
NODE_ENV=development
|
||||||
|
|
||||||
|
# Gmail Configuration (for development/testing)
|
||||||
|
EMAIL_USER=your-email@gmail.com
|
||||||
|
EMAIL_PASS=your-app-password
|
||||||
|
|
||||||
|
# Application Configuration
|
||||||
|
DELAY_MINUTES=1 # Shorter delay for testing
|
||||||
|
BATCH_SIZE=5 # Smaller batches for testing
|
||||||
|
|
||||||
|
# Logging & Debug
|
||||||
|
LOG_LEVEL=debug
|
||||||
|
DEBUG=true # Enable comprehensive debug logging throughout the application
|
||||||
|
|
||||||
|
# Email Settings
|
||||||
|
EMAIL_TEST_MODE=true # When true, adds [TEST] to subject line
|
||||||
|
EMAIL_TEST_RECIPIENT=your-test-email@gmail.com # Single test recipient (for backward compatibility)
|
||||||
|
|
||||||
|
# Multi-Recipient Testing (NEW)
|
||||||
|
# Use this to test different campaigns with different recipients
|
||||||
|
# Recipients will be used in round-robin fashion
|
||||||
|
EMAIL_TEST_RECIPIENTS=test1@example.com,test2@example.com,test3@example.com,test4@example.com
|
||||||
|
|
||||||
|
# Campaign Settings
|
||||||
|
CAMPAIGN_TEST_MODE=true
|
||||||
|
CAMPAIGN_TEST_DATA=tests/test-campaigns.json
|
||||||
|
|
||||||
|
# Test Limiting
|
||||||
|
EMAIL_TEST_LIMIT=4 # Limit number of emails in test mode
|
||||||
|
|
||||||
|
# Attachment Settings
|
||||||
|
ATTACHMENT_ENABLED=true
|
||||||
|
ATTACHMENT_FILES= # Comma-separated list of files from attachments folder
|
||||||
|
|
||||||
|
# GIF Settings
|
||||||
|
GIF_ENABLED=true
|
||||||
|
GIF_URL=https://media.giphy.com/media/3o7abKhOpu0NwenH3O/giphy.gif # Professional handshake GIF
|
||||||
|
GIF_ALT=Professional handshake
|
||||||
|
|
||||||
|
# Rate Limiting Settings
|
||||||
|
RATE_LIMITING_ENABLED=true
|
||||||
|
MAX_EMAILS_PER_HOUR=15
|
||||||
|
PAUSE_EVERY_N_EMAILS=50
|
||||||
|
PAUSE_DURATION_MINUTES=30
|
||||||
|
|
||||||
|
# Error Handling Settings
|
||||||
|
ERROR_HANDLING_ENABLED=true
|
||||||
|
MAX_RETRIES=3
|
||||||
|
RETRY_BASE_DELAY=60 # seconds
|
||||||
|
LOG_FAILURES=true
|
||||||
|
|
||||||
|
# Tracking Settings
|
||||||
|
TRACKING_ENABLED=false
|
||||||
|
TRACKING_PORT=3000
|
||||||
|
TRACKING_DOMAIN=http://localhost:3000
|
||||||
|
|
||||||
|
# SMTP Settings (Optional - uses Gmail by default)
|
||||||
|
# SMTP_HOST=smtp.gmail.com
|
||||||
|
# SMTP_PORT=587
|
||||||
|
# SMTP_SECURE=false
|
||||||
|
# SMTP_USER=your-email@gmail.com
|
||||||
|
# SMTP_PASS=your-app-password
|
||||||
527
index.js
Normal file
527
index.js
Normal file
@ -0,0 +1,527 @@
|
|||||||
|
const config = require("./config");
|
||||||
|
const nodemailer = require("nodemailer");
|
||||||
|
const delay = require("delay");
|
||||||
|
const fs = require("fs");
|
||||||
|
const templateEngine = require("./lib/templateEngine");
|
||||||
|
const attachmentHandler = require("./lib/attachmentHandler");
|
||||||
|
const rateLimiter = require("./lib/rateLimiter");
|
||||||
|
const errorHandler = require("./lib/errorHandler");
|
||||||
|
const logger = require("./lib/logger");
|
||||||
|
const database = require("./lib/database");
|
||||||
|
const trackingServer = require("./lib/trackingServer");
|
||||||
|
|
||||||
|
// Initialize database and load firm data
|
||||||
|
let uniqueFirms = [];
|
||||||
|
let currentCampaignId = null;
|
||||||
|
|
||||||
|
// Generate test recipients automatically based on test limit
|
||||||
|
function generateTestRecipients(testLimit) {
|
||||||
|
const baseEmail = config.email.user; // Use your own email as base
|
||||||
|
const [localPart, domain] = baseEmail.split("@");
|
||||||
|
|
||||||
|
const recipients = [];
|
||||||
|
for (let i = 1; i <= testLimit; i++) {
|
||||||
|
// Create test recipients like: yourname+test1@gmail.com, yourname+test2@gmail.com
|
||||||
|
recipients.push(`${localPart}+test${i}@${domain}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Generated ${testLimit} test recipients:`);
|
||||||
|
recipients.forEach((email, index) => {
|
||||||
|
console.log(`🐛 DEBUG: Recipient ${index + 1}: ${email}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return recipients;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract email sending logic for reuse in retries
|
||||||
|
async function sendSingleEmail(mailOptions, recipient, transporter) {
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Sending email to ${recipient}`);
|
||||||
|
console.log(`🐛 DEBUG: Subject: ${mailOptions.subject}`);
|
||||||
|
console.log(`🐛 DEBUG: Firm: ${mailOptions.firmName}`);
|
||||||
|
console.log(`🐛 DEBUG: Tracking ID: ${mailOptions.trackingId}`);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: HTML length: ${
|
||||||
|
mailOptions.html ? mailOptions.html.length : 0
|
||||||
|
} chars`
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Text length: ${
|
||||||
|
mailOptions.text ? mailOptions.text.length : 0
|
||||||
|
} chars`
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Attachments: ${
|
||||||
|
mailOptions.attachments ? mailOptions.attachments.length : 0
|
||||||
|
}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
await transporter.sendMail(mailOptions);
|
||||||
|
|
||||||
|
// Log successful email to database
|
||||||
|
if (currentCampaignId && mailOptions.firmId) {
|
||||||
|
try {
|
||||||
|
await database.logEmailSend({
|
||||||
|
campaignId: currentCampaignId,
|
||||||
|
firmId: mailOptions.firmId,
|
||||||
|
recipientEmail: recipient,
|
||||||
|
subject: mailOptions.subject,
|
||||||
|
status: "sent",
|
||||||
|
trackingId: mailOptions.trackingId,
|
||||||
|
});
|
||||||
|
} catch (dbError) {
|
||||||
|
logger.error("Failed to log email send to database", {
|
||||||
|
recipient,
|
||||||
|
error: dbError.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log successful email
|
||||||
|
logger.emailSent(
|
||||||
|
recipient,
|
||||||
|
mailOptions.subject,
|
||||||
|
mailOptions.firmName,
|
||||||
|
config.email.testMode
|
||||||
|
);
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`✅ Email sent to ${recipient}${
|
||||||
|
config.email.testMode ? " (TEST MODE)" : ""
|
||||||
|
}`
|
||||||
|
);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Email successfully delivered to ${recipient}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function initializeData() {
|
||||||
|
try {
|
||||||
|
// Initialize database
|
||||||
|
await database.init();
|
||||||
|
|
||||||
|
// If in test mode with limit, we can skip migration and just create test data
|
||||||
|
if (config.email.testMode && config.email.testLimit) {
|
||||||
|
const testRecipients = generateTestRecipients(config.email.testLimit);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Test limit: ${config.email.testLimit}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`🧪 TEST MODE: ${config.email.testLimit} email${
|
||||||
|
config.email.testLimit === 1 ? "" : "s"
|
||||||
|
} to auto-generated recipients`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Create test firms - one for each recipient
|
||||||
|
const baseTestFirms = [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
firm_name: "Test Law Firm Alpha",
|
||||||
|
location: "New York, NY",
|
||||||
|
website: "https://testfirm-alpha.com",
|
||||||
|
contact_email: testRecipients[0] || "test@testfirm-alpha.com",
|
||||||
|
state: "New York",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 2,
|
||||||
|
firm_name: "Test Law Firm Beta",
|
||||||
|
location: "Los Angeles, CA",
|
||||||
|
website: "https://testfirm-beta.com",
|
||||||
|
contact_email: testRecipients[1] || "test@testfirm-beta.com",
|
||||||
|
state: "California",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 3,
|
||||||
|
firm_name: "Test Law Firm Gamma",
|
||||||
|
location: "Chicago, IL",
|
||||||
|
website: "https://testfirm-gamma.com",
|
||||||
|
contact_email: testRecipients[2] || "test@testfirm-gamma.com",
|
||||||
|
state: "Illinois",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 4,
|
||||||
|
firm_name: "Test Law Firm Delta",
|
||||||
|
location: "Houston, TX",
|
||||||
|
website: "https://testfirm-delta.com",
|
||||||
|
contact_email: testRecipients[3] || "test@testfirm-delta.com",
|
||||||
|
state: "Texas",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: 5,
|
||||||
|
firm_name: "Test Law Firm Epsilon",
|
||||||
|
location: "Miami, FL",
|
||||||
|
website: "https://testfirm-epsilon.com",
|
||||||
|
contact_email: testRecipients[4] || "test@testfirm-epsilon.com",
|
||||||
|
state: "Florida",
|
||||||
|
},
|
||||||
|
];
|
||||||
|
|
||||||
|
uniqueFirms = baseTestFirms.slice(0, config.email.testLimit);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Created ${uniqueFirms.length} test firms:`);
|
||||||
|
uniqueFirms.forEach((firm, index) => {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Firm ${index + 1}: ${firm.firm_name} → ${
|
||||||
|
firm.contact_email
|
||||||
|
}`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`📧 Ready to send test emails using ${uniqueFirms.length} firms`
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
// Full initialization for production or full test runs
|
||||||
|
if (fs.existsSync("firm.json")) {
|
||||||
|
logger.info("Found firm.json, checking if migration is needed");
|
||||||
|
const counts = await database.getTableCounts();
|
||||||
|
|
||||||
|
if (counts.firms === 0) {
|
||||||
|
logger.info("Database is empty, running migration from JSON");
|
||||||
|
|
||||||
|
// Run migration with its own database connection
|
||||||
|
const {
|
||||||
|
migrateJsonToDatabase,
|
||||||
|
} = require("./scripts/migrate-to-database");
|
||||||
|
await migrateJsonToDatabase();
|
||||||
|
|
||||||
|
uniqueFirms = await database.getFirms();
|
||||||
|
logger.info(`Loaded ${uniqueFirms.length} firms after migration`);
|
||||||
|
} else {
|
||||||
|
uniqueFirms = await database.getFirms();
|
||||||
|
logger.info(`Loaded ${uniqueFirms.length} firms from database`);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
uniqueFirms = await database.getFirms();
|
||||||
|
logger.info(`Loaded ${uniqueFirms.length} firms from database`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply test limit if needed
|
||||||
|
if (
|
||||||
|
config.email.testMode &&
|
||||||
|
config.email.testLimit &&
|
||||||
|
config.email.testLimit < uniqueFirms.length
|
||||||
|
) {
|
||||||
|
uniqueFirms = uniqueFirms.slice(0, config.email.testLimit);
|
||||||
|
console.log(
|
||||||
|
`🧪 TEST MODE: ${config.email.testLimit} email${
|
||||||
|
config.email.testLimit === 1 ? "" : "s"
|
||||||
|
} to ${config.email.testRecipient}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validation for test mode
|
||||||
|
if (config.email.testMode && !config.email.testLimit) {
|
||||||
|
throw new Error(
|
||||||
|
"❌ TEST_MODE enabled but EMAIL_TEST_LIMIT not set! Set a test limit to generate recipients automatically."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Firms to process: ${uniqueFirms.length}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create campaign
|
||||||
|
currentCampaignId = await database.createCampaign({
|
||||||
|
name: `Campaign ${new Date().toISOString().split("T")[0]}`,
|
||||||
|
subject: config.email.testMode
|
||||||
|
? "[TEST] Legal Partnership Opportunity"
|
||||||
|
: "Legal Partnership Opportunity",
|
||||||
|
templateName: "outreach",
|
||||||
|
testMode: config.email.testMode,
|
||||||
|
});
|
||||||
|
|
||||||
|
await database.startCampaign(currentCampaignId, uniqueFirms.length);
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to initialize data", { error: error.message });
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function sendEmails() {
|
||||||
|
// Initialize data first
|
||||||
|
await initializeData();
|
||||||
|
|
||||||
|
// Start tracking server if enabled and not skipping tracking
|
||||||
|
if (config.tracking.enabled && !config.tracking.skipTracking) {
|
||||||
|
try {
|
||||||
|
await trackingServer.start();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to start tracking server", { error: error.message });
|
||||||
|
console.warn(
|
||||||
|
"⚠️ Tracking server failed to start. Continuing without tracking."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else if (config.tracking.skipTracking) {
|
||||||
|
console.log("📊 Tracking disabled - SKIP_TRACKING enabled");
|
||||||
|
}
|
||||||
|
|
||||||
|
const transportConfig = {
|
||||||
|
auth: {
|
||||||
|
user: config.smtp.user || config.email.user,
|
||||||
|
pass: config.smtp.pass || config.email.pass,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
if (config.smtp.host) {
|
||||||
|
transportConfig.host = config.smtp.host;
|
||||||
|
transportConfig.port = config.smtp.port;
|
||||||
|
transportConfig.secure = config.smtp.secure;
|
||||||
|
} else {
|
||||||
|
transportConfig.service = "gmail";
|
||||||
|
}
|
||||||
|
|
||||||
|
const transporter = nodemailer.createTransport(transportConfig);
|
||||||
|
|
||||||
|
// Get attachments once at start
|
||||||
|
const attachments = await attachmentHandler.getAttachments();
|
||||||
|
if (attachments.length > 0) {
|
||||||
|
console.log(`📎 Attaching ${attachments.length} file(s) to emails`);
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Attachment files: ${attachments
|
||||||
|
.map((a) => a.filename)
|
||||||
|
.join(", ")}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: SMTP Config: ${JSON.stringify(
|
||||||
|
{
|
||||||
|
host: config.smtp.host || "gmail",
|
||||||
|
port: config.smtp.port || 587,
|
||||||
|
secure: config.smtp.secure,
|
||||||
|
user: config.smtp.user || config.email.user,
|
||||||
|
},
|
||||||
|
null,
|
||||||
|
2
|
||||||
|
)}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.campaignStart(uniqueFirms.length, config.email.testMode);
|
||||||
|
console.log(`🚀 Starting email campaign for ${uniqueFirms.length} firms`);
|
||||||
|
|
||||||
|
for (const firm of uniqueFirms) {
|
||||||
|
// Process any pending retries first
|
||||||
|
await errorHandler.processRetries(
|
||||||
|
transporter,
|
||||||
|
async (email, recipient, trans) => {
|
||||||
|
await sendSingleEmail(email, recipient, trans);
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
// Format firm data for template (database has different field names)
|
||||||
|
const templateData = templateEngine.formatFirmData({
|
||||||
|
firmName: firm.firm_name,
|
||||||
|
location: firm.location,
|
||||||
|
website: firm.website,
|
||||||
|
contactEmail: firm.contact_email,
|
||||||
|
email: firm.contact_email,
|
||||||
|
});
|
||||||
|
|
||||||
|
// In test mode, firm.contact_email already contains the generated test recipient
|
||||||
|
// CRITICAL: In test mode, NEVER send to actual firm email
|
||||||
|
const recipient = firm.contact_email;
|
||||||
|
|
||||||
|
// Log what we're doing for transparency
|
||||||
|
if (config.email.testMode) {
|
||||||
|
const firmIndex = uniqueFirms.indexOf(firm);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Firm ${firmIndex + 1}/${uniqueFirms.length} → ${recipient}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`🧪 TEST MODE: Email for ${firm.firm_name} → ${recipient}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const subject = config.email.testMode
|
||||||
|
? "[TEST] Legal Partnership Opportunity"
|
||||||
|
: "Legal Partnership Opportunity";
|
||||||
|
|
||||||
|
// Generate unique tracking ID for this email
|
||||||
|
const trackingId = `${currentCampaignId}_${
|
||||||
|
firm.contact_email
|
||||||
|
}_${Date.now()}`;
|
||||||
|
|
||||||
|
// Render email using template
|
||||||
|
const emailContent = await templateEngine.render("outreach", {
|
||||||
|
...templateData,
|
||||||
|
subject: subject,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add tracking to HTML content (unless skip tracking is enabled)
|
||||||
|
const trackedHtmlContent = trackingServer.addEmailTracking(
|
||||||
|
emailContent.html,
|
||||||
|
trackingId,
|
||||||
|
config.tracking.skipTracking
|
||||||
|
);
|
||||||
|
|
||||||
|
const mailOptions = {
|
||||||
|
from: config.email.user,
|
||||||
|
to: recipient,
|
||||||
|
subject: subject,
|
||||||
|
text: emailContent.text,
|
||||||
|
html: trackedHtmlContent,
|
||||||
|
attachments: attachments,
|
||||||
|
// Store firm data for error handling and tracking
|
||||||
|
firmName: firm.firm_name,
|
||||||
|
firmId: firm.id,
|
||||||
|
trackingId: trackingId,
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
await sendSingleEmail(mailOptions, recipient, transporter);
|
||||||
|
|
||||||
|
// Increment sent count immediately after success
|
||||||
|
rateLimiter.recordSuccess();
|
||||||
|
|
||||||
|
// Show rate limiting stats
|
||||||
|
const stats = rateLimiter.getStats();
|
||||||
|
console.log(
|
||||||
|
`📊 Progress: ${stats.sentCount} sent, ${stats.averageRate} emails/hour`
|
||||||
|
);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Rate limiter stats: ${JSON.stringify(stats, null, 2)}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Email send failed for ${recipient}: ${error.message}`
|
||||||
|
);
|
||||||
|
console.log(`🐛 DEBUG: Error code: ${error.code}`);
|
||||||
|
console.log(`🐛 DEBUG: Error command: ${error.command}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use error handler for intelligent retry logic
|
||||||
|
const willRetry = await errorHandler.handleError(
|
||||||
|
mailOptions,
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!willRetry) {
|
||||||
|
console.error(`💀 Skipping ${recipient} due to permanent failure`);
|
||||||
|
} else if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Email queued for retry: ${recipient}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Show retry queue status
|
||||||
|
const retryStats = errorHandler.getRetryStats();
|
||||||
|
if (retryStats.totalFailed > 0) {
|
||||||
|
console.log(
|
||||||
|
`🔄 Retry queue: ${retryStats.pendingRetries} pending, ${retryStats.readyToRetry} ready`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use rate limiter for intelligent delays (only if there are more emails)
|
||||||
|
if (uniqueFirms.indexOf(firm) < uniqueFirms.length - 1) {
|
||||||
|
const delayMs = await rateLimiter.getNextSendDelay();
|
||||||
|
console.log(`⏱️ Next email in: ${rateLimiter.formatDelay(delayMs)}`);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: Delay calculated: ${delayMs}ms`);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Emails remaining: ${
|
||||||
|
uniqueFirms.length - uniqueFirms.indexOf(firm) - 1
|
||||||
|
}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
await delay(delayMs);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process any remaining retries
|
||||||
|
console.log(`🔄 Processing final retries...`);
|
||||||
|
await errorHandler.processRetries(
|
||||||
|
transporter,
|
||||||
|
async (email, recipient, trans) => {
|
||||||
|
await sendSingleEmail(email, recipient, trans);
|
||||||
|
}
|
||||||
|
);
|
||||||
|
|
||||||
|
// Final stats
|
||||||
|
const finalRetryStats = errorHandler.getRetryStats();
|
||||||
|
const finalRateStats = rateLimiter.getStats();
|
||||||
|
|
||||||
|
const campaignStats = {
|
||||||
|
totalSent: finalRateStats.sentCount,
|
||||||
|
runtime: finalRateStats.runtime,
|
||||||
|
averageRate: finalRateStats.averageRate,
|
||||||
|
retryStats: finalRetryStats,
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.campaignComplete(campaignStats);
|
||||||
|
console.log(`📈 Campaign complete. Final stats:`, campaignStats);
|
||||||
|
|
||||||
|
// Complete campaign in database
|
||||||
|
if (currentCampaignId) {
|
||||||
|
await database.completeCampaign(currentCampaignId, {
|
||||||
|
sentEmails: finalRateStats.sentCount,
|
||||||
|
failedEmails: finalRetryStats.totalFailed,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop tracking server if it was started
|
||||||
|
if (config.tracking.enabled && !config.tracking.skipTracking) {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to stop tracking server", { error: error.message });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle graceful shutdown
|
||||||
|
process.on("SIGINT", async () => {
|
||||||
|
console.log("\n🛑 Received SIGINT. Gracefully shutting down...");
|
||||||
|
|
||||||
|
// Stop tracking server
|
||||||
|
if (process.env.TRACKING_ENABLED === "true") {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to stop tracking server during shutdown", {
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
process.exit(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
sendEmails().catch(async (error) => {
|
||||||
|
console.error("Campaign failed:", error);
|
||||||
|
|
||||||
|
// Stop tracking server on error
|
||||||
|
if (process.env.TRACKING_ENABLED === "true") {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (stopError) {
|
||||||
|
logger.error("Failed to stop tracking server after error", {
|
||||||
|
error: stopError.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
72
lib/attachmentHandler.js
Normal file
72
lib/attachmentHandler.js
Normal file
@ -0,0 +1,72 @@
|
|||||||
|
const fs = require("fs").promises;
|
||||||
|
const path = require("path");
|
||||||
|
const config = require("../config");
|
||||||
|
|
||||||
|
class AttachmentHandler {
|
||||||
|
constructor() {
|
||||||
|
this.attachmentsDir = path.join(__dirname, "..", "attachments");
|
||||||
|
}
|
||||||
|
|
||||||
|
async getAttachments() {
|
||||||
|
if (!config.attachments.enabled || config.attachments.files.length === 0) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
const attachments = [];
|
||||||
|
|
||||||
|
for (const filename of config.attachments.files) {
|
||||||
|
try {
|
||||||
|
const filePath = path.join(this.attachmentsDir, filename);
|
||||||
|
|
||||||
|
// Check if file exists
|
||||||
|
await fs.access(filePath);
|
||||||
|
|
||||||
|
// Get file stats for size validation
|
||||||
|
const stats = await fs.stat(filePath);
|
||||||
|
|
||||||
|
// Skip files larger than 10MB to avoid email size issues
|
||||||
|
if (stats.size > 10 * 1024 * 1024) {
|
||||||
|
console.warn(`Skipping ${filename}: File too large (>10MB)`);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
attachments.push({
|
||||||
|
filename: filename,
|
||||||
|
path: filePath,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.warn(`Unable to attach ${filename}: ${error.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return attachments;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper to validate attachment types
|
||||||
|
isValidFileType(filename) {
|
||||||
|
const validExtensions = [
|
||||||
|
".pdf",
|
||||||
|
".doc",
|
||||||
|
".docx",
|
||||||
|
".txt",
|
||||||
|
".jpg",
|
||||||
|
".jpeg",
|
||||||
|
".png",
|
||||||
|
".gif",
|
||||||
|
];
|
||||||
|
const ext = path.extname(filename).toLowerCase();
|
||||||
|
return validExtensions.includes(ext);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get list of available attachments for logging
|
||||||
|
async listAvailableFiles() {
|
||||||
|
try {
|
||||||
|
const files = await fs.readdir(this.attachmentsDir);
|
||||||
|
return files.filter((file) => this.isValidFileType(file));
|
||||||
|
} catch (error) {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new AttachmentHandler();
|
||||||
398
lib/database.js
Normal file
398
lib/database.js
Normal file
@ -0,0 +1,398 @@
|
|||||||
|
const sqlite3 = require("sqlite3").verbose();
|
||||||
|
const { open } = require("sqlite");
|
||||||
|
const path = require("path");
|
||||||
|
const logger = require("./logger");
|
||||||
|
|
||||||
|
class Database {
|
||||||
|
constructor() {
|
||||||
|
this.db = null;
|
||||||
|
this.dbPath = path.join(__dirname, "..", "data", "outreach.db");
|
||||||
|
}
|
||||||
|
|
||||||
|
async init() {
|
||||||
|
try {
|
||||||
|
// Create data directory if it doesn't exist
|
||||||
|
const fs = require("fs");
|
||||||
|
const dataDir = path.dirname(this.dbPath);
|
||||||
|
if (!fs.existsSync(dataDir)) {
|
||||||
|
fs.mkdirSync(dataDir, { recursive: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Open database connection
|
||||||
|
this.db = await open({
|
||||||
|
filename: this.dbPath,
|
||||||
|
driver: sqlite3.Database,
|
||||||
|
});
|
||||||
|
|
||||||
|
logger.info("Database connected", { dbPath: this.dbPath });
|
||||||
|
|
||||||
|
// Create tables
|
||||||
|
await this.createTables();
|
||||||
|
|
||||||
|
return this.db;
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Database initialization failed", { error: error.message });
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async createTables() {
|
||||||
|
const tables = [
|
||||||
|
// Firms table
|
||||||
|
`CREATE TABLE IF NOT EXISTS firms (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
firm_name TEXT NOT NULL,
|
||||||
|
location TEXT,
|
||||||
|
website TEXT,
|
||||||
|
contact_email TEXT NOT NULL,
|
||||||
|
state TEXT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE(contact_email)
|
||||||
|
)`,
|
||||||
|
|
||||||
|
// Email campaigns table
|
||||||
|
`CREATE TABLE IF NOT EXISTS campaigns (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
subject TEXT NOT NULL,
|
||||||
|
template_name TEXT NOT NULL,
|
||||||
|
test_mode BOOLEAN DEFAULT 0,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
started_at DATETIME,
|
||||||
|
completed_at DATETIME,
|
||||||
|
total_emails INTEGER DEFAULT 0,
|
||||||
|
sent_emails INTEGER DEFAULT 0,
|
||||||
|
failed_emails INTEGER DEFAULT 0
|
||||||
|
)`,
|
||||||
|
|
||||||
|
// Email sends table
|
||||||
|
`CREATE TABLE IF NOT EXISTS email_sends (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
campaign_id INTEGER,
|
||||||
|
firm_id INTEGER,
|
||||||
|
recipient_email TEXT NOT NULL,
|
||||||
|
subject TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL, -- 'sent', 'failed', 'retry', 'permanent_failure'
|
||||||
|
error_type TEXT,
|
||||||
|
error_message TEXT,
|
||||||
|
retry_count INTEGER DEFAULT 0,
|
||||||
|
tracking_id TEXT UNIQUE, -- Add tracking ID for email tracking
|
||||||
|
sent_at DATETIME,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (campaign_id) REFERENCES campaigns (id),
|
||||||
|
FOREIGN KEY (firm_id) REFERENCES firms (id)
|
||||||
|
)`,
|
||||||
|
|
||||||
|
// Tracking events table for email opens and clicks
|
||||||
|
`CREATE TABLE IF NOT EXISTS tracking_events (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
tracking_id TEXT NOT NULL,
|
||||||
|
event_type TEXT NOT NULL, -- 'open', 'click'
|
||||||
|
event_data TEXT, -- JSON data (link_id, target_url, etc.)
|
||||||
|
ip_address TEXT,
|
||||||
|
user_agent TEXT,
|
||||||
|
referer TEXT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
FOREIGN KEY (tracking_id) REFERENCES email_sends (tracking_id)
|
||||||
|
)`,
|
||||||
|
|
||||||
|
// Create indexes for better performance
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_firms_email ON firms(contact_email)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_firms_state ON firms(state)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_email_sends_campaign ON email_sends(campaign_id)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_email_sends_status ON email_sends(status)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_email_sends_tracking ON email_sends(tracking_id)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_tracking_events_tracking_id ON tracking_events(tracking_id)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS idx_tracking_events_type ON tracking_events(event_type)`,
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const table of tables) {
|
||||||
|
await this.db.exec(table);
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.info("Database tables created/verified");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Firm operations
|
||||||
|
async insertFirm(firmData) {
|
||||||
|
const { firmName, location, website, contactEmail, state } = firmData;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await this.db.run(
|
||||||
|
`INSERT OR IGNORE INTO firms (firm_name, location, website, contact_email, state)
|
||||||
|
VALUES (?, ?, ?, ?, ?)`,
|
||||||
|
[firmName, location, website, contactEmail, state]
|
||||||
|
);
|
||||||
|
|
||||||
|
return result.lastID;
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to insert firm", { firmData, error: error.message });
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async insertFirmsBatch(firms) {
|
||||||
|
const stmt = await this.db.prepare(
|
||||||
|
`INSERT OR IGNORE INTO firms (firm_name, location, website, contact_email, state)
|
||||||
|
VALUES (?, ?, ?, ?, ?)`
|
||||||
|
);
|
||||||
|
|
||||||
|
let inserted = 0;
|
||||||
|
for (const firm of firms) {
|
||||||
|
try {
|
||||||
|
const result = await stmt.run([
|
||||||
|
firm.firmName,
|
||||||
|
firm.location,
|
||||||
|
firm.website,
|
||||||
|
firm.contactEmail || firm.email,
|
||||||
|
firm.state,
|
||||||
|
]);
|
||||||
|
if (result.changes > 0) inserted++;
|
||||||
|
} catch (error) {
|
||||||
|
logger.warn("Failed to insert firm", {
|
||||||
|
firm: firm.firmName,
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
await stmt.finalize();
|
||||||
|
logger.info(`Inserted ${inserted} firms into database`);
|
||||||
|
return inserted;
|
||||||
|
}
|
||||||
|
|
||||||
|
async getFirms(limit = null, offset = 0) {
|
||||||
|
const sql = limit
|
||||||
|
? `SELECT * FROM firms ORDER BY id LIMIT ? OFFSET ?`
|
||||||
|
: `SELECT * FROM firms ORDER BY id`;
|
||||||
|
|
||||||
|
const params = limit ? [limit, offset] : [];
|
||||||
|
return await this.db.all(sql, params);
|
||||||
|
}
|
||||||
|
|
||||||
|
async getFirmByEmail(email) {
|
||||||
|
return await this.db.get("SELECT * FROM firms WHERE contact_email = ?", [
|
||||||
|
email,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
async getFirmById(id) {
|
||||||
|
return await this.db.get("SELECT * FROM firms WHERE id = ?", [id]);
|
||||||
|
}
|
||||||
|
|
||||||
|
async removeDuplicateFirms() {
|
||||||
|
// Remove duplicates keeping the first occurrence
|
||||||
|
const result = await this.db.run(`
|
||||||
|
DELETE FROM firms
|
||||||
|
WHERE id NOT IN (
|
||||||
|
SELECT MIN(id)
|
||||||
|
FROM firms
|
||||||
|
GROUP BY contact_email
|
||||||
|
)
|
||||||
|
`);
|
||||||
|
|
||||||
|
logger.info(`Removed ${result.changes} duplicate firms`);
|
||||||
|
return result.changes;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Campaign operations
|
||||||
|
async createCampaign(campaignData) {
|
||||||
|
const { name, subject, templateName, testMode } = campaignData;
|
||||||
|
|
||||||
|
const result = await this.db.run(
|
||||||
|
`INSERT INTO campaigns (name, subject, template_name, test_mode)
|
||||||
|
VALUES (?, ?, ?, ?)`,
|
||||||
|
[name, subject, templateName, testMode ? 1 : 0]
|
||||||
|
);
|
||||||
|
|
||||||
|
logger.info("Campaign created", { campaignId: result.lastID, name });
|
||||||
|
return result.lastID;
|
||||||
|
}
|
||||||
|
|
||||||
|
async startCampaign(campaignId, totalEmails) {
|
||||||
|
await this.db.run(
|
||||||
|
`UPDATE campaigns
|
||||||
|
SET started_at = CURRENT_TIMESTAMP, total_emails = ?
|
||||||
|
WHERE id = ?`,
|
||||||
|
[totalEmails, campaignId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async completeCampaign(campaignId, stats) {
|
||||||
|
await this.db.run(
|
||||||
|
`UPDATE campaigns
|
||||||
|
SET completed_at = CURRENT_TIMESTAMP,
|
||||||
|
sent_emails = ?,
|
||||||
|
failed_emails = ?
|
||||||
|
WHERE id = ?`,
|
||||||
|
[stats.sent, stats.failed, campaignId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Email send tracking
|
||||||
|
async logEmailSend(emailData) {
|
||||||
|
const {
|
||||||
|
campaignId,
|
||||||
|
firmId,
|
||||||
|
recipientEmail,
|
||||||
|
subject,
|
||||||
|
status,
|
||||||
|
errorType,
|
||||||
|
errorMessage,
|
||||||
|
retryCount,
|
||||||
|
trackingId,
|
||||||
|
} = emailData;
|
||||||
|
|
||||||
|
const result = await this.db.run(
|
||||||
|
`INSERT INTO email_sends
|
||||||
|
(campaign_id, firm_id, recipient_email, subject, status, error_type, error_message, retry_count, tracking_id, sent_at)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||||
|
[
|
||||||
|
campaignId,
|
||||||
|
firmId,
|
||||||
|
recipientEmail,
|
||||||
|
subject,
|
||||||
|
status,
|
||||||
|
errorType,
|
||||||
|
errorMessage,
|
||||||
|
retryCount,
|
||||||
|
trackingId,
|
||||||
|
status === "sent" ? new Date().toISOString() : null,
|
||||||
|
]
|
||||||
|
);
|
||||||
|
|
||||||
|
return result.lastID;
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateEmailSendStatus(
|
||||||
|
emailSendId,
|
||||||
|
status,
|
||||||
|
errorType = null,
|
||||||
|
errorMessage = null
|
||||||
|
) {
|
||||||
|
await this.db.run(
|
||||||
|
`UPDATE email_sends
|
||||||
|
SET status = ?, error_type = ?, error_message = ?,
|
||||||
|
sent_at = CASE WHEN ? = 'sent' THEN CURRENT_TIMESTAMP ELSE sent_at END
|
||||||
|
WHERE id = ?`,
|
||||||
|
[status, errorType, errorMessage, status, emailSendId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Statistics and reporting
|
||||||
|
async getCampaignStats(campaignId) {
|
||||||
|
const stats = await this.db.all(
|
||||||
|
`
|
||||||
|
SELECT
|
||||||
|
status,
|
||||||
|
COUNT(*) as count
|
||||||
|
FROM email_sends
|
||||||
|
WHERE campaign_id = ?
|
||||||
|
GROUP BY status
|
||||||
|
`,
|
||||||
|
[campaignId]
|
||||||
|
);
|
||||||
|
|
||||||
|
const result = {
|
||||||
|
sent: 0,
|
||||||
|
failed: 0,
|
||||||
|
retry: 0,
|
||||||
|
permanent_failure: 0,
|
||||||
|
};
|
||||||
|
|
||||||
|
stats.forEach((stat) => {
|
||||||
|
result[stat.status] = stat.count;
|
||||||
|
});
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
async getFailedEmails(campaignId) {
|
||||||
|
return await this.db.all(
|
||||||
|
`
|
||||||
|
SELECT es.*, f.firm_name, f.contact_email
|
||||||
|
FROM email_sends es
|
||||||
|
JOIN firms f ON es.firm_id = f.id
|
||||||
|
WHERE es.campaign_id = ? AND es.status IN ('failed', 'permanent_failure')
|
||||||
|
ORDER BY es.created_at DESC
|
||||||
|
`,
|
||||||
|
[campaignId]
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async close() {
|
||||||
|
if (this.db) {
|
||||||
|
await this.db.close();
|
||||||
|
this.db = null;
|
||||||
|
logger.info("Database connection closed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Utility methods
|
||||||
|
async getTableCounts() {
|
||||||
|
const counts = {};
|
||||||
|
const tables = ["firms", "campaigns", "email_sends", "tracking_events"];
|
||||||
|
|
||||||
|
for (const table of tables) {
|
||||||
|
const result = await this.db.get(
|
||||||
|
`SELECT COUNT(*) as count FROM ${table}`
|
||||||
|
);
|
||||||
|
counts[table] = result.count;
|
||||||
|
}
|
||||||
|
|
||||||
|
return counts;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tracking events methods
|
||||||
|
async storeTrackingEvent(trackingId, eventType, eventData = {}) {
|
||||||
|
const query = `
|
||||||
|
INSERT INTO tracking_events (tracking_id, event_type, event_data, ip_address, user_agent, referer)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?)
|
||||||
|
`;
|
||||||
|
|
||||||
|
await this.db.run(query, [
|
||||||
|
trackingId,
|
||||||
|
eventType,
|
||||||
|
JSON.stringify(eventData),
|
||||||
|
eventData.ip || null,
|
||||||
|
eventData.userAgent || null,
|
||||||
|
eventData.referer || null,
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
|
||||||
|
async getTrackingEvents(trackingId) {
|
||||||
|
const query = `
|
||||||
|
SELECT * FROM tracking_events
|
||||||
|
WHERE tracking_id = ?
|
||||||
|
ORDER BY created_at ASC
|
||||||
|
`;
|
||||||
|
|
||||||
|
const events = await this.db.all(query, [trackingId]);
|
||||||
|
return events.map((event) => ({
|
||||||
|
...event,
|
||||||
|
event_data: JSON.parse(event.event_data || "{}"),
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
async getTrackingStats(campaignId) {
|
||||||
|
const query = `
|
||||||
|
SELECT
|
||||||
|
es.tracking_id,
|
||||||
|
es.recipient_email,
|
||||||
|
COUNT(CASE WHEN te.event_type = 'open' THEN 1 END) as opens,
|
||||||
|
COUNT(CASE WHEN te.event_type = 'click' THEN 1 END) as clicks,
|
||||||
|
MIN(CASE WHEN te.event_type = 'open' THEN te.created_at END) as first_open,
|
||||||
|
MIN(CASE WHEN te.event_type = 'click' THEN te.created_at END) as first_click
|
||||||
|
FROM email_sends es
|
||||||
|
LEFT JOIN tracking_events te ON es.tracking_id = te.tracking_id
|
||||||
|
WHERE es.campaign_id = ? AND es.status = 'sent'
|
||||||
|
GROUP BY es.tracking_id, es.recipient_email
|
||||||
|
ORDER BY es.sent_at ASC
|
||||||
|
`;
|
||||||
|
|
||||||
|
return await this.db.all(query, [campaignId]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new Database();
|
||||||
242
lib/errorHandler.js
Normal file
242
lib/errorHandler.js
Normal file
@ -0,0 +1,242 @@
|
|||||||
|
const config = require("../config");
|
||||||
|
const logger = require("./logger");
|
||||||
|
|
||||||
|
class ErrorHandler {
|
||||||
|
constructor() {
|
||||||
|
this.failedEmails = [];
|
||||||
|
this.retryAttempts = new Map(); // Track retry attempts per email
|
||||||
|
}
|
||||||
|
|
||||||
|
// Classify error types
|
||||||
|
classifyError(error) {
|
||||||
|
const errorMessage = error.message.toLowerCase();
|
||||||
|
|
||||||
|
if (
|
||||||
|
errorMessage.includes("invalid login") ||
|
||||||
|
errorMessage.includes("authentication") ||
|
||||||
|
errorMessage.includes("unauthorized")
|
||||||
|
) {
|
||||||
|
return "AUTH_ERROR";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
errorMessage.includes("rate limit") ||
|
||||||
|
errorMessage.includes("too many requests") ||
|
||||||
|
errorMessage.includes("quota exceeded")
|
||||||
|
) {
|
||||||
|
return "RATE_LIMIT";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
errorMessage.includes("network") ||
|
||||||
|
errorMessage.includes("connection") ||
|
||||||
|
errorMessage.includes("timeout") ||
|
||||||
|
errorMessage.includes("econnrefused")
|
||||||
|
) {
|
||||||
|
return "NETWORK_ERROR";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
errorMessage.includes("invalid recipient") ||
|
||||||
|
errorMessage.includes("mailbox unavailable") ||
|
||||||
|
errorMessage.includes("user unknown")
|
||||||
|
) {
|
||||||
|
return "RECIPIENT_ERROR";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (
|
||||||
|
errorMessage.includes("message too large") ||
|
||||||
|
errorMessage.includes("attachment")
|
||||||
|
) {
|
||||||
|
return "MESSAGE_ERROR";
|
||||||
|
}
|
||||||
|
|
||||||
|
return "UNKNOWN_ERROR";
|
||||||
|
}
|
||||||
|
|
||||||
|
// Determine if error is retryable
|
||||||
|
isRetryable(errorType) {
|
||||||
|
const retryableErrors = ["RATE_LIMIT", "NETWORK_ERROR", "UNKNOWN_ERROR"];
|
||||||
|
|
||||||
|
const nonRetryableErrors = [
|
||||||
|
"AUTH_ERROR",
|
||||||
|
"RECIPIENT_ERROR",
|
||||||
|
"MESSAGE_ERROR",
|
||||||
|
];
|
||||||
|
|
||||||
|
return retryableErrors.includes(errorType);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate exponential backoff delay
|
||||||
|
getRetryDelay(attemptNumber) {
|
||||||
|
// Base delay: 1 minute, exponentially increasing
|
||||||
|
const baseDelay = 60 * 1000; // 1 minute in ms
|
||||||
|
const exponentialDelay = baseDelay * Math.pow(2, attemptNumber - 1);
|
||||||
|
|
||||||
|
// Add jitter (±25%)
|
||||||
|
const jitter = exponentialDelay * 0.25 * (Math.random() - 0.5);
|
||||||
|
|
||||||
|
// Cap at maximum delay (30 minutes)
|
||||||
|
const maxDelay = 30 * 60 * 1000;
|
||||||
|
|
||||||
|
return Math.min(exponentialDelay + jitter, maxDelay);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle email sending error
|
||||||
|
async handleError(email, recipient, error, transporter) {
|
||||||
|
const errorType = this.classifyError(error);
|
||||||
|
const emailKey = `${recipient}_${Date.now()}`;
|
||||||
|
|
||||||
|
logger.emailFailed(
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
errorType,
|
||||||
|
email.firmName || "Unknown"
|
||||||
|
);
|
||||||
|
console.error(`❌ Error sending to ${recipient}: ${error.message}`);
|
||||||
|
console.error(` Error Type: ${errorType}`);
|
||||||
|
|
||||||
|
// Get current retry count
|
||||||
|
const currentAttempts = this.retryAttempts.get(emailKey) || 0;
|
||||||
|
const maxRetries = config.errorHandling?.maxRetries || 3;
|
||||||
|
|
||||||
|
if (this.isRetryable(errorType) && currentAttempts < maxRetries) {
|
||||||
|
// Schedule retry
|
||||||
|
const retryDelay = this.getRetryDelay(currentAttempts + 1);
|
||||||
|
this.retryAttempts.set(emailKey, currentAttempts + 1);
|
||||||
|
|
||||||
|
logger.emailRetry(
|
||||||
|
recipient,
|
||||||
|
currentAttempts + 1,
|
||||||
|
maxRetries,
|
||||||
|
retryDelay,
|
||||||
|
errorType
|
||||||
|
);
|
||||||
|
console.warn(
|
||||||
|
`🔄 Scheduling retry ${
|
||||||
|
currentAttempts + 1
|
||||||
|
}/${maxRetries} for ${recipient} in ${Math.round(retryDelay / 1000)}s`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Add to retry queue
|
||||||
|
this.failedEmails.push({
|
||||||
|
email,
|
||||||
|
recipient,
|
||||||
|
error: errorType,
|
||||||
|
attempts: currentAttempts + 1,
|
||||||
|
retryAt: Date.now() + retryDelay,
|
||||||
|
originalError: error.message,
|
||||||
|
});
|
||||||
|
|
||||||
|
return true; // Indicates retry scheduled
|
||||||
|
} else {
|
||||||
|
// Max retries reached or non-retryable error
|
||||||
|
logger.emailPermanentFailure(recipient, errorType, currentAttempts);
|
||||||
|
console.error(
|
||||||
|
`💀 Permanent failure for ${recipient}: ${errorType} (${currentAttempts} attempts)`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Log permanently failed email
|
||||||
|
this.logPermanentFailure(
|
||||||
|
email,
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
errorType,
|
||||||
|
currentAttempts
|
||||||
|
);
|
||||||
|
|
||||||
|
return false; // Indicates permanent failure
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log permanent failures for later review
|
||||||
|
logPermanentFailure(email, recipient, error, errorType, attempts) {
|
||||||
|
const failure = {
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
recipient,
|
||||||
|
errorType,
|
||||||
|
attempts,
|
||||||
|
error: error.message,
|
||||||
|
emailData: {
|
||||||
|
subject: email.subject,
|
||||||
|
firmName: email.firmName,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
// You could write this to a file or database
|
||||||
|
console.error("🚨 PERMANENT FAILURE:", JSON.stringify(failure, null, 2));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process retry queue
|
||||||
|
async processRetries(transporter, sendEmailFunction) {
|
||||||
|
const now = Date.now();
|
||||||
|
const readyToRetry = this.failedEmails.filter(
|
||||||
|
(item) => item.retryAt <= now
|
||||||
|
);
|
||||||
|
|
||||||
|
if (readyToRetry.length === 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`🔄 Processing ${readyToRetry.length} email retries...`);
|
||||||
|
|
||||||
|
for (const retryItem of readyToRetry) {
|
||||||
|
try {
|
||||||
|
console.log(
|
||||||
|
`🔄 Retrying ${retryItem.recipient} (attempt ${retryItem.attempts})`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Attempt to send email again
|
||||||
|
await sendEmailFunction(
|
||||||
|
retryItem.email,
|
||||||
|
retryItem.recipient,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
// Success - remove from retry queue
|
||||||
|
this.failedEmails = this.failedEmails.filter(
|
||||||
|
(item) => item !== retryItem
|
||||||
|
);
|
||||||
|
console.log(`✅ Retry successful for ${retryItem.recipient}`);
|
||||||
|
} catch (error) {
|
||||||
|
// Handle retry failure
|
||||||
|
await this.handleError(
|
||||||
|
retryItem.email,
|
||||||
|
retryItem.recipient,
|
||||||
|
error,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
// Remove the processed item from queue
|
||||||
|
this.failedEmails = this.failedEmails.filter(
|
||||||
|
(item) => item !== retryItem
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get retry queue status
|
||||||
|
getRetryStats() {
|
||||||
|
const now = Date.now();
|
||||||
|
const pending = this.failedEmails.filter((item) => item.retryAt > now);
|
||||||
|
const ready = this.failedEmails.filter((item) => item.retryAt <= now);
|
||||||
|
|
||||||
|
return {
|
||||||
|
totalFailed: this.failedEmails.length,
|
||||||
|
pendingRetries: pending.length,
|
||||||
|
readyToRetry: ready.length,
|
||||||
|
nextRetryIn:
|
||||||
|
pending.length > 0
|
||||||
|
? Math.min(...pending.map((item) => item.retryAt - now))
|
||||||
|
: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clear retry queue (for testing)
|
||||||
|
clearRetries() {
|
||||||
|
this.failedEmails = [];
|
||||||
|
this.retryAttempts.clear();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new ErrorHandler();
|
||||||
159
lib/logger.js
Normal file
159
lib/logger.js
Normal file
@ -0,0 +1,159 @@
|
|||||||
|
const winston = require("winston");
|
||||||
|
const path = require("path");
|
||||||
|
const config = require("../config");
|
||||||
|
|
||||||
|
// Create logs directory if it doesn't exist
|
||||||
|
const fs = require("fs");
|
||||||
|
const logsDir = path.join(__dirname, "..", "logs");
|
||||||
|
if (!fs.existsSync(logsDir)) {
|
||||||
|
fs.mkdirSync(logsDir);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Custom format for console output
|
||||||
|
const consoleFormat = winston.format.combine(
|
||||||
|
winston.format.timestamp({ format: "HH:mm:ss" }),
|
||||||
|
winston.format.colorize(),
|
||||||
|
winston.format.printf(({ timestamp, level, message, ...meta }) => {
|
||||||
|
let metaStr = "";
|
||||||
|
if (Object.keys(meta).length > 0) {
|
||||||
|
metaStr = " " + JSON.stringify(meta);
|
||||||
|
}
|
||||||
|
return `${timestamp} [${level}] ${message}${metaStr}`;
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
// Custom format for file output
|
||||||
|
const fileFormat = winston.format.combine(
|
||||||
|
winston.format.timestamp(),
|
||||||
|
winston.format.errors({ stack: true }),
|
||||||
|
winston.format.json()
|
||||||
|
);
|
||||||
|
|
||||||
|
// Create the logger
|
||||||
|
const logger = winston.createLogger({
|
||||||
|
level: config.logging.level,
|
||||||
|
format: fileFormat,
|
||||||
|
defaultMeta: {
|
||||||
|
service: "outreach-engine",
|
||||||
|
environment: config.env,
|
||||||
|
},
|
||||||
|
transports: [
|
||||||
|
// Error log file
|
||||||
|
new winston.transports.File({
|
||||||
|
filename: path.join(logsDir, "error.log"),
|
||||||
|
level: "error",
|
||||||
|
maxsize: 5242880, // 5MB
|
||||||
|
maxFiles: 5,
|
||||||
|
}),
|
||||||
|
|
||||||
|
// Combined log file
|
||||||
|
new winston.transports.File({
|
||||||
|
filename: path.join(logsDir, "combined.log"),
|
||||||
|
maxsize: 5242880, // 5MB
|
||||||
|
maxFiles: 5,
|
||||||
|
}),
|
||||||
|
|
||||||
|
// Email activity log
|
||||||
|
new winston.transports.File({
|
||||||
|
filename: path.join(logsDir, "email-activity.log"),
|
||||||
|
level: "info",
|
||||||
|
maxsize: 10485760, // 10MB
|
||||||
|
maxFiles: 10,
|
||||||
|
}),
|
||||||
|
],
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add console transport for development
|
||||||
|
if (config.isDevelopment) {
|
||||||
|
logger.add(
|
||||||
|
new winston.transports.Console({
|
||||||
|
format: consoleFormat,
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Custom logging methods for email activities
|
||||||
|
logger.emailSent = (recipient, subject, firmName, testMode = false) => {
|
||||||
|
logger.info("Email sent successfully", {
|
||||||
|
event: "email_sent",
|
||||||
|
recipient,
|
||||||
|
subject,
|
||||||
|
firmName,
|
||||||
|
testMode,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.emailFailed = (recipient, error, errorType, firmName) => {
|
||||||
|
logger.error("Email failed to send", {
|
||||||
|
event: "email_failed",
|
||||||
|
recipient,
|
||||||
|
error: error.message,
|
||||||
|
errorType,
|
||||||
|
firmName,
|
||||||
|
stack: error.stack,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.emailRetry = (recipient, attempt, maxRetries, retryDelay, errorType) => {
|
||||||
|
logger.warn("Email retry scheduled", {
|
||||||
|
event: "email_retry",
|
||||||
|
recipient,
|
||||||
|
attempt,
|
||||||
|
maxRetries,
|
||||||
|
retryDelay,
|
||||||
|
errorType,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.emailPermanentFailure = (recipient, errorType, attempts) => {
|
||||||
|
logger.error("Email permanent failure", {
|
||||||
|
event: "email_permanent_failure",
|
||||||
|
recipient,
|
||||||
|
errorType,
|
||||||
|
attempts,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.campaignStart = (totalEmails, testMode) => {
|
||||||
|
logger.info("Email campaign started", {
|
||||||
|
event: "campaign_start",
|
||||||
|
totalEmails,
|
||||||
|
testMode,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.campaignComplete = (stats) => {
|
||||||
|
logger.info("Email campaign completed", {
|
||||||
|
event: "campaign_complete",
|
||||||
|
...stats,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.rateLimitPause = (duration, reason) => {
|
||||||
|
logger.warn("Rate limit pause activated", {
|
||||||
|
event: "rate_limit_pause",
|
||||||
|
duration,
|
||||||
|
reason,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
// Helper method to log with email context
|
||||||
|
logger.withEmail = (recipient, firmName) => {
|
||||||
|
return {
|
||||||
|
info: (message, meta = {}) =>
|
||||||
|
logger.info(message, { ...meta, recipient, firmName }),
|
||||||
|
warn: (message, meta = {}) =>
|
||||||
|
logger.warn(message, { ...meta, recipient, firmName }),
|
||||||
|
error: (message, meta = {}) =>
|
||||||
|
logger.error(message, { ...meta, recipient, firmName }),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
module.exports = logger;
|
||||||
107
lib/rateLimiter.js
Normal file
107
lib/rateLimiter.js
Normal file
@ -0,0 +1,107 @@
|
|||||||
|
const config = require("../config");
|
||||||
|
const logger = require("./logger");
|
||||||
|
|
||||||
|
class RateLimiter {
|
||||||
|
constructor() {
|
||||||
|
this.sentCount = 0;
|
||||||
|
this.startTime = Date.now();
|
||||||
|
this.lastSentTime = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Record a successful send
|
||||||
|
recordSuccess() {
|
||||||
|
this.sentCount++;
|
||||||
|
this.lastSentTime = Date.now();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get randomized delay between min and max
|
||||||
|
getRandomDelay() {
|
||||||
|
const baseDelay = config.app.delayMinutes * 60 * 1000; // Convert to ms
|
||||||
|
const minDelay = baseDelay * 0.8; // 20% less than base
|
||||||
|
const maxDelay = baseDelay * 1.2; // 20% more than base
|
||||||
|
|
||||||
|
// Add additional randomization
|
||||||
|
const randomFactor = Math.random() * (maxDelay - minDelay) + minDelay;
|
||||||
|
|
||||||
|
// Add jitter to avoid patterns
|
||||||
|
const jitter = (Math.random() - 0.5) * 30000; // +/- 30 seconds
|
||||||
|
|
||||||
|
return Math.floor(randomFactor + jitter);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if we should pause based on sent count
|
||||||
|
shouldPause() {
|
||||||
|
const hoursSinceStart = (Date.now() - this.startTime) / (1000 * 60 * 60);
|
||||||
|
const emailsPerHour = this.sentCount / hoursSinceStart;
|
||||||
|
|
||||||
|
// Gmail limits: ~500/day = ~20/hour
|
||||||
|
// Be conservative: pause if exceeding 15/hour
|
||||||
|
if (emailsPerHour > 15) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pause every 50 emails for 30 minutes
|
||||||
|
if (this.sentCount > 0 && this.sentCount % 50 === 0) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get pause duration
|
||||||
|
getPauseDuration() {
|
||||||
|
// Standard pause: 30 minutes
|
||||||
|
const basePause = 30 * 60 * 1000;
|
||||||
|
|
||||||
|
// Add randomization to pause duration
|
||||||
|
const randomPause = basePause + Math.random() * 10 * 60 * 1000; // +0-10 minutes
|
||||||
|
|
||||||
|
return randomPause;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate next send time
|
||||||
|
async getNextSendDelay() {
|
||||||
|
if (this.shouldPause()) {
|
||||||
|
const pauseDuration = this.getPauseDuration();
|
||||||
|
const pauseMinutes = Math.round(pauseDuration / 60000);
|
||||||
|
|
||||||
|
logger.rateLimitPause(
|
||||||
|
pauseDuration,
|
||||||
|
`Automatic pause after ${this.sentCount} emails`
|
||||||
|
);
|
||||||
|
console.log(`Rate limit pause: ${pauseMinutes} minutes`);
|
||||||
|
return pauseDuration;
|
||||||
|
}
|
||||||
|
|
||||||
|
return this.getRandomDelay();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get human-readable time
|
||||||
|
formatDelay(ms) {
|
||||||
|
const minutes = Math.floor(ms / 60000);
|
||||||
|
const seconds = Math.floor((ms % 60000) / 1000);
|
||||||
|
return `${minutes}m ${seconds}s`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reset counters (for testing)
|
||||||
|
reset() {
|
||||||
|
this.sentCount = 0;
|
||||||
|
this.startTime = Date.now();
|
||||||
|
this.lastSentTime = null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get current stats
|
||||||
|
getStats() {
|
||||||
|
const runtime = (Date.now() - this.startTime) / 1000; // seconds
|
||||||
|
const avgRate = this.sentCount / (runtime / 3600); // emails per hour
|
||||||
|
|
||||||
|
return {
|
||||||
|
sentCount: this.sentCount,
|
||||||
|
runtime: Math.floor(runtime / 60), // minutes
|
||||||
|
averageRate: avgRate.toFixed(1),
|
||||||
|
nextDelay: this.formatDelay(this.getRandomDelay()),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new RateLimiter();
|
||||||
130
lib/templateEngine.js
Normal file
130
lib/templateEngine.js
Normal file
@ -0,0 +1,130 @@
|
|||||||
|
const fs = require("fs").promises;
|
||||||
|
const path = require("path");
|
||||||
|
const Handlebars = require("handlebars");
|
||||||
|
const config = require("../config");
|
||||||
|
|
||||||
|
class TemplateEngine {
|
||||||
|
constructor() {
|
||||||
|
this.templatesDir = path.join(__dirname, "..", "templates");
|
||||||
|
this.compiledTemplates = new Map();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert HTML to plain text by removing tags and formatting
|
||||||
|
htmlToText(html) {
|
||||||
|
return html
|
||||||
|
.replace(/<style[^>]*>.*?<\/style>/gis, "") // Remove style blocks
|
||||||
|
.replace(/<script[^>]*>.*?<\/script>/gis, "") // Remove script blocks
|
||||||
|
.replace(/<br\s*\/?>/gi, "\n") // Convert <br> to newlines
|
||||||
|
.replace(/<\/p>/gi, "\n\n") // Convert </p> to double newlines
|
||||||
|
.replace(/<\/div>/gi, "\n") // Convert </div> to newlines
|
||||||
|
.replace(/<\/h[1-6]>/gi, "\n\n") // Convert headings to double newlines
|
||||||
|
.replace(/<li[^>]*>/gi, "• ") // Convert <li> to bullet points
|
||||||
|
.replace(/<\/li>/gi, "\n") // End list items with newlines
|
||||||
|
.replace(/<[^>]*>/g, "") // Remove all other HTML tags
|
||||||
|
.replace(/ /g, " ") // Convert to spaces
|
||||||
|
.replace(/&/g, "&") // Convert & to &
|
||||||
|
.replace(/</g, "<") // Convert < to <
|
||||||
|
.replace(/>/g, ">") // Convert > to >
|
||||||
|
.replace(/"/g, '"') // Convert " to "
|
||||||
|
.replace(/'/g, "'") // Convert ' to '
|
||||||
|
.replace(/\n\s*\n\s*\n/g, "\n\n") // Reduce multiple newlines to double
|
||||||
|
.replace(/^\s+|\s+$/gm, "") // Trim whitespace from lines
|
||||||
|
.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
async loadTemplate(templateName) {
|
||||||
|
const cacheKey = templateName;
|
||||||
|
|
||||||
|
if (this.compiledTemplates.has(cacheKey)) {
|
||||||
|
return this.compiledTemplates.get(cacheKey);
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const htmlPath = path.join(this.templatesDir, `${templateName}.html`);
|
||||||
|
const htmlContent = await fs.readFile(htmlPath, "utf-8");
|
||||||
|
|
||||||
|
// Automatically inject GIF if enabled
|
||||||
|
let finalHtmlContent = htmlContent;
|
||||||
|
if (config.gif.enabled && templateName === "outreach") {
|
||||||
|
finalHtmlContent = this.injectGifIntoHtml(htmlContent);
|
||||||
|
}
|
||||||
|
|
||||||
|
const htmlTemplate = Handlebars.compile(finalHtmlContent);
|
||||||
|
|
||||||
|
// Generate text version from HTML
|
||||||
|
const textTemplate = Handlebars.compile(
|
||||||
|
this.htmlToText(finalHtmlContent)
|
||||||
|
);
|
||||||
|
|
||||||
|
const compiledTemplate = {
|
||||||
|
html: htmlTemplate,
|
||||||
|
text: textTemplate,
|
||||||
|
};
|
||||||
|
|
||||||
|
this.compiledTemplates.set(cacheKey, compiledTemplate);
|
||||||
|
return compiledTemplate;
|
||||||
|
} catch (error) {
|
||||||
|
throw new Error(
|
||||||
|
`Failed to load template ${templateName}: ${error.message}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inject GIF into HTML template automatically
|
||||||
|
injectGifIntoHtml(htmlContent) {
|
||||||
|
const gifHtml = `
|
||||||
|
{{#if gifUrl}}
|
||||||
|
<div style="text-align: center; margin: 20px 0;">
|
||||||
|
<img src="{{gifUrl}}" alt="{{gifAlt}}" style="max-width: 100%; height: auto; border-radius: 5px;" />
|
||||||
|
</div>
|
||||||
|
{{/if}}
|
||||||
|
`;
|
||||||
|
|
||||||
|
// Insert GIF after the header or at the beginning of content
|
||||||
|
if (htmlContent.includes('<div class="content">')) {
|
||||||
|
return htmlContent.replace(
|
||||||
|
'<div class="content">',
|
||||||
|
`<div class="content">${gifHtml}`
|
||||||
|
);
|
||||||
|
} else if (htmlContent.includes("<body>")) {
|
||||||
|
return htmlContent.replace("<body>", `<body>${gifHtml}`);
|
||||||
|
} else {
|
||||||
|
// Fallback: add at the beginning
|
||||||
|
return gifHtml + htmlContent;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async render(templateName, data) {
|
||||||
|
const template = await this.loadTemplate(templateName);
|
||||||
|
|
||||||
|
// Add default sender information from config
|
||||||
|
const defaultData = {
|
||||||
|
senderName: "John Smith",
|
||||||
|
senderTitle: "Business Development Manager",
|
||||||
|
senderCompany: "Legal Solutions Inc.",
|
||||||
|
fromEmail: config.email.user,
|
||||||
|
gifUrl: config.gif.enabled ? config.gif.url : null,
|
||||||
|
gifAlt: config.gif.enabled ? config.gif.alt : null,
|
||||||
|
...data,
|
||||||
|
};
|
||||||
|
|
||||||
|
return {
|
||||||
|
html: template.html(defaultData),
|
||||||
|
text: template.text(defaultData),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// Helper to format firm data for template
|
||||||
|
formatFirmData(firm) {
|
||||||
|
return {
|
||||||
|
firmName: firm.firmName || "your firm",
|
||||||
|
location: firm.location,
|
||||||
|
website: firm.website,
|
||||||
|
email: firm.contactEmail || firm.email,
|
||||||
|
greeting: firm.name || "Legal Professional",
|
||||||
|
// Additional fields can be mapped here
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new TemplateEngine();
|
||||||
266
lib/trackingServer.js
Normal file
266
lib/trackingServer.js
Normal file
@ -0,0 +1,266 @@
|
|||||||
|
const http = require("http");
|
||||||
|
const url = require("url");
|
||||||
|
const path = require("path");
|
||||||
|
const logger = require("./logger");
|
||||||
|
const database = require("./database");
|
||||||
|
|
||||||
|
class TrackingServer {
|
||||||
|
constructor() {
|
||||||
|
this.server = null;
|
||||||
|
this.port = process.env.TRACKING_PORT || 3000;
|
||||||
|
this.trackingDomain =
|
||||||
|
process.env.TRACKING_DOMAIN || `http://localhost:${this.port}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 1x1 transparent pixel GIF
|
||||||
|
get trackingPixel() {
|
||||||
|
return Buffer.from(
|
||||||
|
"R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7",
|
||||||
|
"base64"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async start() {
|
||||||
|
this.server = http.createServer((req, res) => {
|
||||||
|
this.handleRequest(req, res);
|
||||||
|
});
|
||||||
|
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
this.server.listen(this.port, (err) => {
|
||||||
|
if (err) {
|
||||||
|
reject(err);
|
||||||
|
} else {
|
||||||
|
console.log(`📊 Tracking server started on ${this.trackingDomain}`);
|
||||||
|
logger.info("Tracking server started", {
|
||||||
|
port: this.port,
|
||||||
|
domain: this.trackingDomain,
|
||||||
|
});
|
||||||
|
resolve();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async stop() {
|
||||||
|
if (this.server) {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
this.server.close(() => {
|
||||||
|
console.log("📊 Tracking server stopped");
|
||||||
|
logger.info("Tracking server stopped");
|
||||||
|
resolve();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async handleRequest(req, res) {
|
||||||
|
const parsedUrl = url.parse(req.url, true);
|
||||||
|
const pathname = parsedUrl.pathname;
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (pathname.startsWith("/track/open/")) {
|
||||||
|
await this.handleOpenTracking(req, res, parsedUrl);
|
||||||
|
} else if (pathname.startsWith("/track/click/")) {
|
||||||
|
await this.handleClickTracking(req, res, parsedUrl);
|
||||||
|
} else if (pathname === "/health") {
|
||||||
|
this.handleHealthCheck(req, res);
|
||||||
|
} else {
|
||||||
|
this.handle404(req, res);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Tracking request error", {
|
||||||
|
error: error.message,
|
||||||
|
url: req.url,
|
||||||
|
userAgent: req.headers["user-agent"],
|
||||||
|
});
|
||||||
|
this.handle500(req, res);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async handleOpenTracking(req, res, parsedUrl) {
|
||||||
|
const pathParts = parsedUrl.pathname.split("/");
|
||||||
|
const trackingId = pathParts[3]; // /track/open/{trackingId}
|
||||||
|
|
||||||
|
if (!trackingId) {
|
||||||
|
return this.handle400(req, res, "Missing tracking ID");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log the email open
|
||||||
|
const trackingData = {
|
||||||
|
trackingId,
|
||||||
|
event: "email_open",
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
ip: req.connection.remoteAddress || req.headers["x-forwarded-for"],
|
||||||
|
userAgent: req.headers["user-agent"],
|
||||||
|
referer: req.headers.referer,
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.info("Email opened", trackingData);
|
||||||
|
|
||||||
|
// Store in database
|
||||||
|
try {
|
||||||
|
await database.storeTrackingEvent(trackingId, "open", trackingData);
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to store tracking event", {
|
||||||
|
trackingId,
|
||||||
|
event: "open",
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return 1x1 transparent pixel
|
||||||
|
res.writeHead(200, {
|
||||||
|
"Content-Type": "image/gif",
|
||||||
|
"Content-Length": this.trackingPixel.length,
|
||||||
|
"Cache-Control": "no-cache, no-store, must-revalidate",
|
||||||
|
Pragma: "no-cache",
|
||||||
|
Expires: "0",
|
||||||
|
});
|
||||||
|
res.end(this.trackingPixel);
|
||||||
|
}
|
||||||
|
|
||||||
|
async handleClickTracking(req, res, parsedUrl) {
|
||||||
|
const pathParts = parsedUrl.pathname.split("/");
|
||||||
|
const trackingId = pathParts[3]; // /track/click/{trackingId}
|
||||||
|
const linkId = pathParts[4]; // /track/click/{trackingId}/{linkId}
|
||||||
|
const targetUrl = parsedUrl.query.url;
|
||||||
|
|
||||||
|
if (!trackingId || !targetUrl) {
|
||||||
|
return this.handle400(req, res, "Missing tracking ID or target URL");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log the click
|
||||||
|
const trackingData = {
|
||||||
|
trackingId,
|
||||||
|
linkId,
|
||||||
|
event: "email_click",
|
||||||
|
targetUrl,
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
ip: req.connection.remoteAddress || req.headers["x-forwarded-for"],
|
||||||
|
userAgent: req.headers["user-agent"],
|
||||||
|
referer: req.headers.referer,
|
||||||
|
};
|
||||||
|
|
||||||
|
logger.info("Email link clicked", trackingData);
|
||||||
|
|
||||||
|
// Store in database
|
||||||
|
try {
|
||||||
|
await database.storeTrackingEvent(trackingId, "click", trackingData);
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to store tracking event", {
|
||||||
|
trackingId,
|
||||||
|
event: "click",
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redirect to target URL
|
||||||
|
res.writeHead(302, {
|
||||||
|
Location: targetUrl,
|
||||||
|
"Cache-Control": "no-cache",
|
||||||
|
});
|
||||||
|
res.end();
|
||||||
|
}
|
||||||
|
|
||||||
|
handleHealthCheck(req, res) {
|
||||||
|
res.writeHead(200, { "Content-Type": "application/json" });
|
||||||
|
res.end(
|
||||||
|
JSON.stringify({
|
||||||
|
status: "healthy",
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
uptime: process.uptime(),
|
||||||
|
})
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
handle400(req, res, message) {
|
||||||
|
res.writeHead(400, { "Content-Type": "text/plain" });
|
||||||
|
res.end(`Bad Request: ${message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
handle404(req, res) {
|
||||||
|
res.writeHead(404, { "Content-Type": "text/plain" });
|
||||||
|
res.end("Not Found");
|
||||||
|
}
|
||||||
|
|
||||||
|
handle500(req, res) {
|
||||||
|
res.writeHead(500, { "Content-Type": "text/plain" });
|
||||||
|
res.end("Internal Server Error");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate tracking URLs
|
||||||
|
generateOpenTrackingUrl(emailId) {
|
||||||
|
return `${this.trackingDomain}/track/open/${emailId}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
generateClickTrackingUrl(emailId, linkId, targetUrl) {
|
||||||
|
const encodedUrl = encodeURIComponent(targetUrl);
|
||||||
|
return `${this.trackingDomain}/track/click/${emailId}/${linkId}?url=${encodedUrl}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add tracking to email content
|
||||||
|
addTrackingToEmail(htmlContent, emailId) {
|
||||||
|
if (!htmlContent) return htmlContent;
|
||||||
|
|
||||||
|
// Add tracking pixel just before closing body tag
|
||||||
|
const trackingPixel = `<img src="${this.generateOpenTrackingUrl(
|
||||||
|
emailId
|
||||||
|
)}" width="1" height="1" style="display:none;" alt="" />`;
|
||||||
|
|
||||||
|
if (htmlContent.includes("</body>")) {
|
||||||
|
return htmlContent.replace("</body>", `${trackingPixel}</body>`);
|
||||||
|
} else {
|
||||||
|
// If no body tag, append at the end
|
||||||
|
return htmlContent + trackingPixel;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Replace links with tracking URLs
|
||||||
|
addClickTrackingToEmail(htmlContent, emailId) {
|
||||||
|
if (!htmlContent) return htmlContent;
|
||||||
|
|
||||||
|
let linkId = 0;
|
||||||
|
return htmlContent.replace(
|
||||||
|
/<a\s+([^>]*href=["']([^"']+)["'][^>]*)>/gi,
|
||||||
|
(match, attributes, href) => {
|
||||||
|
linkId++;
|
||||||
|
|
||||||
|
// Skip if already a tracking URL or mailto/tel links
|
||||||
|
if (
|
||||||
|
href.includes("/track/click/") ||
|
||||||
|
href.startsWith("mailto:") ||
|
||||||
|
href.startsWith("tel:")
|
||||||
|
) {
|
||||||
|
return match;
|
||||||
|
}
|
||||||
|
|
||||||
|
const trackingUrl = this.generateClickTrackingUrl(
|
||||||
|
emailId,
|
||||||
|
linkId,
|
||||||
|
href
|
||||||
|
);
|
||||||
|
return `<a ${attributes.replace(
|
||||||
|
/href=["'][^"']+["']/i,
|
||||||
|
`href="${trackingUrl}"`
|
||||||
|
)}`;
|
||||||
|
}
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Combined method to add all tracking
|
||||||
|
addEmailTracking(htmlContent, emailId, skipTracking = false) {
|
||||||
|
if (!htmlContent) return htmlContent;
|
||||||
|
|
||||||
|
// If skip tracking is enabled, return original content without tracking
|
||||||
|
if (skipTracking) {
|
||||||
|
return htmlContent;
|
||||||
|
}
|
||||||
|
|
||||||
|
let trackedContent = this.addClickTrackingToEmail(htmlContent, emailId);
|
||||||
|
trackedContent = this.addTrackingToEmail(trackedContent, emailId);
|
||||||
|
|
||||||
|
return trackedContent;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = new TrackingServer();
|
||||||
5875
package-lock.json
generated
Normal file
5875
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
36
package.json
Normal file
36
package.json
Normal file
@ -0,0 +1,36 @@
|
|||||||
|
{
|
||||||
|
"name": "outreach-engine",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"description": "",
|
||||||
|
"main": "index.js",
|
||||||
|
"scripts": {
|
||||||
|
"start": "node index.js",
|
||||||
|
"start:dev": "NODE_ENV=development node index.js",
|
||||||
|
"start:prod": "NODE_ENV=production node index.js",
|
||||||
|
"test-email": "EMAIL_TEST_LIMIT=1 NODE_ENV=development node index.js",
|
||||||
|
"test-campaigns": "EMAIL_TEST_MODE=true CAMPAIGN_TEST_MODE=true NODE_ENV=development node scripts/run-campaigns.js",
|
||||||
|
"migrate": "node scripts/migrate-to-database.js",
|
||||||
|
"test": "jest --config tests/jest.config.js",
|
||||||
|
"test:watch": "jest --config tests/jest.config.js --watch",
|
||||||
|
"test:coverage": "jest --config tests/jest.config.js --coverage",
|
||||||
|
"validate-json": "node -e \"try { JSON.parse(require('fs').readFileSync('firm.json', 'utf8')); console.log('✅ JSON is valid'); } catch(e) { console.log('❌ JSON error:', e.message); }\"",
|
||||||
|
"test-config": "node -e \"const config = require('./config'); console.log('Environment:', config.env); console.log('EMAIL_USER:', config.email.user ? 'Set' : 'Missing'); console.log('EMAIL_PASS:', config.email.pass ? 'Set' : 'Missing'); console.log('Test Mode:', config.email.testMode);\""
|
||||||
|
},
|
||||||
|
"keywords": [],
|
||||||
|
"author": "",
|
||||||
|
"license": "ISC",
|
||||||
|
"type": "commonjs",
|
||||||
|
"dependencies": {
|
||||||
|
"delay": "^6.0.0",
|
||||||
|
"dotenv": "^17.2.0",
|
||||||
|
"handlebars": "^4.7.8",
|
||||||
|
"nodemailer": "^7.0.5",
|
||||||
|
"sqlite": "^5.1.1",
|
||||||
|
"sqlite3": "^5.1.7",
|
||||||
|
"winston": "^3.17.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@jest/globals": "^30.0.4",
|
||||||
|
"jest": "^30.0.4"
|
||||||
|
}
|
||||||
|
}
|
||||||
113
scripts/migrate-to-database.js
Normal file
113
scripts/migrate-to-database.js
Normal file
@ -0,0 +1,113 @@
|
|||||||
|
const fs = require("fs");
|
||||||
|
const path = require("path");
|
||||||
|
const logger = require("../lib/logger");
|
||||||
|
|
||||||
|
// Use a separate database instance for migration to avoid conflicts
|
||||||
|
const Database = require("../lib/database");
|
||||||
|
|
||||||
|
async function migrateJsonToDatabase() {
|
||||||
|
const migrationDb = new Database();
|
||||||
|
|
||||||
|
try {
|
||||||
|
console.log("🚀 Starting migration from JSON to database...");
|
||||||
|
|
||||||
|
// Initialize migration database
|
||||||
|
await migrationDb.init();
|
||||||
|
|
||||||
|
// Check if firm.json exists
|
||||||
|
const firmJsonPath = path.join(__dirname, "..", "firm.json");
|
||||||
|
if (!fs.existsSync(firmJsonPath)) {
|
||||||
|
console.log("❌ firm.json not found. No migration needed.");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read and parse firm.json
|
||||||
|
const firmData = JSON.parse(fs.readFileSync(firmJsonPath, "utf8"));
|
||||||
|
console.log("📖 Read firm.json successfully");
|
||||||
|
|
||||||
|
// Flatten the data structure (it's organized by state)
|
||||||
|
const allFirms = [];
|
||||||
|
for (const [state, firms] of Object.entries(firmData)) {
|
||||||
|
if (Array.isArray(firms)) {
|
||||||
|
firms.forEach((firm) => {
|
||||||
|
// Normalize firm data
|
||||||
|
allFirms.push({
|
||||||
|
firmName: firm.firmName,
|
||||||
|
location: firm.location,
|
||||||
|
website: firm.website,
|
||||||
|
contactEmail: firm.contactEmail || firm.email,
|
||||||
|
state: state,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`📊 Found ${allFirms.length} firms across ${
|
||||||
|
Object.keys(firmData).length
|
||||||
|
} states`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Filter out firms without email addresses
|
||||||
|
const validFirms = allFirms.filter((firm) => firm.contactEmail);
|
||||||
|
console.log(`✅ ${validFirms.length} firms have valid email addresses`);
|
||||||
|
|
||||||
|
if (validFirms.length !== allFirms.length) {
|
||||||
|
console.log(
|
||||||
|
`⚠️ Skipped ${
|
||||||
|
allFirms.length - validFirms.length
|
||||||
|
} firms without email addresses`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Insert firms into database
|
||||||
|
const inserted = await migrationDb.insertFirmsBatch(validFirms);
|
||||||
|
console.log(`✨ Inserted ${inserted} new firms into database`);
|
||||||
|
|
||||||
|
// Remove duplicates
|
||||||
|
const duplicatesRemoved = await migrationDb.removeDuplicateFirms();
|
||||||
|
if (duplicatesRemoved > 0) {
|
||||||
|
console.log(`🧹 Removed ${duplicatesRemoved} duplicate firms`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get final counts
|
||||||
|
const counts = await migrationDb.getTableCounts();
|
||||||
|
console.log("📈 Final database counts:", counts);
|
||||||
|
|
||||||
|
// Create backup of JSON file
|
||||||
|
const backupPath = path.join(
|
||||||
|
__dirname,
|
||||||
|
"..",
|
||||||
|
`firm.json.backup.${Date.now()}`
|
||||||
|
);
|
||||||
|
fs.copyFileSync(firmJsonPath, backupPath);
|
||||||
|
console.log(`💾 Created backup: ${path.basename(backupPath)}`);
|
||||||
|
|
||||||
|
console.log("✅ Migration completed successfully!");
|
||||||
|
} catch (error) {
|
||||||
|
console.error("❌ Migration failed:", error.message);
|
||||||
|
logger.error("Migration failed", {
|
||||||
|
error: error.message,
|
||||||
|
stack: error.stack,
|
||||||
|
});
|
||||||
|
throw error;
|
||||||
|
} finally {
|
||||||
|
// Only close if not called from main app
|
||||||
|
await migrationDb.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run migration if script is called directly
|
||||||
|
if (require.main === module) {
|
||||||
|
migrateJsonToDatabase()
|
||||||
|
.then(() => {
|
||||||
|
console.log("Migration script completed");
|
||||||
|
process.exit(0);
|
||||||
|
})
|
||||||
|
.catch((error) => {
|
||||||
|
console.error("Migration script failed:", error);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { migrateJsonToDatabase };
|
||||||
338
scripts/run-campaigns.js
Normal file
338
scripts/run-campaigns.js
Normal file
@ -0,0 +1,338 @@
|
|||||||
|
const config = require("../config");
|
||||||
|
const nodemailer = require("nodemailer");
|
||||||
|
const delay = require("delay");
|
||||||
|
const fs = require("fs");
|
||||||
|
const templateEngine = require("../lib/templateEngine");
|
||||||
|
const attachmentHandler = require("../lib/attachmentHandler");
|
||||||
|
const rateLimiter = require("../lib/rateLimiter");
|
||||||
|
const errorHandler = require("../lib/errorHandler");
|
||||||
|
const logger = require("../lib/logger");
|
||||||
|
const database = require("../lib/database");
|
||||||
|
const trackingServer = require("../lib/trackingServer");
|
||||||
|
|
||||||
|
// Extract email sending logic for reuse
|
||||||
|
async function sendSingleEmail(mailOptions, recipient, transporter) {
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Sending email to ${recipient}`);
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Subject: ${mailOptions.subject}`);
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Firm: ${mailOptions.firmName}`);
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Template: ${mailOptions.templateName}`);
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Tracking ID: ${mailOptions.trackingId}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
await transporter.sendMail(mailOptions);
|
||||||
|
|
||||||
|
// Log successful email
|
||||||
|
logger.emailSent(
|
||||||
|
recipient,
|
||||||
|
mailOptions.subject,
|
||||||
|
mailOptions.firmName,
|
||||||
|
config.email.testMode
|
||||||
|
);
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`✅ Email sent to ${recipient}${
|
||||||
|
config.email.testMode ? " (TEST MODE)" : ""
|
||||||
|
}`
|
||||||
|
);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign] Email successfully delivered to ${recipient}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate test recipients automatically based on campaign count
|
||||||
|
function generateCampaignTestRecipients(campaignCount) {
|
||||||
|
const baseEmail = config.email.user; // Use your own email as base
|
||||||
|
const [localPart, domain] = baseEmail.split("@");
|
||||||
|
|
||||||
|
const recipients = [];
|
||||||
|
for (let i = 1; i <= campaignCount; i++) {
|
||||||
|
// Create test recipients like: yourname+campaign1@gmail.com, yourname+campaign2@gmail.com
|
||||||
|
recipients.push(`${localPart}+campaign${i}@${domain}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: Generated ${campaignCount} campaign test recipients:`
|
||||||
|
);
|
||||||
|
recipients.forEach((email, index) => {
|
||||||
|
console.log(`🐛 DEBUG: Campaign ${index + 1}: ${email}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return recipients;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runCampaigns() {
|
||||||
|
try {
|
||||||
|
// Initialize database
|
||||||
|
await database.init();
|
||||||
|
|
||||||
|
// Load campaign test data
|
||||||
|
const testDataFile = config.campaigns.testDataFile;
|
||||||
|
if (!fs.existsSync(testDataFile)) {
|
||||||
|
throw new Error(`Campaign test data file not found: ${testDataFile}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
const testData = JSON.parse(fs.readFileSync(testDataFile, "utf8"));
|
||||||
|
let campaigns = testData.TestCampaigns || [];
|
||||||
|
|
||||||
|
if (campaigns.length === 0) {
|
||||||
|
throw new Error("No campaigns found in test data file");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate test recipients if in test mode
|
||||||
|
if (config.email.testMode) {
|
||||||
|
const testRecipients = generateCampaignTestRecipients(campaigns.length);
|
||||||
|
|
||||||
|
// Update campaigns with generated test recipients
|
||||||
|
campaigns = campaigns.map((campaign, index) => ({
|
||||||
|
...campaign,
|
||||||
|
contactEmail: testRecipients[index] || campaign.contactEmail,
|
||||||
|
}));
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`🧪 TEST MODE: Updated ${campaigns.length} campaigns with auto-generated recipients`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(`🚀 Running ${campaigns.length} different campaigns`);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign] Loaded campaigns from: ${testDataFile}`
|
||||||
|
);
|
||||||
|
campaigns.forEach((campaign, index) => {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign ${index + 1}] ${campaign.campaign} → ${
|
||||||
|
campaign.firmName
|
||||||
|
} → ${campaign.contactEmail}`
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign ${index + 1}] Subject: ${campaign.subject}`
|
||||||
|
);
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign ${index + 1}] Template: ${campaign.campaign}`
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start tracking server if enabled and not skipping tracking
|
||||||
|
if (config.tracking.enabled && !config.tracking.skipTracking) {
|
||||||
|
try {
|
||||||
|
await trackingServer.start();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to start tracking server", {
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
console.warn(
|
||||||
|
"⚠️ Tracking server failed to start. Continuing without tracking."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else if (config.tracking.skipTracking) {
|
||||||
|
console.log("📊 Tracking disabled - SKIP_TRACKING enabled");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Setup email transporter
|
||||||
|
const transportConfig = {
|
||||||
|
auth: {
|
||||||
|
user: config.smtp.user || config.email.user,
|
||||||
|
pass: config.smtp.pass || config.email.pass,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
if (config.smtp.host) {
|
||||||
|
transportConfig.host = config.smtp.host;
|
||||||
|
transportConfig.port = config.smtp.port;
|
||||||
|
transportConfig.secure = config.smtp.secure;
|
||||||
|
} else {
|
||||||
|
transportConfig.service = "gmail";
|
||||||
|
}
|
||||||
|
|
||||||
|
const transporter = nodemailer.createTransport(transportConfig);
|
||||||
|
|
||||||
|
// Get attachments once at start
|
||||||
|
const attachments = await attachmentHandler.getAttachments();
|
||||||
|
if (attachments.length > 0) {
|
||||||
|
console.log(`📎 Attaching ${attachments.length} file(s) to emails`);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process each campaign
|
||||||
|
for (let i = 0; i < campaigns.length; i++) {
|
||||||
|
const campaign = campaigns[i];
|
||||||
|
console.log(
|
||||||
|
`\n📧 Campaign ${i + 1}/${campaigns.length}: ${campaign.campaign}`
|
||||||
|
);
|
||||||
|
console.log(` Firm: ${campaign.firmName}`);
|
||||||
|
console.log(` Template: ${campaign.campaign}`);
|
||||||
|
console.log(` Subject: ${campaign.subject}`);
|
||||||
|
|
||||||
|
// Create campaign in database
|
||||||
|
const campaignId = await database.createCampaign({
|
||||||
|
name: `${campaign.campaign} - ${
|
||||||
|
new Date().toISOString().split("T")[0]
|
||||||
|
}`,
|
||||||
|
subject: campaign.subject,
|
||||||
|
templateName: campaign.campaign,
|
||||||
|
testMode: config.email.testMode,
|
||||||
|
});
|
||||||
|
|
||||||
|
await database.startCampaign(campaignId, 1);
|
||||||
|
|
||||||
|
// Format firm data for template
|
||||||
|
const templateData = templateEngine.formatFirmData({
|
||||||
|
firmName: campaign.firmName,
|
||||||
|
location: campaign.location,
|
||||||
|
website: campaign.website,
|
||||||
|
contactEmail: campaign.contactEmail,
|
||||||
|
email: campaign.contactEmail,
|
||||||
|
state: campaign.state,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Use campaign contact email directly (contains generated test recipient in test mode)
|
||||||
|
const recipient = campaign.contactEmail;
|
||||||
|
|
||||||
|
// Double check: Ensure we have a valid recipient (auto-generated in test mode)
|
||||||
|
if (config.email.testMode && !recipient) {
|
||||||
|
throw new Error("Test mode error: No recipient generated for campaign");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Log what we're doing
|
||||||
|
if (config.email.testMode) {
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(`🐛 DEBUG: [Campaign] Campaign ${i + 1} → ${recipient}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
console.log(
|
||||||
|
`🧪 TEST MODE: Email for ${campaign.firmName} → ${recipient}`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate unique tracking ID
|
||||||
|
const trackingId = `${campaignId}_${campaign.contactEmail}_${Date.now()}`;
|
||||||
|
|
||||||
|
// Render email using template
|
||||||
|
const emailContent = await templateEngine.render(campaign.campaign, {
|
||||||
|
...templateData,
|
||||||
|
subject: campaign.subject,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Add tracking to HTML content (unless skip tracking is enabled)
|
||||||
|
const trackedHtmlContent = trackingServer.addEmailTracking(
|
||||||
|
emailContent.html,
|
||||||
|
trackingId,
|
||||||
|
config.tracking.skipTracking
|
||||||
|
);
|
||||||
|
|
||||||
|
const mailOptions = {
|
||||||
|
from: config.email.user,
|
||||||
|
to: recipient,
|
||||||
|
subject: campaign.subject,
|
||||||
|
text: emailContent.text,
|
||||||
|
html: trackedHtmlContent,
|
||||||
|
attachments: attachments,
|
||||||
|
firmName: campaign.firmName,
|
||||||
|
trackingId: trackingId,
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
await sendSingleEmail(mailOptions, recipient, transporter);
|
||||||
|
|
||||||
|
// Record success
|
||||||
|
rateLimiter.recordSuccess();
|
||||||
|
|
||||||
|
// Show progress
|
||||||
|
const stats = rateLimiter.getStats();
|
||||||
|
console.log(
|
||||||
|
`📊 Progress: ${stats.sentCount} sent, ${stats.averageRate} emails/hour`
|
||||||
|
);
|
||||||
|
|
||||||
|
// Complete campaign
|
||||||
|
await database.completeCampaign(campaignId, {
|
||||||
|
sentEmails: 1,
|
||||||
|
failedEmails: 0,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.error(
|
||||||
|
`❌ Failed to send campaign ${campaign.campaign}:`,
|
||||||
|
error.message
|
||||||
|
);
|
||||||
|
|
||||||
|
// Complete campaign with failure
|
||||||
|
await database.completeCampaign(campaignId, {
|
||||||
|
sentEmails: 0,
|
||||||
|
failedEmails: 1,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add delay between campaigns (except for the last one)
|
||||||
|
if (i < campaigns.length - 1) {
|
||||||
|
const delayMs = rateLimiter.getRandomDelay();
|
||||||
|
console.log(
|
||||||
|
`⏱️ Next campaign in: ${rateLimiter.formatDelay(delayMs)}`
|
||||||
|
);
|
||||||
|
|
||||||
|
if (config.logging.debug) {
|
||||||
|
console.log(
|
||||||
|
`🐛 DEBUG: [Campaign] Delaying ${delayMs}ms before next campaign`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use setTimeout wrapped in Promise instead of delay package
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, delayMs));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Final stats
|
||||||
|
const finalStats = rateLimiter.getStats();
|
||||||
|
console.log(`\n📈 All campaigns complete! Final stats:`, finalStats);
|
||||||
|
|
||||||
|
// Stop tracking server if it was started
|
||||||
|
if (config.tracking.enabled && !config.tracking.skipTracking) {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to stop tracking server", {
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
console.error("Campaign runner failed:", error);
|
||||||
|
|
||||||
|
// Stop tracking server on error
|
||||||
|
if (process.env.TRACKING_ENABLED === "true") {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (stopError) {
|
||||||
|
logger.error("Failed to stop tracking server after error", {
|
||||||
|
error: stopError.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle graceful shutdown
|
||||||
|
process.on("SIGINT", async () => {
|
||||||
|
console.log("\n🛑 Received SIGINT. Gracefully shutting down...");
|
||||||
|
|
||||||
|
if (process.env.TRACKING_ENABLED === "true") {
|
||||||
|
try {
|
||||||
|
await trackingServer.stop();
|
||||||
|
} catch (error) {
|
||||||
|
logger.error("Failed to stop tracking server during shutdown", {
|
||||||
|
error: error.message,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
process.exit(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
runCampaigns().catch(console.error);
|
||||||
204
suggestions.md
Normal file
204
suggestions.md
Normal file
@ -0,0 +1,204 @@
|
|||||||
|
# 📧 Outreach Engine - Enhancement Suggestions
|
||||||
|
|
||||||
|
This document outlines potential improvements for the email outreach application. Items marked ✅ are implemented.
|
||||||
|
|
||||||
|
## ✅ Implemented Features
|
||||||
|
|
||||||
|
### 1. **HTML Email Support** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Professional HTML templates with Handlebars
|
||||||
|
- ✅ Plain text fallback for compatibility
|
||||||
|
- ✅ Responsive design with styling
|
||||||
|
|
||||||
|
### 2. **Advanced Personalization** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Dynamic content using firm data (name, location, website)
|
||||||
|
- ✅ Template engine with Handlebars
|
||||||
|
- ✅ Conditional content based on available data
|
||||||
|
|
||||||
|
### 3. **Email Templates** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Handlebars template system
|
||||||
|
- ✅ Multiple template support
|
||||||
|
- ✅ HTML and text versions
|
||||||
|
|
||||||
|
### 4. **Configuration Management** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Separate dev/production environments
|
||||||
|
- ✅ Environment-based configuration
|
||||||
|
- ✅ Centralized config module
|
||||||
|
|
||||||
|
### 5. **Enhanced Rate Limiting** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Randomized delays with jitter
|
||||||
|
- ✅ Automatic pausing based on volume
|
||||||
|
- ✅ Real-time statistics and monitoring
|
||||||
|
|
||||||
|
### 6. **Attachment Support** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ PDF, Word, image attachments
|
||||||
|
- ✅ File size validation
|
||||||
|
- ✅ Configurable attachment lists
|
||||||
|
|
||||||
|
### 7. **GIF Support** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Inline GIFs in email body
|
||||||
|
- ✅ Auto-play on email open
|
||||||
|
- ✅ Professional styling
|
||||||
|
|
||||||
|
### 8. **Error Handling & Retries** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Smart error classification (auth, network, recipient, etc.)
|
||||||
|
- ✅ Exponential backoff retry logic
|
||||||
|
- ✅ Permanent failure detection
|
||||||
|
- ✅ Retry queue management
|
||||||
|
|
||||||
|
### 9. **Winston Logging** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Structured JSON logging with timestamps
|
||||||
|
- ✅ Multiple log files (error, combined, email-activity)
|
||||||
|
- ✅ Log rotation and file size management
|
||||||
|
- ✅ Development console output
|
||||||
|
|
||||||
|
### 10. **Database Integration** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ SQLite database with optimized schema
|
||||||
|
- ✅ Campaign and email send tracking
|
||||||
|
- ✅ Automatic JSON migration
|
||||||
|
- ✅ Advanced analytics and reporting
|
||||||
|
|
||||||
|
### 11. **Testing Framework** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Jest testing with comprehensive mocks
|
||||||
|
- ✅ Test coverage reporting (`npm run test:coverage`)
|
||||||
|
- ✅ Module-specific test suites
|
||||||
|
- ✅ Watch mode for development (`npm run test:watch`)
|
||||||
|
- ✅ Continuous integration ready
|
||||||
|
- ✅ Configuration validation scripts (`npm run test-config`)
|
||||||
|
- ✅ JSON validation utilities (`npm run validate-json`)
|
||||||
|
|
||||||
|
### 12. **Open Tracking** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Invisible 1x1 pixel tracking
|
||||||
|
- ✅ HTTP tracking server
|
||||||
|
- ✅ Detailed analytics logging
|
||||||
|
- ✅ IP and user agent capture
|
||||||
|
|
||||||
|
### 13. **Click Tracking** ✅ DONE
|
||||||
|
|
||||||
|
- ✅ Automatic link wrapping
|
||||||
|
- ✅ Redirect tracking with analytics
|
||||||
|
- ✅ Link-specific statistics
|
||||||
|
- ✅ Target URL preservation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Future Enhancement Ideas
|
||||||
|
|
||||||
|
### Phase 1: Remaining Core Features
|
||||||
|
|
||||||
|
#### **Duplicate Management**
|
||||||
|
|
||||||
|
- **What**: Advanced deduplication beyond email addresses
|
||||||
|
- **Implementation**: Fuzzy matching on firm names, domains
|
||||||
|
- **Benefits**: Cleaner data and reduced redundant outreach
|
||||||
|
|
||||||
|
#### **Development Tooling** ✅ IMPLEMENTED
|
||||||
|
|
||||||
|
- **What**: Comprehensive development and testing utilities
|
||||||
|
- **Implementation**:
|
||||||
|
- Jest testing framework with watch mode and coverage
|
||||||
|
- Configuration validation scripts
|
||||||
|
- JSON syntax validation utilities
|
||||||
|
- npm scripts for common tasks
|
||||||
|
- **Benefits**: Improved development workflow, code reliability, and easier debugging
|
||||||
|
|
||||||
|
### Phase 3: Advanced Features
|
||||||
|
|
||||||
|
#### **Campaign Management**
|
||||||
|
|
||||||
|
- **What**: Multiple campaign support with tracking
|
||||||
|
- **Implementation**: Campaign IDs, segmented lists, scheduling
|
||||||
|
- **Benefits**: Organized multi-stage outreach
|
||||||
|
|
||||||
|
#### **Queue Management**
|
||||||
|
|
||||||
|
- **What**: Background job processing
|
||||||
|
- **Implementation**: Bull Queue or similar
|
||||||
|
- **Benefits**: Non-blocking operations
|
||||||
|
|
||||||
|
#### **API Integration**
|
||||||
|
|
||||||
|
- **What**: Pull firm data from external sources
|
||||||
|
- **Implementation**: CRM APIs, legal directories
|
||||||
|
- **Benefits**: Automated data collection
|
||||||
|
|
||||||
|
### Phase 4: Analytics & Monitoring
|
||||||
|
|
||||||
|
#### **Health Monitoring**
|
||||||
|
|
||||||
|
- **What**: Application performance monitoring
|
||||||
|
- **Implementation**: Health check endpoints, error alerting
|
||||||
|
- **Benefits**: Proactive issue detection
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 Delivery Tracking Explanation
|
||||||
|
|
||||||
|
**How Email Delivery Tracking Would Work:**
|
||||||
|
|
||||||
|
### 1. **Open Tracking**
|
||||||
|
|
||||||
|
- Embed 1x1 pixel image in HTML emails
|
||||||
|
- When email opens, pixel loads from your server
|
||||||
|
- Log timestamp, IP, user agent
|
||||||
|
|
||||||
|
### 2. **Click Tracking**
|
||||||
|
|
||||||
|
- Replace all links with tracking URLs
|
||||||
|
- Redirect through your server to log clicks
|
||||||
|
- Track which links are most effective
|
||||||
|
|
||||||
|
### 3. **Bounce Tracking**
|
||||||
|
|
||||||
|
- Configure webhook with email provider
|
||||||
|
- Receive notifications for bounced emails
|
||||||
|
- Update database to mark invalid addresses
|
||||||
|
|
||||||
|
### 4. **Implementation Approach**
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
// Example tracking pixel
|
||||||
|
<img src="https://yourdomain.com/track/open/{{emailId}}" width="1" height="1" />
|
||||||
|
|
||||||
|
// Example click tracking
|
||||||
|
<a href="https://yourdomain.com/track/click/{{emailId}}/{{linkId}}">Original Link</a>
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. **Privacy Considerations**
|
||||||
|
|
||||||
|
- Include tracking disclosure in emails
|
||||||
|
- Respect user privacy preferences
|
||||||
|
- Comply with GDPR/privacy laws
|
||||||
|
- Provide opt-out mechanisms
|
||||||
|
|
||||||
|
### 6. **Technical Requirements**
|
||||||
|
|
||||||
|
- Web server for tracking endpoints
|
||||||
|
- Database for storing tracking data
|
||||||
|
- Analytics dashboard for viewing data
|
||||||
|
- Integration with email sending system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Implementation Notes
|
||||||
|
|
||||||
|
### Next Steps Priority:
|
||||||
|
|
||||||
|
1. **Error Handling** - Critical for production reliability
|
||||||
|
2. **Logging** - Essential for debugging and monitoring
|
||||||
|
3. **Database** - Scalability and performance improvement
|
||||||
|
4. **Testing** - Code quality and maintainability
|
||||||
|
5. **Advanced Features** - Enhanced functionality
|
||||||
|
|
||||||
|
_These enhancements build upon the already implemented foundation to create a professional-grade email outreach system._
|
||||||
42
templates/campaign-1-saas.html
Normal file
42
templates/campaign-1-saas.html
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: Arial, sans-serif; line-height: 1.6; color: #333; max-width: 600px; margin: 0 auto; padding: 20px; }
|
||||||
|
h1 { color: #2c3e50; }
|
||||||
|
.cta { background: #e74c3c; color: white; padding: 12px 24px; text-decoration: none; border-radius: 5px; font-weight: bold; }
|
||||||
|
.price { background: #f8f9fa; padding: 15px; border-left: 4px solid #e74c3c; margin: 15px 0; }
|
||||||
|
@media (max-width: 600px) { body { padding: 10px; } }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>🚀 LinkedIn Employment Intelligence API for {{firmName}}</h1>
|
||||||
|
<p>Hello {{firmName}} team,</p>
|
||||||
|
<p>I've developed an <strong>automated LinkedIn parser</strong> that finds recently terminated employees in {{location}} and {{state}} - perfect for employment law leads.</p>
|
||||||
|
|
||||||
|
<h2>What You Get:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>📡 <strong>Real-time API access</strong> to query layoffs, terminations, wrongful dismissals</li>
|
||||||
|
<li>🎯 <strong>Location filtering</strong> for {{location}} and areas where you practice</li>
|
||||||
|
<li>🤖 <strong>AI analysis</strong> of posts (sentiment, urgency, potential case value)</li>
|
||||||
|
<li>📊 <strong>Clean JSON data</strong> with contact info, company details, timeline</li>
|
||||||
|
<li>🔄 <strong>Daily updates</strong> - never miss a potential client</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<div class="price">
|
||||||
|
<strong>Pricing:</strong> $299/month - Cancel anytime<br>
|
||||||
|
<em>First 50 queries free to test the quality</em>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p>This runs 24/7 and finds cases your competitors miss. Ready to see it in action?</p>
|
||||||
|
|
||||||
|
<p><a href="mailto:{{contactEmail}}?subject=LinkedIn Parser Demo Request" class="cta">Request Free Demo</a></p>
|
||||||
|
|
||||||
|
<p>Best regards,<br>
|
||||||
|
[Your Name]<br>
|
||||||
|
LinkedIn Employment Intelligence</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
42
templates/campaign-2-data-service.html
Normal file
42
templates/campaign-2-data-service.html
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: Arial, sans-serif; line-height: 1.6; color: #333; max-width: 600px; margin: 0 auto; padding: 20px; }
|
||||||
|
h1 { color: #2c3e50; }
|
||||||
|
.cta { background: #27ae60; color: white; padding: 12px 24px; text-decoration: none; border-radius: 5px; font-weight: bold; }
|
||||||
|
.price { background: #f8f9fa; padding: 15px; border-left: 4px solid #27ae60; margin: 15px 0; }
|
||||||
|
@media (max-width: 600px) { body { padding: 10px; } }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>📊 Monthly Employment Intelligence Reports for {{firmName}}</h1>
|
||||||
|
<p>Hello {{firmName}},</p>
|
||||||
|
<p>What if you received a <strong>curated monthly report</strong> of employment law opportunities in {{location}} and {{state}} - delivered to your inbox?</p>
|
||||||
|
|
||||||
|
<h2>What You Receive:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>📋 <strong>Monthly PDF report</strong> with 50-100 qualified leads</li>
|
||||||
|
<li>🎯 <strong>{{location}}-specific</strong> layoffs, terminations, workplace issues</li>
|
||||||
|
<li>🔍 <strong>Pre-screened contacts</strong> with AI-analyzed case potential</li>
|
||||||
|
<li>📈 <strong>Trend analysis</strong> - which companies are laying off most</li>
|
||||||
|
<li>⚡ <strong>Urgent alerts</strong> for high-value cases (mass layoffs, discrimination)</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<div class="price">
|
||||||
|
<strong>Investment:</strong> $499/month<br>
|
||||||
|
<em>Includes setup, monthly reports, and urgent alerts</em>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p>I handle all the technical work - you focus on winning cases. No software to install, no training needed.</p>
|
||||||
|
|
||||||
|
<p><a href="mailto:{{contactEmail}}?subject=Sample Report Request" class="cta">Get Sample Report</a></p>
|
||||||
|
|
||||||
|
<p>Best regards,<br>
|
||||||
|
[Your Name]<br>
|
||||||
|
Employment Intelligence Services</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
42
templates/campaign-3-license.html
Normal file
42
templates/campaign-3-license.html
Normal file
@ -0,0 +1,42 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: Arial, sans-serif; line-height: 1.6; color: #333; max-width: 600px; margin: 0 auto; padding: 20px; }
|
||||||
|
h1 { color: #2c3e50; }
|
||||||
|
.cta { background: #8e44ad; color: white; padding: 12px 24px; text-decoration: none; border-radius: 5px; font-weight: bold; }
|
||||||
|
.price { background: #f8f9fa; padding: 15px; border-left: 4px solid #8e44ad; margin: 15px 0; }
|
||||||
|
@media (max-width: 600px) { body { padding: 10px; } }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>🏆 Own Your LinkedIn Employment Parser - {{firmName}}</h1>
|
||||||
|
<p>Dear {{firmName}} team,</p>
|
||||||
|
<p>Instead of paying monthly fees, what if you could <strong>own</strong> a custom LinkedIn parser built specifically for {{location}} employment law?</p>
|
||||||
|
|
||||||
|
<h2>What You Get:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>💻 <strong>Complete source code</strong> - you own it forever</li>
|
||||||
|
<li>🛠️ <strong>Custom installation</strong> on your servers/computer</li>
|
||||||
|
<li>🎯 <strong>Pre-configured</strong> for {{location}} and {{state}} searches</li>
|
||||||
|
<li>📚 <strong>Full documentation</strong> and training for your team</li>
|
||||||
|
<li>🔧 <strong>6 months support</strong> included for updates/fixes</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<div class="price">
|
||||||
|
<strong>One-time Investment:</strong> $4,999<br>
|
||||||
|
<em>Optional: $299/month maintenance & updates</em>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p>No ongoing fees, no data limits, complete control. Perfect for firms that want to keep everything in-house.</p>
|
||||||
|
|
||||||
|
<p><a href="mailto:{{contactEmail}}?subject=Custom Parser License Inquiry" class="cta">Schedule Demo</a></p>
|
||||||
|
|
||||||
|
<p>Best regards,<br>
|
||||||
|
[Your Name]<br>
|
||||||
|
Custom Legal Technology Solutions</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
44
templates/campaign-4-gui.html
Normal file
44
templates/campaign-4-gui.html
Normal file
@ -0,0 +1,44 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: Arial, sans-serif; line-height: 1.6; color: #333; max-width: 600px; margin: 0 auto; padding: 20px; }
|
||||||
|
h1 { color: #2c3e50; }
|
||||||
|
.cta { background: #3498db; color: white; padding: 12px 24px; text-decoration: none; border-radius: 5px; font-weight: bold; }
|
||||||
|
.price { background: #f8f9fa; padding: 15px; border-left: 4px solid #3498db; margin: 15px 0; }
|
||||||
|
@media (max-width: 600px) { body { padding: 10px; } }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>⚡ LinkedIn Employment Dashboard for {{firmName}}</h1>
|
||||||
|
<p>Hello {{firmName}},</p>
|
||||||
|
<p>Introducing a <strong>web-based dashboard</strong> where you can search for employment law opportunities in {{location}} with just a few clicks.</p>
|
||||||
|
|
||||||
|
<h2>How It Works:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>🌐 <strong>Login to secure portal</strong> - no software to install</li>
|
||||||
|
<li>🔍 <strong>Enter search terms</strong> ("laid off", "wrongful termination", etc.)</li>
|
||||||
|
<li>📍 <strong>Set location</strong> to {{location}}, {{state}}, or expand radius</li>
|
||||||
|
<li>⏰ <strong>Get results in minutes</strong> - contact info, company details, post analysis</li>
|
||||||
|
<li>📤 <strong>Export to CSV</strong> for your CRM or follow-up system</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<div class="price">
|
||||||
|
<strong>Pricing:</strong><br>
|
||||||
|
• Basic: $199/month (100 searches)<br>
|
||||||
|
• Professional: $399/month (500 searches)<br>
|
||||||
|
• Enterprise: $699/month (unlimited)
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p>Easy to use, no technical skills required. Your team can start finding cases today.</p>
|
||||||
|
|
||||||
|
<p><a href="mailto:{{contactEmail}}?subject=Dashboard Demo Request" class="cta">Try Free Demo</a></p>
|
||||||
|
|
||||||
|
<p>Best regards,<br>
|
||||||
|
[Your Name]<br>
|
||||||
|
Employment Search Platform</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
85
templates/employment-layer.html
Normal file
85
templates/employment-layer.html
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body {
|
||||||
|
font-family: Arial, sans-serif;
|
||||||
|
line-height: 1.6;
|
||||||
|
color: #333;
|
||||||
|
max-width: 600px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 20px;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
color: #2c3e50;
|
||||||
|
}
|
||||||
|
.cta {
|
||||||
|
background: #3498db;
|
||||||
|
color: white;
|
||||||
|
padding: 10px 20px;
|
||||||
|
text-decoration: none;
|
||||||
|
border-radius: 5px;
|
||||||
|
}
|
||||||
|
@media (max-width: 600px) {
|
||||||
|
body {
|
||||||
|
padding: 10px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>
|
||||||
|
Revolutionary LinkedIn Parser: Unlock Employment Insights for {{firmName}}
|
||||||
|
</h1>
|
||||||
|
<p>Dear {{firmName}} team in {{location}},</p>
|
||||||
|
<p>
|
||||||
|
Imagine a tool that intelligently queries LinkedIn for professionals who
|
||||||
|
may have been recently let go, fired, or terminated – providing your firm
|
||||||
|
with targeted leads for employment law cases or recruitment. Our
|
||||||
|
<strong>Employment Layer Parser</strong> does exactly that, with powerful
|
||||||
|
features optimized for maximum value.
|
||||||
|
</p>
|
||||||
|
<h2>Key Features & Value:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>
|
||||||
|
<strong>Automated Login & Search</strong>: Seamlessly logs into LinkedIn
|
||||||
|
and searches posts using keywords (e.g., 'laid off', 'terminated') from
|
||||||
|
CSVs or CLI – saves hours of manual browsing.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Flexible & Customizable Queries with Location Targeting</strong
|
||||||
|
>: Adjust date ranges, sorting. Filter by location to target the areas
|
||||||
|
where you can practice, such as {{location}} or {{state}}-specific
|
||||||
|
employment transitions for licensed jurisdictions.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Duplicate Prevention & Clean Data</strong>: Detects duplicates,
|
||||||
|
removes noise (hashtags, emojis, URLs), and timestamps results in JSON –
|
||||||
|
delivers clean, actionable data without redundancy.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Local AI Analysis</strong>: Integrates free local Ollama models
|
||||||
|
for private, fast post-processing (e.g., sentiment analysis on layoff
|
||||||
|
posts) – adds deep insights without cloud costs or privacy risks.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Geographic & Runtime Flexibility</strong>: Validates locations
|
||||||
|
accurately and supports CLI overrides – ideal for targeted campaigns or
|
||||||
|
ad-hoc runs.
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
This parser maximizes value by combining automation, AI, and precision –
|
||||||
|
helping {{firmName}} identify opportunities in employment transitions.
|
||||||
|
Let's partner to integrate it into your workflow!
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<a href="{{website}}" class="cta">Learn More</a> or reply to
|
||||||
|
{{contactEmail}}.
|
||||||
|
</p>
|
||||||
|
<p>Best regards,<br />Your Name<br />Your Company</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
80
templates/general-intro.html
Normal file
80
templates/general-intro.html
Normal file
@ -0,0 +1,80 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body {
|
||||||
|
font-family: Arial, sans-serif;
|
||||||
|
line-height: 1.6;
|
||||||
|
color: #333;
|
||||||
|
max-width: 600px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 20px;
|
||||||
|
}
|
||||||
|
h1 {
|
||||||
|
color: #2c3e50;
|
||||||
|
}
|
||||||
|
.cta {
|
||||||
|
background: #3498db;
|
||||||
|
color: white;
|
||||||
|
padding: 10px 20px;
|
||||||
|
text-decoration: none;
|
||||||
|
border-radius: 5px;
|
||||||
|
}
|
||||||
|
@media (max-width: 600px) {
|
||||||
|
body {
|
||||||
|
padding: 10px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>
|
||||||
|
Exciting Partnership Opportunity: Advanced Data Bot for {{firmName}}
|
||||||
|
</h1>
|
||||||
|
<p>Dear {{firmName}} team in {{location}},</p>
|
||||||
|
<p>
|
||||||
|
I'm reaching out from my development team with an innovative tool that
|
||||||
|
could transform how your firm handles data: a
|
||||||
|
<strong>general-purpose bot</strong> designed for automated data parsing,
|
||||||
|
extraction, filtering, formatting, presentation, and reporting.
|
||||||
|
</p>
|
||||||
|
<h2>Key Features:</h2>
|
||||||
|
<ul>
|
||||||
|
<li>
|
||||||
|
<strong>Smart Parsing & Extraction</strong>: Automatically pulls and
|
||||||
|
structures data from various sources (e.g., websites, APIs, documents).
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>AI-Powered Filtering with Location Targeting</strong>: Uses AI
|
||||||
|
for contextual dimensions (e.g., sentiment, relevance) alongside
|
||||||
|
traditional filters (e.g., keywords, dates). Filter by location to
|
||||||
|
target the areas where you can practice, such as {{location}} or
|
||||||
|
{{state}}-specific data for licensed jurisdictions.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Formatting & Presentation</strong>: Converts raw data into
|
||||||
|
clean, visual formats (charts, tables) for easy insights.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Scheduling & Automation</strong>: Runs on custom schedules,
|
||||||
|
generates reports, and integrates with tools like email or databases.
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>Customization</strong>: Tailored for legal workflows, such as
|
||||||
|
case research or client data management.
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
This bot could streamline your operations at {{firmName}}, saving time and
|
||||||
|
uncovering hidden insights. Let's discuss a potential partnership!
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<a href="{{website}}" class="cta">Visit Our Site</a> or reply to this
|
||||||
|
email at {{contactEmail}}.
|
||||||
|
</p>
|
||||||
|
<p>Best regards,<br />Your Name<br />Your Company</p>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
107
templates/outreach.html
Normal file
107
templates/outreach.html
Normal file
@ -0,0 +1,107 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>{{subject}}</title>
|
||||||
|
<style>
|
||||||
|
body {
|
||||||
|
font-family: Arial, sans-serif;
|
||||||
|
line-height: 1.6;
|
||||||
|
color: #333;
|
||||||
|
max-width: 600px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 20px;
|
||||||
|
}
|
||||||
|
.header {
|
||||||
|
background-color: #f4f4f4;
|
||||||
|
padding: 20px;
|
||||||
|
text-align: center;
|
||||||
|
border-radius: 5px;
|
||||||
|
}
|
||||||
|
.content {
|
||||||
|
padding: 20px 0;
|
||||||
|
}
|
||||||
|
.footer {
|
||||||
|
text-align: center;
|
||||||
|
padding: 20px;
|
||||||
|
font-size: 12px;
|
||||||
|
color: #666;
|
||||||
|
}
|
||||||
|
.button {
|
||||||
|
display: inline-block;
|
||||||
|
padding: 10px 20px;
|
||||||
|
background-color: #007bff;
|
||||||
|
color: white;
|
||||||
|
text-decoration: none;
|
||||||
|
border-radius: 5px;
|
||||||
|
margin: 10px 0;
|
||||||
|
}
|
||||||
|
.firm-info {
|
||||||
|
background-color: #f9f9f9;
|
||||||
|
padding: 10px;
|
||||||
|
border-left: 3px solid #007bff;
|
||||||
|
margin: 10px 0;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div class="header">
|
||||||
|
<h1>Legal Partnership Opportunity</h1>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="content">
|
||||||
|
<p>Dear {{greeting}},</p>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
I hope this email finds you well. I'm reaching out to
|
||||||
|
<strong>{{firmName}}</strong> {{#if location}}in {{location}}{{/if}}
|
||||||
|
regarding an exciting business opportunity.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
{{#if website}}
|
||||||
|
<div class="firm-info">
|
||||||
|
<p>
|
||||||
|
We've reviewed your firm's profile and are impressed by your practice
|
||||||
|
areas and expertise showcased on your website at
|
||||||
|
<a href="{{website}}">{{website}}</a>.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
{{/if}}
|
||||||
|
|
||||||
|
<p>
|
||||||
|
We have a unique proposition that could benefit your firm significantly:
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<ul>
|
||||||
|
<li>Expand your client base with qualified leads</li>
|
||||||
|
<li>Access to cutting-edge legal technology solutions</li>
|
||||||
|
<li>Strategic partnership opportunities</li>
|
||||||
|
<li>Revenue growth potential of 20-30%</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
I would love to schedule a brief 15-minute call to discuss how we can
|
||||||
|
help {{firmName}} achieve its growth objectives.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p style="text-align: center">
|
||||||
|
<a href="mailto:{{fromEmail}}?subject=Re: {{subject}}" class="button">
|
||||||
|
Schedule a Call
|
||||||
|
</a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
Best regards,<br />
|
||||||
|
{{senderName}}<br />
|
||||||
|
{{senderTitle}}<br />
|
||||||
|
{{senderCompany}}
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="footer">
|
||||||
|
<p>This email was sent to {{email}} for {{firmName}}</p>
|
||||||
|
<p>To unsubscribe, reply with "UNSUBSCRIBE" in the subject line</p>
|
||||||
|
</div>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
16
tests/jest.config.js
Normal file
16
tests/jest.config.js
Normal file
@ -0,0 +1,16 @@
|
|||||||
|
module.exports = {
|
||||||
|
testEnvironment: "node",
|
||||||
|
collectCoverage: true,
|
||||||
|
coverageDirectory: "../coverage",
|
||||||
|
coverageReporters: ["text", "lcov", "html"],
|
||||||
|
collectCoverageFrom: [
|
||||||
|
"../lib/**/*.js",
|
||||||
|
"../config/**/*.js",
|
||||||
|
"!../lib/**/*.test.js",
|
||||||
|
"!**/node_modules/**",
|
||||||
|
],
|
||||||
|
testMatch: ["**/__tests__/**/*.js", "**/?(*.)+(spec|test).js"],
|
||||||
|
setupFilesAfterEnv: ["<rootDir>/setup.js"],
|
||||||
|
testTimeout: 10000,
|
||||||
|
verbose: true,
|
||||||
|
};
|
||||||
195
tests/lib/errorHandler.test.js
Normal file
195
tests/lib/errorHandler.test.js
Normal file
@ -0,0 +1,195 @@
|
|||||||
|
const { describe, test, expect, beforeEach } = require("@jest/globals");
|
||||||
|
|
||||||
|
// Mock config
|
||||||
|
jest.mock("../../config", () => ({
|
||||||
|
errorHandling: {
|
||||||
|
maxRetries: 3,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Mock logger
|
||||||
|
jest.mock("../../lib/logger", () => ({
|
||||||
|
emailFailed: jest.fn(),
|
||||||
|
emailRetry: jest.fn(),
|
||||||
|
emailPermanentFailure: jest.fn(),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const errorHandler = require("../../lib/errorHandler");
|
||||||
|
|
||||||
|
describe("ErrorHandler", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
jest.clearAllMocks();
|
||||||
|
errorHandler.clearRetries();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("classifyError", () => {
|
||||||
|
test("should classify authentication errors", () => {
|
||||||
|
const error = new Error("Invalid login credentials");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("AUTH_ERROR");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should classify rate limit errors", () => {
|
||||||
|
const error = new Error("Rate limit exceeded");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("RATE_LIMIT");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should classify network errors", () => {
|
||||||
|
const error = new Error("Connection timeout");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("NETWORK_ERROR");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should classify recipient errors", () => {
|
||||||
|
const error = new Error("Invalid recipient address");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("RECIPIENT_ERROR");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should classify message errors", () => {
|
||||||
|
const error = new Error("Message too large");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("MESSAGE_ERROR");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should classify unknown errors", () => {
|
||||||
|
const error = new Error("Something unexpected happened");
|
||||||
|
const result = errorHandler.classifyError(error);
|
||||||
|
expect(result).toBe("UNKNOWN_ERROR");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("isRetryable", () => {
|
||||||
|
test("should mark retryable errors as retryable", () => {
|
||||||
|
expect(errorHandler.isRetryable("RATE_LIMIT")).toBe(true);
|
||||||
|
expect(errorHandler.isRetryable("NETWORK_ERROR")).toBe(true);
|
||||||
|
expect(errorHandler.isRetryable("UNKNOWN_ERROR")).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should mark non-retryable errors as non-retryable", () => {
|
||||||
|
expect(errorHandler.isRetryable("AUTH_ERROR")).toBe(false);
|
||||||
|
expect(errorHandler.isRetryable("RECIPIENT_ERROR")).toBe(false);
|
||||||
|
expect(errorHandler.isRetryable("MESSAGE_ERROR")).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getRetryDelay", () => {
|
||||||
|
test("should calculate exponential backoff delay", () => {
|
||||||
|
const delay1 = errorHandler.getRetryDelay(1);
|
||||||
|
const delay2 = errorHandler.getRetryDelay(2);
|
||||||
|
const delay3 = errorHandler.getRetryDelay(3);
|
||||||
|
|
||||||
|
// Each delay should be roughly double the previous (with jitter)
|
||||||
|
expect(delay1).toBeGreaterThan(45000); // ~1 minute with jitter
|
||||||
|
expect(delay1).toBeLessThan(90000);
|
||||||
|
|
||||||
|
expect(delay2).toBeGreaterThan(90000); // ~2 minutes with jitter
|
||||||
|
expect(delay2).toBeLessThan(180000);
|
||||||
|
|
||||||
|
expect(delay3).toBeGreaterThan(180000); // ~4 minutes with jitter
|
||||||
|
expect(delay3).toBeLessThan(360000);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should cap delay at maximum", () => {
|
||||||
|
const delay = errorHandler.getRetryDelay(10); // Very high attempt number
|
||||||
|
expect(delay).toBeLessThanOrEqual(30 * 60 * 1000); // 30 minutes max
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("handleError", () => {
|
||||||
|
test("should schedule retry for retryable errors", async () => {
|
||||||
|
const email = { subject: "Test", firmName: "Test Firm" };
|
||||||
|
const recipient = "test@example.com";
|
||||||
|
const error = new Error("Network timeout");
|
||||||
|
const transporter = {};
|
||||||
|
|
||||||
|
const result = await errorHandler.handleError(
|
||||||
|
email,
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result).toBe(true); // Indicates retry scheduled
|
||||||
|
|
||||||
|
const stats = errorHandler.getRetryStats();
|
||||||
|
expect(stats.totalFailed).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should not schedule retry for non-retryable errors", async () => {
|
||||||
|
const email = { subject: "Test", firmName: "Test Firm" };
|
||||||
|
const recipient = "test@example.com";
|
||||||
|
const error = new Error("Invalid login");
|
||||||
|
const transporter = {};
|
||||||
|
|
||||||
|
const result = await errorHandler.handleError(
|
||||||
|
email,
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result).toBe(false); // Indicates permanent failure
|
||||||
|
|
||||||
|
const stats = errorHandler.getRetryStats();
|
||||||
|
expect(stats.totalFailed).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should not retry after max attempts reached", async () => {
|
||||||
|
const email = { subject: "Test", firmName: "Test Firm" };
|
||||||
|
const recipient = "test@example.com";
|
||||||
|
const error = new Error("Network timeout");
|
||||||
|
const transporter = {};
|
||||||
|
|
||||||
|
// Schedule maximum retries
|
||||||
|
await errorHandler.handleError(email, recipient, error, transporter);
|
||||||
|
await errorHandler.handleError(email, recipient, error, transporter);
|
||||||
|
await errorHandler.handleError(email, recipient, error, transporter);
|
||||||
|
|
||||||
|
// This should be a permanent failure
|
||||||
|
const result = await errorHandler.handleError(
|
||||||
|
email,
|
||||||
|
recipient,
|
||||||
|
error,
|
||||||
|
transporter
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getRetryStats", () => {
|
||||||
|
test("should return correct retry statistics", () => {
|
||||||
|
// Initially no retries
|
||||||
|
let stats = errorHandler.getRetryStats();
|
||||||
|
expect(stats.totalFailed).toBe(0);
|
||||||
|
expect(stats.pendingRetries).toBe(0);
|
||||||
|
expect(stats.readyToRetry).toBe(0);
|
||||||
|
|
||||||
|
// Add a retry item
|
||||||
|
errorHandler.failedEmails.push({
|
||||||
|
recipient: "test@example.com",
|
||||||
|
retryAt: Date.now() + 60000, // 1 minute from now
|
||||||
|
});
|
||||||
|
|
||||||
|
stats = errorHandler.getRetryStats();
|
||||||
|
expect(stats.totalFailed).toBe(1);
|
||||||
|
expect(stats.pendingRetries).toBe(1);
|
||||||
|
expect(stats.readyToRetry).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("clearRetries", () => {
|
||||||
|
test("should clear all retry data", () => {
|
||||||
|
// Add some retry data
|
||||||
|
errorHandler.failedEmails.push({ recipient: "test@example.com" });
|
||||||
|
errorHandler.retryAttempts.set("test-key", 2);
|
||||||
|
|
||||||
|
errorHandler.clearRetries();
|
||||||
|
|
||||||
|
expect(errorHandler.failedEmails).toHaveLength(0);
|
||||||
|
expect(errorHandler.retryAttempts.size).toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
160
tests/lib/rateLimiter.test.js
Normal file
160
tests/lib/rateLimiter.test.js
Normal file
@ -0,0 +1,160 @@
|
|||||||
|
const { describe, test, expect, beforeEach } = require("@jest/globals");
|
||||||
|
|
||||||
|
// Mock config
|
||||||
|
jest.mock("../../config", () => ({
|
||||||
|
app: {
|
||||||
|
delayMinutes: 5,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Mock logger
|
||||||
|
jest.mock("../../lib/logger", () => ({
|
||||||
|
rateLimitPause: jest.fn(),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const rateLimiter = require("../../lib/rateLimiter");
|
||||||
|
|
||||||
|
describe("RateLimiter", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
jest.clearAllMocks();
|
||||||
|
rateLimiter.reset();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getRandomDelay", () => {
|
||||||
|
test("should return delay within expected range", () => {
|
||||||
|
const delay = rateLimiter.getRandomDelay();
|
||||||
|
|
||||||
|
// Base delay is 5 minutes = 300,000ms
|
||||||
|
// With 20% variance and jitter, expect roughly 240,000 to 390,000ms
|
||||||
|
expect(delay).toBeGreaterThan(200000);
|
||||||
|
expect(delay).toBeLessThan(400000);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should return different delays each time (due to randomization)", () => {
|
||||||
|
const delay1 = rateLimiter.getRandomDelay();
|
||||||
|
const delay2 = rateLimiter.getRandomDelay();
|
||||||
|
const delay3 = rateLimiter.getRandomDelay();
|
||||||
|
|
||||||
|
// Extremely unlikely to be identical due to randomization
|
||||||
|
expect(delay1).not.toBe(delay2);
|
||||||
|
expect(delay2).not.toBe(delay3);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("shouldPause", () => {
|
||||||
|
test("should not pause initially", () => {
|
||||||
|
expect(rateLimiter.shouldPause()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should pause after sending many emails in short time", () => {
|
||||||
|
// Simulate sending 20 emails in 1 hour (exceeds 15/hour limit)
|
||||||
|
rateLimiter.sentCount = 20;
|
||||||
|
rateLimiter.startTime = Date.now() - 60 * 60 * 1000; // 1 hour ago
|
||||||
|
|
||||||
|
expect(rateLimiter.shouldPause()).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should pause every 50 emails", () => {
|
||||||
|
rateLimiter.sentCount = 50;
|
||||||
|
expect(rateLimiter.shouldPause()).toBe(true);
|
||||||
|
|
||||||
|
rateLimiter.sentCount = 100;
|
||||||
|
expect(rateLimiter.shouldPause()).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should not pause if under rate limit", () => {
|
||||||
|
// Simulate sending 10 emails in 1 hour (under 15/hour limit)
|
||||||
|
rateLimiter.sentCount = 10;
|
||||||
|
rateLimiter.startTime = Date.now() - 60 * 60 * 1000; // 1 hour ago
|
||||||
|
|
||||||
|
expect(rateLimiter.shouldPause()).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getPauseDuration", () => {
|
||||||
|
test("should return pause duration around 30 minutes", () => {
|
||||||
|
const duration = rateLimiter.getPauseDuration();
|
||||||
|
|
||||||
|
// Base pause is 30 minutes = 1,800,000ms
|
||||||
|
// With randomization, expect 30-40 minutes
|
||||||
|
expect(duration).toBeGreaterThan(30 * 60 * 1000);
|
||||||
|
expect(duration).toBeLessThan(40 * 60 * 1000);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getNextSendDelay", () => {
|
||||||
|
test("should increment sent count", async () => {
|
||||||
|
const initialCount = rateLimiter.sentCount;
|
||||||
|
await rateLimiter.getNextSendDelay();
|
||||||
|
expect(rateLimiter.sentCount).toBe(initialCount + 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should return pause duration when should pause", async () => {
|
||||||
|
// Force a pause condition
|
||||||
|
rateLimiter.sentCount = 49; // Next increment will trigger pause at 50
|
||||||
|
|
||||||
|
const delay = await rateLimiter.getNextSendDelay();
|
||||||
|
|
||||||
|
// Should be a long pause, not normal delay
|
||||||
|
expect(delay).toBeGreaterThan(30 * 60 * 1000); // More than 30 minutes
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should return normal delay when not pausing", async () => {
|
||||||
|
rateLimiter.sentCount = 5; // Low count, won't trigger pause
|
||||||
|
|
||||||
|
const delay = await rateLimiter.getNextSendDelay();
|
||||||
|
|
||||||
|
// Should be normal delay (around 5 minutes with variance)
|
||||||
|
expect(delay).toBeGreaterThan(200000);
|
||||||
|
expect(delay).toBeLessThan(400000);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("formatDelay", () => {
|
||||||
|
test("should format milliseconds to readable time", () => {
|
||||||
|
expect(rateLimiter.formatDelay(60000)).toBe("1m 0s");
|
||||||
|
expect(rateLimiter.formatDelay(90000)).toBe("1m 30s");
|
||||||
|
expect(rateLimiter.formatDelay(125000)).toBe("2m 5s");
|
||||||
|
expect(rateLimiter.formatDelay(3661000)).toBe("61m 1s");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("getStats", () => {
|
||||||
|
test("should return correct statistics", () => {
|
||||||
|
rateLimiter.sentCount = 10;
|
||||||
|
rateLimiter.startTime = Date.now() - 60 * 60 * 1000; // 1 hour ago
|
||||||
|
|
||||||
|
const stats = rateLimiter.getStats();
|
||||||
|
|
||||||
|
expect(stats.sentCount).toBe(10);
|
||||||
|
expect(stats.runtime).toBe(60); // 60 minutes
|
||||||
|
expect(stats.averageRate).toBe("10.0"); // 10 emails per hour
|
||||||
|
expect(stats.nextDelay).toMatch(/^\d+m \d+s$/); // Format like "5m 23s"
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should handle zero runtime gracefully", () => {
|
||||||
|
rateLimiter.sentCount = 5;
|
||||||
|
rateLimiter.startTime = Date.now(); // Just started
|
||||||
|
|
||||||
|
const stats = rateLimiter.getStats();
|
||||||
|
|
||||||
|
expect(stats.sentCount).toBe(5);
|
||||||
|
expect(stats.runtime).toBe(0);
|
||||||
|
// Average rate should be very high for instant runtime
|
||||||
|
expect(parseFloat(stats.averageRate)).toBeGreaterThan(1000);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("reset", () => {
|
||||||
|
test("should reset all counters", () => {
|
||||||
|
rateLimiter.sentCount = 10;
|
||||||
|
rateLimiter.lastSentTime = Date.now() - 60000;
|
||||||
|
|
||||||
|
rateLimiter.reset();
|
||||||
|
|
||||||
|
expect(rateLimiter.sentCount).toBe(0);
|
||||||
|
expect(rateLimiter.lastSentTime).toBeNull();
|
||||||
|
expect(rateLimiter.startTime).toBeCloseTo(Date.now(), -1); // Within 10ms
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
148
tests/lib/templateEngine.test.js
Normal file
148
tests/lib/templateEngine.test.js
Normal file
@ -0,0 +1,148 @@
|
|||||||
|
const { describe, test, expect, beforeEach } = require("@jest/globals");
|
||||||
|
const fs = require("fs").promises;
|
||||||
|
const path = require("path");
|
||||||
|
|
||||||
|
// Mock the config
|
||||||
|
jest.mock("../../config", () => ({
|
||||||
|
email: { user: "test@example.com" },
|
||||||
|
gif: { enabled: false, url: "", alt: "" },
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Mock fs for template loading
|
||||||
|
jest.mock("fs", () => ({
|
||||||
|
promises: {
|
||||||
|
readFile: jest.fn(),
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
const templateEngine = require("../../lib/templateEngine");
|
||||||
|
|
||||||
|
describe("TemplateEngine", () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
jest.clearAllMocks();
|
||||||
|
// Clear template cache
|
||||||
|
templateEngine.templates = {};
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("formatFirmData", () => {
|
||||||
|
test("should format firm data correctly", () => {
|
||||||
|
const firmData = {
|
||||||
|
firmName: "Test Law Firm",
|
||||||
|
location: "Test City",
|
||||||
|
website: "https://test.com",
|
||||||
|
contactEmail: "test@testfirm.com",
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = templateEngine.formatFirmData(firmData);
|
||||||
|
|
||||||
|
expect(result).toEqual({
|
||||||
|
firmName: "Test Law Firm",
|
||||||
|
location: "Test City",
|
||||||
|
website: "https://test.com",
|
||||||
|
email: "test@testfirm.com",
|
||||||
|
greeting: "Legal Professional",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should handle missing data gracefully", () => {
|
||||||
|
const firmData = {
|
||||||
|
firmName: "Test Firm",
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = templateEngine.formatFirmData(firmData);
|
||||||
|
|
||||||
|
expect(result).toEqual({
|
||||||
|
firmName: "Test Firm",
|
||||||
|
location: undefined,
|
||||||
|
website: undefined,
|
||||||
|
email: undefined,
|
||||||
|
greeting: "Legal Professional",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should use name as greeting when available", () => {
|
||||||
|
const firmData = {
|
||||||
|
firmName: "Test Firm",
|
||||||
|
name: "John Doe",
|
||||||
|
contactEmail: "john@test.com",
|
||||||
|
};
|
||||||
|
|
||||||
|
const result = templateEngine.formatFirmData(firmData);
|
||||||
|
|
||||||
|
expect(result.greeting).toBe("John Doe");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("loadTemplate", () => {
|
||||||
|
test("should load and compile templates", async () => {
|
||||||
|
const mockHtmlContent = "<h1>{{title}}</h1>";
|
||||||
|
const mockTxtContent = "{{title}}";
|
||||||
|
|
||||||
|
fs.readFile
|
||||||
|
.mockResolvedValueOnce(mockHtmlContent)
|
||||||
|
.mockResolvedValueOnce(mockTxtContent);
|
||||||
|
|
||||||
|
const result = await templateEngine.loadTemplate("test");
|
||||||
|
|
||||||
|
expect(fs.readFile).toHaveBeenCalledTimes(2);
|
||||||
|
expect(result).toHaveProperty("html");
|
||||||
|
expect(result).toHaveProperty("text");
|
||||||
|
expect(typeof result.html).toBe("function");
|
||||||
|
expect(typeof result.text).toBe("function");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should cache loaded templates", async () => {
|
||||||
|
const mockHtmlContent = "<h1>{{title}}</h1>";
|
||||||
|
const mockTxtContent = "{{title}}";
|
||||||
|
|
||||||
|
fs.readFile
|
||||||
|
.mockResolvedValueOnce(mockHtmlContent)
|
||||||
|
.mockResolvedValueOnce(mockTxtContent);
|
||||||
|
|
||||||
|
// Load template twice
|
||||||
|
await templateEngine.loadTemplate("test");
|
||||||
|
await templateEngine.loadTemplate("test");
|
||||||
|
|
||||||
|
// Should only read files once due to caching
|
||||||
|
expect(fs.readFile).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should throw error when template loading fails", async () => {
|
||||||
|
fs.readFile.mockRejectedValue(new Error("File not found"));
|
||||||
|
|
||||||
|
await expect(templateEngine.loadTemplate("nonexistent")).rejects.toThrow(
|
||||||
|
"Failed to load template nonexistent"
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("render", () => {
|
||||||
|
test("should render template with data", async () => {
|
||||||
|
const mockHtmlContent = "<h1>Hello {{name}}</h1>";
|
||||||
|
const mockTxtContent = "Hello {{name}}";
|
||||||
|
|
||||||
|
fs.readFile
|
||||||
|
.mockResolvedValueOnce(mockHtmlContent)
|
||||||
|
.mockResolvedValueOnce(mockTxtContent);
|
||||||
|
|
||||||
|
const result = await templateEngine.render("test", { name: "World" });
|
||||||
|
|
||||||
|
expect(result.html).toContain("Hello World");
|
||||||
|
expect(result.text).toContain("Hello World");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("should include default sender data", async () => {
|
||||||
|
const mockHtmlContent = "From: {{senderName}}";
|
||||||
|
const mockTxtContent = "From: {{senderName}}";
|
||||||
|
|
||||||
|
fs.readFile
|
||||||
|
.mockResolvedValueOnce(mockHtmlContent)
|
||||||
|
.mockResolvedValueOnce(mockTxtContent);
|
||||||
|
|
||||||
|
const result = await templateEngine.render("test", {});
|
||||||
|
|
||||||
|
expect(result.html).toContain("John Smith");
|
||||||
|
expect(result.text).toContain("John Smith");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
41
tests/setup.js
Normal file
41
tests/setup.js
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
// Jest setup file for global test configuration
|
||||||
|
const path = require("path");
|
||||||
|
|
||||||
|
// Set test environment
|
||||||
|
process.env.NODE_ENV = "test";
|
||||||
|
process.env.EMAIL_TEST_MODE = "true";
|
||||||
|
process.env.EMAIL_USER = "test@example.com";
|
||||||
|
process.env.EMAIL_PASS = "test-password";
|
||||||
|
process.env.DELAY_MINUTES = "0"; // No delay in tests
|
||||||
|
process.env.LOG_LEVEL = "error"; // Reduce log noise in tests
|
||||||
|
|
||||||
|
// Mock external dependencies
|
||||||
|
jest.mock("nodemailer", () => ({
|
||||||
|
createTransporter: jest.fn(() => ({
|
||||||
|
sendMail: jest.fn(() => Promise.resolve({ messageId: "test-123" })),
|
||||||
|
})),
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Mock delay module to make tests faster
|
||||||
|
jest.mock("delay", () => jest.fn(() => Promise.resolve()));
|
||||||
|
|
||||||
|
// Console spy to reduce output noise in tests
|
||||||
|
global.consoleSpy = jest.spyOn(console, "log").mockImplementation(() => {});
|
||||||
|
global.consoleErrorSpy = jest
|
||||||
|
.spyOn(console, "error")
|
||||||
|
.mockImplementation(() => {});
|
||||||
|
global.consoleWarnSpy = jest
|
||||||
|
.spyOn(console, "warn")
|
||||||
|
.mockImplementation(() => {});
|
||||||
|
|
||||||
|
// Clean up after each test
|
||||||
|
afterEach(() => {
|
||||||
|
jest.clearAllMocks();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Clean up after all tests
|
||||||
|
afterAll(() => {
|
||||||
|
global.consoleSpy.mockRestore();
|
||||||
|
global.consoleErrorSpy.mockRestore();
|
||||||
|
global.consoleWarnSpy.mockRestore();
|
||||||
|
});
|
||||||
40
tests/test-campaigns.json
Normal file
40
tests/test-campaigns.json
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
{
|
||||||
|
"TestCampaigns": [
|
||||||
|
{
|
||||||
|
"firmName": "SaaS Test Firm",
|
||||||
|
"location": "Birmingham",
|
||||||
|
"website": "http://www.saastest.com",
|
||||||
|
"contactEmail": "test1@yourdomain.com",
|
||||||
|
"state": "Alabama",
|
||||||
|
"campaign": "campaign-1-saas",
|
||||||
|
"subject": "[TEST] LinkedIn Employment Intelligence API"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"firmName": "Data Service Test Firm",
|
||||||
|
"location": "Anchorage",
|
||||||
|
"website": "http://www.datatest.com",
|
||||||
|
"contactEmail": "test2@yourdomain.com",
|
||||||
|
"state": "Alaska",
|
||||||
|
"campaign": "campaign-2-data-service",
|
||||||
|
"subject": "[TEST] Monthly Employment Intelligence Reports"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"firmName": "License Test Firm",
|
||||||
|
"location": "Phoenix",
|
||||||
|
"website": "http://www.licensetest.com",
|
||||||
|
"contactEmail": "test3@yourdomain.com",
|
||||||
|
"state": "Arizona",
|
||||||
|
"campaign": "campaign-3-license",
|
||||||
|
"subject": "[TEST] Own Your LinkedIn Employment Parser"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"firmName": "GUI Test Firm",
|
||||||
|
"location": "Little Rock",
|
||||||
|
"website": "http://www.guitest.com",
|
||||||
|
"contactEmail": "test4@yourdomain.com",
|
||||||
|
"state": "Arkansas",
|
||||||
|
"campaign": "campaign-4-gui",
|
||||||
|
"subject": "[TEST] LinkedIn Employment Dashboard"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
Loading…
x
Reference in New Issue
Block a user