# Job Search Parser - Job Market Intelligence

Specialized parser for job market intelligence, tracking job postings, market trends, and competitive analysis. Focuses on tech roles and industry insights.

## 🎯 Purpose

The Job Search Parser is designed to:

- **Track Job Market Trends**: Monitor demand for specific roles and skills
- **Competitive Intelligence**: Analyze salary ranges and requirements
- **Industry Insights**: Track hiring patterns across different sectors
- **Skill Gap Analysis**: Identify in-demand technologies and frameworks
- **Market Demand Forecasting**: Predict job market trends

## 🚀 Features

### Core Functionality

- **Multi-Source Aggregation**: Collect job data from multiple platforms
- **Role-Specific Tracking**: Focus on tech roles and emerging positions
- **Skill Analysis**: Extract and categorize required skills
- **Salary Intelligence**: Track compensation ranges and trends
- **Company Intelligence**: Monitor hiring companies and patterns

### Advanced Features

- **Market Trend Analysis**: Identify growing and declining job categories
- **Geographic Distribution**: Track job distribution by location
- **Experience Level Analysis**: Entry, mid, senior level tracking
- **Remote Work Trends**: Monitor remote/hybrid work patterns
- **Technology Stack Tracking**: Framework and tool popularity

## 🌐 Supported Job Sites

### ✅ Implemented Parsers

#### SkipTheDrive Parser

Remote job board specializing in work-from-home positions.

**Features:**

- Keyword-based job search with relevance sorting
- Job type filtering (full-time, part-time, contract)
- Multi-page result parsing with pagination
- Featured/sponsored job identification
- AI-powered job relevance analysis
- Automatic duplicate detection

**Usage:**

```bash
# Parse SkipTheDrive for QA automation jobs
node index.js --sites=skipthedrive --keywords="automation qa,qa engineer"

# Filter by job type
JOB_TYPES="full time,contract" node index.js --sites=skipthedrive

# Run demo with limited results
node index.js --sites=skipthedrive --demo
```

#### LinkedIn Jobs Parser

Professional network job postings with comprehensive job data.

**Features:**

- LinkedIn authentication support
- Keyword-based job search
- Location filtering (both LinkedIn location and post-extraction filter)
- Multi-page result parsing with pagination
- Job type and experience level extraction
- Automatic duplicate detection
- Infinite scroll handling

**Requirements:**

- LinkedIn credentials (username and password) must be set in `.env` file:
  ```env
  LINKEDIN_USERNAME=******@gmail.com
  LINKEDIN_PASSWORD=***
  LINKEDIN_JOB_LOCATION=Canada  # Optional: LinkedIn location filter
  ```

**Usage:**

```bash
# Search LinkedIn jobs
node index.js --sites=linkedin --keywords="software engineer,developer"

# Search with location filter
node index.js --sites=linkedin --keywords="co-op" --location="Ontario"

# Search with date filter (jobs posted after specific date)
node index.js --sites=linkedin --keywords="co-op" --min-date="2025-12-01"

# Combine filters
node index.js --sites=linkedin --keywords="co-op" --location="Ontario" --min-date="2025-12-01"

# Combine multiple sites
node index.js --sites=linkedin,skipthedrive,indeed --keywords="intern,co-op"

# Use AND logic - jobs must match ALL keywords (e.g., "co-op" AND "summer 2026")
node index.js --sites=linkedin --keywords="co-op,summer 2026" --and

# Use grouped AND/OR logic - (co-op OR intern) AND (summer 2026)
# Use | (pipe) for OR within groups, , (comma) to separate AND groups
node index.js --sites=linkedin --keywords="co-op|intern,summer 2026" --and

# Multiple AND groups - (co-op OR intern) AND (summer 2026) AND (remote)
node index.js --sites=linkedin --keywords="co-op|intern,summer 2026,remote" --and
```

**Date Filter Notes:**
- The date filter uses LinkedIn's `f_TPR` parameter to filter at the LinkedIn level before parsing
- Format: `YYYY-MM-DD` (e.g., `2025-12-01`)
- LinkedIn supports relative timeframes up to ~30 days
- For dates older than 30 days, LinkedIn may limit results to the maximum supported timeframe

#### Indeed Parser

Comprehensive job aggregator with extensive job listings.

**Features:**

- Keyword-based job search
- Location filtering (both Indeed location and post-extraction filter)
- Multi-page result parsing with pagination
- Salary information extraction
- Date filtering (jobs posted within last 30 days)
- Automatic duplicate detection
- Job type and experience level support

**Usage:**

```bash
# Search Indeed jobs
node index.js --sites=indeed --keywords="software engineer,developer"

# Search with location filter
node index.js --sites=indeed --keywords="co-op" --location="Ontario"

# Search with date filter (jobs posted after specific date)
node index.js --sites=indeed --keywords="co-op" --min-date="2025-12-01"

# Combine filters
node index.js --sites=indeed --keywords="co-op" --location="Ontario" --min-date="2025-12-01"

# Combine multiple sites
node index.js --sites=indeed,linkedin --keywords="intern,co-op"

# Use AND logic - jobs must match ALL keywords
node index.js --sites=indeed --keywords="co-op,summer 2026" --and

# Use grouped AND/OR logic - (co-op OR intern) AND (summer 2026)
node index.js --sites=indeed --keywords="co-op|intern,summer 2026" --and
```

**Date Filter Notes:**
- The date filter converts to Indeed's `fromage` parameter (days ago)
- Format: `YYYY-MM-DD` (e.g., `2025-12-01`)
- Indeed supports up to 30 days for date filtering
- For dates older than 30 days, Indeed limits results to the maximum supported timeframe

**CAPTCHA/Verification Handling:**
- Indeed may show CAPTCHA or human verification pages when detecting automated access
- If you encounter CAPTCHA errors, try:
  1. Run in non-headless mode: Set `HEADLESS=false` in `.env` file (you can manually solve CAPTCHA)
  2. Wait a few minutes between runs to avoid rate limiting
  3. Use a different IP address or VPN if available
  4. Reduce the number of pages or keywords per run
- The parser will automatically detect and report CAPTCHA pages with helpful error messages

### 🚧 Planned Parsers
- **Glassdoor**: Jobs with company reviews and salary data
- **Monster**: Traditional job board
- **SimplyHired**: Job aggregator with salary estimates
- **AngelList**: Startup and tech jobs
- **Remote.co**: Dedicated remote work jobs
- **FlexJobs**: Flexible and remote positions

## 📦 Installation

```bash
# Install dependencies
npm install

# Run tests
npm test

# Run demo
node demo.js
```

## 🔧 Configuration

### Environment Variables

Create a `.env` file in the parser directory:

```env
# Job Search Configuration
SEARCH_KEYWORDS=software engineer,developer,programmer
# For grouped AND/OR logic, use pipe (|) for OR within groups and comma (,) for AND groups:
# SEARCH_KEYWORDS=co-op|intern,summer 2026,remote  # (co-op OR intern) AND (summer 2026) AND (remote)
USE_AND_LOGIC=false  # Set to "true" to enable AND logic (required for grouped keywords)
LOCATION_FILTER=Ontario,Canada
MAX_PAGES=5

# LinkedIn Configuration (required for LinkedIn jobs)
LINKEDIN_USERNAME=your_email@example.com
LINKEDIN_PASSWORD=your_password
LINKEDIN_JOB_LOCATION=Canada  # Optional: LinkedIn location search

# Date Filter (LinkedIn only - filters at LinkedIn level before parsing)
MIN_DATE=2025-12-01  # Format: YYYY-MM-DD (jobs posted after this date)

# Analysis Configuration
ENABLE_AI_ANALYSIS=false
HEADLESS=true

# Output Configuration
OUTPUT_FORMAT=json  # Options: "json", "csv", or "both"
```

**Keyword Examples in .env:**

```env
# Simple OR logic (default) - matches ANY keyword
SEARCH_KEYWORDS=co-op,intern
USE_AND_LOGIC=false

# Simple AND logic - matches ALL keywords
SEARCH_KEYWORDS=co-op,summer 2026
USE_AND_LOGIC=true

# Grouped AND/OR logic - (co-op OR intern) AND (summer 2026) AND (remote)
SEARCH_KEYWORDS=co-op|intern,summer 2026,remote
USE_AND_LOGIC=true
```

### Command Line Options

```bash
# Basic usage
node index.js

# Select sites to parse
node index.js --sites=linkedin,skipthedrive,indeed

# Search keywords
node index.js --keywords="software engineer,developer"

# Location filter
node index.js --location="Ontario"

# Max pages to parse
node index.js --max-pages=10

# Exclude rejected results
node index.js --no-rejected

# Output format (json, csv, or both)
node index.js --output=csv
node index.js --output=both

# Date filter (LinkedIn only - filters at LinkedIn level)
node index.js --sites=linkedin --min-date="2025-12-01"

# Use AND logic for keywords (all keywords must match)
node index.js --sites=linkedin --keywords="co-op,summer 2026" --and

# Use grouped AND/OR logic: (co-op OR intern) AND (summer 2026)
# Use | (pipe) for OR within groups, , (comma) to separate AND groups
node index.js --sites=linkedin --keywords="co-op|intern,summer 2026" --and

# Multiple AND groups: (co-op OR intern) AND (summer 2026) AND (remote)
node index.js --sites=linkedin --keywords="co-op|intern,summer 2026,remote" --and
```

**Available Options:**

- `--sites="site1,site2"`: Job sites to parse (linkedin, skipthedrive, indeed)
- `--keywords="keyword1,keyword2"`: Search keywords
  - Use `|` (pipe) to separate OR keywords within a group: `"co-op|intern"` means "co-op" OR "intern"
  - Use `,` (comma) to separate AND groups when using `--and`: `"co-op|intern,summer 2026"` means (co-op OR intern) AND (summer 2026)
- `--location="LOCATION"`: Location filter
- `--max-pages=NUMBER`: Maximum pages to parse (0 or "all" for unlimited)
- `--min-date="YYYY-MM-DD"`: Minimum posted date filter (LinkedIn only - filters at LinkedIn level before parsing)
- `--no-rejected` or `--exclude-rejected`: Exclude rejected results from output
- `--output=FORMAT` or `--format=FORMAT`: Output format - "json", "csv", or "both" (default: "json")
- `--and` or `--all-keywords`: Use AND logic for keywords (all keywords must match). Default is OR logic (any keyword matches)
  - When combined with `|` (pipe) in keywords, enables grouped AND/OR logic

## 📊 Keywords

### Role-Specific Keywords

Place keyword CSV files in the `keywords/` directory:

```
job-search-parser/
├── keywords/
│   ├── job-search-keywords.csv     # General job search terms
│   ├── tech-roles.csv              # Technology roles
│   ├── data-roles.csv              # Data science roles
│   ├── management-roles.csv        # Management positions
│   └── emerging-roles.csv          # Emerging job categories
└── index.js
```

### Tech Roles Keywords

```csv
keyword
software engineer
frontend developer
backend developer
full stack developer
data scientist
machine learning engineer
devops engineer
site reliability engineer
cloud architect
security engineer
mobile developer
iOS developer
Android developer
react developer
vue developer
angular developer
node.js developer
python developer
java developer
golang developer
rust developer
data engineer
analytics engineer
```

### Data Science Keywords

```csv
keyword
data scientist
machine learning engineer
data analyst
business analyst
data engineer
analytics engineer
ML engineer
AI engineer
statistician
quantitative analyst
research scientist
data architect
BI developer
ETL developer
```

## 📈 Usage Examples

### Basic Job Search

```bash
# Standard job market analysis
node index.js

# Specific tech roles
node index.js --roles="software engineer,data scientist"

# Geographic focus
node index.js --locations="Toronto,Vancouver,Calgary"
```

### Advanced Analysis

```bash
# Senior level positions
node index.js --experience="senior" --salary-min=100000

# Remote work opportunities
node index.js --remote="remote" --roles="frontend developer"

# Trend analysis
node index.js --trends --skills --output=results/trends.json
```

### Market Intelligence

```bash
# Salary analysis
node index.js --salary-min=80000 --salary-max=150000

# Skill gap analysis
node index.js --skills --roles="machine learning engineer"

# Competitive intelligence
node index.js --companies="Google,Microsoft,Amazon"
```

## 📊 Output Format

### JSON Structure

```json
{
  "metadata": {
    "timestamp": "2024-01-15T10:30:00Z",
    "search_parameters": {
      "roles": ["software engineer", "data scientist"],
      "locations": ["Toronto", "Vancouver"],
      "experience_levels": ["mid", "senior"],
      "remote_preference": ["remote", "hybrid"]
    },
    "total_jobs_found": 1250,
    "analysis_duration_seconds": 45
  },
  "market_overview": {
    "total_jobs": 1250,
    "average_salary": 95000,
    "salary_range": {
      "min": 65000,
      "max": 180000,
      "median": 92000
    },
    "remote_distribution": {
      "remote": 45,
      "hybrid": 35,
      "onsite": 20
    },
    "experience_distribution": {
      "entry": 15,
      "mid": 45,
      "senior": 40
    }
  },
  "trends": {
    "growing_skills": [
      { "skill": "React", "growth_rate": 25 },
      { "skill": "Python", "growth_rate": 18 },
      { "skill": "AWS", "growth_rate": 22 }
    ],
    "declining_skills": [
      { "skill": "jQuery", "growth_rate": -12 },
      { "skill": "PHP", "growth_rate": -8 }
    ],
    "emerging_roles": ["AI Engineer", "DevSecOps Engineer", "Data Engineer"]
  },
  "jobs": [
    {
      "id": "job_1",
      "title": "Senior Software Engineer",
      "company": "TechCorp",
      "location": "Toronto, Ontario",
      "remote_type": "hybrid",
      "salary": {
        "min": 100000,
        "max": 140000,
        "currency": "CAD"
      },
      "required_skills": ["React", "Node.js", "TypeScript", "AWS"],
      "preferred_skills": ["GraphQL", "Docker", "Kubernetes"],
      "experience_level": "senior",
      "job_url": "https://example.com/job/1",
      "posted_date": "2024-01-10T09:00:00Z",
      "scraped_at": "2024-01-15T10:30:00Z"
    }
  ],
  "analysis": {
    "skill_demand": {
      "React": { "count": 45, "avg_salary": 98000 },
      "Python": { "count": 38, "avg_salary": 102000 },
      "AWS": { "count": 32, "avg_salary": 105000 }
    },
    "company_insights": {
      "top_hirers": [
        { "company": "TechCorp", "jobs": 25 },
        { "company": "StartupXYZ", "jobs": 18 }
      ],
      "salary_leaders": [
        { "company": "BigTech", "avg_salary": 120000 },
        { "company": "FinTech", "avg_salary": 115000 }
      ]
    }
  }
}
```

### CSV Output

The parser can generate CSV files for easy spreadsheet analysis. Use `--output=csv` or `OUTPUT_FORMAT=csv` to export results as CSV.

**CSV Columns:**
- `jobId`: Unique job identifier
- `title`: Job title
- `company`: Company name
- `location`: Job location
- `jobUrl`: Link to job posting
- `postedDate`: Date job was posted
- `description`: Job description
- `jobType`: Type of job (full-time, part-time, contract, etc.)
- `experienceLevel`: Required experience level
- `keyword`: Search keyword that matched
- `extractedAt`: Timestamp when job was extracted
- `source`: Source site (e.g., "linkedin-jobs", "skipthedrive")
- `aiRelevant`: AI analysis relevance (Yes/No)
- `aiConfidence`: AI confidence score (0-1)
- `aiReasoning`: AI reasoning for relevance
- `aiContext`: AI analysis context
- `aiModel`: AI model used for analysis
- `aiAnalyzedAt`: Timestamp of AI analysis

**Example CSV Output:**

```csv
jobId,title,company,location,jobUrl,postedDate,description,jobType,experienceLevel,keyword,extractedAt,source,aiRelevant,aiConfidence,aiReasoning,aiContext,aiModel,aiAnalyzedAt
4344137241,Web Applications Co-op/Intern,Nokia,Kanata ON (Hybrid),https://www.linkedin.com/jobs/view/4344137241,,"Web Applications Co-op/Intern",,co-op,2025-12-17T04:50:05.600Z,linkedin-jobs,Yes,0.8,"The post mentions a co-op/intern position",co-op and internship opportunities for First year Math students,mistral,2025-12-17T04:58:33.479Z
```

**Usage:**

```bash
# Export as CSV only
node index.js --output=csv

# Export both JSON and CSV
node index.js --output=both

# Using environment variable
OUTPUT_FORMAT=csv node index.js
```

## 🔒 Security & Best Practices

### Data Privacy

- Respect job site terms of service
- Implement appropriate rate limiting
- Store data securely and responsibly
- Anonymize sensitive information

### Rate Limiting

- Implement delays between requests
- Respect API rate limits
- Use multiple data sources
- Monitor for blocking/detection

### Legal Compliance

- Educational and research purposes only
- Respect website terms of service
- Implement data retention policies
- Monitor for legal changes

## 🧪 Testing

### Run Tests

```bash
# All tests
npm test

# Specific test suites
npm test -- --testNamePattern="JobSearch"
npm test -- --testNamePattern="Analysis"
npm test -- --testNamePattern="Trends"
```

### Test Coverage

```bash
npm run test:coverage
```

## 🚀 Performance Optimization

### Recommended Settings

#### Fast Analysis

```bash
node index.js --roles="software engineer" --locations="Toronto"
```

#### Comprehensive Analysis

```bash
node index.js --trends --skills --experience="all"
```

#### Focused Intelligence

```bash
node index.js --salary-min=80000 --remote="remote" --trends
```

### Performance Tips

- Use specific role filters to reduce data volume
- Implement caching for repeated searches
- Use parallel processing for multiple sources
- Optimize data storage and retrieval

## 🔧 Troubleshooting

### Common Issues

#### Rate Limiting

```bash
# Reduce request frequency
export REQUEST_DELAY=2000
node index.js
```

#### Data Source Issues

```bash
# Use specific sources
node index.js --sources="linkedin,indeed"

# Check source availability
node index.js --test-sources
```

#### Output Issues

```bash
# Check output directory
mkdir -p results
node index.js --output=results/analysis.json

# Verify file permissions
chmod 755 results/
```

## 📈 Monitoring & Analytics

### Key Metrics

- **Job Volume**: Total jobs found per search
- **Salary Trends**: Average and median salary changes
- **Skill Demand**: Most requested skills
- **Remote Adoption**: Remote work trend analysis
- **Market Velocity**: Job posting frequency

### Dashboard Integration

- Real-time market monitoring
- Trend visualization
- Salary benchmarking
- Skill gap analysis
- Competitive intelligence

## 🤝 Contributing

### Development Setup

1. Fork the repository
2. Create feature branch
3. Add tests for new functionality
4. Ensure all tests pass
5. Submit pull request

### Code Standards

- Follow existing code style
- Add JSDoc comments
- Maintain test coverage
- Update documentation

## 📄 License

This parser is part of the LinkedOut platform and follows the same licensing terms.

---

**Note**: This tool is designed for educational and research purposes. Always respect website terms of service and implement appropriate rate limiting and ethical usage practices.