tanyar09
00c4cf1b6f
- Added functionality to export job search results in CSV format alongside JSON. - Introduced command line options for specifying output format: "json", "csv", or "both". - Updated README to include usage instructions for CSV output and detailed CSV structure. - Created utility functions for converting job results to CSV format, ensuring proper field escaping.
568 lines
13 KiB
Markdown
568 lines
13 KiB
Markdown
# Job Search Parser - Job Market Intelligence
|
|
|
|
Specialized parser for job market intelligence, tracking job postings, market trends, and competitive analysis. Focuses on tech roles and industry insights.
|
|
|
|
## 🎯 Purpose
|
|
|
|
The Job Search Parser is designed to:
|
|
|
|
- **Track Job Market Trends**: Monitor demand for specific roles and skills
|
|
- **Competitive Intelligence**: Analyze salary ranges and requirements
|
|
- **Industry Insights**: Track hiring patterns across different sectors
|
|
- **Skill Gap Analysis**: Identify in-demand technologies and frameworks
|
|
- **Market Demand Forecasting**: Predict job market trends
|
|
|
|
## 🚀 Features
|
|
|
|
### Core Functionality
|
|
|
|
- **Multi-Source Aggregation**: Collect job data from multiple platforms
|
|
- **Role-Specific Tracking**: Focus on tech roles and emerging positions
|
|
- **Skill Analysis**: Extract and categorize required skills
|
|
- **Salary Intelligence**: Track compensation ranges and trends
|
|
- **Company Intelligence**: Monitor hiring companies and patterns
|
|
|
|
### Advanced Features
|
|
|
|
- **Market Trend Analysis**: Identify growing and declining job categories
|
|
- **Geographic Distribution**: Track job distribution by location
|
|
- **Experience Level Analysis**: Entry, mid, senior level tracking
|
|
- **Remote Work Trends**: Monitor remote/hybrid work patterns
|
|
- **Technology Stack Tracking**: Framework and tool popularity
|
|
|
|
## 🌐 Supported Job Sites
|
|
|
|
### ✅ Implemented Parsers
|
|
|
|
#### SkipTheDrive Parser
|
|
|
|
Remote job board specializing in work-from-home positions.
|
|
|
|
**Features:**
|
|
|
|
- Keyword-based job search with relevance sorting
|
|
- Job type filtering (full-time, part-time, contract)
|
|
- Multi-page result parsing with pagination
|
|
- Featured/sponsored job identification
|
|
- AI-powered job relevance analysis
|
|
- Automatic duplicate detection
|
|
|
|
**Usage:**
|
|
|
|
```bash
|
|
# Parse SkipTheDrive for QA automation jobs
|
|
node index.js --sites=skipthedrive --keywords="automation qa,qa engineer"
|
|
|
|
# Filter by job type
|
|
JOB_TYPES="full time,contract" node index.js --sites=skipthedrive
|
|
|
|
# Run demo with limited results
|
|
node index.js --sites=skipthedrive --demo
|
|
```
|
|
|
|
#### LinkedIn Jobs Parser
|
|
|
|
Professional network job postings with comprehensive job data.
|
|
|
|
**Features:**
|
|
|
|
- LinkedIn authentication support
|
|
- Keyword-based job search
|
|
- Location filtering (both LinkedIn location and post-extraction filter)
|
|
- Multi-page result parsing with pagination
|
|
- Job type and experience level extraction
|
|
- Automatic duplicate detection
|
|
- Infinite scroll handling
|
|
|
|
**Requirements:**
|
|
|
|
- LinkedIn credentials (username and password) must be set in `.env` file:
|
|
```env
|
|
LINKEDIN_USERNAME=******@gmail.com
|
|
LINKEDIN_PASSWORD=***
|
|
LINKEDIN_JOB_LOCATION=Canada # Optional: LinkedIn location filter
|
|
```
|
|
|
|
**Usage:**
|
|
|
|
```bash
|
|
# Search LinkedIn jobs
|
|
node index.js --sites=linkedin --keywords="software engineer,developer"
|
|
|
|
# Search with location filter
|
|
node index.js --sites=linkedin --keywords="co-op" --location="Ontario"
|
|
|
|
# Combine multiple sites
|
|
node index.js --sites=linkedin,skipthedrive --keywords="intern,co-op"
|
|
```
|
|
|
|
### 🚧 Planned Parsers
|
|
|
|
- **Indeed**: Comprehensive job aggregator
|
|
- **Glassdoor**: Jobs with company reviews and salary data
|
|
- **Monster**: Traditional job board
|
|
- **SimplyHired**: Job aggregator with salary estimates
|
|
- **AngelList**: Startup and tech jobs
|
|
- **Remote.co**: Dedicated remote work jobs
|
|
- **FlexJobs**: Flexible and remote positions
|
|
|
|
## 📦 Installation
|
|
|
|
```bash
|
|
# Install dependencies
|
|
npm install
|
|
|
|
# Run tests
|
|
npm test
|
|
|
|
# Run demo
|
|
node demo.js
|
|
```
|
|
|
|
## 🔧 Configuration
|
|
|
|
### Environment Variables
|
|
|
|
Create a `.env` file in the parser directory:
|
|
|
|
```env
|
|
# Job Search Configuration
|
|
SEARCH_KEYWORDS=software engineer,developer,programmer
|
|
LOCATION_FILTER=Ontario,Canada
|
|
MAX_PAGES=5
|
|
|
|
# LinkedIn Configuration (required for LinkedIn jobs)
|
|
LINKEDIN_USERNAME=your_email@example.com
|
|
LINKEDIN_PASSWORD=your_password
|
|
LINKEDIN_JOB_LOCATION=Canada # Optional: LinkedIn location search
|
|
|
|
# Analysis Configuration
|
|
ENABLE_AI_ANALYSIS=false
|
|
HEADLESS=true
|
|
|
|
# Output Configuration
|
|
OUTPUT_FORMAT=json # Options: "json", "csv", or "both"
|
|
```
|
|
|
|
### Command Line Options
|
|
|
|
```bash
|
|
# Basic usage
|
|
node index.js
|
|
|
|
# Select sites to parse
|
|
node index.js --sites=linkedin,skipthedrive
|
|
|
|
# Search keywords
|
|
node index.js --keywords="software engineer,developer"
|
|
|
|
# Location filter
|
|
node index.js --location="Ontario"
|
|
|
|
# Max pages to parse
|
|
node index.js --max-pages=10
|
|
|
|
# Exclude rejected results
|
|
node index.js --no-rejected
|
|
|
|
# Output format (json, csv, or both)
|
|
node index.js --output=csv
|
|
node index.js --output=both
|
|
```
|
|
|
|
**Available Options:**
|
|
|
|
- `--sites="site1,site2"`: Job sites to parse (linkedin, skipthedrive)
|
|
- `--keywords="keyword1,keyword2"`: Search keywords
|
|
- `--location="LOCATION"`: Location filter
|
|
- `--max-pages=NUMBER`: Maximum pages to parse (0 or "all" for unlimited)
|
|
- `--no-rejected` or `--exclude-rejected`: Exclude rejected results from output
|
|
- `--output=FORMAT` or `--format=FORMAT`: Output format - "json", "csv", or "both" (default: "json")
|
|
|
|
## 📊 Keywords
|
|
|
|
### Role-Specific Keywords
|
|
|
|
Place keyword CSV files in the `keywords/` directory:
|
|
|
|
```
|
|
job-search-parser/
|
|
├── keywords/
|
|
│ ├── job-search-keywords.csv # General job search terms
|
|
│ ├── tech-roles.csv # Technology roles
|
|
│ ├── data-roles.csv # Data science roles
|
|
│ ├── management-roles.csv # Management positions
|
|
│ └── emerging-roles.csv # Emerging job categories
|
|
└── index.js
|
|
```
|
|
|
|
### Tech Roles Keywords
|
|
|
|
```csv
|
|
keyword
|
|
software engineer
|
|
frontend developer
|
|
backend developer
|
|
full stack developer
|
|
data scientist
|
|
machine learning engineer
|
|
devops engineer
|
|
site reliability engineer
|
|
cloud architect
|
|
security engineer
|
|
mobile developer
|
|
iOS developer
|
|
Android developer
|
|
react developer
|
|
vue developer
|
|
angular developer
|
|
node.js developer
|
|
python developer
|
|
java developer
|
|
golang developer
|
|
rust developer
|
|
data engineer
|
|
analytics engineer
|
|
```
|
|
|
|
### Data Science Keywords
|
|
|
|
```csv
|
|
keyword
|
|
data scientist
|
|
machine learning engineer
|
|
data analyst
|
|
business analyst
|
|
data engineer
|
|
analytics engineer
|
|
ML engineer
|
|
AI engineer
|
|
statistician
|
|
quantitative analyst
|
|
research scientist
|
|
data architect
|
|
BI developer
|
|
ETL developer
|
|
```
|
|
|
|
## 📈 Usage Examples
|
|
|
|
### Basic Job Search
|
|
|
|
```bash
|
|
# Standard job market analysis
|
|
node index.js
|
|
|
|
# Specific tech roles
|
|
node index.js --roles="software engineer,data scientist"
|
|
|
|
# Geographic focus
|
|
node index.js --locations="Toronto,Vancouver,Calgary"
|
|
```
|
|
|
|
### Advanced Analysis
|
|
|
|
```bash
|
|
# Senior level positions
|
|
node index.js --experience="senior" --salary-min=100000
|
|
|
|
# Remote work opportunities
|
|
node index.js --remote="remote" --roles="frontend developer"
|
|
|
|
# Trend analysis
|
|
node index.js --trends --skills --output=results/trends.json
|
|
```
|
|
|
|
### Market Intelligence
|
|
|
|
```bash
|
|
# Salary analysis
|
|
node index.js --salary-min=80000 --salary-max=150000
|
|
|
|
# Skill gap analysis
|
|
node index.js --skills --roles="machine learning engineer"
|
|
|
|
# Competitive intelligence
|
|
node index.js --companies="Google,Microsoft,Amazon"
|
|
```
|
|
|
|
## 📊 Output Format
|
|
|
|
### JSON Structure
|
|
|
|
```json
|
|
{
|
|
"metadata": {
|
|
"timestamp": "2024-01-15T10:30:00Z",
|
|
"search_parameters": {
|
|
"roles": ["software engineer", "data scientist"],
|
|
"locations": ["Toronto", "Vancouver"],
|
|
"experience_levels": ["mid", "senior"],
|
|
"remote_preference": ["remote", "hybrid"]
|
|
},
|
|
"total_jobs_found": 1250,
|
|
"analysis_duration_seconds": 45
|
|
},
|
|
"market_overview": {
|
|
"total_jobs": 1250,
|
|
"average_salary": 95000,
|
|
"salary_range": {
|
|
"min": 65000,
|
|
"max": 180000,
|
|
"median": 92000
|
|
},
|
|
"remote_distribution": {
|
|
"remote": 45,
|
|
"hybrid": 35,
|
|
"onsite": 20
|
|
},
|
|
"experience_distribution": {
|
|
"entry": 15,
|
|
"mid": 45,
|
|
"senior": 40
|
|
}
|
|
},
|
|
"trends": {
|
|
"growing_skills": [
|
|
{ "skill": "React", "growth_rate": 25 },
|
|
{ "skill": "Python", "growth_rate": 18 },
|
|
{ "skill": "AWS", "growth_rate": 22 }
|
|
],
|
|
"declining_skills": [
|
|
{ "skill": "jQuery", "growth_rate": -12 },
|
|
{ "skill": "PHP", "growth_rate": -8 }
|
|
],
|
|
"emerging_roles": ["AI Engineer", "DevSecOps Engineer", "Data Engineer"]
|
|
},
|
|
"jobs": [
|
|
{
|
|
"id": "job_1",
|
|
"title": "Senior Software Engineer",
|
|
"company": "TechCorp",
|
|
"location": "Toronto, Ontario",
|
|
"remote_type": "hybrid",
|
|
"salary": {
|
|
"min": 100000,
|
|
"max": 140000,
|
|
"currency": "CAD"
|
|
},
|
|
"required_skills": ["React", "Node.js", "TypeScript", "AWS"],
|
|
"preferred_skills": ["GraphQL", "Docker", "Kubernetes"],
|
|
"experience_level": "senior",
|
|
"job_url": "https://example.com/job/1",
|
|
"posted_date": "2024-01-10T09:00:00Z",
|
|
"scraped_at": "2024-01-15T10:30:00Z"
|
|
}
|
|
],
|
|
"analysis": {
|
|
"skill_demand": {
|
|
"React": { "count": 45, "avg_salary": 98000 },
|
|
"Python": { "count": 38, "avg_salary": 102000 },
|
|
"AWS": { "count": 32, "avg_salary": 105000 }
|
|
},
|
|
"company_insights": {
|
|
"top_hirers": [
|
|
{ "company": "TechCorp", "jobs": 25 },
|
|
{ "company": "StartupXYZ", "jobs": 18 }
|
|
],
|
|
"salary_leaders": [
|
|
{ "company": "BigTech", "avg_salary": 120000 },
|
|
{ "company": "FinTech", "avg_salary": 115000 }
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### CSV Output
|
|
|
|
The parser can generate CSV files for easy spreadsheet analysis. Use `--output=csv` or `OUTPUT_FORMAT=csv` to export results as CSV.
|
|
|
|
**CSV Columns:**
|
|
- `jobId`: Unique job identifier
|
|
- `title`: Job title
|
|
- `company`: Company name
|
|
- `location`: Job location
|
|
- `jobUrl`: Link to job posting
|
|
- `postedDate`: Date job was posted
|
|
- `description`: Job description
|
|
- `jobType`: Type of job (full-time, part-time, contract, etc.)
|
|
- `experienceLevel`: Required experience level
|
|
- `keyword`: Search keyword that matched
|
|
- `extractedAt`: Timestamp when job was extracted
|
|
- `source`: Source site (e.g., "linkedin-jobs", "skipthedrive")
|
|
- `aiRelevant`: AI analysis relevance (Yes/No)
|
|
- `aiConfidence`: AI confidence score (0-1)
|
|
- `aiReasoning`: AI reasoning for relevance
|
|
- `aiContext`: AI analysis context
|
|
- `aiModel`: AI model used for analysis
|
|
- `aiAnalyzedAt`: Timestamp of AI analysis
|
|
|
|
**Example CSV Output:**
|
|
|
|
```csv
|
|
jobId,title,company,location,jobUrl,postedDate,description,jobType,experienceLevel,keyword,extractedAt,source,aiRelevant,aiConfidence,aiReasoning,aiContext,aiModel,aiAnalyzedAt
|
|
4344137241,Web Applications Co-op/Intern,Nokia,Kanata ON (Hybrid),https://www.linkedin.com/jobs/view/4344137241,,"Web Applications Co-op/Intern",,co-op,2025-12-17T04:50:05.600Z,linkedin-jobs,Yes,0.8,"The post mentions a co-op/intern position",co-op and internship opportunities for First year Math students,mistral,2025-12-17T04:58:33.479Z
|
|
```
|
|
|
|
**Usage:**
|
|
|
|
```bash
|
|
# Export as CSV only
|
|
node index.js --output=csv
|
|
|
|
# Export both JSON and CSV
|
|
node index.js --output=both
|
|
|
|
# Using environment variable
|
|
OUTPUT_FORMAT=csv node index.js
|
|
```
|
|
|
|
## 🔒 Security & Best Practices
|
|
|
|
### Data Privacy
|
|
|
|
- Respect job site terms of service
|
|
- Implement appropriate rate limiting
|
|
- Store data securely and responsibly
|
|
- Anonymize sensitive information
|
|
|
|
### Rate Limiting
|
|
|
|
- Implement delays between requests
|
|
- Respect API rate limits
|
|
- Use multiple data sources
|
|
- Monitor for blocking/detection
|
|
|
|
### Legal Compliance
|
|
|
|
- Educational and research purposes only
|
|
- Respect website terms of service
|
|
- Implement data retention policies
|
|
- Monitor for legal changes
|
|
|
|
## 🧪 Testing
|
|
|
|
### Run Tests
|
|
|
|
```bash
|
|
# All tests
|
|
npm test
|
|
|
|
# Specific test suites
|
|
npm test -- --testNamePattern="JobSearch"
|
|
npm test -- --testNamePattern="Analysis"
|
|
npm test -- --testNamePattern="Trends"
|
|
```
|
|
|
|
### Test Coverage
|
|
|
|
```bash
|
|
npm run test:coverage
|
|
```
|
|
|
|
## 🚀 Performance Optimization
|
|
|
|
### Recommended Settings
|
|
|
|
#### Fast Analysis
|
|
|
|
```bash
|
|
node index.js --roles="software engineer" --locations="Toronto"
|
|
```
|
|
|
|
#### Comprehensive Analysis
|
|
|
|
```bash
|
|
node index.js --trends --skills --experience="all"
|
|
```
|
|
|
|
#### Focused Intelligence
|
|
|
|
```bash
|
|
node index.js --salary-min=80000 --remote="remote" --trends
|
|
```
|
|
|
|
### Performance Tips
|
|
|
|
- Use specific role filters to reduce data volume
|
|
- Implement caching for repeated searches
|
|
- Use parallel processing for multiple sources
|
|
- Optimize data storage and retrieval
|
|
|
|
## 🔧 Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### Rate Limiting
|
|
|
|
```bash
|
|
# Reduce request frequency
|
|
export REQUEST_DELAY=2000
|
|
node index.js
|
|
```
|
|
|
|
#### Data Source Issues
|
|
|
|
```bash
|
|
# Use specific sources
|
|
node index.js --sources="linkedin,indeed"
|
|
|
|
# Check source availability
|
|
node index.js --test-sources
|
|
```
|
|
|
|
#### Output Issues
|
|
|
|
```bash
|
|
# Check output directory
|
|
mkdir -p results
|
|
node index.js --output=results/analysis.json
|
|
|
|
# Verify file permissions
|
|
chmod 755 results/
|
|
```
|
|
|
|
## 📈 Monitoring & Analytics
|
|
|
|
### Key Metrics
|
|
|
|
- **Job Volume**: Total jobs found per search
|
|
- **Salary Trends**: Average and median salary changes
|
|
- **Skill Demand**: Most requested skills
|
|
- **Remote Adoption**: Remote work trend analysis
|
|
- **Market Velocity**: Job posting frequency
|
|
|
|
### Dashboard Integration
|
|
|
|
- Real-time market monitoring
|
|
- Trend visualization
|
|
- Salary benchmarking
|
|
- Skill gap analysis
|
|
- Competitive intelligence
|
|
|
|
## 🤝 Contributing
|
|
|
|
### Development Setup
|
|
|
|
1. Fork the repository
|
|
2. Create feature branch
|
|
3. Add tests for new functionality
|
|
4. Ensure all tests pass
|
|
5. Submit pull request
|
|
|
|
### Code Standards
|
|
|
|
- Follow existing code style
|
|
- Add JSDoc comments
|
|
- Maintain test coverage
|
|
- Update documentation
|
|
|
|
## 📄 License
|
|
|
|
This parser is part of the LinkedOut platform and follows the same licensing terms.
|
|
|
|
---
|
|
|
|
**Note**: This tool is designed for educational and research purposes. Always respect website terms of service and implement appropriate rate limiting and ethical usage practices.
|