hilitehero/README.md

# PDF Highlight Extractor

A Python tool for extracting highlighted text from PDF files with precise text ordering and intelligent hyphenation handling.

## Features

- **4-Color Support**: Extracts Yellow, Pink, Green, and Blue highlights
- **Smart Text Ordering**: Fixes PDF text extraction order issues using multiple methods
- **Hyphenation Merging**: Automatically combines hyphenated words across lines ("lin-" + "guistics" → "linguistics")
- **Precise Boundaries**: Configurable overlap detection to avoid over-extraction
- **Multiple Extraction Methods**: Fallback system for maximum compatibility
- **Cross-page Support**: Handles highlights that span multiple pages
- **Test Mode**: Quick testing with default settings
- **Export Options**: JSON and CSV output formats

## Installation

Clone the repository:
git clone <repository-url>
cd pdf-highlight-extractor

Install required packages:
pip install PyMuPDF pdfplumber colorama pandas


## Dependencies

- PyMuPDF (fitz) - PDF processing and text extraction
- pdfplumber - Additional PDF annotation support
- colorama - Colored terminal output
- pandas - CSV export functionality

## Usage

### Quick Test Mode
python highlight_extractor.py --test

Uses default file: `/mnt/c/Users/admin/Downloads/test2.pdf` and displays results only.

### Interactive Mode
python highlight_extractor.py

Prompts for PDF file path and output options.

### Command Line Flags
- `--test`, `-t`, or `test` - Enable test mode with defaults
- No flags - Full interactive mode

## Output Formats

### Terminal Display
📄 Page 35
🎨 YELLOW
"We end with some specific suggestions for what we can do as linguists"
🎨 PINK (hyphen-merged)
"linguistics itself"

### JSON Export
{
    "highlights": [
        {
            "page": 35,
            "text": "We end with some specific suggestions",
            "color": "yellow",
            "type": "highlight"
        }
    ]
}

### CSV Export
Tabular format with columns: page, text, color, type, category

## Technical Features

### Text Ordering Algorithm
1. **Method A**: PyMuPDF built-in sorting
2. **Method B**: Text block extraction with geometric sorting
3. **Method C**: Enhanced word-level sorting with line detection

### Hyphenation Detection
- Same-page: Detects hyphens within 8-30 pixel line spacing
- Cross-page: Handles hyphenation across page boundaries
- Smart merging: Only merges clear hyphenation patterns

### Precision Control
- **Overlap Threshold**: 40% word overlap required for inclusion
- **Boundary Expansion**: +2 pixel expansion for edge words
- **Line Tolerance**: 5-pixel tolerance for same-line detection

## Troubleshooting

### Common Issues

**Text Order Problems**: The tool uses multiple methods to fix PDF text ordering issues. If text still appears scrambled, the PDF may have complex layout encoding.

**Missing Words**: Lower the overlap threshold or check if highlights are too light/transparent.

**Over-extraction**: The tool is designed to avoid this, but very close text might be included. Check highlight precision in your PDF.

### Debug Output
Run with detailed logging to see extraction decisions:
python highlight_extractor.py --test

## Contributing

1. Create a feature branch from main
2. Make your changes
3. Test with sample PDFs
4. Submit a pull request

## License

MIT License

## Support

For issues or questions, please open a GitHub issue.

# PDF Highlight Extraction Process - Step by Step

## Phase 1: Initialization and Setup
1. **Script Startup**: Check command line arguments for test mode
2. **Path Resolution**: Determine PDF file path (default or user input)
3. **File Validation**: Verify PDF file exists and is accessible
4. **Object Creation**: Initialize PDFHighlightExtractor with file path

## Phase 2: PDF Analysis and Loading
1. **Document Opening**: Load PDF using PyMuPDF (fitz) library
2. **Page Iteration**: Loop through each page in the document
3. **Annotation Discovery**: Find all annotations on each page
4. **Type Filtering**: Identify highlight-type annotations specifically

## Phase 3: Color Classification
1. **Color Extraction**: Get RGB values from annotation properties
2. **Color Normalization**: Convert to 0-255 range if needed
3. **Color Mapping**: Classify into 4 categories (Yellow, Pink, Green, Blue)
4. **Unknown Filtering**: Skip annotations with unrecognized colors

## Phase 4: Text Extraction (Multi-Method Approach)

### Method A: Built-in Sorting
1. **Rectangle Expansion**: Add 2-pixel buffer around highlight area
2. **PyMuPDF Extraction**: Use page.get_text("text", sort=True)
3. **Text Cleaning**: Remove extra whitespace and normalize
4. **Success Check**: Return if valid text found

### Method B: Text Block Extraction  
1. **Block Discovery**: Get text blocks from highlight area
2. **Geometric Sorting**: Sort blocks by Y-position, then X-position
3. **Block Combination**: Join block texts with spaces
4. **Quality Check**: Verify result makes sense

### Method C: Enhanced Word Sorting
1. **Word Collection**: Get all words intersecting highlight area
2. **Overlap Calculation**: Calculate intersection ratio for each word
3. **Threshold Filtering**: Include words with 40%+ overlap
4. **Line Detection**: Group words by Y-position (5-pixel tolerance)
5. **Line Sorting**: Sort lines top-to-bottom
6. **Word Sorting**: Sort words left-to-right within each line
7. **Text Assembly**: Combine words in proper reading order

## Phase 5: Hyphenation Detection and Merging
1. **Pattern Recognition**: Look for highlights ending with '-'
2. **Proximity Check**: Verify next highlight is same color and nearby
3. **Distance Validation**: Check reasonable line spacing (8-30 pixels)
4. **Page Handling**: Support both same-page and cross-page hyphenation
5. **Text Joining**: Remove hyphen and combine words seamlessly

## Phase 6: Data Organization
1. **Highlight Storage**: Create structured data objects for each highlight
2. **Sorting**: Order by page number, then Y-position, then X-position
3. **Merging**: Apply hyphenation merging where detected
4. **Categorization**: Separate annotations from background highlights

## Phase 7: Output Generation

### Terminal Display
1. **Page Grouping**: Organize results by page number
2. **Color Coding**: Apply terminal colors for visual distinction
3. **Status Indicators**: Show merge status (hyphen-merged, cross-page)
4. **Formatting**: Clean, readable text presentation

### File Export (Optional)
1. **JSON Generation**: Structure data with metadata
2. **CSV Creation**: Tabular format for analysis
3. **File Writing**: Save to specified output paths

## Phase 8: Cleanup and Reporting
1. **Resource Cleanup**: Close PDF document properly
2. **Statistics**: Report extraction counts and timing
3. **Status Messages**: Provide user feedback on results
4. **Memory Management**: Clean up temporary objects

## Error Handling Throughout
- **Try-Catch Blocks**: Graceful handling of PDF parsing errors
- **Fallback Methods**: Alternative extraction approaches
- **Validation Checks**: Verify data integrity at each step
- **User Feedback**: Clear error messages and debugging info

## Debug Information
- **Overlap Ratios**: Show word inclusion/exclusion decisions
- **Method Success**: Indicate which extraction method worked
- **Hyphenation Detection**: Log when word merging occurs
- **Performance Timing**: Track processing duration