PHIN Practitioner Data Scraper & Procedure Code Matcher

A comprehensive Python-based data collection system for the PHIN (Private Healthcare Information Network) that scrapes practitioner profiles, matches them with procedure codes, and provides clean, structured data exports. This project evolved from a simple scraper into a complete data pipeline for healthcare practitioner analysis.

🚀 New: Annual Refresh Workflow

The scraper now supports a two-phase approach that separates cache updating from data export:

1. Cache Update Phase (Network Operations): Refresh indexes and stale profiles
2. Export Phase (Offline Operations): Generate CSV/JSONL from cached data

This workflow is perfect for annual data updates and ensures your cache is always the source of truth.

✨ Key Features

• 📊 Complete Data Collection: 13,487+ practitioner profiles with contact details, specialties, and patient feedback
• 🔗 Procedure Code Matching: 779 procedures with 104,304+ consultant-procedure associations
• 🧹 Data Cleaning: Automated removal of empty procedure fields while preserving all other data
• 💾 Smart Caching: Cache-first design with ETag/Last-Modified tracking
• 📁 Multiple Output Formats: CSV, JSONL with both raw and cleaned versions
• 🕷️ Respectful Scraping: Rate limiting, exponential backoff, and stealth techniques
• 📈 Progress Tracking: Real-time updates and comprehensive logging
• 🔄 Resume Capability: Can resume interrupted operations from cache

🏗️ Project Structure

PHIN Practitioner Data - Procedure Code/
├── phin_production_scraper.py          # Main production scraper
├── append_procedures_to_profiles.py    # Procedure code matching script
├── run_production_scraper.py           # Production workflow runner
├── requirements.txt                    # Python dependencies
├── README.md                          # This documentation
├── cache_phin/                        # 🎯 SOURCE OF TRUTH (never delete)
│   ├── index/                        # Practitioner index files
│   └── profiles/                     # Individual consultant profiles (*.json)
├── production_output/                 # Final data exports
│   ├── consultants_with_procedures.csv           # Complete dataset (115K+ rows)
│   ├── consultants_with_procedures.jsonl         # Complete dataset (13K+ records)
│   ├── consultants_with_procedures_cleaned.csv   # Cleaned version (no procedure NaN)
│   ├── consultants_with_procedures_cleaned.jsonl # Cleaned version (no procedure NaN)
│   └── procedure_results_complete.json          # Procedure data (779 procedures, 23.8MB)
├── test_output*/                      # Various test outputs and iterations
└── PHIN Procedure Group IDs - Procedure groups (1).csv  # Procedure reference data

📊 Data Overview

Verified Data Statistics

• Total Practitioners: 13,487 records
• Total Procedures: 779 procedures
• Consultant-Procedure Associations: 104,304+ associations
• Coverage: 97.6% of procedures have associated consultants
• Average Consultants per Procedure: 133.9 consultants

Top 10 Most Common Procedures

1. Skin lesion removal - 1,200 consultants
2. Removal of metalwork from bone - 1,200 consultants
3. Joint injections for pain - 1,200 consultants
4. Knee arthroscopy - 1,170 consultants
5. Inguinal hernia repair - 1,160 consultants
6. Upper GI endoscopy - diagnostic - 1,159 consultants
7. Colonoscopy - diagnostic - 1,145 consultants
8. Knee replacement (primary) - 1,137 consultants
9. Sigmoidoscopy - diagnostic - 1,068 consultants
10. Umbilical hernia repair - 1,054 consultants

Data Quality

• Complete Coverage: All active PHIN practitioners included
• Clean Structure: Consistent schema across all records
• Validated Data: GMC numbers, contact details, and procedure associations verified
• No Duplicates: Deduplicated practitioner and procedure data

📦 Installation

1. Clone the repository:

git clone <repository-url>
cd PHIN-Practitioner-Data-Procedure-Code

2. Install dependencies:

pip install -r requirements.txt

🎯 Usage

Complete Data Collection Workflow

1. Initial Setup:

pip install -r requirements.txt

2. Run Complete Data Collection:

python run_production_scraper.py

This will:

• Scrape procedure search results from PHIN (using HTML parsing + SAPPER extraction)
• Collect procedure code data and admission counts
• Match procedure data with existing practitioner profiles
• Generate raw CSV and JSONL exports (with NaN values for unmatched procedures)
• Create batch results and complete procedure datasets

Note: This generates raw data with NaN values. Use the cleaned versions in production_output/ for analysis.

3. Data Cleaning (Optional): The raw exports contain NaN values for unmatched procedures. Use the cleaning script:

python clean_procedure_nan.py

This creates cleaned versions:

• consultants_with_procedures_cleaned.csv - NaN values removed for procedure fields only
• consultants_with_procedures_cleaned.jsonl - Same cleaning applied to JSONL format

Note: The cleaning script removes only procedure-related NaN values (fields like procedure_132_id, procedure_160_name, etc.) while preserving all other data including non-procedure NaN values.

Individual Components

Scrape Practitioner Profiles:

python phin_production_scraper.py

Match Procedure Codes:

python append_procedures_to_profiles.py

Export Cached Data:

python phin_production_scraper.py --export-cache-only

📁 Output Files

Main Datasets

1. consultants_with_procedures.csv (115K+ rows)

   • Complete dataset with all practitioner and procedure data
   • Includes procedure code columns with admission counts
   • Contains NaN values for unmatched procedures

2. consultants_with_procedures.jsonl (13K+ records)

   • Same data in JSON Lines format
   • One practitioner per line
   • Includes complete raw profile data

3. consultants_with_procedures_cleaned.csv (115K+ rows)

   • Cleaned version with procedure NaN values removed
   • Maintains all other data integrity
   • Optimized for analysis without empty procedure fields

4. consultants_with_procedures_cleaned.jsonl (13K+ records)

   • Cleaned JSONL version
   • Only includes procedure fields with actual data
   • Preserves all practitioner information

5. procedure_results_complete.json (23.8MB, 775K+ lines)

   • Complete procedure data with consultant associations
   • 779 procedures with 104,304+ consultant-procedure associations
   • Detailed admission statistics and practitioner data

Data Structure

CSV Columns Include:

• Basic info: id, name, gmcNumber, specialty
• Contact: email, phone, remoteConsultations
• Feedback: wouldRecommend_pct, patient experience data
• Location: Hospital details, regions, coordinates
• Procedures: procedure_[ID]_id, procedure_[ID]_name, procedure_[ID]_admissions
• Raw data: Complete JSON profile as raw_data

JSONL Structure:

• Normalized practitioner data
• Procedure associations with admission counts
• Complete profile metadata
• Consistent schema across all records

🔧 Advanced Usage

Custom Data Collection

Scrape Specific Letter Ranges:

python phin_production_scraper.py --letters A-C --max-profiles 100

Update Stale Data Only:

python phin_production_scraper.py --stale-days 30

Force Refresh All Data:

python phin_production_scraper.py --refresh

Data Processing

Clean Procedure Data:

# The cleaning is built into the workflow, but you can run:
python -c "
import pandas as pd
df = pd.read_csv('production_output/consultants_with_procedures.csv')
# Remove procedure columns that are all NaN
procedure_cols = [col for col in df.columns if col.startswith('procedure_')]
df_clean = df.dropna(subset=procedure_cols, how='all')
df_clean.to_csv('production_output/consultants_cleaned.csv', index=False)
"

📈 Data Statistics

• Total Practitioners: 13,487
• Total Procedures: 779
• Data Fields: 2,300+ columns (including procedure codes)
• File Sizes:
  • Raw CSV: ~200MB
  • Cleaned CSV: ~162MB
  • Raw JSONL: ~144MB
• Coverage: All active PHIN practitioners
• Update Frequency: Annual refresh workflow supported

🔧 Advanced Usage

Production Scraper Configuration

Customize the main production scraper settings:

# Edit production_config.json to adjust:
# - batch_size: Number of procedures per batch (default: 20)
# - max_workers: Parallel worker threads (default: 5)
# - min_request_interval: Rate limiting interval (default: 0.5s)
# - max_pages_per_procedure: Pages to scrape per procedure (default: 50)

# Run with custom configuration
python phin_production_scraper.py --config production_config.json

Testing and Debug

# Test with small subset using production scraper
python run_production_scraper.py test

# Test specific procedures
python phin_production_scraper.py --procedure-ids 1-5 --max-pages 3

# Debug mode with verbose logging
python phin_production_scraper.py --debug --procedure-ids 1-2

Stealth Scraper (Alternative)

For conservative scraping when needed:

# Stealth scraper for profile collection
python phin_stealth_scraper.py --letters A-Z --stale-days 30 --out temp_refresh.csv

# Adjust delays for stealth mode
python phin_stealth_scraper.py --delay-min 2.0 --delay-max 4.0

# Export-only mode (no network requests)
python phin_stealth_scraper.py --export-cache-only --out test.csv --max-profiles 10

🕷️ Technical Features

Scraping Strategy

• Respectful Rate Limiting: 1.2-3.2 second delays between requests
• Exponential Backoff: Intelligent retry logic for server errors
• ETag Caching: Only downloads changed data
• Stealth Techniques: Randomized delays and pattern breaking

Data Processing

• Incremental Updates: Only processes changed profiles
• Memory Efficient: Processes large datasets without memory issues
• Error Handling: Robust error recovery and logging
• Data Validation: Ensures data quality and consistency

Caching System

• Cache-First Design: Local cache as single source of truth
• Smart Updates: Only refreshes stale data
• Resume Capability: Can restart interrupted operations
• State Tracking: Monitors update history and progress

⚠️ Important Notes

Data Usage

• Educational/Research: This data is for analysis and research purposes
• Privacy Compliant: Only publicly available PHIN data collected
• Rate Limiting: Respects PHIN API terms of service
• No Personal Data: No sensitive personal information collected

File Management

• Cache Directory: Never delete cache_phin/ - it's your data source
• Output Files: Always overwritten during updates
• Backup Strategy: Regular backups recommended for long-term storage
• Storage Requirements: ~2GB for complete dataset with images

🐛 Troubleshooting

Common Issues

1. Rate Limiting: Increase delays or use chunked updates
2. Memory Issues: Process data in smaller batches
3. Network Errors: Use resume capability to continue interrupted operations
4. File Permissions: Ensure write access to output directories

Debug Commands

# Check cache status
ls cache_phin/profiles/ | wc -l

# Verify data integrity
python -c "
import json
with open('production_output/consultants_with_procedures_cleaned.jsonl') as f:
    lines = [json.loads(line) for line in f]
    print(f'Records: {len(lines)}')
    print(f'Sample keys: {list(lines[0].keys())[:10]}')
"

# Test data processing
python append_procedures_to_profiles.py --max-profiles 10

🔮 Future Enhancements

• Real-time Updates: Live data synchronization
• Database Integration: Direct database exports
• Advanced Analytics: Built-in data analysis tools
• API Interface: REST API for data access
• Machine Learning: Practitioner recommendation algorithms
• Geographic Analysis: Location-based insights

📄 License

This project is for educational and research purposes. Please respect the PHIN API terms of service and implement appropriate rate limiting when using this tool.

🤝 Contributing

This project represents a complete data collection and processing pipeline for healthcare practitioner analysis. The codebase includes multiple iterations and testing phases, demonstrating the evolution from a simple scraper to a comprehensive data system.

Feel free to submit issues and enhancement requests! The modular design makes it easy to extend functionality for specific research needs.

---

Project Evolution: Started as a simple PHIN scraper, evolved into a comprehensive practitioner data collection system with procedure code matching, data cleaning, and multiple export formats. The final output provides a complete dataset of UK private healthcare practitioners with their associated procedures and admission statistics.