Add HackerNews support and improve data quality metrics

[ARJ999]

Summary

This PR adds HackerNews as a new data source for research reports and introduces comprehensive data quality metrics to provide transparency about data verification and freshness. It also includes performance improvements for Reddit enrichment and refinements to scoring algorithms.

Key Changes

New Features

• HackerNews Integration: Added scripts/lib/hn.py module to search HackerNews via Algolia API with configurable depth levels (quick/default/deep)
• Data Quality Metrics: New DataQuality schema class that tracks verified dates, verified engagement, average recency, and available/failed sources
• HNItem Schema: New HNItem dataclass for normalized HackerNews items with engagement metrics (points, comments)

Performance & Reliability

• Parallel Reddit Enrichment: Converted sequential Reddit enrichment to use ThreadPoolExecutor with 5 workers for better throughput while maintaining per-item error handling
• Model Cache Optimization: Reduced model cache TTL from 7 days to 1 day to ensure latest models are always used
• Engagement Verification: Added engagement_verified flag to track whether engagement data comes from actual API responses vs. estimates

Scoring & Ranking Improvements

• Exponential Recency Bias: Replaced linear recency scoring with tiered exponential scoring that strongly prioritizes content from the last 3 days (premium tier: 90-100) while still valuing week-old content (high tier: 75-89)
• Quality-Focused Engagement Weights:
  • Reddit: Increased upvote ratio weight (30%) to reward community agreement
  • X: Prioritized reposts (40%) as a deep engagement signal
  • HN: New formula with 60% points + 40% comments
• Verified Data Bonus: Added +8 point bonus for items with verified engagement data from APIs
• Unverified Penalty: Increased penalty for unknown engagement from 10 to 15 points

Data Validation

• Reddit Engagement Validation: New is_engagement_valid() function to detect deleted posts and invalid metrics
• Stricter Date Requirements: Updated Reddit search prompt to enforce date filtering and reject pre-filtered old content
• HN Date Handling: Proper Unix timestamp conversion with timezone awareness

UI & Reporting

• Data Quality Section: New section in compact render showing total items, verification percentages, average recency, and source availability
• HackerNews Display: Dedicated section showing HN items with points, comments, and discussion links
• Enhanced Error Tracking: Separate error fields for each source (reddit_error, x_error, hn_error, web_error)

Model Selection

• GPT-5 Series Priority: Updated fallback models to prefer latest GPT-5.5, GPT-5.3, GPT-5.2 over older versions
• Grok Model Improvements: Changed xAI latest from grok-4-1-fast to grok-4-1 for higher quality, with proper x_search capability detection

Implementation Details

• HN search uses Algolia's free API (no authentication required)
• Engagement metrics are computed using logarithmic scaling for better distribution across different platforms
• Data quality metrics are computed server-side after all sources complete
• Thread pool executor properly handles futures with as_completed() for progress updates
• All new fields are backward compatible with existing report serialization

Testing

• Updated test expectations for exponential recency scoring
• Updated model selection tests to reflect new GPT-5 and Grok preferences
• Maintained backward compatibility with existing report deserialization

https://claude.ai/code/session_01YNFJJXPx5fDWRdfHAz7k27