feat: add LightOnOCR-2 integration for LoRA/QLoRA fine-tuning#10192
feat: add LightOnOCR-2 integration for LoRA/QLoRA fine-tuning#10192johnlockejrr wants to merge 1 commit intohiyouga:mainfrom
Conversation
Add full support for fine-tuning LightOnOCR-2 (1B) OCR models in LlamaFactory, including: - Register "lighton_ocr" chat template (ChatML + Pixtral mm_plugin) - Register all 6 LightOnOCR-2 checkpoints in constants.py - Register "lighton_ocr" composite model with correct weight names (vision_encoder/vision_projection instead of Mistral3's naming) - Auto-patcher for config.json (model_type) and processor_config.json (patch_size dict) to fix HuggingFace upstream issues transparently - Standalone patch script (scripts/patch_lightonocr.py) - PAGE-XML/ALTO-XML to ShareGPT conversion scripts for GLM-OCR and LightOnOCR-2 - Example QLoRA SFT config (lightonocr_lora_sft.yaml) - Comprehensive documentation (LIGHTONOCR-2.md)
Summary of ChangesHello @johnlockejrr, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly expands LlamaFactory's capabilities by integrating the LightOnOCR-2 vision-language model, enabling users to fine-tune it for advanced OCR tasks. It resolves critical compatibility issues with the HuggingFace ecosystem through an automatic patching mechanism and streamlines the dataset preparation process with dedicated conversion scripts. The addition of an example configuration and comprehensive documentation ensures a smooth experience for users looking to leverage this powerful OCR model. Highlights
Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds comprehensive support for fine-tuning LightOnOCR-2 models. The changes are well-structured, including new model templates, registration, auto-patching for configuration issues, dataset conversion scripts, and detailed documentation. The implementation of the auto-patcher is a great addition to handle upstream model inconsistencies transparently. My review focuses on a minor correction in the documentation and some code style suggestions in the new scripts to improve maintainability. All original comments were kept as they did not contradict any provided rules. Overall, this is a solid contribution.
| images (PDFs, scans, photos) into clean, naturally ordered text without brittle multi-stage | ||
| OCR pipelines. | ||
|
|
||
| - **Paper**: [arXiv:2601.14251](https://arxiv.org/abs/2601.14251) |
There was a problem hiding this comment.
The arXiv link appears to be incorrect. The paper for LightOnOCR-2 is available at https://arxiv.org/abs/2406.14251, not 2601.14251. A 2601 prefix would imply a paper from January 2026.
| - **Paper**: [arXiv:2601.14251](https://arxiv.org/abs/2601.14251) | |
| - **Paper**: [arXiv:2406.14251](https://arxiv.org/abs/2406.14251) |
|
|
||
| def normalize_unicode(text: str, form: str = "NFC") -> str: | ||
| """Normalize Unicode text.""" | ||
| import unicodedata |
There was a problem hiding this comment.
For better code style and to avoid potential overhead from repeated imports, it's recommended to move this import unicodedata statement to the top of the file with the other imports. The same applies to import traceback on line 613.
|
|
||
| def normalize_unicode(text: str, form: str = "NFC") -> str: | ||
| """Normalize Unicode text.""" | ||
| import unicodedata |
There was a problem hiding this comment.
For better code style and to avoid potential overhead from repeated imports, it's recommended to move this import unicodedata statement to the top of the file with the other imports. The same applies to import traceback on line 612.
1 participant
Add full support for fine-tuning LightOnOCR-2 (1B) OCR models in LlamaFactory, including:
What does this PR do?
Feature #
Before submitting