LITA integration by Slyne · Pull Request #9578 · NVIDIA-NeMo/NeMo

Slyne · 2024-07-01T15:43:37Z

What does this PR do ?

This PR aims to add LITA from checkpoint conversion, dataset preprocessing for DVC datasets to finetuning.
It also adds vila to the lita model.

Collection: [Note which collection this PR will affect]

multimodal
nlp

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Slyne · 2024-07-01T15:45:39Z

TO DO LIST:

to move convert_llava_hf_to_nemo.py
to add tutorial

examples/multimodal/multimodal_llm/neva/eval/eval_qa.py

scripts/multimodal_dataset_conversion/generate_qa_data.py

scripts/multimodal_dataset_conversion/convert_dvc_dataset_for_training.py

+
+event_prompts = [
+    "What is the action performed in this video?",
+    "Can you highlight the action performed in this video?" "What is the main event or action captured in this video?",


scripts/multimodal_dataset_conversion/convert_video_qa_dataset.py

+        return time_to_string(value) + f"<!|t{value}t|!>"
+
+    value = re.sub(r"<([\d.]{1,20})s>", repl, value)
+    value = re.sub(r"\s([\d.]{1,20})s[\s|\.|,|>]", repl, value)


nemo/collections/nlp/modules/common/text_generation_utils.py

+        return time_to_string(value) + f"<!|t{value}t|!>"
+
+    text = re.sub(r"<([\d.]{1,20})s>", repl, text)
+    text = re.sub(r"\s([\d.]{1,20})s[\s|\.|,|>]", repl, text)


examples/multimodal/multimodal_llm/convert_hf_llava_to_neva.py

nemo/collections/multimodal/parts/utils.py

nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py

scripts/multimodal_dataset_conversion/convert_dvc_dataset_for_training.py

examples/multimodal/multimodal_llm/neva/convert_llava_to_neva.py

scripts/multimodal_dataset_conversion/prepare_youmakeup.py

examples/multimodal/multimodal_llm/neva/conf/vita_config.yaml

examples/multimodal/multimodal_llm/neva/conf/lita_1_5_config.yaml

examples/multimodal/multimodal_llm/neva/conf/lita_1_5_config_lora.yaml

examples/multimodal/multimodal_llm/neva/conf/lita_config.yaml

examples/multimodal/multimodal_llm/neva/eval/eval_qa.py

examples/multimodal/multimodal_llm/neva/eval/eval_rtl.py

examples/multimodal/multimodal_llm/neva/neva_evaluation.py

nemo/collections/multimodal/parts/utils.py

examples/multimodal/multimodal_llm/neva/conf/lita_1_5_config.yaml

examples/multimodal/multimodal_llm/neva/conf/lita_1_5_config_lora.yaml

Signed-off-by: Slyne Deng <slyned@nvidia.com>

Signed-off-by: Slyne <Slyne@users.noreply.github.com>

Signed-off-by: slyne deng <slyned@nvidia.com>

Signed-off-by: Slyne <Slyne@users.noreply.github.com>

Signed-off-by: Slyne Deng <slyned@nvidia.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Signed-off-by: Malay Nagda <malayn@malayn-mlt.client.nvidia.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Signed-off-by: tonyjie <jl4257@cornell.edu>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* add lita Signed-off-by: Slyne Deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * add part of the tutorial and fix format Signed-off-by: slyne deng <slyned@nvidia.com> * add tutorial Signed-off-by: slyne deng <slyned@nvidia.com> * fix Tutorial ckpt conversion Signed-off-by: slyne deng <slyned@nvidia.com> * Apply isort and black reformatting Signed-off-by: Slyne <Slyne@users.noreply.github.com> * update cicd Signed-off-by: Slyne Deng <slyned@nvidia.com> * add to CIICD test Signed-off-by: Slyne Deng <slyned@nvidia.com> * changes based on review comments Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix bot warning Signed-off-by: Slyne Deng <slyned@nvidia.com> * update cicd main Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix cicd ckpt conversion Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Signed-off-by: Slyne <Slyne@users.noreply.github.com> Signed-off-by: slyne deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne <Slyne@users.noreply.github.com> Co-authored-by: Yu Yao <54727607+yaoyu-33@users.noreply.github.com>

github-actions bot added NLP Multi Modal labels Jul 1, 2024

github-advanced-security bot found potential problems Jul 1, 2024

View reviewed changes

github-advanced-security bot found potential problems Jul 5, 2024

View reviewed changes

examples/multimodal/multimodal_llm/neva/convert_llava_to_neva.py Fixed Show fixed Hide fixed

scripts/multimodal_dataset_conversion/prepare_youmakeup.py Fixed Show fixed Hide fixed

scripts/multimodal_dataset_conversion/prepare_youmakeup.py Fixed Show fixed Hide fixed

Slyne force-pushed the LITA branch from 707585a to f2d3a40 Compare July 5, 2024 09:09

Slyne changed the title ~~[DRAFT]LITA integration~~ LITA integration Jul 5, 2024

github-actions bot added the CI label Jul 5, 2024

yaoyu-33 self-requested a review July 8, 2024 17:26