Adds Tiktoken tokenizer for Nemotron-Mistral 12B#9797

Merged

akoumpa merged 153 commits intomainfrom

tkonuk/tiktoken

Jul 22, 2024

Commits on Jul 19, 2024

Adding context- & expert-parallism to MegatronStrategy (#9525)

marcromeyn
authored and
ertkonuk
committed
Add CICD test for Stable Diffusion (#9464)

michal2409
authored and
ertkonuk
committed
Akoumparouli/nemo ux mixtral (#9446)

akoumpa
authored and
ertkonuk
committed
update mcoreddp call (#9345)

authored and
ertkonuk
committed
[NeMo-UX] Llama and Gemma (#9528)

authored and
ertkonuk
committed
[NeMo-UX] minor logging bug fixes (#9529)

authored and
ertkonuk
committed
mcore distOpt restore fix (#9421)

akoumpa
authored and
ertkonuk
committed
Custom Tiktoken tokenizer.
ertkonuk
committed
Fixed the tokenizer decoding on special tokens.
ertkonuk
committed
Apply isort and black reformatting
ertkonuk
committed
Added token_to_id() method.
ertkonuk
committed
Update neva conversion script from and to HF (#9296)

yaoyu-33
authored and
ertkonuk
committed
vLLM Export Support (#9381)

authored and
ertkonuk
committed
PL: Delete precision if using plugin. TODO switch to MegatronTrainerBuilder (#9535)

akoumpa
authored and
ertkonuk
committed
Add page context fmha (#9526)

meatybobby
authored and
ertkonuk
committed
extend get_gpt_layer_modelopt_spec to support MoE (#9532)

akoumpa
authored and
ertkonuk
committed
fix mock data generation for legacy dataset (#9530)

dimapihtar
authored and
ertkonuk
committed
[Nemo-UX] IO fixes (#9512)

marcromeyn
authored and
ertkonuk
committed
Test C++ runtime on demand in nemo_export.py to avoid possible OOMs (#9544)

janekl
authored and
ertkonuk
committed
Fix lhotse tests for v1.24.2 (#9546)

pzelasko
authored and
ertkonuk
committed
gpu_unitTests_notOptional (#9551)

pablo-garay
authored and
ertkonuk
committed
add reset learning rate functionality (#9372)

dimapihtar
authored and
ertkonuk
committed
Add Python AIStore SDK to container and bump min Lhotse version (#9537)

pzelasko
authored and
ertkonuk
committed
Adding 'use_dynamo' option for export to use onnx.dynamo_export() instead of onnx.export() (#9147)

authored and
ertkonuk
committed
[NeMo-UX] Fix tokenizer IO (#9555)

marcromeyn
authored and
ertkonuk
committed
[NeMo UX] Move mistral_7b.py to mistral.py (#9545)

akoumpa
authored and
ertkonuk
committed
Use closed-formula to round by multiple (#9307)

authored and
ertkonuk
committed
ci: Do not attempt to send slack on fork (#9556)

ko3n1g
authored and
ertkonuk
committed
Fix nemo export test (#9547)

authored and
ertkonuk
committed
Fix SDXL incorrect name in docs (#9534)

suiyoubi
authored and
ertkonuk
committed
GPU unit tests: Mark flaky tests to be fixed (#9559)

pablo-garay
authored and
ertkonuk
committed
Bump PTL version (#9557)

athitten
authored and
ertkonuk
committed
[Resiliency] Straggler detection (#9473)

authored and
ertkonuk
committed
switch to torch_dist as default dist checkpointing backend (#9541)

authored and
ertkonuk
committed
[NeMo-UX] Checkpointing bug fixes (#9562)

authored and
ertkonuk
committed
Add tps and pps params to the export script (#9558)

oyilmaz-nvidia
authored and
ertkonuk
committed
Consolidate gpt continue training script into pretraining script (#9413)

yaoyu-33
authored and
ertkonuk
committed
Add support to change Multi task model prompt (#9542)

authored and
ertkonuk
committed
Add Multimodal Exporter (#9256)

meatybobby
authored and
ertkonuk
committed
Enable encoder adapters for Canary and MultiTaskAED models (#9409)

authored and
ertkonuk
committed
pass option through (#9570)

maanug-nv
authored and
ertkonuk
committed
PTQ refinements (#9574)

janekl
authored and
ertkonuk
committed
Audio model collection (#9263)

anteju
authored and
ertkonuk
committed
[NeMo-UX] Fix Trainer serialization (#9571)

marcromeyn
authored and
ertkonuk
committed
Update click version requirement (#9580)

thomasdhc
authored and
ertkonuk
committed
[Fault tolerance] Heartbeat detection (#9352)

authored and
ertkonuk
committed
Add ModelOpt QAT example for Llama2 SFT model (#9326)

kevalmorabia97
authored and
ertkonuk
committed
Set TE flag in legacy -> mcore conversion script (#9585)

cuichenx
authored and
ertkonuk
committed
[Nemo-UX] Add fabric-API for manual forward-pass (#9577)

marcromeyn
authored and
ertkonuk
committed
[Nemo-UX] Add SDK-factories to llm-collection (#9589)

marcromeyn
authored and
ertkonuk
committed
Multimodal projection layer adapter fix for PP>1 (#9445)

authored and
ertkonuk
committed
Add offline quantization script for QLoRA deployment (#9455)

cuichenx
authored and
ertkonuk
committed
qlora support more models (#9488)

cuichenx
authored and
ertkonuk
committed
[NeMo-UX] Some improvements to NeMoLogger (#9591)

marcromeyn
authored and
ertkonuk
committed
Set n_gpu to None in nemo export (#9593)

oyilmaz-nvidia
authored and
ertkonuk
committed
Inflight nemo model export support (#9527)

JimmyZhang12
authored and
ertkonuk
committed
vLLM Export Improvements (#9596)

authored and
ertkonuk
committed
Set finalize_model_grads_func in on_fit_start instead to make sure it's being called (#9599)

marcromeyn
authored and
ertkonuk
committed
Set no_sync_func & grad_sync_fucn (#9601)

akoumpa
authored and
ertkonuk
committed
small nemo logger bug fix (#9607)

authored and
ertkonuk
committed
fix the dict format returned by scheduler method (#9609)

authored and
ertkonuk
committed
[NeMo-UX] Dataloading enhancements and bug fixes (#9595)

authored and
ertkonuk
committed
Fix serialization of AutoResume (#9616)

sararb
authored and
ertkonuk
committed
Chat template support for megatron_gpt_eval.py (#9354)

authored and
ertkonuk
committed
Jsonl support (#9611)

adityavavre
authored and
ertkonuk
committed
[NeMo-UX] Add PEFT (#9490)

authored and
ertkonuk
committed
Akoumparouli/mistral import instruct chat template fix (#9567)

authored and
ertkonuk
committed
Remove .cuda calls, use device isntead (#9602)

akoumpa
authored and
ertkonuk
committed
fix converter defautl args (#9565)

akoumpa
authored and
ertkonuk
committed
mixtral export (#9603)

akoumpa
authored and
ertkonuk
committed
fix: remove non_blocking from PTL's .cuda call (#9618)

akoumpa
authored and
ertkonuk
committed
Alit/mamba tmp (#9612)

authored and
ertkonuk
committed
TitaNet Batch Verify Speaker (#9337)

authored and
ertkonuk
committed
Enable MCore checkpointing optimizations (#9505)

mikolajblaz
authored and
ertkonuk
committed
Change mixtral moe key name for trt-llm (#9620)

oyilmaz-nvidia
authored and
ertkonuk
committed
fix ckpt load bug (#9621)

dimapihtar
authored and
ertkonuk
committed
NeVA Minor Fixes (#9608)

yaoyu-33
authored and
ertkonuk
committed
fix pretrianing data sizes and weights (#9627)

cuichenx
authored and
ertkonuk
committed
Alit/mamba (#9575)

authored and
ertkonuk
committed
[NeMo-UX] async checkpointing support (#9466)

authored and
ertkonuk
committed
Fix the arguments of forward_for_export function in msdd_models (#9624)

tango4j
authored and
ertkonuk
committed
Change default parallel_save to False (#9632)

mikolajblaz
authored and
ertkonuk
committed
Unwrap ckpt_io for model opt (async save) (#9622)

mikolajblaz
authored and
ertkonuk
committed
MCore T5 support for NeMo - Training (#9432)

authored and
ertkonuk
committed
[Nemo-UX] Expose transformer_layer_spec inside GPTConfig (#9592)

marcromeyn
authored and
ertkonuk
committed
Update NeMo Clip to Use MCore Modules (#9594)

yaoyu-33
authored and
ertkonuk
committed
Add REST API to deploy module (#9539)

authored and
ertkonuk
committed
Mistral + Mixtral Support for NeVa (#9459)

paul-gibbons
authored and
ertkonuk
committed
ci: Timeout per step, not job (#9635)

ko3n1g
authored and
ertkonuk
committed
Adding support for mcore generate (#9566)

shanmugamr1992
authored and
ertkonuk
committed
Improve error messaging during trt-llm export (#9638)

oyilmaz-nvidia
authored and
ertkonuk
committed
Nemotron export - fixing megatron_export.py (#9625)

authored and
ertkonuk
committed
support lora when kv_channel != hidden_size / num_heads (#9636)

suiyoubi
authored and
ertkonuk
committed
[Nemo CICD] Docker temp files auto-cleanup (#9642)

pablo-garay
authored and
ertkonuk
committed
Update Dockerfile.ci (#9651)

huvunvidia
authored and
ertkonuk
committed
SDXL improvements (and support for Draft+) [DRAFT PR] (#9543)

authored and
ertkonuk
committed
Triton deployment improvements for in-framework models (#9600)

authored and
ertkonuk
committed
Use FP8 in GPT TP2 test (#9451)

authored and
ertkonuk
committed
enables default data step in megatron parallel to operate on a wider variety of tensors (#9641)

authored and
ertkonuk
committed
Revert "enables default data step in megatron parallel to operate on a wider …" (#9666)

marcromeyn
authored and
ertkonuk
committed
Contrastive Reranker/Reward model (#9171)

arendu
authored and
ertkonuk
committed
unpin transformers version (#9606)

authored and
ertkonuk
committed
Added CPU offloading docs (#9479)

authored and
ertkonuk
committed
Update llama-3 PEFT notebook to download model from NGC (#9667)

shashank3959
authored and
ertkonuk
committed
fix pipeline parallel dtype bug (#9637) (#9661)

authored and
ertkonuk
committed
LITA integration (#9578)

authored and
ertkonuk
committed
Parametrize FPS group (#9648) (#9669)

authored and
ertkonuk
committed
Huvu/mcore t5 (#9677)

authored and
ertkonuk
committed
chore: Version bump NeMo (#9631)

ko3n1g
authored and
ertkonuk
committed
add a bit more for timeout (#9702)

pablo-garay
authored and
ertkonuk
committed
Alit/mamba (#9696)

authored and
ertkonuk
committed
NeMo performance feature documentation (#9482)

erhoo82
authored and
ertkonuk
committed
[TTS] Add fullband mel codec checkpoints (#9704)

rlangman
authored and
ertkonuk
committed
Adding support for mcore T5 Eval - SFT - PEFT (#9679)

authored and
ertkonuk
committed
Allows non-strict load with distributed checkpoints (#9613) (#9715)

authored and
ertkonuk
committed
refactor: Uniform BRANCH for notebooks (#9710)

ko3n1g
authored and
ertkonuk
committed
fix legacy ds padding bug (#9716)

dimapihtar
authored and
ertkonuk
committed
enables default data step in megatron parallel to operate on a wider variety of tensors - second try (#9671)

authored and
ertkonuk
committed
[NeMo-UX] Fix when optimizers are setup for PEFT (#9619) (#9647)

authored and
ertkonuk
committed
refactor: README (#9712)

ko3n1g
authored and
ertkonuk
committed
Remove mask if use fusion mask (#9723)

hsiehjackson
authored and
ertkonuk
committed
[NeMo-UX] Fix imports so local configuration of runs works again (#9690) (#9694)

authored and
ertkonuk
committed
add contianer (#9731)

authored and
ertkonuk
committed
update pretrained model text (#9724) (#9745)

authored and
ertkonuk
committed
[Nemo-UX] Including all trainable-params in a PEFT-checkpoint (#9650) (#9691)

authored and
ertkonuk
committed
[NeMo-UX] Make TE and Apex dependencies optional (#9732)

authored and
ertkonuk
committed
[NeMo-UX] Minor bug fix when TE/Apex not installed (#9749)

ashors1
authored and
ertkonuk
committed
make 'load_directly_on_device' configurable (#9657) (#9674)

authored and
ertkonuk
committed
TorchAudio installation workaround for incorrect `PYTORCH_VERSION` env variable (#9736) (#9750)

authored and
ertkonuk
committed
Create __init__.py (#9755)

stevehuang52
authored and
ertkonuk
committed
Canary Adapters tutorial (#9670)

titu1994
authored and
ertkonuk
committed
match nemo 1's default behavior for drop_last and pad_samples_to_global_batch_size (#9707) (#9753)

authored and
ertkonuk
committed
ci: Bump MCore tag (#9744)

ko3n1g
authored and
ertkonuk
committed
Fix the serialization of partial functions in nemo 2.0 (#9668)

sararb
authored and
ertkonuk
committed
ci: Add PAT to create-pullrequest action (#9769)

ko3n1g
authored and
ertkonuk
committed
Speeds up copying of necessary artifact files with SaveRestoreConnector (#9682)

authored and
ertkonuk
committed
ci: Remove ko3n1g from reviewers (#9773)

ko3n1g
authored and
ertkonuk
committed
bump mcore commit in Dockerfile (#9766)

ashors1
authored and
ertkonuk
committed
Yuya/add checkpoints section (#9329)

authored and
ertkonuk
committed
Release automation (#9687)

ko3n1g
authored and
ertkonuk
committed
Rename speech dockerfile appropriately (#9778)

pablo-garay
authored and
ertkonuk
committed
Add option to convert PyTriton response to OpenAI format (#9726)

athitten
authored and
ertkonuk
committed
ci: Fix changelog-config (#9788)

ko3n1g
authored and
ertkonuk
committed
Support configurable extra fields for LazyNeMoTarredIterator (#9548)

pzelasko
authored and
ertkonuk
committed
upper bound huggingface-hub version to 0.24.0 (exc.) (#9799)

akoumpa
authored and
ertkonuk
committed
CodeQL fixes

akoumpa
authored and
ertkonuk
committed
import guard

akoumpa
authored and
ertkonuk
committed
add tiktoken to requirements

akoumpa
authored and
ertkonuk
committed
[pre-commit.ci] auto fixes from pre-commit.com hooks

pre-commit-ci[bot]
authored and
ertkonuk
committed
Apply isort and black reformatting

pre-commit-ci[bot]
authored and
ertkonuk
committed
Apply isort and black reformatting
ertkonuk
committed
Merge branch 'main' into tkonuk/tiktoken
akoumpa
authored
Apply isort and black reformatting
akoumpa
committed