CPU based pre-trained model

[johnyoonh]

I am guessing that the model provided is for machines with CUDA-capable device.
Do you guys happen to have a pre-trained CPU version for cnndm_model.bin ?

@@ -165,7 +165,7 @@ def main():
     print(args.model_recover_path)
     for model_recover_path in glob.glob(args.model_recover_path.strip()):
         logger.info("***** Recover model: %s *****", model_recover_path)
-        model_recover = torch.load(model_recover_path)
+        model_recover = torch.load(model_recover_path, map_location="cpu")

DATA_DIR=../cnndm_data
MODEL_RECOVER_PATH=../cnndm_model.bin
EVAL_SPLIT=test
export PYTORCH_PRETRAINED_BERT_CACHE=/tmp/bert-cased-pretrained-cache
# run decoding
python biunilm/decode_seq2seq.py --fp16 --amp --bert_model bert-large-cased --new_segment_ids --mode s2s --need_score_t
races \
  --input_file ${DATA_DIR}/${EVAL_SPLIT}.src --split ${EVAL_SPLIT} --tokenized_input \
  --model_recover_path ${MODEL_RECOVER_PATH} \
  --max_seq_length 768 --max_tgt_length 128 \
  --batch_size 64 --beam_size 5 --length_penalty 0 \
  --forbid_duplicate_ngrams --forbid_ignore_word ".|[X_SEP]"
11/04/2019 15:55:06 - INFO - pytorch_pretrained_bert.tokenization -   loading vocabulary file https://s3.amazonaws.com/
models.huggingface.co/bert/bert-large-cased-vocab.txt from cache at /tmp/bert-cased-pretrained-cache/cee054f6aafe5e2cf8
16d2228704e326446785f940f5451a5b26033516a4ac3d.e13dbb970cb325137104fb2e5f36fe865f27746c6b526f6352861b1980eb80b1
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "biunilm/decode_seq2seq.py", line 254, in <module>
    main()
  File "biunilm/decode_seq2seq.py", line 147, in main
    amp_handle = amp.init(enable_caching=True)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/apex/amp/amp.py", line 65, in init
    handle = AmpHandle(enable_caching, verbose)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/apex/amp/handle.py", line 14, in __init__
    self._default_scaler = LossScaler()
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/apex/amp/scaler.py", line 35, in __init__
    self._overflow_buf = torch.cuda.IntTensor([0])
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:51
[1]    72305 exit 1     python biunilm/decode_seq2seq.py --fp16 --amp --bert_model bert-large-cased

without --amp:

Traceback (most recent call last):
  File "biunilm/decode_seq2seq.py", line 254, in <module>
    main()
  File "biunilm/decode_seq2seq.py", line 216, in main
    position_ids, input_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1409, in forward
    return self.beam_search(input_ids, token_type_ids, position_ids, attention_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1528, in beam_search
    output_all_encoded_layers=True, prev_embedding=prev_embedding, prev_encoded_layers=prev_encoded_layers, mask_qkv=mask_qkv)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1062, in forward
    input_ids, token_type_ids, attention_mask)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1037, in get_extended_attention_mask
    extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/tensor.py", line 371, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
RuntimeError: "add_cpu" not implemented for 'Half'

Packages:

pytorch-pretrained-bert 0.4.0    
torch                   1.1.0 
tensorboardX            1.9
apex                     0.1

[nyanyanya]

Have you tried to run without "--fp16" flag? It seems Pytorch didn't implement half precision for CPU.

[johnyoonh]

Yeah, this is what I get:

11/04/2019 17:59:22 - INFO - pytorch_pretrained_bert.modeling -   Weights from pretrained model not used in BertForSeq2SeqDecoder: ['crit_mask_lm_smoothed.one_hot']
  0%|                                                                                                                                                                                                                  | 0/180 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "biunilm/decode_seq2seq.py", line 254, in <module>
    main()
  File "biunilm/decode_seq2seq.py", line 216, in main
    position_ids, input_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1409, in forward
    return self.beam_search(input_ids, token_type_ids, position_ids, attention_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1528, in beam_search
    output_all_encoded_layers=True, prev_embedding=prev_embedding, prev_encoded_layers=prev_encoded_layers, mask_qkv=mask_qkv)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 1065, in forward
    input_ids, token_type_ids, position_ids, task_idx=task_idx)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/code/unilm/src/pytorch_pretrained_bert/modeling.py", line 251, in forward
    embeddings = self.LayerNorm(embeddings)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/apex/normalization/fused_layer_norm.py", line 149, in forward
    input, self.weight, self.bias)
  File "/home/john/.virtualenvs/unilm/lib/python3.6/site-packages/apex/normalization/fused_layer_norm.py", line 21, in forward
    input_, self.normalized_shape, weight_, bias_, self.eps)
RuntimeError: input must be a CUDA tensor (layer_norm_affine at apex/normalization/csrc/layer_norm_cuda.cpp:120)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f241afc6441 in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f241afc5d7a in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: layer_norm_affine(at::Tensor, c10::ArrayRef<long>, at::Tensor, at::Tensor, double) + 0x66f (0x7f240505f07f in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x18af4 (0x7f240506baf4 in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #4: <unknown function> + 0x1623a (0x7f240506923a in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/fused_layer_norm_cuda.cpython-36m-x86_64-linux-gnu.so)
frame #5: python() [0x50abc5]
<omitting python frames>
frame #8: python() [0x5951c1]
frame #10: THPFunction_do_forward(THPFunction*, _object*) + 0x2bc (0x7f245a760dbc in /home/john/.virtualenvs/unilm/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #12: python() [0x54ac01]
frame #14: python() [0x50ab53]
frame #17: python() [0x5951c1]
frame #20: python() [0x5081d5]
frame #22: python() [0x5951c1]
frame #23: python() [0x54ac01]
frame #25: python() [0x50ab53]
frame #27: python() [0x5081d5]
frame #29: python() [0x5951c1]
frame #32: python() [0x5081d5]
frame #34: python() [0x5951c1]
frame #35: python() [0x54ac01]
frame #37: python() [0x50ab53]
frame #39: python() [0x5081d5]
frame #41: python() [0x5951c1]
frame #44: python() [0x5081d5]
frame #46: python() [0x5951c1]
frame #47: python() [0x54ac01]
frame #49: python() [0x50ab53]
frame #51: python() [0x5081d5]
frame #52: python() [0x50a020]
frame #53: python() [0x50aa1d]
frame #55: python() [0x5081d5]
frame #57: python() [0x5951c1]
frame #60: python() [0x5081d5]
frame #62: python() [0x5951c1]
frame #63: python() [0x54ac01]

[1]    22291 exit 1     python biunilm/decode_seq2seq.py --bert_model bert-large-cased  --mode s2s

[johnyoonh]

There is a commit in apex to make FusedLayerNorm cpu compatible, but it's newer than the commit (by 100 commits) suggested in the README.md
NVIDIA/apex#256
I tried using that commit for installation of apex, but it seems like since the model was built with the older commit, it doesn't help.

[aretius]

Hey @johnyoonh
To run in CPU mode you should run decode_seq2seq.py without --amp and --fp16.
Also don't install apex. Where ever the code is importing apex just comment that part out.
Apex is just supposed to make things faster not necessary for inference.
Also stick to the versions of libraries as suggested in the README.md
I have spent a whole day to figure out the CPU inference for UniLM. If you have more doubts do tell me.

[donglixp]

Hi @aretius ,

Thanks for the pointer. I will also update the FAQ section in README.md to include your solution.

[johnyoonh]

Thanks aretius

The following changes made it work.
Lowering batch_size and beam_size is also crucial.

DATA_DIR=./cnndm_data                                                                                                                                                         
MODEL_RECOVER_PATH=./cnndm_model.bin
EVAL_SPLIT=test
PYTORCH_PRETRAINED_BERT_CACHE=~/tmp/bert-cased-pretrained-cache
python src/biunilm/decode_seq2seq.py --bert_model bert-large-cased --new_segment_ids --mode s2s --need_score_traces \
  --input_file ${DATA_DIR}/${EVAL_SPLIT}.src --split ${EVAL_SPLIT} --tokenized_input \
  --model_recover_path ${MODEL_RECOVER_PATH} \
  --max_seq_length 768 --max_tgt_length 128 \
  --batch_size 1 --beam_size 2 --length_penalty 0 \
  --forbid_duplicate_ngrams --forbid_ignore_word ".|[X_SEP]"

--- a/src/pytorch_pretrained_bert/__init__.py
+++ b/src/pytorch_pretrained_bert/__init__.py
@@ -3,5 +3,5 @@ from .tokenization import BertTokenizer, BasicTokenizer, WordpieceTokenizer
 from .modeling import (BertConfig, BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction, BertForSequenceClassification,
                        BertForMultipleChoice, BertForTokenClassification, BertForQuestionAnswering, BertForPreTrainingLossMask, BertPreTrainingPairRel, BertPreTrainingPairTransform)
 from .optimization import BertAdam, BertAdamFineTune
-from .optimization_fp16 import FP16_Optimizer_State
+# from .optimization_fp16 import FP16_Optimizer_State
 from .file_utils import PYTORCH_PRETRAINED_BERT_CACHE
diff --git a/src/pytorch_pretrained_bert/modeling.py b/src/pytorch_pretrained_bert/modeling.py

[fseasy]

I got errors

Traceback (most recent call last):
  File "biunilm/decode_seq2seq.py", line 263, in <module>
    main()
  File "biunilm/decode_seq2seq.py", line 225, in main
    position_ids, input_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/users/xuwei15/env/py3_pytorch1.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/users/xuwei15/learning/unilm/src/pytorch_pretrained_bert/modeling.py", line 1410, in forward
    return self.beam_search(input_ids, token_type_ids, position_ids, attention_mask, task_idx=task_idx, mask_qkv=mask_qkv)
  File "/home/users/xuwei15/learning/unilm/src/pytorch_pretrained_bert/modeling.py", line 1688, in beam_search
    forbid_word_mask, [batch_size * K, 1, vocab_size]).cuda()
  File "/home/users/xuwei15/env/py3_pytorch1.1/lib/python3.7/site-packages/torch/cuda/__init__.py", line 162, in _lazy_init
    _check_driver()
  File "/home/users/xuwei15/env/py3_pytorch1.1/lib/python3.7/site-packages/torch/cuda/__init__.py", line 75, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

The except place is

1687                         forbid_word_mask = torch.reshape(
1688                             forbid_word_mask, [batch_size * K, 1, vocab_size]).cuda()

I removed .cuda(), Is it OK?

[aretius]

Hey @MeMeDa
Since you are using the CPU version of pytorch, the code base in multiple places will have code in the form of xxxx.cuda() All you need to do is remove the .cuda() from that code.

[anshoomehra]

Hey @johnyoonh
To run in CPU mode you should run decode_seq2seq.py without --amp and --fp16.
Also don't install apex. Where ever the code is importing apex just comment that part out.
Apex is just supposed to make things faster not necessary for inference.
Also stick to the versions of libraries as suggested in the README.md
I have spent a whole day to figure out the CPU inference for UniLM. If you have more doubts do tell me.

@aretius This is great! What is the throughput you are getting post transitioning to CPU for inference only? For example on GPU, one instance is roughly generating 184 questions for me. Secondly, any suggestions on making the GPU inference faster ?? I created a new Issue, and should you have any suggestions, please do post there directly such that all benefit from it.

Thanks!!
Anshoo

[donglixp]

If the large-size model is too slow for inference, please refer to https://github.com/microsoft/unilm/tree/master/s2s-ft for the base-size checkpoints. We will also update a mini-size model at https://github.com/microsoft/unilm/tree/master/minilm (paper: https://arxiv.org/pdf/2002.10957.pdf ) .