RuntimeError:element 0 of tensors does not require grad and does not have a grad_fn

[mingzhu-wu]

Hi,
I got an error while training the MTBert baseline without SQuAD dataset, all the rest settings are the same with the example command except for the number of t_total.

The error is as follows:
File "/home/ukp/mwu/MRQA-Shared-Task-2019/baseline/venv-3.6/lib/python3.6/site-packages/allennlp/commands/train.py", line 243, in train_model
metrics = trainer.train()
File "/home/ukp/mwu/MRQA-Shared-Task-2019/baseline/venv-3.6/lib/python3.6/site-packages/allennlp/training/trainer.py", line 480, in train
train_metrics = self._train_epoch(epoch)
File "/home/ukp/mwu/MRQA-Shared-Task-2019/baseline/venv-3.6/lib/python3.6/site-packages/allennlp/training/trainer.py", line 327, in _train_epoch
loss.backward()
File "/home/ukp/mwu/MRQA-Shared-Task-2019/baseline/venv-3.6/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/ukp/mwu/MRQA-Shared-Task-2019/baseline/venv-3.6/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

The same error happens when I tried to add additional training examples.
I would be really appreciated to know the reason of this error and how to fix it.
Thanks in advance.

[alontalmor]

Hi xfwmz, you referred to ZHO9504 problem. Do you also experience this only on 2 GPUs? what is your running command? which version of AllenNLP do you use? (also there was a code update in the last few weeks to the baseline model, be sure you are using the latest code)

[ZHO9504]

I got this error too..

[alontalmor]

can you please send me the allennlp command you used here? thanks

[mingzhu-wu]

can you please send me the allennlp command you used here? thanks

Hi alontalmor, I reproduced this error with the following command:
python -m allennlp.run train https://multiqa.s3.amazonaws.com/config/MRQA_BERTbase.json -s MRQA-Shared-Task-2019/Models/MultiTrain -o "{'dataset_reader': {'sample_size': 75000}, 'validation_dataset_reader': {'sample_size': 1000}, 'train_data_path': 'https://mrqa.s3.us-east-2.amazonaws.com/data/train/NewsQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/train/HotpotQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/train/SearchQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/train/TriviaQA-web.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/train/NaturalQuestionsShort.jsonl.gz', 'validation_data_path': 'https://mrqa.s3.us-east-2.amazonaws.com/data/dev/SQuAD.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/dev/NewsQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/dev/HotpotQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/dev/SearchQA.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/dev/TriviaQA-web.jsonl.gz,https://mrqa.s3.us-east-2.amazonaws.com/data/dev/NaturalQuestionsShort.jsonl.gz', 'trainer': {'cuda_device': 0, 'num_epochs': '2', 'optimizer': {'type': 'bert_adam', 'lr': 3e-05, 'warmup': 0.1, 't_total': '120000'}}}" --include-package mrqa_allennlp

I was running this command on 1 GPU, using allennlp version 0.8.4, this error appeared after 8 hours of training when the EM score reaches 54, as shown in the following:
EM: 54.6440, f1: 65.0914, qas_used_fraction: 1.0000, loss: 4.2103 ||: : 59878it [7:06:57, 2.50it/s]
EM: 54.6462, f1: 65.0929, qas_used_fraction: 1.0000, loss: 4.2101 ||: : 59906it [7:07:08, 2.54it/s]
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/run.py", line 21, in
run()
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/run.py", line 18, in run
main(prog="allennlp")
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/commands/init.py", line 102, in main
args.func(args)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/commands/train.py", line 116, in train_model_from_args
args.cache_prefix)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/commands/train.py", line 160, in train_model_from_file
cache_directory, cache_prefix)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/commands/train.py", line 243, in train_model
metrics = trainer.train()
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/training/trainer.py", line 480, in train
train_metrics = self._train_epoch(epoch)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/allennlp/training/trainer.py", line 327, in _train_epoch
loss.backward()
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/ukp-storage-1/mwu/venv-mrqa/lib64/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

[mingzhu-wu]

I found the bug that results in this RuntimeError, it happens when a batch contains only one training example, then the expression "len(np.argwhere(span_start.squeeze().cpu() >= 0)) > 0" in BERT_QA.py line 91 would be false even if the gold answer is given. In this case, the output loss will be zero and this error happens.

I fixed this bug by modifying the expression from "if span_start is not None and len(np.argwhere(span_start.squeeze().cpu() >= 0)) > 0:" to "if span_start is not None and len(np.argwhere(span_start.squeeze(-1).squeeze(-1).cpu() >= 0)) > 0:", now it works fine.