generate simplified text

[vanh17]

Hi, I was wondering if there is a way for us to just use this tool and pass in the txt file containing difficult sentences, one per line and get the simplified sentences? If this is doable, can we generate without training the model? Thank you!

[vanh17]

Follow up on this,

sh preprocess.sh

The script assumes that the raw data has the following format

/

|-- train.src

|-- train.dst

|-- test.src

|-- test.dst

|-- valid.src

|-- valid.dst

.src files contain complex sentences and .dst files contain simple sentences.

Adapting this to my question, do you think I need to create the directory has train.src train.dst etc but then I can just make one difficult sentences file let's say train.src and leave the rest of the .src and .dst blank? Thank you!

[mounicam]

@vanh17, you should be able to generate simplified sentences without training the model using our pretrained models (Pretained models section in the README gives you instructions)

Yes, you can have dummy .src and .dst files for valid and train. Then, you can have your data in both test.src and test.dst for preprocessing.

Another option is to modify the preprocess.sh file. You can remove --trainpref and --validpref options in the preprocess line and remove the corresponding tokenization lines.

[vanh17]

Thank you @mounicam for the information! I created these dummies file in the raw files directory.
I installed TensorFlow and torch and I successfully installed pip install --editable . . However, when I ran preprocessing.sh I got this error:

Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess/anonymize_wordpiece.py", line 2, in <module>
    from wordpiece import FullTokenizer
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/preprocess/wordpiece.py", line 25, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'
Traceback (most recent call last):
  File "preprocess.py", line 15, in <module>
    from fairseq import options, tasks, utils
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/fairseq/__init__.py", line 11, in <module>
    import fairseq.criterions
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/fairseq/criterions/__init__.py", line 12, in <module>
    from fairseq.criterions.fairseq_criterion import FairseqCriterion
  File "/net/kate/storage/work/vnhh/wiki-auto/simplification/fairseq/criterions/fairseq_criterion.py", line 8, in <module>
    from torch.nn.modules.loss import _Loss
ModuleNotFoundError: No module named 'torch.nn'

For some reason, look like the word_piece.py file did not recognize the TensorFlow and torch modules available on my machine. To make sure that I have installed TensorFlow and torch correctly I checked and it worked when I imported TensorFlow and torch module on terminal python.

Do I have to do any additional setup on directory wiki-auto/simplification for it to fully work?
Attached is a screenshot to show that I have TensorFlow and torch modules installed successfully on the Linux python terminal

[mounicam]

The script uses the python version from the following path - "/usr/bin/python3". I forgot to change it to just "python3". You can change the python path and run it again.