'NoneType' object has no attribute 'argv' with nbconvert 5.4 and nbsphinx

[astrojuanlu]

Recent nbconvert release broke nbsphinx:

$ sphinx-build -vW -b html docs/source docs-html
Running Sphinx v1.7.9
loading pickled environment... failed: build environment version not current
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from http://docs.astropy.org/en/stable/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/reference/objects.inv...
loading intersphinx inventory from http://matplotlib.org/objects.inv...
intersphinx inventory has moved: http://matplotlib.org/objects.inv -> https://matplotlib.org/objects.inv
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 26 source files that are out of date
updating environment: 26 added, 0 changed, 0 removed
reading sources... [  3%] about
reading sources... [  7%] api
reading sources... [ 11%] changelog
reading sources... [ 15%] core
reading sources... [ 19%] dastcom5 parameters
reading sources... [ 23%] examples/Catch that asteroid!
reading sources... [ 26%] examples/Comparing Hohmann and bielliptic transfers

Traceback (most recent call last):

  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbsphinx.py", line 653, in parse
    rststring, resources = exporter.from_notebook_node(nb, resources)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbsphinx.py", line 575, in from_notebook_node
    nb, resources = pp.preprocess(nb, resources)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 359, in preprocess
    with self.setup_preprocessor(nb, resources, km=km):
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 304, in setup_preprocessor
    self.km, self.kc = self.start_new_kernel(cwd=path)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 253, in start_new_kernel
    km.start_kernel(extra_arguments=self.extra_arguments, **kwargs)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/jupyter_client/manager.py", line 246, in start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/jupyter_client/manager.py", line 170, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
AttributeError: 'NoneType' object has no attribute 'argv'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/cmdline.py", line 304, in main
    app.build(args.force_all, filenames)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/application.py", line 331, in build
    self.builder.build_update()
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 342, in build_update
    'out of date' % len(to_build))
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 355, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir, self.doctreedir))
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 568, in update
    self._read_serial(docnames, self.app)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 587, in _read_serial
    self.read_doc(docname, app)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/environment/__init__.py", line 662, in read_doc
    doctree = read_doc(self.app, self, self.doc2path(docname))
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/sphinx/io.py", line 302, in read_doc
    pub.publish()
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/docutils/core.py", line 217, in publish
    self.settings)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/docutils/readers/__init__.py", line 72, in read
    self.parse()
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/home/juanlu/.miniconda36/envs/_test36_pol/lib/python3.6/site-packages/nbsphinx.py", line 664, in parse
    str(e))
nbsphinx.NotebookError: AttributeError in examples/Comparing Hohmann and bielliptic transfers.ipynb:
'NoneType' object has no attribute 'argv'

Notebook error:
AttributeError in examples/Comparing Hohmann and bielliptic transfers.ipynb:
'NoneType' object has no attribute 'argv'

If I remove the notebooks and the references to nbsphinx everything works fine. If I downgrade to 5.3.1 and keep the notebooks, everything works too.

I report the issue here because I understand that projects that depend on nbconvert should not break from 5.3 to 5.4, but perhaps it's indeed an nbsphinx issue. cc @spatialaudio

Environment details below:

Details

$ conda info -a

     active environment : _test36_pol
    active env location : /home/juanlu/.miniconda36/envs/_test36_pol
            shell level : 1
       user config file : /home/juanlu/.condarc
 populated config files : /home/juanlu/.miniconda36/.condarc
          conda version : 4.5.11
    conda-build version : 3.12.1
         python version : 3.6.5.final.0
       base environment : /home/juanlu/.miniconda36  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/juanlu/.miniconda36/pkgs
                          /home/juanlu/.conda/pkgs
       envs directories : /home/juanlu/.miniconda36/envs
                          /home/juanlu/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.11 requests/2.18.4 CPython/3.6.5 Linux/4.15.0-33-generic linuxmint/18.3 glibc/2.23
                UID:GID : 1000:1000
             netrc file : /home/juanlu/.netrc
           offline mode : False

# conda environments:
#
base                     /home/juanlu/.miniconda36
_numba36_test            /home/juanlu/.miniconda36/envs/_numba36_test
_test27                  /home/juanlu/.miniconda36/envs/_test27
_test36                  /home/juanlu/.miniconda36/envs/_test36
_test36_2                /home/juanlu/.miniconda36/envs/_test36_2
_test36_pol           *  /home/juanlu/.miniconda36/envs/_test36_pol
_testpl27                /home/juanlu/.miniconda36/envs/_testpl27
_testpl36                /home/juanlu/.miniconda36/envs/_testpl36
api36                    /home/juanlu/.miniconda36/envs/api36
astropy36_dev            /home/juanlu/.miniconda36/envs/astropy36_dev
brte36                   /home/juanlu/.miniconda36/envs/brte36
brte36dsk18              /home/juanlu/.miniconda36/envs/brte36dsk18
geostore36               /home/juanlu/.miniconda36/envs/geostore36
gmat36                   /home/juanlu/.miniconda36/envs/gmat36
marvin34                 /home/juanlu/.miniconda36/envs/marvin34
mbd36                    /home/juanlu/.miniconda36/envs/mbd36
pgadmin4                 /home/juanlu/.miniconda36/envs/pgadmin4
poliastro36              /home/juanlu/.miniconda36/envs/poliastro36
poliastro37              /home/juanlu/.miniconda36/envs/poliastro37
py34                     /home/juanlu/.miniconda36/envs/py34
py36                     /home/juanlu/.miniconda36/envs/py36
                         /home/juanlu/.miniconda36/envs/py36/conda-bld/cspice_1529866334362/_build_env
                         /home/juanlu/.miniconda36/envs/py36/conda-bld/cspice_1529866449168/_build_env
                         /home/juanlu/.miniconda36/envs/py36/conda-bld/cspice_1529866543542/_build_env
py37                     /home/juanlu/.miniconda36/envs/py37
py37_def                 /home/juanlu/.miniconda36/envs/py37_def
pypiserver36             /home/juanlu/.miniconda36/envs/pypiserver36
pyrenfe36                /home/juanlu/.miniconda36/envs/pyrenfe36
spice37                  /home/juanlu/.miniconda36/envs/spice37
telluric36               /home/juanlu/.miniconda36/envs/telluric36
unitshack36              /home/juanlu/.miniconda36/envs/unitshack36
                         /home/juanlu/Development/poliastro/benchmarks/env/1e6a43cbb25dbca4e803dce4b31b80e5
                         /home/juanlu/Development/poliastro/benchmarks/env/4ef9678914ff6e01972901ec0de8ad50

sys.version: 3.6.5 | packaged by conda-forge | (defau...
sys.prefix: /home/juanlu/.miniconda36
sys.executable: /home/juanlu/.miniconda36/bin/python
conda location: /home/juanlu/.miniconda36/lib/python3.6/site-packages/conda
conda-build: /home/juanlu/.miniconda36/bin/conda-build
conda-convert: /home/juanlu/.miniconda36/bin/conda-convert
conda-develop: /home/juanlu/.miniconda36/bin/conda-develop
conda-env: /home/juanlu/.miniconda36/bin/conda-env
conda-index: /home/juanlu/.miniconda36/bin/conda-index
conda-inspect: /home/juanlu/.miniconda36/bin/conda-inspect
conda-metapackage: /home/juanlu/.miniconda36/bin/conda-metapackage
conda-render: /home/juanlu/.miniconda36/bin/conda-render
conda-skeleton: /home/juanlu/.miniconda36/bin/conda-skeleton
conda-smithy: /home/juanlu/.miniconda36/bin/conda-smithy
conda-verify: /home/juanlu/.miniconda36/bin/conda-verify
user site dirs: 

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: _test36_pol
CONDA_EXE: /home/juanlu/.miniconda36/bin/conda
CONDA_PREFIX: /home/juanlu/.miniconda36/envs/_test36_pol
CONDA_PROMPT_MODIFIER: (_test36_pol) 
CONDA_PYTHON_EXE: /home/juanlu/.miniconda36/bin/python
CONDA_ROOT: /home/juanlu/.miniconda36
CONDA_SHLVL: 1
DEFAULTS_PATH: /usr/share/gconf/cinnamon.default.path
MANDATORY_PATH: /usr/share/gconf/cinnamon.mandatory.path
PATH: /home/juanlu/.miniconda36/envs/_test36_pol/bin:/home/juanlu/.rbenv/bin:$/home/juanlu/.rbenv/shims:/home/juanlu/.gem/ruby/2.3.0/bin:/home/juanlu/.local/share/umake/bin:/home/juanlu/bin:/home/juanlu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
WINDOWPATH: 8


WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license
$ conda list
# packages in environment at /home/juanlu/.miniconda36/envs/_test36_pol:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.11                    <pip>
asn1crypto                0.24.0                    <pip>
astropy                   3.0.4                     <pip>
astroquery                0.3.8                     <pip>
atomicwrites              1.2.1                     <pip>
attrs                     18.2.0                    <pip>
Babel                     2.6.0                     <pip>
backcall                  0.1.0                     <pip>
beautifulsoup4            4.6.3                     <pip>
bleach                    2.1.4                     <pip>
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.8.24            ha4d7672_0    conda-forge
certifi                   2018.8.24                py36_1    conda-forge
cffi                      1.11.5                    <pip>
chardet                   3.0.4                     <pip>
coverage                  4.5.1                     <pip>
cryptography              2.3.1                     <pip>
cycler                    0.10.0                    <pip>
decorator                 4.3.0                     <pip>
defusedxml                0.5.0                     <pip>
docutils                  0.14                      <pip>
entrypoints               0.2.3                     <pip>
gmp                       6.1.2                hfc679d8_0    conda-forge
html5lib                  1.0.1                     <pip>
idna                      2.7                       <pip>
imagesize                 1.1.0                     <pip>
ipykernel                 4.9.0                     <pip>
ipython                   6.5.0                     <pip>
ipython-genutils          0.2.0                     <pip>
ipywidgets                7.4.1                     <pip>
jedi                      0.12.1                    <pip>
jeepney                   0.3.1                     <pip>
Jinja2                    2.10                      <pip>
jplephem                  2.8                       <pip>
jsonschema                2.6.0                     <pip>
jupyter-client            5.2.3                     <pip>
jupyter-core              4.4.0                     <pip>
keyring                   15.0.0                    <pip>
kiwisolver                1.0.1                     <pip>
libffi                    3.2.1                hfc679d8_5    conda-forge
libgcc-ng                 7.2.0                hdf63c60_3    conda-forge
libstdcxx-ng              7.2.0                hdf63c60_3    conda-forge
llvmlite                  0.24.0                    <pip>
MarkupSafe                1.0                       <pip>
matplotlib                2.2.3                     <pip>
mistune                   0.8.3                     <pip>
more-itertools            4.3.0                     <pip>
nbconvert                 5.4.0                     <pip>
nbformat                  4.4.0                     <pip>
ncurses                   6.1                  hfc679d8_1    conda-forge
notebook                  5.6.0                     <pip>
numba                     0.39.0                    <pip>
numpy                     1.15.1                    <pip>
openssl                   1.0.2p               h470a237_0    conda-forge
packaging                 17.1                      <pip>
pandas                    0.23.4                    <pip>
pandoc                    1.19.2                        0    conda-forge
pandocfilters             1.4.2                     <pip>
parso                     0.3.1                     <pip>
pexpect                   4.6.0                     <pip>
pickleshare               0.7.4                     <pip>
pip                       18.0                     py36_1    conda-forge
plotly                    3.2.0                     <pip>
pluggy                    0.7.1                     <pip>
poliastro                 0.11.dev0                 <pip>
prometheus-client         0.3.1                     <pip>
prompt-toolkit            1.0.15                    <pip>
ptyprocess                0.6.0                     <pip>
py                        1.6.0                     <pip>
pycodestyle               2.4.0                     <pip>
pycparser                 2.18                      <pip>
Pygments                  2.2.0                     <pip>
pyparsing                 2.2.0                     <pip>
pytest                    3.8.0                     <pip>
pytest-cov                2.6.0                     <pip>
python                    3.6.6                h5001a0f_0    conda-forge
python-dateutil           2.7.3                     <pip>
pytz                      2018.5                    <pip>
pyzmq                     17.1.2                    <pip>
readline                  7.0                  haf1bffa_1    conda-forge
requests                  2.19.1                    <pip>
retrying                  1.3.3                     <pip>
scipy                     1.1.0                     <pip>
SecretStorage             3.1.0                     <pip>
Send2Trash                1.5.0                     <pip>
setuptools                40.2.0                   py36_0    conda-forge
simplegeneric             0.8.1                     <pip>
six                       1.11.0                    <pip>
snowballstemmer           1.2.1                     <pip>
Sphinx                    1.7.9                     <pip>
sphinx-rtd-theme          0.4.1                     <pip>
sphinxcontrib-websupport  1.1.0                     <pip>
sqlite                    3.24.0               h2f33b56_1    conda-forge
terminado                 0.8.1                     <pip>
testpath                  0.3.1                     <pip>
tk                        8.6.8                         0    conda-forge
tornado                   5.1                       <pip>
traitlets                 4.3.2                     <pip>
urllib3                   1.23                      <pip>
wcwidth                   0.1.7                     <pip>
webencodings              0.5.1                     <pip>
wheel                     0.31.1                   py36_1    conda-forge
widgetsnbextension        3.4.1                     <pip>
xz                        5.2.4                h470a237_1    conda-forge
zlib                      1.2.11               h470a237_3    conda-forge

[akhmerov]

Can you try to narrow down the issue? From looking at the code I see that:

• The failure is because the execute preprocessor is not instantiated with a kernel spec
• nbsphinx has a default kernel name "", which should make execute preprocessor look into what's stored in the notebook
• the notebook has the kernel correctly specified

Therefore it appears that the problem shouldn't be happening. Also looking at the changes in the ExecutePreprocessor I cannot figure out what could be triggering the problem.

How are you calling nbsphinx? Can you figure out what part of the code specifies the kernel spec to be equal to None?

[astrojuanlu]

Sorry, I meant to ping @mgeier above.

Thanks for your quick answer @akhmerov! I am "just" a nbsphinx user so I don't know those details right away. If Matthias doesn't chime in, I will try to lend a hand in about a couple of weeks.

[mgeier]

As a work-around you could try setting this:

nbsphinx_kernel_name = 'python3'

But normally, this shouldn't be necessary.

[mgeier]

I just updated my nbconvert installation and now I'm also getting the error!

I bisected into it and found that the error was introduced in #816, and more exactly in commit c1cb89c, committted by @mpacer.

[mpacer]

I think this may be due to nbspbinx specifying the kernel directly by setting it equal to “”. That would override the default value (where the kernel name is now being grabbed from the notebook dynamically) and as a result it never hits the code that would have grabbed it from the notebook.

We can add that behaviour back in since we had it before. But it does seem like a previous anti pattern to require specifying an option in order to trigger what should be default behaviour (adhering to the notebook’s kernel name).

This is speculation based on looking at the code based I’ll need to test this out in a bit.

[mgeier]

I've disabled nbconvert release 5.4 in the latest nbsphinx release, so updating nbsphinx should avoid the error for now.

I hope the actual problem will be fixed in #886 before the next nbconvert release.