Test all the doc snippets

[daspecster]

Per @dhermes and @tseaver's conversation.

Services with tested snippets

• Add validation script to check for untested snippets.
• Start running snippets during system tests.
• Bigquery (docs/bigquery_snippets.py) (partial)
• Bigtable
• Datastore
• DNS
• Error Reporting
• Language
• Logging
• Montoring
• Pubsub (docs/pubsub_snippets.py)
• Resource manager
• Speech
• Storage
• Translate
• Vision

[dhermes]

We may want a script which explicitly checks and (complains about / errors in the presence of) snippets in RST files or docstrings?

[daspecster]

@dhermes you mean check for unmarked snippets/code blocks? Can you clarify that?

Use system test environment?

I think @tseaver mentioned something about not wanting to write a lot of mocks for snippets?
With that in mind, couldn't we just test the snippets through the system tests? That seems like the most direct testing strategy.

[dhermes]

@daspecster Not sure what you mean by unmarked. I simply mean a script which makes sure all of our snippets must be valid and test-able. So if there is just some code-block somewhere without doctest, then we flag it as untested.

[daspecster]

@dhermes gotcha.

Snippets are referenced via...

.. literalinclude:: bigquery_snippets.py
   :start-after: [START client_list_datasets]
   :end-before: [END client_list_datasets]

So you could just check for .. code-block::'s or ::'s.
However I don't know if we would want to test things like...

.. code-block:: console

    $ pip install google-cloud

Could just look for .. code-block:: python I suppose.

[dhermes]

@daspecster Yeah I wasn't trying to spec out how the script would work, just trying to say we should have a check. Otherwise snippets may leak in. There might be a Sphinx extension to check for it already?

[daspecster]

Yeah sorry, probably the wrong place to spec it out. I added a checkbox for a validation script.

[dhermes]

Proof of concept: dhermes/ci-diff-helper@bfbff87

I did something similar in google-cloud-python to get an idea of the code-block's we have:
dhermes@f513290

(Worth noting that a literalinclude becomes a code block.)

[tseaver]

@dhermes Our goals for testing snippets inline:

• Snippets have no syntax errors. We could mock everything to satisfy this case.
• Snippets calls to each API match their signatures. Here, we would have to mock things at the connection / stub / helper level. We would still not catch stuff where the values passed in matched the names of expected arguments, but failed to marshal properly, or match what the back-end mandates in some other way.
• Snippets show actually correct processing of results as they would be returned by the API. Here we would be required to mock everything about the back-end for each API.

I don't believe we can reasonably expect to do the second or third options inside the test setup / teardown facilities provided by Sphinx. Exercising the snippets in a system-test-like environment, against the real APIs, seems like the only viable choice.

[dhermes]

@tseaver That's fine with me. (The thing I spec-ed out was really just to show how we could make sure the literal blocks adhere to whatever standards we desire. So if we wanted them all to be literalinclude we could check for that.)