ProgrammingInPython/modules/Testing_advanced.html at gh-pages · UWPCE-PythonCert/ProgrammingInPython

872 lines (820 loc) · 72.7 KB
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Advanced Testing &mdash; Programming in Python 7.0 documentation</title>
      <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
      <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <!--[if lt IE 9]>
    <script src="../_static/js/html5shiv.min.js"></script>
  <![endif]-->
        <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
        <script src="../_static/jquery.js"></script>
        <script src="../_static/underscore.js"></script>
        <script src="../_static/doctools.js"></script>
    <script src="../_static/js/theme.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="Mocking Mailroom" href="../exercises/mailroom/mailroom-mock.html" />
    <link rel="prev" title="18. Advanced Testing" href="../topics/18-advanced_testing/index.html" /> 
<body class="wy-body-for-nav"> 
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #4b2e83" >
            <a href="../index.html">
            <img src="../_static/UWPCE_logo_full.png" class="logo" alt="Logo"/>
          </a>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
        </div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
              <p class="caption" role="heading"><span class="caption-text">Topics in the Program</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../topics/01-setting_up/index.html">1. Setting up your Environment</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/02-basic_python/index.html">2. Basic Python</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/03-recursion_booleans/index.html">3. Booleans and Recursion</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/04-sequences_iteration/index.html">4. Sequences and Iteration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/05-text_handling/index.html">5. Basic Text Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/06-exceptions/index.html">6. Exception Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/07-unit_testing/index.html">7. Unit Testing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/08-dicts_sets/index.html">8. Dictionaries and Sets</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/09-files/index.html">9. File Handling</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/10-modules_packages/index.html">10. Modules and Packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/11-argument_passing/index.html">11. Advanced Argument Passing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/12-comprehensions/index.html">12. Comprehensions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/13-intro_oo/index.html">13. Intro to Object Oriented Programing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/14-magic_methods/index.html">14. Properties and Magic Methods</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/15-subclassing/index.html">15. Subclassing and Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/16-multiple_inheritance/index.html">16. Multiple Inheritance</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/17-functional_programming/index.html">17. Introduction to Functional Programming</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../topics/18-advanced_testing/index.html">18. Advanced Testing</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">Advanced Testing</a></li>
<li class="toctree-l2"><a class="reference internal" href="../exercises/mailroom/mailroom-mock.html">Mocking Mailroom</a></li>
<li class="toctree-l1"><a class="reference internal" href="../topics/99-extras/index.html">19. Extra Topics</a></li>
        </div>
      </div>
    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu"  style="background: #4b2e83" >
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">Programming in Python</a>
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content style-external-links">
          <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
          <li><a href="../topics/18-advanced_testing/index.html"><span class="section-number">18. </span>Advanced Testing</a> &raquo;</li>
      <li>Advanced Testing</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/modules/Testing_advanced.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul><div class="rst-breadcrumbs-buttons" role="navigation" aria-label="Sequential page navigation">
        <a href="../topics/18-advanced_testing/index.html" class="btn btn-neutral float-left" title="18. Advanced Testing" accesskey="p"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../exercises/mailroom/mailroom-mock.html" class="btn btn-neutral float-right" title="Mocking Mailroom" accesskey="n">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
  <div class="section" id="advanced-testing">
<span id="id1"></span><h1>Advanced Testing<a class="headerlink" href="#advanced-testing" title="Permalink to this headline"></a></h1>
<p>Testing in Python</p>
<div class="section" id="what-is-testing">
<h2>What is testing?<a class="headerlink" href="#what-is-testing" title="Permalink to this headline"></a></h2>
<p>Code which runs your application in as close to a real environment as
feasible and validates its behavior</p>
<div class="section" id="terminology-of-testing">
<h3>Terminology of testing<a class="headerlink" href="#terminology-of-testing" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Unit tests</p></li>
<li><p>Integration tests</p></li>
<li><p>High level system tests</p></li>
<li><p>Acceptance tests</p></li>
<li><p>Black box / White box testing</p></li>
<div class="section" id="v-model-and-tests-levels">
<h3>“V” model and tests levels<a class="headerlink" href="#v-model-and-tests-levels" title="Permalink to this headline"></a></h3>
<img alt="../_images/test_v_model.png" src="../_images/test_v_model.png" />
<div class="section" id="unit-testing">
<h3>Unit testing<a class="headerlink" href="#unit-testing" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Test smallest discrete units of source code</p></li>
<li><p>Tests should be independent of each other</p></li>
<li><p>Can separate tests from required resources through fixtures and
mocking</p></li>
<li><p>Automatable</p></li>
<li><p>Integrates with the development process</p></li>
<div class="section" id="what-should-be-tested">
<h3>What should be tested?<a class="headerlink" href="#what-should-be-tested" title="Permalink to this headline"></a></h3>
<p>The percentage of code which gets run in a test is known as the
“coverage”.</p>
<p>100% coverage is an ideal to strive for. But the decision on when and
what to test should take into account the volatility of the project.</p>
<p><strong>NOTE</strong> Even if every line of code is run during tests (100% coverage),
they may not be comprehensive! It is very hard to anticipate every weird
input some code may get.</p>
<div class="section" id="unit-testing-tools">
<h3>Unit Testing tools<a class="headerlink" href="#unit-testing-tools" title="Permalink to this headline"></a></h3>
<li><p>unittest, the test framework that ships with Python. Port of Java jUnit</p>
<p><a class="reference external" href="http://docs.python.org/3/library/unittest.html">http://docs.python.org/3/library/unittest.html</a></p>
<li><p>pytest, a test runner, and also an alternative to unittest, which you should be pretty familiar with now</p>
<p><a class="reference external" href="http://pytest.org/latest/">http://pytest.org/latest/</a></p>
<li><p>mock, an object mocking library. Ships with Python 3.3+</p>
<p><a class="reference external" href="https://docs.python.org/dev/library/unittest.mock.html">https://docs.python.org/dev/library/unittest.mock.html</a></p>
<p>Note that while mock is in the <code class="docutils literal notranslate"><span class="pre">unittest</span></code> package, you do not need to be using <code class="docutils literal notranslate"><span class="pre">unittest</span></code> tests to use it.</p>
<div class="section" id="about-unit-testing">
<h3>About Unit Testing<a class="headerlink" href="#about-unit-testing" title="Permalink to this headline"></a></h3>
<ol class="arabic simple">
<li><p>Tests should be independent.</p></li>
<li><p>Tests do not run in order, which shouldn’t matter, see point 1.</p></li>
<li><p>Test fixtures are available to do any setup / teardown needed for tests.</p></li>
<li><p>Test behavior not implementation dependent.</p></li>
<li><p>Mocking is available to fake stuff you may not want to run in your tests.</p></li>
<p>This all applies regardless of your test framework</p>
<div class="section" id="unittest">
<h3>unittest<a class="headerlink" href="#unittest" title="Permalink to this headline"></a></h3>
<p>The unittest framework comes with the standard library</p>
<p>Unittest is ported from Java’s jUnit – it is therefore OO-heavy, and
requires a lot of boilerplate code.</p>
<p>Many projects built custom testing Frameworks on top of it – e.g. Django</p>
<p>Therefore you will encounter it</p>
<p>So it’s good to be familiar with it.</p>
<p>Key missing features:</p>
<blockquote>
<div><ul class="simple">
<li><p>A test runner</p>
<li><p>many people use nose or pytest to run unittest tests.</p></li>
<li><p>Parameterized tests</p>
<li><p>there are kludges and some third-party tools for this.</p></li>
</div></blockquote>
<div class="section" id="unittest-testcase-anatomy">
<h3>unittest.TestCase anatomy<a class="headerlink" href="#unittest-testcase-anatomy" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>create a new subclass of <code class="docutils literal notranslate"><span class="pre">unittest.TestCase</span></code></p></li>
<li><p>name test methods <code class="docutils literal notranslate"><span class="pre">test_foo</span></code> so the test runner finds them</p></li>
<li><p>make calls to the <code class="docutils literal notranslate"><span class="pre">self.assert*</span></code> family of methods to validate results</p></li>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="k">class</span> <span class="nc">TestMyStuff</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">x</span> <span class="o">=</span> <span class="mi">2</span>
    <span class="k">def</span> <span class="nf">test_add</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">x</span><span class="o">+</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">test_len</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="s1">&#39;foo&#39;</span><span class="p">),</span> <span class="mi">3</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
    <span class="n">unittest</span><span class="o">.</span><span class="n">main</span><span class="p">()</span>
</pre></div>
<div class="section" id="assert-methods">
<h3>Assert Methods<a class="headerlink" href="#assert-methods" title="Permalink to this headline"></a></h3>
<p>TestCase contains a number of methods named <code class="docutils literal notranslate"><span class="pre">assert*</span></code> which can be used
for validation, here are a few common ones:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">assertEqual</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="n">assertNotEqual</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="n">assertTrue</span><span class="p">(</span><span class="n">expr</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="n">assertFalse</span><span class="p">(</span><span class="n">expr</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
<span class="n">assertIn</span><span class="p">(</span><span class="n">first</span><span class="p">,</span> <span class="n">second</span><span class="p">)</span>
<span class="n">assertRaises</span><span class="p">(</span><span class="n">exc</span><span class="p">,</span> <span class="n">fun</span><span class="p">,</span> <span class="n">msg</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
</pre></div>
<p>See a full list at:</p>
<p><a class="reference external" href="http://docs.python.org/3/library/unittest.html#assert-methods">http://docs.python.org/3/library/unittest.html#assert-methods</a> or</p>
<p><code class="docutils literal notranslate"><span class="pre">dir(unittest.TestCase)</span></code> or to get really fancy</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="p">[</span><span class="nb">print</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">dir</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">)</span> <span class="k">if</span> <span class="n">i</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;assert&#39;</span><span class="p">)]</span>
</pre></div>
<div class="section" id="running-your-tests">
<h2>Running your tests<a class="headerlink" href="#running-your-tests" title="Permalink to this headline"></a></h2>
<p>How do you actually run your tests?</p>
<div class="section" id="running-tests-in-a-single-module">
<h3>running tests in a single module<a class="headerlink" href="#running-tests-in-a-single-module" title="Permalink to this headline"></a></h3>
<p>Call unittest.main() right in your module</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;__main__&quot;</span><span class="p">:</span>
    <span class="n">unittest</span><span class="o">.</span><span class="n">main</span><span class="p">()</span>
</pre></div>
<p>or from the command line:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>python -m unittest test_my_module  <span class="c1"># with or without .py on end</span>
python -m unittest test_my_module.TestClass  <span class="c1"># particular class in a module</span>
python -m unittest test_my_module.TestClass.test_method  <span class="c1"># particular test</span>
</pre></div>
<p>If it gets cumbersome with many TestCases, organize the tests into a
test suite (or use a test runner, which we get to soon).</p>
<div class="section" id="test-suites">
<h3>Test Suites<a class="headerlink" href="#test-suites" title="Permalink to this headline"></a></h3>
<p>Test suites group test cases into a single testable unit</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">calculator_test</span> <span class="kn">import</span> <span class="n">TestCalculatorFunctions</span>
<span class="n">suite</span> <span class="o">=</span> <span class="n">unittest</span><span class="o">.</span><span class="n">TestLoader</span><span class="p">()</span><span class="o">.</span><span class="n">loadTestsFromTestCase</span><span class="p">(</span><span class="n">TestCalculatorFunctions</span><span class="p">)</span>
<span class="n">unittest</span><span class="o">.</span><span class="n">TextTestRunner</span><span class="p">(</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">suite</span><span class="p">)</span>
</pre></div>
<p>Tests can also be organized into suites in the</p>
<p><code class="docutils literal notranslate"><span class="pre">if</span> <span class="pre">__name__</span> <span class="pre">==</span> <span class="pre">&quot;__main__&quot;:</span></code></p>
<p>block</p>
<div class="section" id="testrunners-pytest-and-nose2">
<h3>TestRunners: pytest and Nose2<a class="headerlink" href="#testrunners-pytest-and-nose2" title="Permalink to this headline"></a></h3>
<p>Nose2 is the new nose. Nose is no longer maintained, and directs users to nose2.</p>
<p>Both pytest and Nose2 are test runners: they auto-discover test cases.</p>
<p>They will find tests for you so you can focus on writing tests, not
maintaining test suites.</p>
<p>To find tests, pytest and nose look for modules (such as python files)
whose names start with ‘test’. In those modules, they will load tests
from all unittest.TestCase subclasses, as well as functions whose names
start with ‘test’.</p>
<p>So running your tests is as easy as</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pytest
</pre></div>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ nose2
</pre></div>
<p><a class="reference external" href="http://nose2.readthedocs.org/en/latest/getting_started.html#running-tests">http://nose2.readthedocs.org/en/latest/getting_started.html#running-tests</a></p>
<p><a class="reference external" href="https://docs.pytest.org/en/latest/index.html">https://docs.pytest.org/en/latest/index.html</a></p>
<p>A number of projects use nose – so you may encounter it, but we’ll focus
on pytest for now.</p>
<div class="section" id="fixtures-setting-up-your-tests-for-success">
<h3>Fixtures: Setting up your tests for success<a class="headerlink" href="#fixtures-setting-up-your-tests-for-success" title="Permalink to this headline"></a></h3>
<p>(or failure!)</p>
<p>Test fixtures are a fixed baseline for tests to run from consistently,
also known as test context.</p>
<p>Fixtures can (and should) be set up fresh before each test, once before each test
case, or before an entire test suite.</p>
<div class="section" id="fixtures-in-unittest">
<h3>Fixtures in unittest<a class="headerlink" href="#fixtures-in-unittest" title="Permalink to this headline"></a></h3>
<p>unittest provides fixture support via these methods:</p>
<ul class="simple">
<li><p>setUp / tearDown - these are run before and after each test method</p></li>
<li><p>setUpClass / tearDownClass - these are run before/after each TestCase</p></li>
<li><p>setUpModule / tearDownModule - run before/after each TestSuite</p></li>
<li><p>addCleanup / doCleanups - called after tearDown,
in case a test throws an exception</p></li>
<div class="section" id="fixtures-in-pytest">
<h3>Fixtures in pytest<a class="headerlink" href="#fixtures-in-pytest" title="Permalink to this headline"></a></h3>
<p>pytest provides a fixture system that is powerful and flexible:</p>
<p><a class="reference external" href="https://docs.pytest.org/en/latest/fixture.html#fixture">https://docs.pytest.org/en/latest/fixture.html#fixture</a></p>
<p>You use a decorator to create a fixture:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pytest</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">smtp</span><span class="p">():</span>
    <span class="kn">import</span> <span class="nn">smtplib</span>
    <span class="k">return</span> <span class="n">smtplib</span><span class="o">.</span><span class="n">SMTP</span><span class="p">(</span><span class="s2">&quot;smtp.gmail.com&quot;</span><span class="p">)</span>
</pre></div>
<p>A fixture is simply a function that will get run when it is used, and
returns <em>something</em> that your tests need:</p>
<p>To use a fixture, you add it as a parameter to your test function:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_ehlo</span><span class="p">(</span><span class="n">smtp</span><span class="p">):</span>
    <span class="n">response</span><span class="p">,</span> <span class="n">msg</span> <span class="o">=</span> <span class="n">smtp</span><span class="o">.</span><span class="n">ehlo</span><span class="p">()</span>
    <span class="k">assert</span> <span class="n">response</span> <span class="o">==</span> <span class="mi">250</span>
    <span class="k">assert</span> <span class="mi">0</span> <span class="c1"># for demo purposes</span>
</pre></div>
<p>The parameter gets set to the value returned by the fixture function.
The fixture function is automatically run before each test.</p>
<p>Let’s see this in action:</p>
<p><a class="reference download internal" download="" href="../_downloads/cd1f82fcc00d02d1af7aae379d84f37e/pytest_fixtures.py"><code class="xref download docutils literal notranslate"><span class="pre">pytest_fixtures.py</span></code></a></p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pytest -s -v pytest_fixtures.py
</pre></div>
<p>The <code class="docutils literal notranslate"><span class="pre">-s</span></code> tells pytest not to capture stdout – so we can see print statements.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">-v</span></code> is verbose mode – so we can see a bit more what is going on.</p>
<div class="section" id="teardown">
<h3>“Teardown”<a class="headerlink" href="#teardown" title="Permalink to this headline"></a></h3>
<p>If your fixture needs to clean itself up after its done, this is known as
“teardown”</p>
<p>To accomplish this in pytest, you use “yield”, rather than “return”.</p>
<p>The teardown code will run after the yield</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@pytest</span><span class="o">.</span><span class="n">fixture</span>
<span class="k">def</span> <span class="nf">smtp</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
    <span class="n">smtp</span> <span class="o">=</span> <span class="n">smtplib</span><span class="o">.</span><span class="n">SMTP</span><span class="p">(</span><span class="s2">&quot;smtp.gmail.com&quot;</span><span class="p">)</span>
    <span class="k">yield</span> <span class="n">smtp</span>  <span class="c1"># provide the fixture value</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;teardown smtp&quot;</span><span class="p">)</span>
    <span class="n">smtp</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
<p>Remember that putting a yield in a function makes it a generator function – which provides a way to halt execution of the function, return a value, and then pick up where it left off. So in this case, you use whatever code you want to generate your object – then after the yield, all those variables will be there, so you can do whatever clean up you need to do.</p>
<p>See the example again for this…</p>
<div class="section" id="testing-floating-point-values">
<h2>Testing floating point values<a class="headerlink" href="#testing-floating-point-values" title="Permalink to this headline"></a></h2>
<p>Why can’t we just test if .5 == 1/2 ?</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [1]: </span><span class="mi">3</span> <span class="o">*</span> <span class="mf">.15</span> <span class="o">==</span> <span class="mf">.45</span>
<span class="gh">Out[1]: </span><span class="go">False</span>
<span class="gp">In [2]: </span><span class="mi">3</span> <span class="o">*</span> <span class="mf">.15</span>
<span class="gh">Out[2]: </span><span class="go">0.44999999999999996</span>
<span class="gp">In [3]: </span><span class="mi">3</span> <span class="o">*</span> <span class="mf">.15</span> <span class="o">*</span> <span class="mi">10</span> <span class="o">/</span> <span class="mi">10</span>  <span class="o">==</span> <span class="mf">.45</span>
<span class="gh">Out[3]: </span><span class="go">True</span>
</pre></div>
<p>There are an infinite number of real numbers, so they are
stored as an approximation in computing hardware.</p>
<p><a class="reference external" href="https://docs.python.org/3/tutorial/floatingpoint.html">https://docs.python.org/3/tutorial/floatingpoint.html</a></p>
<div class="section" id="levels-of-precision-of-floating-point">
<h3>Levels of precision of floating point<a class="headerlink" href="#levels-of-precision-of-floating-point" title="Permalink to this headline"></a></h3>
<p>Python floating point numbers are stored in <a class="reference external" href="http://en.wikipedia.org/wiki/IEEE_floating_point">IEEE 754</a> 64-bit double precision format, so 1 bit for the sign, 11 bits for the exponent, and the remaining 52 for the fraction.</p>
<p>So we can count on up to about 16 digits of precision in decimal:</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [39]: </span><span class="nb">len</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">52</span><span class="p">))</span>
<span class="gh">Out[39]: </span><span class="go">16</span>
<span class="gp">In [40]: </span><span class="mf">.1</span><span class="o">+</span><span class="mf">.2</span>
<span class="gh">Out[40]: </span><span class="go">0.30000000000000004</span>
<span class="gp">In [41]: </span><span class="nb">len</span><span class="p">(</span><span class="s1">&#39;3000000000000000&#39;</span><span class="p">)</span>
<span class="gh">Out[41]: </span><span class="go">16</span>
<span class="go"># with repeated operations, the errors eventually build up:</span>
<span class="go"># here&#39;s multiplying by &quot;1&quot; 10 million times:</span>
<span class="gp">In [64]: </span><span class="n">x</span><span class="o">=</span><span class="mi">1</span>
<span class="gp">In [69]: </span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10000000</span><span class="p">):</span> <span class="n">x</span> <span class="o">*=</span> <span class="p">(</span><span class="mf">.1</span> <span class="o">+</span> <span class="mf">.2</span><span class="p">)</span><span class="o">/</span><span class="mf">.3</span>
<span class="go">Out [69]: 1.000000002220446</span>
</pre></div>
<div class="section" id="assertalmostequal">
<h3>assertAlmostEqual<a class="headerlink" href="#assertalmostequal" title="Permalink to this headline"></a></h3>
<p>assertAlmostEqual is a custom assert in <code class="docutils literal notranslate"><span class="pre">unittest</span></code> that verifies that two floating point values are close enough to each other.</p>
<p>Add a places keyword argument to specify the number of decimal places.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="k">class</span> <span class="nc">TestAlmostEqual</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">setUp</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">pass</span>
    <span class="k">def</span> <span class="nf">test_floating_point</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="mf">.15</span><span class="p">,</span> <span class="mf">.45</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">test_almost_equal</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertAlmostEqual</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="mf">.15</span><span class="p">,</span> <span class="mf">.45</span><span class="p">,</span> <span class="n">places</span><span class="o">=</span><span class="mi">7</span><span class="p">)</span>
</pre></div>
<div class="section" id="what-is-close-enough">
<h3>What is close enough?<a class="headerlink" href="#what-is-close-enough" title="Permalink to this headline"></a></h3>
<p><strong>Warning</strong></p>
<p><code class="docutils literal notranslate"><span class="pre">assertAlmostEqual</span></code> lets you specify <em>decimal places</em>, i.e. the number of digits after the decimal point.</p>
<p>This works great for numbers that are about magnitude 1.0 (as above)</p>
<p>But what if you have numbers that are very large? (or small):</p>
<blockquote>
<div><ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">1.0e22</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">1.0000000000001e22</span></code></p></li>
</div></blockquote>
<p>are they almost equal?</p>
<p>Remember that python floating point numbers store the exponent and up
to 16 decimal digits.</p>
<p>So those two are almost as close as you can get. But:</p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [30]: </span><span class="n">x</span> <span class="o">=</span> <span class="mf">1e22</span>
<span class="gp">In [31]: </span><span class="n">y</span> <span class="o">=</span> <span class="mf">1.0000000000001e22</span>
<span class="gp">In [32]: </span><span class="s1">&#39;</span><span class="si">%g</span><span class="s1">&#39;</span><span class="o">%</span><span class="p">(</span><span class="n">y</span> <span class="o">-</span> <span class="n">x</span><span class="p">)</span>
<span class="gh">Out[32]: </span><span class="go">&#39;1.00034e+09&#39;</span>
</pre></div>
<p>They are different by about a billion!</p>
<p>In general, we don’t want to compare floating point numbers to within a
certain number of decimal places.</p>
<p>Anyone remember “significant figures” from science classes?</p>
<div class="section" id="isclose">
<h3><code class="docutils literal notranslate"><span class="pre">isclose()</span></code><a class="headerlink" href="#isclose" title="Permalink to this headline"></a></h3>
<p>Python 3.5 introduced the <code class="docutils literal notranslate"><span class="pre">isclose()</span></code> function in the <code class="docutils literal notranslate"><span class="pre">math</span></code> module:</p>
<p><a class="reference external" href="https://www.python.org/dev/peps/pep-0485/">https://www.python.org/dev/peps/pep-0485/</a></p>
<div class="highlight-ipython notranslate"><div class="highlight"><pre><span></span><span class="gp">In [39]: </span><span class="kn">import</span> <span class="nn">math</span>
<span class="gp">In [40]: </span><span class="n">x</span>
<span class="gh">Out[40]: </span><span class="go">1e+22</span>
<span class="gp">In [41]: </span><span class="n">y</span>
<span class="gh">Out[41]: </span><span class="go">1.0000000000001e+22</span>
<span class="gp">In [42]: </span><span class="n">math</span><span class="o">.</span><span class="n">isclose</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="gh">Out[42]: </span><span class="go">True</span>
</pre></div>
<p>So this works for any magnitude number.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">is_close</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">rel_tol</span><span class="o">=</span><span class="mf">1e-09</span><span class="p">,</span> <span class="n">abs_tol</span><span class="o">=</span><span class="mf">0.0</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span>
<span class="n">Determine</span> <span class="n">whether</span> <span class="n">two</span> <span class="n">floating</span> <span class="n">point</span> <span class="n">numbers</span> <span class="n">are</span> <span class="n">close</span> <span class="ow">in</span> <span class="n">value</span><span class="o">.</span>
   <span class="n">rel_tol</span>
       <span class="n">maximum</span> <span class="n">difference</span> <span class="k">for</span> <span class="n">being</span> <span class="n">considered</span> <span class="s2">&quot;close&quot;</span><span class="p">,</span> <span class="n">relative</span> <span class="n">to</span> <span class="n">the</span>
       <span class="n">magnitude</span> <span class="n">of</span> <span class="n">the</span> <span class="nb">input</span> <span class="n">values</span>
    <span class="n">abs_tol</span>
       <span class="n">maximum</span> <span class="n">difference</span> <span class="k">for</span> <span class="n">being</span> <span class="n">considered</span> <span class="s2">&quot;close&quot;</span><span class="p">,</span> <span class="n">regardless</span> <span class="n">of</span> <span class="n">the</span>
       <span class="n">magnitude</span> <span class="n">of</span> <span class="n">the</span> <span class="nb">input</span> <span class="n">values</span>
<span class="n">Return</span> <span class="kc">True</span> <span class="k">if</span> <span class="n">a</span> <span class="ow">is</span> <span class="n">close</span> <span class="ow">in</span> <span class="n">value</span> <span class="n">to</span> <span class="n">b</span><span class="p">,</span> <span class="ow">and</span> <span class="kc">False</span> <span class="n">otherwise</span><span class="o">.</span>
</pre></div>
<p><code class="docutils literal notranslate"><span class="pre">rel_tol</span></code> essentially specifies how many significant figures you want:
<code class="docutils literal notranslate"><span class="pre">1e-09</span></code> is 9 significant figures: about half of what floats can store.</p>
<p><code class="docutils literal notranslate"><span class="pre">abs_tol</span></code> is required for comparisons to zero – nothing is
“relatively close” to zero</p>
<div class="section" id="using-isclose-with-unittest">
<h3>Using <code class="docutils literal notranslate"><span class="pre">isclose()</span></code> with <code class="docutils literal notranslate"><span class="pre">unittest</span></code><a class="headerlink" href="#using-isclose-with-unittest" title="Permalink to this headline"></a></h3>
<p>Ideally, <code class="docutils literal notranslate"><span class="pre">TestCase</span></code> would have an <code class="docutils literal notranslate"><span class="pre">assertIsClose</span></code> method.
But you can use:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">unittest</span>
<span class="kn">from</span> <span class="nn">math</span> <span class="kn">import</span> <span class="n">isclose</span>
<span class="k">class</span> <span class="nc">TestAlmostEqual</span><span class="p">(</span><span class="n">unittest</span><span class="o">.</span><span class="n">TestCase</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">test_floating_point</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">3</span><span class="o">*</span><span class="mf">.15</span><span class="p">,</span> <span class="mf">.45</span><span class="p">)</span>
    <span class="k">def</span> <span class="nf">test_almost_equal</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span> <span class="n">isclose</span><span class="p">(</span> <span class="mi">3</span><span class="o">*</span><span class="mf">.15</span><span class="p">,</span> <span class="mf">.45</span><span class="p">,</span> <span class="n">rel_tol</span><span class="o">=</span><span class="mi">7</span><span class="p">)</span> <span class="p">)</span>
</pre></div>
<p><strong>NOTE</strong> This is one of the key flaws with the unittest module: while
it can test anything with <code class="docutils literal notranslate"><span class="pre">assertTrue</span></code> and the like – if there is no
nifty <code class="docutils literal notranslate"><span class="pre">assert*</span></code> method for your use-case, you lose the advantages of
the <code class="docutils literal notranslate"><span class="pre">assert*</span></code> methods.</p>
<p>What are those advantages? – mostly a prettier printing of information
in the error:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">FAIL</span><span class="p">:</span> <span class="n">test_floating_point</span> <span class="p">(</span><span class="n">__main__</span><span class="o">.</span><span class="n">TestAlmostEqual</span><span class="p">)</span>
<span class="o">----------------------------------------------------------------------</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
  <span class="n">File</span> <span class="s2">&quot;/Users/Chris/PythonStuff/UWPCE/Py300-Spring2017/Examples/testing/test_floats.py&quot;</span><span class="p">,</span> <span class="n">line</span> <span class="mi">17</span><span class="p">,</span> <span class="ow">in</span> <span class="n">test_floating_point</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="mi">3</span> <span class="o">*</span> <span class="mf">.15</span><span class="p">,</span> <span class="mf">.45</span><span class="p">)</span>
<span class="ne">AssertionError</span><span class="p">:</span> <span class="mf">0.44999999999999996</span> <span class="o">!=</span> <span class="mf">0.45</span>
</pre></div>
<p>But when you use assertTrue:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">FAIL</span><span class="p">:</span> <span class="n">test_isclose_tiny</span> <span class="p">(</span><span class="n">__main__</span><span class="o">.</span><span class="n">TestAlmostEqual</span><span class="p">)</span>
<span class="o">----------------------------------------------------------------------</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
  <span class="n">File</span> <span class="s2">&quot;/Users/Chris/PythonStuff/UWPCE/Py300-Spring2017/Examples/testing/test_floats.py&quot;</span><span class="p">,</span> <span class="n">line</span> <span class="mi">32</span><span class="p">,</span> <span class="ow">in</span> <span class="n">test_isclose_tiny</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">isclose</span><span class="p">(</span><span class="mi">4</span> <span class="o">*</span> <span class="mf">.15e-30</span><span class="p">,</span> <span class="mf">.45e-30</span><span class="p">))</span>
<span class="ne">AssertionError</span><span class="p">:</span> <span class="kc">False</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">true</span>
</pre></div>
<p>Not that helpful – is it? I thikn we all already know that False is not true ;-)</p>
<p><code class="docutils literal notranslate"><span class="pre">pytest</span></code> give you nice informative messages when tests fail – without special asserts.</p>
<div class="section" id="parameterized-tests">
<h2>Parameterized Tests<a class="headerlink" href="#parameterized-tests" title="Permalink to this headline"></a></h2>
<p>Often you want to run exactly the same tests, but with different outputs and inputs.</p>
<p>You can do this a really naive way, by putting multiple asserts into one test:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_multiply</span><span class="p">():</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">4</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">6</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
</pre></div>
<p>If they all pass, fine, but if not, it will fail on the first one,
and you’ll have no idea if the others pass.</p>
<p>Plus, it gets a bit tedious to write, particularly if the code is more
complex than a single function call.</p>
<p>You can write a separate test for each case:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_multiply_both_positive</span><span class="p">():</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span> <span class="o">==</span> <span class="mi">4</span>
<span class="k">def</span> <span class="nf">test_multiply_one_negative</span><span class="p">):</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="o">==</span> <span class="o">-</span><span class="mi">4</span>
<span class="k">def</span> <span class="nf">test_multiply_both_negative</span><span class="p">():</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">6</span>
<span class="k">def</span> <span class="nf">test_multiply_second_zero</span><span class="p">():</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
<span class="k">def</span> <span class="nf">test_multiply_first_zero</span><span class="p">():</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">3</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span>
</pre></div>
<p>But talk about tedious!!!</p>
<p>Unfortunately, <code class="docutils literal notranslate"><span class="pre">unittest</span></code> does not have a built-in way to solve this problem.
There is a nifty library called parameterized, which does solve it (and they spell parameterize correctly).
It works with nose, unittest, and pytest.</p>
<p><a class="reference external" href="https://pypi.python.org/pypi/parameterized">https://pypi.python.org/pypi/parameterized</a></p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@parameterized</span><span class="p">([</span>
    <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">8</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
    <span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">0</span><span class="p">),])</span>
<span class="k">def</span> <span class="nf">test_pow</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">exponent</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
    <span class="n">assert_equal</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">pow</span><span class="p">(</span><span class="n">base</span><span class="p">,</span> <span class="n">exponent</span><span class="p">),</span> <span class="n">expected</span><span class="p">)</span>
</pre></div>
<p>You will find many more examples on their website.</p>
<div class="section" id="pytest-mark-parametrize">
<h3><code class="docutils literal notranslate"><span class="pre">pytest.mark.parametrize</span></code><a class="headerlink" href="#pytest-mark-parametrize" title="Permalink to this headline"></a></h3>
<p>With pytest, you don’t need a third party library: as it provides a nifty built-in way to do it:</p>
<p><a class="reference external" href="https://docs.pytest.org/en/latest/parametrize.html#parametrize-basics">https://docs.pytest.org/en/latest/parametrize.html#parametrize-basics</a></p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">param_names</span> <span class="o">=</span> <span class="s2">&quot;arg1, arg2, result&quot;</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
          <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">2</span><span class="p">),</span>
          <span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
          <span class="p">]</span>
<span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">parametrize</span><span class="p">(</span><span class="n">param_names</span><span class="p">,</span> <span class="n">params</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_multiply</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">,</span> <span class="n">result</span><span class="p">):</span>
    <span class="k">assert</span> <span class="n">multiply</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">)</span> <span class="o">==</span> <span class="n">result</span>
</pre></div>
<p>I find this very, very, useful.</p>
<p>See <a class="reference download internal" download="" href="../_downloads/5a5fdff5de7f7f1944ea29e74393d4b9/test_calculator_pytest.py"><code class="xref download docutils literal notranslate"><span class="pre">test_calculator_pytest.py</span></code></a></p>
<div class="section" id="code-coverage">
<h3>Code Coverage<a class="headerlink" href="#code-coverage" title="Permalink to this headline"></a></h3>
<p>“Coverage” is the fraction of your code that is run by your tests.
That is, how much code is “covered” by the tests.</p>
<p>It’s usually reported as a percentage of lines of code that were run.</p>
<p>If a line of code is <em>not</em> run in your tests – you can be pretty
sure it hasn’t been tested – so how do you know it works?</p>
<p>So 100% coverage is a good goal (though harder to achieve than you might think!)</p>
<p>Keep in mind that 100% coverage does <strong>NOT</strong> mean that your code is <em>fully</em> tested – you have no idea how many corner cases may not have been checked.</p>
<p>But it’s a good start.</p>
<div class="section" id="the-coverage-tool">
<h3>The coverage tool<a class="headerlink" href="#the-coverage-tool" title="Permalink to this headline"></a></h3>
<p><code class="docutils literal notranslate"><span class="pre">coverage.py</span></code> is a tool (written by Ned Batchelder) for checking code testing
coverage in python:</p>
<p><a class="reference external" href="https://coverage.readthedocs.io">https://coverage.readthedocs.io</a></p>
<p>It can be installed with <code class="docutils literal notranslate"><span class="pre">pip</span></code>:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ python -m pip install coverage
</pre></div>
<p>To run coverage on your test suite:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ coverage run my_program.py arg1 arg2
</pre></div>
<p>This generates a .coverage file. To analyze it on the console:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ coverage report
</pre></div>
<p>Or you can generate an HTML report in the current directory:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ coverage html
</pre></div>
<p>To find out coverage across the standard library, add -L:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">-</span><span class="n">L</span><span class="p">,</span> <span class="o">--</span><span class="n">pylib</span>   <span class="n">Measure</span> <span class="n">coverage</span> <span class="n">even</span> <span class="n">inside</span> <span class="n">the</span> <span class="n">Python</span> <span class="n">installed</span>
              <span class="n">library</span><span class="p">,</span> <span class="n">which</span> <span class="n">isn</span><span class="s1">&#39;t done by default.</span>
</pre></div>
<div class="section" id="branch-coverage">
<h3>Branch Coverage<a class="headerlink" href="#branch-coverage" title="Permalink to this headline"></a></h3>
<p>consider the following code:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">x</span> <span class="o">=</span> <span class="kc">False</span>  <span class="c1"># 1</span>
<span class="k">if</span> <span class="n">x</span><span class="p">:</span>      <span class="c1"># 2</span>
    <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;in branch&quot;</span><span class="p">)</span>  <span class="c1"># 3</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;out of branch&quot;</span><span class="p">)</span>  <span class="c1"># 4</span>
</pre></div>
<p>We want to make sure the branch is being bypassed correctly in the False
<p>Track which branch destinations were not visited with the –branch
option to run:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>coverage run --branch myprog.py
</pre></div>
<p><a class="reference external" href="http://nedbatchelder.com/code/coverage/branch.html">http://nedbatchelder.com/code/coverage/branch.html</a></p>
<div class="section" id="using-coverage-with-pytest">
<h3>Using coverage with pytest<a class="headerlink" href="#using-coverage-with-pytest" title="Permalink to this headline"></a></h3>
<p>There is a plug-in for pytest that will run coverage for you when you run your tests:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pip install pytest-cov
<span class="c1"># now it can be used</span>
$ pytest --cov code_module test_module.py
</pre></div>
<p><a class="reference external" href="https://pypi.python.org/pypi/pytest-cov">https://pypi.python.org/pypi/pytest-cov</a></p>
<p>There are a number of ways to invoke it and get different reports:</p>
<p>To get a nifty html report:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pytest --cov code_module --cov-report html test_module.py
</pre></div>
<div class="section" id="doctests">
<h2>Doctests<a class="headerlink" href="#doctests" title="Permalink to this headline"></a></h2>
<p>Tests placed in docstrings to demonstrate usage of a component to a
human in a machine testable way</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">square</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;</span>
<span class="sd">    Squares x.</span>
<span class="sd">    &gt;&gt;&gt; square(2)</span>
<span class="sd">    4</span>
<span class="sd">    &gt;&gt;&gt; square(-2)</span>
<span class="sd">    4</span>
<span class="sd">    &quot;&quot;&quot;</span>
    <span class="k">return</span> <span class="n">x</span> <span class="o">*</span> <span class="n">x</span>
</pre></div>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>python -m doctest -v example.py
</pre></div>
<p>Now generate documentation, using epydoc for example:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ epydoc example.py
</pre></div>
<p><a class="reference external" href="http://docs.python.org/3/library/doctest.html">http://docs.python.org/3/library/doctest.html</a></p>
<p><a class="reference external" href="http://www.python.org/dev/peps/pep-0257/">http://www.python.org/dev/peps/pep-0257/</a></p>
<p><a class="reference external" href="http://epydoc.sourceforge.net/">http://epydoc.sourceforge.net/</a></p>
<p>These days, most Python projects use Sphinx to do their documentation:</p>
<p><a class="reference external" href="http://sphinx-doc.org/">http://sphinx-doc.org/</a></p>
<p>Well worth checking out – and you can have Sphinx run your doctests for you.</p>
<div class="section" id="my-take">
<h3>My Take:<a class="headerlink" href="#my-take" title="Permalink to this headline"></a></h3>
<p>doctests are really cool – but they are more a way to test your documentation, than a way to test your code. Which is great – you can have examples in your docs, and know that they are still correct.</p>
<div class="section" id="test-driven-development-tdd">
<h2>Test Driven Development (TDD)<a class="headerlink" href="#test-driven-development-tdd" title="Permalink to this headline"></a></h2>
<p>In TDD, the tests are written to meet the requirements before the code
exists.</p>
<p>Once the collection of tests passes, the requirement is considered met.</p>
<p>We’ve been trying to get you to do this from the beginning of this class :-)</p>
<p>We don’t always want to run the entire test suite. In order to run a
single test with pytest:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pytest -k <span class="s2">&quot;test_divide&quot;</span>
</pre></div>
<p>The -k means:</p>
<blockquote>
<div><p>only run tests which match the given substring expression. An expression is a python evaluatable expression where all names are substring-matched against test names and their parent classes.</p>
</div></blockquote>
<p>So you can pretty easily select a subset of your tests if they have consistent naming scheme.</p>
<div class="section" id="exercises">
<h2>Exercises<a class="headerlink" href="#exercises" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>Add unit tests for each method in calculator_functions.py</p></li>
<li><p>Add fixtures via setUp/tearDown methods and setUpClass/tearDownClass
class methods. Are they behaving how you expect?</p></li>
<ul class="simple">
<li><p>Use pytest fixtures instead.</p></li>
<li><p>Add additional unit tests for floating point calculations</p></li>
<li><p>Fix any failures in the code</p></li>
<li><p>Add doctests to calculator_functions.py</p></li>
<p>Here are the files you’ll need:</p>
<p><a class="reference download internal" download="" href="../_downloads/35f2dfffa997e2acb10d5b41052228f0/calculator.py"><code class="xref download docutils literal notranslate"><span class="pre">calculator.py</span></code></a></p>
<p><a class="reference download internal" download="" href="../_downloads/5cdf55a8048eb097483f483519f663d2/calculator_functions.py"><code class="xref download docutils literal notranslate"><span class="pre">calculator_functions.py</span></code></a></p>
<p><a class="reference download internal" download="" href="../_downloads/366509eb0a8e44364a5ebd1270c18204/calculator_test.sh"><code class="xref download docutils literal notranslate"><span class="pre">calculator_test.sh</span></code></a></p>
<p><a class="reference download internal" download="" href="../_downloads/5a5fdff5de7f7f1944ea29e74393d4b9/test_calculator_pytest.py"><code class="xref download docutils literal notranslate"><span class="pre">test_calculator_pytest.py</span></code></a></p>
<p><a class="reference download internal" download="" href="../_downloads/20c2b8dec0016f4a222d42c9fddfe3b2/calculator_test_suite.py"><code class="xref download docutils literal notranslate"><span class="pre">calculator_test_suite.py</span></code></a></p>
<p><a class="reference download internal" download="" href="../_downloads/151bc0d618fe0bd8cac43e0b183fd66f/test_calculator.py"><code class="xref download docutils literal notranslate"><span class="pre">test_calculator.py</span></code></a></p>
<div class="section" id="mocking">
<h2>Mocking<a class="headerlink" href="#mocking" title="Permalink to this headline"></a></h2>
<div class="section" id="now-we-ve-got-the-tools-to-really-test">
<h3>Now we’ve got the tools to really test<a class="headerlink" href="#now-we-ve-got-the-tools-to-really-test" title="Permalink to this headline"></a></h3>
<p>Consider the application in:</p>
<p><a class="reference download internal" download="" href="../_downloads/5cbd2d019dd7b865fe9e576fc8f42d5b/wikidef.zip"><code class="xref download docutils literal notranslate"><span class="pre">wikidef.zip</span></code></a></p>
<p>Give the command line utility a subject, and it will return a definition.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>./define.py Robot
</pre></div>
<p>How can we test our application code without abusing (and waiting for)
Wikipedia?</p>
<div class="section" id="using-mock-objects">
<h3>Using Mock objects<a class="headerlink" href="#using-mock-objects" title="Permalink to this headline"></a></h3>
<p>Using Mock objects to test an application with service dependencies</p>
<p>Mock objects replace real objects in your code at runtime during test</p>
<p>This allows you to test code which calls these objects without having
their actual code run</p>
<p>Useful for testing objects which depend on unimplemented code, resources
which are expensive, or resources which are unavailable during test
execution</p>
<p><a class="reference external" href="https://docs.python.org/3/library/unittest.mock-examples.html">https://docs.python.org/3/library/unittest.mock-examples.html</a></p>
<div class="section" id="mocks">
<h3>Mocks<a class="headerlink" href="#mocks" title="Permalink to this headline"></a></h3>
<p>The MagicMock class will keep track of calls to it so we can verify
that the class is being called correctly, without having to execute the
code underneath</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">unittest</span> <span class="kn">import</span> <span class="n">mock</span>
<span class="n">mock_object</span> <span class="o">=</span> <span class="n">mock</span><span class="o">.</span><span class="n">MagicMock</span><span class="p">()</span>
<span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="s2">&quot;foo return&quot;</span>
<span class="nb">print</span><span class="p">(</span><span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="o">.</span><span class="n">call_count</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="p">())</span>
<span class="nb">print</span><span class="p">(</span><span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="o">.</span><span class="n">call_count</span><span class="p">)</span>
<span class="c1"># raise an exception by assigning to the side_effect attribute</span>
<span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="o">.</span><span class="n">side_effect</span> <span class="o">=</span> <span class="ne">Exception</span>
<span class="n">mock_object</span><span class="o">.</span><span class="n">foo</span><span class="p">()</span>
</pre></div>
<div class="section" id="easy-mocking-with-mock-patch">
<h3>Easy mocking with mock.patch<a class="headerlink" href="#easy-mocking-with-mock-patch" title="Permalink to this headline"></a></h3>
<p>patch acts as a function decorator, class decorator, or a context
manager</p>
<p>Inside the body of the function or with statement, the target is patched
with a new object. When the function/with statement exits the patch is
<div class="section" id="using-patch">
<h3>Using patch<a class="headerlink" href="#using-patch" title="Permalink to this headline"></a></h3>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># patch with a decorator</span>
<span class="nd">@patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">Wikipedia</span><span class="p">,</span> <span class="s1">&#39;article&#39;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_article_success_decorator_mocked</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">mock_method</span><span class="p">):</span>
    <span class="n">article</span> <span class="o">=</span> <span class="n">Definitions</span><span class="o">.</span><span class="n">article</span><span class="p">(</span><span class="s2">&quot;Robot&quot;</span><span class="p">)</span>
    <span class="n">mock_method</span><span class="o">.</span><span class="n">assert_called_once_with</span><span class="p">(</span><span class="s2">&quot;Robot&quot;</span><span class="p">)</span>
<span class="c1"># patch with a context manager</span>
<span class="k">def</span> <span class="nf">test_article_success_context_manager_mocked</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
    <span class="k">with</span> <span class="n">patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">Wikipedia</span><span class="p">,</span> <span class="s1">&#39;article&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">mock_method</span><span class="p">:</span>
        <span class="n">article</span> <span class="o">=</span> <span class="n">Definitions</span><span class="o">.</span><span class="n">article</span><span class="p">(</span><span class="s2">&quot;Robot&quot;</span><span class="p">)</span>
        <span class="n">mock_method</span><span class="o">.</span><span class="n">assert_called_once_with</span><span class="p">(</span><span class="s2">&quot;Robot&quot;</span><span class="p">)</span>
</pre></div>
<p>There are a number of ways to use <code class="docutils literal notranslate"><span class="pre">mock.patch</span></code> – this is a nice discussion of that: <a class="reference external" href="http://treyhunner.com/2014/10/the-many-flavors-of-mock-dot-patch/">The Many Flavors of mock.patch</a></p>
<div class="section" id="mocking-with-pytest">
<h3>mocking with pytest<a class="headerlink" href="#mocking-with-pytest" title="Permalink to this headline"></a></h3>
<p>pytest uses the same mock library, but has a little different syntax.</p>
<p>Here is an example of mocking <code class="docutils literal notranslate"><span class="pre">input()</span></code> with pytest:</p>
<p><a class="reference download internal" download="" href="../_downloads/a227f1937bb2aa9586ba90b83f79fc72/test_mock_input.py"><code class="xref download docutils literal notranslate"><span class="pre">test_mock_input.py</span></code></a></p>
<p><code class="docutils literal notranslate"><span class="pre">pytest-mock</span></code> is a utility that makes it easier to mock
with pytest.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ pip install pytest-mock
</pre></div>
<p>Here is a nice blog post about using it:</p>
<p><a class="reference external" href="https://medium.com/&#64;bfortuner/python-unit-testing-with-pytest-and-mock-197499c4623c">https://medium.com/&#64;bfortuner/python-unit-testing-with-pytest-and-mock-197499c4623c</a></p>
<div class="section" id="exercise">
<h4>Exercise<a class="headerlink" href="#exercise" title="Permalink to this headline"></a></h4>
<p>When <code class="docutils literal notranslate"><span class="pre">define.py</span></code> is given the name of a non-existent article, an exception
is thrown. This exception causes another exception to occur, and the whole thing is not very readable. Why does this happen?</p>
<p>Use what you know about exceptions to throw a better exception, and
then add a new test that confirms this behavior. Use mock for your test, so you are not hammering Wikipedia.</p>
<div class="section" id="mocking-a-builtin">
<h3>Mocking a builtin<a class="headerlink" href="#mocking-a-builtin" title="Permalink to this headline"></a></h3>
<p>Say you would like to mock input in this function in a file called mock_input.py:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_input</span><span class="p">():</span>
    <span class="n">color</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s2">&quot;What is your favorite color? &quot;</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">color</span>
</pre></div>
<p>In your test file, you would do this:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@mock</span><span class="o">.</span><span class="n">patch</span><span class="p">(</span><span class="s1">&#39;builtins.input&#39;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test_get_input</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">new_mocked_input</span><span class="p">):</span>
    <span class="n">new_mocked_input</span><span class="o">.</span><span class="n">return_value</span> <span class="o">=</span> <span class="s1">&#39;blue&#39;</span>
    <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">mock_input</span><span class="o">.</span><span class="n">get_input</span><span class="p">(),</span> <span class="s1">&#39;blue&#39;</span><span class="p">)</span>
</pre></div>
           </div>
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../topics/18-advanced_testing/index.html" class="btn btn-neutral float-left" title="18. Advanced Testing" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../exercises/mailroom/mailroom-mock.html" class="btn btn-neutral float-right" title="Mocking Mailroom" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
  <div role="contentinfo">
    <p>&#169; Copyright 2020, University of Washington, Natasha Aleksandrova, Christopher Barker, Brian Dorsey, Cris Ewing, Christy Heaton, Jon Jacky, Maria McKinley, Andy Miles, Rick Riehle, Joseph Schilz, Joseph Sheedy, Hosung Song. Creative Commons Attribution-ShareAlike 4.0 license.</p>
  Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
    provided by <a href="https://readthedocs.org">Read the Docs</a>.
        </div>
      </div>
    </section>
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
  </script> 
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FilesExpand file tree

Testing_advanced.html

Latest commit

History

Testing_advanced.html

File metadata and controls
