This task has been started in PR #5405, but ended without success, so I have decided to split it to multiple stages:

1. Make pytest compatible with the current test suite
   • Add support of unitest-like fixtures (setUp and other methods) on classes not derived from unitest.UnitTest (fixed with 8b64294)
   • Add support of image_comparison and knownfailif decorator (fixed with bbaf7dc)
   • Fix raise KnownFailureTest problems (PR Add Py.test testing framework support #6730)
     • ImageComparisonTest.test
     • test_axes.test_formatter_large_small
     • test_backend_pgf.compare_figure
     • test_animation.check_save_animation
     • test_coding_standards.test_pep8_conformance_examples
   • test_basic leaks tests from numpy into global scope with from pylab import * (fixed with 30caf00)
   • Solve strange failures on several tests
     • test_subplots.test_subplots_offsettext (PR Missing cleanup decorator in test_subplots.test_exceptions #6741)
     • test_axes.test_eventplot_problem_kwargs (PR Fixed warnings catching and counting with warnings.catch_warnings #6761)
     • test_pickle.test_complete (PR Fixed broken test_pickle.test_complete test #6840)
   • Ensure that pytest collects all the tests (6150/6150) (actually nose shows 6151 because it counts lib/matplotlib/tests/test_delaunay.py::make_all_2d_testfuncs as a test)
2. Migration to pytest
   • Add new build under allow_failures with pytest
     • Appveyor
     • Travis
     • Use pytest-cov for coverage reports
   • Replace nose in all builds with pytest
3. Post-migration
   • Rename setUp -> setup_method and other methods to pytest analogs
     • Do not inherit tests from unitest.UnitTest class
     • Remove setUp/tearDown workaround
   • Rewrite image_comparison decorator (currently it slowdowns the collection phase due to baseline images copying) or use pytest-mpl plugin (needs some improvements, see paragraph below)
   • Rewrite all assertations to pure assert
   • Get away from cleanup decorator
   • Replace knownfailif with mark.xfail
   • Replace raise nose.SkipTest(...) with xfail(...) call or mark.skipif decorator
4. Enhancements
   • Replace setup_method and etc methods with fixtures
   • Rewrite test_delaunay.make_all_2d_testfuncs
   • Use pytest-pep8 plugin instead of test_coding_standards
   • Use pytest-mock plugin instead of mock and unittest.mock

Tests entry point (raised in #5405 (comment))

I think that running tests with simple py.test command is better than python tests.py, but there are unused tests which will run that way. Solutions:

• A. Get them away from pytest's sight with:
  1. Remove unused tests (test_sankey, test_skew, test_ttconv, and test_usetex from lib/matplotlib/tests)
  2. Rename test_only.py (to setup_test_only.py probably)
• B. Make filter for collection phase.
  1. Implement whitelist
  2. Implement blacklist

I have tried both ways and they work, so what to choose is up to you.

• py.test --collect-filter=none
• py.test --collect-filter=blacklist
• py.test --collect-filter=whitelist

Currently --collect-filter=whitelist is default (to follow same politics as nose does currently).

Image comparison decorator

Why do we have to rewrite image_comparison decorator? Because pytest-mpl plugin in the current state ("as is") is not usable for matplotlib:

• + supports savefig_kwargs
• + supports tolerance
• ‒ need to supply baseline_dir in every decorator
• ‒ does not support image format conversion
• ‒ need something to do with freetype_version, remove_text, and style