test_plan.md

Test plan

Environment

OS: XXX (Windows, macOS, latest Ubuntu LTS)
- Shell: XXX (Command Prompt, PowerShell, bash, fish)
Python
- Distribution: XXX (CPython, miniconda)
- Version: XXX (2.7, latest 3.x)
VS Code: XXX (Insiders)

Tests

ALWAYS:

Check the Output window under Python for logged errors
Have Developer Tools open to detect any errors
Consider running the tests in a multi-folder workspace
Focus on in-development features (i.e. experimental debugger and language server)

Scenarios

Environment

Interpreters

Terminal

Sample file:

import requests
request = requests.get("https://drive.google.com/uc?export=download&id=1_9On2-nsBQIw3JiY43sWbrF8EjrqrR4U")
with open("survey2017.zip", "wb") as file:
    file.write(request.content)
import zipfile
with zipfile.ZipFile('survey2017.zip') as zip:
    zip.extractall('survey2017')
import shutil, os
shutil.move('survey2017/survey_results_public.csv','survey2017.csv')
shutil.rmtree('survey2017')
os.remove('survey2017.zip')

Shift+Enter to send selected code in sample file to terminal works

Virtual environments

ALWAYS:

Environment files

Sample files:

# example.py
import os
print('Hello,', os.environ.get('WHO'), '!')

# .env
WHO=world
PYTHONPATH=some/path/somewhere
SPAM='hello ${WHO}'

ALWAYS:

Make sure to use Reload Window between tests to reset your environment
Note that environment files only apply under the debugger
Environment variables in a .env file are exposed when running under the debugger
"python.envFile" allows for specifying an environment file manually
envFile in a launch.json configuration works
simple variable substitution works

Debugging

pythonPath setting in your launch.json overrides your python.pythonPath default setting

Linting

ALWAYS:

Check under the Problems tab to see e.g. if a linter is raising errors

Language server

LS is downloaded using HTTP (no SSL) when the "http.proxyStrictSSL" setting is false
An item with a cloud icon appears in the status bar indicating progress while downloading the language server
Installing requests in virtual environment is detected
- Import of requests without package installed is flagged as unresolved
- Create a virtual environment
- Install requests into the virtual environment

Pylint/default linting

[Prompting to install Pylint is covered under Environments above]

For testing the disablement of the default linting rules for Pylint:

# pylintrc
[MESSAGES CONTROL]
enable=bad-names

# example.py
foo = 42  # Marked as a disallowed name.

Installation via the prompt installs Pylint as appropriate
- Uses --user for system-install of Python
- Installs into a virtual environment environment directly
Pylint works
The existence of a pylintrc file turns off the default rules

Other linters

Note:

Editing

IntelliSense

Please also test for general accuracy on the most "interesting" code you can find.

"python.autoComplete.extraPaths" works
"python.autocomplete.addBrackets": true causes auto-completion of functions to append ()
Auto-completions works

Formatting

Sample file:

# There should be _some_ change after running `Format Document`.
import os,sys;
def foo():pass

Refactoring

Extract Variable works
- You are prompted to install rope if it is not already available
Extract method works
- You are prompted to install rope if it is not already available
Sort Imports works

Debugging

Unit testing

`unittest`

import unittest

MODULE_SETUP = False


def setUpModule():
    global MODULE_SETUP
    MODULE_SETUP = True


class PassingSetupTests(unittest.TestCase):
    CLASS_SETUP = False
    METHOD_SETUP = False

    @classmethod
    def setUpClass(cls):
        cls.CLASS_SETUP = True

    def setUp(self):
        self.METHOD_SETUP = True

    def test_setup(self):
        self.assertTrue(MODULE_SETUP)
        self.assertTrue(self.CLASS_SETUP)
        self.assertTrue(self.METHOD_SETUP)


class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

Run All Unit Tests triggers the prompt to configure the test runner
Tests are discovered (as shown by code lenses on each test)
- Code lens for a class runs all tests for that class
- Code lens for a method runs just that test
  - Run Test works
  - Debug Test works
  - Module/suite setup methods are also run (run the test_setup method to verify)
while debugging tests, an uncaught exception in a test does not cause debugpy to raise SystemExit exception.

`pytest`

def test_passing():
    assert 42 == 42

def test_failure():
    assert 42 == -13

Run All Unit Tests triggers the prompt to configure the test runner
- pytest gets installed
Tests are discovered (as shown by code lenses on each test)
- Run Test works
- Debug Test works
A Diagnostic is shown in the problems pane for each failed/skipped test
- The Diagnostics are organized according to the file the test was executed from (not necessarily the file it was defined in)
- The appropriate DiagnosticRelatedInformation is shown for each Diagnostic
- The DiagnosticRelatedInformation reflects the traceback for the test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test plan

Environment

Tests

Environment

Interpreters

Terminal

Virtual environments

Environment files

Debugging

Linting

Language server

Pylint/default linting

Other linters

Editing

IntelliSense

Formatting

Refactoring

Debugging

Unit testing

`unittest`

`pytest`

General

FilesExpand file tree

test_plan.md

Latest commit

History

test_plan.md

File metadata and controls

Test plan

Environment

Tests

Interpreters

Terminal

Virtual environments

Language server

Pylint/default linting

Other linters

General