diff --git a/.circleci/config.yml b/.circleci/config.yml new file mode 100644 index 00000000..acb1e62a --- /dev/null +++ b/.circleci/config.yml @@ -0,0 +1,46 @@ +version: 2.1 + +orbs: + python: circleci/python@1.2 + +jobs: + build-and-test: + docker: + - image: cimg/python:3.12 + steps: + - checkout + - python/install-packages: + pip-dependency-file: requirements-test.txt + pkg-manager: pip + - run: + name: Run tests + command: pytest -m "not compiled" --cov=pythonwhat + publish: + docker: + - image: cimg/python:3.9 + steps: + - checkout + - run: + command: | + python setup.py sdist bdist_wheel + pip install pipenv + pipenv install twine + pipenv run twine upload --verbose --repository pypi dist/* + +workflows: + build: + jobs: + - build-and-test: + context: org-global + filters: + tags: + only: /^v\d+\.\d+\.\d+$/ + - publish: + context: org-global + requires: + - build-and-test + filters: + tags: + only: /^v\d+\.\d+\.\d+$/ + branches: + ignore: /.*/ diff --git a/.coveragerc b/.coveragerc new file mode 100644 index 00000000..c2c30563 --- /dev/null +++ b/.coveragerc @@ -0,0 +1,2 @@ +[run] +source=pythonwhat \ No newline at end of file diff --git a/.gitignore b/.gitignore index 17a5e5ba..90e761a1 100644 --- a/.gitignore +++ b/.gitignore @@ -71,3 +71,9 @@ target/ # Mac stuff .DS_Store tests/.DS_Store + +# pytest +.pytest_cache/ + +# datasets +*.csv diff --git a/.readthedocs.yml b/.readthedocs.yml new file mode 100644 index 00000000..7f0089f5 --- /dev/null +++ b/.readthedocs.yml @@ -0,0 +1,20 @@ +# Read the Docs configuration file +# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details + +# Required +version: 2 + +# Build documentation in the docs/ directory with Sphinx +sphinx: + configuration: docs/conf.py + +# Optionally build your docs in additional formats such as PDF and ePub +formats: all + +# Optionally set the version of Python and requirements required to build your docs +python: + version: 3.5 + install: + - requirements: requirements.txt + - method: pip + path: . diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..cd1ab6fc --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,391 @@ +# Changelog + +All notable changes to the pythonwhat project will be documented in this file. This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). + +## 2.24.0 + +- Support Python version 3.9 +- Update dependencies to up-to-date versions + +## 2.23.2 + +- Update behaviour of has_expr() default feedback message. If the student's evaluation is too long, it is now shortened and an ellipsis is added. + +## 2.23.1 + +- Fix string formatting in has_expr(). Strings will now have quotes in error messages. Leading and trailing whitespace is no longer removed. + +## 2.23.0 + +- Update to protowhat v2 (embedding xwhats, `prepare_validation` helper, for checking bash history, autodebug) + +## 2.22.0 + +- Add support for replacing the placeholder `__focus__` with the focused code in the `expr_code` argument +- Expose bash history functionality from protowhat + +## 2.21.0 + +- Use path info in `run` + +## 2.20.2 + +- Improve checking exceptions +- Improve checking file content + +## 2.20.1 + +- Speed improvements + +## 2.20.0 + +- Expose `_debug` function +- Update protowhat + +## 2.19.0 + +- `state` is now the first argument to SCT functions (instead of a keyword argument) +- functionality is shared with protowhat: + - `Reporter`, `Chain`, `F` and logic SCTs are reused + - `State` and `Feedback` inherit + - `Dispatcher` handles parsing (decoupled from `State`) + - `Test` moved + +## 2.18.1 + +### Fixed + +- Local setup with `setup_state` now works correctly (e.g. when the PEC contains a list comprehension) + +## 2.18.0 + +- Add optional `force_diagnose` parameter to `test_exercise` to force passing the `diagnose` tests in `check_correct`. + +## 2.17.2 + +### Improved + +Documentation has been improved significantly (hopefully): + +- Every 'compound statement check' has an example with explanation in the reference now. +- Common usecases have additional examples +- Checking compound statements article has been simplified, limits to context now. +- More helpful message in case check_object or has_printout not called on root state +- More helpful message in case the zoomed in on object is not an AST +- Explain has_equal_value vs has_equal_ast +- Get rid of some articles in favor of more fleshed out reference documentation + +### Fixed + +- CI is now using the `datacamp` account on PyPi + +## 2.17.1 + +### Fixed + +- Tuples of Numpy arrays can now be checked properly. + +### Removed + +- Code in `ObjectAssignmentParser` that is not used. + +## 2.17.0 + +### Added + +- Function `has_no_error()` to check earlier on whether the student did not generate any errors. + +### Improved + +- Messaging between V1 and V2 is entirely consistent now. +- No need for `__JINJA__` prefix in custom messages specified in SCTs anymore. + +### Removed + +- No more support for `expand_message` argument in old 'node checking functions' such as `test_for_loop()`. + +## 2.16.2 + +### Added + +- Ability to check class definitions with ``check_class_def()`` (and ``check_bases()``). Tested and documented. + +## 2.16.1 + +### Changed + +- The `check_object()` only on root check is only done if `PYTHONWHAT_V2_ONLY` environment variable is set. + +### Added + +- More checks that guard against commonly made mistakes (some in v2 only, others not) + +## 2.16.0 + +### Added + +- Function documentation (with examples) for `multi()`, `check_or()`, `check_correct()` and `check_not()` + +### Changed + +- If an SCT is incorrectly coded, it will generate more easily understandable errors so the author can easily fix the issue. +- If an SCT correctly runs but does not make a lot of sense, easily understanble errors will be thrown so the author can make improvements. + +## 2.15.3 + +### Added + +- `check_object()` does not check whether the targeted object is specified in the solution process if + student and solution process are identical (as is the case in the `SingleProcessExercise`). +- `has_expr()`, the function used by `has_equal_value()`, `has_equal_output()` and `has_equal_error()` can take an `override` argument, + that causes the solution expression _not_ to run and use the value specified in `override` instead. + For more information, have a look at the 'SingleProcessExercise' article on the documentation. + +## 2.15.2 + +### Removed + +- `call()` can no longer be used. Use `Ex().check_function_def().check_call().has_equal_x()` instead. +- `has_key()` and `has_equal_key()` can no longer be used. Use `Ex().check_object().check_keys().has_equal_x()` instead. + +## 2.15.1 + +- Fixed: `check_keys()` allows for more exotic ways of indexing as well, even though it's needed very rarely + +## 2.15.0 + +### Added + +- You can now use `Ex().check_object('df').check_keys('a').has_equal_value()` to test DataFrame columns and dictionary elements. +- You can now use `Ex().check_function_def('my_fun').check_call('f(1,3,4)').has_equal_x()` to check the value, output or error that calling a function generates. Soon, the `call()` syntax, although still supported, will be removed. +- More manual signatures have been added for functions in the `numpy.random` submodule, so SCT authors have to specify `signature=False` less and there is more robust argument matching. + +### Changed + +- Update docs to promote new functions introduced above. +- Messaging has improved: if there is crazy nesting, only the last two 'expand messages' are included. That way, you don't get feedback messages like "Check the first for loop. Check the body. Check the first for loop. Check the body. Check the function. ...". +- The function parser (used by `check_function()`) now also discovers function calls in lists and dictionaries. +- `has_import()` is now more flexible by default, not requiring students to use the same alias. + +### Removed + +- Nothing for now, but the following functions will be discontinued in the future: + + `test_function_definition()` + + `test_with()` + + `test_object_after_expression()` + + `has_key()` and `has_equal_key()` + + `call()` + +## 2.14.2 + +- Add `check_df()` to the API again. Turns out quite a lot of live exercises use it by now! + +## 2.14.1 + +- Fix issue in `test_data_frame()` if message is not specified. +- Improve messaging in `has_key()` and `has_equal_key()`. + +## 2.14.0 + +### Changed + +- If `PYTHONWHAT_V2_ONLY = '1'` is set as an environment variable, you can no longer use _any_ of the `test_` functions. + + Instead of `Ex().test_or(...)`, you have to use `Ex().check_or(...)`. + + Instead of `Ex().test_correct(...)`, you have to use `Ex().check_correct(...)`. + + Instead of `test_mc()`, you have to use `Ex().has_chosen()`. + Docs have been updated accordingly. +- The package structure has been updated significantly + + Distributing nearly all new functions over `check_funcs.py`, `has_funcs.py` and `check_logic.py`. + + Grouping all old functions to test compound statements. + + Moving around and rewriting tests to use `pytest` more and be more readable overall. + +### Fixed + +- `test_not()`'s functionality was tested more and bugs that appeared were fixed. + It was nowhere used, so it was removed from the API in favor for `check_not()`, which has the same functionality. + +### Removed + +- `extend()` can not be used anymore. +- `check_df()` can not be used anymore. UPDATE: added again in 2.14.2. + +## 2.13.2 + +### Changed + +- The documentation pages have undergone significant maintenance. + + There is now a tutorial that gradually exposes you to `pythonwhat`. + + The articles have been brushed up to include more examples and more involved examples. + + The articles have been split up into basic and advanced articles to make it clear what is most important. + + `test_correct()` as a tool to add robustness is now featured more prominently + +### Fixed + +- `set_context()` can now be used to specify arguments either by position, either by name, making it a great tool for flexible checking. +- For `has_equal_output()`, the message that was generated didn't always make sense. That is fixed now. +- Highlighting was removed in an SCT chain when `set_context()` was used. This is no longer the case. +- Small fixes for bugs that should not have impacted students in the first place. + +## 2.13.1 + +_Small changes to follow up on `2.13.0`_ + +- Get rid of `typestr` argument in `check_function()` as it's used nowhere +- Change the internals of has_printout, to be more allowing for different ways of doing things +- Improve variable names for readability and understanding + +## 2.13.0 + +### Changed + +- `test_function()` and `test_function_v2()` now use `check_function()` and `check_args()` behind the scenes. + That way, when we make improvements to the messaging logic, all SCTs that use any of these three functions will benefit from them. + In the future, we will deprecate `test_function()` and `test_function_v2()` as they are not explicit enough about what is being tested and how. +- `test_function('print')` and `check_function('print')` use `Ex().has_printout()` behind the scenes when appropriate. + This makes the SCTs much more accepting for different ways of doing printouts. +- You can now use `check_finalbody()` to check the `finally` part of a `try-except` block. +- Drastically improve `has_equal_ast()` messaging. +- If you manually specify `code` argument in `has_equal_ast()`, you _have_ to specify the `incorrect_msg` because the machine-generated one will be meaningless (for now). + +### Fixed + +- **BIG ONE**: You can now test method calls that have subscripts in them. + This is particularly useful for `pandas`, where you for example want to test a call `df[df.b == 'x'].a.sum()`. + You can now do that with: + + ```python + # Check whether the function was called: + Ex().check_function('df.a.sum', signature=False) + # Check whether the function was called and generated the correct result: + Ex().check_function('df.a.sum', signature=False).has_equal_value() + ``` + + This update means that you should no longer need to use two `has_equal_ast()`'s inside a `test_correct()` to allow for two + different ways of doing a pandas operation. If you do, please create an issue! + +- Some fixes to `has_printout()` that caused it not to work in all cases. +- `check_function()` now refers to a function call in the way that the student defined it. + When a student uses `import pandas as pd` and then has to call a function `pd.Series()`, + `check_function()` will refer to the function with `pd.Series()` and not `pandas.Series()`. + +### Removed + +- `test_dict_comp()`, `test_try_except()`, `test_generator_exp()` and `test_lambda_function()` have been removed from the API. + The couple of SCTs on DataCamp that used these functions have been converted to use the modern `check_` functions. + +## 2.12.6 + +### Changed + +- **If manually specifying `incorrect_msg` in has_equal_x: do not prepend previously generated messages. You no longer have to set `expand_msg = ""`.** + +- Overall, improvement of automaticlaly generated messages: + + Bite-size messages that are pasted together + + More meaningful defaults that help (see `tests/test_messaging.py`) + + Get rid of default-generated messages scattered all over the place + + Better description of arguments + + Better description of calls + + Get rid of `has_key()` and `has_equal_key()` docs, as they will be phased out + + Improved handling of getting results, output and values from process + + Simplify old `test_expression_x()` functions and group in one file + +## 2.12.4 + +### Added + +- You can now robustly test printouts with `Ex().has_printout(index = x)`. This function will look for the `x`'th `print()` call in the solution code, rerun that call while capturing the output, and then look for that output in the output that was generated by the student's code submission. + + ```python + Ex().has_printout(index=0) + ``` + + This approach is far easier and more robust than using `Ex().check_function().check_args().has_equal_value()`. + + +### Changed + +- Improvement in messages that are generated by default when checking `*args` and `**kwargs` arguments in function calls. + +### Fixed + +- You can now use `test_or()` inside `test_correct()`: + + ```python + Ex().test_correct( + check_object(...) + test_or( + check_function(...), + has_equal_ast(...) + ) + ) + ``` + + +## 2.12.3 + +### Added + +- Whether or not something should be highlighted can be specified through the state now +- You can use `.disable_highlighting().` anywhere in the SCT chain to disable highlighting. + +### Changed + +- There are no more `highlight` arguments in any pythonwhat SCT functions. +- `Feedback` now takes a state from which it reads which part should be highlighting and whether it should be highlighted. +- Use of `StubState` to trick the system somewhat, should be refactored at some point, but works fine. +- `test_function_v2()` and `test_function()` only higlight if the `index = 1`, i.e. when the first call of a certain function is being checked. Added tests (and updated other tests) accordingly. + +## 2.12.2 + +### Added + +- In `check_args()`, you can now use `['args', 0]` and `['kwargs', 'a']` to look for matched positional star args, and matched named star args. + The docs have been updated accordingly: https://pythonwhat.readthedocs.io/en/stable/articles/checking_function_calls.html + +### Changed + +- The equality checks for lists, dicts, numpy arrays, pandas dataframes and pandas series have been made faster, without compromising backwards compatibility. + +### Fixed + +- There was a nasty bug with signature binding when using `check_function()` for the same function when this function took only positional args. This would alleviate the need of any `signature=False` usage, once and for all! + +## 2.12.1 + +### Added + +- Reference documentation now contains examples for `set_context()` and `set_env()`. +- A `tests/test_debug.py` file has been added to easily test questions that CDs might have. + +### Changed + +- There was a bug in the feedback message generation for errors. It now just refers to the console. + + +## 2.12.0 + +### Added + +- All new `has_()` functions are now properly documented, so there should be no reason for you to use a `test_` function other than `test_or` and `test_correct`. If you do find yourself using it (because you don't know the alternative), ping me and we'll talk! +- Added documentation on the `check_function_def()` related functions. +- `index = 0` is now a default for functions like `check_if_else()`, `check_function()` etc. +- Like there is `set_context`, there is now `set_env` to set environment variables before using something like `has_equal_value()`. These `set_` functions serve as alternatives for the `extra_env` and `context_vals` arguments that appear in many functions and that I plan to discontinue at some point. +- Added a `test_` to `check_` article, that explains how you can go from old-style to new-style SCTs. Feel free to contribute!! + +### Changed + +- If you experiment locally, you will now see the feedback message that the SCT chain would generate (this is explained in the README on GitHub): + +```Python +from pythonwhat.local import setup_state +s = setup_state(sol_code="x = 4", stu_code="x = 5") +s.check_object('x').has_equal_value() +# +# pythonwhat.Test.TestFail: Check the variable `x`. Unexpected expression value: expected `4`, got `5`. +``` + +- The glossary is featured more prominently in the docs, as it is a good resource to see how everything fits together. + +### Removed + +- There is no support for `keep_objs_in_env`, as nobody is using it. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index d663f34e..00000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,8 +0,0 @@ -Some exercises are very challenging to write appropriate SCTs for. In your issue, be sure to include, - -1. Where you looked in [the wiki](https://www.github.com/datacamp/pythonwhat/wiki). -2. A link to the __source code__ (i.e. an RMarkdown file), or a copy of the exact pre-exercise and solution code. -3. What you want to test with the SCT, and which message you want it to return for a given submission. (Be exact). -4. What you have already tried (include the code) and why the documentation is not helping you (that way, the documentation can be updated afterwards). - -Thanks! diff --git a/LICENSE b/LICENSE new file mode 100644 index 00000000..dbbe3558 --- /dev/null +++ b/LICENSE @@ -0,0 +1,661 @@ + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published + by the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +. diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 00000000..096dd503 --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,2 @@ +include LICENSE +include requirements.txt diff --git a/README.md b/README.md index 09daaba3..f50b0e0b 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,68 @@ -![banner](https://s3.amazonaws.com/assets.datacamp.com/img/github/content-engineering-repos/pythonwhat_banner_v2.png) +# pythonwhat -The `pythonwhat` package provides rich functionality to write Submission Correctness Tests for interactive Python exercises on the DataCamp platform. DataCamp operates with **Python 3**. +[![Build Status](https://travis-ci.org/datacamp/pythonwhat.svg?branch=master)](https://travis-ci.org/datacamp/pythonwhat) +[![PyPI version](https://badge.fury.io/py/pythonwhat.svg)](https://badge.fury.io/py/pythonwhat) +[![Documentation Status](https://readthedocs.org/projects/pythonwhat/badge/?version=stable)](http://pythonwhat.readthedocs.io/en/stable/?badge=stable) +[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fdatacamp%2Fpythonwhat.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fdatacamp%2Fpythonwhat?ref=badge_shield) -For a detailed guide on how to use `pythonwhat`, head over to [the online documentation](http://pythonwhat.readthedocs.io). Before, all documentation was on the [wiki](https://github.com/datacamp/pythonwhat/wiki), but things are steadily being moved to the _readthedocs_ format. +Verify Python code submissions and auto-generate meaningful feedback messages. Originally developed for Python exercises on DataCamp for so-called Submission Correctness Tests, but can also be used independently. -Visit [DataCamp Teach](https://www.datacamp.com/teach) to create your own DataCamp Python course, powered by `pythonwhat`. +- New to teaching on DataCamp? Check out https://instructor-support.datacamp.com +- To learn what SCTs are and how they work, visit [this article](https://instructor-support.datacamp.com/courses/course-development/submission-correctness-tests) specifically. +- For a complete overview of all functionality inside pythonwhat and articles about what to use when, consult https://pythonwhat.readthedocs.io. ## Installation +```bash +# latest stable version from PyPi +pip install pythonwhat + +# latest development version from GitHub +pip install git+https://github.com/datacamp/pythonwhat ``` -pip3 install markdown2 -pip3 install numpy -pip3 install pandas -pip3 install matplotlib -pip3 install git+https://github.com/datacamp/pythonwhat + +## Demo + +To experiment locally, you can use `setup_state()` and write SCTs interactively. +The code throws an error when the underlying checks fail. + +```python +# make all checking functions available +from pythonwhat.test_exercise import prep_context +_, ctxt = prep_context() +globals().update(ctxt) + +# initialize state with student and solution submission +from pythonwhat.test_exercise import setup_state +setup_state(stu_code = "x = 5", sol_code = "x = 4") + +Ex().check_object('x') +# No error: x is defined in both student and solution process + +Ex().check_object('x').has_equal_value() +# TestFail: Did you correctly define the variable `x`? Expected `4`, but got `5`. + +# Debugging state +Ex()._state # access state object +dir(Ex()._state) # list all elements available in the state object +Ex()._state.student_code # access student_code of state object ``` +To learn how to include an SCT in a DataCamp course, visit https://instructor-support.datacamp.com. + ## Run tests +```bash +pyenv local 3.12.7 +pip3.12 install -r requirements-test.txt +pip3.12 install -e . +pytest ``` -# install python backend (private) + required packages -sudo pip3 install boto3 -sudo pip3 install bs4 -sudo pip3 install h5py -cd path/to/pythonbackend -python3 setup.py install - -cd /path/to/pythonwhat -python3 setup.py install -cd tests -python3 run_all.py -``` -To disable deprecation warnings: `$ export PYTHONWARNINGS="ignore"` +## Contributing + +Bugs? Questions? Suggestions? [Create an issue](https://github.com/datacamp/pythonwhat/issues/new), or [contact us](mailto:content-engineering@datacamp.com)! + +## License -For more details, questions and suggestions, contact learn-engineering@datacamp.com. +[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fdatacamp%2Fpythonwhat.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fdatacamp%2Fpythonwhat?ref=badge_large) diff --git a/docs/Makefile b/docs/Makefile index 9708020c..e77458a9 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -1,216 +1,20 @@ -# Makefile for Sphinx documentation +# Minimal makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = build +SPHINXPROJ = pythonwhat +SOURCEDIR = . +BUILDDIR = _build -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source - -.PHONY: help +# Put it first so that "make" without argument is like "make help". help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -.PHONY: clean -clean: - rm -rf $(BUILDDIR)/* - -.PHONY: html -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -.PHONY: dirhtml -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -.PHONY: singlehtml -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -.PHONY: pickle -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -.PHONY: json -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -.PHONY: htmlhelp -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -.PHONY: qthelp -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Pythonwhat.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Pythonwhat.qhc" - -.PHONY: applehelp -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -.PHONY: devhelp -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/Pythonwhat" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Pythonwhat" - @echo "# devhelp" - -.PHONY: epub -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -.PHONY: latex -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -.PHONY: latexpdf -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -.PHONY: latexpdfja -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -.PHONY: text -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -.PHONY: man -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -.PHONY: texinfo -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -.PHONY: info -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -.PHONY: gettext -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -.PHONY: changes -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -.PHONY: linkcheck -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -.PHONY: doctest -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -.PHONY: coverage -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -.PHONY: xml -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." +.PHONY: help Makefile -.PHONY: pseudoxml -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/README.rst b/docs/README.rst deleted file mode 100644 index bd82c1c0..00000000 --- a/docs/README.rst +++ /dev/null @@ -1,866 +0,0 @@ -**NOTE**: The pdf version of the docs is better formated. Code parts are a bit messed in the `.rst` file. - - -pythonwhat -********** - - -test_expression_output -====================== - -pythonwhat.test_expression_output.test_expression_output(extra_env=None, context_vals=None, incorrect_msg=None, eq_condition='equal', pre_code=None, keep_objs_in_env=None) - - Test output of expression. - - The code of the student is ran in the active state and the output - it generates is compared with the code of the solution. This can be - used in nested pythonwhat calls like test_if_else. In these kind of - calls, the code of the active state is set to the code in a part of - the sub statement (e.g. the body of an if statement). It has - various parameters to control the execution of the (sub)expression. - - Parameters: - * **extra_env** (*dict*) -- set variables to the extra - environment. They will update the student and solution - environment in the active state before the student/solution - code in the active state is ran. This argument should contain - a dictionary with the keys the names of the variables you want - to set, and the values are the values of these variables. - - * **context_vals** (*list*) -- set variables which are bound - in a for loop to certain values. This argument is only useful - if you use the function in a test_for_loop. It contains a list - with the values of the bound variables. - - * **incorrect_msg** (*str*) -- feedback message if the output - of the expression in the solution doesn't match the one of the - student. This feedback message will be expanded if it is used - in the context of another test function, like test_if_else. - - * **eq_condition** (*str*) -- the condition which is checked - on the eval of the group. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **pre_code** (*str*) -- the code in string form that should - be executed before the expression is executed. This is the - ideal place to set a random seed, for example. - - * **keep_obj_in_env** (*list(*) -- a list of variable names - that should be hold in the copied environment where the - expression is evaluated. All primitive types are copied - automatically, other objects have to be passed explicitely. - - -[ Examples ]- - - Student code - - "a = 12" - "if a > 3:" - "print('test %d' % a)" - - Soltuion code - - "a = 4" - "if a > 3:" - "print('test %d' % a)" - - SCT - - "test_if_else(1," - "body = lambda: test_expression_output(extra_env = { 'a': 5 }," - "incorrect_msg = "Print out the correct things"))" - - This SCT will pass as the subexpression will output 'test 5' in - both student as solution environment, since the extra environment - sets *a* to 5. - - -test_expression_result -====================== - -pythonwhat.test_expression_result.test_expression_result(extra_env=None, context_vals=None, incorrect_msg=None, eq_condition='equal', expr_code=None, pre_code=None, keep_objs_in_env=None) - - Test result of expression. - - The code of the student is ran in the active state and the result - of the evaluation is compared with the result of the solution. This - can be used in nested pythonwhat calls like test_if_else. In these - kind of calls, the code of the active state is set to the code in a - part of the sub statement (e.g. the condition of an if statement). - It has various parameters to control the execution of the - (sub)expression. - - Parameters: - * **extra_env** (*dict*) -- set variables to the extra - environment. They will update the student and solution - environment in the active state before the student/solution - code in the active state is ran. This argument should contain - a dictionary with the keys the names of the variables you want - to set, and the values are the values of these variables. - - * **context_vals** (*list*) -- set variables which are bound - in a for loop to certain values. This argument is only useful - if you use the function in a test_for_loop. It contains a list - with the values of the bound variables. - - * **incorrect_msg** (*str*) -- feedback message if the result - of the expression in the solution doesn't match the one of the - student. This feedback message will be expanded if it is used - in the context of another test function, like test_if_else. - - * **eq_condition** (*str*) -- the condition which is checked - on the eval of the group. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **expr_code** (*str*) -- if this variable is not None, the - expression in the studeont/solution code will not be ran. - Instead, the given piece of code will be ran in the student as - well as the solution environment and the result will be - compared. - - * **pre_code** (*str*) -- the code in string form that should - be executed before the expression is executed. This is the - ideal place to set a random seed, for example. - - * **keep_obj_in_env** (*list(*) -- a list of variable names - that should be hold in the copied environment where the - expression is evaluated. All primitive types are copied - automatically, other objects have to be passed explicitely. - - -[ Examples ]- - - Student code - - "a = 12" - "if a > 3:" - "print('test %d' % a)" - - Solution code - - "a = 4" - "b = 5" - "if (a + 1) > (b - 1):" - "print('test %d' % a)" - - SCT - - "test_if_else(1," - "test = lambda: test_expression_result(extra_env = { 'a': 3 }" - "incorrect_msg = "Test if `a` > 3"))" - - This SCT will pass as the condition in the student's code (*a > 3*) - will evaluate to the same value as the code in the solution code - (*(a + 1) > (b - 1)*), with value of *a* set to *3*. - - -test_for_loop -============= - -pythonwhat.test_for_loop.test_for_loop(index=1, for_iter=None, body=None, orelse=None, expand_message=True) - - Test parts of the for loop. - - This test function will allow you to extract parts of a specific - for loop and perform a set of tests specifically on these parts. A - for loop consists of two parts: the sequence, *for_iter*, which is - the values over which are looped, and the *body*. A for loop can - have a else part as well, *orelse*, but this is almost never used. - - "for i in range(10):" - "print(i)" - - Has *range(10)* as the sequence and *print(i)* as the body. - - Parameters: - * **index** (*int*) -- index of the function call to be - checked. Defaults to 1. - - * **for_iter** -- this argument holds the part of code that - will be ran to check the sequence of the for loop. It should - be passed as a lambda expression or a function. The functions - that are ran should be other pythonwhat test functions, and - they will be tested specifically on only the sequence part of - the for loop. - - * **body** -- this argument holds the part of code that will - be ran to check the body of the for loop. It should be passed - as a lambda expression or a function. The functions that are - ran should be other pythonwhat test functions, and they will - be tested specifically on only the body of the for loop. - - * **orelse** -- this argument holds the part of code that will - be ran to check the else part of the for loop. It should be - passed as a lambda expression or a function. The functions - that are ran should be other pythonwhat test functions, and - they will be tested specifically on only the else part of the - for loop. - - * **expand_message** (*bool*) -- if true, feedback messages - will be expanded with *in the ___ of the for loop on line - ___*. Defaults to True. If False, *test_for_loop()* will - generate no extra feedback. - - -[ Examples ]- - - Student code - - "for i in range(10):" - "print(i)" - - Solution code - - "for n in range(10):" - "print(n)" - - SCT - - "test_for_loop(1," - "for_iter = lamdba: test_function("range")," - "body = lambda: test_expression_output(context_val = [5])" - - This SCT will evaluate to True as the function *"range"* is used in - the sequence and the function *test_exression_output()* will pass - on the body code. - - -test_function_definition -======================== - -pythonwhat.test_function_definition.test_function_definition(name, arg_names=True, arg_defaults=True, body=None, results=None, outputs=None, not_called_msg=None, nb_args_msg=None, arg_names_msg=None, arg_defaults_msg=None, wrong_result_msg=None, wrong_output_msg=None, expand_message=True) - - Test a function definition. - - This function helps you test a function definition. Generally four - things can be tested: - 1. The argument names of the function (including if the - correct defaults are used) - - 2. The body of the functions (does it output correctly, are - the correct functions used) - - 3. The return value with a certain input - - 4. The output value with a certain input - - Custom feedback messages can be set for all these parts, default - messages are generated automatically if none are set. - - Parameters: - * **name** (*str*) -- the name of the function definition to - be tested. - - * **arg_names** (*bool*) -- if True, the argument names will - be tested, if False they won't be tested. Defaults to True. - - * **arg_defaults** (*bool*) -- if True, the default values of - the arguments will be tested, if False they won't be tested. - Defaults to True. - - * **body** -- this arguments holds the part of the code that - will be ran to check the body of the function definition. It - should be passed as a lambda expression or a function. The - functions that are ran should be other pythonwhat test - functions, and they will be tested specifically on only the - body of the for loop. Defaults to None. - - * **results** (*list(tuple*) -- a list of tuples representing - arguments that should be passed to the defined function. These - arguments are passed to the function in the student - environment and the solution environment, the results (what's - returned) are compared. - - * **outputs** (*list(tuple*) -- a list of tuples representing - arguments that should be passed to the defined function. These - arguments are passed to the function in the student - environment and the solution environment, the outpus are - compared. - - * **not_called_msg** (*str*) -- message if the function is not - defined. - - * **nb_args_msg** (*str*) -- message if the number of - arguments do not matched. - - * **arg_names_msg** (*str*) -- message if the argument names - do not match. - - * **arg_defaults_msg** (*str*) -- message if the argument - default values do not match. - - * **wrong_result_msg** (*str*) -- message if one of the tested - function call's result did not match. - - * **wrong_output_msg** (*str*) -- message if one of the tested - functions call's output did not match. - - * **expand_message** (*bool*) -- only relevant if there is a - body test. If True, feedback messages defined in the body test - will be preceded by 'In your definition of ___, '. If False, - *test_function_definition()* will generate no extra feedback - if the body test fails. Defaults to True. - - -[ Examples ]- - - Student code - - "def shout( word, times = 3):" - "shout_word = not_word + '???'" - "print( shout_word )" - "return word * times" - - Solution code - - "def shout( word = 'help', times = 3 ):" - "shout_word = word + '!!!'" - "print( shout_word )" - "return word * times" - - SCT - - "test_function_definition('shout')": fail. - "test_function_definition('shout', arg_defaults = False)": pass. - "test_function_definition('shout', arg_defaults = False," - "outputs = [('help')])": fail. - "test_function_definition('shout', arg_defaults = False," - "results = [('help', 2)])": pass. - "test_function_definition('shout', args_defaults = False" - "body = lambda: test_function('print', args = []]))": pass. - - -test_function -============= - -pythonwhat.test_function.test_function(name, index=1, args=None, keywords=None, eq_condition='equal', do_eval=True, not_called_msg=None, incorrect_msg=None) - - Test if function calls match. - - This function compares a function call in the student's code with - the corresponding one in the solution code. It will cause the - reporter to fail if the corresponding calls do not match. The fail - message that is returned will depend on the sort of fail. - - Parameters: - * **name** (*str*) -- the name of the function to be tested. - - * **index** (*int*) -- index of the function call to be - checked. Defaults to 1. - - * **args** (*list(int*) -- the indices of the positional - arguments that have to be checked. If it is set to None, all - positional arguments which are in the solution will be - checked. - - * **keywords** (*list(str*) -- the indices of the keyword - arguments that have to be checked. If it is set to None, all - keyword arguments which are in the solution will be checked. - - * **eq_condition** (*str*) -- The condition which is checked - on the eval of the group. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **do_eval** (*bool*) -- Boolean representing whether the - group should be evaluated and compared or not. Defaults to - True. - - * **not_called_msg** (*str*) -- feedback message if the - function is not called. - - * **incorret_msg** (*str*) -- feedback message if the - arguments of the function in the solution doesn't match the - one of the student. - - Raises: - * "NameError" -- the eq_condition you passed is not "equal" or - "equivalent". - - * "NameError" -- function is not called in the solution - - -[ Examples ]- - - Student code - - "import numpy as np" - "np.mean([1,2,3])" - "np.std([2,3,4])" - - Solution code - - "import numpy" - "numpy.mean([1,2,3], axis = 0)" - "numpy.std([4,5,6])" - - SCT - - "test_function("numpy.mean", index = 1, keywords = [])": pass. - "test_function("numpy.mean", index = 1)": fail. - "test_function(index = 1, incorrect_op_msg = "Use the correct operators")": fail. - "test_function(index = 1, used = [], incorrect_result_msg = "Incorrect result")": fail. - - -test_if_else module -=================== - -pythonwhat.test_if_else.test_if_else(index=1, test=None, body=None, orelse=None, expand_message=True) - - Test parts of the if statement. - - This test function will allow you to extract parts of a specific if - statement and perform a set of tests specifically on these parts. A - for loop consists of three potential parts: the condition test, - *test*, which specifies the condition of the if statement, the - *body*, which is what's executed if the condition is True and a - else part, *orelse*, which will be executed if the condition is not - True. - - "if 5 == 3:" - "print("success")" - "else:" - "print("fail")" - - Has *5 == 3* as the condition test, *print("success")* as the body - and *print("fail")* as the else part. - - Parameters: - * **index** (*int*) -- index of the function call to be - checked. Defaults to 1. - - * **test** -- this argument holds the part of code that will - be ran to check the condition test of the if statement. It - should be passed as a lambda expression or a function - definition. The functions that are ran should be other - pythonwhat test functions, and they will be tested - specifically on only the condition test of the if statement. - - * **body** -- this argument holds the part of code that will - be ran to check the body of the if statement. It should be - passed as a lambda expression or a function definition. The - functions that are ran should be other pythonwhat test - functions, and they will be tested specifically on only the - body of the if statement. - - * **orelse** -- this argument holds the part of code that will - be ran to check the else part of the if statement. It should - be passed as a lambda expression or a function definition. The - functions that are ran should be other pythonwhat test - functions, and they will be tested specifically on only the - else part of the if statement. - - * **expand_message** (*bool*) -- if true, feedback messages - will be expanded with *in the ___ of the if statement on line - ___*. Defaults to True. If False, *test_if_else()* will - generate no extra feedback. - - -[ Examples ]- - - Student code - - "a = 12" - "if a > 3:" - "print('test %d' % a)" - - Solution code - - "a = 4" - "if a > 3:" - "print('test %d' % a)" - - SCT - - "test_if_else(1," - "body = lambda: test_expression_output(extra_env = { 'a': 5 }" - "incorrect_msg = "Print out the correct things"))" - - This SCT will pass as *test_expression_output()* is ran on the body - of the if statement and it will output the same thing in the - solution as in the student code. - - -test_import -=========== - -pythonwhat.test_import.test_import(name, same_as=True, not_imported_msg=None, incorrect_as_msg=None) - - Test import. - - Test whether an import statement is used the same in the student's - environment as in the solution environment. - - Parameters: - * **name** (*str*) -- the name of the package that has to be - checked. - - * **same_as** (*bool*) -- if false, the alias of the package - doesn't have to be the same. Defaults to True. - - * **not_imported_msg** (*str*) -- feedback message when the - package is not imported. - - * **incorrect_as_msg** (*str*) -- feedback message if the - alias is wrong. - - -[ Examples ]- - - Student code - - "import numpy as np" - "import pandas as pa" - - Solution code - - "import numpy as np" - "import pandas as pd" - - SCT - - "test_import("numpy")": pass. - "test_import("pandas")": fail. - "test_import("pandas", same_as = False)": pass. - - -test_mc -======= - -pythonwhat.test_mc.test_mc(correct, msgs) - - Test multiple choice exercise. - - Test for a MultipleChoiceExercise. The correct answer (as an - integer) and feedback messages are passed to this function. - - Parameters: - * **correct** (*int*) -- the index of the correct answer - (should be an instruction). Starts at 1. - - * **msgs** (*list(str*) -- a list containing all feedback - messages belonging to each choice of the - - * **The list should have the same length as the number of - instructions.** (*student.*) -- - - -test_object -=========== - -pythonwhat.test_object.test_object(name, eq_condition='equal', do_eval=True, undefined_msg=None, incorrect_msg=None) - - Test object. - - The value of an object in the ending environment is compared in the - student's environment and the solution environment. - - Parameters: - * **name** (*str*) -- the name of the object which value has - to be checked. - - * **eq_condition** (*str*) -- the condition which is checked - on the eval of the object. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **do_eval** (*bool*) -- if False, the object will only be - checked for existence. Defaults to True. - - * **undefined_msg** (*str*) -- feedback message when the - object is not defined - - * **incorrect_msg** (*str*) -- feedback message if the value - of the object in the solution environment doesn't match the - one in the student environment. - - -[ Examples ]- - - Student code - - "a = 1" - "b = 5" - - Solution code - - "a = 1" - "b = 2" - - SCT - - "test_object("a")": pass. - "test_object("b")": fail. - - -test_object_after_expression -============================ - -pythonwhat.test_object_after_expression.test_object_after_expression(name, extra_env=None, context_vals=None, undefined_msg=None, incorrect_msg=None, eq_condition='equal', pre_code=None, keep_objs_in_env=None) - - Test object after expression. - - The code of the student is ran in the active state and the the - value of the given object is compared with the value of that object - in the solution. This can be used in nested pythonwhat calls like - test_for_loop. In these kind of calls, the code of the active state - is set to the code in a part of the sub statement (e.g. the body of - a for loop). It has various parameters to control the execution of - the (sub)expression. This test function is ideal to check if a - value is updated correctly in the body of a for loop. - - Parameters: - * **name** (*str*) -- the name of the object which value has - to be checked after evaluation of the expression. - - * **extra_env** (*dict*) -- set variables to the extra - environment. They will update the student and solution - environment in the active state before the student/solution - code in the active state is ran. This argument should contain - a dictionary with the keys the names of the variables you want - to set, and the values are the values of these variables. - - * **context_vals** (*list*) -- set variables which are bound - in a for loop to certain values. This argument is only useful - if you use the function in a test_for_loop. It contains a list - with the values of the bound variables. - - * **incorrect_msg** (*str*) -- feedback message if the value - of the object in the solution environment doesn't match the - one in the student environment. This feedback message will be - expanded if it is used in the context of another test - function, like test_for_loop. - - * **eq_condition** (*str*) -- the condition which is checked - on the eval of the object. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **expr_code** (*str*) -- if this variable is not None, the - expression in the studeont/solution code will not be ran. - Instead, the given piece of code will be ran in the student as - well as the solution environment and the result will be - compared. - - * **pre_code** (*str*) -- the code in string form that should - be executed before the expression is executed. This is the - ideal place to set a random seed, for example. - - * **keep_obj_in_env** (*list(*) -- a list of variable names - that should be hold in the copied environment where the - expression is evaluated. All primitive types are copied - automatically, other objects have to be passed explicitely. - - -[ Examples ]- - - Student code - - "count = 1" - "for i in range(100):" - "count = count + i" - - Solution code - - "count = 15" - "for n in range(30):" - "count = count + n" - - SCT - - "test_for_loop(1," - "body = lambda: test_object_after_expression("count"," - "extra_env = { 'count': 20 }," - "contex_vals = [ 10 ])" - - This SCT will pass as the value of *count* is updated - identically in the body of the for loop in the student code and - solution code. - - -test_operator -============= - -pythonwhat.test_operator.test_operator(index=1, eq_condition='equal', used=None, do_eval=True, not_found_msg=None, incorrect_op_msg=None, incorrect_result_msg=None) - - Test if operator groups match. - - This function compares an operator group in the student's code with - the corresponding one in the solution code. It will cause the - reporter to fail if the corresponding operators do not match. The - fail message that is returned will depend on the sort of fail. We - say that one operator group correpsonds to a group of operators - that is evaluated to one value (e.g. 3 + 5 * (1/3)). - - Parameters: - * **index** (*int*) -- Index of the operator group to be - checked. Defaults to 1. - - * **eq_condition** (*str*) -- The condition which is checked - on the eval of the group. Can be "equal" -- meaning that the - operators have to evaluate to exactly the same value, or - "equivalent" -- which can be used when you expect an integer - and the result can differ slightly. Defaults to "equal". - - * **used** (*List[str]*) -- A list of operators that have to - be in the group. Valid operators are: "+", "-", "*", "/", "%", - "**", "<<", ">>", "|", "^", "&" and "//". If the list is None, - operators that are in the group in the solution have to be in - the student code. Defaults to None. - - * **do_eval** (*bool*) -- Boolean representing whether the - group should be evaluated and compared or not. Defaults to - True. - - * **not_found_msg** (*str*) -- Feedback message if not enough - operators groups are found in the student's code. - - * **incorrect_op_msg** (*str*) -- Feedback message if the - wrong operators are used in the student's code. - - * **incorrect_result_msg** (*str*) -- Feedback message if the - operator group evaluates to the wrong result in the student's - code. - - Raises: - * "NameError" -- the eq_condition you passed is not "equal" or - "equivalent". - - * "IndexError" -- not enough operation groups in the solution - environment. - - -[ Examples ]- - - Student code - - "1 + 5 * (3+5)" - "1 + 1 * 238" - - Solution code - - "3.1415 + 5" - "1 + 238" - - SCT - - "test_operator(index = 2, used = ["+"])": pass. - "test_operator(index = 2)": fail. - "test_operator(index = 1, incorrect_op_msg = "Use the correct operators")": fail. - "test_operator(index = 1, used = [], incorrect_result_msg = "Incorrect result")": fail. - - -test_output_contains -==================== - -pythonwhat.test_output_contains.test_output_contains(text, pattern=True, no_output_msg=None) - - Test the output. - - Tests if the output contains a (pattern of) text. - - Parameters: - * **text** (*str*) -- the text that is searched for - - * **pattern** (*bool*) -- if True, the text is treated as a - pattern. If False, it is treated as plain text. Defaults to - False. - - * **no_output_msg** (*str*) -- feedback message to be - displayed if the output is not found. - - -test_student_typed -================== - -pythonwhat.test_student_typed.test_student_typed(text, pattern=True, not_typed_msg=None) - - Test the student code. - - Tests if the student typed a (pattern of) text. - - Parameters: - * **text** (*str*) -- the text that is searched for - - * **pattern** (*bool*) -- if True, the text is treated as a - pattern. If False, it is treated as plain text. Defaults to - False. - - * **not_typed_msg** (*str*) -- feedback message to be - displayed if the student did not type the text. - - -test_while_loop module -====================== - -pythonwhat.test_while_loop.test_while_loop(index=1, test=None, body=None, orelse=None, expand_message=True) - - Test parts of the while loop. - - This test function will allow you to extract parts of a specific - while loop and perform a set of tests specifically on these parts. - A while loop generally consists of two parts: the condition test, - *test*, which is the condition that is tested each loop, and the - *body*. A for while can have a else part as well, *orelse*, but - this is almost never used. - - "a = 10" - "while a < 5:" - "print(a)" - "a -= 1" - - Has *a < 5* as the condition test and *print(i)* as the body. - - Parameters: - * **index** (*int*) -- index of the function call to be - checked. Defaults to 1. - - * **test** -- this argument holds the part of code that will - be ran to check the condition test of the while loop. It - should be passed as a lambda expression or a function - definition. The functions that are ran should be other - pythonwhat test functions, and they will be tested - specifically on only the condition test of the while loop. - - * **body** -- this argument holds the part of code that will - be ran to check the body of the while loop. It should be - passed as a lambda expression or a function definition. The - functions that are ran should be other pythonwhat test - functions, and they will be tested specifically on only the - body of the while loop. - - * **orelse** -- this argument holds the part of code that will - be ran to check the else part of the while loop. It should be - passed as a lambda expression or a function definition. The - functions that are ran should be other pythonwhat test - functions, and they will be tested specifically on only the - else part of the while loop. - - * **expand_message** (*bool*) -- if true, feedback messages - will be expanded with *in the ___ of the while loop on line - ___*. Defaults to True. If False, *test_for_loop()* will - generate no extra feedback. - - -[ Examples ]- - - Student code - - "a = 10" - "while a < 5:" - "print(a)" - "a -= 1" - - Solution code - - "a = 20" - "while a < 5:" - "print(a)" - "a -= 1" - - SCT - - "test_while_loop(1," - "test = lamdba: test_expression_result({"a": 5})," - "body = lambda: test_expression_output({"a": 5}))" - - This SCT will evaluate to True as condition test will have thes - same result in student and solution code and - *test_exression_output()* will pass on the body code. diff --git a/docs/articles/checking_compound_statements.rst b/docs/articles/checking_compound_statements.rst new file mode 100644 index 00000000..13c7022e --- /dev/null +++ b/docs/articles/checking_compound_statements.rst @@ -0,0 +1,159 @@ +Checking compound statements +---------------------------- + +As described in the `official Python documentation `_, +*compound statements contain (groups of) other statements; they affect or control the execution of those other statements in some way. +In general, compound statements span multiple lines, although in simple incarnations a whole compound statement may be contained in one line.* + +``if``, ``while``, ``for``, ``try``, and ``with`` statements are all examples of compounds statements, and pythonwhat contains functionality to check all of these, +- as well as function definitions, list and dictionary comprehensions, generator expressions and lambda functions - in a consistent fashion. + +Inner workings +============== + +The ``if`` statement example in the tutorial describes how different ``check_`` functions zoom into a specific part of a submission and solution, +producing a child state, to which additional SCT functions can be chained. The ``check_if_else()`` function scanned the code for an ``if`` statement, +and broke it into three parts: a ``test``, the ``body`` and the ``orelse`` part; the former two were dived into with the SCT functions ``check_test`` and ``check_body``. +Notice that the naming is consistent: the ``test`` part that ``check_if_else()`` surfaces can be inspected with ``check_test()``. +The ``body`` part that ``check_if_else()`` unearths can be inspected with ``check_body``. + +Similar to how ``if`` statements has a ``check_if_else`` associated with it, +all other compound statements have corresponding ``check_`` functions to perform this action of looking up a statement, +and chopping it up into its constituents that can be inspected with ``check_()``: + +- ``check_for_loop()`` will look for a ``for`` loop, an break it up into a ``iter``, ``body`` and ``orelse`` part, that can be zoomed in on using ``check_iter()``, ``check_body()`` and ``check_orelse()`` respectively. +- ``check_list_comp()`` will look for a list comprehension and break it up into a ``iter``, ``ifs`` and ``body`` part, that can be zoomed in on using ``check_iter()``, ``check_ifs()`` and ``check_body()`` respectively. +- etc. + +For specific examples on checking for loops, list comprehensions, function definitions etc., +visit the reference. Every function is documented with a full example and corresponding explanation. +All of these examples are specific to a single construct, but of course you can combine things up to crazy levels. + +Crazy combo, example 1 +====================== + +Suppose you want to check whether a function definition containing a for loop was coded correctly as follows: + +.. code:: + + def counter(lst, key): + count = 0 + for l in lst: + count += l[key] + return count + +The following SCT would robustly verify this: + +.. code:: + + Ex().check_function_def('counter').check_correct( + multi( + check_call("f([{'a': 1}], 'a')").has_equal_value(), + check_call("f([{'b': 1}, {'b': 2}], 'b')").has_equal_value() + ), + check_body().set_context([{'a': 1}, {'a': 2}], 'a').set_env(count = 0).check_for_loop().multi( + check_iter().has_equal_value(), + check_body().set_context({'a': 1}).has_equal_value(name = 'count') + ) + ) + +Some notes about this SCT: + +- ``check_correct()`` is used so the body is not further checked if calling the function in different ways produces the same value in both student and solution process. +- ``set_context()`` is used twice. Once to set the context variables introduced by the function definition, and once to set the context variable introducted by the for loop. +- ``set_env()`` had to be used to initialize ``count`` to a variable that was scoped only to the function definition. + + +Overview of all supported compound statements +============================================= + +The table below summarizes all checks that pythonwhat supports to test compound statements. + +- Code in all caps indicates the name of a piece of code that may be inspected using ``check_{part}``, + where ``{part}`` is replaced by the name in caps (e.g. ``check_if_else().check_test()``). +- If the statement produces context variables, these are referred to in the parts column and listed + in the context variables column. The names used are just to refer to which context variable comes + from where; you are totally free in naming your context variables. + + ++------------------------+------------------------------------------------------+-------------------+ +| check | parts | context variables | ++========================+======================================================+===================+ +|check_if_else() | .. code:: | | +| | | | +| | if TEST: | | +| | BODY | | +| | else: | | +| | ORELSE | | +| | | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_while() | .. code:: | | +| | | | +| | while TEST: | | +| | BODY | | +| | else: | | +| | ORELSE | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_list_comp() | .. code:: | ``i`` | +| | | | +| | [BODY for i in ITER if IFS[0] if IFS[1]] | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_generator_exp() | .. code:: | ``i`` | +| | | | +| | (BODY for i in ITER if IFS[0] if IFS[1]) | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_dict_comp() | .. code:: | ``k``, ``v`` | +| | | | +| | {KEY : VALUE for k, v in ITER if IFS[0]} | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_for_loop() | .. code:: | ``i``, ``j`` | +| | | | +| | for i, j in ITER: | | +| | BODY | | +| | else: | | +| | ORELSE | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_try_except() | .. code:: | ``e`` | +| | | | +| | try: | | +| | BODY | | +| | except BaseException as e: | | +| | HANDLERS['BaseException'] | | +| | except: | | +| | HANDLERS['all'] | | +| | else: | | +| | ORELSE | | +| | finally: | | +| | FINALBODY | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_with() | .. code:: | ``f`` | +| | | | +| | with CONTEXT[0] as f1, CONTEXT[1] as f2: | | +| | BODY | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_function_def('f') | .. code:: | argument names | +| | | | +| | def f(ARGS[0], ARGS[1]): | | +| | BODY | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_lambda_function() | .. code:: | argument names | +| | | | +| | lambda ARGS[0], ARGS[1]: BODY | | +| | | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ +|check_class_def('f') | .. code:: | | +| | | | +| | class KLS(BASES[0], BASES[1]): | | +| | BODY | | +| | | | ++------------------------+------------------------------------------------------+-------------------+ diff --git a/docs/articles/checking_function_calls.rst b/docs/articles/checking_function_calls.rst new file mode 100644 index 00000000..c7808a23 --- /dev/null +++ b/docs/articles/checking_function_calls.rst @@ -0,0 +1,381 @@ +Checking function calls +----------------------- + +Basic functionality +=================== + +Take the following example that checks whether a student used the ``round()`` function correctly: + +.. code:: + + # solution + round(2.718282, ndigits = 3) + + # sct + Ex().check_function("round").multi( + check_args("number").has_equal_value(), + check_args("ndigits").has_equal_value() + ) + + # submissions that pass: + round(2.718282, 3) + round(2.718282, ndigits = 3 + round(number=2.718282, ndigits=3) + round(ndigits=3, number=2.718282) + val=2.718282; dig=3; round(val, dig) + val=2.718282; dig=3; round(number=val, dig) + int_part = 2; dec_part = 0.718282; round(int_part + dec_part, 3) + + +- `check_function()` checks whether ``round()`` is called by the student, and parses all the arguments. +- ``check_args()`` checks whether a certain argument was specified, and zooms in on the expression used to specify that argument. +- ``has_equal_value()`` will rerun the expressions used to specify the arguments in both student and solution process, and compare the results. + +.. note:: + + In ``check_args()`` you can refer to the argument of a function call both by argument name and by position. + +Customizations +~~~~~~~~~~~~~~ + +If you only want to check the ``number`` parameter, just don't include a second chain with ``check_args("ndigits")``: + +.. code:: + + Ex().check_function("round").check_args("number").has_equal_value() + +If you only want to check whether the ``number`` parameter was specified, but not that it was specified correctly, drop ``has_equal_value()``: + +.. code:: + + Ex().check_function("round").check_args("number") + +If you just want to check whether the function was called, drop ``check_args()``: + +.. code:: + + Ex().check_function("round") + +If you want to compare the 'string versions' of the expressions used to set the arguments instead of the evaluated result of these expressions, +you can use ``has_equal_ast()`` instead of ``has_equal_value()``: + +.. code: + + Ex().check_function("round").multi( + check_args("number").has_equal_ast(), + check_args("ndigits").has_equal_value() + ) + +Now, the following submissions would fail: + +- ``val=2.718282; dig=3; round(val, dig)`` -- the string representation of ``val`` in the student code is compared to ``2.718282`` in the solution code. +- ``val=2.718282; dig=3; round(number=val, dig)`` -- same +- ``int_part = 2; dec_part = 0.718282; round(int_part + dec_part, 3)`` -- the string representation of ``int_part + dec_part`` in the student code is compered to ``2.718282`` in the solution code. + +As you can see, doing exact string comparison of arguments is not a good idea here, as it is very inflexible. +There are cases, however, where it makes sense to use this, e.g. when there are very big objects passed to functions, +and you don't want to spend the processing power to fetch these objects from the student and solution processes. + +Functions in packages +===================== + +If you're testing whether function calls of particular packages are used correctly, you should always refer to these functions with their 'full name'. +Suppose you want to test whether the function ``show`` of ``matplotlib.pyplot`` was called, use this SCT: + +.. code:: + + Ex().check_function("matplotlib.pyplot.show") + +``check_function()`` can handle it when a student used aliases for the python packages (all ``import`` and ``import * from *`` calls are supported). +If the student did not properly call the function, ``check_function()`` will automatically generate a feedback message that corresponds to how the student imported the modules/functions. + +.. note: + + No matter how you import the function, you always have to refer to the function with its full name, e.g. ``package.subpackage1.subpackage2.function``. + +has_equal_value? has_equal_ast? +=============================== + +In the customizations section above, you could already notice the difference between ``has_equal_value()`` and ``has_equal_ast()`` for checking +whether arguments are correct. The former **reruns** the expression used to specify the argument in both student and solution process +and compares their results, while the latter simply compares the expression's AST representations. Clearly, the former is more robust, but there +are some cases in which ``has_equal_ast()`` can be useful: + +- For better feedback. When using ``has_equal_ast()``, the 'expected x got y' message that is automatically generated when the arguments + don't match up will use the actual expressions used. ``has_equal_value()`` will use string representations of the evaluations of the expressions, + if they make sense, and this is typically less useful. +- To avoid very expensive object comparisons. If you are 100% sure that the object people have to pass as an argument is already correct (because + you checked it earlier in the SCT or because it was already specified in the pre exercise code) and doing an equality check on this object between + student and solution project is likely going to be expensive, then you can safely use ``has_equal_ast()`` to speed things up. +- If you want to save yourself the trouble of building exotic contexts. You'll often find yourself checking function calls in e.g. a for loop. + Typically, these function calls will use objects that were generated inside the loop. To easily unit test the body of a for loop, you'll typically + have to use ``set_context()`` and ``set_env()``. For exotic for loops, this can become tricky, and it might be a quick fix to be a little more + specific about the object names people should use, and just use ``has_equal_ast()`` for the argument comparison. That way, you're bypassing the need + to build up a context in the student/solution process and do object comparisions. + + +Signatures +========== + +The ``round()`` example earlier in this article showed that a student can call the function in a multitude of ways, +specifying arguments by position, by keyword or a mix of those. To be robust against this, pythonwhat uses the concept of argument binding. + +More specifically, each function has a function signature. Given this signature and the way the function was called, +argument binding can map each parameter you specified to an argument. This small demo fetches the signature of the ``open`` function and tries to +bind arguments that have been specified in two different ways. Notice how the resulting bound arguments are the same: + +.. code:: + + >>> import inspect + + >>> sig = inspect.signature(open) + + >>> sig + + + >>> sig.bind('my_file.txt', mode = 'r') + + + >>> sig.bind(file = 'my_file.txt', mode = 'r') + + + +When you're using ``check_args()`` you are actually selecting these bound arguments. +This works fine for functions like ``round()`` and ``open()`` that have a list of named arguments, +but things get tricky when dealing with functions that take ``*args`` and ``*kwargs``. + +``*args`` example +~~~~~~~~~~~~~~~~~ + +Python allows functions to take a variable number of unnamed arguments through ``*args``, like this function: + +.. code:: + + def multiply(*args): + res = 1 + for num in args: + res *= num + return res + +Let's see what happens when different calls are bound to their arguments: + +.. code:: + + >>> import inspect + + >>> inspect.signature(multiply) + + + >>> sig = inspect.signature(multiply) + + >>> sig + + + >>> sig.bind(1, 2) + + + >>> sig.bind(3, 4, 5) + + +Notice how now the list of arguments is grouped under a tuple with the name ``args`` in the bound arguments. +To be able to check each of these arguments individually, pythonwhat allows you to do repeated indexing in ``check_args()``. +Instead of specifying the name of an argument, you can specify a list of indices: + +.. code:: + + # solution to check against + multiply(2, 3, 4) + + # corresponding SCT + Ex().check_function("multiply").multi( + check_args(["args", 0]).has_equal_value(), + check_args(["args", 1]).has_equal_value(), + check_args(["args", 2]).has_equal_value() + ) + +The ``check_args()`` subchains each zoom in on a particular tuple element of the bound ``args`` argument. + +``**kwargs`` example +~~~~~~~~~~~~~~~~~~~~ + +Python allows functions to take a variable number of named arguments through ``**kwargs``, like this function: + +.. code:: + + def my_dict(**kwargs): + return dict(**kwargs) + +Let's see what happens when different calls are bound to their arguments: + +.. code:: + + >>> import inspect + + >>> sig = inspect.signature(my_dict) + + >>> sig.bind(a = 1, b = 2) + + + >>> sig.bind(c = 2, b = 3) + + +Notice how now the list of arguments is grouped under a dictionary name ``kwargs`` in the bound arguments. +To be able to check each of these arguments individually, pythonwhat allows you to do repeated indexing in ``check_args()``. +Instead of specifying the name of an argument, you can specify a list of indices: + +.. code:: + + # solution to check against + my_dict(a = 1, b = 2) + + # corresponding SCT + Ex().check_function("my_dict").multi( + check_args(["kwargs", "a"]).has_equal_value(), + check_args(["kwargs", "b"]).has_equal_value() + ) + +The ``check_args()`` subchains each zoom in on a particular dictionary element of the bound ``kwargs`` argument. + +Manual signatures +~~~~~~~~~~~~~~~~~ + +Unfortunately for a lot of Python's built-in functions no function signature is readily available because the function has been implemented in C code. +To work around this, pythonwhat already includes manually specified signatures for functions such as ``print()``, ``str()``, ``hasattr()``, etc, +but it's still possible that some signatures are missing. + +That's why ``check_function()`` features a ``signature`` parameter, that is ``True`` by default. +If pythonwhat can't retrieve a signature for the function you want to test, +you can pass an object of the class ``inspect.Signature`` to the ``signature`` parameter. + +Suppose, for the sake of example, that ``check_function()`` can't find a signature for the ``round()`` function. +In a real situation, you will be informed about a missing signature through a backend error. +To be able to implement this SCT, you can use the ``sig_from_params()`` function: + +.. code:: + + sig = sig_from_params(param("number", param.POSITIONAL_OR_KEYWORD), + param("ndigits", param.POSITIONAL_OR_KEYWORD, default=0)) + Ex().check_function("round", signature=sig).multi( + check_args("number").has_equal_value(), + check_args("ndigits").has_equal_value() + ) + +You can pass ``sig_from_params()`` as many parameters as you want. + +``param`` is an alias of the ``Parameter`` class that's inside the ``inspect`` module. +- The first argument of ``param()`` should be the name of the parameter, +- The second argument should be the 'kind' of parameter. ``param.POSITIONAL_OR_KEYWORD`` tells ``check_function`` that the parameter can be specified either through a positional argument or through a keyword argument. +Other common possibilities are ``param.POSITIONAL_ONLY`` and ``param.KEYWORD_ONLY`` (for a full list, refer to the `docs `_). +- The third optional argument allows you to specify a default value for the parameter. + +.. note:: + + If you find vital Python functions that are used very often and that are not included in pythonwhat by default, you can `let us know `_ and we'll add the function to our `list of manual signatures `_. + +Multiple function calls +======================= + +Inside ``check_function()`` the ``index`` argument (``0`` by default), becomes important when there are several calls of the same function. +Suppose that your exercise requires the student to call the ``round()`` function twice: once on ``pi`` and once on Euler's number: + +.. code:: + + # Call round on pi + round(3.14159, 3) + + # Call round on e + round(2.71828, 3) + +To test both these function calls, you'll need the following SCT: + +.. code:: + + Ex().check_function("round", 0).multi( + check_args("number").has_equal_value() + check_args("ndigits").has_equal_value() + ) + Ex().check_function("round", 1).multi( + check_args("number").has_equal_value() + check_args("ndigits").has_equal_value() + ) + +The first ``check_function()`` chain, where ``index=0``, looks for the first call of ``round()`` in both student solution code, +while ``check_funtion()`` with ``index=1`` will look for the second function call. After this, the rest of the SCT chain behaves as before. + +Methods +======= + +Methods are Python functions that are called on objects. For testing this, you can also use ``check_function()``. +Consider the following examples, that calculates the ``mean()`` of the column ``a`` in the pandas data frame ``df``: + +.. code:: + + # pec + import pandas as pd + df = pd.DataFrame({ 'a': [1, 2, 3, 4] }) + + # solution + df.a.mean() + + # sct + Ex().check_function('df.a.mean').has_equal_value() + ``` + +The SCT is checking whether the method ``df.a.mean`` was called in the student code, and whether rerunning the call in both student and solution process is returning the same result. + +As a more advanced example, consider this example of chained method calls: + +.. code:: + + # pec + import pandas as pd + df = pd.DataFrame({ 'type': ['a', 'b', 'a', 'b'], 'val': [1, 2, 3, 4] }) + + # solution + df.groupby('type').mean() + + # sct + Ex().check_function('df.groupby').check_args(0).has_equal_value() + Ex().check_function('df.groupby.mean', signature=sig_from_obj('df.mean')).has_equal_value() + +Here: + +- The first SCT is checking whether ``df.groupby()`` was called and whether the argument for ``df.groupby()`` was specified correctly to be ``'type'``. +- The second SCT is first checking whether ``df.groupby.mean()`` was called and whether calling it gives the right result. Notice several things: + + + We describe the entire chain of method calls, leaving out the parentheses and arguments used for method calls in between. + + We use ``sig_from_obj()`` to manually specify a Python expression that pythonwhat can use to derive the signature from. + If the string you use to describe the function to check evaluates to a method or function in the solution process, like for ``'df.groupby'``, + pythonwhat can figure out the signature. However, for ``'df.groupby.mean'`` will `not` evaluate to a method object in the solution process, + so we need to manually specify a valid expression that `will` evaluate to a valid signature with ``sig_from_obj()``. + +In this example, you are only checking whether the function is called and whether rerunning it gives the correct result. +You are not checking the actual arguments, so there's actually no point in trying to match the function call to its signature. +In cases like this, you can set ``signature=False``, which skips the fetching of a signature and the binding or arguments altogether: + +.. code:: + + # pec + import pandas as pd + df = pd.DataFrame({ 'type': ['a', 'b', 'a', 'b'], 'val': [1, 2, 3, 4] }) + + # solution + df.groupby('type').mean() + + # sct + Ex().check_function('df.groupby').check_args(0).has_equal_value() + Ex().check_function('df.groupby.mean', signature=False).has_equal_value() + +.. warning:: + + Watch out with disabling signature binding as a one-stop solution to make your SCT run without errors. + If there are arguments to check, argument binding makes sure that various ways of + calling the function can all work. Setting ``signature=False`` will skip this binding, which can + cause your SCT to mark perfectly valid student submissions as incorrect! + +.. note:: + + You can also use the ``sig_from_params()`` function to manually build the signature from scratch, + but this this more work than simply specifying the function object as a string from which to extract the signature. + + diff --git a/docs/articles/checking_through_string_matching.rst b/docs/articles/checking_through_string_matching.rst new file mode 100644 index 00000000..5781e8a3 --- /dev/null +++ b/docs/articles/checking_through_string_matching.rst @@ -0,0 +1,110 @@ +Checking through string matching +-------------------------------- + +has_code +======== + +With ``has_code()``, you can look through the student's submission to find a match with a search pattern you have specified. + +- If ``pattern = True``, the default, the ``text`` is used as a regular expression to match against. +- If ``pattern = False``, ``has_code()`` will consider the text you pass as an actual string that has to be found exactly. + +.. caution:: + + It is often tempting to use ``has_code()`` as it's straightforward to use, + but **you should avoid using this function**, as it imposes severe restrictions on how a student can solve an exercise. + Often, there are many different ways to solve an exercise. Unless you have a very advanced regular expression, + ``has_code()`` will not be able to accept all these different approaches. + Always think about better ways to test a student submission before you resort to ``has_code()``. + +Take the following example: + +.. code:: + + # solution + s = sum(range(10)) + + # sct that checks whether sum(range( is in the code + Ex().has_code("sum\s*\(\s*range\s*\(", not_typed_msg="You didn't use ``range()`` inside ``sum()``.") + +We also used ``not_typed_msg`` here to specify the feedback message shown to the student if ``has_code()`` doesn't pass. + +has_equal_ast +============= + +AST stands for `abstract syntax tree`; it is a way of representing the high-level structure of python code. +As the name suggests, ``has_equal_ast()`` verifies whether the code portion under consideration has the same AST representation in student and solution. +Compared to ``has_code()``, it is more robust to small syntactical details that are equivalent. + +- Quotes: the AST for ``x = "1"`` or ``x = '1'`` will be the same. +- Parentheses: Grouping by parentheses produces the same AST, when the same statement would work the same without them. + ``(True or False) and True``, and ``True or False and True``, are the same due to operator precedence. +- Spacing: ``x = 1`` or ``x = 1`` have the same AST. + +The AST does **not** represent is values that are found through evaluation. For example, the first item in the list in + +.. code:: + + x = 1 + [x, 2, 3] + +and + +.. code:: + + [1, 2, 3] + +Is not the same. In the first case, the AST represents that a variable ``x`` needs to be evaluated in order to find out what its value is. +In the second case, it just represents the value ``1``. + +.. caution:: + + Note that it is `not` a good idea to use ``Ex().has_equal_ast()``, effectively comparing the entire solution with the entire student submission. + It `is` a good idea, however, to use `has_equal_ast` for checking small excerpts of code when checking compound statements, + for example to inspect the test part of an ``if`` statement. + +As an example, consider this example that checks whether a student correclty coded a condition in a for loop (note that there are better ways to check this with ``has_equal_value()``!): + +.. code:: + + # solution + x = 3 + if x % 2 == 0: + print('x is even') + + # sct + Ex().check_if_else().multi( + check_test().has_equal_ast(), + check_body().has_equal_output() + ) + + # passing submission 1 + x = 3 + if (x % 2 == 0): + print('x is even') + + # passing submission 2 + x = 3 + if x%2==0: + print('x is even') + + # failing submission + x = 3 + if 0 == x % 2: + print('x is even') + +Here, the ``Ex().check_if_else().check_test()`` chain zooms in on the test part of the if statement. +With ``has_equal_ast()`` you are checking whether the AST representation of the test in the solution, ``x % 2 == 0`` is also found in the test specified by the student. +Notice that ``has_equal_ast()`` is not robust against a simple switching the order of the operands of the ``==`` operator. +A better SCT here would not use string (or AST) matching in the first place and rerun the test for different values of ```x``: + +.. code:: + + Ex().check_if_else().multi( + check_test().multi( + set_env(x = 3).has_equal_value(), + set_env(x = 4).has_equal_value(), + set_env(x = 4).has_equal_value() + ), + check_body().has_equal_output() + ) \ No newline at end of file diff --git a/docs/articles/electives.rst b/docs/articles/electives.rst new file mode 100644 index 00000000..9e577aed --- /dev/null +++ b/docs/articles/electives.rst @@ -0,0 +1,209 @@ +Electives +--------- + +Success message +=============== + +When all tests in an SCT pass, pythonwhat will automatically generate a congratulatory message to present to the student. If you want to override this 'success message', you can use the ``success_msg()`` function. + +.. code:: + + Ex().check_object("x").has_equal_value() + success_msg("You are a hero when it comes to variable assignment!") + + +`This article `_ on the authoring docs describes how to write good success messages. + + +Multiple choice exercises +========================= + +Multiple choice exercises are straightforward to test. +Use ``has_chosen()`` to provide tailored feedback for both the incorrect options, as the correct option. +Below is the markdown source for a multiple choice exercise example, with an SCT that uses ``has_chosen``: + +.. code-block:: none + + ## The author of Python + + ```yaml + type: MultipleChoiceExercise + ``` + + Who is the author of the Python programming language? + + `@instructions` + + - Roy Co + - Ronald McDonald + - Guido van Rossum + + `@sct` + + ```{python} + Ex().has_chosen(correct = 3, + msgs = ["That's someone who makes soups.", + "That's a clown who likes burgers.", + "Correct! Head over to the next exercise!"]) + ``` + +- ``correct`` specifies the number of the correct answer in this list (1-base indexed). +- ``msgs`` argument should be a list of strings with a length equal to the number of options. We encourage you to provide feedback messages that are informative and tailored to the (incorrect) option that people selected. + +Notice that there's no need for ``success_msg()`` in multiple choice exercises, as you have to specify the success message inside ``has_chosen()``, +along with the feedback for incorrect options. + +Capabilities of multi +===================== + +``multi()`` is always used to 'branch' different chains of SCT functions, so that the same state is passed to two sub-chains. There are different ways + +Comma separated arguments +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Most commonly, ``multi()`` is used to convert this code + +.. code:: + + Ex().check_if_exp().check_body().has_equal_value() + Ex().check_if_exp().check_test().has_equal_value() + + +into this equivalent (and more performant) SCT: + +.. code:: + + Ex().check_if_exp().multi( + check_body().has_equal_value(), + check_test().has_equal_value() + ) + +List or generator of subtests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Rather than one or more subtest args, multi can take a single list or generator of subtests. +For example, the code below checks that the body of a list comprehension has equal value +for 10 possible values of the iterator variable, ``i``. + +.. code:: + + Ex().check_list_comp() + .check_body() + .multi(set_context(i=x).has_equal_value() for x in range(10)) + +Chaining off multi +~~~~~~~~~~~~~~~~~~ + +Multi returns the same state, or focus, it was given, so whatever comes after multi will run +the same as if multi wasn't used. For example, the code below tests a list comprehension's body, +followed by its iterator. + +.. code:: + + Ex().check_list_comp() \ + .multi(check_body().has_equal_value()) \ + .check_iter().has_equal_value() + +has_context +=========== + +Tests whether context variables defined by the student match the solution, for a selected block of code. +A context variable is one that is defined in a looping or block statement. +For example, ``ii`` in the code below. + +.. code:: + + [ii + 1 for ii in range(3)] + +By default, the test fails if the submission code does not have the same number of context variables. +This is illustrated below. + +.. code:: + + # solution + # ii and ltr are context variables + for ii, ltr in enumerate(['a']): pass + + # sct + Ex().check_for_loop().check_body().has_context() + + # passing submission + # still 2 variables, just different names + for jj, Ltr in enumerate(['a']): pass + + # failing submission + # only 1 variable + for ii in enumerate(['a']): pass + +.. note:: + + If you use ``has_context(exact_names = True)``, then the submission must use the same names for the context variables, + which would cause the passing submission above to fail. + +set_context +=========== + +Sets the value of a temporary variable, such as ``ii`` in the list comprehension below. + +.. code:: + + [ii + 1 for ii in range(3)] + +Variable names may be specified using positional or keyword arguments. + + +Example +~~~~~~~ + +.. code:: + + # solution + ltrs = ['a', 'b'] + for ii, ltr in enumerate(ltrs): + print(ii) + + # sct + Ex().check_for_loop().check_body() \ + .set_context(ii=0, ltr='a').has_equal_output() \ + .set_context(ii=1, ltr='b').has_equal_output() + +Note that if a student replaced ``ii`` with ``jj`` in their submission, ``set_context`` would still work. +It uses the solution code as a reference. While we specified the target variables ``ii`` and ``ltr`` +by name in the SCT above, they may also be given by position.. + +.. code:: + + Ex().check_for_loop().check_body().set_context(0, 'a').has_equal_output() + +with_context +============ + +.. autofunction:: pythonwhat.checks.check_funcs.with_context + :noindex: + +Runs subtests after setting the context for a ``with`` statement. + +This function takes arguments in the same form as ``multi``. + +Context Managers Explained +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +With statements are special in python in that they enter objects called a context manager at the beginning of the block, +and exit them at the end. For example, the object returned by ``open('fname.txt')`` below is a context manager. + +.. code:: + + with open('fname.txt') as f: + print(f.read()) + +This code runs by + +1. assigning ``f`` to the context manager returned by ``open('fname.txt')`` +2. calling ``f.__enter__()`` +3. running the block +4. calling ``f.__exit__()`` + +``with_context`` was designed to emulate this sequence of events, by setting up context values as in step (1), +and replacing step (3) with any sub-tests given as arguments. + + diff --git a/docs/articles/expression_tests.rst b/docs/articles/expression_tests.rst new file mode 100644 index 00000000..23a502a6 --- /dev/null +++ b/docs/articles/expression_tests.rst @@ -0,0 +1,309 @@ +Expression tests +---------------- + +Expression tests run pieces of the student and solution code, and then check the resulting value, printed output, or errors they produce. + +``has_equal`` syntax +==================== + +Once student/submission code has been selected using a check function, we can run it using one of three functions. +They all take the same arguments, and run the student and submission code in the same way. +However, they differ in how they compare the outcome: + +* ``has_equal_value()`` - compares the value returned by the code. +* ``has_equal_output()`` - compares printed output. +* ``has_equal_error()`` - compares any errors raised. + +Basic Usage +=========== + +Running the whole code submission +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the example below, we re-run the entire student and submission code, and check that they print out the same output. + +.. code:: + + # solution + x = [1,2,3] + print(x) + + # sct + Ex().has_equal_output() + +Note that while we could have used ``has_output()`` to verify that the student printed ``"[1, 2, 3]"``, +using ``has_equal_output`` simply requires that the student output matches the solution output. + +Running part of the code +~~~~~~~~~~~~~~~~~~~~~~~~ + +Combining an expression test with part checks will run only a piece of the submitted code. +The example below first uses ``has_equal_value`` to run an entire if expression, and then to run only its body. + +.. code:: + + # solution + x = [1,2,3] + sum(x) if x else None + + # sct to test body of if expression + (Ex().check_if_exp() # focus on if expression + .has_equal_value() # run entire if expression, check value + .check_body() # focus on body "sum(x)" + .has_equal_value() # run body, check value + ) + +.. note:: + + Because ``has_equal_value()`` returns the exact same state as it was passed, + commands chaining off of ``has_equal_value`` behave as they would have if ``has_equal_value`` weren't used. + +Context Values +============== + +Suppose we want the student to define a function, that loops over the elements in a dictionary, and prints out each key and value, as follows: + +.. code:: + + # solution + def print_dict(my_dict): + for key, value in my_dict.items(): + print(key + " - " + str(value)) + +An appropriate SCT for this exercise could be the following (for clarity, we're not using any default messages): + +.. code:: + + # get for loop code, set context for my_dict argument + for_loop = (Ex() + .check_function_def('print_dict') # ensure 'print_dict' is defined + .check_body() # get student/solution code in body + .set_context(my_dict = {'a': 2, 'b': 3}) # set print_dict's my_dict arg + .check_for_loop() # ensure for loop is defined + ) + + # test for loop iterator + for_loop.check_iter().has_equal_value() # run iterator (my_dict.items()) + # test for loop body + for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value() + +Assuming the student coded the function in the exact same way as the solution, the following things happen: + +- checks whether ``print_dict`` is defined, then gets the code for the function definition body. +- because ``print_dict`` takes an argument ``my_dict``, which would be undefined if we ran the body code, ``set_context`` defines what ``my_dict`` should be when running the code. Note that its okay if the submitted code named the argument ``my_dict`` something else, since set_context matches submission / solution arguments up by position. + +When running the bottom two SCTs for the for_loop + +- ``for_loop.check_iter().has_equal_value()`` - runs the code for the iterator, ``my_dict.items()`` in the solution and its corresponding code in the submission, and compares the values they return. +- ``for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value()`` - runs the code in the for loop body, ``print(key + " - " + str(value))`` in the solution, and compares outputs. + Since this code may use variables the for loop defined, ``key`` and ``value``, we need to define them using ``set_context``. + +How are context values matched? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Context values are matched by position. For example, the submission and solution codes... + +.. code:: + + # solution + for ii, x in enumerate(range(3)): print(ii) + + # student submission + for jj, y in enumerate(range(3)): print(jj) + +Using ``Ex().check_for_loop().check_body().set_context(...)`` will do the following... + +====================== ======================= ========================== + statement solution (ii, x) submission (jj, y) +====================== ======================= ========================== +set_context(ii=1, x=2) ii = 1, x = 2 jj = 1, y = 2 +set_context(ii=1) ii = 1, x is undefined jj = 1, y is undefined +set_context(x=2) ii is undefined, x = 2 jj is undefined, y = 2 +====================== ======================= ========================== + +.. note:: + + If ``set_context`` does not define a variable, nothing is done with it. + This means that in the code examples above, running the body of the for loop would call print with ::ii:: or ::jj:: left at 2 (the values they have in the solution/submission environments). + +Context values for nested parts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Context values may now be defined for nested parts. For example, the print statement below, + +.. code:: + + for i in range(2): # outer for loop part + for j in range(3): # inner for loop part + print(i + j) + +may be tested by setting context values at each level, + +.. code:: + + (Ex() + .check_for_loop().check_body().set_context(i = 1) # outer for + .check_for_loop().check_body().set_context(j = 2) # inner for + .has_equal_output() + ) + + +``pre_code``: fixing mutations +=============================== + +Python code commonly mutates, or changes values within an object. +For example, the variable ``x`` points to an object that is mutated every time a function is called. + + +.. code:: + + x = {'a': 1} + + def f(d): d['a'] += 1 + + f(x) # x['a'] == 2 now + f(x) # x['a'] == 3 now + +In this case, when ``f`` is run, it changes the contents of ``x`` as a side-effect and returns None. +When using SCTs that run expressions, mutations in either the solution or submission environment can cause very confusing results. +For example, calling ``np.random.random()`` will advance numpy's random number generator. Consider the markdown source for an exercise that illustrates this. + +.. code-block:: none + + `@pre_exercise_code` + ```{python} + import numpy as np + np.random.seed(42) # set random generator seed to 42 + ``` + + `@solution` + ```{python} + if True: np.random.random() # 1st random call: .37 + + np.random.random() # 2nd random call: .95 + ``` + + `@sct` + ```{python} + # Should pass but fails, because random generator has advanced + # twice in solution, but only once in submission + Ex().check_if_else().check_body().has_equal_value() + ``` + +Assume this student submission: + +.. code:: + + if True: np.random.random() # 1st random call: .37 + + # forgot 2nd call to np.random.random() + + +In this situation the random seed is set to 42, but the solution code advances the random generator further than the submission code. +As a result the SCT will fail. In order to test random code, the random generator needs to be at the same state between submission and solution environments. +Since their generators can be thrown out of sync, the most reliable way to do this is to set the seed using the ``pre_code`` argument to ``has_equal_value``. +In the case above, the SCT may be fixed as follows + +.. code:: + + Ex().check_if_else().check_body().has_equal_value(pre_code = "np.random.seed(42)") + +More generally, it can be helpful to define a pre_code variable to use before expression tests... + +.. code:: + + pre_code = """ + np.random.seed(42) + """ + + Ex().has_equal_output(pre_code=pre_code) + Ex().check_if_else().check_body().has_equal_value(pre_code = pre_code) + + +``extra_env`` +============= + +As illustrated in the `Advanced part checking section `_ of the Checking compound statements article, +``set_env()`` (as a function) or ``extra_env`` (as an arugment) can be used to temporarily override the student and solution process to +run an expression in multiple situations. + +Setting extra environment variables is similar to ``pre_code``, in that you can (re)define objects in the student and submission environment before running an expression. +The difference is that, rather than passing a string that is executed in each environment, ``extra_env`` lets you pass objects directly. +For example, the three SCT chains below are equivalent... + +.. code:: + + Ex().has_equal_value(pre_code="x = 10") + Ex().set_env(x = 10).has_equal_value() + Ex().has_equal_value(extra_env = {'x': 10}) + +In practice they can often be used interchangably. +However, one area where ``extra_env`` may shine is in mocking up data objects before running tests. +For example, if the SCT below didn't use ``extra_env``, then it would take a long time to run. + +.. code:: + + `@pre_exercise_code` + ```{python} + a_list = list(range(10000000)) + ``` + + `@solution` + ```{python} + print(a_list[1]) + ``` + + `@sct` + ```{python} + Ex().set_env(a_list = list(range(10))).has_equal_output() + ``` + +The reason extra_env is important here, is that pythonwhat tries to make a deepcopy of lists, so that course developers don't get bit by unexpected mutations. +However, the larger the list, the longer it takes to make a deepcopy. +If an SCT is running slowly, there's a good chance it uses a very large object that is being copied for every expression test. + +``expr_code``: change expression +================================ + +The ``expr_code`` argument takes a string, and uses it to replace the code that would be run by an expression test. +For example, the markdown source for the following exercise simply runs ``len(x)`` in the solution and student environments. + +.. code:: + + `@solution` + ```{python} + # keep x the same length + x = [1,2,3] + ``` + + `@sct` + ```{python} + Ex().check_object('x').has_equal_value(expr_code="len(x)") + ``` + +.. note:: + + Using ``expr_code`` does not change how expression tests perform highlighting. + This means that ``Ex().for_loop().has_equal_value(expr_code="x[0]")`` would highlight the body of the checked for loop. + +``func``: Override the equality function +======================================== + +After running the expression in question, the ``has_equal_x`` function will compare the result/output/error of the expression using a built-in equality function. +This equality function is geared towards the types of objects you are trying to compare and does its job just fine in 99% of the cases. +However, there are cases where you want to customize the equality operation. To do this, you can set ``func`` to be function that takes two arguments and returns a boolean. + +Reiterating over the example from the ``expr_code`` section above, you can write an equivalent SCT with ``func`` instead of ``expr_code``: + +.. code:: + + `@solution` + ```{python} + # keep x the same length + x = [1,2,3] + ``` + + `@sct` + ```{python} + Ex().check_object('x').has_equal_value(func = lambda x, y: len(x) == len(y)) + ``` diff --git a/docs/articles/make_your_sct_robust.rst b/docs/articles/make_your_sct_robust.rst new file mode 100644 index 00000000..f3d94d50 --- /dev/null +++ b/docs/articles/make_your_sct_robust.rst @@ -0,0 +1,120 @@ +Make your SCT robust +-------------------- + +For larger exercises, you'll often want to be flexible: if students get the end result right, you don't want to be picky about how they got there. +However, when they do make a mistake, you want to be specific about the mistake they are making. +These seemingly conflicting requirements can be satisfied with ``check_correct()`` and ``check_or()``. + +``check_correct()`` +=================== + +To explain the concept of ``check_correct()``, consider this example: + +.. code:: + + # setup + import numpy as np + arr = np.array([1, 2, 3, 4, 5, 6]) + + # calculate result + result = np.mean(arr) + +You want the SCT to pass when the student manages to store the correct value in the object ``result``. +How ``result`` was calculated, does not matter to you: as long as ``result`` is correct, the SCT should accept the submission. +If something about ``result`` is not correct, you want to dig a little deeper and see if the student used the ``np.mean()`` function correctly. +The following SCT will do just that: + +.. code:: + + Ex().check_correct( + check_object("result").has_equal_value(), + check_function("numpy.mean").check_args("a").has_equal_value() + ) + + +Inside ``check_correct()``, two SCT chains are specified, separated by a comma: + +- A ``check`` chain, that has to pass in all cases, but when it fails, it doesn't immediately stop the SCT execution and fail the exercise. +- A ``diagnose`` chain, that is only execute if the ``check`` chain failed silently. + +In the example, we're checking the end value of ``result`` first. Only if this is not correct, will the ``check_function()`` chain be run, +to verify if the student used ``numpy.mean``. If the ``diagnose`` chain does not fail, the ``check``` chain is executed again 'loudly'. + +Let's see what happens in case of different student submissions: + +- The student submits ``result = np.mean(arr)`` + + - ``check_correct()`` runs the ``check_object()`` chain. + - This test passes, so ``check_correct()`` stops. + - The SCT passes. + +- The student submits ``result = np.sum(arr) / arr.size`` + + - ``check_correct()`` runs the ``check_object()`` chain. + - This test passes, so ``check_correct()`` stops before running ``check_function()``. + - The entire SCT passes even though ``np.mean()`` was not used. + +- The student submits ``result = np.mean(arr + 1)`` + + - ``check_correct()`` runs the ``check_object()`` chain. + - This test fails, so ``check_correct()`` continues with the ``diagnose`` part, running the ``check_function()`` chain. + - This chain fails, since the argument passed to ``numpy.mean()`` in the student submission does not correspond to the argument passed in the solution. + - A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside ``np.mean()``. + +- The student submits ``result = np.mean(arr) + 1`` + + - ``check_correct()`` runs the ``check_object()`` chain. + - This test fails, so ``check_correct()`` continues with the ``diagnose`` part, running the ``check_function()`` chain. + - This function passes, because ``np.mean()`` is called in exactly the same way in the student code as in the solution. + - Because there is something wrong - ``result`` is not correct - the ``check`` chain is executed again, and this time its feedback on failure is presented to the student. + - The student gets the message that ``result`` does not contain the correct value. + + +Multiple functions in ``diagnose`` and `check` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is perfectly possible for your ``check`` and ``diagnose`` SCT chains to branch out into different sub-branches with ``multi()``: + +.. code:: + + Ex().check_correct( + multi( + check_object('a').has_equal_value(), # multiple check SCTs + check_object('b').has_equal_value() + ), + check_function("numpy.mean").check_args("a").has_equal_value() + ) + + +Why use `check_correct()` +~~~~~~~~~~~~~~~~~~~~~~~~~ + +You will find that ``check_correct()`` is an **extremely powerful function&** to allow for different ways of solving the same problem. +You can use ``check_correct()`` to check the end result of a calculation. +If the end result is correct, you can go ahead and accept the entire exercise. +If the end result is incorrect, you can use the ``diagnose`` part of ``check_correct()`` to dig a little deeper. + +It is also perfectly possible to use ``check_correct()`` inside another ``check_correct()``. + +``check_or()`` +============== + +``check_or()`` tests whether one of the SCTs you specify inside it passes. Suppose you want to check whether people correctly printed out any integer between 3 and 7. A solution could be: + +.. code:: + + print(4) + + +To test this in a robust way, you could use ``has_code()`` with a suitable regular expression that covers everything, +or you can use ``check_or()`` with three separate ``has_code()`` functions: + +.. code:: + + Ex().check_or(has_code('4'), + has_code('5'), + has_code('6')) + +You can consider ``check_or()`` a logic-inducing function. The different calls to pythonwhat functions that are in your SCT are actually all tests that _have_ to pass: +they are ``AND`` tests. With ``check_or()`` you can add chunks of ``OR`` tests in there. + diff --git a/docs/articles/processes.rst b/docs/articles/processes.rst new file mode 100644 index 00000000..2823b5a4 --- /dev/null +++ b/docs/articles/processes.rst @@ -0,0 +1,95 @@ +Processes +--------- + +As explained on the `SCT authoring homepage `_, DataCamp's Python coding backends use uses two separate processes: one process to run the solution code, and one process to run the student's submission. +As such, pythonwhat has access to the 'ideal ending scenario' of an exercises, which in turn makes it easier to write SCTs. +Instead of having to specify which value an object should be, we can have pythonwhat look into the solution process and compare the object in that process with the object in the student process. + +Problem +======= + +Fetching Python objects or the results of running expressions inside a process is not straightforward. +To be able to pull data from a process, Python needs to 'dill' and 'undill' files: it converts the Python objects to a byte representation (dilling) that can be passed between processes, and then, inside the process that you want to work with the object, builds up the object from the byte representation again (undilling). + +For the majority of Python objects, this conversion to and from a byte representation works fine, but for some more complex objects, it doesn't. + +If you're writing an SCT with functions that require work in the solution process, such as ``has_equal_value()``, and you try it out in an exercise, it is possible that you'll get the following backend error: + +.. code-block:: none + + ... dilling inside process failed - write manual converter + ... undilling of bytestream failed - write manual converter + +The first error tells you that 'dilling' - converting the object to a bytestream representation - failed. +The second error tells you that 'undilling' - converting the byte representation back to a Python object - failed. +These errors will typically occur if you're dealing with exotic objects, such as objects that interface to files, connections to databases, etc. + +Solution +======== + +To be able to handle these errors, pythonwhat allows you to write your own converters for Python objects. +Say, for example, that you're writing an exercise to import Excel data into Python, and you're using the ``pandas`` package: + +.. code:: + + import pandas as pd + xl = pd.ExcelFile('battledeath.xlsx') + +This is the corresponding SCT: + + Ex().check_object('xl').has_equal_value() + +Suppose now that objects such as ``xl``, which are of the type ``pandas.io.excel.ExcelFile``, can't be properly dilled and undilled. +(Because of hardcoded converters inside pythonwhat, they can, see below). +To make sure that you can still use ``check_object('xl')`` to test the equality of the ``xl`` object between student and solution process, +you can manually define a converter with the ``set_converter()`` function. You can extend the SCT as follows: + +.. code:: + + def my_converter(x): + return(x.sheet_names) + set_converter(key = "pandas.io.excel.ExcelFile", fundef = my_converter) + Ex().check_object('xl').has_equal_value() + +With a lambda function, it's even easier: + +.. code:: + + set_converter(key = "pandas.io.excel.ExcelFile", fundef = lambda x: x.sheet_names) + Ex().check_object('xl').has_equal_value() + +The first arguemnt of ``set_converter()``, the ``key`` takes the type of the object you want to add a manual converter for as a string. +The second argument, ``fundef``, takes a function definition, taking one argument and returning a single object. This function definition converts the exotic object into something more standard. In this case, the function converts the object of type ``pandas.io.excel.ExcelFile`` into a simple list of strings. A list of strings is something that can easily be converted into a bytestream and back into a Python object again, hence solving the problem. + +If you want to reuse the same manual converter over different exercises, you'll have to use ``set_converter()`` in every SCT. + +Hardcoded converters +==================== + +Next to primitive classes like ``str``, ``int``, ``list``, ``dict``, ... and objects with a semantically correct implemenation of ``==``, there are also a bunch of often-used complex objects that don't have a proper implementation of ``==``. +For example, the result of calling ``.keys()`` and ``.items()`` on dictionaries can't be dilled and undilled without extra work. +To handle these common yet problematic situations, pythonwhat features a list of hardcoded converters, so that you don't have to manually specify them each time. +This list is `available in the source code `_. +Feel free to do a pull request if you want to add more converts to this list, which will reduce the amount of code duplication you have to do if you want to reuse the same converter in different exercises. + +Customize equality +================== + +The ``set_converter()`` function opens up possibilities for objects that can actually be dilled and undilled perfectly fine. +Say you want to test a ``numpy`` array, but you only want to check only if the dimensions of the array the student codes up match those in the solution process. +You can easily write a manual converter that overrides the typical dilling and undilling of Numpy arrays, implementing your custom equality behavior: + +.. code:: + + # solution + import numpy as np + my_array = np.array([[1,2], [3,4], [5,6]]) + + # sct + set_converter(key = "numpy.ndarray", fundef = lambda x: x.shape) + Ex().check_object('my_array').has_equal_value() + + # both these submissions will pass + my_array = np.array([[1,2], [3,4], [5,6]]) + my_array = np.array([[0,0], [0,0], [5,6]]) + diff --git a/docs/articles/single_process_exercise.rst b/docs/articles/single_process_exercise.rst new file mode 100644 index 00000000..e9a65a43 --- /dev/null +++ b/docs/articles/single_process_exercise.rst @@ -0,0 +1,106 @@ +SingleProcessExercise +--------------------- + +Introduction +============ + +Typical interactive exercises on DataCamp will be of the type ``NormalExercise`` or something similar. + +For these normal exercises, the pythonbackend (the Python package responsible +for running Python code that the student submitted) will execute: + +- the solution code in a solution process + (once, at exercise initialization), +- the student's submission in a student process + (every time the student hits submit, after which the process is restarted from scratch) +- the student's experimentation commands in the console in a console process + (every time the user executes a command, without restarting afterwards) + +These completely separate processes make sure that: + +- the different commands do not interfere with one another; + if you import a package in one process, + the package will not become available in the other process. +- pythonwhat has access to a 'target solution process' to easily do comparisons; + to compare an object ``x``, you simply have to use ``Ex().check_object('x').has_equal_value()`` and + pythonwhat will figure out the value ``x`` should have from the solution process. + +To learn more about how the backend works, you can visit +`this wiki article `_. + +Why does this exercise type exist? +================================== + +There are Python courses that make extensive use of programs running outside of Python. +Sometimes, these programs cannot handle it well when different +Python process are trying to interface with it. +An example of this is PySpark, where on container startup, +a Spark cluster is started up, that you can then interface with. +Things go horribly wrong if you try to access this PySpark cluster from different Python processes. + +To solve for this, a new exercise type was built, +that does not create three separate Python processes (solution, student, console). +Instead, only one process is created: + +- the solution code is not executed in this process. +- the student's sumission is executed in this process, but the process is not restarted afterwards +- the student's experimentation commands in the console are executed in the same process. + +From a user perspective, this shouldn't pose too much difficulties, +with the exception that the code execution is now stateful. + +So, what's the problem then? +============================ + +As mentioned earlier, pythonwhat depends heavily on the existence of +two separate process: a 'target' solution process, and a student process. +Functions such as ```has_equal_value()`` compare values and the results of expression in these processes. +In the ``SingleProcessExercise``, the student process and +the solution process are identical, it's one and the same process, +so these comparisons don't make any sense. + +Therefore, the 'process-based checks' in pythonwhat have to be used with care when +writing SCTs for a ``SingleProcessExercise``. More specifically: + +- ``check_object()`` should work okay, as there is some magic happening behind the scenes. +- ``has_equal_value()`` and ``has_equal_output()`` should be used with the ``override`` argument. + When this argument is specified, the expression that is 'zoomed in on' in the solution code will not be executed in the solution proces. + Instead, it will just take the value you pass to ``override`` to compare the result/output of the expression that is zoomed in on in the student code to. + +Example +======= + +As an example, suppose we want to check whether a student correctly created a list ``x``: + +.. code:: + + # solution + x = [1, 2, 3, 4, 5] + +If this solution were part of a traditional ``NormalExercise``, your SCT would be simple: + +.. code:: + + # SCT + Ex().check_object('x').has_equal_value() + +However, if this solution were part of a ``SingleProcessExercise``, the above SCT would not work. +Instead, you'll want to do the following: + +.. code:: + + # SCT + Ex().check_object('x').has_equal_value(override = [1, 2, 3, 4, 5]) + +Here, we use ``override`` to tell pythonwhat not to go look for the value of ``x`` in the solution process. +Instead, it uses the manually specified value in ``override`` to compare to. + +You can use ``override`` in combination with other arguments in ``has_equal_x()``, such as ``expr_code``. +Suppose you're only interested in the element at index 2 of the list ``x``: + +.. code:: + + # SCT + Ex().check_object('x').has_equal_value(expr_code = 'x[2]', override = 3) + +Tricky stuff, but it works! \ No newline at end of file diff --git a/docs/articles/test_to_check.rst b/docs/articles/test_to_check.rst new file mode 100644 index 00000000..d1fdb416 --- /dev/null +++ b/docs/articles/test_to_check.rst @@ -0,0 +1,138 @@ +Test to Check +------------- + +If you are looking at the SCTs of old DataCamp courses, you'll notice they use ``test_x()`` functions instead of ``check_x()`` functions, +and there is no usage of ``Ex()``. The ``test_x()`` way of doing things has now been phased out in favor of the more transparent and composable +``check_x()`` functions that start with ``Ex()`` and are chained together with the ``.`` operator. + +Common cases +============ + +Whenever you come across an SCT that uses ``test_x()`` functions, +you'll make everybody's life easier by converting it to a ``check_x()``-based SCT. +Below are the most common cases you will encounter, together with instructions on how to translate from one to the other. + +Something you came across that you didn't find in this list? +Just create an issue on GitHub. Content Engineering will explain how to translate the SCT and update this article. + +``test_student_typed`` +~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # Solution + y = 1 + 2 + 3 + + # old SCT + test_student_typed(r'1\s*\+2\s*\+3') + + # new SCT + Ex().has_code(r'1\s*\+2\s*\+3') + + +``test_object`` +~~~~~~~~~~~~~~~ + +.. code:: + + # Solution + x = 4 + + # old SCT (checks equality by default) + test_object('x') + + # new SCT + Ex().check_object('x').has_equal_value() + +.. code:: + + # Solution + x = 4 + + # old SCT + test_object('x', do_eval=False) + + # new SCT + Ex().check_object('x') + + +``test_function`` +~~~~~~~~~~~~~~~~~ + +.. code:: + + # Solution + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + # old SCT (checks all args specified in solution) + test_function('numpy.array') + + # new SCT + Ex().check_function('numpy.array').check_args('a').has_equal_value() + + +.. code:: + + # Solution + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + np.mean(arr + arr) + + # old SCT (1-based indexed) + test_function('numpy.array', index=1) + test_function('numpy.array', index=2) + + # new SCT (0-based indexed) + Ex().check_function('numpy.array', index=0).check_args('a').has_equal_value() + Ex().check_function('numpy.array', index=1).check_args('a').has_equal_value() + + +``test_function_v2`` +~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # Solution + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + # old SCT (explicitly specify args) + test_function_v2('numpy.array', params=['a'], index=1) + + # new SCT + Ex().check_function('numpy.array', index=0).check_args('a').has_equal_value() + +``test_correct`` +~~~~~~~~~~~~~~~~ + +.. code:: + + # Solution + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + + # old SCT (use lambdas to defer execution) + test_correct(lambda: test_object('arr'), + labmda: test_function('numpy.array')) + + # new SCT (no need for lambdas) + Ex().check_correct(check_object('arr').has_equal_value(), + check_function('numpy.array').check_args('a').has_equal_value()) + +Enforcing check functions +========================= + +For newer courses an updated version of the base Docker image is used that sets the ``PYTHONWHAT_V2_ONLY`` environment variable. +When this variable is set, pythonwhat will no longer allow sct authors to use the old ``test_x()`` functions. +If you are updating the Docker image for courses that have old skool SCTs, +this would mean you have to rewrite all the SCTs to their check equivalents to make the build pass. +If you want to work around this, thus being able to use test functions even with the latest base image, +you can include the following line of code in your ``requirements.sh`` file: + +.. code-block:: bash + + echo "import os; os.environ['PYTHONWHAT_V2_ONLY'] = '0'" > /home/repl/.startup.py \ No newline at end of file diff --git a/docs/articles/tutorial.rst b/docs/articles/tutorial.rst new file mode 100644 index 00000000..ba97d5b4 --- /dev/null +++ b/docs/articles/tutorial.rst @@ -0,0 +1,231 @@ +Tutorial +-------- + +pythonwhat uses the ``.`` to 'chain together' SCT functions. Every chain starts with the ``Ex()`` function call, which holds the exercise state. +This exercise state contains all the information that is required to check if an exercise is correct, which are: + ++ the student submission and the solution as text, and their corresponding parse trees. ++ a reference to the student process and the solution process. ++ the output and errors that were generated when executing the student code. + +As SCT functions are chained together with ``.``, the ``Ex()`` exercise state is copied and adapted into 'sub states' to zoom in on particular parts of the state. +Before this terminology blows your brains out, let's have a look at some basic examples. + +Example 1: output +================= + +Assume we want to robustly check whether a student correctly printed out a sentence: + +.. code:: + + print('hi, my name is DataCamp') + +The following SCT would do that: + +.. code:: + + Ex().has_output(r'[H|h]i,\s+my name is \w+') + +Let's see what happens when the SCT runs: + +- ``Ex()`` returns the 'root state', which considers the entire student submission and solution code, + a reference to the student and solution process, and the output and errors generated. +- ``has_output(r'')`` fetches the output the student generated from the root state and checks whether it can match the specified regular expression against it. + + + If the student had submitted ``print('Hi, my name is Filip')``, the regex will match, the SCT will pass, and the student is presented with a congratulatory message. + + If the student had submitted ``print('Hi,mynameis'_``, the regex will not have found a match, the SCT will fail, and pythonwhat will automatically generate a feedback message. + +Example 2: function call +======================== + +Assume we want to check whether a student correctly called the ``DataFrame`` function of the ``pandas`` package. + +.. code:: + + import pandas as pd + pd.DataFrame([1, 2, 3]) + +The following SCT would do that: + +.. code:: + + Ex().check_function('pandas.DataFrame').check_args('data').has_equal_value() + +Assume the student submits the following (incorrect) script: + +.. code:: + + import pandas as pd + pd.DataFrame([1, 2, 3, 4]) + +Let's see what happens when the SCT runs: + +- ``Ex()`` returns the 'root state', which considers the entire student submission and solution code: + + .. code:: + + # solution + import pandas as pd + pd.DataFrame([1, 2, 3]) + + # student + import pandas as pd + pd.DataFrame([1, 2, 3, 4]) + +- ``check_function('pandas.DataFrame')`` continues from the root state (considering the entire student submission and solution), + and looks for a call of ``pd.DataFrame`` in both. It finds them, and 'zooms in' on the arguments. + In simplified terms, this is the state that ``check_function()`` produces: + + .. code:: + + # solution args + { "data": [1, 2, 3] } + + # student arg + { "data": [1, 2, 3, 4] } + +- ``check_args('data')`` continues from the state produced by ``check_function()`` and looks for the ``"data"`` argument in both the student and solution arguments. + It finds it in both and produces a state that zooms in on the expression used to specify this argument: + + .. code:: + + # solution expression for data arg + [1, 2, 3] + + # student expression for data arg + [1, 2, 3, 4] + +- Finally, ``has_equal_value()`` takes the state produced by ``check_args()``, + executes the student and solution expression in their respective processes, and verifies if they give the same result. + In this example, the results of the expressions don't match: a 3-element array vs a 4-element array. + Hence, the SCT fails and automatically generates a meaningful feedback message. + +Example 3: if statement +======================= + +As a more advanced example, assume we want to check that the student coded up an `if` statement correctly: + +.. code:: + + x = 4 + if x > 0: + print("x is strictly positive") + +The following SCT would do that: + +.. code:: + + Ex().check_if_else().multi( + check_test().has_code(r'x\s+>\s+0'), # chain A + check_body().check_function('print').check_args(0).has_equal_value() # chain B + ) + +Notice how this time, ``multi()`` is used to have the SCT chains 'branch out'; +both ``check_body()`` and ``check_test()`` continue from the state produced by ``check_if_else()``. + +Case 1 +~~~~~~ + +In the first case, assume the following incorrect student submission: + +.. code:: + + x = 4 + if x < 0: + print("x is negative") + +In chain A, this is what happens: + +- ``check_if_else()`` considers the entire submission received from ``Ex()``, + looks for the first if-else statement in both student and solution code, + and produces a child state that zooms in on onlty these ``if`` statements: + + .. code:: + + # solution + if x > 0: + print("x is strictly positive") + + # student + if x < 0: + print("x is negative") + +- ``check_test()`` considers the state above produced by ``check_if_else()`` + and produces a child state that zooms in on the condition parts of the ``if`` statements: + + .. code:: + + # solution + x > 0 + + # student + x < 0 + +- ``has_code()`` considers the state above produced by ``check_test()`` + and tries to match the regexes to the ``x < 0`` student snippet. The regex does not match, so the test fails. + +Case 2 +~~~~~~ + +Assume now that the student corrects the mistake and submits the following (which is still not correct): + +.. code:: + + x = 4 + if x > 0: + print("x is negative") + +Chain A will go through the same steps and will pass this time as ``x > 0`` in the student submission now matches the regex. In Chain B: + +- ``check_body()`` considers the state produced by ``check_if_else()``, and produces a child state that zooms in on the body parts of the ``if`` statements: + + .. code:: + + # solution + print("x is strictly positive") + + # student + print("x is negative") + +- ``check_function()`` considers the state above produced by ``check_if_else()``, and tries to find the function ``print()``. + Next, it produces a state that refers to the different function arguments and the expressions used to specify them: + + .. code:: + + # solution + { "value": "x is strictly positive" } + + # student + { "value": "x is negative" } + +- ``check_args(0)`` looks for the first argument in the state produced by ``check_function()`` and produces a child state that zooms in on the expressions for the ``value`` argument: + + .. code:: + + # solution + "x is strictly positive" + + # student + "x is negative" + +- Finally, ``has_equal_value()`` takes the state produced by ``check_args()``, + executes the student and solution expression in their respective processes, and verifies if they give the same result. + The result of executing ``"x is strictly positive"`` and ``"x is negative"`` don't match so the SCT fails. + +.. caution:: + + We strongly advise against using ``has_code()`` to verify the correctness of excerpts of a student submission. + Visit the 'checking compount statements' article to take a deeper dive. + + +What is good feedback? +====================== + +For larger exercises, you'll often want to be flexible: if students get the end result right, you don't want to be picky about how they got there. +However, when they do make a mistake, you want to be specific about the mistake they are making. In other words, a good SCT is robust against different ways of solving a problem, but specific when something's wrong. + +These seemingly conflicting requirements can be satisfied with ``check_correct()``. It is an **extremely powerful function** that should be used whenever it makes sense. +The `Make your SCT robust `_ article is highly recommended reading. + +For other guidelines on writing good SCTs, check out the 'How to write good SCTs' section on DataCamp's `general SCT documentation page `_. + diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 00000000..bace3095 --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +# +# pythonwhat documentation build configuration file, created by +# sphinx-quickstart on Thu Feb 23 11:22:59 2017. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import os +import sys + +import json +import pythonwhat + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. + +# enables finding custom extensions in docs directory +sys.path.insert(0, os.path.abspath(".")) + +sys.path.insert(0, os.path.abspath("..")) + +TEST_DATA_FILE = "test_data.json" + +if not os.path.isfile(TEST_DATA_FILE): + import subprocess + + subprocess.call("pytest", cwd=os.path.abspath(os.path.join(os.getcwd(), os.pardir))) + +with open(TEST_DATA_FILE, "r") as read_file: + test_data = json.load(read_file) + +jinja_contexts = {"test_ctx": {"test_data": test_data}} + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + "sphinx.ext.autodoc", + "sphinx.ext.napoleon", + "sphinxcontrib.jinja", +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ["_templates"] + +# The suffix(es) of source filenames. +source_suffix = ".rst" + +# The master toctree document. +master_doc = "index" + +# General information about the project. +project = "pythonwhat" +copyright = "2019, DataCamp" +author = "DataCamp" + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = pythonwhat.__version__ +# The full version, including alpha/beta/rc tags. +release = pythonwhat.__version__ + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = ["_build"] + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +add_module_names = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = "sphinx" + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = "sphinx_rtd_theme" + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = ['_static'] + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = "pythonwhatdoc" + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = {} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, "pythonwhat.tex", "pythonwhat Documentation", "DataCamp", "manual") +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [(master_doc, "pythonwhat", "pythonwhat Documentation", [author], 1)] + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + ( + master_doc, + "pythonwhat", + "pythonwhat Documentation", + author, + "pythonwhat", + "One line description of project.", + "Miscellaneous", + ) +] diff --git a/docs/data_to_doc.py b/docs/data_to_doc.py new file mode 100644 index 00000000..d6d94f99 --- /dev/null +++ b/docs/data_to_doc.py @@ -0,0 +1,13 @@ +def render_data(app, docname, source): + """ + Render our pages as a jinja template for fancy templating goodness. + """ + if app.builder.format != "html": + return + src = source[0] + rendered = app.builder.templates.render_string(src, app.config.html_context) + source[0] = rendered + + +def setup(app): + app.connect("source-read", render_data) diff --git a/docs/environment.yml b/docs/environment.yml deleted file mode 100644 index 1c9f3539..00000000 --- a/docs/environment.yml +++ /dev/null @@ -1,7 +0,0 @@ -name: py35 -dependencies: -- python=3.5.1=0 -- sphinx>1.4.0 -- sphinx_rtd_theme>=0.1.9 -- pip: - - recommonmark>=0.4.0 diff --git a/docs/glossary.rst b/docs/glossary.rst new file mode 100644 index 00000000..3633ae83 --- /dev/null +++ b/docs/glossary.rst @@ -0,0 +1,246 @@ +Glossary +-------- + +This article lists some example solutions. For each of these solutions, an SCT +is included, as well as some example student submissions that would pass and fail. In all of these, +a submission that is identical to the solution will pass. + +.. note:: + + These SCT examples are not golden bullets that are perfect for your situation. + Depending on the exercise, you may want to focus on certain parts of a statement, or be + more accepting for different alternative answers. + +Check object +~~~~~~~~~~~~ + +.. code:: + + # solution + x = 10 + + # sct + Ex().check_object('x').has_equal_value() + + # passing submissions + x = 5 + 5 + x = 6 + 4 + y = 4; x = y + 6 + + +Check function call +~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # solution + import pandas as pd + pd.DataFrame([1, 2, 3], columns=['a']) + + # sct + Ex().check_function('pandas.DataFrame')\ + .multi( + check_args('data').has_equal_value(), + check_args('columns').has_equal_value() + ) + + # passing submissions + pd.DataFrame([1, 1+1, 3], columns=['a']) + pd.DataFrame(data=[1, 2, 3], columns=['a']) + pd.DataFrame(columns=['a'], data=[1, 2, 3]) + +Check pandas chain (1) +~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # solution + import pandas as pd + df = pd.DataFrame([1, 2, 3], columns=['a']) + df.a.sum() + + # sct + Ex().check_function("df.a.sum").has_equal_value() + +Check pandas chain (2) +~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # pec + import pandas as pd + df = pd.DataFrame({'a': [1, 2, 3], 'b': ['x', 'x', 'y']}) + + # solution + df.groupby('b').sum() + + # sct + sig = sig_from_obj("df.groupby('b').sum") + Ex().check_correct( + # check if group by works + check_function("df.groupby.sum", signature = sig).has_equal_value(), + # check if group_by called correctly + check_function("df.groupby").check_correct( + has_equal_value(func = lambda x,y: x.keys == y.keys), + check_args(0).has_equal_value() + ) + ) + + # passing submissions + df.groupby('b').sum() + df.groupby(['b']).sum() + + # failing submissions + df # Did you call df.groupby()? + df.groupby('a') # arg of groupby is incorrect + df.groupby('b') # did you call df.groupby.sum()? + +Check pandas plotting +~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # pec + import matplotlib.pyplot as plt + import pandas as pd + import numpy as np + np.random.seed(42) + df = pd.DataFrame({'val': np.random.rand(300) }) + + # solution + df.val.plot(kind='hist') + plt.title('my plot') + plt.show() + plt.clf() + + # sct + Ex().check_or( + multi( + check_function('df.val.plot').check_args('kind').has_equal_value(), + check_function('matplotlib.pyplot.title').check_args(0).has_equal_value() + ), + override("df.val.plot(kind='hist', title='my plot')").check_function('df.val.plot').multi( + check_args('kind').has_equal_value(), + check_args('title').has_equal_value() + ), + override("df['val'].plot(kind = 'hist'); plt.title('my plot')").multi( + check_function('df.plot').check_args('kind').has_equal_value(), + check_function('matplotlib.pyplot.title').check_args(0).has_equal_value() + ), + override("df['val'].plot(kind='hist', title='my plot')").check_function('df.plot').multi( + check_args('kind').has_equal_value(), + check_args('title').has_equal_value() + ) + ) + Ex().check_function('matplotlib.pyplot.show') + Ex().check_function('matplotlib.pyplot.clf') + + +Check object created through function call +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # pec + import numpy as np + arr = np.array([1, 2, 3, 4, 5, 6]) + + # solution + result = np.mean(arr) + + # sct + Ex().check_correct( + check_object("result").has_equal_value(), + check_function("numpy.mean").check_args("a").has_equal_value() + ) + + # passing submissions + result = np.mean(arr) + result = np.sum(arr) / arr.size + +Check DataFrame +~~~~~~~~~~~~~~~ + +.. code:: + + # solution + import pandas as pd + my_df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}) + + # sct + Ex().check_df("my_df").check_keys("a").has_equal_value() + + # passing submissions + my_df = pd.DataFrame({"a": [1, 1 + 1, 3], "b": [4, 5, 6]}) + my_df = pd.DataFrame({"b": [4, 5, 6], "a": [1, 2, 3]}) + +Check printout +~~~~~~~~~~~~~~ + +.. code:: + + # solution + x = 3 + print(x) + + # sct + Ex().has_printout(0) + + # passing submissions + print(3) + print(1 + 1) + x = 4; print(x - 1) + +Check output +~~~~~~~~~~~~ + +.. code:: + + # solution + print("This is weird stuff") + + # sct + Ex().has_output(r"This is \w* stuff") + + # passing submissions + print("This is weird stuff") + print("This is fancy stuff") + print("This is cool stuff") + + # failing submissions + print("this is weird stuff") + print("Thisis weird stuff") + +Check Multiple Choice +~~~~~~~~~~~~~~~~~~~~~ + +.. code:: + + # solution (implicit) + # 3 is the correct answer + + # sct + Ex().has_chosen(correct = 3, # 1-base indexed + msgs = ["That's someone who makes soups.", + "That's a clown who likes burgers.", + "Correct! Head over to the next exercise!"]) + +Check import +~~~~~~~~~~~~ + +`See has_import doc `_ + +Check if statement +~~~~~~~~~~~~~~~~~~ + +`See check_if_else doc `_ + +Check function definition +~~~~~~~~~~~~~~~~~~~~~~~~~ + +`See check_function_def doc `_ + +Check list comprehensions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +`See check_list_comp doc `_ diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 00000000..2a04317b --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,53 @@ +pythonwhat +---------- + +For an introduction to SCTs and how they use pythonwhat, visit the `README `_. + +This documentation features: + +- A glossary with typical use-cases and corresponding SCT constructs. +- Reference documentation of all actively maintained pythonwhat functions. +- A set of basic and advanced articles that gradually expose you to all of pythonwhat's functionality and best practices. + +If you are new to writing SCTs for Python exercises, start with the tutorial and work your way through the other basic articles. +The glossary is good to get a quick overview of how all functions play together after you have a basic understanding. +The reference docs become useful when you grasp all concepts and want to look up details on how to call certain functions and specify custom feedback messages. + +.. toctree:: + :maxdepth: 1 + :caption: Glossary + + glossary + +.. toctree:: + :maxdepth: 2 + + reference + +.. toctree:: + :maxdepth: 1 + :caption: Basic articles + + articles/tutorial.rst + articles/checking_function_calls.rst + articles/make_your_sct_robust.rst + articles/checking_through_string_matching.rst + +.. toctree:: + :maxdepth: 1 + :caption: Advanced articles + + articles/checking_compound_statements.rst + articles/expression_tests.rst + articles/processes.rst + articles/single_process_exercise.rst + articles/electives.rst + articles/test_to_check.rst + +.. toctree:: + :maxdepth: 1 + :caption: Tests + + tests + +For details, questions and suggestions, `contact us `_. diff --git a/docs/source/pythonwhat.wiki/test_object_accessed.md b/docs/old/test_object_accessed.md similarity index 100% rename from docs/source/pythonwhat.wiki/test_object_accessed.md rename to docs/old/test_object_accessed.md diff --git a/docs/reference.rst b/docs/reference.rst new file mode 100644 index 00000000..dc13556e --- /dev/null +++ b/docs/reference.rst @@ -0,0 +1,105 @@ +Reference +========= + +.. note:: + + - ``check_`` functions typically 'dive' deeper into a part of the state it was passed. They are typically chained for further checking. + - ``has_`` functions always return the state that they were intially passed and are used at the 'end' of a chain. + +Objects +------- + +.. autofunction:: pythonwhat.checks.check_object.check_object +.. autofunction:: pythonwhat.checks.check_object.is_instance +.. autofunction:: pythonwhat.checks.check_object.check_df +.. autofunction:: pythonwhat.checks.check_object.check_keys + +Function calls +-------------- + +.. autofunction:: pythonwhat.checks.check_function.check_function +.. autofunction:: pythonwhat.checks.check_funcs.check_args + +Output +------ + +.. autofunction:: pythonwhat.checks.has_funcs.has_output +.. autofunction:: pythonwhat.checks.has_funcs.has_printout +.. autofunction:: pythonwhat.checks.has_funcs.has_no_error + +Code +---- + +.. autofunction:: pythonwhat.checks.has_funcs.has_code +.. autofunction:: pythonwhat.checks.has_funcs.has_import + +has_equal_x +----------- + +.. autofunction:: pythonwhat.checks.has_funcs.has_equal_value +.. autofunction:: pythonwhat.checks.has_funcs.has_equal_output +.. autofunction:: pythonwhat.checks.has_funcs.has_equal_error +.. autofunction:: pythonwhat.checks.has_funcs.has_equal_ast + +Combining SCTs +-------------- + +.. autofunction:: protowhat.checks.check_logic.multi +.. autofunction:: protowhat.checks.check_logic.check_correct +.. autofunction:: protowhat.checks.check_logic.check_or +.. autofunction:: protowhat.checks.check_logic.check_not + +Function/Class/Lambda definitions +--------------------------------- + +.. autofunction:: pythonwhat.checks.check_wrappers.check_function_def +.. autofunction:: pythonwhat.checks.has_funcs.has_equal_part_len +.. autofunction:: pythonwhat.checks.check_funcs.check_call +.. autofunction:: pythonwhat.checks.check_wrappers.check_class_def +.. autofunction:: pythonwhat.checks.check_wrappers.check_lambda_function + +Control flow +------------ + +.. autofunction:: pythonwhat.checks.check_wrappers.check_if_else +.. autofunction:: pythonwhat.checks.check_wrappers.check_try_except +.. autofunction:: pythonwhat.checks.check_wrappers.check_if_exp +.. autofunction:: pythonwhat.checks.check_wrappers.check_with + +Loops +----- + +.. autofunction:: pythonwhat.checks.check_wrappers.check_for_loop +.. autofunction:: pythonwhat.checks.check_wrappers.check_while +.. autofunction:: pythonwhat.checks.check_wrappers.check_list_comp +.. autofunction:: pythonwhat.checks.check_wrappers.check_dict_comp +.. autofunction:: pythonwhat.checks.check_wrappers.check_generator_exp + +State management +---------------- + +.. autofunction:: pythonwhat.checks.check_logic.override +.. autofunction:: pythonwhat.checks.check_logic.disable_highlighting +.. autofunction:: pythonwhat.checks.check_logic.set_context +.. autofunction:: pythonwhat.checks.check_logic.set_env + +Checking files +-------------- + +.. autofunction:: pythonwhat.checks.check_wrappers.check_file +.. autofunction:: pythonwhat.checks.check_wrappers.has_dir +.. autofunction:: pythonwhat.local.run + +Bash history checks +------------------- + +.. automodule:: protowhat.checks.check_bash_history + :members: + +Electives +--------- + +.. autofunction:: pythonwhat.checks.has_funcs.has_chosen +.. autofunction:: pythonwhat.test_exercise.success_msg +.. autofunction:: protowhat.checks.check_simple.allow_errors +.. autofunction:: protowhat.checks.check_logic.fail diff --git a/docs/source/Home.md b/docs/source/Home.md deleted file mode 100644 index c4bb782a..00000000 --- a/docs/source/Home.md +++ /dev/null @@ -1,45 +0,0 @@ -Home -==== - -At DataCamp we build tools to [learn data science](https://www.datacamp.com) interactively. See e.g. our online [R tutorial](https://www.datacamp.com/courses/free-introduction-to-r) to learn R Programming and our Python For Data Science tutorial to [learn Python](https://www.datacamp.com/courses/intro-to-python-for-data-science). - -pythonwhat? ------------ - -A major part of DataCamp's interactive learning is centered around automated and meaningful feedback. When a student submits an incorrect answer, the system tells the student what he or she is doing wrong. This happens through so-called submission correctness tests, or SCTs. An SCT is a test script that compares the different steps in a student's submission to the ideal solution, and generates meaningful feedback along the way. - -`pythonwhat` is a Python package that can help you write these SCTs for interactive Python exercises on DataCamp. It allows you to easily compare parts in the student's submission with the solution code. `pythonwhat` provides a bunch of functions to test object definitions, function calls, function definitions, for loops, while loops, and many more. `pythonwhat` automatically generates meaningful feedback that's specific to the student's mistake; you can also choose to override this feedback with custom messages. - -Writing SCTs, for which `pythonwhat` is built, is only one part of creating DataCamp exercises. For general documentation on creating Python courses on DataCamp, visit the [Teach Documentation](https://www.datacamp.com/teach/documentation). To write SCTs for R exercises on DataCamp, have a look at [testwhat](https://github.com/datacamp/testwhat). - -How does it work? ------------------ - -When a student starts an exercise on DataCamp, a Python session is started and the `pre_exercise_code` (PEC) is run. This code, that the author specifies, initializes the Python workspace with data, loads relevant packages etc, such that students can start coding the essence of the topics treated. Next, a separate solution process is created, in which the same PEC and actual solution code, also coded by the author, is executed. - -When a student submits an answer, his or her submission is executed and the output is shown in the IPython Shell. Then, the correctness of the submission is checked by executing the Submission Correctness Test, or SCT. Basically, your SCT is a Python script with calls to `pythonwhat` test functions. `pythonwhat` features a variety of functions to test a user's submission in different ways; examples are `test_object()`, `test_function()` and `test_output_contains()`. To do this properly, `pythonwhat` uses several resources: - -- The student submission as text, to e.g. figure out which functions have been called. -- The solution code as text, to e.g. figure out whether the student called a particular function in the same way as it is called in the solution. -- The student process, where the student code is executed, to e.g. figure out whether a certain object was created. -- The solution process, where the solution code is executed, to e.g. figure out whether an object that the student created corresponds to the object that was created by the solution. -- The output that's generated when executing the student code, to e.g. figure out if the student printed out something. - -If, during execution of the SCT, a test function notices a mistake, an appropriate feedback will be generated and presented to the student. It is always possible to override these feedback messages with your own messages. Defining custom feedback will make your SCTs longer and they may be error prone (typos, etc.), but they typically give the exercise a more natural and personalized feel. - -If all test functions pass, a success message is presented to the student. `pythonwhat` has some messages in store from which it can choose randomly, but you can override this with the `success_msg()` function. - -Overview --------- - -To get started, make sure to check out the [Quickstart Guide](quickstart_guide.md). - -To robustly test the equality of objects, and results of evaluations, it has to fetch the information from the respective processes, i.e. the student and solution processes. By default, this is done through a process of 'dilling' and 'undilling', but it's also possible to define your own converters to customize the way objects and results are compared. For more background on this, check out the [Processes article](expression_tests.md). For some more background on the principle of 'sub-SCTs', i.e. sets of tests to be called on a particular part or a particular state of a student's submission, have a look at the [Part Checks article](part_checks.rst). - -The remainder of the wiki goes over every test function that `pythonwhat` features, explaining all arguments and covering different use cases. They will give you an idea of how, why and when to use them. - -For more full examples of SCTs for Python exercises on DataCamp, check out the [source files of the introduction to Python course](http://www.github.com/datacamp/courses-intro-to-python). In the chapter files there, you can can see the SCTs that have been written for several exercises. - -To test your understanding of writing SCTs for Python exercises on the DataCamp platform, you can take the course [Writing SCTs with pythonwhat](https://www.datacamp.com/courses/writing-scts-with-pythonwhat) course. - -After reading through this documentation, we hope writing SCTs for Python exercises on DataCamp becomes a painless experience. If this is not the case and you think improvements to `pythonwhat` and this documentation are possible, [please let us know](mailto:content-engineering@datacamp.com)! diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css deleted file mode 100644 index 24899663..00000000 --- a/docs/source/_static/theme_overrides.css +++ /dev/null @@ -1,3 +0,0 @@ -code.docutils.literal { - color: black; -} diff --git a/docs/source/conf.py b/docs/source/conf.py deleted file mode 100644 index 510bc33d..00000000 --- a/docs/source/conf.py +++ /dev/null @@ -1,318 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Pythonwhat documentation build configuration file, created by -# sphinx-quickstart on Sun May 22 17:26:27 2016. -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os - -# Mock heavy duty pythonwhat dependencies for autodoc -import mock - -MOCK_MODULES = ['numpy', 'pandas', 'dill', 'markdown2'] -for mod_name in MOCK_MODULES: - sys.modules[mod_name] = mock.Mock() -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath('../..')) - - -# -- General configuration ------------------------------------------------ - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.3.5' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.napoleon', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo' -] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -from recommonmark.parser import CommonMarkParser -source_parsers = { - '.md': CommonMarkParser - } -source_suffix = ['.rst', '.md'] - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'Pythonwhat' -copyright = u'2016, Vincent Vankrunkelsven' -author = u'Vincent Vankrunkelsven' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = u'1.2' -# The full version, including alpha/beta/rc tags. -release = u'1.2.0' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['build'] - -# The reST default role (used for this markup: `text`) to use for all -# documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True - - -# -- Options for HTML output ---------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -html_theme = 'default' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'Pythonwhatdoc' - -# -- Options for LaTeX output --------------------------------------------- - -latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -#'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -#'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -#'preamble': '', - -# Latex figure (float) alignment -#'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'Pythonwhat.tex', u'Pythonwhat Documentation', - u'Vincent Vankrunkelsven', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output --------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'pythonwhat', u'Pythonwhat Documentation', - [author], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'Pythonwhat', u'Pythonwhat Documentation', - author, 'Pythonwhat', 'One line description of project.', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - -# If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False - -add_module_names = False - -from recommonmark.transform import AutoStructify -def setup(app): - github_doc_root = 'https://github.com/datacamp/pythonwhat/blob/master/docs/source/' - app.add_config_value('recommonmark_config', { - 'url_resolver': lambda url: github_doc_root + url, - 'auto_toc_tree_section': 'Contents', - }, True) - app.add_transform(AutoStructify) - app.add_stylesheet('theme_overrides.css') - -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' -if not on_rtd: # only import and set the theme if we're building docs locally - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] - def setup(app): - app.add_stylesheet('theme_overrides.css') diff --git a/docs/source/expression_tests.md b/docs/source/expression_tests.md deleted file mode 100644 index 0ded5b4f..00000000 --- a/docs/source/expression_tests.md +++ /dev/null @@ -1,406 +0,0 @@ -Expressions -=========== - -Expression tests run pieces of the student and solution code, and then check the resulting value, printed output, or errors they produce. - -`has_equal` Syntax ------------------- - -Once student/submission code has been selected using a check test, we can run it using one of three functions. -They all take the same arguments, and run the student and submission code in the same way. -However, they differ in how they compare the outcome: - -* has_equal_value - compares the value returned by the code. -* has_equal_output - compares printed output. -* has_equal_error - compares any errors raised. - -```eval_rst -.. autofunction:: pythonwhat.check_funcs.has_expr -``` - -### Basic Usage - -#### Running the whole code submission - -In the example below, we re-run the entire student and submission code, and check that they print out the same output. - - - *** =solution - ```{python} - x = [1,2,3] - print(x) - ``` - - *** =sct - ```{python} - # run all code and compare output - Ex().has_equal_output() - # equivalent to - # Ex().test_output_contains('[1,2,3]') - ``` - -Note that while we could have used `test_output_contains` to verify that the student printed `"[1, 2, 3]"`, -using `has_equal_output` simply requires that the student output matches the solution output. - -#### Running part of the code - -Combining an expression test with part checks will run only a piece of the submitted code. -The example below first uses `has_equal_value` to run an entire if expression, and then to run only its body. - - *** =solution - ```{python} - x = [1,2,3] - sum(x) if x else None - ``` - - *** =sct - ```{python} - # test body of if expression - (Ex().check_if_exp(0) # focus on if expression - .has_equal_value() # run entire if expression, check value - .check_body() # focus on body "sum(x)" - .has_equal_value() # run body, check value - ) - ``` - -Note that commands chaining off of `has_equal_value` behave as they would have if `has_equal_value` weren't used. -In this sense, the `check_body` behaves the same in - -```python -Ex().check_if_exp(0).has_equal_value().check_body() -``` - -and - -```python -Ex().check_if_exp(0).check_body() -``` - -in that it gets "sum(x)" in the solution code (and its corresponding code in the submission). - -### Context Values - -Suppose we want the student to define a function, that loops over the elements in a dictionary, and prints out each key and value, as follows: - - *** =solution - ```{python} - def print_dict(my_dict): - for key, value in my_dict.items(): - print(key + " - " + str(value)) - ``` - -An appropriate SCT for this exercise could be the following (for clarity, we're not using any default messages): - - *** =sct - ```{python} - # get for loop code, set context for my_dict argument - for_loop = (Ex() - .check_function_def('print_dict') # ensure 'print_dict' is defined - .check_body() # get student/solution code in body - .set_context(my_dict = {'a': 2, 'b': 3}) # set print_dict's my_dict arg - .check_for_loop(0) # ensure for loop is defined - ) - - # test for loop iterator - for_loop.check_iter().has_equal_value() # run iterator (my_dict.items()) - # test for loop body - for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value() - - ``` - -Assuming the student coded the function in the exact same way as the solution, the following things happen: - -- checks whether `print_dict` is defined, then gets the code for the function definition body. -- because `print_dict` takes an argument `my_dict`, which would be undefined if we ran the body code, `set_context` defines what `my_dict` should be when running the code. Note that its okay if the submitted code named the argument `my_dict` something else, since set_context matches submission / solution arguments up by position. - -When running the bottom two SCTs for the for_loop - -- `for_loop.check_iter().has_equal_value()` - runs the code for the iterator, `my_dict.items()` in the solution and its corresponding code in the submission, and compares the values they return. -- `for_loop.check_body().set_context(key = 'c', value = 3).has_equal_value()` - runs the code in the for loop body, `print(key + " - " + str(value))` in the solution, and compares outputs. - Since this code may use variables the for loop defined, `key` and `value`, we need to define them using `set_context`. - -### How are Context Values Matched? - -Context values are matched by position. For example, the submission and solution codes... - - *** =solution - ```{python} - for ii, x in enumerate(range(3)): print(ii) - ``` - - *** =submission - ```{python} - for jj, y in enumerate(range(3)): print(jj) - ``` - -Using `Ex().check_for_loop(0).check_body().set_context(...)` will do the following... - -```eval_rst -====================== ======================= ========================== - statement solution (ii, x) submission (jj, y) -====================== ======================= ========================== -set_context(ii=1, x=2) ii = 1, x = 2 jj = 1, y = 2 -set_context(ii=1) ii = 1, x is undefined jj = 1, y is undefined -set_context(x=2) ii is undefined, x = 2 jj is undefined, y = 2 -====================== ======================= ========================== - -.. note:: - - If ::set_context:: does not define a variable, nothing is done with it. - This means that in the code examples above, running the body of the for loop would call print with ::ii:: or ::jj:: left at 2 (the values they have in the solution/submission environments). -``` - -### pre_code: fixing mutations - -Python code commonly mutates, or changes values within an object. -For example, the variable `x` points to an object that is mutated every time a function is called. - -```python -x = {'a': 1} - -def f(d): d['a'] += 1 - -f(x) # x['a'] == 2 now -f(x) # x['a'] == 3 now -``` - -In this case, when `f` is run, it changes the contents of `x` as a side-effect and returns None. -When using SCTs that run expressions, mutations in either the solution or submission environment can cause very confusing results. -For example, calling `np.random.random()` will advance numpy's random number generator. -In the code below the random seed is set to 42, but the solution code advances the random generator further than the submission code. As a result the SCT will fail. - - *** =pre_exercise_code - ```{python} - import numpy as np - np.random.seed(42) # set random generator seed to 42 - ``` - - *** =solution - ```{python} - if True: np.random.random() # 1st random call: .37 - - np.random.random() # 2nd random call: .95 - ``` - - *** =submission - ```{python} - if True: np.random.random() # 1st random call: .37 - - # forgot 2nd call to np.random.random() - ``` - - *** =sct - ```{python} - # Should pass but fails, because random generator has advanced - # twice in solution, but only once in submission - Ex().check_if_else(0).check_body().has_equal_value() - ``` - -In order to test random code, the random generator needs to be at the same state between submission and solution environments. -Since their generators can be thrown out of sync, the most reliable way to do this is to set the seed using the `pre_code` argument to `has_equal_value`. -In the case above, the sct may be fixed as follows - - *** =sct - ```{python} - Ex().check_if_else(0).check_body().has_equal_value(pre_code = "np.random.seed(42)") - ``` - -More generally, it can be helpful to define a pre_code variable to use before expression tests... - - *** =sct - ```{python} - pre_code = """ - np.random.seed(42) - """ - - Ex().has_equal_output(pre_code=pre_code) - Ex().check_if_else(0).check_body().has_equal_value(pre_code = pre_code) - ``` - -### extra_env: fixing slow SCTs - -The `extra_env` argument is similar to `pre_code`, in that you can (re)define objects in the student and submission environment before running an expression. -The difference is that, rather than passing a string that is executed in each environment, extra_env lets you pass objects directly. -For example, the two SCTs below are equivalent... - - *** =sct - ```{python} - Ex().has_equal_value(pre_code="x = 10") - Ex().has_equal_value(extra_env = {'x': 10}) - ``` - -In practice they can often be used interchangably. -However, one area where `extra_env` may shine is in mocking up data objects before running tests. -For example, if the SCT below didn't use extra_env, then it would take a long time to run. - - *** =pre_exercise_code - ```{python} - a_list = list(range(10000000)) - ``` - - *** =solution - ```{python} - print(a_list[1]) - ``` - - *** =sct - ```{python} - extra_env = {'a_list': list(range(10))} - Ex().has_equal_output(extra_env = extra_env) - ``` - -The reason extra_env is important here, is that pythonwhat tries to make a deepcopy of lists, so that course developers don't get bit by unexpected mutations. -However, the larger the list, the longer it takes to make a deepcopy. -If an SCT is running slowly, there's a good chance it uses a very large object that is being copied for every expression test. - -### name: run tests after expression - -### expr_code: change expression - -The `expr_code` argument takes a string, and uses it to replace the code that would be run by an expression test. -For example, the following SCT simply runs `len(x)` in the solution and student environments. - - *** =solution - ```{python} - # keep x the same length - x = [1,2,3] - ``` - - *** =SCT - ```{python} - Ex().has_equal_value(expr_code="len(x)") - ``` - -```eval_rst -.. note:: - - Using `expr_code` does not change how expression tests perform highlighting. - This means that `Ex().for_loop(0).has_equal_value(expr_code="x[0]")` would highlight the body of the checked for loop. -``` - -`call` Syntax -------------- - -Testing a function definition or lambda may require calling it with some arguments. -In order to do this, use the `call()` SCT. -There are two ways to tell it what arguments to pass to the function/lambda, - -* `call("f (1, 2, x = 3)")` - as a string, where `"f"` gets substituted with the function's name. -* `call([1,2,3])` - as a list of positional arguments. - -Below, two alternative ways of specifying the arguments to pass are shown. - - *** =solution - ```{python} - def my_fun(x, y = 4, z = ('a', 'b'), *args, **kwargs): - return [x, y, *z, *args] - ``` - - *** =sct - ```{python} - Ex().check_function_def('my_fun').call("f(1, 2, (3,4), 5, kw_arg='ok')") # as string - Ex().check_function_def('my_fun').call([1, 2, (3,4), 5]) # as list - ``` - -```eval_rst -.. note:: - - Technically, you can get crazy and replace the list approach with a dictionary of the form ``{'args': [POSARG1, POSARG2], 'kwargs': {KWARGS}}``. -``` - -### Additional Parameters - -In addition to its first argument, `call()` accepts all the parameters that the expression tests above can (i.e. `has_equal_value`, `has_equal_error`, `has_equal_output`). -The function call is run at the point where these functions would evaluate an expression. -Moreover, setting the argument `test` to either "value", "output", or "error" controls which expression test it behaves like. - -For example, the SCT below shows how to run some `pre_code`, and then evaluate the output of a call. - -``` -Ex().check_function_def('my_fun').call("f(1, 2)", test="output", pre_code="x = 1") -``` - - -Managing Processes ------------------ - -As mentioned on the [Homepage](Home.md), DataCamp uses two separate processes. One process to run the solution code, and one process to run the student's submission. This way, `pythonwhat` has access to the 'ideal ending scenario' of an exercises; this makes it easier to write SCTs. Instead of having to specify which value an object should be, we can have `test_object()` look into the solution process and compare the object in that process with the object in the student process. - -### Problem - -Fetching Python objects or the results of running expressions inside a process is not straightforward. To be able to pull data from a process, Python needs to 'pickle' and 'unpickle' files: it converts the Python objects to a byte representation (pickling) that can be passed between processes, and then, inside the process that you want to work with the object, builds up the object from the byte representation again (unpickling). - -For the majority of Python objects, this conversion to and from a byte representation works fine, but for some objects, it doesn't. Even `dill`, and improved implementation of `pickle` that's being used in `pythonwhat`, doesn't flawlessly convert all Python objects out there. - -If you're writing an SCT with functions that require work in the solution process, such as `test_object()`, `test_function()`, and `test_function_definition()`, and then upload the exercise and test it on DataCamp, that you get backend errors that look like this: - - ... dilling inside process failed - write manual converter - ... undilling of bytestream failed - write manual converter - -The first error tells you that 'dilling' - or 'pickling', converting the object to a bytestream representation, failed. The second error tells you that 'undilling' - or 'unpickling', converting the byte representation back to a Python object, failed. These errors will typically occur if you're dealing with exotic objects, such as objects that interface to files, connections to databases, etc. - -### Solution - -To be able to handle these errors, `pythonwhat` allows you to write your own converters for Python objects. Say, for example, that you're writing an exercise to import Excel data into Python, and you're using the `pandas` package. This is the solution and the corresponding SCT: - - *** =solution - ```{python} - import pandas as pd - xl = pd.ExcelFile('battledeath.xlsx') - ``` - - *** =sct - ```{python} - Ex().test_object('xl') - ``` - -Suppose now that objects such as `xl`, which are of the type `pandas.io.excel.ExcelFile`, can't be properly dilled and undilled. (Note: because of hardcoded converters inside `pythonwhat`, they can, see below). To make sure that you can still use `test_object('xl')` to test the equality of the `xl` object between student and solution process, you can manually define a converter with the `set_converter()` function. You can extend the SCT as follows: - - *** =sct - ``` - def my_converter(x): - return(x.sheet_names) - set_converter(key = "pandas.io.excel.ExcelFile", fundef = my_converter) - Ex().test_object('xl') - ``` - -With a lambda function, it's even easier: - - *** =sct - ``` - set_converter(key = "pandas.io.excel.ExcelFile", fundef = lambda x: x.sheet_names) - Ex().test_object('xl') - ``` - -The first arguemnt of `set_converter()`, the `key` takes the type of the object you want to add a manual converter for as a string. The second argument, `fundef`, takes a function definition, taking one argument and returning a single object. This function definition converts the exotic object into something more standard. In this case, the function converts the object of type `pandas.io.excel.ExcelFile` into a simple list of strings. A list of strings is something that can easily be converted into a bytestream and back into a Python object again, hence solving the problem. - -If you want to reuse the same manual converter over different exercises, you'll have to use `set_converter()` in every SCT. - -### Hardcoded converters - -Some converters will be required often. For example, the result of calling `.keys()` and `.items()` on dictionaries can't be dilled and undilled without extra work. To handle these common yet problematic situations, `pythonwhat` features a list of hardcoded converters. This list is [available in the source code](https://github.com/datacamp/pythonwhat/blob/master/pythonwhat/converters.py); feel free to do a pull request if you want to add more converts to this list. This will reduce the amount of code duplication you have to do if you want to reuse the same converter in different exercises. - -### Custom Equality - -The `set_converter()` function opens up possibilities for objects that can actually be dilled and undilled perfectly fine. Say you want to test a `numpy` array, but you only want to check only if the dimensions of the array the student codes up match those in the solution process. You can easily write a manual converter that overrides the typical dilling and undilling of Numpy arrays, implementing your custom equality behavior: - - *** =solution - ```{python} - import numpy as np - my_array = np.array([[1,2], [3,4], [5,6]]) - ``` - - *** =sct - ``` - set_converter(key = "numpy.ndarray", fundef = lambda x: x.shape) - Ex().test_object('my_array') - ``` - -Both of the following submissions will be accepted by this SCT: - -- `my_array = np.array([[1,2], [3,4], [5,6]])` -- `my_array = np.array([[0,0], [0,0], [5,6]])` - - - - diff --git a/docs/source/index.rst b/docs/source/index.rst deleted file mode 100644 index d30a9c36..00000000 --- a/docs/source/index.rst +++ /dev/null @@ -1,33 +0,0 @@ -pythonwhat -================================================ - -Contents: ---------- - -.. toctree:: - :maxdepth: 2 - - Home - quickstart_guide - simple_tests/index.rst - part_checks - expression_tests - logic_tests/index.rst - spec2_summary - - -Pythonwhat V1 -------------- - -.. toctree:: - - pythonwhat.wiki/index.rst - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/docs/source/logic_tests/index.rst b/docs/source/logic_tests/index.rst deleted file mode 100644 index 8227204b..00000000 --- a/docs/source/logic_tests/index.rst +++ /dev/null @@ -1,9 +0,0 @@ -Logic Tests -=========== - -.. toctree:: - :maxdepth: 2 - - test_correct - test_or - test_no diff --git a/docs/source/logic_tests/test_correct.md b/docs/source/logic_tests/test_correct.md deleted file mode 100644 index 0c1350ae..00000000 --- a/docs/source/logic_tests/test_correct.md +++ /dev/null @@ -1,87 +0,0 @@ -test_correct ------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_correct - :members: -``` - -A wrapper function around `test_or()`, `test_correct()` allows you to add logic to your SCT. Normally, your SCT is simply a script with subsequent `pythonwhat` function calls, all of which have to pass. `test_correct()` allows you to bypass this: you can specify a "sub-SCT" in the `check` part, that should pass. If these tests pass, the "sub-SCT" in `diagnose` is not executed. If the tests don't pass, the "sub-SCT" in `diagnose` is run, typically to dive deeper into what the error might be and give more specific feedback. - -To accomplish this, the SCT in `check` is executed silently, so that failure will not cause the SCT to stop and generate a feedback message. If the execution passes, all is good and `test_correct()` is abandoned. If it fails, `diagnose` is executed, not silently. If the `diagnose` part fails, the feedback message that it generates is presented to the student. If it passes, the `check` part is executed again, this time not silently, to make sure that a `test_correct()` that contains a failing `check` part leads to a failing SCT. - -### Example 1 - -As an example, suppose you want the student to calculate the mean of a Numpy array `arr` and store it in `res`. A possible solution could be: - - *** =solution - ```{python} - # Import numpy and create array - import numpy as np - arr = np.array([1, 2, 3, 4, 5, 6]) - - # Calculate result - result = np.mean(arr) - ``` - -You want the SCT to pass when the student manages to store the correct value in the object `result`. How `result` was calculated, does not matter to you: as long as `result` is correct, the SCT should accept the submission. If something about `result` is not correct, you want to dig a little deeper and see if the student used the `np.mean()` function correctly. The following SCT will do just that: - - *** =sct - ```{python} - test_correct(test_object('result'), - test_function('numpy.mean')) - success_msg("You own numpy!") - ``` - -Let's go over what happens when the student submits different pieces of code: - -- The student submits `result = np.mean(arr)`, exactly the same as the solution. - `test_correct()` runs `test_object('result')`. - This test passes, so `test_correct()` stops. - The SCT passes. -- The student submits `result = np.sum(arr) / arr.size`, which also leads to the correct value in `result`. - `test_correct()` runs `test_object('result')`. - This test passes, so `test_correct()` stops before running `test_function()`. - The entire SCT passes even though `np.mean()` was not used. -- The student submits `result = np.mean(arr + 1)`. - `test_correct()` runs `test_object('result')`. - This test fails, so `test_correct()` continues with 'diagnose', running `test_function('numpy.mean')`. - This function fails, since the argument passed to `numpy.mean()` in the student submission does not correspond to the argument passed in the solution. - A meaningful, specific feedback message is presented to the student: you did not correctly specify the arguments inside `np.mean()`. -- The student submits `result = np.mean(arr) + 1`. - `test_correct()` runs `test_object('result')`. - This test fails, so `test_correct()` continues with'diagnose', running `test_function('numpy.mean'). - This function passes, because `np.mean()` is called in exactly the same way in the student code as in the solution. - Because there is something wrong - `result` is not correct - the 'check' SCT, `test_object('result')` is executed again, and this time its feedback on failure is presented to the student. - The student gets the message that `result` does not contain the correct value. - - -### Multiple functions in `diagnose` and `check` - -You can also use `test_correct()` with entire 'sub-SCTs' that are composed of several SCT calls. In this case, you may put multiple tests inside `multi()`, as below.. - - *** =sct - ```{python} - Ex().test_correct( - multi(test_object('a'), test_object('b')), # multiple check SCTs - test_function('numpy.mean') - ) - ``` - -### Why to use `test_correct()` - -You will find that `test_correct()` is an extremely powerful function to allow for different ways of solving the same problem. You can use `test_correct()` to check the end result of a calculation. If the end result is correct, you can go ahead and accept the entire exercise. If the end result is incorrect, you can use the `diagnose` part of `test_correct()` to dig a little deeper. - -It is also perfectly possible to use `test_correct()` inside another `test_correct()`. - -### Wrapper around `test_or()` - -`test_correct()` is a wrapper around `test_or()`. `test_correct(diagnose, check)` is equivalent with: - - def diagnose_and_check() - diagnose() - check() - - test_or(diagnose_and_check, check) - -Note that in each of the `test_or` cases here, the submission has to pass the SCTs specified in `check`. diff --git a/docs/source/logic_tests/test_not.md b/docs/source/logic_tests/test_not.md deleted file mode 100644 index 9a1c2fcf..00000000 --- a/docs/source/logic_tests/test_not.md +++ /dev/null @@ -1,2 +0,0 @@ -test_not --------- diff --git a/docs/source/logic_tests/test_or.md b/docs/source/logic_tests/test_or.md deleted file mode 100644 index 6b3cc5b6..00000000 --- a/docs/source/logic_tests/test_or.md +++ /dev/null @@ -1,28 +0,0 @@ -test_or -------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_or - :members: -``` - -This function simply tests whether one of the SCTs you specify inside it passes. - -Suppose you want to check whether people correctly printed out any integer between 3 and 7. A solution could be: - - *** =solution - ```{python} - print(4) - ``` - -To test this in a robust way, you could use `test_output_contains()` with a suitable regular expression that covers everything, or you can use `test_or()` with three separate `test_output_contains()` functions. - - *** =sct - ```{python} - test_or(test_output_contains('4'), - test_output_contains('5'), - test_output_contains('6')) - success_msg("Nice job!") - ``` - -You can consider `test_or()` a logic-inducing function. The different calls to `pythonwhat` functions that are in your SCT are actually all tests that _have_ to pass: they are `AND` tests. With `test_or()` you can add chunks of `OR` tests in there. diff --git a/docs/source/part_checks.rst b/docs/source/part_checks.rst deleted file mode 100644 index 1d8d0896..00000000 --- a/docs/source/part_checks.rst +++ /dev/null @@ -1,568 +0,0 @@ -Part Checks -================ - -.. role:: python(code) - :language: python - -Check Syntax --------------- - -In Brief -~~~~~~~~ - -While functions beginning with ``test_``, such as ``test_student_typed`` look over some code or output, ``check_`` functions allow us to zoom in on parts of that student and solution code. - -For example, ``check_list_comp`` examines list comprehensions by breaking them into 3 parts: ``body``, ``comp_iter``, and ``ifs``. This is shown below. - - -:code:`[i*2 for i in range(10) if i>2]` => :code:`[BODY for i in COMP_ITER if IFS]` - -Each of these 3 parts may be tested individually using the simple test functions. -For example, in order to test the body of the comprehension above, we could create the following exercise. - -.. code:: python - - *** =solution - ```{python} - L2 = [i**2 for i in range(0,10) if i>2] - ``` - - *** =sct - ```{python} - (Ex().check_list_comp(0) # focus on first list comp - .check_body().test_student_typed('i\*2') # focus on its body for test - ) - ``` - -In the SCT, ``check_list_comp`` gets the first comprehension, and will fail with feedback if no comprehensions were used in th submission code. ``check_body`` gets ``i**2`` in the solution code, and whatever corresponds to BODY in the submission code. - -(Note: the parentheses around the entire statement are just syntactic sugar, to let us chain commands in python without using ``\`` at the end of each line.) - -Full Example -~~~~~~~~~~~~ - -This section expands the above example to run tests on each part: body, iter, and ifs. - -.. code-block:: python - - *** =solution - ```{python} - L2 = [i*2 for i in range(0,10) if i>2] - ``` - - *** =sct - ```{python} - list_comp = Ex().check_list_comp(0, missing_msg="Did you include a list comprehension?") - list_comp.check_body().test_student_typed('i\*2') - list_comp.check_iter().has_equal_value() - list_comp.check_ifs(0).multi([has_equal_value(context_vals=[i]) for i in range(0,10)]) - ``` - -In this SCT, the first line focuses on the first list comprehension, and assigns it to ``list_comp``, so we can test each part in turn. As a reminder, the code corresponding to each part in the solution code is.. - -* BODY: ``i**2`` -* COMP_ITER: ``range(0,10)`` -* IFS: [``i>2``] - -Note that IFS is represented as a list, and the index 1 was passed to `check_ifs` because a list comprehension may have multiple if statements. Since the test on BODY, is explained in the [In Brief section](#In_Brief), we will focus on the tests on ITER and and IFS. - -check_iter -^^^^^^^^^^^^^^ - -In the line ``list_comp.check_iter().equal_value()``, ``check_iter`` gets the ITER part in the solution and submission code, while ``has_equal_value`` tells pythonwhat to run those parts and see if they return equal values. Below are example solution and submission codes, with the ITER part they would produce - -================ ============================================ ==================== - type code ITER part -================ ============================================ ==================== - **solution** :python:`[i*2 for i in range(0,10) if i>2]` :code:`range(0,10)` -**submission** :python:`[i*2 for i in range(10) if i>2]` :code:`range(10)` -================ ============================================ ==================== - -In this case, ``equal_value`` will run each part, and then confirm that :python:`range(0,10) == range(10)`. For more on functions that run code, like ``has_equal_value`` see [Expressions Tests](processes). - -check_ifs -^^^^^^^^^^^^^ - -The line - -.. code-block:: python - - list_comp.check_ifs(0).multi([has_equal_value(context_vals=[i] for i in range(0,10))]) - -is a doozy, but can be broken down into - -.. code-block:: python - - equal_tests = [has_equal_value(context_vals=[i] for i in range(0,10))] # collection of has_equal_tests - list_comp.check_ifs(0).multi(equal_tests) # focus on IFS run equal_tests` - -In this case ``equal_tests`` is a list of ``has_equal_value`` tests that we'll want to perform. ``check_ifs(1)`` grabs the first IFS part, and ``multi(equal_tests)`` runs each ``has_equal_value`` test on that part. - -Notice that ``has_equal_value`` was given a context_val argument. This is because the list comprehension creates a temporary variable that needs to be defined when we run the IFS code. - -================ ============================================== ================ =============== - type code IFS part context value -================ ============================================== ================ =============== - **solution** :python:`[i*2 for i in range(0,10) if i>2]` :python:`if i>2` ``i`` - **submission** :python:`[j*2 for j in range(0,10) if j>2]` :python:`if j>2` ``j`` -================ ============================================== ================ =============== - -In this case, the context_vals argument is a list of values, with one for each (in this case only a single) context value. In this way, ``has_equal_value`` assigns ``i`` and ``j`` to the same value, before running the IFs part. By creating a list of ``has_equal_tests`` with context vals spanning ``range(0,10)``, we test the IFS across a range of values. - -Nested Part Example -~~~~~~~~~~~~~~~~~~~~ - -Check functions may be combined to focus on parts within parts, such as - -.. code:: python - - *** =solution - ```{python} - [i*2 if i> 5 else 0 for i in range(0,10)] - ``` - -In this case, a representation with the parts in caps and wrapping the inline if expression with ``{BODY=...}`` is - -.. code:: - - [{BODY=BODY if TEST else ORELSE} for i in ITER] - -in order to test running the inline if expression we could go from list_comp => body => if_exp. One possible SCT is shown below. - -.. code:: python - - *** =sct - ```{python} - (Ex().check_list_comp(0) # first comprehension - .check_body().set_context(i=6) # comp's body - .check_if_exp(0).has_equal_value() # body's inline IFS - ) - ``` - -Note that rather than using the ``context_vals`` argument of ``has_equal_value`` we use ``set_context`` to define the context variable (``i`` in the solution code) on the body of the list comprehension. This makes it very clear when the context value was introduced. It is worth pointing out that of the parts a list comprehension has, BODY and IFS, but not ITER have ``i`` as a context value. This is because in python ``i`` is undefined in the ITER part. Context values are listed in the [see cheatsheet below]. - -Testing only the body of the list comprehension -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If we left out the ``check_if_exp`` above, the resulting SCT, - -.. code:: python - - (Ex().check_list_comp(0).check_body().set_context(i=6) - #.check_if_exp(1) - .has_equal_value() - ) - -would still run the same code for the solution (the inline if expression), since it's the only thing in the BODY of the list comprehension. However it wouldn't check if an if expression was used, allowing a wider range of passing and failing submissions (for better or worse!). Moreover, `has_equal_value` may be used multiple times during the chaining, as it doesn't change what the focus is. - -Helper Functions ----------------- - -multi -~~~~~~~ - -Runs multiple subtests. - - -Comma separated arguments -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For example, this code without multi, - -.. code:: - - Ex().check_if_exp(0).check_body().has_equal_value() - Ex().check_if_exp(0).check_test().has_equal_value() - - -is equivalent to - -.. code:: - - Ex().check_if_exp(0).multi( - check_body().has_equal_value(), - check_test().has_equal_value() - ) - -List or generator of subtests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Rather than one or more subtest args, multi can take a single list or generator of subtests. -For example, the code below checks that the body of a list comprehension has equal value -for 10 possible values of the iterator variable, ``i``. - -.. code:: - - Ex().check_list_comp(0) - .check_body() - .multi(set_context(i=x).has_equal_value() for x in range(10)) - -Chaining off multi -^^^^^^^^^^^^^^^^^^^ - -Multi returns the same state, or focus, it was given, so whatever comes after multi will run -the same as if multi wasn't used. For example, the code below tests a list comprehension's body, -followed by its iterator. - -.. code:: - - Ex().check_list_comp(0) \ - .multi(check_body().has_equal_value()) \ - .check_iter().has_equal_value() - - -set_context -~~~~~~~~~~~~~ - -Sets the value of a temporary variable, such as ``ii`` in the list comprehension below. - -.. code:: - - [ii + 1 for ii in range(3)] - -Variable names may be specified using positional or keyword arguments. - -Example -^^^^^^^^ - -**Solution Code** - -.. code:: - - ltrs = ['a', 'b'] - for ii, ltr in enumerate(ltrs): - print(ii) - -**SCT** - -.. code:: - - Ex().check_for_loop(0).check_body() \ - .set_context(ii=0, ltr='a').has_equal_output() \ - .set_context(ii=1, ltr='b').has_equal_output() - -Note that if a student replaced `ii` with `jj` in their submission, `set_context` would still work. -It uses the solution code as a reference. While we specified the target variables ``ii`` and ``ltr`` -by name in the SCT above, they may also be given by position.. - -.. code:: - - Ex().check_for_loop(0).check_body().set_context(0, 'a').has_equal_output() - -Instructor Errors -^^^^^^^^^^^^^^^^^^^ - -If you are unsure what variables can be set, it's often easiest to take a guess. -When you try to set context values that don't match any target variables in the solution code, -``set_context`` raises an exception that lists the ones available. - - - -with_context -~~~~~~~~~~~~~~ - -Runs subtests after setting the context for a ``with`` statement. - -This function takes arguments in the same form as ``multi``. -Note also that ``with_context`` was the default behavior for ``test_with`` in pythonwhat version 1. - -Context Managers Explained -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -With statements are special in python in that they enter objects called a context manager at the beginning of the block, -and exit them at the end. For example, the object returned by ``open('fname.txt')`` below is a context manager. - -.. code:: - - with open('fname.txt') as f: - print(f.read()) - -This code runs by - -1. assigning ``f`` to the context manager returned by ``open('fname.txt')`` -2. calling ``f.__enter__()`` -3. running the block -4. calling ``f.__exit__()`` - -``with_context`` was designed to emulate this sequence of events, by setting up context values as in step (1), -and replacing step (3) with any sub-tests given as arguments. - - -fail -~~~~~~ - -Fails. This function takes a single argument, ``msg``, that is the feedback given to the student. -Note that this would be a terrible idea for grading submissions, but may be useful while writing SCTs. -For example, failing a test will highlight the code as if the previous test/check had failed. - -As a trivial SCT example, - -.. code:: - - Ex().check_for_loop(0).check_body().fail() # fails boo - -This can also be helpful for debugging SCTs, as it can be used to stop testing as a given point. - -Check Functions ----------------- - -**Arguments** - -* **index**: index or key corresponding to the node or part of interest. - This applies to all functions in the **check** column in the table below. - However, apart from that, it only applies when there is more than one of a specific part to choose from --- - ``check_ifs``, ``check_args``, ``check_handlers``, and ``check_context``. - (e.g. ``Ex().check_list_comp(0).check_ifs(0)``) -* **missing_msg**: optional feedback message if node or part doesn't exist. - - -Note that code in all caps indicates the name of a piece of code that may be inspected using, ``check_{part}``, -where ``{part}`` is replaced by the name in caps (e.g. ``check_if_else(0).check_test()``). -Target variables are those that may be set using ``set_context``. -These variables may only be set in places where python would set them. -For example, this means that a list comprehension's ITER part has no target variables, -but its BODY does. - -+------------------------+------------------------------------------------------+-------------------+ -| check | parts | target variables | -+========================+======================================================+===================+ -|check_if_else(0) | .. code:: | | -| | | | -| | if TEST: | | -| | BODY | | -| | else: | | -| | ORELSE | | -| | | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_while(0) | .. code:: python | | -| | | | -| | while TEST: | | -| | BODY | | -| | else: | | -| | ORELSE | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_list_comp(0) | .. code:: | ``i`` | -| | | | -| | [BODY for i in ITER if IFS[0] if IFS[1]] | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_generator_exp(0) | .. code:: | ``i`` | -| | | | -| | (BODY for i in ITER if IFS[0] if IFS[1]) | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_dict_comp(0) | .. code:: | ``k``, ``v`` | -| | | | -| | {KEY : VALUE for k, v in ITER if IFS[0]} | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_for_loop(0) | .. code:: | ``i`` | -| | | | -| | for i in ITER: | | -| | BODY | | -| | else: | | -| | ORELSE | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_try_except(0) | .. code:: python | ``e`` | -| | | | -| | try: | | -| | BODY | | -| | except BaseException as e: | | -| | HANDLERS['BaseException'] | | -| | except: | | -| | HANDLERS['all'] | | -| | else: | | -| | ORELSE | | -| | finally: | | -| | FINALBODY | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_with(0) | .. code:: python | `f`` | -| | | | -| | with CONTEXT_TEST as f: | | -| | BODY | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_function_def('f') | .. code:: python | argument names | -| | | | -| | def f(ARGS[0], ARGS[1]): | | -| | BODY | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_lambda(0) | .. code:: | argument names | -| | | | -| | lambda ARGS[0], ARGS[1]: BODY | | -| | | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ -|check_function('f', 0) | .. code:: | argument names | -| | | | -| | f(ARGS[0], ARGS[1]) | | -| | | | -| | | | -+------------------------+------------------------------------------------------+-------------------+ - -More ------- - -elif statements -~~~~~~~~~~~~~~~~ - -In python, when an if-else statement has an elif clause, it is held in the ORELSE part, - -.. code:: python - - if TEST: - BODY - ORELSE # elif and else portion - -In this sense, an if-elif-else statement is represented by python as nested if-elses. For example, the final ``else`` below - -.. code:: python - - if x: print(x) # line 1 - elif y: print(y) # "" 2 - else: print('none') # "" 3 - -can be checked with the following SCT - -.. code:: python - - (Ex().check_if_else(0) # lines 1-3 - .check_orelse().check_if_else(0) # lines 2-3 - .check_orelse().has_equal_output() # line 3 - ) - - -function definition / lambda args -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -the ARGS part in function definitions and lambdas may be selected by position or keyword. -For example, the arguments `a` and `b` below, - -.. code:: python - - def f(a, b=2, *some_name): - BODY - -Could be tested using, - -.. code:: python - - Ex().check_function_def('f').multi( - check_args('a').is_default(), - check_args('b').is_default().has_equal_value(), - check_args('*args', 'missing a starred argument!') - ) - -Note that ``check_args('*args')`` and ``check_args('**kwargs')`` may be used to test *args, and **kwargs style parameters, regardless of their name in the function definition. - -function call args -~~~~~~~~~~~~~~~~~~~ - -Behind the scenes, ``check_function`` uses the same logic for matching arguments to function signatures as `test_function_v2 `__. -It also has a ``signature`` argument that accepts a custom signature. - -Matching Signatures -^^^^^^^^^^^^^^^^^^^^ - -By default, ``check_function`` tries to match each argument in the function call with the appropriate parameters in that function's call signature. -For example, all the calls to ``f`` below use ``a = 1`` and ``b = 2``. - -.. code:: - - def f(a, b): pass - - f(1, 2) # by position - f(a = 1, b = 2) # by keyword - f(1, b = 2) # mixed - -However, when testing a submission, we may not care how the argument was specified. - -.. code:: - - *** =pre_exercise_code - ```{python} - def f(a, b): pass - ``` - - *** =solution - ```{python} - f(1, b=2) - ``` - - *** =sct - ```{python} - Ex().check_function('f', 0).check_args('a').has_equal_value() - ``` - -will pass for all the ways of calling ``f`` listed above. - -signature = False -^^^^^^^^^^^^^^^^^^ - -Setting signature to false, as below, only allows you to check an argument by name, if the name was explicitly specified in the function call. -For example, - -.. code:: - - *** =solution - ```{python} - dict( [('a', 1)], c = 2) - ``` - - *** =sct - ```{python} - Ex().check_function('dict', 0, signature=False)\ - .multi( - check_args(0), # can only select by position - check_args('c') # could use check_args(1) - ) - ``` - -Note that here, an argument's position is referring to its position in the function call (not its signature). - -Example: testing a list passed as an argument -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Suppose you want to test the first argument passed to `sum`. -Below, we show how this can be down, using `has_equal_ast()` to check that the abstract syntax trees for the 1st argument match. - -.. code:: python - - *** =solution - ```{python} - sum([1, 2, 3]) - ``` - - *** =sct - ```{python} - (Ex().check_function('sum', 0) - .check_args(0) - .has_equal_ast("ast fail") # compares abstract representations - .test_student_typed("\[1, 2, 3\]", "typed fail") # alternative, more rigid test - ) - ``` - -Notice that testing the argument is similar to testing, say, the body of an if statement. -In this sense, we could even do deeper checks into an argument. -Below, the SCT verifies that the first argument passed to sum is a list comprehension. - -.. code:: python - - *** =solution - ```{python} - sum([i for i in range(10)]) - ``` - - *** =sct - ```{python} - (Ex().check_function('sum', 0) - .check_args(0) - .check_list_comp(0) - .has_equal_ast() - ) - ``` diff --git a/docs/source/pythonwhat.wiki/index.rst b/docs/source/pythonwhat.wiki/index.rst deleted file mode 100644 index bdff85ed..00000000 --- a/docs/source/pythonwhat.wiki/index.rst +++ /dev/null @@ -1,29 +0,0 @@ -Legacy Tests -============ - -The functions below combine both part checks and simple tests from pythonwhat v1. -In some cases, they allow very specific checks that are not yet exposed to SCT creators in v2 -(such as whether iterator variables have the exact same names). - -.. toctree:: - :maxdepth: 2 - - test_data_frame - test_dictionary - test_operator - test_expression_output - test_expression_result - test_object_after_expression - test_object_accessed - parts_cheatsheet - test_comprehension - test_for_loop - test_if_else - test_if_exp - test_try_except - test_while_loop - test_with - test_function_definition - test_function - test_function_v2.md - test_lambda_function diff --git a/docs/source/pythonwhat.wiki/parts_cheatsheet.md b/docs/source/pythonwhat.wiki/parts_cheatsheet.md deleted file mode 100644 index c79a0537..00000000 --- a/docs/source/pythonwhat.wiki/parts_cheatsheet.md +++ /dev/null @@ -1,89 +0,0 @@ -parts_cheatsheet ----------------- - -### test_list_comp - -```{python} -[BODY for i in COMP_ITER if IFS[0] if IFS[1]] -``` - -### test_dict_comp - -```{python} -{ KEY : VALUE for k, v in COMP_ITER if IFS[0] if IFS[1] } -``` -### test_generator_exp - -```{python} -(BODY for i in COMP_ITER if IFS[0] if IFS[1]) -``` - -### test_for_loop -```{python} -for i in FOR_ITER: - BODY -else: - ORELSE -``` - -_yes, you can put an else statement at the end!_ - -### test_if_else - -```{python} -if TEST: - BODY -else: - ORELSE -``` - -or, in the case of elif statements... - -```{python} -if TEST: - BODY -ORELSE -``` - -### test_lambda -```{python} -lambda x: BODY -``` - -### test_try_except - -```{python} -try: - BODY -except BaseException: - HANDLERS['BaseException'] -except: - HANDLERS['all'] -else: - ORELSE -finally: - FINALBODY -``` - -### test_while - -```{python} -while TEST: - BODY -else: - ORELSE -``` - -### test_with - -```{python} -with CONTEXT_TEST as context_var: - BODY -``` - -### test_function_definition - -```{python} -def f(a, b): - BODY -``` diff --git a/docs/source/pythonwhat.wiki/test_comprehension.md b/docs/source/pythonwhat.wiki/test_comprehension.md deleted file mode 100644 index 0c26f37a..00000000 --- a/docs/source/pythonwhat.wiki/test_comprehension.md +++ /dev/null @@ -1,201 +0,0 @@ -test comprehensions -------------------- - - def test_list_comp(index=1, - not_called_msg=None, - comp_iter=None, - iter_vars_names=False, - incorrect_iter_vars_msg=None, - body=None, - ifs=None, - insufficient_ifs_msg=None, - expand_message=True) - - def test_generator_exp(index=1, - not_called_msg=None, - comp_iter=None, - iter_vars_names=False, - incorrect_iter_vars_msg=None, - body=None, - ifs=None, - insufficient_ifs_msg=None, - expand_message=True) - - def test_dict_comp(index=1, - not_called_msg=None, - comp_iter=None, - iter_vars_names=False, - incorrect_iter_vars_msg=None, - key=None, - value=None, - ifs=None, - insufficient_ifs_msg=None, - expand_message=True) - -Currently, functionality to test list comprehensions, generator expressions and dictionary comprehensions is implemented. If you look at the signatures, you'll see that the arguments for `test_list_comp()` and `test_generator_exp()` are identical. Syntactically, there is close to no difference between list comprehensions and generator expressions, so all tests and settings apply for both cases. For `test_dict_comp()` there's only a small difference: the arguments `key` and `value` instead of the `body` argument, so that you can test the `key` part of the dictionary comprehension seperately from the `value` comprehension. - -The above functions work pretty similarly to `test_for_loop()`, with some additions and customizations here and there. Let's go over the argments: - -- `index`: the number of the comprehension in the submission to test. (this is specific to each comprehension, if there's one list and one dict comprehension you need `index=1` twice.) -- `not_called_msg`: Custom message in case the comprehension was not coded (or there weren't enough comprehensions). -- `comp_iter`: sub SCT to check the sequence part of the comprehension. Specify this through another function definition or a lambda function. -- `iter_vars_names`: whether or not the iterator variables should match the ones in the solution. -- `incorrect_iter_vars_msg`: Custom message in case the iterator variables don't match the solution (if `iter_vars_names` is `True`) or if the number of iterator variables doesn't correspond to the solution. -- `body`, `key`, `value`: sub SCTs to check the body part (for list comps and generator expressions) or the key and value part of a dictionary comprehension. -- `ifs`: list of sub-SCTs to check each of the ifs specified inside the comprehension. If you specify `ifs`, make sure that the number of sub-SCTs corresponds exactly to the number of ifs that are in the solution. -- `insufficient_ifs`: custom message in case the student coded less ifs than the corresponding comp in the solution. -- `expand_message`: whether or not to expand feedback messages from sub-SCTs with more information about where in the list comprehension they occur. - -### Example 1: List comprehension - -Suppose you want the student to code a list comprehension like below: - - *** =solution - ```{python} - x = {'a': 2, 'b':3, 'c':4, 'd':'test'} - [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(val, int)] - ``` - -The following SCT will test several parts of this list comprehension, and relies on automatic feedback messages everywhere: - - *** =sct - ```{python} - test_list_comp(index=1, - comp_iter=lambda: test_expression_result(), - iter_vars_names=True, - body=lambda: test_expression_result(context_vals = ['a', 2]), - ifs=[lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False]), - lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False])]) - ``` - -By setting `iter_vars_names` to `True`, `pythonwhat` will check that the student actually used the iterator variables `key` and `val`. Notice that in the sub SCT for the body, `context_vals` are used to set the `key` and `val` iterator variables before the expression is tested. This is similar to how things work in `test_for_loop()`. Notice also that inside the list of if sub-SCTs, `do_eval` is false, because the values `key` and `val` are not available there (setting context vals is currently only possible inside `test_expression_*()` functions). - -`test_list_comp()` will generate a bunch of meaningful automated messages depending on which error the student made: - - submission: - feedback: "The system wants to check the first list comprehension you defined but hasn't found it." - - submission: [key for key in x.keys()] - feedback: "Check your code in the iterable part of the first list comprehension. Unexpected expression: expected `dict_items([('a', 2), ('b', 3), ('c', 4), ('d', 'test')])`, got `dict_keys(['a', 'b', 'c', 'd'])` with values." - - submission: [a + str(b) for a,b in x.items()] - feedback: "Have you used the correct iterator variables in the first list comprehension? Make sure you use the correct names!" - - submission: [key + '_' + str(val) for key,val in x.items()] - feedback: "Check your code in the body of the first list comprehension. Unexpected expression: expected `a2`, got `a_2` with values." - - submission: [key + str(val) for key,val in x.items()] - feedback: "Have you used 2 ifs inside the first list comprehension?" - - submission: [key + str(val) for key,val in x.items() if hasattr(key, 'test') if hasattr(key, 'test')] - feedback: "Check your code in the first if of the first list comprehension. Have you called `isinstance()`?" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if hasattr(key, 'test')] - feedback: "Check your code in the second if of the first list comprehension. Have you called `isinstance()`?" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(key, str)] - feedback: "Check your code in the second if of the first list comprehension. Did you call `isinstance()` with the correct arguments?" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(val, str)] - feedback: "Great work!" - -NOTE: the "check your code in the ... of the first list comprehension" parts are included because `expand_message = True`. - -You can also update SCT to override all automatically generated messages, either inside `test_list_comp()` itself or inside the sub-SCTs: - - *** =sct - ```{python} - test_list_comp(index=1, - not_called_msg='notcalled', - comp_iter=lambda: test_expression_result(incorrect_msg = 'iterincorrect'), - iter_vars_names=True, - incorrect_iter_vars_msg='incorrectitervars', - body=lambda: test_expression_result(context_vals = ['a', 2], incorrect_msg = 'bodyincorrect'), - ifs=[lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False], not_called_msg = 'notcalled1', incorrect_msg = 'incorrect2'), - lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False], not_called_msg = 'notcalled2', incorrect_msg = 'incorrect2')], - insufficient_ifs_msg='insufficientifs') - ``` - -In this case, you get the following feedback for different submissions: - - submission: - feedback: "notcalled" - - submission: [key for key in x.keys()] - feedback: "Check your code in the iterable part of the first list comprehension. iterincorrect" - - submission: [a + str(b) for a,b in x.items()] - feedback: "incorrectitervars" - - submission: [key + '_' + str(val) for key,val in x.items()] - feedback: "Check your code in the body of the first list comprehension. bodyincorrect" - - submission: [key + str(val) for key,val in x.items()] - feedback: "insufficientifs" - - submission: [key + str(val) for key,val in x.items() if hasattr(key, 'test') if hasattr(key, 'test')] - feedback: "Check your code in the first if of the first list comprehension. notcalled1" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if hasattr(key, 'test')] - feedback: "Check your code in the second if of the first list comprehension. notcalled2" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(key, str)] - feedback: "Check your code in the second if of the first list comprehension. incorrect2" - - submission: [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(val, str)] - feedback: "Great work!" - - -### Example 2: Generator Expressions - -An example here won't be necessary, because it works the exact same way as in Example 1, with the only difference that in automated feedback, "list comprehension" is replaced with "generator expression". - -### Example 3: Dictionary Comprehensions - -Suppose you want the student to code a dictionary comprehension like below: - - *** =solution - ```{python} - x = {'a': 2, 'b':3, 'c':4, 'd':'test'} - [key + str(val) for key,val in x.items() if isinstance(key, str) if isinstance(val, int)] - ``` - -The following SCT will test several parts of this list comprehension, and relies on automatic feedback messages everywhere: - - *** =sct - ```{python} - test_list_comp(index=1, - comp_iter=lambda: test_expression_result(), - iter_vars_names=True, - body=lambda: test_expression_result(context_vals = ['a', 2]), - ifs=[lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False]), - lambda: test_function_v2('isinstance', params = ['obj'], do_eval = [False])]) - ``` - -Again, customized messages are generated for different cases: - - submission: - feedback: "The system wants to check the first dictionary comprehension you defined but hasn't found it." - - submission: { a:a for a in lst[1:2] } - feedback: "Check your code in the iterable part of the first dictionary comprehension. Unexpected expression: expected `['this', 'is', 'a', 'list']`, got `['is']` with values." - - submission: { a:a for a in lst } - feedback: "Have you used the correct iterator variables in the first dictionary comprehension? Make sure you use the correct names!" - - submission: { el + 'a':str(el) for el in lst } - feedback: "Check your code in the key part of the first dictionary comprehension. Unexpected expression: expected `a`, got `aa` with values." - - submission: { el:str(el) for el in lst } - feedback: "Check your code in the value part of the first dictionary comprehension. Unexpected expression: expected `1`, got `a` with values." - - submission: { el:len(el) for el in lst } - feedback: "Have you used 1 ifs inside the first dictionary comprehension?" - - submission: { el:len(el) for el in lst if isinstance('a', str)} - feedback: "Check your code in the first if of the first dictionary comprehension. Did you call `isinstance()` with the correct arguments?" - - submission: { el:len(el) for el in lst if isinstance(el, str)} - feedback: "Great work!" - - diff --git a/docs/source/pythonwhat.wiki/test_data_frame.md b/docs/source/pythonwhat.wiki/test_data_frame.md deleted file mode 100644 index efb8944b..00000000 --- a/docs/source/pythonwhat.wiki/test_data_frame.md +++ /dev/null @@ -1,37 +0,0 @@ -test_data_frame ---------------- - - def test_data_frame(name, - columns=None, - undefined_msg=None, - not_data_frame_msg=None, - undefined_cols_msg=None, - incorrect_msg=None) - -Test a pandas DataFrame. This methods makes it possible to test the columns of a DataFrame object independently. Only the contents will be tested. Customisable error messages are possible for when there is no object in the process with name `name`, for when that object is no pandas DataFrame, when there are columns you want to test for which are not defined and when some columns contain bad values. `columns` contains a list of column names, and defaults to `None`. If it's `None`, all columns that are found in the data frame created by the solution will be tested. - -### Example 1 - -Suppose we have the following solution: - - *** =solution - ```{python} - # import pandas - import pandas as pd - - # Create dataframe with columns a and b - my_df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]}) - ``` - -To test this we simply use: - - *** =sct - ```{python} - test_import("pandas") - test_data_frame("my_df", columns = ["a", "b"]) - success_msg("Great job!") - ``` - -This SCT will first test if `pandas` is correctly imported, and will then check if the student created a Pandas DataFrame called `my_df`. If it was not defined, a message is generated that you can override with `undefined_msg`. If the object was defined but it isn't a Pandas DataFrame, as message is generated that you can override wiht `not_data_frame_msg`. If `my_df` is a Pandas DataFrame, `test_data_frame()` goes on to check if all columns that are specified in the `columns` argument are defined in the data frame, and next whether these columns are correct. The messages that are generated in case of an incorrect submission can be overrided with `undefined_cols_msg` and `incorrect_msg`, respectively. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.rst). diff --git a/docs/source/pythonwhat.wiki/test_dictionary.md b/docs/source/pythonwhat.wiki/test_dictionary.md deleted file mode 100644 index bf59cc92..00000000 --- a/docs/source/pythonwhat.wiki/test_dictionary.md +++ /dev/null @@ -1,43 +0,0 @@ -test_dictionary ---------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_dictionary.test_dictionary -``` - - def test_dictionary(name, - keys=None, - undefined_msg=None, - not_dictionary_msg=None, - key_missing_msg=None, - incorrect_value_msg=None) - -Test a dictionary. Consider this function an advanced version of `test_object`, where you can specify messages that are explicit to test dictionaries. `test_dictionary` takes a step-by-step approach to checking the correspondence of the dictionary between student and solution process: - -- Step 1: Is the object specified in `name` actually defined? -- Step 2: Is the object specified in `name` actually a dictionary? -- Step 3: For each key, is the key specified in the dictionary? -- Step 4: For each key, is the value corresponding to the key correct when comparing to the solution? - -For Step 3 and Step 4, you can control which keys have to be tested through the `keys` argument. If you don't specify this argument, `test_dictionary()` will look for all keys and compare the values that are specified in the corresponding dictionary in the solution process. - -### Example: step by step - -Suppose you want the student to create a dictionary `x`, that contains three keys: `"a"`, `"b"` and `"c"`. The following solution and sct could be used for this: - - *** =solution - ```{python} - x = {'a': 123, 'b':456, 'c':789} - ``` - - *** =sct - ```{python} - test_dictionary('x') - ``` - -- Step 1: if the student submits an empty script, the feedback _Are you sure you defined the dictionary `x`?_ will be presented. You can override this by specifying `undefined_msg` yourself. -- Step 2: if the student submits `x = 123`, the feedback _`x` is not a dictionary._ will be presented. You can override this message by specifying `not_dictionary_msg` yourself. -- Step 3: if the student submits `x = {'a':123, 'b':456, 'd':78}`, the feedback _Have you specified a key `c` inside `x`?_ will be presented. You can override this by specifying `key_missing_msg` yourself. -- Step 4: if the student submits `x = {'a':123, 'b':456, 'c':78}`, the feedback _Have you specified the correct value for the key `c` inside `x`?_ will be presented. You can override this by specifying `incorrect_value_msg` yourself. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_expression_output.md b/docs/source/pythonwhat.wiki/test_expression_output.md deleted file mode 100644 index df88df23..00000000 --- a/docs/source/pythonwhat.wiki/test_expression_output.md +++ /dev/null @@ -1,86 +0,0 @@ -test_expression_output ----------------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_expression_output.test_expression_output -``` - - def test_expression_output(extra_env=None, - context_vals=None, - incorrect_msg=None, - eq_condition="equal", - expr_code=None, - pre_code=None, - keep_objs_in_env=None) - - -`test_expression_output()` is similar to `test_expression_result()`, but instead of checking the result, it checks the output that a single or a set of expressions generates. Typically, this function is used as a sub-test inside other test functions, such as `test_for_loop()` and `test_with()`. - -By default, the `test_expression_output()` will execute the 'active expression(s)'; if it's used as a top-level SCT function, that is the entire student submission. If it's used inside the `body` of the `test_for_loop()` function, for example, the entire for loop's body will executed and the output will be compared to the solution. With `expr_code`, you can override this default expression tree. With `pre_code`, you can prepend the execution of the default expression tree with some extra code, for example to set some variables. - -Oftentimes, the expression you want to check the output for does not have all variables to its disposal that it requires. Remember that the process in which the expression is evaluated only contains the variables that are available in the global scope. If you're for example running the body of a function definition, this means that the local variables, that are for example passed into the function as arguments, are not all available during execution. To make these variables available, you can set the `extra_env` and `context_vals` arguments. The former is to specify extra variables, with a dictionary. The latter is to specify so called context values; the objects you specify here should not be named; this is to make function definition arguments, iterators in for loops, context variables in `with` constructs, etc, name-independent. - -### Example 1 - -Suppose we want the student to define a function, that loops over the elements in a dictionary, and prints out each key and value, as follows: - - *** =solution - ```{python} - def print_dict(my_dict): - for key, value in my_dict.items(): - print(key + " - " + str(value)) - ``` - -An appropriate SCT for this exercise could be the following (for clarity, we're not using any default messages): - - *** =sct - ```{python} - def fun_body_test(): - def for_iter_test(): - example_dict = {'a': 2, 'b': 3} - test_expression_result(context_vals = [example_dict]) - def for_body_test(): - test_expression_output(context_vals = ['c', 3]) - test_for_loop(for_iter = for_iter_test, body = for_body_test) - - test_function_definition('print_dict', body = fun_body_test) - ``` - -Assuming the student coded the function in the exact same way as the solution, the following things happen: - -- `test_function_definition()` is run first: it checks whether `print_dict` is defined, whether the arguments are correctly named and with the correct defaults. Next, it checks the function definition body: it extracts the body of both the student and the solution code, sets the context values for this 'substate', i.e. `"my_dict"`, and then runs `fun_body_test()`. -- Inside `fun_body_test()`, `test_for_loop()` is executed. This function will find the for loop in the function definition body of both student and solution code, and will then run different tests: - - First, the `for_iter` test is run, which is specified with `for_iter_test()` in this SCT. The `for_iter` part of the `for` loop is extracted, which is `my_dict.items()` in the case of the solution. The context values are still `"my_dict"`. Inside `test_expression_result()`, the context vals are specified, so through `context_vals = [example_dict]`, the variable `my_dict` will now have the value `{'a': 2, 'b':3}` inside the student and solution processes. Next, the currently active expression (`my_dict.items()`) is executed. The result of calling this expression in both student and solution process is compared. - - Second, the `body` test is run, which is specified iwth `for_body_test()` in this SCT. The `body` part of the `for` loop is extracted, which is `print(key + " - " + str(value))` in the case of the solution. Now, the context values are set to the iterator variables of the `for` loop, so `"key"` and `"value"`. Inside `test_expression_output()`, the context vals are specified: `key` is set to be `'c'`, `value` is set to be `3`. Next, the currently active expression (`print(key + " - " + str(value))`) is executed, and the output it generates is fetched. The output of calling this expression in both student and solution process is compared. - -### Example 2 - -Suppose now that inside the `for` loop of `print_dict()` from the previous example, you each time want to print out the length of the entire dictionary: - - *** =solution - ```{python} - def print_dict(my_dict): - for key, value in my_dict.items(): - print("total length: " + str(len(my_dict))) - print(key + " - " + str(value)) - ``` - -The SCT from before won't work out of the box, because now you also need a value for `my_dict` inside `test_expression_output()`, the test of the body of the `for` loop, but this value is not available. You cannot specify this value through `context_vals`, because the context variables are already updated to be `"key"` and `"value"`. To be able to test this appropriately, you'll have to set `extra_env` inside `test_expression_output()`: - - *** =sct - ```{python} - def fun_body_test(): - def for_iter_test(): - example_dict = {'a': 2, 'b': 3} - test_expression_result(context_vals = [example_dict]) - def for_body_test(): - example_dict = {'a': 2, 'b': 3} - test_expression_output(context_vals = ['c', 3], extra_env = {'my_dict': example_dict}) - - test_for_loop(for_iter = for_iter_test, body = for_body_test) - - test_function_definition('print_dict', body = fun_body_test) - ``` - -With this update of the SCT, the exercise will still run fine. - diff --git a/docs/source/pythonwhat.wiki/test_expression_result.md b/docs/source/pythonwhat.wiki/test_expression_result.md deleted file mode 100644 index 4f82c059..00000000 --- a/docs/source/pythonwhat.wiki/test_expression_result.md +++ /dev/null @@ -1,19 +0,0 @@ -test_expression_result ----------------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_expression_result.test_expression_result -``` - - def test_expression_result(extra_env=None, - context_vals=None, - incorrect_msg=None, - eq_condition="equal", - expr_code=None, - pre_code=None, - keep_objs_in_env=None, - error_msg=None) - -`test_expression_result()` works pretty much the same as `test_expression_output()` and takes the same arguments. However, in this case, the expression should be a single expression and can't be a 'tree of expressions', such as the entire body of a function definition for example. Currently, the only places where `test_expression_result()` is used, is inside inherently 'single expression parts' of your code, such as the sequence specification of a `for` loop, the expression of a lambda function, etc. - -The example in the [`test_expression_output()` article](test_expression_output.md) also explains the use of `test_expression_result()`. diff --git a/docs/source/pythonwhat.wiki/test_for_loop.md b/docs/source/pythonwhat.wiki/test_for_loop.md deleted file mode 100644 index 4b4545d1..00000000 --- a/docs/source/pythonwhat.wiki/test_for_loop.md +++ /dev/null @@ -1,88 +0,0 @@ -test_for_loop -------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_for_loop - :members: -``` - - - test_for_loop(index=1, - for_iter=None, - body=None, - orelse=None, - expand_message=True) - -As the name suggesets, you can use `test_for_loop()` to test if a for loop was properly coded. Similar to how `test_if_else()` and `test_while_loop()` works, `test_for_loop()` parses the for loop in the student's submission and breaks it up into its composing parts. Next, it also parses the for loop in the solution solution, and compares the parts between student submission and solution. It does this through sub-SCTs that you specify in `cond_test` and `expr_test`. - -### Example 1 - -Suppose you want the student to implement an algorithm that calculates fibonacci's row (until `n = 20`) using a simple for loop. The solution could look like this: - - *** =solution - ```{python} - # Initialise the row - fib = [0, 1] - - # Update the row correctly each loop - for n in range(2, 20): - fib.append(fib[n-2] + fib[n-1]) - ``` - -An SCT to accompany this exercise could be the following: - - *** =sct - ```{python} - def test_for_iter(): - "You have to iterate over `range(2, 20)`" - test_function("range", - not_called_msg=msg, - incorrect_msg=msg) - - def test_for_body(): - msg = "Make sure your row, `fib`, updates correctly" - test_object_after_expression("fib", - extra_env={ "fib": [0, 1, 1, 2] }, - context_vals=[4], - undefined_msg=msg, - incorrect_msg=msg) - test_for_loop(index=1, - for_iter=test_for_iter, - body=test_for_body) - ``` - - -Notice that two self-defined functions, `test_for_iter()` and `test_for_body()` are used to specify the sub-SCTs for the different parts in the `for` loop. With `index = 1`, you tell `pythonwhat` that you want to check the first `for` loop you find in the student submission with the first `for` loop in the solution. - - -The `for_iter` part of `test_for_loop()` tests whether the loop with index 1 loops over the correct range. The tests in this sub-SCT are run on the sequence part of the loop, which in this case for the solution is `range(2, 20)`. With `test_function()`, we can test this. In other cases, you could use e.g. `test_expression_result()`, to test the result of the sequence part. - -The `body` part of `test_for_loop()` tests whether `fib` is updated correctly. The tests in this sub-SCT are run on the body of the loop. The `test_object_after_expression()`. This function will test an object after the active expression is run in the student and solution process. In this case it will check if `fib` is updated the same in the student and solution process after one loop through the body of the `for`. Two important arguments for `test_object_after_expression()` are: - -- `extra_env = { "fib": [0, 1, 1, 2] }`: when running the body of the for loop, the process will be updated with these extra environment variables. In this case this means that before the body is ran, `fib` will be initialised to `[0, 1, 1, 2]`. -- `context_vals = [4]`: this argument contains the values of the loop's variable. In the solution code, for example, there will be one: `n`. This means that `n` will be initialised to `4` in the solution process when the body of the for loop is run. The student can give any name to `n`, as long as the functionality remains the same. - -You may have noticed that the helper functions that are used within `test_for_loop()` contain feedback messages as well. When they are used within a `test_for_loop()`, these messages will automatically be extended with "in the ___ of the for loop on line ___.". To avoid this extension, you could set the option `expand_message = False` in `test_for_loop()`. - -Example 2: Multiple context vals - -If you have multiple context vals, things largely work the same way. Suppose you want somebody to print out the keys and values of a dictionary as follows: - - *** =solution - ```{python} - my_dict = {'a': 1, 'b': 2, 'c': 3} - for k, v in my_dict.items(): - print(k + ' - ' + str(v)) - ``` - -An appropriate SCT would be: - - *** =sct - ```{python} - test_object('my_dict') - test_for_loop(index=1, - for_iter = lambda: test_expression_result(), - body = lambda: test_expression_output(context_vals = ['a', 1])) - ``` - -In this case, when you're checking the output of the body of the `for` loop, you're telling `k` to be `'a'` and `v` to be `1`. diff --git a/docs/source/pythonwhat.wiki/test_function.md b/docs/source/pythonwhat.wiki/test_function.md deleted file mode 100644 index ff5d43f6..00000000 --- a/docs/source/pythonwhat.wiki/test_function.md +++ /dev/null @@ -1,208 +0,0 @@ -test_function -------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_function.test_function -``` - - test_function(name, - index=1, - args=None, - keywords=None, - eq_condition="equal", - do_eval=True, - not_called_msg=None, - incorrect_msg=None) - -`test_function()` enables you to test whether the student called a function correctly. The function first tests if the specified function is actually called by the student, and then compares the call with calls of the function in the solution code. Next, it can compare the parameters passed to these functions. Because `test_function()` also uses the student and solution process, this can be done in a very concise way. - -### Example 1 - -Suppose you want the student to call the `round()` function on pi, as follows: - - *** =solution - ```{python} - # This is pi - pi = 3.14159 - - # Round pi to 3 digits - r_pi = round(pi, 3) - ``` - -The following SCT tests whether the `round()` function is used correctly: - - *** =sct - ```{python} - test_function("round") - success_msg("Great job!") - ``` - -This is a very robust way to test whether `round()` is used, much more robust when comparing to `test_student_typed()`. `test_function()` tests whether the student has called the function `round()` and checks whether the values of the arguments are the same as in the solution. So in this case, it tests wether `round()` is used with its first argument equal to `3.14159` and the second argument equal to `3`. `test_function()` figures out the values of these arguments from the solution code and the solution processes that corresponds with it. The above SCT would accept all of the following student submissions: - -- `round(3.14159, 3)` -- `pi = 3.14159; dig = 3; round(pi, dig)` -- `int_part = 3; dec_part = 0.14159; round(int_part + dec_part, 3)` - -By default, `test_function()` tests all arguments that are specified in the solution code. It is also possible to check whether a function is used and only check specific positional arguments. For example, - - *** =sct - ```{python} - test_function("round", args=[0]) - success_msg("Great job!") - ``` - -will only test whether the first argument's value is `3.14159`. A student submission that is `round(pi, 5)` would also pass this SCT. - -With `args`, you can also control whether or not to actually check the values that were passed as parameters. Say you only want to check that the function `round()` was called: - - *** =sct - ```{python} - test_function("round", args=[]) - success_msg("Great job!") - ``` - - -`test_function()` will automatically generate meaningful feedback, but you can also override these messages with `not_called_msg` and `incorrect_msg`. The former controls the message that is thrown if the student didn't call the specified function in the first place. The latter is thrown if the student did not correctly set the arguments in the function call: - - *** =sct - ```{python} - test_function("round", - not_called_msg = "You did not call `round()` to round the irrational number, `pi`.", - incorrect_msg = "Be sure to round `pi` to `3` digits.`) - success_msg("Great job!") - ``` - - - -### Example 2: Multiple function calls - -`index`, which is 1 by default, becomes important when there are several calls of the same function. Suppose that your exercise requires the student to call the `round()` function twice: once on `pi` and once on `e`, Euler's number. A possible solution could be the following: - - *** =solution - ```{python} - # Call round on pi - round(3.14159, 3) - - # Call round on e - round(2.71828, 3) - ``` - -To test both these function calls, you'll need the following SCT: - - *** =sct - ```{python} - test_function("round", index=1) - test_function("round", index=2) - success_msg("Two in a row, great!") - ``` - -The first `test_function()` call, where `index=1`, checks the solution code for the first function call of `round()`, finds it - `round(3.14159, 3)` - and then goes to look through the student code to find a function call of `round()` that matches the arguments. It is perfectly possible that there are 5 function calls of `round()` in the student's submission, and that only the fourth call matches the requirements for `test_function()`. As soon as a function call is found in the student code that passes all tests, `pythonwhat` heads over to the second `test_function()` call, where `index=2`. The same thing happens: the second call of `round()` is found from the solution code, and a match is sought for in the student code. This time, however, the function call that was matched before is now 'blacklisted'; it is not possible that the same function call in the student code causes both `test_function()` calls to pass. - -This means that all of the following student submissions would be accepted: - - - `round(3.14159, 3); round(2.71828, 3)` - - `round(2.71828, 3); round(3.14159, 3)` - - `round(3.14159, 3); round(123.456); round(2.71828, 3)` - - `round(2.71828, 3); round(123.456); round(3.14159, 3)` - -Of course, you can also specify all other arguments to customize your test, such as `do_eval`, `args`, `not_called_msg` and `incorrect_msg`. - -### Example 3: Custom feedback - -By default `test_function()` checks all arguments and keywords that are specified in the solution; if you specify `incorrect_msg`, any error to one of these arguments will replaced by the same custom message. If you want to provide different custom error messages for different arguments, you can do so with multiple function calls. To, for example, provide different feedback for the first and second argument of the `round()` function: - - *** =sct - ```{python} - test_function("round", args = [0], index=1, incorrect_msg = 'first arg wrong!') - test_function("round", args = [1], index=1, incorrect_msg = 'second arg wrong!') - success_msg("Well done") - ``` - -**NOTE**: currently, `test_function()` automatically checks all arguments and keywords that you specify in corresponding function call in the solution. Therefore, if you want to give specific feedback, make sure to select a single argument or a single keyword. To check the first argument, you can best use `args = [0], keywords = []`, to test a keyword named `check`, you'll want to use `args = [], keywords = ['check']`. - -### Example 4: Methods - -Python also features methods, i.e. functions that are called on objects. For testing such a thing, you can also use `test_function()`. Consider the following solution code, that creates a connection to an SQLite Database with `sqlalchemy`. - - *** =solution - ```{python} - from urllib.request import urlretrieve - from sqlalchemy import create_engine, MetaData, Table - engine = create_engine('sqlite:///census.sqlite') - metadata = MetaData() - connection = engine.connect() - from sqlalchemy import select - census = Table('census', metadata, autoload=True, autoload_with=engine) - stmt = select([census]) - - # execute the query and fetch the results. - connection.execute(stmt).fetchall() - ``` - -To test the last chained method calls, you can use the following SCT. Notice from the second `test_function()` call here that you have to describe the entire chain (leaving out the arguments that are passed to `execute()`). This way, you explicitly list the order in which the methods should be called. - - *** =sct - ``` - test_function("connection.execute", do_eval = False) - test_function("connection.execute.fetchall") - ``` - -**NOTE**: currently, it is not possible to easily test the arguments inside chained method calls, methods inside arguments, etc. We are working on a massive update of `pythonwhat` to easily support this very customized testing, with virtually no limit to 'how deep you want the tests to go'. More on this later! - -### `do_eval` - -With `do_eval`, you can control how arguments are compared between student and solution code. - -- If `do_eval` is `True`, the evaluated version of the arguments are compared; -- If `do_eval` is `False`, the 'string version' of the argumetns are compared; -- If `do_eval` is `None`, the arguments are not compared; in this case, `test_function()` simply checks if you specified the arguments, without further checks. - - -### Function calls in packages - -If you're testing whether function calls of particular packages are used correctly, you should always refer to these functions with their 'full name'. Suppose you want to test whether the function `show` of `matplotlib.pyplot` was used correctly, you should use - - *** =sct - ```{python} - test_function("matplotlib.pyplot.show") - ``` - -The `test_function()` call can handle it when a student used aliases for the python packages (all `import` and `import * from *` calls are supported). In case there is an error, `test_function()` will automatically generated a feedback message that uses the alias of the student. - -**NOTE:** No matter how you import the function, you always have to refer to the function with its full name, e.g. `package.subpackage1.subpackage2.function`. - -### Argument equality - -Just like with `test_object()`, evaluated arguments are compared using the `==` operator (check out [the section about Object equality](https://github.com/datacamp/pythonwhat/wiki/test_object#object-equality)). For a lot of complex objects, the implementation of `==` causes the object instances to be compared... not their underlying meaning. For example when the solution is: - - *** =solution - from urllib.request import urlretrieve - fn1 = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_998/datasets/Chinook.sqlite' - urlretrieve(fn1, 'Chinook.sqlite') - - # Import packages - from sqlalchemy import create_engine - import pandas as pd - - # Create engine: engine - engine = create_engine('sqlite:///Chinook.sqlite') - - # Execute query and store records in dataframe: df - df = pd.read_sql_query("SELECT * FROM Album", engine) - -And the SCT is: - - *** =sct - test_function("pandas.read_sql_query") - -The SCT will fail even if the student uses this exact solution code. The reason being that the `engine` object is compared in the solution and student process. The engine object is evaluated by `create_engine('sqlite:///Chinook.sqlite')`. As you can try out yourself, `create_engine('sqlite:///Chinook.sqlite') == create_engine('sqlite:///Chinook.sqlite')` will always be `False`, even though they are semantically exactly the same. A better way of testing this code would be: - - *** =sct - test_correct( - lambda: test_object("df"), - lambda: test_function("pandas.read_sql_query", do_eval=False) - ) - -This SCT will not do exactly the same, but it will test enough in practice 99% of the time. Check out [the section about Object equality](https://github.com/datacamp/pythonwhat/wiki/test_object#object-equality) for complex objects that do have a good equality implementation. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). diff --git a/docs/source/pythonwhat.wiki/test_function_definition.md b/docs/source/pythonwhat.wiki/test_function_definition.md deleted file mode 100644 index 293b4792..00000000 --- a/docs/source/pythonwhat.wiki/test_function_definition.md +++ /dev/null @@ -1,194 +0,0 @@ -test_function_definition ------------------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_function_definition - :members: -``` - - def test_function_definition(name, - arg_names=True, - arg_defaults=True, - body=None, - results=None, - outputs=None, - errors=None, - not_called_msg=None, - nb_args_msg=None, - other_args_msg=None, - arg_names_msg=None, - arg_defaults_msg=None, - wrong_result_msg=None, - wrong_output_msg=None, - no_error_msg=None, - wrong_error_msg=None, - expand_message=True): - - -In more advanced courses, you'll sometimes want students to define their own functions. With `test_function_definition()` it is possible to test such user-defined functions in a robust way. This function allows you to test four things: - -1. The argument names of the function (including if the correct defaults are used) -2. The body of the functions (does it output correctly, are the correct functions used) -3. The return value with a certain input -4. The output value with a certain input - -### Example 1 - -Say you want a student to write a very basic function to set numbers in a base from 1 up until 9 to a decimal. To not overcomplicate things you just ask them to implement the basic functionality; they don't have to catch any exceptions. A solution to the exercise can like like this: - - *** =solution - ```{python} - def to_decimal(number, base = 2): - print("Converting %d from base %s to base 10" % (number, base)) - number_str = str(number) - number_range = range(len(number_str)) - multipliers = [base ** ((len(number_str) - 1) - i) for i in number_range] - decimal = sum([int(number_str[i]) * multipliers[i] for i in number_range]) - return decimal - ``` - -You could test the function like this: - - *** =sct - ```{python} - # All of the following test_function_definition() functions are done on the same - # function definition. - - # Test the function, see that the defaults of the arguments are the same. - # For this function, we don't care about the argument names of the function. - # Note: generally, we DO care about the names of the arguments, since they can - # be used as keywords. arg_defaults and arg_names will be set to True by default. - - test_function_definition("to_decimal", arg_defaults = True, arg_names = False) - - # Here, a feedback message will be generated. You can overwrite this feedback - # message by using: - # test_function_definition("to_decimal", arg_defaults = True, arg_names = False, - # arg_defaults_msg = "Use the correct default argument values!") - # In the following tests, I'll always use the standard feedback messages, remember they - # can almost always be overwritten. - - # We want to test whether the function returns the correct things with certain inputs. - - test_function_definition("to_decimal", arg_names = False, arg_defaults = False, # Already tested this - results = [ - [1001101, 2], - ]1212357, 8] - ) - - # This will run to_decimal(1001101, 2) and to_decimal(1212357, 8) in student and solution - # process, and match the results. If they don't match, a feedback message will be generated. - # Note: here we've set arg_defaults to False, because we already tested this in the first - # test_function_definition. - - # We want to test the output of the function with certain inputs. - - test_function_definition("to_decimal", arg_names = False, arg_defaults = False, # Already tested this - outputs = [ - [1234, 6], - [8888888, 9] - ) - - # This will run to_decimal(1234, 6) and to_decimal(8888888, 9) in solution and student - # process and compare their printed output. - - # Finally, we might want them to use a certain function. For this we can do tests specifically - # on the body of the function. Remember you can use lambda functions or custom functions for this - # (also see wiki about test_if_else(), test_for_loop() and test_while_loop(). - - test_function_definition("to_decimal", arg_names = False, arg_defaults = False, # Already tested this - body = lambda: test_function("sum", args = [], incorrect_msg = "you should use the `sum()` function.")) - - # This will test the body of the function definition, and see if the function sum() is used. - # Note that the generated feedback will be preceded by: 'In your definition of `to_decimal()`, ...' - # So if the last test doesn't pass, this feedback will be generated: - # In your definition of `to_decimal()`, you should use the `sum()` function. - ``` - -Pitfall: you have to watch out when using `test_function()` in a body test, you should never test arguments -that are only defined within the scope of the function (e.g. function parameters). This is the reason why -we used `args = []` in the last test, because the argument used in `sum()` can not be calculated to verify -in the global scope. This is something which would require architectural changes in the `pythonwhat` package. - - -### Example 2: User-defined errors - -In some cases, you'll want the student to code resilience against incorrect inputs or behavior. To test this, you can use the `errors`, `no_error_msg` and `wrong_error_msg` arguments. The first is similar to `results`, and specifies the input arguments as a list of tuples or a list of lists, that have to generate an error. With `no_error_msg` you can control the message that is presented if running one of these argument sets does not generate an error, while it should. With `wrong_error_msg`, you control the message that is presented if the type of the error (or exception) that is thrown does not correspond to the type that is thrown when the function is called in the solution process. - -Suppose you want the student to code up a function `inc`, that increments a number if it's positive. If it's not, you want the function to raise a `ValueError`. A solution could look like this: - - *** =solution - ```{python} - def inc(num): - if num < 0: - raise ValueError('num is negative') - return(num + 1) - ``` - -To test this, we can use the following SCT (we're only focussing on the `errors` part here; of course you can extend the `test_function_definition()` call with more checks on arguments, `results`, body, etc.): - - *** =sct - ```{python} - test_function_definition("inc", errors = [[-1]]) - ``` - -If the student submits the following code: - -``` -def inc(num): - return(num + 1) -``` - -the SCT will see it's incorrect and throw the message: _Calling `inc(-1)` doesn't result in an error, but it should!_ - -If the student submits the following code: - -``` -def inc(num): - if num < 0: - raise NameError('num is negative') - return(num + 1) -``` - -the SCT will see it's incorrect and throw the message: _Calling `inc(-1)` should result in a `ValueError`, instead got a `NameError`._ - -Currently, there isn't a way to test the actual message you pass with errors you raise. - -### Example 3: `*args` and `**kwargs` - -When defining a function in Python, it also possible to specify so-called 'unordered non-keyword arguments', with a `*`, and 'unordered keyword arguments'. Typically, these are called `args` and `kwargs` respectively, but this is not required. - -Have a look at the following example: - - *** =solution - ```{python} - def my_fun(x, y = 4, z = ['a', 'b'], *args, **kwargs): - k = len(args) - l = len(kwargs) - print("just checking") - return k + l - ``` - -An SCT to check this function definition: - - *** =sct - ```{python} - def inner_test(): - context = ['r', 's', ['c', 'd'], ['t', 'u'], {'a': 2, 'b': 3, 'd':4}] - test_object_after_expression('k', context_vals = context) - test_object_after_expression('l', context_vals = context) - test_function_definition("my_fun", body = inner_test, - results = [{'args': ['r', 's', ['c', 'd'], 't', 'u', 'v'], 'kwargs': {'a': 2, 'b': 3, 'd': 4}}], - outputs = [{'args': ['r', 's', ['c', 'd'], 't', 'u', 'v'], 'kwargs': {'a': 2, 'b': 3, 'd': 4}}]) - ``` - -There are different things to note: - -- By default, the names of the `*` argument and the `**` argument are checked, if they are defined in the solution. This is controlled through `arg_names`, just like for 'regular' arguments. To override the automatic message that is thrown if the `*` or `**` arg is not specified or not appropriately named, use `other_args_msg`. -- The `*` and `**` args are also part of the context values that you can specify in 'inner tests'. They are appended to the normal arguments: first the `*`, then the `**` argument. You can see in the `context` object, that the penultimate element is used to specify the `*args` argument, and the last element, a dictionary, is used to specify the `**` argument. -- Before, you saw that `results`, `outputs`, and `errors` should be a list of lists, where the inner list is the list of arguments. To also cater for explicitly keyworded arguments, you can also specify a list of dictionaries. Each dictionary represents one call of the user-defined fucntion and should contain two elements: `'args'` and `'kwargs'`. Behind the scenes, the function will be called as: `my_fun([*d['args'], **d['kwargs']])`, where `d` is the two-key dictionary. - - -### Sidenote - -Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_function_v2.md b/docs/source/pythonwhat.wiki/test_function_v2.md deleted file mode 100644 index 4e3c6395..00000000 --- a/docs/source/pythonwhat.wiki/test_function_v2.md +++ /dev/null @@ -1,297 +0,0 @@ -test_function_v2 ----------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_function.test_function_v2 -``` - - test_function_v2(name, - index=1, - params=None, - signature=None, - eq_condition="equal", - do_eval=True, - not_called_msg=None, - params_not_matched_msg=None, - params_not_specified_msg=None, - incorrect_msg=None) - -`test_function_v2()` enables you to test whether the student called a function correctly. The function first tests if the specified function is actually called by the student, and then compares the call with calls of the function in the solution code. Next, it can compare the arguments passed to these functions. Because `test_function_v2()` also uses the student and solution process, this can be done in a very concise way. `test_function_v2()` is an improved version of [`test_function()`](test_function.md), where: - -- there is resilience against different ways of calling a function (arguments vs keywords), -- you have to be specific about which parameters you want to check, -- you can specify parameter-specific evaluation forms (`do_eval` can be a list), -- you can specify parameter-specific custom messages (`params_not_matched_msg` and `params_not_specified_msg` can be lists), -- you have more control over messaging in general. - -### Example 1 - -Suppose you want the student to call the `round()` function on pi, as follows: - - *** =solution - ```{python} - # This is pi - pi = 3.14159 - - # Round pi to 3 digits - r_pi = round(pi, 3) - ``` - -The following SCT tests whether the `round()` function is used correctly: - - *** =sct - ```{python} - test_function_v2("round", params=["number", "ndigits"]) - success_msg("Great job!") - ``` - -`test_function_v2()` tests whether the student has called the function `round()` and checks whether the values of the arguments are the same as in the solution. So in this case, it tests whether `round()` is used and the `number` and `ndigits` parameters, that `round()` expects, are specified correctly, i.e. equal to `3.14159` and `3` respectively. `test_function_v2()` figures out the values of these arguments from the solution code and the solution process that corresponds with it. The above SCT would accept all of the following student submissions: - -- `round(3.14159, 3)` -- `round(number=3.14159, 3)` -- `round(number=3.14159, ndigits=3)` -- `round(ndigits=3, number=3.14159)` -- `pi=3.14159; dig=3; round(pi, dig)` -- `pi=3.14159; dig=3; round(number=pi, dig)` -- `int_part = 3; dec_part = 0.14159; round(int_part + dec_part, 3)` - -In `params`, you have to explicitly list all the parameters that you want to test. If you only want to check the `number` parameter, for example, you can use: - - *** =sct - ```{python} - test_function_v2("round", params=["number"]) - success_msg("Great job!") - ``` - -This SCT will only test whether the `number` parameter was specified to be `3.14159`. If a student submits `round(pi, 5)`, this would also pass this SCT. - -If you specify `params` to be an empty list, which is the default, you are simply checking whether the `round()` function was called in the first place: - - *** =sct - ```{python} - test_function_v2("round") # same as test_function_v2("round", params=[]) - success_msg("Great job!") - ``` - -`test_function_v2()` will automatically generate meaningful feedback, but you can also override these messages through the different `*_msg` parameters that `test_function_v2()` features: - -- `not_called_msg`: message if the student didn't call the specified function or didn't call the specified function often enough (if you're testing multiple calls of the same function in the same submission). -- `params_not_matched_msg`: message if the function call of the student was invalid, i.e. if the way of specifying the different parameters was invalid. -- `params_not_specified_msg`: message if the student did not specify all parameters that are specified inside `params`. This argument can either be a string, to give the same message for each parameter that is missing, or a list of strings with the same length as `params`. In case of a missing parameter, `test_function_v2()` will present the corresponding message. -- `incorrect_msg`: message if the student did not specify all parameters correctly, so when his or her specifications don't correspond with the solution. This argument can again be a single string, or a list of parameter-specific feedback messages. - -Below is an example of an SCT that specified all feedback messages. This is not required, though; you can depend on the automatic feedback messages for the `not_called_msg`, `params_not_specified_msg` and `incorrect_msg` and only manually specify the `params_not_matched_msg`, for example. - - *** =sct - ```{python} - test_function_v2("round", params=["number", "ndigits"] - not_called_msg="You did not call `round()` to round the irrational number, `pi`.", - params_not_matched_msg="Are you sure you correctly called the `round()` function?", - params_not_specified_msg="Make sure to specify both the `number` and `ndigits` parameter!", - incorrect_msg=["Make sure to correctly specify `number`; it should be `pi`, or `3.14159`.", - "Have you specified `ndigits` so that `pi` is rounded to 3 digits?"]) - success_msg("Great job!") - ``` - - -### Example 2: Multiple function calls - -`index`, which is 1 by default, becomes important when there are several calls of the same function. Suppose that your exercise requires the student to call the `round()` function twice: once on `pi` and once on `e`, Euler's number. A possible solution could be the following: - - *** =solution - ```{python} - # Call round on pi - round(3.14159, 3) - - # Call round on e - round(2.71828, 3) - ``` - -To test both these function calls, you'll need the following SCT: - - *** =sct - ```{python} - test_function_v2("round", params=["number","ndigits"], index=1) - test_function_v2("round", params=["number","ndigits"], index=2) - success_msg("Two in a row, great!") - ``` - -The first `test_function_v2()` call, where `index=1`, checks the solution code for the first function call of `round()`, finds it - `round(3.14159, 3)` - and then goes to look through the student code to find a function call of `round()` that matches the arguments. It is perfectly possible that there are 5 function calls of `round()` in the student's submission, and that only the fourth call matches the requirements for `test_function_v2()`. As soon as a function call is found in the student code that passes all tests, `pythonwhat` heads over to the second `test_function_v2()` call, where `index=2`. The same thing happens: the second call of `round()` is found from the solution code, and a match is sought for in the student code. This time, however, the function call that was matched before is now 'blacklisted'; it is not possible that the same function call in the student code causes both `test_function_v2()` calls to pass. - -This means that all of the following student submissions would be accepted: - - - `round(3.14159, 3); round(2.71828, 3)` - - `round(2.71828, 3); round(3.14159, 3)` - - `round(number=3.14159, ndigts=3); round(number=2.71828, 3)` - - `round(number=2.71828, 3); round(number=3.14159, 3)` - - `round(3.14159, 3); round(123.456); round(2.71828, 3)` - - `round(2.71828, 3); round(123.456); round(3.14159, 3)` - -Of course, you can also specify all other arguments to customize your test to perfection, such as custom messages and `do_eval` (example 3). - -### Example 3: `do_eval` - -With `do_eval`, you can control how parameter specifications are compared between student and solution code. There are two ways to specify `do_eval`: you can specify a single value, that will be used for comparing all `params` that you specified. However, you can also specify a list of values, with the same length as `params`; the way in which parameter specifications are compared becomes parameter specific. In both cases, there are three valid values: - -- `True`, where the evaluated version of the student and solution arguments is compared. -- `False`, where the 'string version' of the arguments is compared; -- `None`, in which case the arguments are not compared; `test_function_v2` simply checks if the parameter(s) in question has/have been specified. - -Say, for example, you want to check if a student called the `round()` function and specified the parameters `number` and `ndigits`. You want to test the actual equality of `number`, but you don't care about the value of `ndigits`, you just want to make sure the student specified it, nothing more. - -The following solution and SCT implement this train of thought (custom feedback messages have not been specified, although this is perfectly possible): - - *** =solution - ```{python} - # This is pi - pi = 3.14159 - - # Round pi to 3 digits - r_pi = round(pi, 3) - ``` - - *** =sct - ```{python} - test_function_v2("round", - params=["number", "ndigits"], - do_eval=[True, None]) - success_msg("Great job!") - ``` - -All of the following submissions would be accepted by this SCT: - -- `round(pi, 3)` -- `round(number=pi, ndigits=3)` -- `round(number=pi, ndigits=4)` -- `round(pi, 4)` -- `round(pi, 0)` - -### Example 4: Function calls in packages - -If you're testing whether function calls of particular packages are used correctly, you should always refer to these functions with their 'full name'. Suppose you want to test whether the function `show` of `matplotlib.pyplot` was used correctly, you should use - - *** =sct - ```{python} - test_function_v2("matplotlib.pyplot.show") - ``` - -The `test_function_v2()` call can handle it when a student used aliases for the python packages (all `import` and `import * from *` calls are supported). In case there is an error, `test_function_v2()` will automatically generated a feedback message that uses the alias that the student used. - -**NOTE:** No matter how you import the function, you always have to refer to the function with its full name, e.g. `package.subpackage1.subpackage2.function`. - -### Example 5: Manual signatures - -To implement resilience against different ways of specify function parameters, the `inspect` module is used, that is part of Python's basic distribution. Through `inspect.signature()` a function's parameters can be inferred, and then 'bound' to the arguments that the student specified. However, this signature is not available for all of Python's functions. More specifically, Python's built-in functions that are implemented in C don't allow a signature to be extracted from them. `pythonwhat` already includes manually specified signatures for functions such as `print()`, `str()`, `hasattr()`, etc, but it's still possible that some signatures are missing. - -That's why `test_function_v2()` features a `signature` parameter, that is `None` by default. If `pythonwhat` can't retrieve a signature for the function you want to test, you can pass an object of the class `inspect.Signature` to the `signature` parameter. - -Suppose, for the sake of example, that `test_function_v2()` can't find a signature for the `round()` function (you will be informed by this through automated testing; running the solution against an SCT that depends on a signature that is not found will throw a backend error). To be able to implement this function test, you can use the `sig_from_params()` function: - - *** =sct - ```{python} - sig = sig_from_params(param("number", param.POSITIONAL_OR_KEYWORD), - param("ndigits", param.POSITIONAL_OR_KEYWORD, default=0)) - test_function_v2("round", params=["number", "ndigits"], signature=sig) - ``` - -`param` is an alias of the `Parameter` class that's inside the `inspect` module. You can pass `sig_from_params()` as many parameters as you want. The first argument of `param()` should be the name of the parameter, the second argument should be the 'kind' of parameter. `param.POSITIONAL_OR_KEYWORD` tells `test_function_v2` that the parameter can be specified either through a positional argument or through a keyword argument. Other common possibilities are `param.POSITIONAL_ONLY` and `param.KEYWORD_ONLY` (for a full list, refer to the [Python docs on `inspect`](https://docs.python.org/3.4/library/inspect.html#inspect.Parameter)). The third, optional argument, allows you to specify a default value for the parameter. - -**NOTE:** If you find vital Python functions that are used very often and that are not included in `pythonwhat` by default, you can [let us know](mailto:content-engineering@datacamp.com) and we'll add the function to our [list of manual signatures](https://github.com/datacamp/pythonwhat/blob/master/pythonwhat/signatures.py). - -### Example 6: Methods - -Python also features methods, i.e. functions that are called on objects. For testing such a thing, you can also use `test_function_v2()`. Consider the following solution code, that creates a connection to an SQLite Database with `sqlalchemy`. - - *** =solution - ```{python} - # Prepare everything -from urllib.request import urlretrieve -from sqlalchemy import create_engine, MetaData, Table -engine = create_engine('sqlite:///census.sqlite') -metadata = MetaData() -connection = engine.connect() -from sqlalchemy import select -census = Table('census', metadata, autoload=True, autoload_with=engine) -stmt = select([census]) - - # execute the query and fetch the results. - connection.execute(stmt).fetchall() - ``` - -To test the last chained method calls, you can use the following SCT. Notice from the second `test_function_v2()` call here that you have to describe the entire chain (leaving out the arguments that are passed to `execute()`). This way, you explicitly list the order in which the methods should be called. - - *** =sct - ```{python} - test_function_v2("connection.execute", params = ["object"], do_eval = False) - test_function_v2("connection.execute.fetchall") - ``` - -**NOTE**: currently, it is not possible to easily test the arguments inside chained method calls, methods inside arguments, etc. We are working on a massive update of `pythonwhat` to easily support this very customized testing, with virtually no limit to 'how deep you want the tests to go'. More on this later! - -### Example 7: Signatures for methods - -In the previous example, you might have noticed that `test_funtion_v2()` was capable to infer that `connection` is a `Connection` object, and that `execute()` is a method of the `Connection` class. For checking method calls that aren't chained, this is possible, but for chained method calls, such as `connection.execute.fetchall`, this is not possible. In those cases you'll have to manually specify a signature. With `sig_from_obj()` you can specify the function from which to extract a signature. - -The following full example shows how it's done: - - *** =pre_exercise_code - ```{python} - class Test(): - def __init__(self, a): - self.a = a - - def set_a(self, value): - self.a = value - return(self) - x = Test(123) - ``` - - *** =solution - ```{python} - x.set_a(843).set_a(102) - ``` - - *** =sct - ```{python} - sig = sig_from_obj('x.set_a') - test_function_v2('x.set_a.set_a', params=['value'], signature=sig) - ``` - -**NOTE**: You can also use the `sig_from_params()` function to manually build the signature from scratch, but this this more work than simply specifying the function object as a string from which to extract the signature. - - -### Extra: Argument equality - -Just like with `test_object()`, evaluated arguments are compared using the `==` operator (check out [the section about Object equality](test_object.md#object-equality)). For a lot of complex objects, the implementation of `==` causes the object instances to be compared... not their underlying meaning. For example when the solution is: - - *** =solution - ``` - from urllib.request import urlretrieve - fn1 = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_998/datasets/Chinook.sqlite' - urlretrieve(fn1, 'Chinook.sqlite') - from sqlalchemy import create_engine - import pandas as pd - engine = create_engine('sqlite:///Chinook.sqlite') - - # Execute query and store records in dataframe: df - df = pd.read_sql_query("SELECT * FROM Album", engine) - ``` - -And the SCT is: - - *** =sct - ``` - test_function_v2("pandas.read_sql_query", params = ['sql', 'con'], do_eval = [True, False]) - ``` - -The SCT will fail even if the student uses this exact solution code. The reason being that the `engine` object is compared in the solution and student process. The engine object is evaluated by `create_engine('sqlite:///Chinook.sqlite')`. As you can try out yourself, `create_engine('sqlite:///Chinook.sqlite') == create_engine('sqlite:///Chinook.sqlite')` will always be `False`, even though they are semantically exactly the same. A better way of testing this code would be: - - *** =sct - test_correct( - lambda: test_object("df"), - lambda: test_function_v2("pandas.read_sql_query", do_eval=False) - ) - -This SCT will not do exactly the same, but it will test enough in practice 99% of the time. Check out [the section about Object equality](test_object.md#object-equality) for complex objects that DO have a good equality implementation. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_if_else.md b/docs/source/pythonwhat.wiki/test_if_else.md deleted file mode 100644 index df894987..00000000 --- a/docs/source/pythonwhat.wiki/test_if_else.md +++ /dev/null @@ -1,64 +0,0 @@ -test_if_else ------------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_if_else.test_if_else -``` - - test_if_else(index=1, - test=None, - body=None, - orelse=None, - expand_message=True) - -`test_if_else()` allows you to robustly check `if` statements, optionally extended with `elif` and `else` components. For each of the components of an if-else construct `test_if_else()` takes several 'sub-SCTs'. These 'sub-SCTs', that you have to pass in the form of lambda functions or through a function that defines all tests, are executed on these separate parts of the submission. - -### Example 1 - -Suppose an exercise asks the student to code up the following if-else construct: - - *** =solution - ```{python} - # a is set to 5 - a = 5 - - # If a < 5, print out "It's small", else print out "It's big" - if a < 5: - - else: - print("It's big") - ``` - -The `if-else` construct here consists of three parts: - -- The condition to check: `a < 5`. The `test` argument of `test_if_else()` specifies the sub-SCT to test this. -- The body of the `if` statement: `print("It's small")`. The `body` argument of `test_if_else()` specifies the sub-SCT to test this. -- The else part: `print("It's big")`. The `orelse` argument of `test_if_else()` specifies the sub-SCT to ttest this. - -You can thus write our SCT as follows. Notice that for the `test` argument a function is used to specify different tests; for the `body` and `orelse` arguments two lambda functions suffise. - - *** =sct - ```{python} - def sct_on_condition_test(): - test_expression_result({"a": 4}) - test_expression_result({"a": 5}) - test_expression_result({"a": 6}) - - test_if_else(index = 1, - test = sct_on_condition_test, - body = lambda: test_function("print") - orelse = lambda: test_function("print")) - ``` - -#### The `test` part - -Have a look at the `sct_on_condition_test()`, that is used to specify the sub-SCT for the `test` part of the if-else-construct, so `a < 5`. It contains three calls of the `test_expression_result` function. These functions are executed in a 'narrow scope' that only considers the condition of the student code, and the condition of the solution code. - -More specifically, `test_expression_result({"a": 5})` will check whether executing the `if` condition that the student coded when `a` equals 5 leads to the same result as executing the `if` condition that is coded in the solution when `a` equals 5. That way, you can robustly check the validity of the `if` test. There are three `test_expression_result()` calls to see if the condition makes sense for different inputs. - -Suppose that the student incorrectly used the condition `a < 6` instead of `a < 5`. `test_expression_result({"a": 5})` will see what the result is of `a < 6` if `a` equals 5. The result is `True`. Next, it checks the result of `a < 5`, the `if` condition of the solution, which is `False`. There is a mismatch between the 'student result' and the 'solution result', and a meaningful feedback messages is generated. - -#### The `body` and `orelse` parts - -In a similar way, the functions that are used as lambda functions in both the `body` and `orelse` part, will also be executed in a 'narrow scope', where only the `body` and `orelse` part of the student's submission and the solution are used. - diff --git a/docs/source/pythonwhat.wiki/test_if_exp.md b/docs/source/pythonwhat.wiki/test_if_exp.md deleted file mode 100644 index 9ff8f097..00000000 --- a/docs/source/pythonwhat.wiki/test_if_exp.md +++ /dev/null @@ -1,58 +0,0 @@ -test_if_exp ------------ - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_if_else.test_if_exp -``` - -`test_if_exp` is a wrapper around `test_if_else`, which tells it to look for inline `if` expressions. As such, it uses the same arguments. [See `test_if_else` for more info](test_if_else.md). - -### What is an inline `if` expression? - -An inline `if` expression looks like.. - -```{python} -x = 'a' if True else 'b' -``` - -This is in contrast to an `if` block, which looks like.. - -```{python} -if True: - x = 'a' -else: - x = 'b' -``` - -### Parts - -This test tries to break code into 3 parts, BODY, TEST, and ORELSE. -The table below shows an example inline `if` expression on the left, -and the parts that would be extracted on the right. - -| code | parts breakdown | -| ------------------------- | --------------------- | -| `x = 'a' if True else 'b'` | `x = BODY if TEST else ORELSE` | - -### Nested `if` expressions - -Just like `test_if_else`, `test_if_exp` will not find a nested `if` expression. -Instead, the nested portion will be inside one of the parts. -For example, below is an exercise with an `if` expression in the ORELSE part of another `if` expression. - -*** =solution -```{python} -x = 'a' if True else ('b' if False else 'c') -``` - -*** =sct -```{python} -test_if_exp(orelse=lambda: test_if_exp(orelse=lambda: test_student_typed('c'))) -``` - -The SCT above tests that the student typed 'c' in the ORELSE part of the inner `if` expression. -In parts, this looks like.. - -```{python} -BODY1 if TEST1 else (ORELSE1 = BODY2 if TEST2 else ORELSE2) -``` diff --git a/docs/source/pythonwhat.wiki/test_lambda_function.md b/docs/source/pythonwhat.wiki/test_lambda_function.md deleted file mode 100644 index b5a9bca5..00000000 --- a/docs/source/pythonwhat.wiki/test_lambda_function.md +++ /dev/null @@ -1,86 +0,0 @@ -test_lambda_function --------------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_lambda_function - :members: -``` - - def test_lambda_function(index, - arg_names=True, - arg_defaults=True, - body=None, - results=None, - errors=None, - not_called_msg=None, - nb_args_msg=None, - arg_names_msg=None, - arg_defaults_msg=None, - wrong_result_msg=None, - no_error_msg=None, - expand_message=True) - -With `test_function_definition()`, you can only test user-defined functions that have a name. There is an important class of functions in Python that go by the name of lambda functions. These functions are anonymous, so they don't necessarily require a name. To be able to test user-coded lambda functions, the `test_lambda_function()` is available. If you're familiar with `test_function_definition()`, you'll notice some similarities. However, instead of the `name`, you now have to pass the `index`; this means you have to specify the lambda function definition to test by number (test the first, second, third ...). Also, because we don't necessarily have an object represents a lambda function (because it can be anonymous), some tricky things are required to correctly specify the arguments `errors` and `results`; the example will give more details. - -### Example 1 - -Suppose we want the student to code a lambda function that takes two arguments, `word` and `echo`, the latter of which should have default value 1. The lambda function should return the product of `word` and `echo`. A solution to this challenge could be the following: - - *** =solution - ```{python} - echo_word = lambda word, echo = 1: word * echo - ``` - -To test this lambda function definition, you can use the following SCT: - - *** =sct - ``` - test_lambda_function(1, - body = lambda: test_student_typed('word'), - results = ["lam('test', 2)"], - errors = ["lam('a', '2')"]) - ``` - -With `1`, we tell `pythonwhat` to test the first lambda function it comes across. Through body, we can specify sub-SCTs to be tested on the body of the lambda function (similar to how `test_function_definition` does it). With `results` and `errors`, you can test the lambda function definition for different input arguments. Notice here that you have to specify a list of function calls as a string. The function you have to call is `lam()`; behind the scenes, this `lam` will be replaced by the actual lambda function the student and solution defined. This means that `lam('test', 2)` will be converted into: - - ``` - (lambda word, echo = 1: word * echo)('test', 2) - ``` - -That way, the system can run the function call, and compare the results between function and solution. Things work the same way for `errors`. - -As usual, the `test_lambda_function()` will generate a bunch of meaningful automated messages depending on which error the student made (you can override all these messages through the `*_msg` argument): - - submission: - feedback: "The system wants to check the first lambda function you defined but hasn't found it." - - submission: echo_word = lambda wrd: wrd * 1 - feedback: "You should define the first lambda function with 2 arguments, instead got 1." - - submission: echo_word = lambda wrd, echo: wrd * echo - feedback: "In your definition of the first lambda function, the first argument should be called word, instead got wrd." - - submission: echo_word = lambda word, echo = 2: word * echo - feedback: "In your definition of the first lambda function, the second argument should have 1 as default, instead got 2." - - submission: echo_word = lambda word, echo = 1: 2 * echo - feedback: "In your definition of the first lambda function, could not find the correct pattern in your code." - - submission: echo_word = lambda word, echo = 1: word * echo + 1 - feedback: "Calling the the first lambda function with arguments ('test', 2) should result in testtest, instead got an error." - - submission: echo_word = lambda word, echo = 1: word * echo * 2 - feedback = "Calling the first lambda function with arguments ('test', 2) should result in testtest, instead got testtesttesttest" - - submission: echo_word = lambda word, echo = 1: word * int(echo) - feedback: "Calling the first lambda function with the arguments ('a', '2') doesn't result in an error, but it should!" - - submission: echo_word = lambda word, echo = 1: word * echo - feedback: "Great job!" (pass) - - -### What about testing usage? - -This is practically impossible to do in a robust way; we suggest you do this in an indirect way (checking the output that should be generated, checking the object that should be created, etc). - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_object_after_expression.md b/docs/source/pythonwhat.wiki/test_object_after_expression.md deleted file mode 100644 index 43a08897..00000000 --- a/docs/source/pythonwhat.wiki/test_object_after_expression.md +++ /dev/null @@ -1,91 +0,0 @@ -test_object_after_expression ----------------------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_object_after_expression - :members: -``` - - test_object_after_expression(name, - extra_env=None, - context_vals=None, - undefined_msg=None, - incorrect_msg=None, - eq_condition="equal", - pre_code=None, - keep_objs_in_env=None) - -`test_object_after_expression()` is a function that is primarily used to check the correctness of the body of control statements. Through `extra_env` and `context_vals` you can adapt the student/solution environment with manual elements. Next, the 'currently active expression tree', such as the body of a for loop, is executed, and the resulting environment is inspected. This is done for both the student and the solution code, and afterwards the value of the object that you specify in `name` is checked for equality. With pre_code, you can prepend the execution of the default expression tree with some extra code, for example to set some variables. - -### Example 1: Function defintion - -Suppose you want to student to code up a function `shout()`, that adds three exclamation marks to every word you pass it: - - *** =solution - ```{python} - def shout(word): - shout_word = word + '!!!' - return shout_word - ``` - -To test whether the student did this appropriately, you want to first test whether `shout` is a user-defined function, and then whether inside the function, a new variable `shout_word` is created. Finally, you also want to check whether the result of calling `shout('hello')` is correct. The following SCT will do that for us: - - *** =sct - ```{python} - test_function_definition('shout', - body = test_object_after_expression('shout_word', context_vals = ['anything']), - results = [('hello')]) - ``` - -Let's focus on the `body` argument of `test_function_definition()` here, that uses `test_object_after_expression()`. For the other elements, refer to the [`test_function_definition()`](https://github.com/datacamp/pythonwhat/wiki/test_function_definition) article. - -The first argument of `test_object_after_expression()` tells the system to check the value of `shout_word` after executing the body of the function definition. Which part of the code to execute, the 'expression', is implicitly specified by `pythonwhat`. However, to run correctly, this expression has to know what `word` is. You can specify a value of `word` through the `context_vals` argument. It's a simple list: the first element of the list will be the value for the first argument of the function definition, the second element of the list will be the value for the second argument of the list, and so on. Here, there's only one argument, so a list with a single element, a string (that will be the value of the `word` variable), suffises. - -`test_object_after_expression()` will execute the expression, and run it on the solution side and the student side. On the solution side, the value of `shout_word` after the execution will be `'anything!!!'`. If the value on the student code is the same, we can rest assured that `shout_word` has been appropriately defined by the student and the test passes. - -### Example 2: for loop - -Suppose you want the student to build up a dictionary of word counts based on a list, as follows: - - *** =solution - ```{python} - words = ['it', 'is', 'a', 'the', 'is', 'the', 'a', 'the', 'it'] - counts = {} - for word in words: - if word in counts: - counts[word] += 1 - else: - counts[word] = 1 - ``` - -To check whether the `counts` list was correctly built, you can simply use `test_object()`, but you can also go deeper if it goes wrong. This calls for a `test_correct()` in combination with `test_object()` and `test_for()`, that in its turn uses `test_object_after_expression()`: - - - ``` =solution - def check_test(): - test_object('counts') - - def diagnose_test(): - body_test(): - test_object_after_expression('counts', - extra_env = {'counts': {'it': 1}}, - context_vals = ['it']) - test_object_after_expression('counts', - extra_env = {'counts': {'it': 1}}, - context_vals = ['is']) - test_for_loop(index = 1, - test = test_expression_result(), # Check if correct iterable used - body = body_test) - - test_correct(check_test, diagnose_test) - ``` - -Let's focus on the `body_test` for the for loop. Here, we're using `test_object_after_expression()` twice. - -In the first function call, we override the environment so that `counts` is a dictionary with a single key and value. Also, the context value, `word` in this case (the iterator of the `for` loop), is set to `it`. In this case, the body of the for loop - making abstraction of the if-else test - should increment the value of the value, without adding a new key. - -In the second function call, we override the environment so that `counts` is again a dictionary with a single key and value. This time, the context value is set to `is`, so a value that is not yet in the `counts` dictionary, so this should lead to a `counts` dictionary with two elements. - -As in the first example, `test_object_after_expression()` sets the environment variables and context values, runs the expression (in this case the entire body of the `for` loop), and then inspects the value of `counts` after this expression. The combination of the two `test_object_after_expression()` calls here, will indirectly check whether both the if and else part of the body has been correctly implemented. - - diff --git a/docs/source/pythonwhat.wiki/test_operator.md b/docs/source/pythonwhat.wiki/test_operator.md deleted file mode 100644 index dff7e113..00000000 --- a/docs/source/pythonwhat.wiki/test_operator.md +++ /dev/null @@ -1,62 +0,0 @@ -test_operator -------------- - -**THIS FUNCTION IS DEPRECATED AND WILL BE REMOVED IN A FUTURE RELEASE** - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_operator - :members: - -``` - - def test_operator(index=1, - eq_condition="equal", - used=None, - do_eval=True, - not_found_msg=None, - incorrect_op_msg=None, - incorrect_result_msg=None) - -Suppose you want the student to do some very basic operations using the `*` and the `**` operator. You could just ask the student to do some calculations and assign the result to a variable, `result` for example, and check that variable using `test_object()`. However this won't allow you to give the student very tailored feedback. Suppose you want to check if the student uses `**` and tell him/her if he/she doesn't! `test_object()` won't allow you to check this kind of specifics as it only checks resulting objects in both processes. Luckily, you can use another helper function, `test_operator()`. - -Say you want the student to calculate the future value of \$100 after 6 years. The interest rate 6% and you are using compound interest. This means the result has to be `100 * 1.06 ** 6`, so the solution code would be. - - *** =solution - ```{python} - # Calculate the future value of 100 dollar: result - result = 100 * 1.06 ** 6 - - # Print out the result - print(result) - ``` - -The SCT might look something like this, - - *** =sct - ```{python} - test_operator(index=1) - test_object("result") - test_function("print") - success_msg("Great!") - ``` - -You can learn about `test_object()` and `test_function()` in the other articles, so those won't be deatiled here. Let's focus on `test_operator()` instead. This function will extract the first operator group from the solution code (`100 * 1.06 ** 6`), run it in the solution process, and compare the result with the result from running the first operator in the student code in the student process. In total, three steps will be tested: - -- Did the student define enough operations? -- Does the student use the same operators as the solution? -- Is the result of the operation for the student the same as the one in the solution? - -`test_operator()` takes some additional arguments for further customization and tailored feedback messages. For example, you can use it as follows to just check whether the student used the `**` operator in his/her first operation and give custom feedback: - - *** =sct - ```{python} - test_operator(index=1, used=["**"], do_eval=False, - incorrect_op_msg="A little tip: you should use `**` to do this calculation.") - test_object("result") - test_function("print") - success_msg("Great!") - ``` - -This SCT will be more forgiving, but the result is still checked with `test_object()` so the student will still have to calculate the correct value. This time, however, it is not checked by `test_operator()` because `do_eval = False`. `used = ["**"]` is used to tell the system to only check on the `**` operator for the first operation group. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/pythonwhat.wiki/test_try_except.md b/docs/source/pythonwhat.wiki/test_try_except.md deleted file mode 100644 index 1dab8e8d..00000000 --- a/docs/source/pythonwhat.wiki/test_try_except.md +++ /dev/null @@ -1,71 +0,0 @@ -test_try_except ---------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_try_except - :members: -``` - - def test_try_except(index=1, - not_called_msg=None, - body=None, - handlers={}, - except_missing_msg = None, - orelse=None, - orelse_missing_msg=None, - finalbody=None, - finalbody_missing_msg=None, - expand_message=True) - -With `test_try_except`, you can check whether the student correctly coded a `try-except` block. - -As usual, `index` controls which try-except block to check. With `not_called_msg` you can choose a custom message to override the automatically defined message in case not enough try-except blocks weren't found in the student code. `body` is a sub-sct to test the code of the `try` block. `orelse` and `finalbody` work the same way, but here there are also `_msg` arguments to provide custom messages in case these parts ar missing. Finally, there's also `handlers` and `except_missing_msg`. `handlers` should be a dictionary, where the keys are the error classes you expect the student to capture (for the general `except:`, use `'all'`), and the values are sub-SCTs for each of these `except` blocks. An `except` block is only checked for existence and correctness if you mention it inside `handlers`. If it is not available, an automatic message will be generated, but this can ge overriden with `expect_missing_msg`. - - -Note: For more information on sub-SCTs, visit [part checks](../part_checks.rst). - -### Example 1 - -Suppose you want to student to code up the following (completely useless) piece of Python code: - - *** =solution - ```{python} - try: - x = max([1, 2, 'a']) - except TypeError as e: - x = 'typeerror' - except ValueError: - x = 'valueerror' - except (ZeroDivisionError, IOError) as e: - x = e - except : - x = 'someerror' - else : - passed = True - finally: - print('done') - ``` - -To test each and every part of this model solution, you can use the following SCT: - - *** =sct - ```{python} - import collections - handlers = collections.OrderedDict() - handlers['TypeError'] = lambda: test_object_after_expression('x') - handlers['ValueError'] = lambda: test_object_after_expression('x') - handlers['ZeroDivisionError'] = lambda: test_object_after_expression('x', context_vals = ['anerror']) - handlers['IOError'] = lambda: test_object_after_expression('x', context_vals = ['anerror']) - handlers['all'] = lambda: test_object_after_expression('x') - test_try_except(index = 1, - body = lambda: test_function("max"), - handlers = handlers, - orelse = lambda: test_object_after_expression('passed'), - finalbody = lambda: test_function('print')) - ``` - -Notice that: - -- We use the `OrderedDict()` from the `collections` module so that the dictionary we pass in the `handlers` argument is always gone through in the same order. -- We can use `context_vals` to initalize the context value, `e` in this case. - diff --git a/docs/source/pythonwhat.wiki/test_while_loop.md b/docs/source/pythonwhat.wiki/test_while_loop.md deleted file mode 100644 index ea8a90bd..00000000 --- a/docs/source/pythonwhat.wiki/test_while_loop.md +++ /dev/null @@ -1,37 +0,0 @@ -test_while_loop ---------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_while_loop - :members: -``` - - test_while_loop(index=1, - test=None, - body=None, - orelse=None, - expand_message=True) - -Since a lot of the logic of `test_if_else()` and `test_for_loop()` can be applied to `test_while_loop()`, this article is limited to an example. For more info see the wiki on `test_if_else()` and `test_for_loop()`, or the documentation of `test_while_loop()`. - - *** =solution - ```{python} - a = 10 - while a > 5: - print("%s is bigger than 5" % a) - a -= 1 - ``` - - *** =sct - ```{python} - def sct_on_condition_test(): - test_expression_result({"a": 4}) - test_expression_result({"a": 5}) - test_expression_result({"a": 6}) - - test_while_loop(index = 1, - test = sct_on_condition_test, - body = lambda: test_expression_output({"a":4})) - ``` - - diff --git a/docs/source/pythonwhat.wiki/test_with.md b/docs/source/pythonwhat.wiki/test_with.md deleted file mode 100644 index a52df604..00000000 --- a/docs/source/pythonwhat.wiki/test_with.md +++ /dev/null @@ -1,59 +0,0 @@ -test_with ---------- - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_with.test_with -``` - - def test_with(index, - context_vals=False, - context_tests=None, - body=None, - undefined_msg=None, - context_vals_len_msg=None, - context_vals_msg=None, - expand_message=True) - -In Python, one can build so-called context managers with the `with` statement. - -Have a look at an example of such a context manager: - - with open('something.txt') as file1, open('something_else.csv') as file2: # the contexts - # body of the with statement - # do something with file1 and file2 - # ... - -Two important parts can be distinguished: the contexts that are being opened and the body, in which operations are done with these contexts. In this example, two contexts are defined: `open('something.txt')` and `open('something_else.csv')`. The context can be given names (this is optional). In the example, the first one will be called `file1` after the `with` statement, and the second one `file2`. - -`test_with()` is written to allow you to test all these parts of the `with` statement separately. - -### Example 1 - -Suppose you want the student to code something as follows: - - *** =solution - ```{python} - with open('moby_dick.txt') as moby, open('lotr.txt') as lotr: - print("First line of Moby Dick: %r." % moby.readline()) - print("First line of The Lord of The Rings: The Two Towers: %r." % lotr.readline()) - - -In this case you want to test two things: you want the student to open up the correct context and you want them to print out the correct information. Let's assume that how the context are named is not important to you. The solution uses `moby` and `lotr`, but the student can use any name he or she wants. Note these names will not be tested by default, but you can change that by setting `context_vals = True`. - -In the SCT, we specify a sub-SCT for `context_tests` and for `body`. The former tests the contexts, the latter tests the body. As before, you can specify these sub-SCTs through lambda functions or a separate function definition: - - *** =sct - ```{python} - def test_with_body(): - test_function('print', 1) - test_function('print', 2) - - test_with(1, - context_tests = [ - lambda: test_function('open'), - lambda: test_function('open') - ], - body = test_with_body) - ``` - -Different from before, htough, `context_tests` expects a list of lambda functions or customly defined functions. The index in this list of functions represents the context against which the SCTs will be tested. The first lambda/custom function in `context_tests` will be tested against the first context. The second lambda/custom function in `context_tests` will be tested against the second context. If only one function is given in `context_tests`, only the first context will be tested. The `body` argument requires one lambda/custom function to be passed, this contains the sub-SCT that is run against the `with` statements' body. diff --git a/docs/source/quickstart_guide.md b/docs/source/quickstart_guide.md deleted file mode 100644 index 8fdcb08c..00000000 --- a/docs/source/quickstart_guide.md +++ /dev/null @@ -1,66 +0,0 @@ -Quickstart Guide -================ - -Course Setup ------------- - -This guide will cover the basics of creating submission correctness tests (SCTs) for DataCamp exercise. SCTs deal with the running and testing code submissions, in order to give useful feedback. For help on the entire exercise creation process, check out https://www.datacamp.com/teach/documentation. If this is your first time creating a course, see their [Getting Started screencast](https://www.datacamp.com/teach/documentation#tab_getting_started) and [Code Exercises article](https://www.datacamp.com/teach/documentation#tab_code_exercises). - -Your First Exercise -------------------- - -As a basic example, suppose we have an exercise that requires the student to print a variable named `x`. This exercise could look something like this: - - -``````python - *** =pre_exercise_code - ```{python} - x = 5 - ``` - - *** =sample_code - ```{python} - # Print x - - ``` - - *** =solution - ```{python} - # Print x - print(x) - ``` - - *** =sct - ```{python} - Ex().check_object('x').has_equal_value() - Ex().test_output_contains('5') - success_msg('Great job!') - ``` -`````` - -The SCT uses three `pythonwhat` chains to test the correctness of the student's submission. - -1. `check_object` is used to test whether `x` was defined in the submission. In addition, the `has_equal_value()` statement tests whether the value of `x` is equal between the submission and solution. -2. `test_output_contains()` tests whether the student printed out `x` correctly. The function looks at the output the student generated with his or her submission, and then checks whether the string '5' is found in this output. -3. `success_msg()` is used to give positive feedback when all `pythonwhat` tests passed. If you do not use `success_msg()`, `pythonwhat` will generate a kind message itself :). - -In all the test statements above, feedback messages will be automatically generated when something goes wrong. However, it is possible to manually set these feedback messages. For example, in the code below, - -```python -Ex().check_object(undefined_msg="`x` is undefined!") \ - .has_equal_value(incorrect_msg="wrong value for `x`") -``` - -the automatic messages for when `x` is undefined or incorrect are replaced with manual feedback. Now, if students submit `x = 4` instead of `x = 5`, they will see the message, "wrong value for `x`". Finally, notice that you can use Markdown syntax inside the strings here. - -The same holds for `test_output_contains()`: you can use the `no_output_msg` argument to specify a custom message. For more information on all the different arguments you can set in the different `pythonwhat` functions, have a look at the articles in this wiki, describing them in detail. - -Next Steps ----------- - -Test functions in pythonwhat are broken into 4 groups: - -* [Simple tests](simple_tests/index.rst): look at, e.g., the output produced by an entire code submission. -* [Part checks](part_checks.rst): focus on specific pieces of code, like a particular for loop. -* [Expression tests](expression_tests.md): combined with part checks, these run pieces of code and evaluate the outcome. -* [Logic tests](logic_tests/index.rst): these allow logic like an or statement to be used with SCTs. diff --git a/docs/source/simple_tests/has_equal_ast.md b/docs/source/simple_tests/has_equal_ast.md deleted file mode 100644 index 25cfda6b..00000000 --- a/docs/source/simple_tests/has_equal_ast.md +++ /dev/null @@ -1,39 +0,0 @@ -has_equal_ast --------------- - -```eval_rst -.. automodule:: pythonwhat.check_funcs.has_equal_ast - :members: -``` - -An abstract syntax tree (AST) is a way of representing the high-level structure of python code. - -### Example: quotes - -Whether you use the concrete syntax `x = "1"` or `x = '1'`, the abstract syntax is the same: x is being assigned to the string "1". - -### Example: parenthesis - -Grouping by parentheses produces the same AST, when the same statement would work the same without them. -For example, `(True or False) and True`, and `True or False and True`, are the same due to operator precedence. - -### Example: spacing - -The same holds for different types of spacing that essentially specify the same statement: `x = 1` or `x = 1`. - -### Caveat: evaluating - -What the AST doesn't represent is values that are found through evaluation. For example, the first item in the list in - -```python -x = 1 -[x, 2, 3] -``` - -and - -```python -[1, 2, 3] -``` - -Is not the same. In the first case, the AST represents that a variable `x` needs to be evaluated in order to find out what its value is. In the second case, it just represents the value `1`. diff --git a/docs/source/simple_tests/index.rst b/docs/source/simple_tests/index.rst deleted file mode 100644 index cf037998..00000000 --- a/docs/source/simple_tests/index.rst +++ /dev/null @@ -1,17 +0,0 @@ -Simple Tests -============ - -Simple tests are the most basic tests available in pythonwhat. -They usually don't focus on specific pieces of a submission (:doc:`like part checks `), or re-run any code (:doc:`like expression tests ). -Instead, they simply look at things like imports, printed output, or raw code text. -A final, common use is to test the value of a variable in the final environment (that is, after the submission of solution code have been run). - -.. toctree:: - :maxdepth: 2 - - test_import - test_object - test_output_contains - test_student_typed - has_equal_ast - test_mc diff --git a/docs/source/simple_tests/test_import.md b/docs/source/simple_tests/test_import.md deleted file mode 100644 index 9b199cff..00000000 --- a/docs/source/simple_tests/test_import.md +++ /dev/null @@ -1,52 +0,0 @@ -test_import ------------ - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_import - :members: -``` - - def test_import(name, - same_as=True, - not_imported_msg=None, - incorrect_as_msg=None): - -With `test_import` you can test whether a student correctly imported a certain package. As an option, you can also specify whether or not the same alias should be used. - -Python features many ways to import packages. All of these different methods revolve around the `import`, `from` and `as` keywords. Suppose you want students to import `matplotlib.pyplot` as `plt` (the common way of importing the plotting tools in `matplotlib`. A possible solution of your exercises could be the following: - - *** =solution - ```{python} - # Import plotting tools - import matplotlib.pyplot as plt - ``` - -Below is a possible SCT for this exercise: - - *** =sct - ```{python} - test_import("matplotlib.pyplot") - success_msg("You nailed it!") - ``` - -Here, `test_import` will parse both the student's submission as well as the solution, and figure out which packages were imported and how. Next, it checks if the `matplotlib.pyplot` package was imported and under which alias. If the student did this and imported it as `plt`, all is good. If, however, the student submitted `import matplotlib` (import entire package instead of module) or `import matplotlib.pyplot as pppplot` (incorrect alias), `test_import()` will fail and generate the appropriate messages. - -As usual, you can override these messages with your own: - - *** =sct - ```{python} - test_import("matplotlib.pyplot"), - not_imported_msg = "You can import pyplot by using `import matplotlib.pyplot`.", - incorrect_as_msg = "You should set the correct alias for `matplotlib.pyplot`, import it `as plt`.") - success_msg("You nailed it!") - ``` - -With `same_as`, you can control whether or not the alias should be exactly the same. By default `same_as=True`, so the alias (`plt` in the example) should also be used by student. If you set it to `False`: - - *** =sct - ```{python} - test_import("matplotlib.pyplot", same_as=False) - success_msg("You nailed it!") - ``` - -The SCT will also pass if the student uses `import matplotlib.pyplot as pppplot`, a submission that wouldn't be accepted if `same_as=True`. diff --git a/docs/source/simple_tests/test_mc.md b/docs/source/simple_tests/test_mc.md deleted file mode 100644 index 7d9f8154..00000000 --- a/docs/source/simple_tests/test_mc.md +++ /dev/null @@ -1,34 +0,0 @@ -test_mc -------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_mc - :members: -``` - - test_mc(correct, msgs) - -Multiple choice exercises are straightforward to test. Use `test_mc()` to provide tailored feedback for both the incorrect options, as the correct option. Below is the code for a multiple choice exercise example, with an SCT that uses `test_mc`: - - --- type:MultipleChoiceExercise lang:python xp:50 skills:2 - ## The author of Python - - Who is the author of the Python programming language? - - *** =instructions - - Roy Co - - Ronald McDonald - - Guido van Rossum - - *** =hint - Just google it! - - *** =sct - ```{python} - test_mc(correct = 3, - msgs = ["That's someone who makes soups.", - "That's a clown who likes burgers.", - "Correct! Head over to the next exercise!"]) - ``` - -The first argument of `test_mc()`, `correct`, should be the number of the correct answer in this list. Here, the correct answer is Guido van Rossum, corresponding to 3. The `msgs` argument should be a list of strings with a length equal to the number of options. We encourage you to provide feedback messages that are informative and tailored to the (incorrect) option that people selected. Make sure to correctly order the feedback message such that it corresponds to the possible answers that are listed in the instructions tab. Notice that there's no need for `success_msg()` in multiple choice exercises, as you have to specify the success message inside `test_mc()`, along with the feedback for incorrect options. diff --git a/docs/source/simple_tests/test_object.md b/docs/source/simple_tests/test_object.md deleted file mode 100644 index 4e0120fd..00000000 --- a/docs/source/simple_tests/test_object.md +++ /dev/null @@ -1,99 +0,0 @@ -test_object ------------ - -```eval_rst -.. autofunction:: pythonwhat.test_funcs.test_object.test_object -``` - - test_object(name, - eq_condition="equal", - do_eval=True, - undefined_msg=None, - incorrect_msg=None) - -`test_object()` enables you to test whether a student correctly defined an object. - -As explained on the [docs home](/Home.md), both the student's submission as well as the solution code are executed, in separate processes. `test_object()` looks at these processes and checks if the object specified in `name` is available in the student process. Next, it checks whether the object in the student and solution process correspond. In case of a failure along the way, `test_object()` will generate a meaningful feedback message that you can override. - -### Example 1 - -Suppose we have the following solution: - - *** =solution - ```{python} - # Create a variable x, equal to 3 * 5 - x = 3 * 15 - ``` - -To test this we simply use: - - *** =sct - ```{python} - test_object("x") - success_msg("Great job!") - ``` - -This SCT will test if the variable `x` is defined, and has the same ending value in the student process as in the solution process. All of the following student submissions would be accepted by `test_object()`: - -- `x = 15` -- `x = 12 + 3` -- `x = 3; x += 12` - -How the object `x` came about in the student's submission, does not matter: only the end result, the actual content of `x`, matters. - -`do_eval=True` by default; if you set it to `False`, only the existence of an object `x` will be checked; its contents will not be compared to the object `x` that's in the solution process. - -### Object equality - -When comparing more complex objects in Python, chances are they don't use the equality operation you desire. Python objects are compared using the `==` operator, and objects can overwrite its implementation to fit the object's needs. Internally, `test_object()` uses the `==` operation to compare objects, this means you could encounter undesirable behaviour. Sometimes `==` just compares the actual object instances, and objects which are semantically alike wont be according to `test_object()`. - -Say, for example, that you have the following solution: - - *** =solution - from urllib.request import urlretrieve - fn1 = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_998/datasets/Chinook.sqlite' - urlretrieve(fn1, 'Chinook.sqlite') - - # Import packages - from sqlalchemy import create_engine - import pandas as pd - - # Create engine: engine - engine = create_engine('sqlite:///Chinook.sqlite') - - # Open engine connection - con = engine.connect() - -An SCT for this exercise could be the following: - - *** =sct - test_object("engine") - test_object("con") - -Now, if the student enters the exact same code as the solution, the SCT will still fail. How? Well if you try this out: `create_engine('sqlite:///Chinook.sqlite') == create_engine('sqlite:///Chinook.sqlite')` you will notice that it returns `False`. This means the exact same execution doesn't lead to the the exact same object (although they might be semantically equal). We can't use `test_object` like that here. There are several ways to solve this: - -#### Workaround - - *** =sct - test_object("engine", do_eval=False) - test_function("create_engine") - test_object("con", do_eval=False) - test_function("engine.connect") - -This will check whether the objects `engine` and `con` are declared, without checking for it's value. With `test_function()` we check whether they used the correct functions. This will not test the exact same thing as the first SCT, but it's effective 99% of the time. - -#### Equality operations hardcoded in `pythonwhat` - -A side note here, complex objects that are used a lot have an custom implementation of equality built for them in `pythonwhat`. These objects can be tested with a regular `test_object(...)`, without having to use `do_eval=False`. At the moment, the more complex classes that can be tested are: - -- `numpy.ndarray` -- `pandas.DataFrame` -- `pandas.Series` - -Of course primitive classes like `str`, `int`, `list`, `dict`, ... can be tested without any problems too, as well as objects of which the class has a semantically correct implementation of the `==` operator. - -#### Manually define a converter - -As explained in the [Processes article](../expression_tests.md), objects are extracted from their respected processes by 'dilling' and 'undilling' them. However, you can manually set a 'converter' with the `set_converter()` function. This will override the default dilling and undilling behavior, and enables you to make simplified representations of custom objects, testing only exactly what you want to test. - -**NOTE**: Behind the scenes, `pythonwhat` has to fetch the value of objects from sub-processes. The required 'dilling' and 'undilling' can cause issues for exotic objects. For more information on this and possible errors that can occur, read the [Processes article](../expression_tests.md). diff --git a/docs/source/simple_tests/test_output_contains.md b/docs/source/simple_tests/test_output_contains.md deleted file mode 100644 index 75bc8ae4..00000000 --- a/docs/source/simple_tests/test_output_contains.md +++ /dev/null @@ -1,40 +0,0 @@ -test_output_contains --------------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_output_contains - :members: -``` - - test_output_contains(text, - pattern=True, - no_output_msg=None) - -We can test the output of the student contains with `test_output_contains()`. This function will compare the given text with the text in the student's output and see if we have a match. You can use regular expressions or not, that's completely up to you. - -Here's an example of an exercise with `test_student_typed()`, suppose the solution looks like this: - - *** =solution - ```{python} - # Print the "This is some ... stuff" to the shell - print("This is some weird stuff") - ``` - -The following SCT tests whether the student outputs `This is some weird stuff`: - - *** =sct - ```{python} - test_output_contains("This is some weird stuff", pattern = False) - success_msg("Great job!") - ``` - -Notice that we set `pattern` to `False`, this will cause `test_output_contains()` to search for the pure string, no patterns are used. This SCT is not robust, because it won't be accepted if the student submits `print("This is some cool stuff")`, for example. Therefore, it's a good idea to use [regular expressions](https://docs.python.org/3.5/library/re.html). `pattern=True` by default, so there's no need to specify this: - - *** =sct - ```{python} - test_output_contains(/This is some \w* stuff/, - no_output_msg = "Print out `This is some ... stuff` to the output, fill in `...` with a word you like.") - success_msg("Great job!") - ``` - -Now, different printouts will be accepted. Notice that we also specified `no_output_msg` here. If the pattern is not found in the output generated, this message will be shown instead of a message that's automatically generated by `pythonwhat`. diff --git a/docs/source/simple_tests/test_student_typed.md b/docs/source/simple_tests/test_student_typed.md deleted file mode 100644 index 9b5f73c0..00000000 --- a/docs/source/simple_tests/test_student_typed.md +++ /dev/null @@ -1,44 +0,0 @@ -test_student_typed ------------------- - -```eval_rst -.. automodule:: pythonwhat.test_funcs.test_student_typed - :members: -``` - - test_student_typed(text, - pattern=True, - not_typed_msg=None) - -`test_student_typed()` will look through the student's submission to find a match with the string specified in `text`. With `pattern`, you can declare whether or not to use regular expressions. - -Suppose the solution of an exercise looks like this: - - *** =solution - ```{python} - # Calculate the sum of all single digit numbers and assign the result to 's' - s = sum(range(10)) - - # Print the result to the shell - print(s) - ``` - -The following SCT tests whether the student typed `"sum(range("`: - - *** =sct - ```{python} - test_student_typed("sum(range(", pattern = False) - success_msg("Great job!") - ``` - -Notice that we set `pattern` to `False`, this will cause `test_student_typed()` to search for the pure string, no patterns are used. This SCT is not that robust though, it won't accept something like `sum( range(10) )`. This is why we should almost always use [regular expressions](https://docs.python.org/3.5/library/re.html) in `test_student_typed`. For example: - - *** =sct - ```{python} - test_student_typed("sum\s*\(\s*range\s*\(", not_typed_msg="You didn't use `range()` inside `sum()`.") - success_msg("Great job!") - ``` - -We also used `not_typed_msg` here, which will control the feedback given to the student when `test_student_typed()` doesn't pass. Note that also `success_msg()` is used here, this is the message that is shown when the SCT has passed. - -In general, **you should avoid using `test_student_typed()`**, as it imposes severe restrictions on how a student can solve an exercise. Often, there are different ways to solve an exercise. Unless you have a very advanced regular expression, `test_student_typed()` will not be able to accept all these different approaches. For the example above, `test_function()` would be more appropriate. diff --git a/docs/source/spec2_summary.rst b/docs/source/spec2_summary.rst deleted file mode 100644 index 63b8cc3f..00000000 --- a/docs/source/spec2_summary.rst +++ /dev/null @@ -1,226 +0,0 @@ -Spec2 Changes -================== - -.. role:: python(code) - :language: python - - -Lambdaless test functions ---------------------------- - -Sometimes you want to pass a test function as an argument to another test function. For the examples below, we'll use the following solution code - -.. code:: python - - # solution code - [1 if True else 0 for i in range(10)] - -Removing Lambdas -~~~~~~~~~~~~~~~~ - -Using an SCT that works in pythonwhat v2, - -.. code:: python - - # pythonwhat v2 SCT - test_list_comp( - body=test_if_exp( - body=test_student_typed('1')) # line 4 - -would do the following (in order)... - -1. run the test_student_typed function on line 4, which runs over the whole code, rather than just the body of the inline if. -2. pass its return value (None) to the body argument of ``test_if_exp``. -3. run ``test_if_exp``, whose body argument is None, rather than a sub-test. -4. run test_list_comp, whose body argument is None, rather than a sub-test. - -Instead, in pythonwhat v1, testing the inline if expression (`1 if True else 0`) requires an SCT peppered with lambdas, such as - -.. code:: python - - # v1 SCT - test_list_comp( - body=lambda: test_if_exp( - body = lambda: test_student_typed('1')) - -Removing temporary functions for multiple sub-tests -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In pythonwhat v2, multiple sub-tests may be run by putting them in a list, such as - -.. code:: python - - # v2 SCT - # list of sub-tests - if_body_tests = [test_student_typed('1'), test_expression_result(context_vals=[1])] - # main test - test_list_comp( - body = test_if_exp( - body = if_body_tests) # list could also be put here directly - -which, in pythonwhat v1 would try to run the tests in ``if_body_tests`` first, rather than as sub-tests. -In order to accomplish this in pythonwhat v1, temporary functions were necessary, such as, - -.. code:: python - - # v1 SCT - # temporary function for testing inline if expression - def inner_test(): - test_student_typed('1') - test_expression_result(context_vals=[1]) - # main test - test_list_comp( - body=lambda: test_if_exp(body = inner_test)) - -while not too different, this approach can spiral out of control for complex SCTs (temporary functions within temporary functions, etc..). - - - -How it works -^^^^^^^^^^^^ -+---------+--------------------------------------+-------------------------------------+ -| spec | SCT | effect | -+=========+======================================+=====================================+ -| v1 test | :python:`test_list_comp()` | runs test | -+---------+--------------------------------------+-------------------------------------+ -| v1 test | :python:`lambda: test_list_comp()` | waits to run | -+---------+--------------------------------------+-------------------------------------+ -| v2 check| :python:`check_list_comp()` | waits to run | -+---------+--------------------------------------+-------------------------------------+ -| v2 check| :python:`Ex().check_list_comp()` | runs test | -+---------+--------------------------------------+-------------------------------------+ -| v2 test | :python:`F().test_list_comp()` | waits to run | -+---------+--------------------------------------+-------------------------------------+ -| v2 test | :python:`test_list_comp()` | runs test if not argument to another| -+---------+--------------------------------------+-------------------------------------+ - -The critical message is in pythonwhat - -* **v1**: you have to do something special (use a lambda) to **opt-out** of running a test immediately. -* **v2**: you have to do something special (use ``Ex()``) to **opt-in** to running a test immediately. - -pythonwhat v2 is Backwards Compatibile -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -for all test\_ functions, pythonwhat v2's behavior is completely backwards compatible (and in fact was put in pythonwhat v1 several weeks before releasing v2). If you want to be explicit about any test function not being run, you can use the function chain object ``F``, for example - -.. code:: - - # Implicit - sub_test = test_if_exp(ETC...) # waits to run only if passed to another SCT - test_list_comp(body=sub_test) # comment out this line, and sub_test will run (as in pythonwhat v1) - - # Explicit - sub_test = F().test_if_exp(ETC...) # always waits to run - Ex().test_list_comp(body=sub_test) - -Never mix Explicit and Implicit approaches -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you choose to use the explicit approach (``Ex()`` and ``F()``), **don't expect the implicit approach to work**. -That is, if you want ``test_if_exp`` below to run immediately, do not write - -.. code:: - - test_if_exp(1) # implicit, should use Ex() or F() - Ex().check_list_comp(1) # explicit - -and expect the SCTs to run in a predictable order. - -If you want to create a bunch of sub-tests, but don't want to preface each with F(), you can use the pythonwhat v2 function multi, as below. - -.. code:: - - subtest = multi(test_if_exp(ETC...), test_list_comp(ETC...)) - - -Context values for nested parts -------------------------------- - -Context values may now be defined for nested parts. For example, the print statement below, - -.. code:: - - for i in range(2): # outer for loop part - for j in range(3): # inner for loop part - print(i + j) - -may be tested by setting context values at each level, - -.. code:: - - (Ex() - .check_for_loop(0).check_body().set_context(i = 1) # outer for - .check_for_loop(0).check_body().set_context(j = 2) # inner for - .has_equal_output() - ) - -For more on context valus see [PROCESSES LINK HERE]. - -Can call code chunks that before could only be split up -------------------------------------------------------- - -Entire code pieces, such as the inline if statement below, - -.. code:: - - 'yes' if True else 'no' - -may be tested using something like, - -.. code:: - - Ex().check_if_exp(0).has_equal_value() - -Argument checking ------------------ - -The arguments of a function definition, such as - -.. code:: - - def f(a=1): print(a) - -are now parts and may be checked as below.. - -.. code:: - - (Ex().check_function_def('f') # does f exist? - .check_args('a') # does a exist? - .is_default() # is it a default argument? - .has_equal_value() # is it's default equal to solution? - ) - -For more on the argument part, see [PART CHEATSHEET LINK HERE]. - - -Deprecate test_expression_result and friends --------------------------------------------- - -In pythonwhat v1, the functions - -* test_expression_result -* test_expression_output -* test_object_after_expression - -and various arguments of test_function_definition, test_lambda_function ran code and then -checked the result, printed output, or errors against eachother. - -These functions have been deprecated in favor of similar function.. - -* `has_equal_value` -* `has_equal_output` -* `has_equal_error` - -These functions include identical arguments as the above. - -Feedback messages may use templating (via str.format or Jinja2) ------------------------------------------------------------------ - -**This feature is not stable, and should not be used in production** - - -Cleaned up internals --------------------- - -Yayyyy. diff --git a/docs/tests.rst b/docs/tests.rst new file mode 100644 index 00000000..c6d9c5d6 --- /dev/null +++ b/docs/tests.rst @@ -0,0 +1,79 @@ +Tests +===== + +.. note:: + + The examples are numbered and linkable, + but numbers (and links) can change between builds of the documentation. + +.. jinja:: test_ctx + + {% for file, tests in test_data.items() %} + + {{ file }} + {{ "-" * 100 }} + + {% for test in tests %} + + Example {{loop.index}} + ~~~~~~~~~~~~~~~~~~~~~~ + {% if test.pre_exercise_code %} + PEC :: + + {{ test.pre_exercise_code | indent(4) }} + + {% else %} + No PEC + {% endif %} + {% if test.solution_code %} + Solution code :: + + {{ test.solution_code | indent(4) }} + + {% else %} + No solution code + {% endif %} + {% if test.student_code %} + Student code :: + + {{ test.student_code | indent(4) }} + + {% else %} + No student code + {% endif %} + {% if test.raw_student_output %} + Student output :: + + {{ test.raw_student_output | indent(4) }} + + {% else %} + No output + {% endif %} + {% if test.sct %} + SCT :: + + {{ test.sct | indent(4) }} + + {% else %} + No SCT + {% endif %} + {% if test.result %} + Result :: + + {{ test.result.message | indent(4) }} + + {% else %} + No result + {% endif %} + {% if test.error %} + Error :: + + {{ test.error | indent(4) }} + + {% else %} + No error + {% endif %} + + {% endfor %} + + {% endfor %} diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 00000000..724b2ae5 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,6 @@ +[pytest] +testpaths = tests/ +addopts =-m "not compiled" +markers = + slow + compiled diff --git a/pythonwhat/Feedback.py b/pythonwhat/Feedback.py deleted file mode 100644 index 54103ba9..00000000 --- a/pythonwhat/Feedback.py +++ /dev/null @@ -1,112 +0,0 @@ -import re -from pythonwhat import utils -from pythonwhat import utils_ast -import _ast - -class Feedback(object): - - def __init__(self, message, astobj = None): - self.message = message - self.line_info = {} - try: - if astobj is not None: - if issubclass(type(astobj), (_ast.Module, _ast.Expression)): - astobj = astobj.body - if isinstance(astobj, list) and len(astobj) > 0: - start = astobj[0] - end = astobj[-1] - else: - start = astobj - end = astobj - if hasattr(start, "lineno") and \ - hasattr(start, "col_offset") and \ - hasattr(end, "end_lineno") and \ - hasattr(end, "end_col_offset"): - self.line_info["line_start"] = start.lineno - self.line_info["column_start"] = start.col_offset - self.line_info["line_end"] = end.end_lineno - self.line_info["column_end"] = end.end_col_offset - except: - pass - -# TODO FILIP: No used for now, come back to this later. -class FeedbackMessage(object): - """Generate feedback. - - Don't use this yet! - - This class will hold all functionality which is related to feedback messaging. - At the moment it is NOT used, feedback generation is still HIGLY interwoven with - test_... files. Should be decoupled. - - Class should be refactored to use .format() instead. - - Will be documented when it's refactored. - """ - def __init__(self, message_string): - self.set(message_string) - self.information = {} - - def add_information(self, key, value): - if (not(key in self.information)): - self.set_information(key, value) - - def set_information(self, key, value): - self.information[key] = utils.shorten_str(str(value)) - - def remove_information(self, key): - if (key in self.information): - self.information.pop(key) - - def set(self, message_string): - self.message_string = str(message_string) - - def append(self, message_string): - self.message_string += str(message_string) - - def cond_append(self, cond, message_string): - self.message_string += "${{" + \ - str(cond) + " ? " + str(message_string) + "}}" - - def generateString(self): - generated_string = FeedbackMessage.replaceRegularTags( - self.message_string, self.information) - generated_string = FeedbackMessage.replaceConditionalTags( - generated_string, self.information) - return(generated_string) - - def replaceRegularTags(message_string, information): - generated_string = message_string - - pattern = "\${([a-zA-Z]*?)}" - - keywords = re.findall(pattern, generated_string) - for keyword in keywords: - replace = "\${" + keyword + "}" - if (keyword in information): - generated_string = re.sub( - replace, information[keyword], generated_string) - else: - generated_string = re.sub(replace, "", generated_string) - - return(generated_string) - - def replaceConditionalTags(message_string, information): - generated_string = message_string.replace("\n", "\\\\n") - pattern = "\${{([a-zA-Z]*?) \? (.*?)}}" - - cond_keywords = re.findall(pattern, generated_string) - for (keyword, k_string) in cond_keywords: - replace = "\${{" + keyword + " \? " + re.escape(k_string) + "}}" - if (keyword in information): - generated_string = re.sub( - replace, - " " + - FeedbackMessage.replaceRegularTags( - k_string, - information), - generated_string) - else: - generated_string = re.sub(replace, "", generated_string) - - return(generated_string) diff --git a/pythonwhat/Reporter.py b/pythonwhat/Reporter.py deleted file mode 100644 index 9c075200..00000000 --- a/pythonwhat/Reporter.py +++ /dev/null @@ -1,153 +0,0 @@ -from pythonwhat.Feedback import Feedback -import re -import markdown2 -from pythonwhat.Test import TestFail, Test - -""" -This file holds the reporter class. -""" - -class Reporter(object): - """Do reporting. - - This class holds the feedback- or success message and tracks whether there are failed tests - or not. All tests are executed trough do_test() in the Reporter. - """ - active_reporter = None - - def __init__(self): - self.failed_test = False - self.feedback = Feedback("Oh no, your solution is incorrect! Please, try again.") - self.success_msg = "Great work!" - self.errors_allowed = False - self.tags = {} - self.failure_msg = "" - self.fallback_ast = None - self.test_stack = [] - self.test_mode = None - - def set_success_msg(self, success_msg): - self.success_msg = success_msg - - def allow_errors(self): - self.errors_allowed = True - - def reject_errors(self): - self.errors_allowed = False - - def fail(self, failure_msg): - self.failed_test = True - self.feedback = Feedback(failure_msg) - - def do_test(self, testobj, prepend_on_fail="", fallback_ast=None): - """Do test. - - Execute a given test, unless some previous test has failed. If the test has failed, - the state of the reporter changes and the feedback is kept. - """ - - if self.test_mode is 'or': - return self.test_stack.append([testobj, prepend_on_fail, fallback_ast]) - - if prepend_on_fail: self.failure_msg = prepend_on_fail - if fallback_ast: self.fallback_ast = fallback_ast - - if self.failed_test: - self.feedback.message = self.failure_msg + self.feedback.message - raise TestFail - return - if isinstance(testobj, Test): - testobj.test() - result = testobj.result - if (not result): - self.failed_test = True - self.feedback = testobj.get_feedback() - self.feedback.message = self.failure_msg + self.feedback.message - if not self.feedback.line_info and self.fallback_ast: - self.feedback = Feedback(self.feedback.message, self.fallback_ast) - raise TestFail - - else: - result = None - testobj() # run function for side effects - - #self.failure_msg_stack.pop() - return result - - def do_tests(self, testobjs): - """Do multiple tests. - - Execute an array of tests. - """ - for testobj in testobjs: - if self.failed_test: - break - - self.do_test(testobj) - - def start_or_test(self): - self.test_mode = 'or' - self.test_stack = [] - - def end_or_test(self): - self.test_mode = None - first_message = None - success = False - for sct_args in self.test_stack: - try: - self.do_test(*sct_args) - success = True - except TestFail as e: - if not first_message: first_message = self.feedback.message - self.failed_test = False - - if success: return - - self.failed_test = True - self.feedback.message = first_message - raise TestFail - - def set_tag(self, key, value): - self.tags[key] = value - - - def build_payload(self, error): - if (error and not self.failed_test and not self.errors_allowed): - feedback_msg = "Your code contains an error: `%s`" % str(error[1]) - return({ - "correct": False, - "message": Reporter.to_html(feedback_msg), - "tags": {"fun": "runtime_error"}}) - - if self.failed_test: - if not self.feedback.line_info: - return({ - "correct": False, - "message": Reporter.to_html(self.feedback.message), - "tags": self.tags}) - else: - # Hack to make it work with campus app implementation - if self.feedback.line_info["column_start"] is None: - col_start = None - else: - col_start = self.feedback.line_info["column_start"] + 1 - - return({ - "correct": False, - "message": Reporter.to_html(self.feedback.message), - "line_start": self.feedback.line_info["line_start"], - "column_start": col_start, - "line_end": self.feedback.line_info["line_end"], - "column_end": self.feedback.line_info["column_end"], - "tags": self.tags}) - - - else: - return({ - "correct": True, - "message": Reporter.to_html(self.success_msg) - }) - - @staticmethod - def to_html(msg): - return(re.sub("

(.*)

", "\\1", markdown2.markdown(msg)).strip()) diff --git a/pythonwhat/State.py b/pythonwhat/State.py index 037d0605..86a0f426 100644 --- a/pythonwhat/State.py +++ b/pythonwhat/State.py @@ -1,17 +1,24 @@ -import ast -import inspect -import string -from copy import copy -from functools import partial -from pythonwhat.parsing import TargetVars, FunctionParser, ObjectAccessParser, parser_dict -from pythonwhat.Reporter import Reporter -from pythonwhat.Feedback import Feedback -from pythonwhat import utils_ast +import asttokens + +from functools import partialmethod +from collections.abc import Mapping + +from protowhat.failure import debugger +from protowhat.Feedback import FeedbackComponent +from protowhat.selectors import DispatcherInterface +from protowhat.State import State as ProtoState +from protowhat.utils import parameters_attr from pythonwhat import signatures from pythonwhat.converters import get_manual_converters -from collections.abc import Mapping -from itertools import chain -from jinja2 import Template +from pythonwhat.feedback import Feedback +from pythonwhat.parsing import ( + TargetVars, + FunctionParser, + ObjectAccessParser, + parser_dict, +) +from pythonwhat.utils_ast import wrap_in_module + class Context(Mapping): def __init__(self, context=None, prev=None): @@ -21,8 +28,7 @@ def __init__(self, context=None, prev=None): self._items = {**self.prev, **self.context.defined_items()} def update_ctx(self, new_ctx): - upd_prev = {**self.prev, **self.context.defined_items()} - return self.__class__(new_ctx, upd_prev) + return self.__class__(new_ctx, self._items) def __getitem__(self, x): return self._items[x] @@ -34,292 +40,299 @@ def __len__(self): return len(self._items) -#class MsgFormatter(string.Formatter): -# def vformat(self, format_string, args, kwargs): -# """Restricted vformat, which does not format entries with converters or format specs""" -# used_args = set() -# result = [] -# for chunk in string._string.formatter_parser(format_string): -# orig = self._orig_from_chunk(*chunk) -# # return original string if there are converters or format specs, -# # otherwise, parse as normal -# if chunk[1] and any(chunk[2:]): -# result.append(orig) -# elif chunk[0] and not any(chunk[1:]): -# result.append(chunk[0]) -# else: -# res, _ = self._vformat(orig, args, kwargs, used_args, 1) -# result.append(res) -# return "".join(result) -# -# def get_field(self, field_name, args, kwargs): -# try: -# return super().get_field(field_name, args, kwargs) -# except (KeyError, AttributeError): -# return "{"+field_name+"}", "NA" -# -# @staticmethod -# def _orig_from_chunk(literal_text, field_name, format_spec, conversion): -# # of form, literal_str {var_name!conversion:format_spec} -# conversion = '!' + conversion if conversion else "" -# format_spec = ":" + format_spec if format_spec else "" -# return "%s{%s%s%s}"%(literal_text, field_name, conversion, format_spec) - -class State(object): +@parameters_attr +class State(ProtoState): """State of the SCT environment. This class holds all information relevevant to test the correctness of an exercise. It is coded suboptimally and it will be refactored soon, and documented thouroughly after that. - """ - def __init__(self, - student_context=None, solution_context=None, - student_parts=None, solution_parts=None, - highlight = None, messages=None, - **kwargs): - - # Set basic fields from kwargs - self.__dict__.update(kwargs) - - self.student_parts = student_parts - self.solution_parts = solution_parts - self.messages = messages if messages else [] - - # parse code if didn't happen yet - if not hasattr(self, 'student_tree'): - self.student_tree = State.parse_ext(self.student_code) + kwargs: + ... + - reporter - if not hasattr(self, 'solution_tree'): - self.solution_tree = State.parse_int(self.solution_code) - - if not hasattr(self, 'pre_exercise_tree'): - self.pre_exercise_tree = State.parse_int(self.pre_exercise_code) - - if not hasattr(self, 'parent_state'): - self.parent_state = None - - self.student_context = Context(student_context) if student_context is None else student_context - self.solution_context = Context(solution_context) if solution_context is None else solution_context - - self.highlight = self.student_tree if (not highlight) and self.parent_state else highlight + """ - self.converters = get_manual_converters() # accessed only from root state + feedback_cls = Feedback + + def __init__( + self, + student_code, + solution_code, + pre_exercise_code, + student_process, + solution_process, + raw_student_output, + # solution output + reporter, + force_diagnose=False, + highlight=None, + highlight_offset=None, + highlighting_disabled=None, + feedback_context=None, + creator=None, + student_ast=None, + solution_ast=None, + student_ast_tokens=None, + solution_ast_tokens=None, + student_parts=None, + solution_parts=None, + student_context=Context(), + solution_context=Context(), + student_env=Context(), + solution_env=Context(), + ): + args = locals().copy() + self.debug = False + + for k, v in args.items(): + if k != "self": + setattr(self, k, v) + + self.ast_dispatcher = self.get_dispatcher() + + # Parse solution and student code + # if possible, not done yet and wanted (ast arguments not False) + if isinstance(self.student_code, str) and student_ast is None: + self.student_ast = self.parse(student_code) + if isinstance(self.solution_code, str) and solution_ast is None: + with debugger(self): + self.solution_ast = self.parse(solution_code) + + if highlight is None: # todo: check parent_state? (move check to reporting?) + self.highlight = self.student_ast + + self.converters = get_manual_converters() # accessed only from root state - self.fun_usage = {} self.manual_sigs = None - self._parser_cache = {} - - def set_used(self, name, stud_index, sol_index): - if name in self.fun_usage.keys(): - self.fun_usage[name][sol_index] = stud_index - else: - self.fun_usage[name] = {sol_index: stud_index} - - def get_options(self, name, stud_indices, sol_index): - if name in self.fun_usage.keys(): - if sol_index in self.fun_usage[name].keys(): - # sol_index has already been used - return [self.fun_usage[name][sol_index]] - else: - # sol_index hasn't been used yet - # exclude stud_index that have been hit elsewhere - used = set(list(self.fun_usage[name].values())) - return list(set(stud_indices) - used) - else: - return stud_indices def get_manual_sigs(self): if self.manual_sigs is None: self.manual_sigs = signatures.get_manual_sigs() - return(self.manual_sigs) - - def build_message(self, tail="", fmt_kwargs=None): - if not fmt_kwargs: fmt_kwargs = {} - out_list = [] - # add trailing message to msg list - msgs = self.messages[:] + [{'msg': tail or "", 'kwargs':fmt_kwargs}] - # format messages in list, by iterating over previous, current, and next message - for prev_d, d, next_d in zip([{}, *msgs[:-1]], msgs, [*msgs[1:], {}]): - tmp_kwargs = {'parent': prev_d.get('kwargs'), - 'child': next_d.get('kwargs'), - 'this': d['kwargs'], - **d['kwargs']} - if d['msg'].startswith('FMT:'): - out = d['msg'].replace('FMT:', "").format(**tmp_kwargs) - elif d['msg'].startswith('__JINJA__:'): - out = Template(d['msg'].replace('__JINJA__:', "")).render(**tmp_kwargs) - else: - out = d['msg'] - - out_list.append(out) - - return "".join(out_list) - - def to_child_state(self, student_subtree, solution_subtree, - student_context=None, solution_context=None, - student_parts=None, solution_parts=None, - highlight = None, - append_message="", node_name=""): + return self.manual_sigs + + def to_child(self, append_message=None, node_name="", **kwargs): """Dive into nested tree. Set the current state as a state with a subtree of this syntax tree as student tree and solution tree. This is necessary when testing if statements or for loops for example. """ + bad_parameters = set(kwargs) - set(self.parameters) + if bad_parameters: + raise ValueError( + "Invalid init parameters for State: %s" % ", ".join(bad_parameters) + ) + + base_kwargs = { + attr: getattr(self, attr) + for attr in self.parameters + if hasattr(self, attr) and attr not in ["ast_dispatcher", "highlight"] + } + + if append_message and not isinstance(append_message, FeedbackComponent): + append_message = FeedbackComponent(append_message) + kwargs["feedback_context"] = append_message + kwargs["creator"] = {"type": "to_child", "args": {"state": self}} + + def update_kwarg(name, func): + kwargs[name] = func(kwargs[name]) + + def update_context(name): + update_kwarg(name, getattr(self, name).update_ctx) + + for ast_arg in ["student_ast", "solution_ast"]: + if isinstance(kwargs.get(ast_arg), list): + update_kwarg(ast_arg, wrap_in_module) + + if kwargs.get("student_ast") and kwargs.get("student_code") is None: + kwargs["student_code"] = self.student_ast_tokens.get_text( + kwargs["student_ast"] + ) + if kwargs.get("solution_ast") and kwargs.get("solution_code") is None: + kwargs["solution_code"] = self.solution_ast_tokens.get_text( + kwargs["solution_ast"] + ) + + for context in [ + "student_context", + "solution_context", + "student_env", + "solution_env", + ]: + if context in kwargs: + if kwargs[context] is not None: + update_context(context) + else: + kwargs.pop(context) + + klass = self.SUBCLASSES[node_name] if node_name else State + init_kwargs = {**base_kwargs, **kwargs} + child = klass(**init_kwargs) + + extra_attrs = set(vars(self)) - set(self.parameters) + for attr in extra_attrs: + # don't copy attrs set on new instances in init + # the cached manual_sigs is passed + if attr not in {"ast_dispatcher", "converters"}: + setattr(child, attr, getattr(self, attr)) - if isinstance(student_subtree, list): - student_subtree = ast.Module(student_subtree) - if isinstance(solution_subtree, list): - solution_subtree = ast.Module(solution_subtree) - - # get new contexts - if solution_context is not None: - solution_context = self.solution_context.update_ctx(solution_context) - else: - solution_context = self.solution_context - - if student_context is not None: - student_context = self.student_context.update_ctx(student_context) - else: - student_context = self.student_context - - if not isinstance(append_message, dict): - append_message = {'msg': append_message, 'kwargs': {}} - - messages = [*self.messages, append_message] - - if not (solution_subtree and student_subtree): - return self.update(student_context = student_context, solution_context = solution_context, - student_parts = student_parts, solution_parts = solution_parts, - highlight = highlight, messages = messages) - - klass = State if not node_name else self.SUBCLASSES[node_name] - child = klass(student_code = utils_ast.extract_text_from_node(self.full_student_code, student_subtree), - full_student_code = self.full_student_code, - pre_exercise_code = self.pre_exercise_code, - student_context = student_context, - solution_context = solution_context, - student_process = self.student_process, - solution_process = self.solution_process, - raw_student_output = self.raw_student_output, - pre_exercise_tree = self.pre_exercise_tree, - student_tree = student_subtree, - solution_tree = solution_subtree, - student_parts = student_parts, - solution_parts = solution_parts, - highlight = highlight, - messages = messages, - parent_state = self) - return(child) - - def update(self, **kwargs): - """Return a copy of set, setting kwargs as attributes""" - child = copy(self) - for k, v in kwargs.items(): - setattr(child, k, v) return child - @staticmethod - def parse_ext(x): - rep = Reporter.active_reporter - - res = None + def has_different_processes(self): + # process classes have an _identity field that is a tuple try: - res = ast.parse(x) - # enrich tree with end lines and end columns - utils_ast.mark_text_ranges(res, x + '\n') - + return ( + self.student_process._identity[0] != self.solution_process._identity[0] + ) + except: + # play it safe (most common) + return True + + def assert_execution_root(self, fun, extra_msg=""): + if not (self.is_root or self.is_creator_type("run")): + with debugger(self): + self.report( + "`%s()` should only be called focusing on a full script, following `Ex()` or `run()`. %s" + % (fun, extra_msg) + ) + + def is_creator_type(self, type): + return self.creator and self.creator.get("type") == type + + def assert_is(self, klasses, fun, prev_fun): + if self.__class__.__name__ not in klasses: + with debugger(self): + self.report( + "`%s()` can only be called on %s." + % (fun, " or ".join(["`%s()`" % pf for pf in prev_fun])) + ) + + def assert_is_not(self, klasses, fun, prev_fun): + if self.__class__.__name__ in klasses: + with debugger(self): + self.report( + "`%s()` should not be called on %s." + % (fun, " or ".join(["`%s()`" % pf for pf in prev_fun])) + ) + + def parse_external(self, code): + res = (None, None) + try: + return self.ast_dispatcher.parse(code) except IndentationError as e: - rep.set_tag("fun", "indentation_error") e.filename = "script.py" # no line info for now - rep.feedback = Feedback("Your code could not be parsed due to an error in the indentation:
`%s.`" % str(e)) - rep.failed_test = True + self.report( + "Your code could not be parsed due to an error in the indentation:
`%s.`" + % str(e) + ) except SyntaxError as e: - rep.set_tag("fun", "syntax_error") e.filename = "script.py" # no line info for now - rep.feedback = Feedback("Your code can not be executed due to a syntax error:
`%s.`" % str(e)) - rep.failed_test = True + self.report( + "Your code can not be executed due to a syntax error:
`%s.`" % str(e) + ) # Can happen, can't catch this earlier because we can't differentiate between # TypeError in parsing or TypeError within code (at runtime). except: - rep.set_tag("fun", "other_error") - rep.feedback.message = "Something went wrong while parsing your code." - rep.failed_test = True + self.report("Something went wrong while parsing your code.") + + return res + + def parse_internal(self, code): + try: + return self.ast_dispatcher.parse(code) + except Exception as e: + self.report( + "Something went wrong when parsing the solution code: %s" % str(e) + ) + + def parse(self, text): + if self.debug: + parse_method = self.parse_internal + token_attr = "solution_ast_tokens" + else: + parse_method = self.parse_external + token_attr = "student_ast_tokens" - finally: - if (res is None): - res = False + tokens, ast = parse_method(text) + setattr(self, token_attr, tokens) - return(res) + return ast - @staticmethod - def parse_int(x): - res = None + def get_dispatcher(self): try: - res = ast.parse(x) - utils_ast.mark_text_ranges(res, x + '\n') + return Dispatcher(self.pre_exercise_code) + except Exception as e: + with debugger(self): + self.report("Something went wrong when parsing the PEC: %s" % str(e)) + + +class Dispatcher(DispatcherInterface): + _context_cache = dict() + + def __init__(self, context_code=""): + self._parser_cache = dict() + context_ast = getattr(self._context_cache, context_code, None) + if context_ast is None: + context_ast = self._context_cache[context_code] = self.parse(context_code)[ + 1 + ] + self.context_mappings = self._getx(FunctionParser, "mappings", context_ast) + + def find(self, name, node, *args, **kwargs): + return getattr(self, name)(node) + + def parse(self, code): + res = asttokens.ASTTokens(code, parse=True) + return res, res.tree + + # add methods for retrieving parser outputs -------------------------- + def _getx(self, Parser, ext_attr, tree): + """getter for Parser outputs""" + # return cached output if possible + cache_key = Parser.__name__ + str(hash(tree)) + if self._parser_cache.get(cache_key): + p = self._parser_cache[cache_key] + else: + # otherwise, run parser over tree + p = Parser() + # set mappings for parsers that inspect attribute access + if ext_attr != "mappings" and Parser in [ + FunctionParser, + ObjectAccessParser, + ]: + p.mappings = self.context_mappings.copy() + # run parser + p.visit(tree) + # cache + self._parser_cache[cache_key] = p + return getattr(p, ext_attr) + + +# put a function on the dispatcher +for k, Parser in parser_dict.items(): + setattr(Dispatcher, k, partialmethod(Dispatcher._getx, Parser, "out")) + +# mappings from ObjectAccessParser +prop_oa_map = partialmethod(Dispatcher._getx, ObjectAccessParser, "mappings") +setattr(Dispatcher, "oa_mappings", prop_oa_map) + +# mappings from FunctionParser +prop_map = partialmethod(Dispatcher._getx, FunctionParser, "mappings") +setattr(Dispatcher, "mappings", prop_map) - except SyntaxError as e: - raise SyntaxError(str(e)) - except TypeError as e: - raise TypeError(str(e)) - finally: - if (res is None): - res = False - - return(res) - -# add property methods for retrieving parser outputs -------------------------- -# note that this code is an alternative means of using something like.. -# @property -# def student_withs(self): ... -# when defining the State class. -def getx(tree_name, Parser, ext_attr, self): - """getter for Parser outputs""" - # return cached output if possible - cache_key = tree_name + Parser.__name__ - if self._parser_cache.get(cache_key): - p = self._parser_cache[cache_key] - else: - # otherwise, run parser over tree - p = Parser() - # set mappings for parsers that inspect attribute access - if ext_attr != 'mappings' and Parser in [FunctionParser, ObjectAccessParser]: - p.mappings = self.pre_exercise_mappings.copy() - # run parser - p.visit(getattr(self, tree_name)) - # cache - self._parser_cache[cache_key] = p - return getattr(p, ext_attr) - -# put a property getter on state for each parsed ast tree output. -# since the getter takes only one argument, self, partial functions -# are used to set all other arguments on getx -for s in ['student', 'solution']: - tree_name = s+'_tree' - for k, Parser in parser_dict.items(): - setattr(State, s+'_'+k, property(partial(getx, tree_name, Parser, 'out'))) - - # mappings from ObjectAccessParser - prop_oa_map = property(partial(getx, tree_name, ObjectAccessParser, 'mappings')) - setattr(State, s+'_oa_mappings', prop_oa_map) - - # mappings from FunctionParser - prop_map = property(partial(getx, tree_name, FunctionParser, 'mappings')) - setattr(State, s+'_mappings', prop_map) - -# mappings for pre exercise code from FunctionParser -pec_prop_map = property(partial(getx, 'pre_exercise_tree', FunctionParser, 'mappings')) -setattr(State, 'pre_exercise_mappings', pec_prop_map) # State subclasses based on parsed output ------------------------------------- -State.SUBCLASSES = {node_name: type(node_name, (State,), {}) for node_name in parser_dict} +State.SUBCLASSES = { + node_name: type(node_name, (State,), {}) for node_name in parser_dict +} + # global setters on State ----------------------------------------------------- def set_converter(key, fundef): diff --git a/pythonwhat/Test.py b/pythonwhat/Test.py index 9706da10..47c0703c 100644 --- a/pythonwhat/Test.py +++ b/pythonwhat/Test.py @@ -1,71 +1,18 @@ import re -from pythonwhat.Feedback import Feedback -import numpy as np -import pandas as pd from pythonwhat.tasks import * +from protowhat.Test import Test """ -This file contains all tests that can be done on specific objects. All tests are represented +This file contains all tests that can be done on specific objects. All tests are represented as an object. Tests that are alike can inherit from the same superclass. A test is first initialized and can then be performed by calling the 'test()' function. The result will be stored inside the result boolean. A test contains a failure message, which can be used by the reporter to show when the test failed. """ -class TestFail(Exception): - pass -class Test(object): - """ - The basic Test. It should only contain a failure message, as all tests should result in - a failure message when they fail. - - Note: - This test should not be used by itself, subclasses should be used. - - Attributes: - feedback (str): A string containing the failure message in case the test fails. - result (bool): True if the test succeed, False if it failed. None if it hasn't been tested yet. - """ - - def __init__(self, feedback): - """ - Initialize the standard test. +# Testing definition - Args: - feedback: string or Feedback object - """ - if (issubclass(type(feedback), Feedback)): - self.feedback = feedback - elif (issubclass(type(feedback), str)): - self.feedback = Feedback(feedback) - else: - raise TypeError("When creating a test, specify either a string or a Feedback object") - - self.result = None - - def test(self): - """ - Wrapper around specific tests. Tests only get one chance. - """ - if self.result is None: - try: - self.specific_test() - self.result = np.array(self.result).all() - except: - self.result = False - - def specific_test(self): - """ - Perform the actual test. For the standard test, result will be set to False. - """ - self.result = False - - def get_feedback(self): - return(self.feedback) - - -## Testing definition class DefinedProcessTest(Test): def __init__(self, name, process, feedback): @@ -73,7 +20,7 @@ def __init__(self, name, process, feedback): self.name = name self.process = process - def specific_test(self): + def test(self): self.result = isDefinedInProcess(self.name, self.process) @@ -87,14 +34,16 @@ class DefinedCollTest(Test): collection (list/dict/set): Contains any object on which the 'in' operator can be performed. result (bool): True if the test succeed, False if it failed. None if it hasn't been tested yet. """ + def __init__(self, name, collection, feedback): super().__init__(feedback) self.name = name self.collection = collection - def specific_test(self): + def test(self): self.result = self.name in self.collection + class DefinedCollProcessTest(Test): def __init__(self, name, key, process, feedback): super().__init__(feedback) @@ -102,20 +51,12 @@ def __init__(self, name, key, process, feedback): self.key = key self.process = process - def specific_test(self): + def test(self): self.result = isDefinedCollInProcess(self.name, self.key, self.process) -## Testing class +# Testing class -class InstanceTest(Test): - def __init__(self, obj, cls, feedback): - super().__init__(feedback) - self.obj = obj - self.cls = cls - - def specific_test(self): - self.result = isinstance(self.obj, self.cls) class InstanceProcessTest(Test): def __init__(self, name, klass, process, feedback): @@ -124,14 +65,13 @@ def __init__(self, name, klass, process, feedback): self.klass = klass self.process = process - def specific_test(self): + def test(self): self.result = isInstanceInProcess(self.name, self.klass, self.process) +# Testing equality -## Testing equality - class EqualTest(Test): """ Check if two objects are equal. Equal means the objects are exactly the same. @@ -144,113 +84,118 @@ class EqualTest(Test): result (bool): True if the test succeed, False if it failed. None if it hasn't been tested yet. """ - def __init__(self, obj1, obj2, feedback): + def __init__(self, obj1, obj2, feedback, func=None): super().__init__(feedback) self.obj1 = obj1 self.obj2 = obj2 + self.func = func if func is not None else is_equal - def specific_test(self): + def test(self): """ Perform the actual test. result is set to False if the objects differ, True otherwise. """ - self.result = is_equal(self.obj1, self.obj2) - + result = self.func(self.obj1, self.obj2) + try: + import numpy as np -class EqualProcessTest(Test): - - def __init__(self, name, student_process, sol_obj, feedback): - super().__init__(feedback) - self.name = name - self.student_process = student_process - self.sol_obj = sol_obj + self.result = np.array(result).all() + except ImportError: + self.result = result - def specific_test(self): - stud_obj = getRepresentation(self.name, self.student_process) - if isinstance(stud_obj, ReprFail): - self.result = False - else: - self.result = is_equal(stud_obj, self.sol_obj) -class EqualValueProcessTest(Test): - def __init__(self, name, key, student_process, sol_value, feedback): - super().__init__(feedback) - self.name = name - self.key = key - self.student_process = student_process - self.sol_value = sol_value +# Helpers for testing equality - def specific_test(self): - stud_value, stud_str = getValueInProcess(self.name, self.key, self.student_process) - if isinstance(stud_value, ReprFail): - self.result = False - else: - self.result = is_equal(stud_value, self.sol_value) -## Helpers for testing equality +def areinstance(x, y, tuple_of_classes): + return isinstance(x, tuple_of_classes) and isinstance(y, tuple_of_classes) -def objs_are(x, y, list_of_classes): - return ( - any([isinstance(x, klass) for klass in list_of_classes]) & - any([isinstance(y, klass) for klass in list_of_classes]) - ) +def is_primitive(x): + return isinstance(x, (str, int, float, bool, type(None))) +def is_collection_of_primitives(x): + return isinstance(x, (list, tuple, set)) and all(is_primitive(element) for element in x) +# For equality of ndarrays, list, dicts, pd Series and pd DataFrames: +# First try to the faster equality functions. If these don't pass, +# Run the assertions that are typically slower. def is_equal(x, y): + try: + if areinstance(x, y, (str, int, float, bool, type(None))): + return x == y + + if is_collection_of_primitives(x): + return x == y + + if areinstance(x, y, (Exception,)): + # Types of errors don't matter (this is debatable) + return str(x) == str(y) + + if areinstance(x, y, (list, tuple,)): + if len(x) != len(y): + return False + return all(is_equal(x_element, y_element) for x_element, y_element in zip(x, y)) + + if areinstance(x, y, (map, filter,)): + x_list, y_list = list(x), list(y) + if len(x_list) != len(y_list): + return False + return all(is_equal(x_element, y_element) for x_element, y_element in zip(x_list, y_list)) + + if areinstance(x, y, (set,)): + return is_equal(sorted(x), sorted(y)) + + if areinstance(x, y, (dict,)): + if x.keys() != y.keys(): + return False + return all(is_equal(x[key], y[key]) for key in x) + + # Delay importing pandas / numpy until absolutely necessary. This is important for performance in Pyodide. + # Also, assume they may not be available, as Pyodide won't install them unless they are needed. try: - if objs_are(x, y, [np.ndarray, dict, list]): + import numpy as np + + if areinstance(x, y, (np.ndarray,)): np.testing.assert_equal(x, y) return True - elif objs_are(x, y, [map, filter]): - np.testing.assert_equal(list(x), list(y)) - return True - elif objs_are(x, y, [pd.DataFrame]): - pd.util.testing.assert_frame_equal(x, y) - return True - elif objs_are(x, y, [pd.Series]): - pd.util.testing.assert_series_equal(x, y) + except ImportError: + if areinstance(x, y, (np.ndarray,)): + raise RuntimeError("NumPy is required for comparing NumPy objects.") + + try: + import pandas as pd + from pandas.testing import assert_frame_equal, assert_series_equal + + if areinstance(x, y, (pd.DataFrame,)): + if x.equals(y): + return True + assert_frame_equal(x, y) return True - elif objs_are(x, y, [Exception]): - assert type(x) == type(y) and str(x) == str(y) + elif areinstance(x, y, (pd.Series,)): + if x.equals(y): + return True + assert_series_equal(x, y) return True - else: - return x == y + except ImportError: + if areinstance(x, y, (pd.DataFrame, pd.Series)): + raise RuntimeError("pandas is required for comparing pandas objects.") - except Exception: - self.result = False + return x == y + except Exception: + return False -## Others -class BiggerTest(Test): - """ - Check if one object is greater than another. This test should only be used with numeric variables (for now). +# Others - Attributes: - feedback (str): A string containing the failure message in case the test fails. - obj1 (str): The first object, that should be the greatest - obj2 (str): The second object, that should be smaller - result (bool): True if the test succeed, False if it failed. None if it hasn't been tested yet. - """ - def __init__(self, obj1, obj2, feedback): - """ - Initialize with two objects. - - Args: - obj1 (str): The first object, obj1 will be set to this. - obj2 (str): The second object, obj2 will be set to this. - feedback (str): The failure message will be set to this. - """ - super().__init__(feedback) - self.obj1 = obj1 - self.obj2 = obj2 +class BiggerTest(EqualTest): + """ + Check if the first object is greater than another. + """ - def specific_test(self): - """ - Perform the actual test. result is set to False if the objects differ, True otherwise. - """ - self.result = (self.obj1 > self.obj2) + def __init__(self, *args): + super().__init__(*args, func=lambda obj1, obj2: obj1 > obj2) class StringContainsTest(Test): @@ -280,18 +225,12 @@ def __init__(self, string, search_string, pattern, feedback): self.search_string = search_string self.pattern = pattern - def specific_test(self): + def test(self): """ Perform the actual test. result will be True if string is found (whether or not with a pattern), False otherwise. """ if self.pattern: - self.result = ( - re.search( - self.search_string, - self.string) is not None) + self.result = re.search(self.search_string, self.string) is not None else: - self.result = (self.string.find(self.search_string) is not -1) - - - + self.result = self.string.find(self.search_string) != -1 diff --git a/pythonwhat/__init__.py b/pythonwhat/__init__.py index 8929c32d..47b6d8fd 100644 --- a/pythonwhat/__init__.py +++ b/pythonwhat/__init__.py @@ -1,2 +1,3 @@ -from .test_exercise import test_exercise, allow_errors +__version__ = "2.30.1" +from .test_exercise import test_exercise, allow_errors diff --git a/pythonwhat/check_funcs.py b/pythonwhat/check_funcs.py deleted file mode 100644 index a8596c56..00000000 --- a/pythonwhat/check_funcs.py +++ /dev/null @@ -1,485 +0,0 @@ -from pythonwhat.Reporter import Reporter -from pythonwhat.Test import Test, EqualTest -from pythonwhat.Feedback import Feedback -from pythonwhat.utils import get_ord -from types import GeneratorType -from functools import partial -import copy - -def part_to_child(stu_part, sol_part, append_message, state, node_name=None): - # stu_part and sol_part will be accessible on all templates - append_message['kwargs'].update({'stu_part': stu_part, 'sol_part': sol_part}) - - # if the parts are dictionaries, use to deck out child state - if all(isinstance(p, dict) for p in [stu_part, sol_part]): - return state.to_child_state(stu_part['node'], sol_part['node'], - stu_part.get('target_vars'), sol_part.get('target_vars'), - stu_part, sol_part, - highlight = stu_part.get('highlight'), - append_message = append_message, node_name=node_name) - - # otherwise, assume they are just nodes - return state.to_child_state(stu_part, sol_part, append_message = append_message) - - -def check_part(name, part_msg, state=None, missing_msg="Are you sure it's defined?", expand_msg=""): - """Return child state with name part as its ast tree""" - rep = Reporter.active_reporter - - if not part_msg: part_msg = name - append_message = {'msg': expand_msg, 'kwargs': {'part': part_msg,}} - - has_part(name, missing_msg, state, append_message['kwargs']) - - stu_part = state.student_parts[name] - sol_part = state.solution_parts[name] - - return part_to_child(stu_part, sol_part, append_message, state) - -def check_part_index(name, index, part_msg, - missing_msg="FMT:Are you sure it is defined?", - state=None, expand_msg=""): - """Return child state with indexed name part as its ast tree""" - - rep = Reporter.active_reporter - - # create message - ordinal = "" if isinstance(index, str) else get_ord(index+1) - fmt_kwargs = {'index': index, 'ordinal': ordinal} - fmt_kwargs['part'] = part_msg.format(**fmt_kwargs) - - append_message = {'msg': expand_msg, - 'kwargs': fmt_kwargs} - - # check there are enough parts for index - stu_parts = state.student_parts[name] - try: stu_parts[index] - except (KeyError, IndexError): - _msg = state.build_message(missing_msg, append_message['kwargs']) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - # get part at index - stu_part = state.student_parts[name][index] - sol_part = state.solution_parts[name][index] - - # return child state from part - return part_to_child(stu_part, sol_part, append_message, state) - -MSG_MISSING = "FMT:The system wants to check the {typestr} you defined but hasn't found it." -MSG_PREPEND = "__JINJA__:Check your code in the {{child['part']+ ' of the' if child['part']}} {{typestr}}. " -def check_node(name, index, typestr, missing_msg=MSG_MISSING, expand_msg=MSG_PREPEND, state=None): - rep = Reporter.active_reporter - stu_out = getattr(state, 'student_'+name) - sol_out = getattr(state, 'solution_'+name) - - # check if there are enough nodes for index - fmt_kwargs = {'ordinal': get_ord(index+1) if isinstance(index, int) else "", - 'index': index, - 'name': name} - fmt_kwargs['typestr'] = typestr.format(**fmt_kwargs) - - # test if node can be indexed succesfully - try: stu_out[index] - except (KeyError, IndexError): # TODO comment errors - _msg = state.build_message(missing_msg, fmt_kwargs) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - # get node at index - stu_part = stu_out[index] - sol_part = sol_out[index] - - append_message = {'msg': expand_msg, - 'kwargs': fmt_kwargs - } - - return part_to_child(stu_part, sol_part, append_message, state, node_name=name) - - -# Part tests ------------------------------------------------------------------ - -def has_part(name, msg, state=None, fmt_kwargs=None): - rep = Reporter.active_reporter - d = {'sol_part': state.solution_parts, - 'stu_part': state.student_parts, - **fmt_kwargs - } - - try: - part = state.student_parts[name] - if part is None: raise KeyError - except (KeyError, IndexError): - _msg = state.build_message(msg, d) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - return state - - -def has_equal_part(name, msg, state): - rep = Reporter.active_reporter - d = {'stu_part': state.student_parts, - 'sol_part': state.solution_parts, - 'name': name} - - _msg = state.build_message(msg, d) - rep.do_test(EqualTest(d['stu_part'][name], d['sol_part'][name], Feedback(_msg, state.highlight))) - - return state - - -def has_equal_part_len(name, insufficient_msg, state=None): - rep = Reporter.active_reporter - d = dict(stu_len = len(state.student_parts[name]), - sol_len = len(state.solution_parts[name])) - - if d['stu_len'] != d['sol_len']: - _msg = state.build_message(insufficient_msg, d) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - return state - - -# functions for running multiple sub-tests ------------------------------------ - -def extend(*args, state=None): - """Run multiple subtests in sequence, each using the output state of the previous.""" - - # when input is a single list of subtests - args = args[0] if len(args) == 1 and hasattr(args[0], '__iter__') else args - - for test in args: state = test(state=state) # run tests sequentially - return state # return final state for chaining - -def multi(*args, state=None): - """Run multiple subtests. Return original state (for chaining).""" - if any(args): - rep = Reporter.active_reporter - - # when input is a single list of subtests - if len(args) == 1 and isinstance(args[0], (list, tuple, GeneratorType)): - args = args[0] - - for test in args: - # assume test is function needing a state argument - # partial state so reporter can test - closure = partial(test, state=state) - rep.do_test(closure, "", state.highlight) - - # return original state, so can be chained - return state - -from pythonwhat.Test import TestFail - -def test_not(*args, msg, state=None): - """Pass if all of the subtests fail""" - rep = Reporter.active_reporter - - try: multi(*args, state=state) - except TestFail as e: - rep.failed_test = False # protect against old behavior - return state - - _msg = state.build_message(msg) - return rep.do_test(Test(_msg)) - -# utility functions ----------------------------------------------------------- - -def quiet(n = 0, state=None): - """Turn off prepended messages. Defaults to turning all off.""" - cpy = copy.copy(state) - hushed = [{**m, 'msg': ""} for m in cpy.messages] - cpy.messages = hushed - return cpy - -def fail(msg="", state=None): - """Fail test with message""" - rep = Reporter.active_reporter - _msg = state.build_message(msg) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - return state - -import ast -def override(solution, state=None): - """Change the focused solution code.""" - - # the old ast may be a number of node types, but generally either a - # (1) ast.Module, or for single expressions... - # (2) whatever was grabbed using module.body[0] - # (3) module.body[0].value, when module.body[0] is an Expr node - old_ast = state.solution_tree - new_ast = ast.parse(solution) - if not isinstance(old_ast, ast.Module) and len(new_ast.body) == 1: - expr = new_ast.body[0] - candidates = [expr, expr.value] if isinstance(expr, ast.Expr) else [expr] - for node in candidates: - if isinstance(node, old_ast.__class__): - new_ast = node - break - - kwargs = state.messages[-1] if state.messages else {} - child = state.to_child_state( - solution_subtree = new_ast, - student_subtree = state.student_tree, - highlight = state.highlight, - append_message = {'msg': "", 'kwargs': kwargs} - ) - - return child - - -# context functions ----------------------------------------------------------- - -from pythonwhat.tasks import setUpNewEnvInProcess, breakDownNewEnvInProcess -def with_context(*args, state=None): - # set up context in processes - solution_res = setUpNewEnvInProcess(process = state.solution_process, - context = state.solution_parts['with_items']) - if isinstance(solution_res, Exception): - raise Exception("error in the solution, running test_with() on with %d: %s" % (index - 1, str(solution_res))) - - student_res = setUpNewEnvInProcess(process = state.student_process, - context = state.student_parts['with_items']) - if isinstance(student_res, AttributeError): - rep.do_test(Test(Feedback("In your %s `with` statement, you're not using a correct context manager." % (get_ord(index)), child.highlight))) - - if isinstance(student_res, (AssertionError, ValueError, TypeError)): - rep.do_test(Test(Feedback("In your %s `with` statement, the number of values in your context manager " + \ - "doesn't correspond to the number of variables you're trying to assign it to." % (get_ord(index)), child.highlight))) - - # run subtests - try: - multi(*args, state=state) - finally: - # exit context - if breakDownNewEnvInProcess(process = state.solution_process): - raise Exception("error in the solution, closing the %s with fails with: %s" % - (get_ord(index), close_solution_context)) - - if breakDownNewEnvInProcess(process = state.student_process): - - rep.do_test(Test(Feedback("Your %s `with` statement can not be closed off correctly, you're " + \ - "not using the context manager correctly." % (get_ord(index)), state.highlight)), - fallback_ast = state.highlight) - return state - -def set_context(*args, state=None, **kwargs): - """Update context values for student and solution environments. - - Note that excess args and unmatched kwargs will be unused in the student environment. - If an argument is specified both by name and position args, will use named arg. - """ - stu_crnt = state.student_context.context - sol_crnt = state.solution_context.context - # set args specified by pos ----------------------------------------------- - # stop if too many pos args for solution - if len(args) > len(sol_crnt): - raise IndexError("Too many positional args. There are {} context vals, but tried to set {}" - .format(len(sol_crnt), len(args))) - # set pos args - upd_sol = sol_crnt.update(dict(zip(stu_crnt.keys(), args))) - upd_stu = stu_crnt.update(dict(zip(sol_crnt.keys(), args))) - - # set args specified by keyword ------------------------------------------- - if set(kwargs) - set(upd_sol): - raise KeyError("Context val names are {}, but tried to set {}" - .format(upd_sol or "none", kwargs.keys())) - out_sol = upd_sol.update(kwargs) - # need to match keys in kwargs with corresponding keys in stu context - # in case they used, e.g., different loop variable names - match_keys = dict(zip(sol_crnt.keys(), stu_crnt.keys())) - out_stu = upd_stu.update({match_keys[k]: v for k,v in kwargs.items() if k in match_keys}) - - return state.to_child_state(student_subtree = None, solution_subtree = None, - student_context = out_stu, solution_context = out_sol) - - -def check_args(name, missing_msg='FMT:Are you sure it is defined?', state=None): - if name in ['*args', '**kwargs']: - return check_part(name, name, state=state, missing_msg = missing_msg) - else: - arg_str = "%s argument"%get_ord(name+1) if isinstance(name, int) else "argument `%s`"%name - return check_part_index('args', name, arg_str, state=state, missing_msg = missing_msg) - - -# CALL CHECK ================================================================== - -from pythonwhat.tasks import getResultInProcess, getOutputInProcess, getErrorInProcess, ReprFail -import ast - -evalCalls = {'value': getResultInProcess, - 'output': getOutputInProcess, - 'error': getErrorInProcess} - -call_warnings = { - 'value': 'in the solution process resulted in an error', - 'error': 'did not generate an error in the solution environment', - 'output': 'in the solution process resulted in an error' - } - -def fix_format(arguments): - if isinstance(arguments, str): - arguments = (arguments, ) - if isinstance(arguments, tuple): - arguments = list(arguments) - - if isinstance(arguments, list): - arguments = {'args': arguments, 'kwargs': {}} - - if not isinstance(arguments, dict) or 'args' not in arguments or 'kwargs' not in arguments: - raise ValueError("Wrong format of arguments in 'results', 'outputs' or 'errors'; either a list, or a dictionary with names args (a list) and kwargs (a dict)") - - return(arguments) - -# TODO: test string syntax with check_function_def -# test argument syntax with check_lambda -# implement for error and output -def run_call(args, node, process, get_func, **kwargs): - # Get function expression - if isinstance(node, ast.FunctionDef): # function name - func_expr = ast.Name(id=node.name, ctx=ast.Load()) - elif isinstance(node, ast.Lambda): # lambda body expr - func_expr = node - else: raise TypeError("Only function definition or lambda may be called") - - # args is a call string or argument list/dict - if isinstance(args, str): - parsed = ast.parse(args).body[0].value - parsed.func = func_expr - ast.fix_missing_locations(parsed) - return get_func(process = process, tree = parsed, **kwargs) - else: - # e.g. list -> {args: [...], kwargs: {}} - fmt_args = fix_format(args) - ast.fix_missing_locations(func_expr) - return get_func(process = process, tree=func_expr, call = fmt_args, **kwargs) - - -MSG_CALL_INCORRECT = "FMT:Calling it should result in {str_sol}, instead got {str_stu}" -MSG_CALL_ERROR = "FMT:Calling it should result in {str_sol}, instead got an error" -def call(args, - test='value', - incorrect_msg=MSG_CALL_INCORRECT, - error_msg=MSG_CALL_ERROR, - # TODO kept for backwards compatibility in test_function_definition/lambda - argstr='', - state=None, **kwargs): - rep = Reporter.active_reporter - test_type = ('value', 'output', 'error') - - get_func = evalCalls[test] - - # Run for Solution -------------------------------------------------------- - eval_sol, str_sol = run_call(args, state.solution_parts['node'], state.solution_process, get_func, **kwargs) - - if (test == 'error') ^ isinstance(str_sol, Exception): - _msg = state.build_message("FMT:Calling for arguments {args} resulted in an error (or not an error if testing for one). Error message: {type_err} {str_sol}", - dict(args=args, type_err=type(str_sol), str_sol=str_sol)) - raise ValueError(_msg) - - if isinstance(eval_sol, ReprFail): - _msg = state.build_message("FMT:Can't get the result of calling it for arguments {args}: {eval_sol.info}", - dict(args = args, eval_sol=eval_sol)) - raise ValueError(_msg) - - # Run for Submission ------------------------------------------------------ - eval_stu, str_stu = run_call(args, state.student_parts['node'], state.student_process, get_func, **kwargs) - fmt_kwargs = {'part': argstr, 'argstr': argstr, 'str_sol': str_sol, 'str_stu': str_stu} - - # either error test and no error, or vice-versa - stu_node = state.student_parts['node'] - if (test == 'error') ^ isinstance(str_stu, Exception): - _msg = state.build_message(error_msg, fmt_kwargs) - rep.do_test(Test(Feedback(_msg, stu_node))) - - # incorrect result - _msg = state.build_message(incorrect_msg, fmt_kwargs) - rep.do_test(EqualTest(eval_sol, eval_stu, Feedback(_msg, stu_node))) - - return state - -# Expression tests ------------------------------------------------------------ -from pythonwhat.tasks import ReprFail, UndefinedValue -from pythonwhat import utils - -def has_equal_ast(incorrect_msg="FMT: Your code does not seem to match the solution.", state=None): - rep = Reporter.active_reporter - - stu_rep = ast.dump(state.student_tree) - sol_rep = ast.dump(state.solution_tree) - - _msg = state.build_message(incorrect_msg) - rep.do_test(EqualTest(stu_rep, sol_rep, Feedback(_msg, state.highlight))) - - return state - -def has_expr(incorrect_msg="FMT:Unexpected expression {test}: expected `{sol_eval}`, got `{stu_eval}` with values{extra_env}.", - error_msg="Running an expression in the student process caused an issue.", - undefined_msg="FMT:Have you defined `{name}` without errors?", - extra_env=None, - context_vals=None, - expr_code=None, - pre_code=None, - keep_objs_in_env=None, - name=None, - highlight=None, - state=None, - test=None): - rep = Reporter.active_reporter - - # run function to highlight a block of code - if callable(highlight): - try: highlight = highlight(state=state).student_tree - except: pass - highlight = highlight or state.highlight - - get_func = partial(evalCalls[test], - extra_env = extra_env, - context_vals = context_vals, - pre_code = pre_code, - expr_code = expr_code, - keep_objs_in_env = keep_objs_in_env, - name=name, - do_exec = True if test == 'output' else False) - - eval_sol, str_sol = get_func(tree = state.solution_tree, - process = state.solution_process, - context = state.solution_context) - - if (test == 'error') ^ isinstance(str_sol, Exception): - raise ValueError("evaluating expression raised error in solution process (or not an error if testing for one). " - "Error: %s - %s"%(type(str_sol), str_sol)) - if isinstance(eval_sol, ReprFail): - raise ValueError("Couldn't figure out the value of a default argument: " + eval_sol.info) - - eval_stu, str_stu = get_func(tree = state.student_tree, - process = state.student_process, - context = state.student_context) - - # kwargs --- - fmt_kwargs = {'stu_part': state.student_parts, 'sol_part': state.solution_parts, - 'name': name, 'test': test, - 'extra_env': " "+str(extra_env or ""), 'context_vals': context_vals} - fmt_kwargs['stu_eval'] = utils.shorten_str(str(eval_stu)) - fmt_kwargs['sol_eval'] = utils.shorten_str(str(eval_sol)) - - # tests --- - # error in process - if (test == 'error') ^ isinstance(str_stu, Exception): - _msg = state.build_message(error_msg, fmt_kwargs) - feedback = Feedback(_msg, highlight) - rep.do_test(Test(feedback)) - - # name is undefined after running expression - if isinstance(str_stu, UndefinedValue): - _msg = state.build_message(undefined_msg, fmt_kwargs) - rep.do_test(Test(Feedback(_msg, highlight))) - - # test equality of results - _msg = state.build_message(incorrect_msg, fmt_kwargs) - rep.do_test(EqualTest(eval_stu, eval_sol, Feedback(_msg, highlight))) - - return state - -has_equal_value = partial(has_expr, test = 'value') -has_equal_output = partial(has_expr, test = 'output') -has_equal_error = partial(has_expr, test = 'error') diff --git a/pythonwhat/check_function.py b/pythonwhat/check_function.py deleted file mode 100644 index 2d72a159..00000000 --- a/pythonwhat/check_function.py +++ /dev/null @@ -1,72 +0,0 @@ -from pythonwhat.Reporter import Reporter -from pythonwhat.check_funcs import part_to_child -from pythonwhat.test_funcs.test_function import bind_args -from pythonwhat.tasks import getSignatureInProcess -from pythonwhat.utils import get_ord -from pythonwhat.Test import Test -from pythonwhat.Feedback import Feedback -from pythonwhat.parsing import IndexedDict -from functools import partial - -def bind_args(signature, args_part): - pos_args = []; kw_args = {} - for k, arg in args_part.items(): - if isinstance(k, int): pos_args.append(arg) - else: kw_args[k] = arg - - bound_args = signature.bind(*pos_args, **kw_args) - - return (IndexedDict(bound_args.arguments), signature) - -MSG_PREPEND = "__JINJA__:Check your code in the {{child['part']+ ' of the' if child['part']}} {{typestr}}. " -def check_function(name, index, - missing_msg = "FMT:Did you define {typestr}?", - params_not_matched_msg = "FMT:Something went wrong in figuring out how you specified the " - "arguments for `{name}`; have another look at your code and its output.", - expand_msg = MSG_PREPEND, - signature=True, - typestr = "{ordinal} function call", - state=None): - rep = Reporter.active_reporter - stu_out = state.student_function_calls - sol_out = state.solution_function_calls - - fmt_kwargs = {'ordinal': get_ord(index+1), - 'index': index, - 'name': name} - fmt_kwargs['typestr'] = typestr.format(**fmt_kwargs) - - # Get Parts ---- - try: - stu_parts = stu_out[name][index] - except (KeyError, IndexError): - _msg = state.build_message(missing_msg, fmt_kwargs) - rep.do_test(Test(Feedback(_msg, state.highlight))) - - sol_parts = sol_out[name][index] - - # Signatures ----- - if signature: - signature = None if isinstance(signature, bool) else signature - get_sig = partial(getSignatureInProcess, name=name, signature=signature, - manual_sigs = state.get_manual_sigs()) - - try: - sol_sig = get_sig(mapped_name=sol_parts['name'], process=state.solution_process) - sol_parts['args'], _ = bind_args(sol_sig, sol_parts['args']) - except: - raise ValueError("Something went wrong in matching call index {index} of {name} to its signature. " - "You might have to manually specify or correct the signature." - .format(index=index, name=name)) - - try: - stu_sig = get_sig(mapped_name=stu_parts['name'], process=state.student_process) - stu_parts['args'], _ = bind_args(stu_sig, stu_parts['args']) - except Exception as e: - _msg = state.build_message(params_not_matched_msg, fmt_kwargs) - rep.do_test(Test(Feedback(_msg, stu_parts['node']))) - - # three types of parts: pos_args, keywords, args (e.g. these are bound to sig) - append_message = {'msg': expand_msg, 'kwargs': fmt_kwargs} - child = part_to_child(stu_parts, sol_parts, append_message, state, node_name='function_calls') - return child diff --git a/pythonwhat/check_object.py b/pythonwhat/check_object.py deleted file mode 100644 index 5db3f461..00000000 --- a/pythonwhat/check_object.py +++ /dev/null @@ -1,82 +0,0 @@ -from pythonwhat.parsing import ObjectAssignmentParser -from pythonwhat.Test import DefinedProcessTest, InstanceProcessTest, DefinedCollProcessTest, EqualValueProcessTest -from pythonwhat.Reporter import Reporter -from pythonwhat.Feedback import Feedback -from pythonwhat.tasks import isDefinedInProcess, isInstanceInProcess, getValueInProcess, isDefinedCollInProcess, ReprFail -from pythonwhat.check_funcs import part_to_child, has_equal_value - - -MSG_PREPEND = "FMT:Check the variable `{index}`. " -MSG_UNDEFINED = "FMT:Are you sure you defined the {typestr}, `{index}`?" -MSG_INCORRECT_VAL = """FMT: Have you specified the correct value for "{key}" inside `{parent[sol_part][name]}`?""" -MSG_KEY_MISSING = "__JINJA__:There is no {{ 'column' if 'DataFrame' in parent.typestr else 'key' }} inside {{parent.index}}." - -def check_object(index, missing_msg=MSG_UNDEFINED, expand_msg=MSG_PREPEND, state=None, typestr="variable"): - rep = Reporter.active_reporter - - if not isDefinedInProcess(index, state.solution_process): - raise NameError("%r not in solution environment " % index) - - append_message = {'msg': expand_msg, 'kwargs': {'index': index, 'typestr': typestr}} - - # create child state, using either parser output, or create part from name - fallback = lambda: ObjectAssignmentParser.get_part(index) - stu_part = state.student_object_assignments.get(index, fallback()) - sol_part = state.solution_object_assignments.get(index, fallback()) - - # test object exists - _msg = state.build_message(missing_msg, append_message['kwargs']) - rep.do_test(DefinedProcessTest(index, state.student_process, Feedback(_msg))) - - child = part_to_child(stu_part, sol_part, append_message, state) - - return child - -def is_instance(inst, not_instance_msg="FMT:Is it a {inst.__name__}?", name=None, state=None): - rep = Reporter.active_reporter - - sol_name = name or state.solution_parts.get('name') - stu_name = name or state.student_parts.get('name') - - if not isInstanceInProcess(sol_name, inst, state.solution_process): - raise ValueError("%r is not a %s in the solution environment" % (sol_name, type(inst))) - - _msg = state.build_message(not_instance_msg, {'inst': inst}) - feedback = Feedback(_msg, state.highlight) - rep.do_test(InstanceProcessTest(stu_name, inst, state.student_process, feedback)) - - return state - -def has_key(key, key_missing_msg=MSG_KEY_MISSING, name = None, state=None): - rep = Reporter.active_reporter - - sol_name = name or state.solution_parts.get('name') - stu_name = name or state.student_parts.get('name') - - if not isDefinedCollInProcess(sol_name, key, state.solution_process): - raise NameError("Not all keys you specified are actually keys in %s in the solution process" % sol_name) - - # check if key available - _msg = state.build_message(key_missing_msg, {'key': key}) - rep.do_test(DefinedCollProcessTest(stu_name, key, state.student_process, - Feedback(_msg, state.highlight))) - - return state - -def has_equal_key(key, incorrect_value_msg=MSG_INCORRECT_VAL, key_missing_msg=MSG_KEY_MISSING, name=None, state=None): - rep = Reporter.active_reporter - - sol_name = name or state.solution_parts.get('name') - stu_name = name or state.student_parts.get('name') - - has_key(key, key_missing_msg, state=state) - - sol_value, sol_str = getValueInProcess(sol_name, key, state.solution_process) - if isinstance(sol_value, ReprFail): - raise NameError("Value from %r can't be fetched from the solution process: %s" % c(sol_name, sol_value.info)) - - # check if value ok - _msg = state.build_message(incorrect_value_msg, {'key': key}) - rep.do_test(EqualValueProcessTest(stu_name, key, state.student_process, sol_value, Feedback(_msg, state.highlight))) - - return state diff --git a/pythonwhat/check_syntax.py b/pythonwhat/check_syntax.py deleted file mode 100644 index 886a0b30..00000000 --- a/pythonwhat/check_syntax.py +++ /dev/null @@ -1,106 +0,0 @@ -from pythonwhat.check_wrappers import scts -from pythonwhat.State import State -from pythonwhat.probe import Node, Probe, TEST_NAMES -from pythonwhat import test_funcs -from functools import partial, reduce, wraps -import inspect -import copy - -# TODO: could define scts for check_wrappers at the module level -ATTR_SCTS = scts.copy() - -def multi_dec(f): - """Decorator for multi to remove nodes for original test functions from root node""" - - @wraps(f) - def wrapper(*args, **kwargs): - args = args[0] if len(args) == 1 and isinstance(args[0], (list, tuple)) else args - for arg in args: - if isinstance(arg, Node) and arg.parent.name is 'root': - arg.parent.remove_child(arg) - arg.update_child_calls() - return f(*args, **kwargs) - return wrapper - - -def state_dec(f): - """Decorate check_* functions to return F chain if no state passed""" - - @wraps(f) - def wrapper(*args, **kwargs): - ba = inspect.signature(f).bind(*args, **kwargs) - ba.apply_defaults() - - state_arg = ba.arguments.get('state') - if isinstance(state_arg, State): # proper state, run function - return f(*args, **kwargs) - elif state_arg is None: # default state arg, make partial - return F._from_func(partial(f, *args, **kwargs)) - else: # passed improper state arg - raise BaseException("Did you use the right number of arguments in your SCT?") - - return wrapper - -class Chain: - def __init__(self, state): - self._state = state - self._crnt_sct = None - self._waiting_on_call = False - - def __getattr__(self, attr): - if attr not in ATTR_SCTS: raise AttributeError("No SCT named %s"%attr) - elif self._waiting_on_call: - raise AttributeError("Did you forget to call a statement? " - "e.g. Ex().check_list_comp.check_body()") - else: - # make a copy to return, - # in case someone does: a = chain.a; b = chain.b - chain = copy.copy(self) - chain._crnt_sct = ATTR_SCTS[attr] - chain._waiting_on_call = True - return chain - - def __call__(self, *args, **kwargs): - self._state = self._crnt_sct(state=self._state, *args, **kwargs) - self._waiting_on_call = False - return self - -class F(Chain): - def __init__(self, stack = None): - self._crnt_sct = None - self._stack = [] if stack is None else stack - self._waiting_on_call = False - - def __call__(self, *args, **kwargs): - if not self._crnt_sct: - state = kwargs.get('state') or args[0] - return reduce(lambda s, f: f(state=s), self._stack, state) - else: - pf = partial(self._crnt_sct, *args, **kwargs) - return self.__class__(self._stack + [pf]) - - @classmethod - def _from_func(cls, f): - func_chain = cls() - func_chain._stack.append(f) - return func_chain - -def Ex(): - return Chain(State.root_state) - -# Prepare SCTs that may be chained attributes ---------------------- -# decorate functions that may try to run test_* function nodes as subtests -# so they remove those nodes from the tree -for k in ['multi', 'with_context', 'test_not']: - ATTR_SCTS[k] = multi_dec(ATTR_SCTS[k]) -# allow test_* functions as chained attributes -for k in TEST_NAMES: - ATTR_SCTS[k] = Probe(tree = None, f = getattr(test_funcs, k), eval_on_call=True) -# original logical test_* functions behave like multi, test_not -# this is necessary to allow them to take check_* funcs as args -# since probe behavior will try to call all SCTs passed (assuming they're also probes) -for k in ['test_or', 'test_correct']: - ATTR_SCTS[k] = multi_dec(getattr(test_funcs, k)) - -# Prepare check_funcs to be used alone (e.g. test = check_with().check_body()) -spec_2_context = {k : state_dec(v) for k, v in scts.items()} diff --git a/pythonwhat/check_wrappers.py b/pythonwhat/check_wrappers.py deleted file mode 100644 index deae9ace..00000000 --- a/pythonwhat/check_wrappers.py +++ /dev/null @@ -1,74 +0,0 @@ -from pythonwhat.check_funcs import check_part, check_part_index, check_node, has_equal_part -from pythonwhat import check_funcs, check_object -from pythonwhat.check_function import check_function -from pythonwhat.test_funcs.test_data_frame import check_df -from pythonwhat.test_funcs.test_dictionary import check_dict -from pythonwhat import test_funcs -from functools import partial -import inspect -#from jinja2 import Template - -__PART_WRAPPERS__ = { - 'iter': 'iterable part', - 'body': 'body', - 'key' : 'key part', - 'value': 'value part', - 'orelse': 'else part', - 'test': 'condition' - } - -__PART_INDEX_WRAPPERS__ = { - 'ifs': '{ordinal} if', - 'handlers': '{index} `except` block', - 'context': '{ordinal} context' - } - -__NODE_WRAPPERS__ = { - 'list_comp': '{ordinal} list comprehension', - 'generator_exp': '{ordinal} generator expression', - 'dict_comp': '{ordinal} dictionary comprehension', - 'for_loop': '{ordinal} for statement', - 'function_def': 'definition of `{index}()`', - 'if_exp': '{ordinal} if expression', - 'if_else': '{ordinal} if statement', - 'lambda_function': '{ordinal} lambda function', - 'try_except': '{ordinal} try statement', - 'while': '{ordinal} `while` loop', - 'with': '{ordinal} `with` statement' - } - -scts = {} - -# make has_equal_part wrappers - -scts['has_equal_name'] = partial(has_equal_part, 'name', msg='Make sure to use the correct {name}, was expecting {sol_part[name]}, instead got {stu_part[name]}.') -scts['is_default'] = partial(has_equal_part, 'is_default', msg="__JINJA__:Make sure it {{ 'has' if sol_part.is_default else 'does not have'}} a default argument.") - -# include rest of wrappers -for k, v in __PART_WRAPPERS__.items(): - scts['check_'+k] = partial(check_part, k, v) - -for k, v in __PART_INDEX_WRAPPERS__.items(): - scts['check_'+k] = partial(check_part_index, k, part_msg=v) - - -for k, v in __NODE_WRAPPERS__.items(): - scts['check_'+k] = partial(check_node, k+'s', typestr=v) -scts['check_function'] = check_function - -for k in ['set_context', - 'has_equal_value', 'has_equal_output', 'has_equal_error', 'has_equal_ast', 'call', - 'extend', 'multi', 'test_not', 'fail', 'quiet', 'override', - 'with_context', - 'check_args', - 'has_equal_part']: - scts[k] = getattr(check_funcs, k) - -# include check_object and friends ------ -for k in ['check_object', 'is_instance', 'has_equal_key', 'has_key']: - scts[k] = getattr(check_object, k) - -scts['check_df'] = check_df -scts['check_dict'] = check_dict - - diff --git a/pythonwhat/output.py b/pythonwhat/checks/__init__.py similarity index 100% rename from pythonwhat/output.py rename to pythonwhat/checks/__init__.py diff --git a/pythonwhat/checks/check_funcs.py b/pythonwhat/checks/check_funcs.py new file mode 100644 index 00000000..be618aa4 --- /dev/null +++ b/pythonwhat/checks/check_funcs.py @@ -0,0 +1,329 @@ +from protowhat.Feedback import FeedbackComponent +from pythonwhat.checks.check_logic import multi +from pythonwhat.checks.has_funcs import has_part +from protowhat.failure import debugger +from pythonwhat.tasks import setUpNewEnvInProcess, breakDownNewEnvInProcess +from protowhat.utils_messaging import get_ord +from pythonwhat.utils_ast import assert_ast +import ast +from jinja2 import Template + + +def render(template, kwargs): + return Template(template).render(**kwargs) + + +def part_to_child(stu_part, sol_part, append_message, state, node_name=None): + # stu_part and sol_part will be accessible on all templates + append_message.kwargs.update({"stu_part": stu_part, "sol_part": sol_part}) + + # if the parts are dictionaries, use to deck out child state + if all(isinstance(p, dict) for p in [stu_part, sol_part]): + child_state = state.to_child( + student_ast=stu_part["node"], + solution_ast=sol_part["node"], + student_context=stu_part.get("target_vars"), + solution_context=sol_part.get("target_vars"), + student_parts=stu_part, + solution_parts=sol_part, + highlight=stu_part.get("highlight"), + append_message=append_message, + node_name=node_name, + ) + else: + # otherwise, assume they are just nodes + child_state = state.to_child( + student_ast=stu_part, + solution_ast=sol_part, + append_message=append_message, + node_name=node_name, + ) + + return child_state + + +def check_part(state, name, part_msg, missing_msg=None, expand_msg=None): + """Return child state with name part as its ast tree""" + + if missing_msg is None: + missing_msg = "Are you sure you defined the {{part}}? " + if expand_msg is None: + expand_msg = "Did you correctly specify the {{part}}? " + + if not part_msg: + part_msg = name + append_message = FeedbackComponent(expand_msg, {"part": part_msg}) + + has_part(state, name, missing_msg, append_message.kwargs) + + stu_part = state.student_parts[name] + sol_part = state.solution_parts[name] + + assert_ast(state, sol_part, append_message.kwargs) + + return part_to_child(stu_part, sol_part, append_message, state) + + +def check_part_index(state, name, index, part_msg, missing_msg=None, expand_msg=None): + """Return child state with indexed name part as its ast tree. + + ``index`` can be: + + - an integer, in which case the student/solution_parts are indexed by position. + - a string, in which case the student/solution_parts are expected to be a dictionary. + - a list of indices (which can be integer or string), in which case the student parts are indexed step by step. + """ + + if missing_msg is None: + missing_msg = "Are you sure you defined the {{part}}? " + if expand_msg is None: + expand_msg = "Did you correctly specify the {{part}}? " + + # create message + ordinal = get_ord(index + 1) if isinstance(index, int) else "" + fmt_kwargs = {"index": index, "ordinal": ordinal} + fmt_kwargs.update(part=render(part_msg, fmt_kwargs)) + + append_message = FeedbackComponent(expand_msg, fmt_kwargs) + + # check there are enough parts for index + has_part(state, name, missing_msg, fmt_kwargs, index) + + # get part at index + stu_part = state.student_parts[name] + sol_part = state.solution_parts[name] + + if isinstance(index, list): + for ind in index: + stu_part = stu_part[ind] + sol_part = sol_part[ind] + else: + stu_part = stu_part[index] + sol_part = sol_part[index] + + assert_ast(state, sol_part, fmt_kwargs) + + # return child state from part + return part_to_child(stu_part, sol_part, append_message, state) + + +def check_node( + state, name, index=0, typestr="{{ordinal}} node", missing_msg=None, expand_msg=None +): + + if missing_msg is None: + missing_msg = "The system wants to check the {{typestr}} but hasn't found it." + if expand_msg is None: + expand_msg = "Check the {{typestr}}. " + + stu_out = state.ast_dispatcher.find(name, state.student_ast) + sol_out = state.ast_dispatcher.find(name, state.solution_ast) + + # check if there are enough nodes for index + fmt_kwargs = { + "ordinal": get_ord(index + 1) if isinstance(index, int) else "", + "index": index, + "name": name, + } + fmt_kwargs["typestr"] = render(typestr, fmt_kwargs) + + # test if node can be indexed succesfully + try: + stu_out[index] + except (KeyError, IndexError): # TODO comment errors + state.report(missing_msg, fmt_kwargs) + + # get node at index + stu_part = stu_out[index] + sol_part = sol_out[index] + + append_message = FeedbackComponent(expand_msg, fmt_kwargs) + + return part_to_child(stu_part, sol_part, append_message, state, node_name=name) + + +# context functions ----------------------------------------------------------- +# TODO: check if still useful +def with_context(state, *args, child=None): + + # set up context in processes + solution_res = setUpNewEnvInProcess( + process=state.solution_process, context=state.solution_parts["with_items"] + ) + if isinstance(solution_res, Exception): + with debugger(state): + state.report( + "error in the solution, running test_with(): %s" % str(solution_res) + ) + + student_res = setUpNewEnvInProcess( + process=state.student_process, context=state.student_parts["with_items"] + ) + if isinstance(student_res, AttributeError): + child.report( + "In your `with` statement, you're not using a correct context manager." + ) + + if isinstance(student_res, (AssertionError, ValueError, TypeError)): + child.report( + "In your `with` statement, the number of values in your context manager " + "doesn't correspond to the number of variables you're trying to assign it to." + ) + + # run subtests + try: + multi(state, *args) + finally: + # exit context + close_solution_context = breakDownNewEnvInProcess( + process=state.solution_process + ) + if isinstance(close_solution_context, Exception): + with debugger(state): + state.report( + "error in the solution, closing the `with` fails with: %s" + % close_solution_context + ) + + close_student_context = breakDownNewEnvInProcess(process=state.student_process) + if isinstance(close_student_context, Exception): + state.report( + "Your `with` statement can not be closed off correctly, you're " + "not using the context manager correctly." + ) + return state + + +def check_args(state, name, missing_msg=None): + """Check whether a function argument is specified. + + This function can follow ``check_function()`` in an SCT chain and verifies whether an argument is specified. + If you want to go on and check whether the argument was correctly specified, you can can continue chaining with + ``has_equal_value()`` (value-based check) or ``has_equal_ast()`` (AST-based check) + + This function can also follow ``check_function_def()`` or ``check_lambda_function()`` to see if arguments have been + specified. + + Args: + name (str): the name of the argument for which you want to check if it is specified. This can also be + a number, in which case it refers to the positional arguments. Named arguments take precedence. + missing_msg (str): If specified, this overrides the automatically generated feedback message in case + the student did specify the argument. + state (State): State object that is passed from the SCT Chain (don't specify this). + + :Examples: + + Student and solution code:: + + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + SCT:: + + # Verify whether arr was correctly set in np.mean + # has_equal_value() checks the value of arr, used to set argument a + Ex().check_function('numpy.mean').check_args('a').has_equal_value() + + # Verify whether arr was correctly set in np.mean + # has_equal_ast() checks the expression used to set argument a + Ex().check_function('numpy.mean').check_args('a').has_equal_ast() + + Student and solution code:: + + def my_power(x): + print("calculating sqrt...") + return(x * x) + + SCT:: + + Ex().check_function_def('my_power').multi( + check_args('x') # will fail if student used y as arg + check_args(0) # will still pass if student used y as arg + ) + + """ + if missing_msg is None: + missing_msg = "Did you specify the {{part}}?" + + if name in ["*args", "**kwargs"]: # for check_function_def + return check_part(state, name, name, missing_msg=missing_msg) + else: + if isinstance(name, list): # dealing with args or kwargs + if name[0] == "args": + arg_str = "{} argument passed as a variable length argument".format( + get_ord(name[1] + 1) + ) + else: + arg_str = "argument `{}`".format(name[1]) + else: + arg_str = ( + "{} argument".format(get_ord(name + 1)) + if isinstance(name, int) + else "argument `{}`".format(name) + ) + return check_part_index(state, "args", name, arg_str, missing_msg=missing_msg) + + +# CALL CHECK ================================================================== + + +def build_call(callstr, node): + if isinstance(node, ast.FunctionDef): # function name + func_expr = ast.Name(id=node.name, ctx=ast.Load()) + argstr = "`%s`" % callstr.replace("f", node.name) + elif isinstance(node, ast.Lambda): # lambda body expr + func_expr = node + argstr = "it with the arguments `%s`" % callstr.replace("f", "") + else: + raise TypeError("Can't handle AST that is passed.") + + parsed = ast.parse(callstr).body[0].value + parsed.func = func_expr + ast.fix_missing_locations(parsed) + return parsed, argstr + + +def check_call(state, callstr, argstr=None, expand_msg=None): + """When checking a function definition of lambda function, + prepare has_equal_x for checking the call of a user-defined function. + + Args: + callstr (str): call string that specifies how the function should be called, e.g. `f(1, a = 2)`. + ``check_call()`` will replace ``f`` with the function/lambda you're targeting. + argstr (str): If specified, this overrides the way the function call is refered to in the expand message. + expand_msg (str): If specified, this overrides any messages that are prepended by previous SCT chains. + state (State): state object that is chained from. + + :Example: + + Student and solution code:: + + def my_power(x): + print("calculating sqrt...") + return(x * x) + + SCT:: + + Ex().check_function_def('my_power').multi( + check_call("f(3)").has_equal_value() + check_call("f(3)").has_equal_output() + ) + """ + + state.assert_is( + ["function_defs", "lambda_functions"], + "check_call", + ["check_function_def", "check_lambda_function"], + ) + + if expand_msg is None: + expand_msg = "To verify it, we reran {{argstr}}. " + + stu_part, _argstr = build_call(callstr, state.student_parts["node"]) + sol_part, _ = build_call(callstr, state.solution_parts["node"]) + + append_message = FeedbackComponent(expand_msg, {"argstr": argstr or _argstr}) + child = part_to_child(stu_part, sol_part, append_message, state) + + return child diff --git a/pythonwhat/checks/check_function.py b/pythonwhat/checks/check_function.py new file mode 100644 index 00000000..ca093c24 --- /dev/null +++ b/pythonwhat/checks/check_function.py @@ -0,0 +1,172 @@ +from protowhat.Feedback import FeedbackComponent +from pythonwhat.checks.check_funcs import part_to_child +from pythonwhat.tasks import getSignatureInProcess +from protowhat.utils_messaging import get_ord, get_times +from protowhat.failure import debugger +from pythonwhat.parsing import IndexedDict +from functools import partial + + +def bind_args(signature, args_part): + pos_args = [] + kw_args = {} + for k, arg in args_part.items(): + if isinstance(k, int): + pos_args.append(arg) + else: + kw_args[k] = arg + + bound_args = signature.bind(*pos_args, **kw_args) + return IndexedDict(bound_args.arguments) + + +def get_mapped_name(name, mappings): + # get name by splitting on periods + if "." in name: + for orig, full_name in mappings.items(): + if name.startswith(full_name): + return name.replace(full_name, orig) + return name + + +MISSING_MSG = "Did you call `{{mapped_name}}()`{{' ' + times if index>0}}?" +SIG_ISSUE_MSG = ( + "Have you specified the arguments for `{{mapped_name}}()` using the right syntax?" +) +PREPEND_MSG = "Check your {{ord + ' ' if index>0}}call of `{{mapped_name}}()`. " + + +def check_function( + state, + name, + index=0, + missing_msg=None, + params_not_matched_msg=None, + expand_msg=None, + signature=True, +): + """Check whether a particular function is called. + + ``check_function()`` is typically followed by: + + - ``check_args()`` to check whether the arguments were specified. + In turn, ``check_args()`` can be followed by ``has_equal_value()`` or ``has_equal_ast()`` + to assert that the arguments were correctly specified. + - ``has_equal_value()`` to check whether rerunning the function call coded by the student + gives the same result as calling the function call as in the solution. + + Checking function calls is a tricky topic. Please visit the + `dedicated article `_ for more explanation, + edge cases and best practices. + + Args: + name (str): the name of the function to be tested. When checking functions in packages, always + use the 'full path' of the function. + index (int): index of the function call to be checked. Defaults to 0. + missing_msg (str): If specified, this overrides an automatically generated feedback message in case + the student did not call the function correctly. + params_not_matched_msg (str): If specified, this overrides an automatically generated feedback message + in case the function parameters were not successfully matched. + expand_msg (str): If specified, this overrides any messages that are prepended by previous SCT chains. + signature (Signature): Normally, check_function() can figure out what the function signature is, + but it might be necessary to use ``sig_from_params()`` to manually build a signature and pass this along. + state (State): State object that is passed from the SCT Chain (don't specify this). + + :Examples: + + Student code and solution code:: + + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + SCT:: + + # Verify whether arr was correctly set in np.mean + Ex().check_function('numpy.mean').check_args('a').has_equal_value() + + # Verify whether np.mean(arr) produced the same result + Ex().check_function('numpy.mean').has_equal_value() + """ + + append_missing = missing_msg is None + append_params_not_matched = params_not_matched_msg is None + if missing_msg is None: + missing_msg = MISSING_MSG + if expand_msg is None: + expand_msg = PREPEND_MSG + if params_not_matched_msg is None: + params_not_matched_msg = SIG_ISSUE_MSG + + stu_out = state.ast_dispatcher.find("function_calls", state.student_ast) + sol_out = state.ast_dispatcher.find("function_calls", state.solution_ast) + + student_mappings = state.ast_dispatcher.find("mappings", state.student_ast) + + fmt_kwargs = { + "times": get_times(index + 1), + "ord": get_ord(index + 1), + "index": index, + "mapped_name": get_mapped_name(name, student_mappings), + } + + # Get Parts ---- + # Copy, otherwise signature binding overwrites sol_out[name][index]['args'] + with debugger(state): + try: + sol_parts = {**sol_out[name][index]} + except KeyError: + state.report( + "`check_function()` couldn't find a call of `%s()` in the solution code. Make sure you get the mapping right!" + % name + ) + except IndexError: + state.report( + "`check_function()` couldn't find %s calls of `%s()` in your solution code." + % (index + 1, name) + ) + + try: + # Copy, otherwise signature binding overwrites stu_out[name][index]['args'] + stu_parts = {**stu_out[name][index]} + except (KeyError, IndexError): + state.report(missing_msg, fmt_kwargs, append=append_missing) + + # Signatures ----- + if signature: + signature = None if isinstance(signature, bool) else signature + get_sig = partial( + getSignatureInProcess, + name=name, + signature=signature, + manual_sigs=state.get_manual_sigs(), + ) + + try: + sol_sig = get_sig( + mapped_name=sol_parts["name"], process=state.solution_process + ) + sol_parts["args"] = bind_args(sol_sig, sol_parts["args"]) + except Exception as e: + with debugger(state): + state.report( + "`check_function()` couldn't match the %s call of `%s` to its signature:\n%s " + % (get_ord(index + 1), name, e) + ) + + try: + stu_sig = get_sig( + mapped_name=stu_parts["name"], process=state.student_process + ) + stu_parts["args"] = bind_args(stu_sig, stu_parts["args"]) + except Exception: + state.to_child(highlight=stu_parts["node"]).report( + params_not_matched_msg, fmt_kwargs, append=append_params_not_matched + ) + + # three types of parts: pos_args, keywords, args (e.g. these are bound to sig) + append_message = FeedbackComponent(expand_msg, fmt_kwargs) + child = part_to_child( + stu_parts, sol_parts, append_message, state, node_name="function_calls" + ) + return child diff --git a/pythonwhat/checks/check_has_context.py b/pythonwhat/checks/check_has_context.py new file mode 100644 index 00000000..2a2fef24 --- /dev/null +++ b/pythonwhat/checks/check_has_context.py @@ -0,0 +1,92 @@ +from pythonwhat.Test import EqualTest +from protowhat.Feedback import FeedbackComponent +from protowhat.failure import debugger +from pythonwhat.State import State +from functools import singledispatch +from pythonwhat.checks.check_funcs import check_part_index + +MSG_INCORRECT_LOOP = "Have you used the correct iterator variable names? Was expecting `{{sol_vars}}` but got `{{stu_vars}}`." +MSG_INCORRECT_WITH = "Make sure to use the correct context variable names. Was expecting `{{sol_vars}}` but got `{{stu_vars}}`." + + +def has_context(state, incorrect_msg=None, exact_names=False): + # call _has_context, since the built-in singledispatch can only use 1st pos arg + return _has_context(state, incorrect_msg, exact_names) + + +def _test(state, incorrect_msg, exact_names, tv_name, highlight_name): + # get parts for testing from state + # TODO: this could be rewritten to use check_part_index -> has_equal_part, etc.. + stu_vars = state.student_parts[tv_name] + sol_vars = state.solution_parts[tv_name] + + child_state = state.to_child( + student_ast=state.student_parts.get(highlight_name), + solution_ast=state.solution_parts.get(highlight_name), + ) + + # variables exposed to messages + d = {"stu_vars": stu_vars, "sol_vars": sol_vars, "num_vars": len(sol_vars)} + + if exact_names: + # feedback for wrong iter var names + child_state.do_test( + EqualTest(stu_vars, sol_vars, FeedbackComponent(incorrect_msg, d)) + ) + else: + # feedback for wrong number of iter vars + child_state.do_test( + EqualTest(len(stu_vars), len(sol_vars), FeedbackComponent(incorrect_msg, d)) + ) + + return state + + +@singledispatch +def _has_context(state, incorrect_msg, exact_names): + with debugger(state): + state.report( + "first argument to _has_context must be a State instance or subclass" + ) + + +@_has_context.register(State) +def has_context_state(*args, **kwargs): + return _test(*args, tv_name="target_vars", highlight_name="highlight", **kwargs) + + +@_has_context.register(State.SUBCLASSES["for_loops"]) +@_has_context.register(State.SUBCLASSES["whiles"]) +@_has_context.register(State.SUBCLASSES["dict_comps"]) +@_has_context.register(State.SUBCLASSES["generator_exps"]) +@_has_context.register(State.SUBCLASSES["list_comps"]) +def has_context_loop(state, incorrect_msg, exact_names): + """When dispatched on loops, has_context the target vars are the attribute _target_vars. + + Note: This is to allow people to call has_context on a node (e.g. for_loop) rather than + one of its attributes (e.g. body). Purely for convenience. + """ + return _test( + state, + incorrect_msg or MSG_INCORRECT_LOOP, + exact_names, + tv_name="_target_vars", + highlight_name="target", + ) + + +@_has_context.register(State.SUBCLASSES["withs"]) +def has_context_with(state, incorrect_msg, exact_names): + """When dispatched on with statements, has_context loops over each context manager. + + Note: This is to allow people to call has_context on the with statement, rather than + having to manually loop over each context manager. + + e.g. Ex().check_with(0).has_context() vs Ex().check_with(0).check_context(0).has_context() + """ + + for i in range(len(state.solution_parts["context"])): + ctxt_state = check_part_index(state, "context", i, "{{ordinal}} context") + _has_context(ctxt_state, incorrect_msg or MSG_INCORRECT_WITH, exact_names) + + return state diff --git a/pythonwhat/checks/check_logic.py b/pythonwhat/checks/check_logic.py new file mode 100644 index 00000000..e9b86ab5 --- /dev/null +++ b/pythonwhat/checks/check_logic.py @@ -0,0 +1,312 @@ +from protowhat.Feedback import FeedbackComponent +from protowhat.checks.check_logic import ( + multi, + check_not, + check_or, + check_correct, + disable_highlighting, + fail, +) +from protowhat.failure import InstructorError +import ast + + +multi.__doc__ = ( + str(multi.__doc__) + + """ + :Example: + + Suppose we want to verify the following function call: :: + + round(1.2345, ndigits=2) + + The following SCT would verify this, using ``multi`` to + 'branch out' the state to two sub-SCTs: :: + + Ex().check_function('round').multi( + check_args(0).has_equal_value(), + check_args('ndigits').has_equal_value() + ) + """ +) + + +check_not.__doc__ = ( + str(check_not.__doc__) + + """ + :Example: + The SCT fails with feedback for a specific incorrect value, defined using an override: :: + + Ex().check_object('result').multi( + check_not( + has_equal_value(override=100), + msg='100 is incorrect for reason xyz.' + ), + has_equal_value() + ) + + Notice that ``check_not`` comes before the ``has_equal_value`` test + that checks if the student value is equal to the solution value. + + :Example: + The SCT below runs two ``has_code`` cases: :: + + Ex().check_not( + has_code('mean'), + has_code('median'), + msg='Check your code' + ) + + If students use ``mean`` or ``median`` anywhere in their code, this SCT will fail. + + Note: + - This function is not yet tested with all checks, please report unexpected behaviour. + - This function can be thought as a NOT(x OR y OR ...) statement, since all tests it runs must fail + - This function can be considered a direct counterpart of multi. + + """ +) + + +check_or.__doc__ = ( + str(check_or.__doc__) + + """ + :Example: + + The SCT below tests that the student typed either 'mean' or 'median': :: + + Ex().check_or( + has_code('mean'), + has_code('median') + ) + + If the student didn't type either, the feedback message generated by ``has_code(mean)``, + the first SCT, will be presented to the student. + + """ +) + + +check_correct.__doc__ = ( + str(check_correct.__doc__) + + """ + :Example: + + The SCT below tests whether an object is correct. Only if the object is not correct, will + the function calling checks be executed :: + + Ex().check_correct( + check_object('x').has_equal_value(), + check_function('round').check_args(0).has_equal_value() + ) + + """ +) + + +# utility functions ----------------------------------------------------------- + + +fail.__doc__ = ( + str(fail.__doc__) + + """ + :Example: + + As a trivial SCT example, :: + + Ex().check_for_loop().check_body().fail() + + This can also be helpful for debugging SCTs, as it can be used to stop testing as a given point. + """ +) + + +def override(state, solution): + """Override the solution code with something arbitrary. + + There might be cases in which you want to temporarily override the solution code + so you can allow for alternative ways of solving an exercise. + When you use ``override()`` in an SCT chain, the remainder of that SCT chain will + run as if the solution code you specified is the only code that was in the solution. + + Check the glossary for an example (pandas plotting) + + Args: + solution: solution code as a string that overrides the original solution code. + state: State instance describing student and solution code. Can be omitted if used with Ex(). + """ + + # the old ast may be a number of node types, but generally either a + # (1) ast.Module, or for single expressions... + # (2) whatever was grabbed using module.body[0] + # (3) module.body[0].value, when module.body[0] is an Expr node + old_ast = state.solution_ast + new_ast = ast.parse(solution) + if not isinstance(old_ast, ast.Module) and len(new_ast.body) == 1: + expr = new_ast.body[0] + candidates = [expr, expr.value] if isinstance(expr, ast.Expr) else [expr] + for node in candidates: + if isinstance(node, old_ast.__class__): + new_ast = node + break + + kwargs = state.feedback_context.kwargs if state.feedback_context else {} + child = state.to_child( + solution_ast=new_ast, + student_ast=state.student_ast, + highlight=state.highlight, + append_message=FeedbackComponent("", kwargs), + ) + + return child + + +def set_context(state, *args, **kwargs): + """Update context values for student and solution environments. + + When ``has_equal_x()`` is used after this, the context values (in ``for`` loops and function definitions, for example) + will have the values specified through his function. It is the function equivalent of the ``context_vals`` argument of + the ``has_equal_x()`` functions. + + - Note 1: excess args and unmatched kwargs will be unused in the student environment. + - Note 2: When you try to set context values that don't match any target variables in the solution code, + ``set_context()`` raises an exception that lists the ones available. + - Note 3: positional arguments are more robust to the student using different names for context values. + - Note 4: You have to specify arguments either by position, either by name. A combination is not possible. + + :Example: + + Solution code:: + + total = 0 + for i in range(10): + print(i ** 2) + + Student submission that will pass (different iterator, different calculation):: + + total = 0 + for j in range(10): + print(j * j) + + SCT:: + + # set_context is robust against different names of context values. + Ex().check_for_loop().check_body().multi( + set_context(1).has_equal_output(), + set_context(2).has_equal_output(), + set_context(3).has_equal_output() + ) + + # equivalent SCT, by setting context_vals in has_equal_output() + Ex().check_for_loop().check_body().\\ + multi([s.has_equal_output(context_vals=[i]) for i in range(1, 4)]) + + """ + + stu_crnt = state.student_context.context + sol_crnt = state.solution_context.context + + # for now, you can't specify both + if len(args) > 0 and len(kwargs) > 0: + raise InstructorError.from_message( + "In `set_context()`, specify arguments either by position, either by name." + ) + + # set args specified by pos ----------------------------------------------- + if args: + # stop if too many pos args for solution + if len(args) > len(sol_crnt): + raise InstructorError.from_message( + "Too many positional args. There are {} context vals, but tried to set {}".format( + len(sol_crnt), len(args) + ) + ) + # set pos args + upd_sol = sol_crnt.update(dict(zip(sol_crnt.keys(), args))) + upd_stu = stu_crnt.update(dict(zip(stu_crnt.keys(), args))) + else: + upd_sol = sol_crnt + upd_stu = stu_crnt + + # set args specified by keyword ------------------------------------------- + if kwargs: + # stop if keywords don't match with solution + if set(kwargs) - set(upd_sol): + raise InstructorError.from_message( + "`set_context()` failed: context val names are {}, but you tried to set {}.".format( + upd_sol or "missing", sorted(list(kwargs.keys())) + ) + ) + out_sol = upd_sol.update(kwargs) + # need to match keys in kwargs with corresponding keys in stu context + # in case they used, e.g., different loop variable names + match_keys = dict(zip(sol_crnt.keys(), stu_crnt.keys())) + out_stu = upd_stu.update( + {match_keys[k]: v for k, v in kwargs.items() if k in match_keys} + ) + else: + out_sol = upd_sol + out_stu = upd_stu + + return state.to_child( + student_context=out_stu, solution_context=out_sol, highlight=state.highlight + ) + + +def set_env(state, **kwargs): + """Update/set environemnt variables for student and solution environments. + + When ``has_equal_x()`` is used after this, the variables specified through this function will + be available in the student and solution process. Note that you will not see these variables + in the student process of the state produced by this function: the values are saved on the state + and are only added to the student and solution processes when ``has_equal_ast()`` is called. + + :Example: + + Student and Solution Code:: + + a = 1 + if a > 4: + print('pretty large') + + SCT:: + + # check if condition works with different values of a + Ex().check_if_else().check_test().multi( + set_env(a = 3).has_equal_value(), + set_env(a = 4).has_equal_value(), + set_env(a = 5).has_equal_value() + ) + + # equivalent SCT, by setting extra_env in has_equal_value() + Ex().check_if_else().check_test().\\ + multi([has_equal_value(extra_env={'a': i}) for i in range(3, 6)]) + """ + + stu_crnt = state.student_env.context + sol_crnt = state.solution_env.context + + stu_new = stu_crnt.update(kwargs) + sol_new = sol_crnt.update(kwargs) + + return state.to_child( + student_env=stu_new, solution_env=sol_new, highlight=state.highlight + ) + + +disable_highlighting.__doc__ = ( + str(disable_highlighting.__doc__) + + """ + :Examples: + + SCT that will mark the 'number' portion if it is incorrect:: + + Ex().check_function('round').check_args(0).has_equal_ast() + + SCT chains that will not mark certain mistakes. The earlier you put the function, the more types of mistakes will no longer be highlighted:: + + Ex().disable_highlighting().check_function('round').check_args(0).has_equal_ast() + Ex().check_function('round').disable_highlighting().check_args(0).has_equal_ast() + Ex().check_function('round').check_args(0).disable_highlighting().has_equal_ast() + """ +) diff --git a/pythonwhat/checks/check_object.py b/pythonwhat/checks/check_object.py new file mode 100644 index 00000000..2b0b8856 --- /dev/null +++ b/pythonwhat/checks/check_object.py @@ -0,0 +1,372 @@ +from pythonwhat.parsing import ObjectAssignmentParser +from pythonwhat.Test import ( + DefinedProcessTest, + InstanceProcessTest, + DefinedCollProcessTest, +) +from protowhat.Feedback import FeedbackComponent +from protowhat.failure import InstructorError +from pythonwhat.tasks import ( + isDefinedInProcess, + isInstanceInProcess, + isDefinedCollInProcess, +) +from pythonwhat.checks.check_funcs import part_to_child +from pythonwhat.utils import v2_only +import ast + + +def check_object(state, index, missing_msg=None, expand_msg=None, typestr="variable"): + """Check object existence (and equality) + + Check whether an object is defined in the student's process, and zoom in on its value in both + student and solution process to inspect quality (with has_equal_value(). + + In ``pythonbackend``, both the student's submission as well as the solution code are executed, in separate processes. + ``check_object()`` looks at these processes and checks if the referenced object is available in the student process. + Next, you can use ``has_equal_value()`` to check whether the objects in the student and solution process correspond. + + Args: + index (str): the name of the object which value has to be checked. + missing_msg (str): feedback message when the object is not defined in the student process. + expand_msg (str): If specified, this overrides any messages that are prepended by previous SCT chains. + + :Example: + + Suppose you want the student to create a variable ``x``, equal to 15: :: + + x = 15 + + The following SCT will verify this: :: + + Ex().check_object("x").has_equal_value() + + - ``check_object()`` will check if the variable ``x`` is defined in the student process. + - ``has_equal_value()`` will check whether the value of ``x`` in the solution process is the same as in the student process. + + Note that ``has_equal_value()`` only looks at **end result** of a variable in the student process. + In the example, how the object ``x`` came about in the student's submission, does not matter. + This means that all of the following submission will also pass the above SCT: :: + + x = 15 + x = 12 + 3 + x = 3; x += 12 + + :Example: + + As the previous example mentioned, ``has_equal_value()`` only looks at the **end result**. If your exercise is + first initializing and object and further down the script is updating the object, you can only look at the final value! + + Suppose you want the student to initialize and populate a list `my_list` as follows: :: + + my_list = [] + for i in range(20): + if i % 3 == 0: + my_list.append(i) + + There is no robust way to verify whether `my_list = [0]` was coded correctly in a separate way. + The best SCT would look something like this: :: + + msg = "Have you correctly initialized `my_list`?" + Ex().check_correct( + check_object('my_list').has_equal_value(), + multi( + # check initialization: [] or list() + check_or( + has_equal_ast(code = "[]", incorrect_msg = msg), + check_function('list') + ), + check_for_loop().multi( + check_iter().has_equal_value(), + check_body().check_if_else().multi( + check_test().multi( + set_context(2).has_equal_value(), + set_context(3).has_equal_value() + ), + check_body().set_context(3).\\ + set_env(my_list = [0]).\\ + has_equal_value(name = 'my_list') + ) + ) + ) + ) + + - ``check_correct()`` is used to robustly check whether ``my_list`` was built correctly. + - If ``my_list`` is not correct, **both** the initialization and the population code are checked. + + :Example: + + Because checking object correctness incorrectly is such a common misconception, we're adding another example: :: + + import pandas as pd + df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}) + df['c'] = [7, 8, 9] + + The following SCT would be **wrong**, as it does not factor in the possibility that the 'add column ``c``' step could've been wrong: :: + + Ex().check_correct( + check_object('df').has_equal_value(), + check_function('pandas.DataFrame').check_args(0).has_equal_value() + ) + + The following SCT would be better, as it is specific to the steps: :: + + # verify the df = pd.DataFrame(...) step + Ex().check_correct( + check_df('df').multi( + check_keys('a').has_equal_value(), + check_keys('b').has_equal_value() + ), + check_function('pandas.DataFrame').check_args(0).has_equal_value() + ) + + # verify the df['c'] = [...] step + Ex().check_df('df').check_keys('c').has_equal_value() + + :Example: + + pythonwhat compares the objects in the student and solution process with the ``==`` operator. + For basic objects, this ``==`` is operator is properly implemented, so that the objects can be effectively compared. + For more complex objects that are produced by third-party packages, however, it's possible that this equality operator is not implemented in a way you'd expect. + Often, for these object types the ``==`` will compare the actual object instances: :: + + # pre exercise code + class Number(): + def __init__(self, n): + self.n = n + + # solution + x = Number(1) + + # sct that won't work + Ex().check_object().has_equal_value() + + # sct + Ex().check_object().has_equal_value(expr_code = 'x.n') + + # submissions that will pass this sct + x = Number(1) + x = Number(2 - 1) + + The basic SCT like in the previous example will notwork here. + Notice how we used the ``expr_code`` argument to _override_ which value `has_equal_value()` is checking. + Instead of checking whether `x` corresponds between student and solution process, it's now executing the expression ``x.n`` + and seeing if the result of running this expression in both student and solution process match. + + """ + + # Only do the assertion if PYTHONWHAT_V2_ONLY is set to '1' + if v2_only(): + extra_msg = "If you want to check the value of an object in e.g. a for loop, use `has_equal_value(name = 'my_obj')` instead." + state.assert_execution_root("check_object", extra_msg=extra_msg) + + if missing_msg is None: + missing_msg = "Did you define the {{typestr}} `{{index}}` without errors?" + + if expand_msg is None: + expand_msg = "Did you correctly define the {{typestr}} `{{index}}`? " + + if ( + not isDefinedInProcess(index, state.solution_process) + and state.has_different_processes() + ): + raise InstructorError.from_message( + "`check_object()` couldn't find object `%s` in the solution process." + % index + ) + + append_message = FeedbackComponent(expand_msg, {"index": index, "typestr": typestr}) + + # create child state, using either parser output, or create part from name + fallback = lambda: ObjectAssignmentParser.get_part(index) + stu_part = state.ast_dispatcher.find("object_assignments", state.student_ast).get( + index, fallback() + ) + sol_part = state.ast_dispatcher.find("object_assignments", state.solution_ast).get( + index, fallback() + ) + + # test object exists + state.do_test( + DefinedProcessTest( + index, + state.student_process, + FeedbackComponent(missing_msg, append_message.kwargs), + ) + ) + + child = part_to_child( + stu_part, sol_part, append_message, state, node_name="object_assignments" + ) + + return child + + +def is_instance(state, inst, not_instance_msg=None): + """Check whether an object is an instance of a certain class. + + ``is_instance()`` can currently only be used when chained from ``check_object()``, the function that is + used to 'zoom in' on the object of interest. + + Args: + inst (class): The class that the object should have. + not_instance_msg (str): When specified, this overrides the automatically generated message in case + the object does not have the expected class. + state (State): The state that is passed in through the SCT chain (don't specify this). + + :Example: + + Student code and solution code:: + + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + + SCT:: + + # Verify the class of arr + import numpy + Ex().check_object('arr').is_instance(numpy.ndarray) + """ + + state.assert_is(["object_assignments"], "is_instance", ["check_object"]) + + sol_name = state.solution_parts.get("name") + stu_name = state.student_parts.get("name") + + if not_instance_msg is None: + not_instance_msg = "Is it a {{inst.__name__}}?" + + if not isInstanceInProcess(sol_name, inst, state.solution_process): + raise InstructorError.from_message( + "`is_instance()` noticed that `%s` is not a `%s` in the solution process." + % (sol_name, inst.__name__) + ) + + feedback = FeedbackComponent(not_instance_msg, {"inst": inst}) + state.do_test(InstanceProcessTest(stu_name, inst, state.student_process, feedback)) + + return state + + +def check_df(state, index, missing_msg=None, not_instance_msg=None, expand_msg=None): + """Check whether a DataFrame was defined and it is the right type + + ``check_df()`` is a combo of ``check_object()`` and ``is_instance()`` that checks whether the specified object exists + and whether the specified object is pandas DataFrame. + + You can continue checking the data frame with ``check_keys()`` function to 'zoom in' on a particular column in the pandas DataFrame: + + Args: + index (str): Name of the data frame to zoom in on. + missing_msg (str): See ``check_object()``. + not_instance_msg (str): See ``is_instance()``. + expand_msg (str): If specified, this overrides any messages that are prepended by previous SCT chains. + + :Example: + + Suppose you want the student to create a DataFrame ``my_df`` with two columns. + The column ``a`` should contain the numbers 1 to 3, + while the contents of column ``b`` can be anything: :: + + import pandas as pd + my_df = pd.DataFrame({"a": [1, 2, 3], "b": ["a", "n", "y"]}) + + The following SCT would robustly check that: :: + + Ex().check_df("my_df").multi( + check_keys("a").has_equal_value(), + check_keys("b") + ) + + - ``check_df()`` checks if ``my_df`` exists (``check_object()`` behind the scenes) and is a DataFrame (``is_instance()``) + - ``check_keys("a")`` zooms in on the column ``a`` of the data frame, and ``has_equal_value()`` checks if the columns correspond between student and solution process. + - ``check_keys("b")`` zooms in on hte column ``b`` of the data frame, but there's no 'equality checking' happening + + The following submissions would pass the SCT above: :: + + my_df = pd.DataFrame({"a": [1, 1 + 1, 3], "b": ["a", "l", "l"]}) + my_df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6], "c": [7, 8, 9]}) + + """ + import pandas as pd + child = check_object( + state, + index, + missing_msg=missing_msg, + expand_msg=expand_msg, + typestr="pandas DataFrame", + ) + is_instance(child, pd.DataFrame, not_instance_msg=not_instance_msg) + return child + + +def check_keys(state, key, missing_msg=None, expand_msg=None): + """Check whether an object (dict, DataFrame, etc) has a key. + + ``check_keys()`` can currently only be used when chained from ``check_object()``, the function that is + used to 'zoom in' on the object of interest. + + Args: + key (str): Name of the key that the object should have. + missing_msg (str): When specified, this overrides the automatically generated + message in case the key does not exist. + expand_msg (str): If specified, this overrides any messages that are prepended by previous SCT chains. + state (State): The state that is passed in through the SCT chain (don't specify this). + + :Example: + + Student code and solution code:: + + x = {'a': 2} + + SCT:: + + # Verify that x contains a key a + Ex().check_object('x').check_keys('a') + + # Verify that x contains a key a and a is correct. + Ex().check_object('x').check_keys('a').has_equal_value() + + """ + + state.assert_is(["object_assignments"], "is_instance", ["check_object", "check_df"]) + + if missing_msg is None: + missing_msg = "There is no {{ 'column' if 'DataFrame' in parent.typestr else 'key' }} `'{{key}}'`." + if expand_msg is None: + expand_msg = "Did you correctly set the {{ 'column' if 'DataFrame' in parent.typestr else 'key' }} `'{{key}}'`? " + + sol_name = state.solution_parts.get("name") + stu_name = state.student_parts.get("name") + + if not isDefinedCollInProcess(sol_name, key, state.solution_process): + raise InstructorError.from_message( + "`check_keys()` couldn't find key `%s` in object `%s` in the solution process." + % (key, sol_name) + ) + + # check if key available + state.do_test( + DefinedCollProcessTest( + stu_name, key, state.student_process, FeedbackComponent(missing_msg, {"key": key}) + ) + ) + + def get_part(name, key, highlight): + if isinstance(key, str): + slice_val = ast.Constant(value=key) + else: + slice_val = ast.parse(str(key)).body[0].value + expr = ast.Subscript( + value=ast.Name(id=name, ctx=ast.Load()), + slice=ast.Index(value=slice_val), + ctx=ast.Load(), + ) + ast.fix_missing_locations(expr) + return {"node": expr, "highlight": highlight} + + stu_part = get_part(stu_name, key, state.student_parts.get("highlight")) + sol_part = get_part(sol_name, key, state.solution_parts.get("highlight")) + append_message = FeedbackComponent(expand_msg, {"key": key}) + child = part_to_child(stu_part, sol_part, append_message, state) + return child diff --git a/pythonwhat/checks/check_wrappers.py b/pythonwhat/checks/check_wrappers.py new file mode 100644 index 00000000..f10067a9 --- /dev/null +++ b/pythonwhat/checks/check_wrappers.py @@ -0,0 +1,799 @@ +from protowhat.failure import _debug +from protowhat.checks.check_simple import allow_errors +from protowhat.checks.check_bash_history import has_command +from protowhat.checks.check_files import check_file, has_dir +from pythonwhat.checks.check_funcs import check_part, check_part_index, check_node +from pythonwhat.checks.has_funcs import has_equal_part +from pythonwhat.checks.check_function import check_function +from pythonwhat.checks.check_has_context import has_context +from pythonwhat.checks import check_object, check_logic, check_funcs, has_funcs +from pythonwhat.local import run + +from inspect import signature, Parameter +from functools import partial, wraps +from jinja2 import Template + +__PART_WRAPPERS__ = { + "iter": "iterable part", + "body": "body", + "key": "key part", + "value": "value part", + "orelse": "else part", + "finalbody": "finally part", + "test": "condition", +} + +__PART_INDEX_WRAPPERS__ = { + "ifs": "{{ordinal}} if", + "bases": "{{ordinal}} base class", + "handlers": "`{{index}}` `except` block", + "context": "{{ordinal}} context", +} + +__NODE_WRAPPERS__ = { + "list_comp": { + "typestr": "{{ordinal}} list comprehension", + "docstr": """Check whether a list comprehension was coded and zoom in on it. + + Can be chained with ``check_iter()``, ``check_body()``, and ``check_ifs()``. + + Args: + index: Index of the list comprehension (0-based) + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you expect students to create a list ``my_list`` as follows: :: + + my_list = [ i*2 for i in range(0,10) if i>2 ] + + The following SCT would robustly verify this: :: + + Ex().check_correct( + check_object('my_list').has_equal_value(), + check_list_comp().multi( + check_iter().has_equal_value(), + check_body().set_context(4).has_equal_value(), + check_ifs(0).multi( + set_context(0).has_equal_value(), + set_context(3).has_equal_value(), + set_context(5).has_equal_value() + ) + ) + ) + + - With ``check_correct()``, we're making sure that the list comprehension + checking is not executed if ``my_list`` was calculated properly. + - If ``my_list`` is not correct, the 'diagnose' chain will run: ``check_list_comp()`` looks + for the first list comprehension in the student's submission. + - Next, ``check_iter()`` zooms in on the iterator, ``range(0, 10)`` in the case of the solution. + ``has_equal_value()`` verifies whether the expression that the student used evaluates to the + same value as the expression that the solution used. + - ``check_body()`` zooms in on the body, ``i*2`` in the case of the solution. + ``set_context()`` sets the iterator to 4, allowing for the fact that the student used another name instead of ``i`` for this iterator. + ``has_equal_value()`` reruns the body in the student and solution code with the iterator set to 4, and checks if the results are the same. + - ``check_ifs(0)`` zooms in on the first ``if`` of the list comprehension, ``i>2`` in case of the solution. + With a series of ``set_context()`` and ``has_equal_value()``, it is verifies whether this condition evaluates to the same value in student + and solution code for different values of the iterator (`i` in the case of the solution, whatever in the case of the student). + + """, + }, + "generator_exp": { + "typestr": "{{ordinal}} generator expression", + "docstr": """Check whether a generator expression was coded and zoom in on it. + + Can be chained with ``check_iter()``, ``check_body()``, and ``check_ifs()``. + + Args: + index: Index of the generator expression (0-based) + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you expect students to create a generator ``my_gen`` as follows: :: + + my_gen = ( i*2 for i in range(0,10) ) + + The following SCT would robustly verify this: :: + + Ex().check_correct( + check_object('my_gen').has_equal_value(), + check_generator_exp().multi( + check_iter().has_equal_value(), + check_body().set_context(4).has_equal_value() + ) + ) + + Have a look at ``check_list_comp`` to understand what's going on; it is very similar. + + """, + }, + "dict_comp": { + "typestr": "{{ordinal}} dictionary comprehension", + "docstr": """Check whether a dictionary comprehension was coded and zoom in on it. + + Can be chained with ``check_key()``, ``check_value()``, and ``check_ifs()``. + + Args: + index: Index of the dictionary comprehension (0-based) + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you expect students to create a dictionary ``my_dict`` as follows: :: + + my_dict = { m:len(m) for m in ['a', 'ab', 'abc'] } + + The following SCT would robustly verify this: :: + + Ex().check_correct( + check_object('my_dict').has_equal_value(), + check_dict_comp().multi( + check_iter().has_equal_value(), + check_key().set_context('ab').has_equal_value(), + check_value().set_context('ab').has_equal_value() + ) + ) + + - With ``check_correct()``, we're making sure that the dictionary comprehension + checking is not executed if ``my_dict`` was created properly. + - If ``my_dict`` is not correct, the 'diagnose' chain will run: ``check_dict_comp()`` looks + for the first dictionary comprehension in the student's submission. + - Next, ``check_iter()`` zooms in on the iterator, ``['a', 'ab', 'abc']`` in the case of the solution. + ``has_equal_value()`` verifies whether the expression that the student used evaluates to the + same value as the expression that the solution used. + - ``check_key()`` zooms in on the key of the comprehension, ``m`` in the case of the solution. + ``set_context()`` temporaritly sets the iterator to ``'ab'``, allowing for the fact that the student used another name instead of ``m`` for this iterator. + ``has_equal_value()`` reruns the key expression in the student and solution code with the iterator set to ``'ab'``, and checks if the results are the same. + - ``check_value()`` zooms in on the value of the comprehension, ``len(m)`` in the case of the solution. + ``has_equal_value()`` reruns the value expression in the student and solution code with the iterator set to ``'ab'``, and checks if the results are the same. + + """, + }, + "for_loop": { + "typestr": "{{ordinal}} for loop", + "docstr": """Check whether a for loop was coded and zoom in on it. + + Can be chained with ``check_iter()`` and ``check_body()``. + + Args: + index: Index of the for loop (0-based). + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want a student to iterate over a predefined dictionary ``my_dict`` and do the appropriate printouts: :: + + for key, value in my_dict.items(): + print(key + " - " + str(value)) + + The following SCT would verify this: :: + + Ex().check_for_loop().multi( + check_iter().has_equal_value(), + check_body().multi( + set_context('a', 1).has_equal_output(), + set_context('b', 2).has_equal_output() + ) + ) + + - ``check_for_loop()`` zooms in on the ``for`` loop, and makes its parts available for further checking. + - ``check_iter()`` zooms in on the iterator part of the for loop, ``my_dict.items()`` in the solution. + ``has_equal_value()`` re-executes the expressions specified by student and solution and compares their results. + - ``check_body()`` zooms in on the body part of the for loop, ``print(key + " - " + str(value))``. + For different values of ``key`` and ``value``, the student's body and solution's body are executed again and the printouts are captured and compared to see if they are equal. + + Notice how you do not need to specify the variables by name in ``set_context()``. pythonwhat can figure out the variable names used in both student and solution code, and + can do the verification independent of that. That way, we can make the SCT robust against submissions that code the correct logic, but use different names for the context values. + In other words, the following student submissions that would also pass the SCT: :: + + # passing submission 1 + my_dict = {'a': 1, 'b': 2} + for k, v in my_dict.items(): + print(k + " - " + str(v)) + + # passing submission 2 + my_dict = {'a': 1, 'b': 2} + for first, second in my_dict.items(): + mess = first + " - " + str(second) + print(mess) + + :Example: + + As another example, suppose you want the student to build a list of doubles as follows: :: + + even = [] + for i in range(10): + even.append(2*i) + + The following SCT would robustly verify this: :: + + Ex().check_correct( + check_object('even').has_equal_value(), + check_for_loop().multi( + check_iter().has_equal_value(), + check_body().set_context(2).set_env(even = []).\\ + has_equal_value(name = 'even') + ) + ) + + - ``check_correct()`` makes sure that we do not dive into the ``for`` loop if the array ``even`` is correctly populated in the end. + - If ``even`` was not correctly populated, ``check_for_loop()`` will zoom in on the for loop. + - The ``check_iter()`` chain verifies whether `range(10)` (or something equivalent) was used to iterate over. + - ``check_body()`` zooms in on the body, and reruns the body (``even.append(2*i)`` in the solution) for ``i`` equal to 2, and even temporarily set to an empty array. + Notice how we use ``set_context()`` to robustly set the context value (the student can use a different variable name), while we have to explicitly set ``even`` with ``set_env()``. + Also notice how we use ``has_equal_value(name = 'even')`` instead of the usual ``check_object()``; ``check_object()`` can only be called from the root state ``Ex()``. + + :Example: + + As a follow-up example, suppose you want the student to build a list of doubles of the even numbers only: :: + + even = [] + for i in range(10): + if i % 2 == 0: + even.append(2*i) + + The following SCT would robustly verify this: :: + + Ex().check_correct( + check_object('even').has_equal_value(), + check_for_loop().multi( + check_iter().has_equal_value(), + check_body().check_if_else().multi( + check_test().multi( + set_context(1).has_equal_value(), + set_context(2).has_equal_value() + ), + check_body().set_context(2).\\ + set_env(even = []).has_equal_value(name = 'even') + ) + ) + ) + + """, + }, + "function_def": { + "typestr": "definition of `{{index}}()`", + "docstr": """Check whether a function was defined and zoom in on it. + + Can be chained with ``check_call()``, ``check_args()`` and ``check_body()``. + + Args: + index: the name of the function definition. + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want a student to create a function ``shout_echo()``: :: + + def shout_echo(word1, echo=1): + echo_word = word1 * echo + shout_words = echo_word + '!!!' + return shout_words + + The following SCT robustly checks this: :: + + Ex().check_function_def('shout_echo').check_correct( + multi( + check_call("f('hey', 3)").has_equal_value(), + check_call("f('hi', 2)").has_equal_value(), + check_call("f('hi')").has_equal_value() + ), + check_body().set_context('test', 1).multi( + has_equal_value(name = 'echo_word'), + has_equal_value(name = 'shout_words') + ) + ) + + Here: + + - ``check_function_def()`` zooms in on the function definition of ``shout_echo`` in both student and solution code (and process). + - ``check_correct()`` is used to + + + First check whether the function gives the correct result when called in different ways (through ``check_call()``). + + Only if these 'function unit tests' don't pass, ``check_correct()`` will run the `check_body()` chain that dives deeper into the + function definition body. This chain sets the context variables - ``word1`` and ``echo``, the arguments of the function - to + the values ``'test'`` and ``1`` respectively, again while being agnostic to the actual name of these context variables. + + Notice how ``check_correct()`` is used to great effect here: why check the function definition internals if the I/O of the function works fine? + Because of this construct, all the following submissions will pass the SCT: :: + + # passing submission 1 + def shout_echo(w, e=1): + ew = w * e + return ew + '!!!' + + # passing submission 2 + def shout_echo(a, b=1): + return a * b + '!!!' + + :Example: + + ``check_args()`` is most commonly used in combination with ``check_function()`` + to verify the arguments of function **calls**, but it can also be used + to verify the arguments specified in the signature of a function definition. + + We can extend the SCT for the previous example to explicitly verify the signature: :: + + + msg1 = "Make sure to specify 2 arguments!" + msg2 = "don't specify default arg!" + msg3 = "specify a default arg!" + Ex().check_function_def('shout_echo').check_correct( + multi( + check_call("f('hey', 3)").has_equal_value(), + check_call("f('hi', 2)").has_equal_value(), + check_call("f('hi')").has_equal_value() + ), + multi( + has_equal_part_len("args", unequal_msg=1), + check_args(0).has_equal_part('is_default', msg=msg2), + check_args('word1').has_equal_part('is_default', msg=msg2), + check_args(1).\\ + has_equal_part('is_default', msg=msg3).has_equal_value(), + check_args('echo').\\ + has_equal_part('is_default', msg=msg3).has_equal_value(), + check_body().set_context('test', 1).multi( + has_equal_value(name = 'echo_word'), + has_equal_value(name = 'shout_words') + ) + ) + ) + + - ``has_equal_part_len("args")`` verifies whether student and solution function + definition have the same number of arguments. + - ``check_args(0)`` refers to the first argument in the signature by position, + and the chain checks whether the student did not specify a default as in the solution. + - An alternative for the ``check_args(0)`` chain is to use ``check_args('word1')`` + to refer to the first argument. This is more restrictive, as the requires the + student to use the exact same name. + - ``check_args(1)`` refers to the second argument in the signature by position, + and the chain checks whether the student specified a default, as in the solution, and + whether the value of this default corresponds to the one in the solution. + - The ``check_args('echo')`` chain is a more restrictive alternative for the ``check_args(1)`` + chain. + + Notice that support for verifying arguments is not great yet: + + - A lot of work is needed to verify the number of arguments and whether or not defaults are set. + - You have to specify custom messages because pythonwhat doesn't automatically generate messages. + + We are working on it! + + """, + }, + "class_def": { + "typestr": "class definition of `{{index}}`", + "docstr": """Check whether a class was defined and zoom in on its definition + + Can be chained with ``check_bases()`` and ``check_body()``. + + Args: + index: the name of the function definition. + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want to check whether a class was defined correctly: :: + + class MyInt(int): + def __init__(self, i): + super().__init__(i + 1) + + The following SCT would verify this: :: + + Ex().check_class_def('MyInt').multi( + check_bases(0).has_equal_ast(), + check_body().check_function_def('__init__').multi( + check_args('self'), + check_args('i'), + check_body().set_context(i = 2).multi( + check_function('super', signature=False), + check_function('super.__init__').check_args(0).has_equal_value() + ) + ) + ) + + - ``check_class_def()`` looks for the class definition itself. + - With ``check_bases()``, you can zoom in on the different basse classes that the class definition inherits from. + - With ``check_body()``, you zoom in on the class body, after which you can use other functions such + as ``check_function_def()`` to look for class methods. + - Of course, just like for other examples, you can use ``check_correct()`` where necessary, + e.g. to verify whether class methods give the right behavior with ``check_call()`` + before diving into the body of the method itself. + + """, + }, + "if_exp": { + "typestr": "{{ordinal}} if expression", + "docstr": """Check whether an if expression was coded zoom in on it. + + This function works the exact same way as ``check_if_else()``. + """, + }, + "if_else": { + "typestr": "{{ordinal}} if statement", + "docstr": """Check whether an if statement was coded zoom in on it. + + Args: + index: the index of the if statement to look for (0 based) + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want students to print out a message if ``x`` is larger than 0: :: + + x = 4 + if x > 0: + print("x is strictly positive") + + The following SCT would verify that: :: + + Ex().check_if_else().multi( + check_test().multi( + set_env(x = -1).has_equal_value(), + set_env(x = 1).has_equal_value(), + set_env(x = 0).has_equal_value() + ), + check_body().check_function('print', 0).\\ + check_args('value').has_equal_value() + ) + + - ``check_if_else()`` zooms in on the first if statement in the student and solution submission. + - ``check_test()`` zooms in on the 'test' portion of the if statement, ``x > 0`` in case of the solution. + ``has_equal_value()`` reruns this expression and the corresponding expression in the student code for + different values of ``x`` (set with ``set_env()``) and compare there results. + This way, you can robustly verify whether the if test was coded up correctly. If the student + codes up the condition as ``0 < x``, this would also be accepted. + - ``check_body()`` zooms in on the 'body' portion of the if statement, ``print("...")`` in case of the solution. + With a classical ``check_function()`` chain, it is verified whether the if statement contains a + function ``print()`` and whether its argument is set correctly. + + :Example: + + In Python, when an if-else statement has an ``elif`` clause, it is held in the `orelse` part. + In this sense, an if-elif-else statement is represented by python as nested if-elses. + More specifically, this if-else statement: :: + + if x > 0: + print(x) + elif y > 0: + print(y) + else: + print('none') + + Is syntactically equivalent to: :: + + if x > 0: + print(x) + else: + if y > 0: + print(y) + else: + print('none') + + The second representation has to be followed when writing the corresponding SCT: :: + + Ex().check_if_else().multi( + check_test(), # zoom in on x > 0 + check_body(), # zoom in on print(x) + check_orelse().check_if_else().multi( + check_test(), # zoom in on y > 0 + check_body(), # zoom in on print(y) + check_orelse() # zoom in on print('none') + ) + ) + + """, + }, + "lambda_function": { + "typestr": "{{ordinal}} lambda function", + "docstr": """Check whether a lambda function was coded zoom in on it. + + Can be chained with ``check_call()``, ``check_args()`` and ``check_body()``. + + Args: + index: the index of the lambda function (0-based). + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want a student to create a lambda function + that returns the length of an array times two: :: + + lambda x: len(x)*2 + + The following SCT robustly checks this: :: + + Ex().check_lambda_function().check_correct( + multi( + check_call("f([1])").has_equal_value(), + check_call("f([1, 2])").has_equal_value() + ), + check_body().set_context([1, 2, 3]).has_equal_value() + ) + + Here: + + - ``check_lambda_function()`` zooms in on the first lambda function in both student and solution code. + - ``check_correct()`` is used to + + + First check whether the lambda function gives the correct result when called in different ways (through ``check_call()``). + + Only if these 'function unit tests' don't pass, ``check_correct()`` will run the `check_body()` chain that dives deeper into the + lambda function's body. This chain sets the context variable `x`, the argument of the function, to + the values ``[1, 2, 3]``, while being agnostic to the actual name the student used for this context variable. + + Notice how ``check_correct()`` is used to great effect here: why check the function definition internals if the I/O of the function works fine? + Because of this construct, all the following submissions will pass the SCT: :: + + # passing submission 1 + lambda x: len(x) + len(x) + + # passing submission 2 + lambda y, times=2: len(y) * times + """, + }, + "try_except": { + "typestr": "{{ordinal}} try statement", + "docstr": """Check whether a try except statement was coded zoom in on it. + + Can be chained with ``check_body()``, ``check_handlers()``, ``check_orelse()`` and ``check_finalbody()``. + + Args: + index: the index of the try except statement (0-based). + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want to verify whether the student did a `try-except` statement properly: :: + + do_dangerous_thing = lambda n: n + + try: + x = do_dangerous_thing(n = 4) + except ValueError as e: + x = 'something wrong with inputs' + except: + x = 'something went wrong' + finally: + print('ciao!') + + The following SCT can be used to verify this: :: + + Ex().check_try_except().multi( + check_body().\\ + check_function('do_dangerous_thing').\\ + check_args('n').has_equal_value(), + check_handlers('ValueError').\\ + has_equal_value(name = 'x'), + check_handlers('all').\\ + has_equal_value(name = 'x'), + check_finalbody().\\ + check_function('print').check_args(0).has_equal_value() + ) + + """, + }, + "while": { + "typestr": "{{ordinal}} `while` loop", + "docstr": """Check whether a while loop was coded and zoom in on it. + + Can be chained with ``check_test()``, ``check_body()`` and ``check_orelse()``. + + Args: + index: the index of the while loop to verify (0-based). + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + :Example: + + Suppose you want a student to code a while loop that counts down a counter from 50 until + a multilpe of 11 is found. If it is found, the value should be printed out. :: + + i = 50 + while i % 11 != 0: + i -= 1 + + The following SCT robustly verifies this: :: + + Ex().check_correct( + check_object('i').has_equal_value(), + check_while().multi( + check_test().multi( + set_env(i = 45).has_equal_value(), + set_env(i = 44).has_equal_value() + ), + check_body().set_env(i = 3).has_equal_value(name = 'i') + ) + ) + + - ``check_correct()`` first checks whether the end result of ``i`` is correct. If it is, the entire chain that checks the ``while`` loop is skipped. + - If ``i`` is not correctly calculated, ``check_while_loop()`` zooms in on the while loop. + - ``check_test()`` zooms in on the condition of the ``while`` loop, ``i % 11 != 0`` in the solution, and verifies whether + the expression gives the same results for different values of ``i``, set through ``set_env()``, when comparing student and solution. + - ``check_body()`` zooms in on the body of the ``while`` loop, and ``has_equal_value()`` checks whether rerunning this body + updates ``i`` as expected when ``i`` is temporarily set to 3 with ``set_env()``. + + """, + }, + "with": { + "typestr": "{{ordinal}} `with` statement", + "docstr": """Check whether a with statement was coded zoom in on it. + + Args: + index: the index of the``with`` statement to verify (0-based) + {{typestr}} + {{missing_msg}} + {{expand_msg}} + + """, + }, +} + +scts = dict() + +# make has_equal_part wrappers + + +def partial_with_offset(offset=1): + def bound_partial_with_offset(func, *partial_args, **partial_kwargs): + kwargs_partial = partial(func, **partial_kwargs) + + @wraps(func) + def full_partial(*args, **kwargs): + full_args = args[:offset] + partial_args + args[offset:] + return kwargs_partial(*full_args, **kwargs) + + # set correct signature of returned partial + # todo: pass arguments as keywords to partial, instead of this decorator? + # (where args are always the same) + func_sig = signature(full_partial) + parameter_names = tuple(func_sig.parameters) + + partialed_positional_indices = [] + for kwarg in partial_kwargs: + param = func_sig.parameters[kwarg] + if param.default is param.empty: + partialed_positional_indices.append(parameter_names.index(kwarg)) + + partial_params = list(func_sig.parameters.values()) + for index in sorted(partialed_positional_indices, reverse=True): + # appending isn't needed for functionality, but more similar to partial + # and it shows that these arguments can still be updated as kwargs + partial_params.append( + partial_params[index].replace( + kind=Parameter.KEYWORD_ONLY, + default=partial_kwargs[partial_params[index].name], + ) + ) + del partial_params[index] + del partial_params[offset : offset + len(partial_args)] + + full_partial.__signature__ = func_sig.replace(parameters=partial_params) + + return full_partial + + return bound_partial_with_offset + + +state_partial = partial_with_offset() + + +def rename_function(func, name): + # see functools.wraps + func.__name__ = func.__qualname__ = name + + +def add_partial_sct(func, name): + rename_function(func, name) + scts[name] = func + + +add_partial_sct( + state_partial( + has_equal_part, + "name", + msg="Make sure to use the correct {{name}}, was expecting {{sol_part[name]}}, instead got {{stu_part[name]}}.", + ), + "has_equal_name", +) +add_partial_sct( + state_partial( + has_equal_part, + "is_default", + msg="Make sure it {{ 'has' if sol_part.is_default else 'does not have'}} a default argument.", + ), + "is_default", +) + +# include rest of wrappers +for k, v in __PART_WRAPPERS__.items(): + check_fun = state_partial(check_part, k, v) + add_partial_sct(check_fun, "check_" + k) + +for k, v in __PART_INDEX_WRAPPERS__.items(): + check_fun = state_partial(check_part_index, k, part_msg=v) + add_partial_sct(check_fun, "check_" + k) + +for k, v in __NODE_WRAPPERS__.items(): + check_fun = state_partial(check_node, k + "s", typestr=v["typestr"]) + check_fun.__doc__ = Template(v["docstr"]).render( + typestr="typestr: If specified, this overrides the standard way of referring to the construct you're zooming in on.", + missing_msg="missing_msg: If specified, this overrides the automatically generated feedback message in case the construct could not be found.", + expand_msg="expand_msg: If specified, this overrides the automatically generated feedback message that is prepended to feedback messages that are thrown further in the SCT chain.", + ) + add_partial_sct(check_fun, "check_" + k) + +for k in [ + "set_context", + "set_env", + "disable_highlighting", + "check_not", + "check_or", + "check_correct", + "fail", + "override", + "multi", +]: + scts[k] = getattr(check_logic, k) + +for k in ["with_context", "check_args", "check_call"]: + scts[k] = getattr(check_funcs, k) + +for k in [ + "has_equal_value", + "has_equal_output", + "has_equal_error", + "has_equal_ast", + "has_equal_part_len", + "has_equal_part", + "has_import", + "has_output", + "has_printout", + "has_code", + "has_no_error", + "has_chosen", +]: + sct = getattr(has_funcs, k) + if not hasattr(sct, "__name__"): + rename_function(sct, k) + scts[k] = sct + +# include check_object and friends ------ +for k in ["check_object", "is_instance", "check_df", "check_keys"]: + scts[k] = getattr(check_object, k) + +scts["has_context"] = has_context +scts["check_function"] = check_function + +scts["run"] = run + +scts["has_command"] = has_command + +scts["check_file"] = check_file +scts["has_dir"] = has_dir + +scts["allow_errors"] = allow_errors + +scts["_debug"] = _debug + +locals().update(scts) diff --git a/pythonwhat/checks/has_funcs.py b/pythonwhat/checks/has_funcs.py new file mode 100644 index 00000000..870e1069 --- /dev/null +++ b/pythonwhat/checks/has_funcs.py @@ -0,0 +1,852 @@ +from protowhat.utils_messaging import get_ord +from pythonwhat.tasks import ( + getResultInProcess, + getOutputInProcess, + getErrorInProcess, + ReprFail, + isDefinedInProcess, + getOptionFromProcess, + UndefinedValue, +) +from pythonwhat.Test import EqualTest, DefinedCollTest +from protowhat.Feedback import Feedback, FeedbackComponent +from protowhat.failure import InstructorError, debugger +from pythonwhat import utils +from functools import partial +import re +import copy +import ast + +evalCalls = { + "value": getResultInProcess, + "output": getOutputInProcess, + "error": getErrorInProcess, +} + + +def has_part(state, name, msg, fmt_kwargs=None, index=None): + d = { + "sol_part": state.solution_parts, + "stu_part": state.student_parts, + **fmt_kwargs, + } + + def verify(part, index): + if index is not None: + if isinstance(index, list): + for ind in index: + part = part[ind] + else: + part = part[index] + if part is None: + raise KeyError + + # TODO: instructor error if msg is not str + # Check if it's there in the solution + try: + verify(state.solution_parts[name], index) + except (KeyError, IndexError): + with debugger(state): + err_msg = "SCT fails on solution: {}".format(msg) + state.report(err_msg, d) + + try: + verify(state.student_parts[name], index) + except (KeyError, IndexError): + state.report(msg, d) + + return state + + +def has_equal_part(state, name, msg): + d = { + "stu_part": state.student_parts, + "sol_part": state.solution_parts, + "name": name, + } + + state.do_test( + EqualTest(d["stu_part"][name], d["sol_part"][name], FeedbackComponent(msg, d)) + ) + + return state + + +# TODO: shouldn't have to hardcode message +def has_equal_part_len(state, name, unequal_msg): + """Verify that a part that is zoomed in on has equal length. + + Typically used in the context of ``check_function_def()`` + + Arguments: + name (str): name of the part for which to check the length to the corresponding part in the solution. + unequal_msg (str): Message in case the lengths do not match. + state (State): state as passed by the SCT chain. Don't specify this explicitly. + + :Examples: + + Student and solution code:: + + def shout(word): + return word + '!!!' + + SCT that checks number of arguments:: + + Ex().check_function_def('shout').has_equal_part_len('args', 'not enough args!') + """ + d = dict( + stu_len=len(state.student_parts[name]), sol_len=len(state.solution_parts[name]) + ) + + if d["stu_len"] != d["sol_len"]: + state.report(unequal_msg, d) + + return state + + +# Expression tests ----------------------------------------------------------- + + +def has_equal_ast(state, incorrect_msg=None, code=None, exact=True, append=None): + """Test whether abstract syntax trees match between the student and solution code. + + ``has_equal_ast()`` can be used in two ways: + + * As a robust version of ``has_code()``. By setting ``code``, you can look for the AST representation of ``code`` in the student's submission. + But be aware that ``a`` and ``a = 1`` won't match, as reading and assigning are not the same in an AST. + Use ``ast.dump(ast.parse(code))`` to see an AST representation of ``code``. + * As an expression-based check when using more advanced SCT chain, e.g. to compare the equality of expressions to set function arguments. + + Args: + incorrect_msg: message displayed when ASTs mismatch. When you specify ``code`` yourself, you have to specify this. + code: optional code to use instead of the solution AST. + exact: whether the representations must match exactly. If false, the solution AST + only needs to be contained within the student AST (similar to using test student typed). + Defaults to ``True``, unless the ``code`` argument has been specified. + + :Example: + + Student and Solution Code:: + + dict(a = 'value').keys() + + SCT:: + + # all pass + Ex().has_equal_ast() + Ex().has_equal_ast(code = "dict(a = 'value').keys()") + Ex().has_equal_ast(code = "dict(a = 'value')", exact = False) + + Student and Solution Code:: + + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + SCT:: + + # Check underlying value of arugment a of np.mean: + Ex().check_function('numpy.mean').check_args('a').has_equal_ast() + + # Only check AST equality of expression used to specify argument a: + Ex().check_function('numpy.mean').check_args('a').has_equal_ast() + + """ + if utils.v2_only(): + state.assert_is_not(["object_assignments"], "has_equal_ast", ["check_object"]) + state.assert_is_not(["function_calls"], "has_equal_ast", ["check_function"]) + + if code and incorrect_msg is None: + raise InstructorError.from_message( + "If you manually specify the code to match inside has_equal_ast(), " + "you have to explicitly set the `incorrect_msg` argument." + ) + + if ( + append is None + ): # if not specified, set to False if incorrect_msg was manually specified + append = incorrect_msg is None + if incorrect_msg is None: + incorrect_msg = ( + "Expected \n```\n{{sol_str}}\n```\n, but got \n```\n{{stu_str}}\n```\n" + if utils.is_multiline_code(state.student_code, state.solution_code) + else "Expected `{{sol_str}}`, but got `{{stu_str}}`." + ) + + def parse_tree(tree): + # get contents of module.body if only 1 element + crnt = ( + tree.body[0] + if isinstance(tree, ast.Module) and len(tree.body) == 1 + else tree + ) + + # remove Expr if it exists + return ast.dump(crnt.value if isinstance(crnt, ast.Expr) else crnt) + + stu_rep = parse_tree(state.student_ast) + sol_rep = parse_tree(state.solution_ast if not code else ast.parse(code)) + + if utils.is_multiline_code(state.student_code, state.solution_code): + fmt_kwargs = { + "sol_str": utils.format_code(state.solution_code) + if not code + else utils.format_code(code), + "stu_str": utils.format_code(state.student_code), + } + else: + fmt_kwargs = { + "sol_str": state.solution_code if not code else code, + "stu_str": state.student_code, + } + + if exact and not code: + state.do_test( + EqualTest( + stu_rep, + sol_rep, + FeedbackComponent(incorrect_msg, fmt_kwargs, append=append), + ) + ) + elif sol_rep not in stu_rep: + state.report(incorrect_msg, fmt_kwargs, append=append) + + return state + + +DEFAULT_INCORRECT_MSG = "Expected {{test_desc}}`{{sol_eval}}`, but got `{{stu_eval}}`." +DEFAULT_ERROR_MSG = "Running {{'it' if parent['part'] else 'the highlighted expression'}} generated an error: `{{stu_str}}`." +DEFAULT_ERROR_MSG_INV = "Running {{'it' if parent['part'] else 'the highlighted expression'}} didn't generate an error, but it should!" +DEFAULT_UNDEFINED_NAME_MSG = "Running {{'it' if parent['part'] else 'the highlighted expression'}} should define a variable `{{name}}` without errors, but it doesn't." +DEFAULT_INCORRECT_NAME_MSG = ( + "Are you sure you assigned the correct value to `{{name}}`?" +) +DEFAULT_INCORRECT_EXPR_CODE_MSG = ( + "Running the expression `{{expr_code}}` didn't generate the expected result." +) + +args_string = """ + + Args: + incorrect_msg (str): feedback message if the {0} of the expression in the solution + doesn't match the one of the student. This feedback message will be expanded if it is used + in the context of another check function, like ``check_if_else``. + error_msg (str): feedback message if there was an error when running the targeted student code. + Note that when testing for an error, this message is displayed when none is raised. + undefined_msg (str): feedback message if the ``name`` argument is defined, but a variable + with that name doesn't exist after running the targeted student code. + extra_env (dict): set variables to the extra environment. They will update the student and solution environment in + the active state before the student/solution code in the active state is ran. This argument should contain a + dictionary with the keys the names of the variables you want to set, and the values are the values of these variables. + You can also use ``set_env()`` for this. + context_vals (list): set variables which are bound in a ``for`` loop to certain values. + This argument is only useful when checking a for loop (or list comprehensions). + It contains a list with the values of the bound variables. + You can also use ``set_context()`` for this. + pre_code (str): the code in string form that should be executed before the expression is executed. + This is the ideal place to set a random seed, for example. + expr_code (str): If this argument is set, the expression in the student/solution code will not + be ran. Instead, the given piece of code will be ran in the student as well as the solution environment + and the result will be compared. However if the string contains one or more placeholders ``__focus__``, + they will be substituted by the currently focused code. + name (str): If this is specified, the {0} of running this expression after running the focused expression + is returned, instead of the {0} of the focused expression in itself. This is typically used to inspect the + {0} of an object after executing the body of e.g. a ``for`` loop. + copy (bool): whether to try to deep copy objects in the environment, such as lists, that could + accidentally be mutated. Disable to speed up SCTs. Disabling may lead to cryptic mutation issues. + func (function): custom binary function of form f(stu_result, sol_result), for equality testing. + override: If specified, this avoids the execution of the targeted code in the solution process. Instead, it + will compare the {0} of the expression in the student process with the value specified in ``override``. + Typically used in a ``SingleProcessExercise`` or if you want to allow for different solutions other than + the one coded up in the solution. + """ + + +def has_expr( + state, + incorrect_msg=None, + error_msg=None, + undefined_msg=None, + append=None, + extra_env=None, + context_vals=None, + pre_code=None, + expr_code=None, + name=None, + copy=True, + func=None, + override=None, + test=None, # todo: default or arg before state +): + + if ( + append is None + ): # if not specified, set to False if incorrect_msg was manually specified + append = incorrect_msg is None + if incorrect_msg is None: + if name: + incorrect_msg = DEFAULT_INCORRECT_NAME_MSG + elif expr_code: + incorrect_msg = DEFAULT_INCORRECT_EXPR_CODE_MSG + else: + incorrect_msg = DEFAULT_INCORRECT_MSG + if undefined_msg is None: + undefined_msg = DEFAULT_UNDEFINED_NAME_MSG + if error_msg is None: + if test == "error": + error_msg = DEFAULT_ERROR_MSG_INV + else: + error_msg = DEFAULT_ERROR_MSG + + if state.solution_code is not None and isinstance(expr_code, str): + expr_code = expr_code.replace("__focus__", state.solution_code) + + get_func = partial( + evalCalls[test], + extra_env=extra_env, + context_vals=context_vals, + pre_code=pre_code, + expr_code=expr_code, + name=name, + copy=copy, + ) + + if override is not None: + # don't bother with running expression and fetching output/value + # eval_sol, str_sol = eval + eval_sol, str_sol = override, str(override) + else: + eval_sol, str_sol = get_func( + tree=state.solution_ast, + process=state.solution_process, + context=state.solution_context, + env=state.solution_env, + ) + + if (test == "error") ^ isinstance(eval_sol, Exception): + raise InstructorError.from_message( + "Evaluating expression raised error in solution process (or didn't raise if testing for one). " + "Error: {} - {}".format(type(eval_sol), str_sol) + ) + if isinstance(eval_sol, ReprFail): + raise InstructorError.from_message( + "Couldn't extract the value for the highlighted expression from the solution process: " + + eval_sol.info + ) + + eval_stu, str_stu = get_func( + tree=state.student_ast, + process=state.student_process, + context=state.student_context, + env=state.student_env, + ) + + # kwargs --- + fmt_kwargs = { + "stu_part": state.student_parts, + "sol_part": state.solution_parts, + "name": name, + "test": test, + "test_desc": "" if test == "value" else "the %s " % test, + "expr_code": expr_code, + } + + fmt_kwargs["stu_eval"] = str(eval_stu) + fmt_kwargs["sol_eval"] = str(eval_sol) + + # wrap in quotes if eval_sol or eval_stu are strings + if test == "value": + if isinstance(eval_stu, str): + fmt_kwargs["stu_eval"] = "'{}'".format(fmt_kwargs["stu_eval"]) + if isinstance(eval_sol, str): + fmt_kwargs["sol_eval"] = "'{}'".format(fmt_kwargs["sol_eval"]) + + # reformat student evaluation string if it is too long + fmt_kwargs["stu_eval"] = utils.shorten_string(fmt_kwargs["stu_eval"]) + + # check if student or solution evaluations are too long or contain newlines + if incorrect_msg == DEFAULT_INCORRECT_MSG and ( + len(fmt_kwargs["sol_eval"]) > 50 + or utils.has_newline(fmt_kwargs["stu_eval"]) + or utils.has_newline(fmt_kwargs["sol_eval"]) + or fmt_kwargs["stu_eval"] == fmt_kwargs["sol_eval"] + ): + fmt_kwargs["stu_eval"] = None + fmt_kwargs["sol_eval"] = None + incorrect_msg = "Expected something different." + + # tests --- + # error in process + if (test == "error") ^ isinstance(eval_stu, Exception): + fmt_kwargs["stu_str"] = str_stu + state.report(error_msg, fmt_kwargs, append=append) + + # name is undefined after running expression + if isinstance(eval_stu, UndefinedValue): + state.report(undefined_msg, fmt_kwargs, append=append) + + # test equality of results + state.do_test( + EqualTest( + eval_stu, + eval_sol, + FeedbackComponent(incorrect_msg, fmt_kwargs, append=append), + func, + ) + ) + + return state + + +has_equal_value = partial(has_expr, test="value") +has_equal_value.__name__ = "has_equal_value" +has_equal_value.__doc__ = ( + """Run targeted student and solution code, and compare returned value. + + When called on an SCT chain, ``has_equal_value()`` will execute the student and solution + code that is 'zoomed in on' and compare the returned values. + """ + + args_string.format("returned value", "value") + + """ + :Example: + + Student code and solution code:: + + import numpy as np + arr = np.array([1, 2, 3, 4, 5]) + np.mean(arr) + + SCT:: + + # Verify equality of arr: + Ex().check_object('arr').has_equal_value() + + # Verify whether arr was correctly set in np.mean + Ex().check_function('numpy.mean').check_args('a').has_equal_value() + + # Verify whether np.mean(arr) produced the same result + Ex().check_function('numpy.mean').has_equal_value() + + """ +) + + +has_equal_output = partial(has_expr, test="output") +has_equal_output.__name__ = "has_equal_output" +has_equal_output.__doc__ = """Run targeted student and solution code, and compare output. + + When called on an SCT chain, ``has_equal_output()`` will execute the student and solution + code that is 'zoomed in on' and compare the output. + """ + args_string.format( + "output" +) + +has_equal_error = partial(has_expr, test="error") +has_equal_error.__name__ = "has_equal_error" +has_equal_error.__doc__ = """Run targeted student and solution code, and compare generated errors. + + When called on an SCT chain, ``has_equal_error()`` will execute the student and solution + code that is 'zoomed in on' and compare the errors that they generate. + """ + args_string.format( + "error" +) + +## Various has tests ---------------------------------------------------------- + +from pythonwhat.Test import StringContainsTest + + +def has_code(state, text, pattern=True, not_typed_msg=None): + """Test the student code. + + Tests if the student typed a (pattern of) text. It is advised to use ``has_equal_ast()`` instead of ``has_code()``, + as it is more robust to small syntactical differences that don't change the code's behavior. + + Args: + text (str): the text that is searched for + pattern (bool): if True (the default), the text is treated as a pattern. If False, it is treated as plain text. + not_typed_msg (str): feedback message to be displayed if the student did not type the text. + + :Example: + + Student code and solution code:: + + y = 1 + 2 + 3 + + SCT:: + + # Verify that student code contains pattern (not robust!!): + Ex().has_code(r"1\\s*\\+2\\s*\\+3") + + """ + if not not_typed_msg: + if pattern: + not_typed_msg = "Could not find the correct pattern in your code." + else: + not_typed_msg = "Could not find the following text in your code: %r" % text + + student_code = state.student_code + + state.do_test(StringContainsTest(student_code, text, pattern, not_typed_msg)) + + return state + + +def has_import( + state, + name, + same_as=False, + not_imported_msg="Did you import `{{pkg}}`?", + incorrect_as_msg="Did you import `{{pkg}}` as `{{alias}}`?", +): + """Checks whether student imported a package or function correctly. + + Python features many ways to import packages. + All of these different methods revolve around the ``import``, ``from`` and ``as`` keywords. + ``has_import()`` provides a robust way to check whether a student correctly imported a certain package. + + By default, ``has_import()`` allows for different ways of aliasing the imported package or function. + If you want to make sure the correct alias was used to refer to the package or function that was imported, + set ``same_as=True``. + + Args: + name (str): the name of the package that has to be checked. + same_as (bool): if True, the alias of the package or function has to be the same. Defaults to False. + not_imported_msg (str): feedback message when the package is not imported. + incorrect_as_msg (str): feedback message if the alias is wrong. + + :Example: + + Example 1, where aliases don't matter (defaut): :: + + # solution + import matplotlib.pyplot as plt + + # sct + Ex().has_import("matplotlib.pyplot") + + # passing submissions + import matplotlib.pyplot as plt + from matplotlib import pyplot as plt + import matplotlib.pyplot as pltttt + + # failing submissions + import matplotlib as mpl + + Example 2, where the SCT is coded so aliases do matter: :: + + # solution + import matplotlib.pyplot as plt + + # sct + Ex().has_import("matplotlib.pyplot", same_as=True) + + # passing submissions + import matplotlib.pyplot as plt + from matplotlib import pyplot as plt + + # failing submissions + import matplotlib.pyplot as pltttt + + """ + student_imports = state.ast_dispatcher.find("imports", state.student_ast) + solution_imports = state.ast_dispatcher.find("imports", state.solution_ast) + + if name not in solution_imports: + raise InstructorError.from_message( + "`has_import()` couldn't find an import of the package %s in your solution code." + % name + ) + + fmt_kwargs = {"pkg": name, "alias": solution_imports[name]} + + state.do_test( + DefinedCollTest( + name, student_imports, FeedbackComponent(not_imported_msg, fmt_kwargs) + ) + ) + + if same_as: + state.do_test( + EqualTest( + solution_imports[name], + student_imports[name], + FeedbackComponent(incorrect_as_msg, fmt_kwargs), + ) + ) + + return state + + +def has_output(state, text, pattern=True, no_output_msg=None): + r"""Search student output for a pattern. + + Among the student and solution process, the student submission and solution code as a string, + the ``Ex()`` state also contains the output that a student generated with his or her submission. + + With ``has_output()``, you can access this output and match it against a regular or fixed expression. + + Args: + text (str): the text that is searched for + pattern (bool): if True (default), the text is treated as a pattern. If False, it is treated as plain text. + no_output_msg (str): feedback message to be displayed if the output is not found. + + :Example: + + As an example, suppose we want a student to print out a sentence: :: + + # Print the "This is some ... stuff" + print("This is some weird stuff") + + The following SCT tests whether the student prints out ``This is some weird stuff``: :: + + # Using exact string matching + Ex().has_output("This is some weird stuff", pattern = False) + + # Using a regular expression (more robust) + # pattern = True is the default + msg = "Print out ``This is some ... stuff`` to the output, " + \\ + "fill in ``...`` with a word you like." + Ex().has_output(r"This is some \w* stuff", no_output_msg = msg) + + """ + if not no_output_msg: + no_output_msg = "You did not output the correct things." + + state.do_test( + StringContainsTest(state.raw_student_output, text, pattern, no_output_msg) + ) + + return state + + +def has_printout( + state, index, not_printed_msg=None, pre_code=None, name=None, copy=False +): + """Check if the right printouts happened. + + ``has_printout()`` will look for the printout in the solution code that you specified with ``index`` (0 in this case), rerun the ``print()`` call in + the solution process, capture its output, and verify whether the output is present in the output of the student. + + This is more robust as ``Ex().check_function('print')`` initiated chains as students can use as many + printouts as they want, as long as they do the correct one somewhere. + + Args: + index (int): index of the ``print()`` call in the solution whose output you want to search for in the student output. + not_printed_msg (str): if specified, this overrides the default message that is generated when the output + is not found in the student output. + pre_code (str): Python code as a string that is executed before running the targeted student call. + This is the ideal place to set a random seed, for example. + copy (bool): whether to try to deep copy objects in the environment, such as lists, that could + accidentally be mutated. Disabled by default, which speeds up SCTs. + state (State): state as passed by the SCT chain. Don't specify this explicitly. + + :Example: + + Suppose you want somebody to print out 4: :: + + print(1, 2, 3, 4) + + The following SCT would check that: :: + + Ex().has_printout(0) + + All of the following SCTs would pass: :: + + print(1, 2, 3, 4) + print('1 2 3 4') + print(1, 2, '3 4') + print("random"); print(1, 2, 3, 4) + + :Example: + + Watch out: ``has_printout()`` will effectively **rerun** the ``print()`` call in the solution process after the entire solution script was executed. + If your solution script updates the value of `x` after executing it, ``has_printout()`` will not work. + + Suppose you have the following solution: :: + + x = 4 + print(x) + x = 6 + + The following SCT will not work: :: + + Ex().has_printout(0) + + Why? When the ``print(x)`` call is executed, the value of ``x`` will be 6, and pythonwhat will look for the output `'6`' in the output the student generated. + In cases like these, ``has_printout()`` cannot be used. + + :Example: + + Inside a for loop ``has_printout()`` + + Suppose you have the following solution: :: + + for i in range(5): + print(i) + + The following SCT will not work: :: + + Ex().check_for_loop().check_body().has_printout(0) + + The reason is that ``has_printout()`` can only be called from the root state. ``Ex()``. + If you want to check printouts done in e.g. a for loop, you have to use a `check_function('print')` chain instead: :: + + Ex().check_for_loop().check_body().\\ + set_context(0).check_function('print').\\ + check_args(0).has_equal_value() + + """ + + extra_msg = "If you want to check printouts done in e.g. a for loop, you have to use a `check_function('print')` chain instead." + state.assert_execution_root("has_printout", extra_msg=extra_msg) + + if not_printed_msg is None: + not_printed_msg = ( + "Have you used `{{sol_call}}` to do the appropriate printouts?" + ) + + try: + sol_call_ast = state.ast_dispatcher.find("function_calls", state.solution_ast)[ + "print" + ][index]["node"] + except (KeyError, IndexError): + raise InstructorError.from_message( + "`has_printout({})` couldn't find the {} print call in your solution.".format( + index, get_ord(index + 1) + ) + ) + + out_sol, str_sol = getOutputInProcess( + tree=sol_call_ast, + process=state.solution_process, + context=state.solution_context, + env=state.solution_env, + pre_code=pre_code, + copy=copy, + ) + + sol_call_str = state.solution_ast_tokens.get_text(sol_call_ast) + + if isinstance(str_sol, Exception): + with debugger(state): + state.report( + "Evaluating the solution expression {} raised error in solution process." + "Error: {} - {}".format(sol_call_str, type(out_sol), str_sol) + ) + + has_output( + state, + out_sol.strip(), + pattern=False, + no_output_msg=FeedbackComponent(not_printed_msg, {"sol_call": sol_call_str}), + ) + + return state + + +def has_no_error( + state, + incorrect_msg="Have a look at the console: your code contains an error. Fix it and try again!", +): + """Check whether the submission did not generate a runtime error. + + If all SCTs for an exercise pass, before marking the submission as correct pythonwhat will automatically check whether + the student submission generated an error. This means it is not needed to use ``has_no_error()`` explicitly. + + However, in some cases, using ``has_no_error()`` explicitly somewhere throughout your SCT execution can be helpful: + + - If you want to make sure people didn't write typos when writing a long function name. + - If you want to first verify whether a function actually runs, before checking whether the arguments were specified correctly. + - More generally, if, because of the content, it's instrumental that the script runs without + errors before doing any other verifications. + + Args: + incorrect_msg: if specified, this overrides the default message if the student code generated an error. + + :Example: + + Suppose you're verifying an exercise about model training and validation: :: + + # pre exercise code + import numpy as np + from sklearn.model_selection import train_test_split + from sklearn import datasets + from sklearn import svm + + iris = datasets.load_iris() + iris.data.shape, iris.target.shape + + # solution + X_train, X_test, y_train, y_test = train_test_split( + iris.data, iris.target, test_size=0.4, random_state=0) + + If you want to make sure that ``train_test_split()`` ran without errors, + which would check if the student typed the function without typos and used + sensical arguments, you could use the following SCT: :: + + Ex().has_no_error() + Ex().check_function('sklearn.model_selection.train_test_split').multi( + check_args(['arrays', 0]).has_equal_value(), + check_args(['arrays', 0]).has_equal_value(), + check_args(['options', 'test_size']).has_equal_value(), + check_args(['options', 'random_state']).has_equal_value() + ) + + If, on the other hand, you want to fall back onto pythonwhat's built in behavior, + that checks for an error before marking the exercise as correct, you can simply + leave of the ``has_no_error()`` step. + + """ + state.assert_execution_root("has_no_error") + + if state.reporter.errors: + state.report(incorrect_msg, {"error": str(state.reporter.errors[0])}) + + return state + + +MC_VAR_NAME = "selected_option" + + +def has_chosen(state, correct, msgs): + """Test multiple choice exercise. + + Test for a MultipleChoiceExercise. The correct answer (as an integer) and feedback messages + are passed to this function. + + Args: + correct (int): the index of the correct answer (should be an instruction). Starts at 1. + msgs (list(str)): a list containing all feedback messages belonging to each choice of the + student. The list should have the same length as the number of options. + """ + if not issubclass(type(correct), int): + raise InstructorError.from_message( + "Inside `has_chosen()`, the argument `correct` should be an integer." + ) + + student_process = state.student_process + if not isDefinedInProcess(MC_VAR_NAME, student_process): + raise InstructorError.from_message( + "Option not available in the student process" + ) + else: + selected_option = getOptionFromProcess(student_process, MC_VAR_NAME) + if not issubclass(type(selected_option), int): + raise InstructorError.from_message("selected_option should be an integer") + + if selected_option < 1 or correct < 1: + raise InstructorError.from_message( + "selected_option and correct should be greater than zero" + ) + + if selected_option > len(msgs) or correct > len(msgs): + raise InstructorError.from_message( + "there are not enough feedback messages defined" + ) + + feedback_msg = msgs[selected_option - 1] + + state.reporter.success_msg = msgs[correct - 1] + + state.do_test(EqualTest(selected_option, correct, feedback_msg)) diff --git a/pythonwhat/converters.py b/pythonwhat/converters.py index 954144e5..16d48ba2 100644 --- a/pythonwhat/converters.py +++ b/pythonwhat/converters.py @@ -1,19 +1,23 @@ import inspect import pythonwhat + def get_manual_converters(): converters = { - 'pandas.io.excel.ExcelFile': lambda x: x.io, - 'builtins.dict_keys': lambda x: sorted(x), - 'builtins.dict_items': lambda x: sorted(x), - 'bs4.BeautifulSoup': lambda x: str(x), - 'bs4.element.Tag': lambda x: str(x), - 'bs4.element.NavigableString': lambda x: str(x), - 'bs4.element.ResultSet': lambda x: [str(res) for res in x], - 'h5py._hl.files.File': lambda x: x.file.filename, - 'h5py._hl.group.Group': lambda x: x.file.filename + '_' + str([x for x in x.keys()]), - 'sqlalchemy.engine.base.Engine': lambda x: x.url.database + "pandas.io.excel.ExcelFile": lambda x: x.io, + "pandas.io.excel._base.ExcelFile": lambda x: x.io, + "builtins.dict_keys": lambda x: sorted(x), + "builtins.dict_items": lambda x: sorted(x), + "bs4.BeautifulSoup": lambda x: str(x), + "bs4.element.Tag": lambda x: str(x), + "bs4.element.NavigableString": lambda x: str(x), + "bs4.element.ResultSet": lambda x: [str(res) for res in x], + "h5py._hl.files.File": lambda x: x.file.filename, + "h5py._hl.group.Group": lambda x: x.file.filename + + "_" + + str([x for x in x.keys()]), + "sqlalchemy.engine.base.Engine": lambda x: x.url.database, } - #' - return(converters) + + return converters diff --git a/pythonwhat/feedback.py b/pythonwhat/feedback.py new file mode 100644 index 00000000..dae753f0 --- /dev/null +++ b/pythonwhat/feedback.py @@ -0,0 +1,19 @@ +from typing import Dict + +from protowhat.Feedback import Feedback as ProtoFeedback + + +class Feedback(ProtoFeedback): + ast_highlight_offset = {"column_start": 1} + + @classmethod + def get_highlight_position(cls, highlight) -> Dict[str, int]: + if getattr(highlight, "first_token", None) and getattr( + highlight, "last_token", None + ): + return { + "line_start": highlight.first_token.start[0], + "column_start": highlight.first_token.start[1], + "line_end": highlight.last_token.end[0], + "column_end": highlight.last_token.end[1], + } diff --git a/pythonwhat/local.py b/pythonwhat/local.py new file mode 100644 index 00000000..7dee518e --- /dev/null +++ b/pythonwhat/local.py @@ -0,0 +1,293 @@ +import io +import os +import random +from pathlib import Path +from contextlib import redirect_stdout + +from multiprocessing import Process, Queue +from protowhat.Reporter import Reporter + +try: + from pythonbackend.shell_utils import create + from pythonbackend.tasks import TaskCaptureFullOutput + + BACKEND_AVAILABLE = True +except: + BACKEND_AVAILABLE = False + + +class StubShell: + def __init__(self, init_code=None): + self.user_ns = {} + if init_code: + self.run_code(init_code) + + def run_code(self, code): + exec(code, self.user_ns) + + +class StubProcess: + def __init__(self, init_code=None, pid=None): + self.shell = StubShell(init_code) + self._identity = (pid,) if pid else (random.randint(0, int(1e12)),) + + def executeTask(self, task): + return task(self.shell) + + +class TaskCaptureOutput: + def __init__(self, code): + self.code = code + + def __call__(self, shell): + return run_code(shell.run_code, self.code) + + +class TaskKillProcess: + def __call__(self, shell): + return None + + +class CaptureErrors: + def __init__(self, output): + self.output = output + + def __enter__(self): + pass + + def __exit__(self, exc_type, exception, traceback): + if exc_type is not None: + self.output.append({"type": "backend-error", "payload": str(exception)}) + return True + + +class WorkerProcess(Process): + instances = [] + + def __init__(self, pid=None): + Process.__init__(self) + self.task_queue = Queue() + self.result_queue = Queue() + self.daemon = ( + True + ) # when parent process is killed, sub/childprocess get also killed + self.instances.append(self) + # used to detect single process exercise + self._identity = (pid,) if pid else (random.randint(0, int(1e12)),) + + def get_shell(self): + return create({}) + + def run(self): + shell = self.get_shell() + while True: + output = [] + with CaptureErrors(output): + next_task = self.task_queue.get() + answer = next_task(shell) + if len(output) > 0: # means backend error happened + answer = output + output = [] + with CaptureErrors(output): + self.result_queue.put_nowait(answer) + if len(output) > 0: # means backend error happened + self.result_queue.put_nowait(output) + if isinstance(next_task, TaskKillProcess): + break # break while loop -> we do not wait upon new task + return + + def executeTask(self, task): + self.task_queue.put_nowait(task) + return self.result_queue.get() # wait and fetches next item in queue + + def kill(self): + try: + if self.is_alive(): + self.executeTask(TaskKillProcess()) + self.join(timeout=3.0) + if self.is_alive(): + self.terminate() + self.join(timeout=3.0) + if self in self.instances: + self.instances.remove(self) + finally: + pass + # python 3.7: + # self.close() + + @classmethod + def kill_all(cls): + for instance in list(cls.instances): + instance.kill() + + +class SimpleProcess(WorkerProcess): + def get_shell(self): + return StubShell() + + +class ChDir(object): + """ + Step into a directory temporarily. + """ + + def __init__(self, path): + self.old_dir = os.getcwd() + self.new_dir = str(path) + + def __enter__(self): + os.chdir(self.new_dir) + + def __exit__(self, *args): + os.chdir(self.old_dir) + + +def run_code(executor, code): + with io.StringIO() as output: + try: + with redirect_stdout(output): + executor(code) + raw_output = output.getvalue() + error = None + except BaseException as e: + raw_output = "" + error = str(e) + return raw_output, error + + +def run_single_process(pec, code, pid=None, mode="simple"): + if mode == "stub": + # no isolation + process = StubProcess(init_code=pec, pid=pid) + raw_output, error = run_code(process.shell.run_code, code) + + elif mode == "simple": + # no advanced functionality + process = SimpleProcess(pid) + process.start() + _ = process.executeTask(TaskCaptureOutput(pec)) + raw_output, error = process.executeTask(TaskCaptureOutput(code)) + + elif mode == "full" and BACKEND_AVAILABLE: + # slow + process = WorkerProcess(pid) + process.start() + _ = process.executeTask( + TaskCaptureFullOutput((pec,), "", None, silent=True) + ) + output, raw_output = process.executeTask( + TaskCaptureFullOutput((code,), "script.py", None, silent=True) + ) + raw_output = raw_output["output_stream"] + error = raw_output["error"] + + else: + raise ValueError("Invalid mode") + + return process, raw_output, error + + +def run_exercise(pec, sol_code, stu_code, sol_wd=None, stu_wd=None, **kwargs): + with ChDir(sol_wd or os.getcwd()): + sol_process, _, _ = run_single_process(pec, sol_code, **kwargs) + + with ChDir(stu_wd or os.getcwd()): + stu_process, raw_stu_output, error = run_single_process(pec, stu_code, **kwargs) + + return sol_process, stu_process, raw_stu_output, error + + +# todo: +# imports from local modules (solution needs to be materialised somewhere) +# converge with xbackend (pythonbackend + look at scalabackend) +# move towards xwhat controlling all execution and xbackend providing the execution interface? +# running with arbitrary wd + path + flags (now only wd) needed? +# e.g. `python -m project.run +# allow setting env vars? e.g. PYTHONPATH, could help running more complex setup +# allow prepending code? set_env? e.g. (automatically) setting __file__? +def run(state, relative_working_dir=None, solution_dir="../solution", run_solution=True): + """Run the focused student and solution code in the specified location + + This function can be used after ``check_file`` to execute student and solution code. + The arguments allow configuring the correct context for execution. + + SCT functions chained after this one that execute pieces of code (custom expressions or the focused part of a file) + execute in the same student and solution locations as the file. + + .. note:: + + This function does not execute the file itself, but code in memory. + This can have an impact when: + + - the solution code imports from a different file in the expected solution (code that is not installed) + - using functionality depending on e.g. ``__file__`` and ``inspect`` + + When the expected code has imports from a different file that is part of the exercise, + it can only work if the solution code provided earlier does not have these imports but instead + has all that functionality inlined. + + Args: + relative_working_dir (str): if specified, this relative path is the subdirectory + inside the student and solution context in which the code is executed + solution_dir (str): a relative path, ``solution`` by default, + that sets the root of the solution context, relative to that of the student execution context + state (State): state as passed by the SCT chain. Don't specify this explicitly. + + If ``relative_working_dir`` is not set, it will be the directory the file was loaded from by ``check_file`` + and fall back to the root of the student execution context (the working directory pythonwhat runs in). + + The ``solution_dir`` helps to prevent solution side effects from conflicting with those of the student. + If the set or derived value of ``relative_working_dir`` is an absolute path, + ``relative_working_dir`` will not be used to form the solution execution working directory: + the solution code will be executed in the root of the solution execution context. + + :Example: + + Suppose the student and solution have a file ``script.py`` in ``/home/repl/``:: + + if True: + a = 1 + + print("Hi!") + + We can check it with this SCT (with ``file_content`` containing the expected file content):: + + Ex().check_file( + "script.py", + solution_code=file_content + ).run().multi( + check_object("a").has_equal_value(), + has_printout(0) + ) + """ + # todo: + # look into executing the file itself + # and keeping the process alive to extract values + if relative_working_dir is None: + if getattr(state, "path", False): + relative_working_dir = state.path.parent + else: + relative_working_dir = "" + + if not os.path.isabs(str(relative_working_dir)): + sol_wd = Path(os.getcwd(), solution_dir, relative_working_dir) + else: + sol_wd = Path(os.getcwd(), solution_dir) + + os.makedirs(str(sol_wd), exist_ok=True) + stu_wd = Path(os.getcwd(), relative_working_dir) + sol_code = state.solution_code or "" if run_solution else "" + + sol_process, stu_process, raw_stu_output, error = run_exercise( + pec="", + sol_code=sol_code, + stu_code=state.student_code, + sol_wd=sol_wd, + stu_wd=stu_wd, + ) + return state.to_child( + student_process=stu_process, + solution_process=sol_process, + raw_student_output=raw_stu_output, + reporter=Reporter(state.reporter, errors=[error] if error else []), + ) diff --git a/pythonwhat/parsing.py b/pythonwhat/parsing.py index 001f9264..cd9102d4 100644 --- a/pythonwhat/parsing.py +++ b/pythonwhat/parsing.py @@ -1,4 +1,5 @@ import ast +from pythonwhat.utils_ast import wrap_in_module from collections.abc import Sequence, Mapping from collections import OrderedDict from contextlib import ExitStack @@ -15,7 +16,10 @@ https://greentreesnakes.readthedocs.org/en/latest/ """ -class EmptyTargetVar: pass + +class EmptyTargetVar: + pass + class TargetVars(Mapping): """Immutable ordered mapping from target variables to their values.""" @@ -23,15 +27,20 @@ class TargetVars(Mapping): EMPTY = EmptyTargetVar() def __init__(self, target_vars=tuple(), is_empty=True): - if is_empty: + if is_empty: target_vars = [(v, self.EMPTY) for v in target_vars] self._od = OrderedDict(target_vars) # getitem, len, iter wrap OrderedDict behavior - def __getitem__(self, k): return self._od.__getitem__(k) - def __len__(self): return self._od.__len__() - def __iter__(self): return self._od.__iter__() + def __getitem__(self, k): + return self._od.__getitem__(k) + + def __len__(self): + return self._od.__len__() + + def __iter__(self): + return self._od.__iter__() def update(self, *args, **kwargs): cpy = self.copy() @@ -43,12 +52,17 @@ def copy(self): def __str__(self): """Format target vars for printing""" - if len(self) > 1: return "({})".format(", ".join(self._od.keys())) - else: return "".join(self._od.keys()) + if len(self) > 1: + return "({})".format(", ".join(self._od.keys())) + else: + return "".join(self._od.keys()) def defined_items(self): """Return copy of instance, omitting entries that are EMPTY""" - return self.__class__([(k, v) for k,v in self.items() if v is not self.EMPTY], is_empty=False) + return self.__class__( + [(k, v) for k, v in self.items() if v is not self.EMPTY], is_empty=False + ) + class IndexedDict(Mapping): """Wrapper around OrderedDict that allows access via item position or key""" @@ -56,15 +70,19 @@ class IndexedDict(Mapping): def __init__(self, *args, **kwargs): self._od = OrderedDict(*args, **kwargs) - def __getitem__(self, k): - try: return list(self._od.values())[k] - except TypeError: return self._od[k] - - def __len__(self): return self._od.__len__() - def __iter__(self): return self._od.__iter__() + def __getitem__(self, k): + try: + return list(self._od.values())[k] + except TypeError: + return self._od[k] + + def __len__(self): + return self._od.__len__() + + def __iter__(self): + return self._od.__iter__() - class Parser(ast.NodeVisitor): """Basic parser. @@ -105,7 +123,7 @@ def generic_visit(self, node): node (ast.Node): The node which is visited. """ pass # This ignore is necessary to keep the parser at base level, also look comment above in - # the visit_Module function body. + # the visit_Module function body. def visit_each(self, lst): for el in lst: @@ -114,11 +132,12 @@ def visit_each(self, lst): @staticmethod def get_target_vars(target): get_id = lambda n: n.id if not isinstance(n, ast.Starred) else n.value.id - if isinstance(target, (ast.Name, ast.Starred)): + if isinstance(target, (ast.Name, ast.Starred)): tv = [get_id(target)] - elif isinstance(target, ast.Tuple): + elif isinstance(target, ast.Tuple): tv = [get_id(node) for node in target.elts] - else: tv = [] + else: + tv = [] return TargetVars(tv) @@ -126,7 +145,7 @@ def get_target_vars(target): def get_arg(el): if el is None: return None - else : + else: return el.arg @staticmethod @@ -137,7 +156,7 @@ def get_arg_tuples(arguments, defaults): @staticmethod def get_arg_parts(arguments, defaults, type): - # only difference is that it doesn't pull out arg.arg, so we can + # only difference is that it doesn't pull out arg.arg, so we can # use all the information on the arg node down the road match_def = [None] * (len(arguments) - len(defaults)) + defaults part_list = [] @@ -148,90 +167,91 @@ def get_arg_parts(arguments, defaults, type): @staticmethod def get_arg_part(_arg, _def, type=None): # type is arg, kwonly, kwarg, vararg - if not _arg: return None + if not _arg: + return None # part uses default highlighting, so will highlight "node" entry return { - 'node': _def or _arg, - 'arg': _arg, - # TODO: need to fill out - 'type': type, - 'is_default': True if _def else False, - 'name': _arg.arg, - 'annotation': _arg.annotation - } - - -class OperatorParser(Parser): - """Find operations. - - A parser which inherits from the basic parser to find binary operators. - - Attributes: - out (list(tuple(num, ast.BinOp, list(str)))): A list of tuples containing the linenumber, node and list of used binary operations. - level (num): A number representing the level at which the parser is parsing. - used (list(str)): The operators that are used in the BinOp that we're handling. - """ - - - # All possible operations and their sign - O_MAP = {} - O_MAP['Add'] = '+' - O_MAP['Sub'] = '-' - O_MAP['Mult'] = '*' - O_MAP['Div'] = '/' - O_MAP['Mod'] = '%' - O_MAP['Pow'] = '**' - O_MAP['LShift'] = '<<' - O_MAP['RShift'] = '>>' - O_MAP['BitOr'] = '|' - O_MAP['BitXor'] = '^' - O_MAP['BitAnd'] = '&' - O_MAP['FloorDiv'] = '//' - - def __init__(self): - """ - Initialize the parser and its attributes. - """ - self.out = [] - self.level = 0 - self.used = [] - - def visit_Expr(self, node): - self.visit(node.value) - - def visit_Call(self, node): - for arg in node.args: - self.visit(arg) - - def visit_Assign(self, node): - self.visit(node.value) - - def visit_Num(self, node): - if not self.level: - self.out.append(( # A number can be seen as a operator on base level. - node, # When student is asked to use operators but just puts in a number instead, - self.used)) # this will help creating a consistent feedback message. - - def visit_UnaryOp(self, node): - self.visit(node.operand) # Unary operations, like '-', should not be added, but they should be - # looked into. They can contain more binary operations. This is important - # during the nesting process. + "node": _def or _arg, + "arg": _arg, + # TODO: need to fill out + "type": type, + "is_default": True if _def else False, + "name": _arg.arg, + "annotation": _arg.annotation, + } - def visit_BinOp(self, node): - self.used.append(OperatorParser.O_MAP[type(node.op).__name__]) - self.level = self.level + 1 - # Nest to other operations, but increase the level. We only - self.visit(node.left) - # want to now which operations are used at a deeper level, but - self.visit(node.right) - self.level = self.level - 1 # we don't need all the explicit nodes. - if not self.level: # We should only add the binary operations of the base level, - self.out.append(( # information about nested operations is included in the used list. - node, - self.used)) - self.used = [] +# class OperatorParser(Parser): +# """Find operations. + +# A parser which inherits from the basic parser to find binary operators. + +# Attributes: +# out (list(tuple(num, ast.BinOp, list(str)))): A list of tuples containing the linenumber, node and list of used binary operations. +# level (num): A number representing the level at which the parser is parsing. +# used (list(str)): The operators that are used in the BinOp that we're handling. +# """ + + +# # All possible operations and their sign +# O_MAP = {} +# O_MAP['Add'] = '+' +# O_MAP['Sub'] = '-' +# O_MAP['Mult'] = '*' +# O_MAP['Div'] = '/' +# O_MAP['Mod'] = '%' +# O_MAP['Pow'] = '**' +# O_MAP['LShift'] = '<<' +# O_MAP['RShift'] = '>>' +# O_MAP['BitOr'] = '|' +# O_MAP['BitXor'] = '^' +# O_MAP['BitAnd'] = '&' +# O_MAP['FloorDiv'] = '//' + +# def __init__(self): +# """ +# Initialize the parser and its attributes. +# """ +# self.out = [] +# self.level = 0 +# self.used = [] + +# def visit_Expr(self, node): +# self.visit(node.value) + +# def visit_Call(self, node): +# for arg in node.args: +# self.visit(arg) + +# def visit_Assign(self, node): +# self.visit(node.value) + +# def visit_Num(self, node): +# if not self.level: +# self.out.append(( # A number can be seen as a operator on base level. +# node, # When student is asked to use operators but just puts in a number instead, +# self.used)) # this will help creating a consistent feedback message. + +# def visit_UnaryOp(self, node): +# self.visit(node.operand) # Unary operations, like '-', should not be added, but they should be +# # looked into. They can contain more binary operations. This is important +# # during the nesting process. + +# def visit_BinOp(self, node): +# self.used.append(OperatorParser.O_MAP[type(node.op).__name__]) +# self.level = self.level + 1 +# # Nest to other operations, but increase the level. We only +# self.visit(node.left) +# # want to now which operations are used at a deeper level, but +# self.visit(node.right) +# self.level = self.level - 1 # we don't need all the explicit nodes. + +# if not self.level: # We should only add the binary operations of the base level, +# self.out.append(( # information about nested operations is included in the used list. +# node, +# self.used)) +# self.used = [] class ImportParser(Parser): @@ -260,8 +280,8 @@ class FunctionParser(Parser): """ def __init__(self): - self.gen_name = '' - self.raw_name = '' + self.gen_name = "" + self.raw_name = "" self.mappings = {} self.out = {} self.call_lookup_active = False @@ -277,6 +297,7 @@ def visit_AugAssign(self, node): self.visit(node.value) def visit_Compare(self, node): + self.visit(node.left) self.visit_each(node.comparators) def visit_UnaryOp(self, node): @@ -287,7 +308,7 @@ def visit_Import(self, node): if imp.asname is not None: self.mappings[imp.asname] = imp.name else: - pass # e.g. numpy import as numpy, so no action needed. + pass # e.g. numpy import as numpy, so no action needed. def visit_ImportFrom(self, node): for imp in node.names: @@ -296,19 +317,27 @@ def visit_ImportFrom(self, node): def visit_Expr(self, node): self.visit(node.value) + def visit_List(self, node): + [self.visit(el) for el in node.elts] + + def visit_Dict(self, node): + [self.visit(el) for el in node.values] + def visit_Call(self, node): if self.call_lookup_active: self.visit(node.func) - else : + else: self.call_lookup_active = True - self.visit(node.func) # Need to visit func to start recording the current function name. + self.visit( + node.func + ) # Need to visit func to start recording the current function name. if self.gen_name: - if (self.gen_name not in self.out): + if self.gen_name not in self.out: self.out[self.gen_name] = [] self.out[self.gen_name].append(self.get_call_part(node)) - #self.out[self.current].append((node, node.args, node.keywords)) + # self.out[self.current].append((node, node.args, node.keywords)) self.gen_name = self.raw_name = "" self.call_lookup_active = False @@ -327,48 +356,53 @@ def visit_Attribute(self, node): self.gen_name += "." + node.attr # Add the function name self.raw_name += "." + node.attr + def visit_Subscript(self, node): + # jump over subscripts for the sake of method calls + self.visit(node.value) + def visit_Name(self, node): self.gen_name = self.mappings.get(node.id) or node.id self.raw_name = node.id - + def get_call_part(self, node): args = [self.get_pos_arg_part(n, ii) for ii, n in enumerate(node.args)] keywords = [self.get_kw_arg_part(n) for n in node.keywords] - return {'node': node, - # TODO: right now, args and keywords can be indexed by pos or name. - # Note that a pos args name is its position. - # Problems will arise if SCT tests a position, but the submission - # has too few positional arguments, since it will then grab a kw arg :( - # This is not necessarily a bad thing, but instructors would need to be - # Careful deciding when to test a pos arg, and when to test using kw. - # Could use check_pos_args with pos_args entry below to solve. - 'args': IndexedDict((n['name'], n) for n in [*args, *keywords]), - #'pos_args': args, - #'keywords': keywords, - 'name': self.raw_name, - '_spec1': (node, node.args, node.keywords, self.raw_name) - } - - def get_pos_arg_part(self, arg, indx_pos): + return { + "node": node, + # TODO: right now, args and keywords can be indexed by pos or name. + # Note that a pos args name is its position. + # Problems will arise if SCT tests a position, but the submission + # has too few positional arguments, since it will then grab a kw arg :( + # This is not necessarily a bad thing, but instructors would need to be + # Careful deciding when to test a pos arg, and when to test using kw. + # Could use check_pos_args with pos_args entry below to solve. + "args": IndexedDict((n["name"], n) for n in [*args, *keywords]), + #'pos_args': args, + #'keywords': keywords, + "name": self.raw_name, + } + + @staticmethod + def get_pos_arg_part(arg, indx_pos): is_star = isinstance(arg, ast.Starred) return { - 'node': arg if not is_star else arg.value, - 'highlight': arg, - 'type': 'argument', - 'is_starred': is_star, - 'name': indx_pos - } - - def get_kw_arg_part(self, arg): + "node": arg if not is_star else arg.value, + "highlight": arg, + "type": "argument", + "is_starred": is_star, + "name": indx_pos, + } + + @staticmethod + def get_kw_arg_part(arg): is_kwarg = arg.arg is None return { - 'node': arg.value, - 'highlight': arg, - 'type': 'keyword', - 'is_kwarg': is_kwarg, - 'name': arg.arg - } - + "node": arg.value, + "highlight": arg, + "type": "keyword", + "is_kwarg": is_kwarg, + "name": arg.arg, + } class ObjectAccessParser(FunctionParser): @@ -406,12 +440,13 @@ def visit_Name(self, node): # if name refers to an import, replace prefix = self.mappings.get(node.id) or node.id - self.gen_name = prefix + "." + self.gen_name if self.gen_name else prefix + self.gen_name = prefix + "." + self.gen_name if self.gen_name else prefix self.raw_name = node.id + "." + self.raw_name if self.raw_name else node.id self.out.append(self.gen_name) self.gen_name = self.raw_name = "" + class ObjectAssignmentParser(Parser): """Find object assignmnts @@ -428,8 +463,7 @@ def visit_Name(self, node): if node.id not in self.out: self.out[node.id] = self.get_part(node, self.active_assignment) else: - self.out[node.id]['highlight'] = None - self.out[node.id]['assignments'].append(self.active_assignment) + self.out[node.id]["highlight"] = None self.active_assignment = None def visit_Attribute(self, node): @@ -462,22 +496,13 @@ def visit_Try(self, node): self.visit_each(node.body) self.visit_each(node.finalbody) - def visit_TryFinally(self, node): - self.visit_each(node.body) - self.visit_each(node.finalbody) - @staticmethod def get_part(name_node, ass_node=None): # either name node or simply str or name itself - name = getattr(name_node, 'id', name_node) + name = getattr(name_node, "id", name_node) load_name = ast.Name(id=name, ctx=ast.Load()) ast.fix_missing_locations(load_name) - # - return {'name': name, - 'node': load_name, - 'highlight': ass_node or name_node, - 'assignments': [] if not ass_node else [ass_node] - } + return {"name": name, "node": load_name, "highlight": ass_node or name_node} class IfParser(Parser): @@ -491,12 +516,9 @@ def __init__(self): self.out = [] def visit_If(self, node): - self.out.append({ - 'node': node, - 'test': node.test, - 'body': node.body, - 'orelse': node.orelse, - }) + self.out.append( + {"node": node, "test": node.test, "body": node.body, "orelse": node.orelse} + ) class IfExpParser(IfParser): @@ -506,7 +528,8 @@ class IfExpParser(IfParser): Only 'top-level' if structures will be found! """ - def visit_If(self, node): return + def visit_If(self, node): + return def visit_IfExp(self, node): super().visit_If(node) @@ -527,15 +550,6 @@ def visit_Compare(self, node): def visit_UnaryOp(self, node): self.visit(node.operand) - def visit_Expr(self, node): - self.visit(node.value) - - def visit_Call(self, node): - self.visit(node.func) - - def visit_Return(self, node): - self.visit(node.value) - class WhileParser(Parser): """Find while structures. @@ -548,12 +562,9 @@ def __init__(self): self.out = [] def visit_While(self, node): - self.out.append({ - 'node': node, - 'test': node.test, - 'body': node.body, - 'orelse': node.orelse - }) + self.out.append( + {"node": node, "test": node.test, "body": node.body, "orelse": node.orelse} + ) class ForParser(Parser): @@ -568,13 +579,31 @@ def __init__(self): def visit_For(self, node): tv = Parser.get_target_vars(node.target) - self.out.append({ - 'node': node, - 'iter': node.iter, - 'body': {'node': node.body, 'target_vars': tv}, - 'orelse': {'node': node.orelse, 'target_vars': tv}, - 'target': node.target, - }) + self.out.append( + { + "node": node, + "iter": node.iter, + "body": {"node": node.body, "target_vars": tv}, + "orelse": {"node": node.orelse, "target_vars": tv}, + "target": node.target, + "_target_vars": tv, + } + ) + + +class ClassDefParser(Parser): + """Find class definitions + """ + + def __init__(self): + self.out = {} + + def visit_ClassDef(self, node): + self.out[node.name] = { + "node": node, + "bases": [{"node": node} for node in node.bases], + "body": node.body, + } class FunctionDefParser(Parser): @@ -583,44 +612,50 @@ class FunctionDefParser(Parser): A parser which inherits from the basic parser to find function definitions. Only 'top-level' for structures will be found! """ + def __init__(self): self.out = {} def visit_FunctionDef(self, node): self.out[node.name] = self.parse_node(node) - @classmethod def parse_node(cls, node): normal_args = cls.get_arg_tuples(node.args.args, node.args.defaults) kwonlyargs = cls.get_arg_tuples(node.args.kwonlyargs, node.args.kw_defaults) # TODO: all single args should be tuples like this vararg = cls.get_arg(node.args.vararg) - kwarg = cls.get_arg(node.args.kwarg) + kwarg = cls.get_arg(node.args.kwarg) # create context variables target_vars = [arg[0] for arg in normal_args] - if vararg: target_vars.append(vararg) - if kwarg: target_vars.append(kwarg) - - args = cls.get_arg_parts(node.args.args, node.args.defaults, 'arg') - kw_args = cls.get_arg_parts(node.args.kwonlyargs, node.args.kw_defaults, 'kwonly') - varargs = cls.get_arg_part(node.args.vararg, None, 'vararg') - kwargs = cls.get_arg_part(node.args.kwarg, None, 'kwarg') + if vararg: + target_vars.append(vararg) + if kwarg: + target_vars.append(kwarg) + + args = cls.get_arg_parts(node.args.args, node.args.defaults, "arg") + kw_args = cls.get_arg_parts( + node.args.kwonlyargs, node.args.kw_defaults, "kwonly" + ) + varargs = cls.get_arg_part(node.args.vararg, None, "vararg") + kwargs = cls.get_arg_part(node.args.kwarg, None, "kwarg") all_args = [*args, varargs, *kw_args, kwargs] - - if isinstance(node, ast.Lambda): body_node = node.body - else: body_node = FunctionBodyTransformer().visit(ast.Module(node.body)) + + if isinstance(node, ast.Lambda): + body_node = node.body + else: + bodyMod = wrap_in_module(node.body) + body_node = FunctionBodyTransformer().visit(bodyMod) return { "node": node, - "name": getattr(node, 'name', None), - "args": IndexedDict([ (p['name'], p) for p in all_args if p is not None]), + "name": getattr(node, "name", None), + "args": IndexedDict([(p["name"], p) for p in all_args if p is not None]), # TODO: arg is the node counterpart to target_vars "_spec1_args": args, "*args": varargs, - "**kwargs": kwargs, - "body": {'node': body_node, - 'target_vars': TargetVars(target_vars)} + "**kwargs": kwargs, + "body": {"node": body_node, "target_vars": TargetVars(target_vars)}, } @@ -646,29 +681,6 @@ def visit_Call(self, node): for key in node.keywords: self.visit(key.value) - def visit_If(self, node): - self.visit_each(node.body) - self.visit_each(node.orelse) - - def visit_While(self, node): - self.visit_each(node.body) - self.visit_each(node.orelse) - - def visit_For(self, node): - self.visit_each(node.body) - self.visit_each(node.orelse) - - def visit_With(self, node): - self.visit_each(node.body) - - def visit_Try(self, node): - self.visit_each(node.body) - self.visit_each(node.finalbody) - - def visit_TryFinally(self, node): - self.visit_each(node.body) - self.visit_each(node.finalbody) - def visit_Lambda(self, node): self.out.append(FunctionDefParser.parse_node(node)) @@ -683,34 +695,30 @@ def visit_Assign(self, node): def visit_AugAssign(self, node): self.visit(node.value) - def visit_Try(self, node): - self.visit_each(node.body) - self.visit_each(node.finalbody) - - def visit_TryFinally(self, node): - self.visit_each(node.body) - self.visit_each(node.finalbody) - def build_comp(self, node): target = node.generators[0].target tv = Parser.get_target_vars(target) ifs = node.generators[0].ifs - self.out.append({ + self.out.append( + { "node": node, - "body": {'node': node.elt, 'target_vars': tv}, + "body": {"node": node.elt, "target_vars": tv}, "target": target, "iter": node.generators[0].iter, - "ifs": [{'node': ifnode, 'target_vars': tv} for ifnode in ifs], + "ifs": [{"node": ifnode, "target_vars": tv} for ifnode in ifs], # TODO: 'private' _target_vars, since it shouldn't be set when selecting node, # see remarks in test_list_comp on rewriting - "_target_vars": tv - }) + "_target_vars": tv, + } + ) + class ListCompParser(CompParser): """Find list comprehensions A parser which inherits from the CompParser to find list comprehensions. """ + def visit_ListComp(self, node): self.build_comp(node) @@ -724,47 +732,49 @@ class GeneratorExpParser(CompParser): def visit_GeneratorExp(self, node): self.build_comp(node) + class DictCompParser(CompParser): """Find dictionary comprehensions A parser which inherits from the CompParser to find dict comprehensions. """ + def visit_DictComp(self, node): target = node.generators[0].target tv = Parser.get_target_vars(target) ifs = node.generators[0].ifs - self.out.append({ + self.out.append( + { "node": node, - "key": {'node': node.key, 'target_vars': tv}, - "value": {'node': node.value, 'target_vars': tv}, + "key": {"node": node.key, "target_vars": tv}, + "value": {"node": node.value, "target_vars": tv}, "target": target, "iter": node.generators[0].iter, - "ifs": [{'node': ifnode, 'target_vars': tv} for ifnode in ifs], + "ifs": [{"node": ifnode, "target_vars": tv} for ifnode in ifs], # TODO: 'private' _target_vars, since it shouldn't be set when selecting node, # see remarks in test_list_comp on rewriting - "_target_vars": tv - }) + "_target_vars": tv, + } + ) class FunctionBodyTransformer(ast.NodeTransformer): # TODO this does not automatically contain line_end information! def visit_Nonlocal(self, node): - new_node = ast.copy_location(ast.Global(names = node.names), node) + new_node = ast.copy_location(ast.Global(names=node.names), node) return FunctionBodyTransformer.decorate(new_node, node) def visit_Return(self, node): - new_node = ast.copy_location(ast.Expr(value = node.value), node) + new_node = ast.copy_location(ast.Expr(value=node.value), node) return FunctionBodyTransformer.decorate(new_node, node) + @staticmethod def decorate(new_node, node): - try: - # only possible on the student side! - new_node.end_lineno = node.end_lineno - new_node.end_col_offset = node.end_col_offset - except: - pass + new_node.first_token = node.first_token + new_node.last_token = node.last_token return new_node + class WithParser(Parser): def __init__(self): self.out = [] @@ -772,22 +782,26 @@ def __init__(self): def visit_With(self, node): items = node.items context = [ - {"node" : item.context_expr, - "target_vars": self.get_target_vars(item.optional_vars), - "with_items": item, - "highlight": node - } - for item in items] - - #body_tv = [] - #for c in context: body_tv.extend(c['target_vars']) - - self.out.append({ - "context": context, - "body": {'node': node.body, 'with_items': items}, - "node": node, - "n_vars": len(items) - }) + { + "node": item.context_expr, + "target_vars": self.get_target_vars(item.optional_vars), + "with_items": item, + "highlight": node, + } + for item in items + ] + + tv_all = TargetVars(sum([list(c["target_vars"].items()) for c in context], [])) + + self.out.append( + { + "context": context, + "body": {"node": node.body, "with_items": items}, + "node": node, + "n_vars": len(items), + } + ) + class TryExceptParser(Parser): def __init__(self): @@ -803,38 +817,52 @@ def visit_Try(self, node): handlers[el.id] = self.parse_handler(handler) else: # either general handler, or single error handler - k = 'all' if not handler.type else handler.type.id + k = "all" if not handler.type else self.get_identifier(handler.type) handlers[k] = self.parse_handler(handler) - self.out.append({ - "node": node, - "body": node.body, - "orelse": node.orelse or None, - "finalbody": node.finalbody or None, - "handlers": handlers, - }) + self.out.append( + { + "node": node, + "body": node.body, + "orelse": node.orelse or None, + "finalbody": node.finalbody or None, + "handlers": handlers, + } + ) + + @staticmethod + def get_identifier(obj): + # if only name needs to be correct + return getattr(obj, "id", None) or getattr(obj, "attr") + # if full attr chain must match + # name = "" + # while hasattr(obj, "attr"): + # name = "." + obj.attr + name + # obj = obj.value + # name = obj.id + name + # return name @staticmethod - def parse_handler(handler): return { - 'node': handler.body, - 'target_vars': TargetVars([handler.name]) - } + def parse_handler(handler): + return {"node": handler.body, "target_vars": TargetVars([handler.name])} + parser_dict = { - "object_accesses": ObjectAccessParser, - "object_assignments": ObjectAssignmentParser, - "operators": OperatorParser, - "imports": ImportParser, - "if_elses": IfParser, - "if_exps": IfExpParser, - "whiles": WhileParser, - "for_loops": ForParser, - "function_defs": FunctionDefParser, - "lambda_functions": LambdaFunctionParser, - "list_comps": ListCompParser, - "dict_comps": DictCompParser, - "generator_exps": GeneratorExpParser, - "withs": WithParser, - "try_excepts": TryExceptParser, - "function_calls": FunctionParser + "object_accesses": ObjectAccessParser, + "object_assignments": ObjectAssignmentParser, + # "operators": OperatorParser, + "imports": ImportParser, + "if_elses": IfParser, + "if_exps": IfExpParser, + "whiles": WhileParser, + "for_loops": ForParser, + "class_defs": ClassDefParser, + "function_defs": FunctionDefParser, + "lambda_functions": LambdaFunctionParser, + "list_comps": ListCompParser, + "dict_comps": DictCompParser, + "generator_exps": GeneratorExpParser, + "withs": WithParser, + "try_excepts": TryExceptParser, + "function_calls": FunctionParser, } diff --git a/pythonwhat/probe.py b/pythonwhat/probe.py index a2b916de..9a041e4d 100644 --- a/pythonwhat/probe.py +++ b/pythonwhat/probe.py @@ -3,54 +3,45 @@ import inspect from functools import partial from collections import OrderedDict +from pythonwhat import test_funcs +from pythonwhat.State import State +from pythonwhat.checks.check_wrappers import state_partial TEST_NAMES = [ "test_mc", "test_or", "test_with", - "test_list_comp", - "test_dict_comp", - "test_generator_exp", "test_import", "test_object", "test_correct", "test_if_else", - "test_if_exp", "test_for_loop", "test_function", - "test_print", + "test_list_comp", "test_function_v2", - "test_operator", - "test_try_except", "test_data_frame", - "test_dictionary", "test_while_loop", "test_student_typed", "test_object_accessed", "test_output_contains", - "test_lambda_function", "test_expression_result", "test_expression_output", "test_function_definition", - "test_object_after_expression" + "test_object_after_expression", ] SUB_TESTS = { - "test_if_else": ['test', 'body', 'orelse'], - "test_if_exp": ['test', 'body', 'orelse'], - "test_list_comp": ['comp_iter', 'body', 'ifs'], - "test_dict_comp": ['comp_iter', 'key', 'value', 'ifs'], - "test_correct": ['check', 'diagnose'], - "test_generator_exp": ['comp_iter', 'body', 'ifs'], - "test_for_loop": ['for_iter', 'body', 'orelse'], - "test_try_except": ['body', 'handlers', 'orelse', 'finalbody'], - "test_while_loop": ['test', 'body', 'orelse'], - "test_with": ['context_tests', 'body'], - "test_function_definition": ['body'], - "test_or": ['tests'], - "test_lambda_function": ['body'] + "test_if_else": ["test", "body", "orelse"], + "test_list_comp": ["comp_iter", "body", "ifs"], + "check_correct": ["check", "diagnose"], + "test_for_loop": ["for_iter", "body", "orelse"], + "test_while_loop": ["test", "body", "orelse"], + "test_with": ["context_tests", "body"], + "test_function_definition": ["body"], + "check_or": ["tests"], } + class Tree(object): def __init__(self): """ @@ -66,10 +57,19 @@ def __init__(self): self.crnt_node = self.root @classmethod - def str_branch(cls, node, str_func=lambda s: ""): - f = node.data.get('func') - this_node = " "*node.depth + getattr(f, '__name__', node.name) + str_func(node) + "\n" - return this_node + "".join(map(lambda x: cls.str_branch(x, str_func), node.child_list)) + def str_branch(cls, node, str_func=lambda s: str(s)): + # dict(getattr(s.data.get("bound_args", {}), 'arguments', {})) + f = node.data.get("func") + this_node = ( + " " * node.depth + + "(" + + getattr(f, "__name__", node.name) + + str_func(node) + + ")\n" + ) + return this_node + "".join( + map(lambda x: cls.str_branch(x, str_func), node.child_list) + ) def __str__(self): return self.str_branch(self.crnt_node) @@ -81,10 +81,12 @@ def descend(self, node=None): return sum(children, base) def __iter__(self): - for ii in self.descend(self.crnt_node): yield ii + for ii in self.descend(self.crnt_node): + yield ii + class Node(object): - def __init__(self, child_list = None, data = None, name="unnamed", arg_name=""): + def __init__(self, child_list=None, data=None, name="unnamed", arg_name=""): """ Hold a function call with its bound arguments, along with child nodes. @@ -99,45 +101,55 @@ def __init__(self, child_list = None, data = None, name="unnamed", arg_name=""): def __call__(self, state=None): """Call original function with its arguments, and optional state""" - ba = self.data['bound_args'] + ba = self.data["bound_args"] if state: - self.data['func'](state=state, *ba.args, **ba.kwargs) + func = self.data["func"] + ba = inspect.signature(func).bind(state, *ba.args[1:], **ba.kwargs) + func(*ba.args, **ba.kwargs) return state else: - self.data['func'](*ba.args, **ba.kwargs) + self.data["func"](*ba.args, **ba.kwargs) ba.apply_defaults() - return ba.arguments['state'] + return ba.arguments["state"] + + def __str__(self): + return pp.pformat( + dict(getattr(self.data.get("bound_args", {}), "arguments", {})) + ) def __iter__(self): - return iter(self.child_list) + for c in self.child_list: + yield c def partial(self): """Return partial of original function call""" - ba = self.data['bound_args'] - return partial(self.data['func'], *ba.args, **ba.kwargs) + ba = self.data["bound_args"] + return state_partial(self.data["func"], *ba.args[1:], **ba.kwargs) def update_child_calls(self): """Replace child nodes on original function call with their partials""" for node in filter(lambda n: len(n.arg_name), self.child_list): - self.data['bound_args'].arguments[node.arg_name] = node.partial() + self.data["bound_args"].arguments[node.arg_name] = node.partial() self.updated = True def remove_child(self, node): - indx = self.child_list.index(node) - del self.child_list[indx] - return indx + index = self.child_list.index(node) + del self.child_list[index] + return index def add_child(self, child): - # since it is a tree, there is only one parent + # since it is a tree, there is only one parent # note this means we do not allow edges between same layer units - if child.parent: child.parent.remove_child(child) + if child.parent: + child.parent.remove_child(child) child.parent = self self.child_list.append(child) def descend(self, include_me=True): """Descend depth first into all child nodes""" - if include_me: yield self + if include_me: + yield self for child in self.child_list: yield child @@ -145,15 +157,11 @@ def descend(self, include_me=True): @property def depth(self): - if self.parent: return self.parent.depth + 1 - else: return 0 - - def __str__(self): - # TODO print function signature without defaults (or with) - return pp.pformat(self.data) + if self.parent: + return self.parent.depth + 1 + else: + return 0 - def __iter__(self): - for c in self.child_list: yield c class NodeList(Node): def partial(self): @@ -162,12 +170,6 @@ def partial(self): def update_child_calls(self): pass -class NodeDict(Node): - def partial(self): - return OrderedDict((node.arg_name, node.partial()) for node in self.child_list) - - def update_child_calls(self): - pass class Probe(object): def __init__(self, tree, f, eval_on_call=False): @@ -177,7 +179,7 @@ def __init__(self, tree, f, eval_on_call=False): self.eval_on_call = eval_on_call # TODO: auto sub_test detection self.sub_tests = SUB_TESTS.get(self.test_name) or [] - + def __call__(self, *args, **kwargs): """Bind arguments to original function signature, and store in a Node @@ -186,52 +188,55 @@ def __call__(self, *args, **kwargs): instances are assembled into a tree. """ + if (len(args) > 0 and not isinstance(args[0], State)) or len(args) == 0: + # no state placeholder if a state is passed + args = ["state_placeholder"] + list(args) bound_args = inspect.signature(self.f).bind(*args, **kwargs) - data = dict( - bound_args = bound_args, - func = self.f) + data = dict(bound_args=bound_args, func=self.f) this_node = Node(data=data, name=self.test_name) if self.tree is not None: self.tree.crnt_node.add_child(this_node) # First pass to set up branches off node - da = bound_args.arguments - for st in self.sub_tests: # TODO: auto sub test detection - if st in da and da[st]: - self.build_sub_test_nodes(da[st], self.tree, this_node, st) + arguments = bound_args.arguments + for subtest in self.sub_tests: # TODO: auto sub test detection + if subtest in arguments and arguments[subtest]: + self.build_sub_test_nodes( + arguments[subtest], self.tree, this_node, subtest + ) + # Second pass to build node and all its children into a subtest - for n in this_node.descend(include_me=True): - if n.updated: # already built, e.g. node used multiple times + for node in this_node.descend(include_me=True): + if node.updated: # already built, e.g. node used multiple times continue else: - n.update_child_calls() - - if self.eval_on_call: return this_node() - else: return this_node + node.update_child_calls() + + if self.eval_on_call: + return this_node() + else: + return this_node @staticmethod def build_sub_test_nodes(test, tree, node, arg_name): # note that I've made the strong assumption that # if not a function, then test is a dict, list or tuple of them - if isinstance(test, dict): - nd = NodeDict(name = "Dict", arg_name = arg_name) - node.add_child(nd) - for k, f in test.items(): Probe.build_sub_test_nodes(f, tree, nd, k) - elif isinstance(test, (list, tuple)): - nl = NodeList(name = "List", arg_name = arg_name) + if isinstance(test, (list, tuple)): + nl = NodeList(name="List", arg_name=arg_name) node.add_child(nl) - for ii, f in enumerate(test): Probe.build_sub_test_nodes(f, tree, nl, str(ii)) - elif isinstance(test, Node): + for ii, f in enumerate(test): + Probe.build_sub_test_nodes(f, tree, nl, str(ii)) + elif isinstance(test, Node): # test was a lambdaless subtest call, which produced a node # so need to tell it what its arg_name was on parent test test.arg_name = arg_name node.add_child(test) - elif callable(test): - # test was inside a lambda or function containing subtests + elif callable(test): + # test was inside a lambda, function containing subtests or v2 F() chain object with subtests # since either may contain multiple subtests, we put them in a node list - nl = NodeList(name = "ListDeferred", arg_name = arg_name) + nl = NodeList(name="ListDeferred", arg_name=arg_name) node.add_child(nl) if tree is not None: prev_node, tree.crnt_node = tree.crnt_node, nl @@ -243,10 +248,7 @@ def build_sub_test_nodes(test, tree, node, arg_name): raise Exception("Expected a function or list/tuple/dict of functions") -def create_test_probes(context): +def build_probe_context(): tree = Tree() - all_tests = [context[s] for s in TEST_NAMES] - new_context = {f.__name__: Probe(tree, f) for f in all_tests} - new_context.update({k:v for k,v in context.items() if k not in new_context}) - #new_context['success_msg'] = lambda s: s - return tree, new_context + probe_context = {s: Probe(tree, getattr(test_funcs, s)) for s in TEST_NAMES} + return tree, probe_context diff --git a/pythonwhat/sct_syntax.py b/pythonwhat/sct_syntax.py new file mode 100644 index 00000000..2ff2af6c --- /dev/null +++ b/pythonwhat/sct_syntax.py @@ -0,0 +1,70 @@ +from protowhat.sct_syntax import EagerChain, ExGen, LazyChainStart, state_dec_gen, LazyChain +from pythonwhat.checks.check_wrappers import scts +from pythonwhat.State import State +from pythonwhat.probe import Node, Probe, TEST_NAMES +from pythonwhat.utils import include_v1 +from pythonwhat import test_funcs +from functools import wraps + +# TODO: could define scts for check_wrappers at the module level +sct_dict = scts.copy() + + +def multi_dec(f): + """Decorator for multi to remove nodes for original test functions from root node""" + + @wraps(f) + def wrapper(*args, **kwargs): + args = ( + args[0] if len(args) == 1 and isinstance(args[0], (list, tuple)) else args + ) + for arg in args: + if isinstance(arg, Node) and arg.parent.name == "root": + arg.parent.remove_child(arg) + arg.update_child_calls() + return f(*args, **kwargs) + + return wrapper + + +state_dec = state_dec_gen(sct_dict) + +# todo: __all__? +assert ExGen +assert LazyChainStart + + +def Ex(state=None): + return EagerChain(state=state or State.root_state, chainable_functions=sct_dict) + + +def F(): + return LazyChain(chainable_functions=sct_dict) + + +def get_chains(): + return { + "Ex": ExGen(sct_dict, State.root_state), + "F": LazyChainStart(sct_dict), + } + + +if include_v1(): + # Prepare SCTs that may be chained attributes ---------------------- + # decorate functions that may try to run test_* function nodes as subtests + # so they remove those nodes from the tree + for k in ["multi", "with_context"]: + sct_dict[k] = multi_dec(sct_dict[k]) + + # allow test_* functions as chained attributes + for k in TEST_NAMES: + sct_dict[k] = Probe(tree=None, f=getattr(test_funcs, k), eval_on_call=True) + + # original logical test_* functions behave like multi + # this is necessary to allow them to take check_* funcs as args + # since probe behavior will try to call all SCTs passed (assuming they're also probes) + for k in ["test_or", "test_correct"]: + sct_dict[k] = multi_dec(getattr(test_funcs, k)) + +# Prepare check_funcs to be used alone (e.g. test = check_with().check_body()) +v2_check_functions = {k: state_dec(v) for k, v in scts.items()} diff --git a/pythonwhat/signatures.py b/pythonwhat/signatures.py index d7ea8a24..94ffaa72 100644 --- a/pythonwhat/signatures.py +++ b/pythonwhat/signatures.py @@ -3,118 +3,175 @@ import pythonwhat from pythonwhat.tasks import getSignatureFromObjInProcess + def sig_from_params(*args): - return(inspect.Signature(list(args))) + return inspect.Signature(list(args)) + def sig_from_obj(obj_char): - return getSignatureFromObjInProcess(obj_char, - pythonwhat.State.State.root_state.solution_process) + return getSignatureFromObjInProcess( + obj_char, pythonwhat.State.State.root_state.solution_process + ) + def get_manual_sigs(): manual_sigs = { # builtins - 'abs': [param('x', param.POSITIONAL_ONLY)], - 'all': [param('iterable', param.POSITIONAL_ONLY)], - 'any': [param('iterable', param.POSITIONAL_ONLY)], - 'ascii': [param('obj', param.POSITIONAL_ONLY)], - 'bin': [param('number', param.POSITIONAL_ONLY)], - 'bool': [param('x', param.POSITIONAL_OR_KEYWORD)], - 'chr': [param('i', param.POSITIONAL_ONLY)], - 'callable': [param('obj', param.POSITIONAL_ONLY)], - 'classmethod': [param('function', param.POSITIONAL_ONLY)], - 'complex': [param('imag', param.POSITIONAL_OR_KEYWORD, default=0), - param('real', param.POSITIONAL_OR_KEYWORD, default=0)], - 'delattr': [param('obj', param.POSITIONAL_ONLY), - param('name', param.POSITIONAL_ONLY)], - 'dir': [param('object', param.POSITIONAL_OR_KEYWORD, default=None)], - 'divmod': [param('x', param.POSITIONAL_ONLY), - param('y', param.POSITIONAL_ONLY)], - 'enumerate': [param('iterable', param.POSITIONAL_ONLY), - param('start', param.POSITIONAL_OR_KEYWORD, default=0)], - 'float': [param('x', param.POSITIONAL_OR_KEYWORD)], - 'getattr': [param('object', param.POSITIONAL_ONLY), - param('name', param.POSITIONAL_ONLY), - param('default', param.POSITIONAL_ONLY, default=None)], - 'hasattr': [param('obj', param.POSITIONAL_ONLY), - param('name', param.POSITIONAL_ONLY)], - 'hash': [param('obj', param.POSITIONAL_ONLY)], - 'hex': [param('number', param.POSITIONAL_ONLY)], - 'id': [param('obj', param.POSITIONAL_ONLY)], - 'int': [param('x', param.POSITIONAL_OR_KEYWORD), - param('base', param.POSITIONAL_OR_KEYWORD, default=10)], - 'isinstance': [param('obj', param.POSITIONAL_ONLY), - param('class_or_tuple', param.POSITIONAL_ONLY)], - 'issubclass': [param('cls', param.POSITIONAL_ONLY), - param('class_or_tuple', param.POSITIONAL_ONLY)], - 'list': [param('iterable', param.POSITIONAL_ONLY, default=None)], - 'len': [param('obj', param.POSITIONAL_ONLY)], - 'oct': [param('number', param.POSITIONAL_ONLY)], - 'open': [param('file', param.POSITIONAL_OR_KEYWORD), - param('mode', param.POSITIONAL_OR_KEYWORD, default='r'), - param('buffering', param.POSITIONAL_OR_KEYWORD, default=1), - param('encoding', param.POSITIONAL_OR_KEYWORD, default=None), - param('errors', param.POSITIONAL_OR_KEYWORD, default=None), - param('newline', param.POSITIONAL_OR_KEYWORD, default=None), - param('closefd', param.POSITIONAL_OR_KEYWORD, default=None), - param('opener', param.POSITIONAL_OR_KEYWORD, default=None)], - 'ord': [param('c', param.POSITIONAL_ONLY)], - 'pow': [param('x', param.POSITIONAL_ONLY), - param('y', param.POSITIONAL_ONLY), - param('z', param.POSITIONAL_ONLY, default=None)], - 'print': [param('value', param.POSITIONAL_ONLY)], - 'repr': [param('obj', param.POSITIONAL_ONLY)], - 'reversed': [param('sequence', param.POSITIONAL_ONLY)], - 'round': [param('number', param.POSITIONAL_OR_KEYWORD), - param('ndigits', param.POSITIONAL_OR_KEYWORD, default=0)], - 'set': [param('iterable', param.POSITIONAL_ONLY, default=None)], - + "abs": [param("x", param.POSITIONAL_ONLY)], + "all": [param("iterable", param.POSITIONAL_ONLY)], + "any": [param("iterable", param.POSITIONAL_ONLY)], + "ascii": [param("obj", param.POSITIONAL_ONLY)], + "bin": [param("number", param.POSITIONAL_ONLY)], + "bool": [param("x", param.POSITIONAL_OR_KEYWORD)], + "chr": [param("i", param.POSITIONAL_ONLY)], + "callable": [param("obj", param.POSITIONAL_ONLY)], + "classmethod": [param("function", param.POSITIONAL_ONLY)], + "complex": [ + param("imag", param.POSITIONAL_OR_KEYWORD, default=0), + param("real", param.POSITIONAL_OR_KEYWORD, default=0), + ], + "delattr": [ + param("obj", param.POSITIONAL_ONLY), + param("name", param.POSITIONAL_ONLY), + ], + "dir": [param("object", param.POSITIONAL_OR_KEYWORD, default=None)], + "divmod": [ + param("x", param.POSITIONAL_ONLY), + param("y", param.POSITIONAL_ONLY), + ], + "enumerate": [ + param("iterable", param.POSITIONAL_ONLY), + param("start", param.POSITIONAL_OR_KEYWORD, default=0), + ], + "float": [param("x", param.POSITIONAL_OR_KEYWORD)], + "getattr": [ + param("object", param.POSITIONAL_ONLY), + param("name", param.POSITIONAL_ONLY), + param("default", param.POSITIONAL_ONLY, default=None), + ], + "hasattr": [ + param("obj", param.POSITIONAL_ONLY), + param("name", param.POSITIONAL_ONLY), + ], + "hash": [param("obj", param.POSITIONAL_ONLY)], + "hex": [param("number", param.POSITIONAL_ONLY)], + "id": [param("obj", param.POSITIONAL_ONLY)], + "int": [ + param("x", param.POSITIONAL_OR_KEYWORD), + param("base", param.POSITIONAL_OR_KEYWORD, default=10), + ], + "isinstance": [ + param("obj", param.POSITIONAL_ONLY), + param("class_or_tuple", param.POSITIONAL_ONLY), + ], + "issubclass": [ + param("cls", param.POSITIONAL_ONLY), + param("class_or_tuple", param.POSITIONAL_ONLY), + ], + "list": [param("iterable", param.POSITIONAL_ONLY, default=None)], + "len": [param("obj", param.POSITIONAL_ONLY)], + "oct": [param("number", param.POSITIONAL_ONLY)], + "open": [ + param("file", param.POSITIONAL_OR_KEYWORD), + param("mode", param.POSITIONAL_OR_KEYWORD, default="r"), + param("buffering", param.POSITIONAL_OR_KEYWORD, default=1), + param("encoding", param.POSITIONAL_OR_KEYWORD, default=None), + param("errors", param.POSITIONAL_OR_KEYWORD, default=None), + param("newline", param.POSITIONAL_OR_KEYWORD, default=None), + param("closefd", param.POSITIONAL_OR_KEYWORD, default=None), + param("opener", param.POSITIONAL_OR_KEYWORD, default=None), + ], + "ord": [param("c", param.POSITIONAL_ONLY)], + "pow": [ + param("x", param.POSITIONAL_ONLY), + param("y", param.POSITIONAL_ONLY), + param("z", param.POSITIONAL_ONLY, default=None), + ], + "print": [param("value", param.POSITIONAL_ONLY)], + "repr": [param("obj", param.POSITIONAL_ONLY)], + "reversed": [param("sequence", param.POSITIONAL_ONLY)], + "round": [ + param("number", param.POSITIONAL_OR_KEYWORD), + param("ndigits", param.POSITIONAL_OR_KEYWORD, default=0), + ], + "set": [param("iterable", param.POSITIONAL_ONLY, default=None)], # Difference v3.4 vs v3.5!!! - 'setattr': [param('obj', param.POSITIONAL_ONLY), - param('name', param.POSITIONAL_ONLY), - param('value', param.POSITIONAL_ONLY)], - 'sorted': [param('iterable', param.POSITIONAL_ONLY), - param('key', param.POSITIONAL_OR_KEYWORD, default=None), - param('reverse', param.POSITIONAL_OR_KEYWORD, default=False)], - 'str': [param('object', param.POSITIONAL_OR_KEYWORD)], - 'sum': [param('iterable', param.POSITIONAL_ONLY), - param('start', param.POSITIONAL_ONLY, default=0)], - 'tuple': [param('iterable', param.POSITIONAL_ONLY, default=None)], - 'type': [param('object', param.POSITIONAL_ONLY)], - 'vars': [param('object', param.POSITIONAL_ONLY)], - + "setattr": [ + param("obj", param.POSITIONAL_ONLY), + param("name", param.POSITIONAL_ONLY), + param("value", param.POSITIONAL_ONLY), + ], + "sorted": [ + param("iterable", param.POSITIONAL_ONLY), + param("key", param.POSITIONAL_OR_KEYWORD, default=None), + param("reverse", param.POSITIONAL_OR_KEYWORD, default=False), + ], + "str": [param("object", param.POSITIONAL_OR_KEYWORD)], + "sum": [ + param("iterable", param.POSITIONAL_ONLY), + param("start", param.POSITIONAL_ONLY, default=0), + ], + "tuple": [param("iterable", param.POSITIONAL_ONLY, default=None)], + "type": [param("object", param.POSITIONAL_ONLY)], + "vars": [param("object", param.POSITIONAL_ONLY)], # int - # str - 'str.center': [param('width', param.POSITIONAL_ONLY), - param('fillchar', param.POSITIONAL_ONLY, default=" ")], - + "str.center": [ + param("width", param.POSITIONAL_ONLY), + param("fillchar", param.POSITIONAL_ONLY, default=" "), + ], # list - 'list.append': [param('object', param.POSITIONAL_ONLY)], - 'list.count': [param('value', param.POSITIONAL_ONLY)], - + "list.append": [param("object", param.POSITIONAL_ONLY)], + "list.count": [param("value", param.POSITIONAL_ONLY)], # dict - # numpy - 'numpy.array': [param('object', param.POSITIONAL_OR_KEYWORD), - param('dtype', param.POSITIONAL_OR_KEYWORD, default=None), - param('copy', param.POSITIONAL_OR_KEYWORD, default=True), - param('order', param.POSITIONAL_OR_KEYWORD, default=None), - param('subok', param.POSITIONAL_OR_KEYWORD, default=False), - param('ndmin', param.POSITIONAL_OR_KEYWORD, default=0)], - 'numpy.random.seed': [param('seed', param.POSITIONAL_OR_KEYWORD, default=None)], - 'numpy.random.rand': [param('d0', param.POSITIONAL_ONLY, default=None), - param('d1', param.POSITIONAL_ONLY, default=None), - param('d2', param.POSITIONAL_ONLY, default=None), - param('d3', param.POSITIONAL_ONLY, default=None), - param('d4', param.POSITIONAL_ONLY, default=None), - param('d5', param.POSITIONAL_ONLY, default=None), - param('d6', param.POSITIONAL_ONLY, default=None)], - 'numpy.random.randint': [param('low', param.POSITIONAL_OR_KEYWORD), - param('high', param.POSITIONAL_OR_KEYWORD, default=None), - param('size', param.POSITIONAL_OR_KEYWORD, default=None), - param('dtype', param.POSITIONAL_OR_KEYWORD, default='l')], - + "numpy.array": [ + param("object", param.POSITIONAL_OR_KEYWORD), + param("dtype", param.POSITIONAL_OR_KEYWORD, default=None), + param("copy", param.POSITIONAL_OR_KEYWORD, default=True), + param("order", param.POSITIONAL_OR_KEYWORD, default=None), + param("subok", param.POSITIONAL_OR_KEYWORD, default=False), + param("ndmin", param.POSITIONAL_OR_KEYWORD, default=0), + ], + "numpy.random.seed": [param("seed", param.POSITIONAL_OR_KEYWORD, default=None)], + "numpy.random.rand": [ + param("d0", param.POSITIONAL_ONLY, default=None), + param("d1", param.POSITIONAL_ONLY, default=None), + param("d2", param.POSITIONAL_ONLY, default=None), + param("d3", param.POSITIONAL_ONLY, default=None), + param("d4", param.POSITIONAL_ONLY, default=None), + param("d5", param.POSITIONAL_ONLY, default=None), + param("d6", param.POSITIONAL_ONLY, default=None), + ], + "numpy.random.randint": [ + param("low", param.POSITIONAL_OR_KEYWORD), + param("high", param.POSITIONAL_OR_KEYWORD, default=None), + param("size", param.POSITIONAL_OR_KEYWORD, default=None), + param("dtype", param.POSITIONAL_OR_KEYWORD, default="l"), + ], + "numpy.random.choice": [ + param("a", param.POSITIONAL_OR_KEYWORD), + param("size", param.POSITIONAL_OR_KEYWORD, default=None), + param("replace", param.POSITIONAL_OR_KEYWORD, default=True), + param("p", param.POSITIONAL_OR_KEYWORD, default=None), + ], + "numpy.random.normal": [ + param("loc", param.POSITIONAL_OR_KEYWORD, default=0.0), + param("scale", param.POSITIONAL_OR_KEYWORD, default=1.0), + param("size", param.POSITIONAL_OR_KEYWORD, default=None), + ], + "numpy.random.poisson": [ + param("lam", param.POSITIONAL_OR_KEYWORD, default=1.0), + param("size", param.POSITIONAL_OR_KEYWORD, default=None), + ], + "numpy.random.binomial": [ + param("n", param.POSITIONAL_OR_KEYWORD), + param("p", param.POSITIONAL_OR_KEYWORD), + param("size", param.POSITIONAL_OR_KEYWORD, default=None), + ], + "numpy.random.shuffle": [param("x", param.POSITIONAL_OR_KEYWORD)], + "numpy.random.permutation": [param("x", param.POSITIONAL_OR_KEYWORD)], # others - 'math.radians': [param('x', param.POSITIONAL_ONLY)] + "math.radians": [param("x", param.POSITIONAL_ONLY)], } - return(manual_sigs) + return manual_sigs diff --git a/pythonwhat/tasks.py b/pythonwhat/tasks.py index 53c13dc1..3516ff7b 100644 --- a/pythonwhat/tasks.py +++ b/pythonwhat/tasks.py @@ -1,45 +1,55 @@ from pythonwhat import utils -import os import dill import pickle import pythonwhat import ast import inspect -import copy +from copy import deepcopy from pickle import PicklingError from pythonwhat.utils_env import set_context_vals, assign_from_ast from contextlib import contextmanager -from functools import partial, wraps, update_wrapper +from functools import partial, wraps +from protowhat.failure import InstructorError + + +# Shell is passed as a parameter to partially applied functions in executeTask +# Process is passed as a parameter in SCT function + def process_task(f): """Decorator to (optionally) run function in a process.""" sig = inspect.signature(f) + @wraps(f) def wrapper(*args, **kwargs): # get bound arguments for call ba = sig.bind_partial(*args, **kwargs) # when process is specified, remove from args and use to execute - process = ba.arguments.get('process') + process = ba.arguments.get("process") if process: - ba.arguments['process'] = None + ba.arguments["process"] = None # partial function since shell argument may have been left # unspecified, as it will be passed when the process executes pf = partial(wrapper, *ba.args, **ba.kwargs) return process.executeTask(pf) # otherwise, run original function return f(*ba.args, **ba.kwargs) + return wrapper + def get_env(ns): - if '__env__' in ns: - return ns['__env__'] + if "__env__" in ns: + return ns["__env__"] else: return ns + @contextmanager def capture_output(): import sys from io import StringIO + oldout, olderr = sys.stdout, sys.stderr out = [StringIO(), StringIO()] sys.stdout, sys.stderr = out @@ -49,14 +59,6 @@ def capture_output(): out[1] = out[1].getvalue() -## DEBUGGING - -# import pythonwhat; pythonwhat.tasks.listElementsInProcess(state.student_process) -@process_task -def listElementsInProcess(process, shell): - return list(get_env(shell.user_ns).keys()) - - # MC @process_task def getOptionFromProcess(process, name, shell): @@ -68,48 +70,41 @@ def getOptionFromProcess(process, name, shell): def isDefinedInProcess(name, process, shell): return name in get_env(shell.user_ns) + # Is a variable is of a certain class in the process? @process_task def isInstanceInProcess(name, klass, process, shell): return isinstance(get_env(shell.user_ns)[name], klass) -# Get the keys() of a dictionary in the process -@process_task -def getKeysInProcess(name, process, shell): - try: - return list(get_env(shell.user_ns)[name].keys()) - except: - return None - # Get the columns of a Pandas data frame in the process @process_task def getColumnsInProcess(name, process, shell): - try: - return list(get_env(shell.user_ns)[name].columns) - except: - return None + return list(get_env(shell.user_ns)[name].columns) + # Is a key defined in a collection in the process? @process_task def isDefinedCollInProcess(name, key, process, shell): return key in get_env(shell.user_ns)[name] + # Get the signature of a function inside the process + def get_signature(name, mapped_name, signature, manual_sigs, env): if isinstance(signature, str): if signature in manual_sigs: signature = inspect.Signature(manual_sigs[signature]) else: - raise ValueError('signature error - specified signature not found') + raise InstructorError.from_message("signature error - specified signature not found") if signature is None: # establish function try: fun = eval(mapped_name, env) except: - raise ValueError("%s() was not found." % mapped_name) + raise InstructorError.from_message("%s() was not found." % mapped_name) # first go through manual sigs # try to get signature @@ -124,33 +119,36 @@ def get_signature(name, mapped_name, signature, manual_sigs, env): els[0] = type(eval(els[0], env)).__name__ generic_name = ".".join(els[:]) except: - raise ValueError('signature error - cannot convert call') + raise InstructorError.from_message("signature error - cannot convert call") if generic_name in manual_sigs: signature = inspect.Signature(manual_sigs[generic_name]) else: - raise ValueError('signature error - %s not in builtins' % generic_name) + raise InstructorError.from_message( + "signature error - %s not in builtins" % generic_name + ) else: - raise ValueError('manual signature not found') - except: + raise InstructorError.from_message("manual signature not found") + except Exception as e: try: signature = inspect.signature(fun) except: - raise ValueError('signature error - cannot determine signature') + raise InstructorError.from_message(e.args[0] + " and cannot determine signature") - return signature + params = [param.replace(annotation=inspect._empty) for param in signature.parameters.values()] + return signature.replace(parameters=params) # Get the signature of a function based on an object inside the process @process_task def getSignatureInProcess(name, mapped_name, signature, manual_sigs, process, shell): - try: - return get_signature(name = name, - mapped_name = mapped_name, - signature = signature, - manual_sigs = manual_sigs, - env = get_env(shell.user_ns)) - except: - return None + return get_signature( + name=name, + mapped_name=mapped_name, + signature=signature, + manual_sigs=manual_sigs, + env=get_env(shell.user_ns), + ) + @process_task def getSignatureFromObjInProcess(obj_char, process, shell): @@ -160,17 +158,17 @@ def getSignatureFromObjInProcess(obj_char, process, shell): return None - - # Stuff for test_with from contextlib import ExitStack + + def context_env_update(context_list, env): es = ExitStack() for item in context_list: # create context manager and enter - tmp_name = '__pw_cm' - cm_code = compile(ast.Expression(item.context_expr), '', 'eval') + tmp_name = "__pw_cm" + cm_code = compile(ast.Expression(item.context_expr), "", "eval") env[tmp_name] = es.enter_context(eval(cm_code, env)) # assign to its optional_vars in separte dict @@ -183,33 +181,35 @@ def context_env_update(context_list, env): @process_task def setUpNewEnvInProcess(context, process, shell): - shell.user_ns['__env__'] = utils.copy_env(shell.user_ns) + shell.user_ns["__env__"] = utils.copy_env(shell.user_ns) try: - es = context_env_update(context, shell.user_ns['__env__']) - shell.user_ns['__exit_stack__'] = es + es = context_env_update(context, shell.user_ns["__env__"]) + shell.user_ns["__exit_stack__"] = es return True except Exception as e: return e + # break down environment def context_objs_exit(es): try: es.close() - return False + return True except Exception as e: - raise e return e + @process_task def breakDownNewEnvInProcess(process, shell): try: - res = context_objs_exit(shell.user_ns['__exit_stack__']) - del shell.user_ns['__exit_stack__'] - del shell.user_ns['__env__'] + res = context_objs_exit(shell.user_ns["__exit_stack__"]) + del shell.user_ns["__exit_stack__"] + del shell.user_ns["__env__"] return res except: return False + # Tasks that may need to serialize across processes =========================== # Get a bytes or string representation of an object in the process @@ -222,10 +222,12 @@ def getClass(name, process, shell): except: return None + @process_task def convert(name, converter, process, shell): return dill.loads(converter)(get_env(shell.user_ns)[name]) + @process_task def getStreamPickle(name, process, shell): try: @@ -233,6 +235,7 @@ def getStreamPickle(name, process, shell): except: return None + @process_task def getStreamDill(name, process, shell): try: @@ -241,70 +244,87 @@ def getStreamDill(name, process, shell): return None -class ReprFail(object): +class ReprFail: def __init__(self, info): self.info = info + def getRepresentation(name, process): obj_class = getClass(name, process) converters = pythonwhat.State.State.root_state.converters if obj_class in converters: repres = convert(name, dill.dumps(converters[obj_class]), process) - if (errored(repres)): - return ReprFail("manual conversion failed") - else: + if errored(repres): + return ReprFail("manual conversion failed: {}".format(repres)) + else: return repres else: # first try to pickle try: stream = getStreamPickle(name, process) - if not errored(stream): return pickle.loads(stream) - except: + if not errored(stream): + return pickle.loads(stream) + except: pass # if it failed, try to dill try: stream = getStreamDill(name, process) - if not errored(stream): return dill.loads(stream) - return ReprFail("dilling inside process failed for %s - write manual converter" % obj_class) + if not errored(stream): + return dill.loads(stream) + return ReprFail( + "dilling inside process failed for %s - write manual converter" + % obj_class + ) except PicklingError: - return ReprFail("undilling of bytestream failed with PicklingError - write manual converter") + return ReprFail( + "undilling of bytestream failed with PicklingError - write manual converter" + ) except Exception as e: - return ReprFail("undilling of bytestream failed for class %s - write manual converter." - "Error: %s - %s" % (obj_class, type(e), e)) + return ReprFail( + "undilling of bytestream failed for class %s - write manual converter." + "Error: %s - %s" % (obj_class, type(e), e) + ) + def errored(el): - return el is None or (isinstance(el, list) and 'backend-error' in str(el)) + return el is None or (isinstance(el, list) and "backend-error" in str(el)) # Make wrapper for getting an object representation from process -------------- -class UndefinedValue: pass +class UndefinedValue: + pass + def getResultFromProcess(res, tempname, process): """Get a value from process, return tuple of value, res if succesful""" if not isinstance(res, (UndefinedValue, Exception)): value = getRepresentation(tempname, process) - return (value, res) - else: - return (None, res) + return value, res + else: + return res, str(res) + # decorator to automatically get value after running process task function def get_rep(f): sig = inspect.signature(f) + @wraps(f) - def wrapper(*args, **kwargs): + def wrapper(*args, **kwargs): # get bound arguments for call ba = sig.bind_partial(*args, **kwargs) ba.apply_defaults() # get tempname, process arg values - tempname = ba.arguments['tempname'] - process = ba.arguments['process'] + tempname = ba.arguments["tempname"] + process = ba.arguments["process"] # run process task - res = f(*args,**kwargs) + res = f(*args, **kwargs) # get result from task return getResultFromProcess(res, tempname, process) + return wrapper + ## Get the output of a tree (with setting envs, pre_code and/er expr_code) @process_task def get_output(f, process, shell, *args, **kwargs): @@ -313,83 +333,145 @@ def get_output(f, process, shell, *args, **kwargs): out_str = out[0].strip() if not isinstance(res, Exception): - str_rep = out_str or "no output" - return (out_str, str_rep) + toret = out_str or "no printouts" + return toret, toret else: - return (None, res) + return res, str(res) + @process_task def get_error(f, *args, **kwargs): res = f(*args, **kwargs) - return (res, res) if isinstance(res, Exception) else (None, res) + return (res, str(res)) if isinstance(res, Exception) else (None, res) + # General tasks to eval or exec code, with decorated counterparts ------------- -# Eval an expression tree or node (with setting envs, pre_code and/or expr_code) + @process_task -def taskRunEval(tree, - process, shell, - keep_objs_in_env = None, extra_env = None, context=None, context_vals=None, - pre_code = "", expr_code = "", name="", tempname='_evaluation_object_', do_exec=False, - call=None): - new_env = utils.copy_env(get_env(shell.user_ns), keep_objs_in_env) - if extra_env is not None: - new_env.update(copy.deepcopy(extra_env)) - if context is not None: - set_context_vals(new_env, context, context_vals) - try: - # Execute pre_code if specified - if pre_code: exec(pre_code, new_env) - - # If no name given, the object of interest is the output of eval - # otherwise, we'll use name to get the object from the environment - if not (name or do_exec): - mode = 'eval' - tree = ast.Expression(tree) +def taskRunEval( + tree, + process, + shell, + env=None, + extra_env=None, + context=None, + context_vals=None, + pre_code="", + expr_code="", + name="", + copy=True, + tempname="_evaluation_object_", + call=None, +): + """ + Eval an expression tree (with setting envs, pre_code and/or expr_code) + Utility function later wrapped to extract either result (end state of a variable), output (stdout) or error + + Args: + tree (ast): current focused ast, used to get code to execute + process: manages shell (see local.py) + shell: link to get process namespace from execution up until now + env: update value in focused code by name + extra_env: variables to be replaced in focused code by name from extra_env in has_expr + context: sum of set_context in sct chain + context_vals: extra context argument in has_expr + pre_code: argument in has_expr to execute code before evaluating, for example to set a seed + expr_code: code to execute instead of focused code + name: extract value after executing focused expr_code (~post_code) + copy: copy entire env because our expr_code could have side effects + tempname: key for the result when it is added to context, only for v1 sct's + call: only used in v1 sct's + + Returns: + str: output of the executed code + """ + try: + # Prepare code and mode ----------------------------------------------- + # Verify if expr_code is expression code (returning a value) or just runnable code. + if ( # expr_code returns nothing and then we will extract a value + expr_code and name + ) or ( # No expr_code and the tree is of a node type that does not evaluate to have output + not expr_code and isinstance(tree, ast.Module) + ): + # We are not focused on an expression (no output) + mode = "exec" else: - mode = 'exec' + mode = "eval" + # Wrap the focused node in the tree so it can be run with eval() + if not isinstance(tree, (ast.Module, ast.Expression, ast.Expr)): + tree = ast.Expression(tree) # Expression code takes precedence over tree code - if expr_code: code = expr_code - else: code = compile(tree, "