Skip to content

Adds tracking for failed tests, re-running failed tests.#16701

Closed
rbuckton wants to merge 3 commits intomasterfrom
failedTestTracking
Closed

Adds tracking for failed tests, re-running failed tests.#16701
rbuckton wants to merge 3 commits intomasterfrom
failedTestTracking

Conversation

@rbuckton
Copy link
Copy Markdown
Contributor

This change adds the following capabilities:

  • After running tests (either via runtests or runtests-parallel), any failing tests will be written to a .failed-tests file.
    • This file is in a response-file format understood by mocha's --opts option.
    • The names of failing tests are escaped and combined to be used with mocha's --grep option.
  • Running either jake runtests failed=true or gulp runtests --failed will run only the failing tests from the last test run.
    • The failed option will be superseded by the tests option, if both are provided.
    • The failed option is not currently supported on the runtests-parallel task.
  • When a test run completes successfully, the .failed-tests file is removed unless you also pass keepFailed=true to jake or --keepFailed to gulp.

Examples (jake)

# Run tests with test failure tracking
jake runtests
jake runtests-parallel

# Run failed tests (remove .failed-tests file on success)
jake runtests failed=true

# Run failed tests (do not remove .failed-tests file on success)
jake runtests failed=true keepFailed=true

Examples (gulp)

# Run tests with test failure tracking
gulp runtests
gulp runtests-parallel

# Run failed tests (remove .failed-tests file on success)
gulp runtests --failed

# Run failed tests (do not remove .failed-tests file on success)
gulp runtests --failed --keepFailed

@yuit
Copy link
Copy Markdown
Contributor

yuit commented Jun 22, 2017

Gotta take a look but this is great :) 🍰

Comment thread Gulpfile.ts
inspect: process.env.inspect || process.env["inspect-brk"] || process.env.i,
host: process.env.TYPESCRIPT_HOST || process.env.host || "node",
browser: process.env.browser || process.env.b || "IE",
bail: process.env.bail || false,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is bail

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exit on the first failed test. This already exists in jakefile.js

Comment thread scripts/mocha-file-reporter.js Outdated
}
}

/**
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: either we remove this or we add it everywhere with comments

Comment thread scripts/mocha-file-reporter.js Outdated
* @param keepFailed {Boolean}
* @param fn {Function}
*/
FileReporter.writeFailures = function (file, failures, keepFailed, fn) {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rename file to fileName or filePath

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only added it here for better intellisense.

@rbuckton rbuckton force-pushed the failedTestTracking branch from c341c4b to ede814b Compare June 27, 2017 23:46
Comment thread scripts/mocha-file-reporter.js Outdated
var reporterOptions = this.reporterOptions = options.reporterOptions || {};
reporterOptions.file = reporterOptions.file || ".failed-tests";
reporterOptions.keepFailed = reporterOptions.keepFailed || false;
reporterOptions.reporter = reporterOptions.reporter;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like a rigid idiom, but it still bugs me. Can you remove it?

Copy link
Copy Markdown
Member

@sandersn sandersn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one comment. Otherwise looks good.

@mhegazy
Copy link
Copy Markdown
Contributor

mhegazy commented Nov 7, 2017

@rbuckton can you refresh this PR?

Copy link
Copy Markdown
Member

@RyanCavanaugh RyanCavanaugh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving a review to get this out of my pending PR reviews list

@rbuckton
Copy link
Copy Markdown
Contributor Author

Superseded by #25004.

@rbuckton rbuckton closed this Jun 26, 2018
@rbuckton rbuckton deleted the failedTestTracking branch June 26, 2018 00:47
@microsoft microsoft locked as resolved and limited conversation to collaborators Oct 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants