Skip to content

Fix phantomjs request aborted edge case - #82

Merged
kategengler merged 1 commit into
ember-cli-code-coverage:masterfrom
jesseditson:phantom-sync-xhr
Nov 30, 2016
Merged

Fix phantomjs request aborted edge case#82
kategengler merged 1 commit into
ember-cli-code-coverage:masterfrom
jesseditson:phantom-sync-xhr

Conversation

@jesseditson

Copy link
Copy Markdown
Contributor

I experienced a difficult to reproduce edge case on an Ember application with > 500 tests:

When running with coverage on the v0.3.8 tag, my test server crashes right after the tests complete with the following error:

Error: request aborted
    at IncomingMessage.onAborted ([snip]/ember-cli-code-coverage/node_modules/raw-body/index.js:269:10)
    at emitNone (events.js:86:13)
    at IncomingMessage.emit (events.js:185:7)
    at abortIncoming (_http_server.js:283:11)
    at Socket.serverSocketCloseListener (_http_server.js:296:5)
    at emitOne (events.js:101:20)
    at Socket.emit (events.js:188:7)
    at TCP._handle.close [as _onclose] (net.js:498:12)

This wasn't happening on the version we were using before (v0.2.2), so I figured it was a change in the recent minor bumps that caused the breakage. I bisected tags, and reproduced the issue in v0.3.7 and not v0.3.5 (v0.3.6 doesn't work with our codebase at all due to a throw that was downgraded to a warn in v0.3.7).

Looking at the compare between those versions, I couldn't see anything that looked even the slightest bit innocuous, so I started a long and fruitless journey through ember-cli, testem, and friends to try to see if I could pin down a source anywhere.

I also set up a new ember application and prodded it for a good amount of time looking for a repro case to no avail. Unfortunately it's a consistent error in a private repo (every jenkins build since I bumped ember-cli-code-coverage has broken along with local builds), but I have no repro case to offer.

As an additional annoyance, I was unsuccessful (in a workday) in finding the real source of the issue in the testem flow, but can say with certainty that it is not caused by:

• a long timeout on the /write-coverage endpoint
• onload use rather than readystatechange events
• calling the sendCoverage callback in the same event loop as the coverage XHR onload event (I tried setting various timeouts before calling callback, same result)

The only difference I could find in my large suite vs a brand new one was the fact that (oddly), Testem was emitting 'all-test-results' multiple (3) times. I figured this was the issue (it resolves the first XHR with 3 in flight, and the second one is cancelled by a disconnect). However, I was unable to fix this by only allowing sendCoverage to be called once, leading me to believe that either my suite is spinning up multiple runners, or the issue is unrelated (or in an alternative listener).

Through this exploration, I found a few different ways to workaround the failure, the simplest of which being contained in this PR (just make the XHR sync when inside PhantomJS) .

I'm almost 100% certain the issue I'm experiencing is not caused by this repo, but this seems like the most reasonable place to add a workaround in lieu of fixing the root cause in a dependency lib, since the use case (sending an XHR from test-body-footer.html on test complete) is fairly esoteric and hunting down the root cause is likely going to take longer than I have available to work on this case.

@rwjblue

rwjblue commented Nov 30, 2016

Copy link
Copy Markdown
Collaborator

Thanks for putting so much time into tracking this down! I'm 👍 ...

@kategengler - Any objections?

@kategengler

Copy link
Copy Markdown
Collaborator

No objections; the request was originally always synchronous.

@jesseditson Thanks so much for looking into this! I've seen this error in apps that aren't using this addon, so I knew the root cause wasn't this addon but didn't really know where to look next.

@kategengler
kategengler merged commit 4e388f7 into ember-cli-code-coverage:master Nov 30, 2016
@jesseditson

Copy link
Copy Markdown
Contributor Author

Neato! @kategengler feel free to ping me if you find a stable repro case for that error, I believe the multiple end events from testem is a good vector, and with a more focused suite could likely track down the root. Hopefully this will hold us over until then!

@DingoEatingFuzz

Copy link
Copy Markdown
Contributor

So my tests were passing dandily in CI on v0.3.8. Then I upgraded a bunch of packages while upgrading ember, and suddenly my tests were hanging in CI. All tests would pass, but the command would never finish.

I eventually tracked it down to this v0.3.9 patch. I imagine, now looking at the patch, the request hangs forever and eventually my CI server fails the build.

It would be good to at the very least have this synchronous request timeout and write an error. Just to save some time for the next sorry sucker that has to manually bisect all their dependency upgrades to find the culprit.

Note that this has been mentioned and +1'd over here: #81 (comment)

@DingoEatingFuzz

Copy link
Copy Markdown
Contributor

Some additional info: we use Codecov.io, which has its own bash script that sniffs a filesystem for coverage reports and merges them all together on their end. For this reason, we don't use the POST /write-coverage behavior at all.

For the curious: http://docs.codecov.io/docs/merging-reports

@kategengler

Copy link
Copy Markdown
Collaborator

Since 0.3.9 fixed this issue for some but apparently introduced it for others, I don't think it's a clear win to just revert the change.

@DingoEatingFuzz The /write-coverage endpoint is the only way any coverage report from this addon gets written to the file system. It's not optional behavior.

@DingoEatingFuzz

Copy link
Copy Markdown
Contributor

Well then. Clearly my understanding of how the addon works is incomplete. I'll look into the situation further tomorrow and report my (hopefully accurate) findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants