Are there any benefits to this other than a bit of organization?

Cutting down on pyflakes output...

Removing unused imports could improve matplotlib's import time and memory usage. I'm not saying necessarily significantly, but I'll take what I can get for free. 😁

PEP 8 doesn't like animation.py

It only helps if they are thing that would not be imported other wise. If things are already in sys.modules the import is basically free.

Note that I never claimed this was for performance :p
PEP8 should be appeased now...

I think we explicitly do not check this bit of pep8, if we are going to do this travis should enforce it.

The dark side of 'discover-the-api-via-tab-complete' is that users will have found and used some of those un-used imports so I expect this to break some user code. It should get some level of mention in the API changes docs.

Not saying that this PR fixes pep8; rather, pep8 was unhappy because it wanted conditional imports (vs Python version) to be at the bottom of the import list.
Added api changes note.

@tacaswell I agree...that's why I said "could". 😁 Also, the unused import adds to the module's globals dict (not to mention takes a few instructions to execute), so that's technically more memory usage (and CPU). I'm not saying it's significant, but it's a "free" fix to not ask for things you don't use.

grouped some more imports (in a separate commit)