Skip to content

Persistent failure when submitting many API requests concurrently #580

Closed
Closed
Persistent failure when submitting many API requests concurrently#580
Assignees
jreiberkyle
Labels
feedback
Milestone
2.0a3: Iterative Alpha Release

Description

@tbarsballe
tbarsballe
opened on Jun 3, 2022

When submitting many (1000+) quick-search requests concurrently using asyncio.gather, they consistently fail with a planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms error.

I have also occasionally seen a planet.exceptions.ServerError: Internal Server Error instead, but only when using a shared session among all requests; using a unique session for each request seems to only produce the TooManyRequests error.

This failure is consistent, and happens every time I attempt this.

Stacktrace of the TooManyRequests error is:

unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-111' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:108> exception=CloseError(SSLError(1, '[SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)'))>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 97, in do_search
    items.append(item)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 183, in aclose
    await self.connection.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 202, in aclose
    await self.socket.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_backends/asyncio.py", line 198, in aclose
    await self.stream_writer.wait_closed()  # type: ignore
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.CloseError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-79' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:108> exception=CloseError(SSLError(1, '[SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)'))>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 97, in do_search
    items.append(item)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 183, in aclose
    await self.connection.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 202, in aclose
    await self.socket.aclose()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_backends/asyncio.py", line 198, in aclose
    await self.stream_writer.wait_closed()  # type: ignore
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.CloseError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2756)
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 317, in __anext__
    item = self._items.pop(0)
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 118, in do_searches
    results = await asyncio.gather(*tasks)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 113, in do_search_geom
    itms = await do_search(start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 95, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 333, in _get_pages
    resp = await self._do_request(self.request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/clients/data.py", line 86, in _do_request
    return await self._session.request(request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 230, in request
    return await self._retry(self._request, request, stream=stream)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 178, in _retry
    raise e
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1422, in send
    await hook(response)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 135, in araise_for_status
    return self._raise_for_status(*args, **kwargs)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 77, in _raise_for_status
    raise exception(msg)
planet.exceptions.TooManyRequests: max rate reached: retry-in 200ms

Stacktrace of the ServerError is:

unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-938' coro=<do_searches.<locals>.do_search_geom() done, defined at /Users/torben.barsballe/repos/Planet/order-util/order_aois.py:109> exception=RemoteProtocolError("can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE")>
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_exceptions.py", line 326, in map_exceptions
    yield
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 218, in arequest
    response = await connection.arequest(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/connection.py", line 106, in arequest
    return await self.connection.arequest(method, url, headers, stream, ext)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 72, in arequest
    ) = await self._receive_response(timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 133, in _receive_response
    event = await self._receive_event(timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_async/http11.py", line 169, in _receive_event
    event = self.h11_state.next_event()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 114, in do_search_geom
    itms = await do_search(sess, start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 96, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1406, in send
    response = await self._send_handling_auth(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1444, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_exceptions.py", line 343, in map_exceptions
    raise mapped_exc(message, **kwargs) from exc  # type: ignore
httpx.RemoteProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
Traceback (most recent call last):
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 317, in __anext__
    item = self._items.pop(0)
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 120, in do_searches
    results = await asyncio.gather(*tasks)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 114, in do_search_geom
    itms = await do_search(sess, start, end, geom)
  File "/Users/torben.barsballe/repos/Planet/order-util/order_aois.py", line 96, in do_search
    async for item in await search_results:
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 321, in __anext__
    page = await self._pages.__anext__()
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/models.py", line 333, in _get_pages
    resp = await self._do_request(self.request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/clients/data.py", line 86, in _do_request
    return await self._session.request(request)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 230, in request
    return await self._retry(self._request, request, stream=stream)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 174, in _retry
    resp = await func(*a, **kw)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 234, in _request
    http_resp = await self._client.send(request.http_request,
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/httpx/_client.py", line 1422, in send
    await hook(response)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 135, in araise_for_status
    return self._raise_for_status(*args, **kwargs)
  File "/Users/torben.barsballe/repos/Planet/order-util/venv-v2/lib/python3.9/site-packages/planet/http.py", line 77, in _raise_for_status
    raise exception(msg)
planet.exceptions.ServerError: Internal Server Error
python-BaseException

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

feedback

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions