jpayne@7: from __future__ import annotations jpayne@7: jpayne@7: import email jpayne@7: import logging jpayne@7: import random jpayne@7: import re jpayne@7: import time jpayne@7: import typing jpayne@7: from itertools import takewhile jpayne@7: from types import TracebackType jpayne@7: jpayne@7: from ..exceptions import ( jpayne@7: ConnectTimeoutError, jpayne@7: InvalidHeader, jpayne@7: MaxRetryError, jpayne@7: ProtocolError, jpayne@7: ProxyError, jpayne@7: ReadTimeoutError, jpayne@7: ResponseError, jpayne@7: ) jpayne@7: from .util import reraise jpayne@7: jpayne@7: if typing.TYPE_CHECKING: jpayne@7: from ..connectionpool import ConnectionPool jpayne@7: from ..response import BaseHTTPResponse jpayne@7: jpayne@7: log = logging.getLogger(__name__) jpayne@7: jpayne@7: jpayne@7: # Data structure for representing the metadata of requests that result in a retry. jpayne@7: class RequestHistory(typing.NamedTuple): jpayne@7: method: str | None jpayne@7: url: str | None jpayne@7: error: Exception | None jpayne@7: status: int | None jpayne@7: redirect_location: str | None jpayne@7: jpayne@7: jpayne@7: class Retry: jpayne@7: """Retry configuration. jpayne@7: jpayne@7: Each retry attempt will create a new Retry object with updated values, so jpayne@7: they can be safely reused. jpayne@7: jpayne@7: Retries can be defined as a default for a pool: jpayne@7: jpayne@7: .. code-block:: python jpayne@7: jpayne@7: retries = Retry(connect=5, read=2, redirect=5) jpayne@7: http = PoolManager(retries=retries) jpayne@7: response = http.request("GET", "https://example.com/") jpayne@7: jpayne@7: Or per-request (which overrides the default for the pool): jpayne@7: jpayne@7: .. code-block:: python jpayne@7: jpayne@7: response = http.request("GET", "https://example.com/", retries=Retry(10)) jpayne@7: jpayne@7: Retries can be disabled by passing ``False``: jpayne@7: jpayne@7: .. code-block:: python jpayne@7: jpayne@7: response = http.request("GET", "https://example.com/", retries=False) jpayne@7: jpayne@7: Errors will be wrapped in :class:`~urllib3.exceptions.MaxRetryError` unless jpayne@7: retries are disabled, in which case the causing exception will be raised. jpayne@7: jpayne@7: :param int total: jpayne@7: Total number of retries to allow. Takes precedence over other counts. jpayne@7: jpayne@7: Set to ``None`` to remove this constraint and fall back on other jpayne@7: counts. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry. jpayne@7: jpayne@7: Set to ``False`` to disable and imply ``raise_on_redirect=False``. jpayne@7: jpayne@7: :param int connect: jpayne@7: How many connection-related errors to retry on. jpayne@7: jpayne@7: These are errors raised before the request is sent to the remote server, jpayne@7: which we assume has not triggered the server to process the request. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry of this type. jpayne@7: jpayne@7: :param int read: jpayne@7: How many times to retry on read errors. jpayne@7: jpayne@7: These errors are raised after the request was sent to the server, so the jpayne@7: request may have side-effects. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry of this type. jpayne@7: jpayne@7: :param int redirect: jpayne@7: How many redirects to perform. Limit this to avoid infinite redirect jpayne@7: loops. jpayne@7: jpayne@7: A redirect is a HTTP response with a status code 301, 302, 303, 307 or jpayne@7: 308. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry of this type. jpayne@7: jpayne@7: Set to ``False`` to disable and imply ``raise_on_redirect=False``. jpayne@7: jpayne@7: :param int status: jpayne@7: How many times to retry on bad status codes. jpayne@7: jpayne@7: These are retries made on responses, where status code matches jpayne@7: ``status_forcelist``. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry of this type. jpayne@7: jpayne@7: :param int other: jpayne@7: How many times to retry on other errors. jpayne@7: jpayne@7: Other errors are errors that are not connect, read, redirect or status errors. jpayne@7: These errors might be raised after the request was sent to the server, so the jpayne@7: request might have side-effects. jpayne@7: jpayne@7: Set to ``0`` to fail on the first retry of this type. jpayne@7: jpayne@7: If ``total`` is not set, it's a good idea to set this to 0 to account jpayne@7: for unexpected edge cases and avoid infinite retry loops. jpayne@7: jpayne@7: :param Collection allowed_methods: jpayne@7: Set of uppercased HTTP method verbs that we should retry on. jpayne@7: jpayne@7: By default, we only retry on methods which are considered to be jpayne@7: idempotent (multiple requests with the same parameters end with the jpayne@7: same state). See :attr:`Retry.DEFAULT_ALLOWED_METHODS`. jpayne@7: jpayne@7: Set to a ``None`` value to retry on any verb. jpayne@7: jpayne@7: :param Collection status_forcelist: jpayne@7: A set of integer HTTP status codes that we should force a retry on. jpayne@7: A retry is initiated if the request method is in ``allowed_methods`` jpayne@7: and the response status code is in ``status_forcelist``. jpayne@7: jpayne@7: By default, this is disabled with ``None``. jpayne@7: jpayne@7: :param float backoff_factor: jpayne@7: A backoff factor to apply between attempts after the second try jpayne@7: (most errors are resolved immediately by a second try without a jpayne@7: delay). urllib3 will sleep for:: jpayne@7: jpayne@7: {backoff factor} * (2 ** ({number of previous retries})) jpayne@7: jpayne@7: seconds. If `backoff_jitter` is non-zero, this sleep is extended by:: jpayne@7: jpayne@7: random.uniform(0, {backoff jitter}) jpayne@7: jpayne@7: seconds. For example, if the backoff_factor is 0.1, then :func:`Retry.sleep` will jpayne@7: sleep for [0.0s, 0.2s, 0.4s, 0.8s, ...] between retries. No backoff will ever jpayne@7: be longer than `backoff_max`. jpayne@7: jpayne@7: By default, backoff is disabled (factor set to 0). jpayne@7: jpayne@7: :param bool raise_on_redirect: Whether, if the number of redirects is jpayne@7: exhausted, to raise a MaxRetryError, or to return a response with a jpayne@7: response code in the 3xx range. jpayne@7: jpayne@7: :param bool raise_on_status: Similar meaning to ``raise_on_redirect``: jpayne@7: whether we should raise an exception, or return a response, jpayne@7: if status falls in ``status_forcelist`` range and retries have jpayne@7: been exhausted. jpayne@7: jpayne@7: :param tuple history: The history of the request encountered during jpayne@7: each call to :meth:`~Retry.increment`. The list is in the order jpayne@7: the requests occurred. Each list item is of class :class:`RequestHistory`. jpayne@7: jpayne@7: :param bool respect_retry_after_header: jpayne@7: Whether to respect Retry-After header on status codes defined as jpayne@7: :attr:`Retry.RETRY_AFTER_STATUS_CODES` or not. jpayne@7: jpayne@7: :param Collection remove_headers_on_redirect: jpayne@7: Sequence of headers to remove from the request when a response jpayne@7: indicating a redirect is returned before firing off the redirected jpayne@7: request. jpayne@7: """ jpayne@7: jpayne@7: #: Default methods to be used for ``allowed_methods`` jpayne@7: DEFAULT_ALLOWED_METHODS = frozenset( jpayne@7: ["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"] jpayne@7: ) jpayne@7: jpayne@7: #: Default status codes to be used for ``status_forcelist`` jpayne@7: RETRY_AFTER_STATUS_CODES = frozenset([413, 429, 503]) jpayne@7: jpayne@7: #: Default headers to be used for ``remove_headers_on_redirect`` jpayne@7: DEFAULT_REMOVE_HEADERS_ON_REDIRECT = frozenset(["Cookie", "Authorization"]) jpayne@7: jpayne@7: #: Default maximum backoff time. jpayne@7: DEFAULT_BACKOFF_MAX = 120 jpayne@7: jpayne@7: # Backward compatibility; assigned outside of the class. jpayne@7: DEFAULT: typing.ClassVar[Retry] jpayne@7: jpayne@7: def __init__( jpayne@7: self, jpayne@7: total: bool | int | None = 10, jpayne@7: connect: int | None = None, jpayne@7: read: int | None = None, jpayne@7: redirect: bool | int | None = None, jpayne@7: status: int | None = None, jpayne@7: other: int | None = None, jpayne@7: allowed_methods: typing.Collection[str] | None = DEFAULT_ALLOWED_METHODS, jpayne@7: status_forcelist: typing.Collection[int] | None = None, jpayne@7: backoff_factor: float = 0, jpayne@7: backoff_max: float = DEFAULT_BACKOFF_MAX, jpayne@7: raise_on_redirect: bool = True, jpayne@7: raise_on_status: bool = True, jpayne@7: history: tuple[RequestHistory, ...] | None = None, jpayne@7: respect_retry_after_header: bool = True, jpayne@7: remove_headers_on_redirect: typing.Collection[ jpayne@7: str jpayne@7: ] = DEFAULT_REMOVE_HEADERS_ON_REDIRECT, jpayne@7: backoff_jitter: float = 0.0, jpayne@7: ) -> None: jpayne@7: self.total = total jpayne@7: self.connect = connect jpayne@7: self.read = read jpayne@7: self.status = status jpayne@7: self.other = other jpayne@7: jpayne@7: if redirect is False or total is False: jpayne@7: redirect = 0 jpayne@7: raise_on_redirect = False jpayne@7: jpayne@7: self.redirect = redirect jpayne@7: self.status_forcelist = status_forcelist or set() jpayne@7: self.allowed_methods = allowed_methods jpayne@7: self.backoff_factor = backoff_factor jpayne@7: self.backoff_max = backoff_max jpayne@7: self.raise_on_redirect = raise_on_redirect jpayne@7: self.raise_on_status = raise_on_status jpayne@7: self.history = history or () jpayne@7: self.respect_retry_after_header = respect_retry_after_header jpayne@7: self.remove_headers_on_redirect = frozenset( jpayne@7: h.lower() for h in remove_headers_on_redirect jpayne@7: ) jpayne@7: self.backoff_jitter = backoff_jitter jpayne@7: jpayne@7: def new(self, **kw: typing.Any) -> Retry: jpayne@7: params = dict( jpayne@7: total=self.total, jpayne@7: connect=self.connect, jpayne@7: read=self.read, jpayne@7: redirect=self.redirect, jpayne@7: status=self.status, jpayne@7: other=self.other, jpayne@7: allowed_methods=self.allowed_methods, jpayne@7: status_forcelist=self.status_forcelist, jpayne@7: backoff_factor=self.backoff_factor, jpayne@7: backoff_max=self.backoff_max, jpayne@7: raise_on_redirect=self.raise_on_redirect, jpayne@7: raise_on_status=self.raise_on_status, jpayne@7: history=self.history, jpayne@7: remove_headers_on_redirect=self.remove_headers_on_redirect, jpayne@7: respect_retry_after_header=self.respect_retry_after_header, jpayne@7: backoff_jitter=self.backoff_jitter, jpayne@7: ) jpayne@7: jpayne@7: params.update(kw) jpayne@7: return type(self)(**params) # type: ignore[arg-type] jpayne@7: jpayne@7: @classmethod jpayne@7: def from_int( jpayne@7: cls, jpayne@7: retries: Retry | bool | int | None, jpayne@7: redirect: bool | int | None = True, jpayne@7: default: Retry | bool | int | None = None, jpayne@7: ) -> Retry: jpayne@7: """Backwards-compatibility for the old retries format.""" jpayne@7: if retries is None: jpayne@7: retries = default if default is not None else cls.DEFAULT jpayne@7: jpayne@7: if isinstance(retries, Retry): jpayne@7: return retries jpayne@7: jpayne@7: redirect = bool(redirect) and None jpayne@7: new_retries = cls(retries, redirect=redirect) jpayne@7: log.debug("Converted retries value: %r -> %r", retries, new_retries) jpayne@7: return new_retries jpayne@7: jpayne@7: def get_backoff_time(self) -> float: jpayne@7: """Formula for computing the current backoff jpayne@7: jpayne@7: :rtype: float jpayne@7: """ jpayne@7: # We want to consider only the last consecutive errors sequence (Ignore redirects). jpayne@7: consecutive_errors_len = len( jpayne@7: list( jpayne@7: takewhile(lambda x: x.redirect_location is None, reversed(self.history)) jpayne@7: ) jpayne@7: ) jpayne@7: if consecutive_errors_len <= 1: jpayne@7: return 0 jpayne@7: jpayne@7: backoff_value = self.backoff_factor * (2 ** (consecutive_errors_len - 1)) jpayne@7: if self.backoff_jitter != 0.0: jpayne@7: backoff_value += random.random() * self.backoff_jitter jpayne@7: return float(max(0, min(self.backoff_max, backoff_value))) jpayne@7: jpayne@7: def parse_retry_after(self, retry_after: str) -> float: jpayne@7: seconds: float jpayne@7: # Whitespace: https://tools.ietf.org/html/rfc7230#section-3.2.4 jpayne@7: if re.match(r"^\s*[0-9]+\s*$", retry_after): jpayne@7: seconds = int(retry_after) jpayne@7: else: jpayne@7: retry_date_tuple = email.utils.parsedate_tz(retry_after) jpayne@7: if retry_date_tuple is None: jpayne@7: raise InvalidHeader(f"Invalid Retry-After header: {retry_after}") jpayne@7: jpayne@7: retry_date = email.utils.mktime_tz(retry_date_tuple) jpayne@7: seconds = retry_date - time.time() jpayne@7: jpayne@7: seconds = max(seconds, 0) jpayne@7: jpayne@7: return seconds jpayne@7: jpayne@7: def get_retry_after(self, response: BaseHTTPResponse) -> float | None: jpayne@7: """Get the value of Retry-After in seconds.""" jpayne@7: jpayne@7: retry_after = response.headers.get("Retry-After") jpayne@7: jpayne@7: if retry_after is None: jpayne@7: return None jpayne@7: jpayne@7: return self.parse_retry_after(retry_after) jpayne@7: jpayne@7: def sleep_for_retry(self, response: BaseHTTPResponse) -> bool: jpayne@7: retry_after = self.get_retry_after(response) jpayne@7: if retry_after: jpayne@7: time.sleep(retry_after) jpayne@7: return True jpayne@7: jpayne@7: return False jpayne@7: jpayne@7: def _sleep_backoff(self) -> None: jpayne@7: backoff = self.get_backoff_time() jpayne@7: if backoff <= 0: jpayne@7: return jpayne@7: time.sleep(backoff) jpayne@7: jpayne@7: def sleep(self, response: BaseHTTPResponse | None = None) -> None: jpayne@7: """Sleep between retry attempts. jpayne@7: jpayne@7: This method will respect a server's ``Retry-After`` response header jpayne@7: and sleep the duration of the time requested. If that is not present, it jpayne@7: will use an exponential backoff. By default, the backoff factor is 0 and jpayne@7: this method will return immediately. jpayne@7: """ jpayne@7: jpayne@7: if self.respect_retry_after_header and response: jpayne@7: slept = self.sleep_for_retry(response) jpayne@7: if slept: jpayne@7: return jpayne@7: jpayne@7: self._sleep_backoff() jpayne@7: jpayne@7: def _is_connection_error(self, err: Exception) -> bool: jpayne@7: """Errors when we're fairly sure that the server did not receive the jpayne@7: request, so it should be safe to retry. jpayne@7: """ jpayne@7: if isinstance(err, ProxyError): jpayne@7: err = err.original_error jpayne@7: return isinstance(err, ConnectTimeoutError) jpayne@7: jpayne@7: def _is_read_error(self, err: Exception) -> bool: jpayne@7: """Errors that occur after the request has been started, so we should jpayne@7: assume that the server began processing it. jpayne@7: """ jpayne@7: return isinstance(err, (ReadTimeoutError, ProtocolError)) jpayne@7: jpayne@7: def _is_method_retryable(self, method: str) -> bool: jpayne@7: """Checks if a given HTTP method should be retried upon, depending if jpayne@7: it is included in the allowed_methods jpayne@7: """ jpayne@7: if self.allowed_methods and method.upper() not in self.allowed_methods: jpayne@7: return False jpayne@7: return True jpayne@7: jpayne@7: def is_retry( jpayne@7: self, method: str, status_code: int, has_retry_after: bool = False jpayne@7: ) -> bool: jpayne@7: """Is this method/status code retryable? (Based on allowlists and control jpayne@7: variables such as the number of total retries to allow, whether to jpayne@7: respect the Retry-After header, whether this header is present, and jpayne@7: whether the returned status code is on the list of status codes to jpayne@7: be retried upon on the presence of the aforementioned header) jpayne@7: """ jpayne@7: if not self._is_method_retryable(method): jpayne@7: return False jpayne@7: jpayne@7: if self.status_forcelist and status_code in self.status_forcelist: jpayne@7: return True jpayne@7: jpayne@7: return bool( jpayne@7: self.total jpayne@7: and self.respect_retry_after_header jpayne@7: and has_retry_after jpayne@7: and (status_code in self.RETRY_AFTER_STATUS_CODES) jpayne@7: ) jpayne@7: jpayne@7: def is_exhausted(self) -> bool: jpayne@7: """Are we out of retries?""" jpayne@7: retry_counts = [ jpayne@7: x jpayne@7: for x in ( jpayne@7: self.total, jpayne@7: self.connect, jpayne@7: self.read, jpayne@7: self.redirect, jpayne@7: self.status, jpayne@7: self.other, jpayne@7: ) jpayne@7: if x jpayne@7: ] jpayne@7: if not retry_counts: jpayne@7: return False jpayne@7: jpayne@7: return min(retry_counts) < 0 jpayne@7: jpayne@7: def increment( jpayne@7: self, jpayne@7: method: str | None = None, jpayne@7: url: str | None = None, jpayne@7: response: BaseHTTPResponse | None = None, jpayne@7: error: Exception | None = None, jpayne@7: _pool: ConnectionPool | None = None, jpayne@7: _stacktrace: TracebackType | None = None, jpayne@7: ) -> Retry: jpayne@7: """Return a new Retry object with incremented retry counters. jpayne@7: jpayne@7: :param response: A response object, or None, if the server did not jpayne@7: return a response. jpayne@7: :type response: :class:`~urllib3.response.BaseHTTPResponse` jpayne@7: :param Exception error: An error encountered during the request, or jpayne@7: None if the response was received successfully. jpayne@7: jpayne@7: :return: A new ``Retry`` object. jpayne@7: """ jpayne@7: if self.total is False and error: jpayne@7: # Disabled, indicate to re-raise the error. jpayne@7: raise reraise(type(error), error, _stacktrace) jpayne@7: jpayne@7: total = self.total jpayne@7: if total is not None: jpayne@7: total -= 1 jpayne@7: jpayne@7: connect = self.connect jpayne@7: read = self.read jpayne@7: redirect = self.redirect jpayne@7: status_count = self.status jpayne@7: other = self.other jpayne@7: cause = "unknown" jpayne@7: status = None jpayne@7: redirect_location = None jpayne@7: jpayne@7: if error and self._is_connection_error(error): jpayne@7: # Connect retry? jpayne@7: if connect is False: jpayne@7: raise reraise(type(error), error, _stacktrace) jpayne@7: elif connect is not None: jpayne@7: connect -= 1 jpayne@7: jpayne@7: elif error and self._is_read_error(error): jpayne@7: # Read retry? jpayne@7: if read is False or method is None or not self._is_method_retryable(method): jpayne@7: raise reraise(type(error), error, _stacktrace) jpayne@7: elif read is not None: jpayne@7: read -= 1 jpayne@7: jpayne@7: elif error: jpayne@7: # Other retry? jpayne@7: if other is not None: jpayne@7: other -= 1 jpayne@7: jpayne@7: elif response and response.get_redirect_location(): jpayne@7: # Redirect retry? jpayne@7: if redirect is not None: jpayne@7: redirect -= 1 jpayne@7: cause = "too many redirects" jpayne@7: response_redirect_location = response.get_redirect_location() jpayne@7: if response_redirect_location: jpayne@7: redirect_location = response_redirect_location jpayne@7: status = response.status jpayne@7: jpayne@7: else: jpayne@7: # Incrementing because of a server error like a 500 in jpayne@7: # status_forcelist and the given method is in the allowed_methods jpayne@7: cause = ResponseError.GENERIC_ERROR jpayne@7: if response and response.status: jpayne@7: if status_count is not None: jpayne@7: status_count -= 1 jpayne@7: cause = ResponseError.SPECIFIC_ERROR.format(status_code=response.status) jpayne@7: status = response.status jpayne@7: jpayne@7: history = self.history + ( jpayne@7: RequestHistory(method, url, error, status, redirect_location), jpayne@7: ) jpayne@7: jpayne@7: new_retry = self.new( jpayne@7: total=total, jpayne@7: connect=connect, jpayne@7: read=read, jpayne@7: redirect=redirect, jpayne@7: status=status_count, jpayne@7: other=other, jpayne@7: history=history, jpayne@7: ) jpayne@7: jpayne@7: if new_retry.is_exhausted(): jpayne@7: reason = error or ResponseError(cause) jpayne@7: raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] jpayne@7: jpayne@7: log.debug("Incremented Retry for (url='%s'): %r", url, new_retry) jpayne@7: jpayne@7: return new_retry jpayne@7: jpayne@7: def __repr__(self) -> str: jpayne@7: return ( jpayne@7: f"{type(self).__name__}(total={self.total}, connect={self.connect}, " jpayne@7: f"read={self.read}, redirect={self.redirect}, status={self.status})" jpayne@7: ) jpayne@7: jpayne@7: jpayne@7: # For backwards compatibility (equivalent to pre-v1.9): jpayne@7: Retry.DEFAULT = Retry(3)