Ever had your service attacked by a retry storm from your clients? Or your clients’ clients? Or has your service ever been the attacker in one of those situations?
To help you avoid this, in release 6.31 we’ve introduced the notion of
that controls when it is okay for a client to retry a failed request.
The default budget allows for 20% of requests to be retried on top of a minimum of 10 retries per second. This is built on top of a token bucket where credits expire after 10 seconds. In practice this should allow for plenty of retries in the face of transient errors without causing cascading failures when there are persistent issues.
Developers can wire up their own budget:
import com.twitter.finagle.Http import com.twitter.finagle.service.RetryBudget val budget: RetryBudget = ??? val twitter = Http.client .withRetryBudget(budget) .newService("twitter.com")
The following code shows how to use the factory method
RetryBudget.apply in order to construct a custom instance of
import com.twitter.conversions.time._ import com.twitter.finagle.service.RetryBudget val budget: RetryBudget = RetryBudget( ttl = 10.seconds, minRetriesPerSec = 5, percentCanRetry = 0.1 )
RetryBudget factory method takes three arguments:
ttl— a time to live for deposited tokens
minRetriesPerSec— the minimum rate of retries allowed
percentCanRetry— the percentage of requests that might be retried
RetryBudgets are wired up to
which allows them to be used outside of a Finagle client as well.
Further details on retries and retry budgets are covered in the
Please let us know if you have any questions, either by filing a GitHub issue or getting in touch through @finagle or the Finaglers mailing list.