1GRPC Connection Backoff Protocol 2================================ 3 4When we do a connection to a backend which fails, it is typically desirable to 5not retry immediately (to avoid flooding the network or the server with 6requests) and instead do some form of exponential backoff. 7 8We have several parameters: 9 1. INITIAL_BACKOFF (how long to wait after the first failure before retrying) 10 1. MULTIPLIER (factor with which to multiply backoff after a failed retry) 11 1. JITTER (by how much to randomize backoffs). 12 1. MAX_BACKOFF (upper bound on backoff) 13 1. MIN_CONNECT_TIMEOUT (minimum time we're willing to give a connection to 14 complete) 15 16## Proposed Backoff Algorithm 17 18Exponentially back off the start time of connection attempts up to a limit of 19MAX_BACKOFF, with jitter. 20 21``` 22ConnectWithBackoff() 23 current_backoff = INITIAL_BACKOFF 24 current_deadline = now() + INITIAL_BACKOFF 25 while (TryConnect(Max(current_deadline, now() + MIN_CONNECT_TIMEOUT)) 26 != SUCCESS) 27 SleepUntil(current_deadline) 28 current_backoff = Min(current_backoff * MULTIPLIER, MAX_BACKOFF) 29 current_deadline = now() + current_backoff + 30 UniformRandom(-JITTER * current_backoff, JITTER * current_backoff) 31 32``` 33 34With specific parameters of 35MIN_CONNECT_TIMEOUT = 20 seconds 36INITIAL_BACKOFF = 1 second 37MULTIPLIER = 1.6 38MAX_BACKOFF = 120 seconds 39JITTER = 0.2 40 41Implementations with pressing concerns (such as minimizing the number of wakeups 42on a mobile phone) may wish to use a different algorithm, and in particular 43different jitter logic. 44 45Alternate implementations must ensure that connection backoffs started at the 46same time disperse, and must not attempt connections substantially more often 47than the above algorithm. 48 49## Reset Backoff 50 51The back off should be reset to INITIAL_BACKOFF at some time point, so that the 52reconnecting behavior is consistent no matter the connection is a newly started 53one or a previously disconnected one. 54 55We choose to reset the Backoff when the SETTINGS frame is received, at that time 56point, we know for sure that this connection was accepted by the server. 57