应付网络抖动等临时故障的重试策略

18-09-19 banq
                   

REST调用或同步是服务器之间通讯的经常方式,在没有分布式事务机制保障情况下,需要我们开发人员手工进行重试,重试几次失败后进行业务回退操作,重试非常重要,容易造成网络堵塞,引入断路器又过于重量,完善重试算法也许是一条出路:

在处理我们的应用程序时,我们不得不依赖远程资源,这些远程资源很容易出现故障,这种故障称为暂时或临时故障,那么如何通过重试避免一些呢?

临时故障

瞬态故障是指暂时发生并在短时间内发生的故障,由于网络拥塞或高系统负载,远程资源可能仅暂时不可用,并且这种情况完全可能会快速自我纠正。在这种情况下,我们必须等待一段时间并重新获取资源,重试以后如果失败就将其视为永久性故障。

正常的过程可能是这样的:

1. 我们获取资源,

2. 如果我们找到它,我们继续处理它。

3. 如果没有,我们等待一段时间再次获取资源,并继续这样做,直到我们拥有资源。

4. 我们显然不能永远等待获取到资源,因此在经过一定数量的重试之后,我们认为它是一个永久性的失败并在我们的代码中适当地处理它。

临时瞬态故障的一个例子可能是API访问限制,许多第三方API(包括Amazon Web Services API)对执行特定任务的API每秒可执行的请求数量有限制。例如,Route 53的每秒限制为5个请求,尽管它们允许在一个请求中批处理多个操作。当达到此限制时,你很可能会收到某种Rate Limit Exceeded错误,表明已超过允许的请求数阈值。那你准备怎么办?你认为它就是一个失败并在你的应用程序中处理?实际上,你可以等待一段时间,然后重试该操作。有许多策略可以让你“退后”一段时间,并在延迟后重试操作。

退避Backoff策略

让我们考虑一个do_something()函数,其中代码执行一些容易出现瞬态失败的操作。我们将使用示例来考虑策略,github源码

1. 没有退避

这是默认方案。如果失败,可以立即重试,不要等待。请记住使用最大允许重试限制尝试次数,否则这可能会永远持续下去。代码可能如下所示:

def no_backoff(max_retry: int):
""" No backoff. If things fail, retry immediately. """

counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:
        print('> No Backoff > Retry number: %s' % counter)

    counter += 1
<p>

2.恒定Constant退避

每次尝试后,恒定退避是指增加固定延迟,在这种情况下,我们会在每次等待1秒钟以后再尝试。

def constant_backoff(max_retry: int):
""" Constant backoff. If things fail, retry after a fixed amount of delay. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = 1
        print('> Constant Backoff > Retry number: %s, sleeping for %s seconds.' % (counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Constant Backoff > Total delay: %s seconds.' % total_delay)
<p>

3.线性退避

在线性回退中,延迟随着每次尝试而增加,遵循线性曲线。

def linear_backoff(max_retry: int):
""" Linear backoff. If things fail, retry with linearly increasing delays. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = counter
        print('> Linear Backoff > Retry number: %s, sleeping for %s seconds.' % (counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Linear Backoff > Total delay: %s seconds.' % total_delay)
<p>

4.斐波那契退避

我们也可以根据重试计数器的Fibonacci系列的总和进行延迟。

def fibonacci_backoff(max_retry: int):
""" Fibonacci backoff. If things fail, retry with delays increasing by fibonacci numbers. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = get_fib(counter)
        print(
                '> Fibonacci Backoff > Retry number: %s, sleeping for %s seconds.' % (
                    counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Fibonacci Backoff > Total delay: %s seconds.' % total_delay)
<p>

5. 二次退避

延迟也可以遵循二次曲线。

def quadratic_backoff(max_retry: int):
""" Quadratic backoff. If things fail, retry with polynomially increasing delays. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = counter ** 2
        print(
                '> Quadratic Backoff > Retry number: %s, sleeping for %s seconds.' % (
                    counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Quadratic Backoff > Total delay: %s seconds.' % total_delay)
<p>

6. 指数退避

延迟也可以遵循指数曲线,如下例所示。

def exponential_backoff(max_retry: int):
""" Exponential backoff. If things fail, retry with exponentially increasing delays. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = 2 ** counter
        print(
                '> Exponential Backoff > Retry number: %s, sleeping for %s seconds.' % (
                    counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Exponential Backoff > Total delay: %s seconds.' % total_delay)
<p>

7. 多项式退避

延迟也可以遵循多项式函数。

def polynomial_backoff(max_retry: int):
""" Polynomial backoff. If things fail, retry with polynomially increasing delays. """

total_delay = 0
counter = 0
while counter < max_retry:

    response = do_something()
    if response:

        return True
    else:

        sleepy_time = counter ** 3
        print(
                '> Polynomial Backoff > Retry number: %s, sleeping for %s seconds.' % (
                    counter, sleepy_time))
        total_delay += sleepy_time
        sleep(sleepy_time)

    counter += 1

print('> Polynomial Backoff > Total delay: %s seconds.' % total_delay)
<p>

8. 总延误

使用的策略取决于你想要的延迟类型。以下是上述算法的运行,以及它们在几秒钟内导致的总延迟,适用于各种最大重试次数。

Starting experiments with maximum 1 retries.
> Constant Backoff > Total delay: 1 seconds.
> Linear Backoff > Total delay: 0 seconds.
> Fibonacci Backoff > Total delay: 0 seconds.
> Quadratic Backoff > Total delay: 0 seconds.
> Exponential Backoff > Total delay: 1 seconds.
> Polynomial Backoff > Total delay: 0 seconds.

Starting experiments with maximum 3 retries.
> Constant Backoff > Total delay: 3 seconds.
> Linear Backoff > Total delay: 3 seconds.
> Fibonacci Backoff > Total delay: 2 seconds.
> Quadratic Backoff > Total delay: 5 seconds.
> Exponential Backoff > Total delay: 7 seconds.
> Polynomial Backoff > Total delay: 9 seconds.

Starting experiments with maximum 5 retries.
> Constant Backoff > Total delay: 5 seconds.
> Linear Backoff > Total delay: 10 seconds.
> Fibonacci Backoff > Total delay: 7 seconds.
> Quadratic Backoff > Total delay: 30 seconds.
> Exponential Backoff > Total delay: 31 seconds.
> Polynomial Backoff > Total delay: 100 seconds.

Starting experiments with maximum 10 retries.
> Constant Backoff > Total delay: 10 seconds.
> Linear Backoff > Total delay: 45 seconds.
> Fibonacci Backoff > Total delay: 88 seconds.
> Quadratic Backoff > Total delay: 285 seconds.
> Exponential Backoff > Total delay: 1023 seconds.
> Polynomial Backoff > Total delay: 2025 seconds.

Starting experiments with maximum 20 retries.
> Constant Backoff > Total delay: 20 seconds.
> Linear Backoff > Total delay: 190 seconds.
> Fibonacci Backoff > Total delay: 10945 seconds.
> Quadratic Backoff > Total delay: 2470 seconds.
> Exponential Backoff > Total delay: 1048575 seconds.
> Polynomial Backoff > Total delay: 36100 seconds.
<p>

9. 封顶/截断延迟

使用算法为重试逻辑添加延迟时,请记住限制这些延迟。在我们的示例中,我们将重试尝试限制/限制为10次重试,这可能就足够了。请记住,我们最终添加的延迟总量取决于我们的应用程序do_something()在下一次迭代中实际运行和到达它所花费的时间,以及我们使用的算法。在线性退避的情况下,10次重试增加45秒,但使用多项式退避策略超过30分钟。在选择方法之前运行关于实际延迟的方案非常重要。

此外,不是按重试次数限制延迟,最好将延迟限制在最大允许延迟时间,以便它们在一段时间后保持平稳。你可以检查一下这个值sleepy_time 变量并确保它永远不会设置为大于您指定的阈值的值。

抖动/随机性的案例

如果你发现即使在使用上述方法之后,也存在多个客户端争用相同资源并面临瞬态故障的情况,考虑添加一些随机性sleepy_time以分散调用的时间。

Retry Strategies for Transient Failures - DEV Comm