采用异常重试实现故障恢复

• 网络延迟或者抖动
• 服务器资源不足（CPU、内存走高、连接池满）
• 服务器故障
• 符合某些特定条件下的服务程序bug（大都非必现）
  

2 系统稳定性等级划分

大部分服务容忍低频、偶发的5xx错误，并使用可用性级别来衡量系统的健壮性，级别系数越高，健壮性越好，如下：

等级描述	故障时长（年）	可用行等级
基本可用性	87.6h	99%
较高可用	8.8h	99.9%
非常高的可用性（大部分故障可自动恢复）	52m	99.99%
极高可用性	5m	99.999%

对于强系统可靠性、强结果预期性 要求的系统，如转账、下单、付款，即使微小的可用性降级也是不可接受，用户强烈需要接收到正确的结果。
可以想想你付款的时候发现付款失败有多么惊慌，订外卖的时候获取信息失败有多么沮丧，这些都是用户痛点。

请求异常的治理手段

3.1 采用异常重试实现故障恢复

通过上面的故障原因分析我们知道，排除了必现的程序逻辑错误，大部分环境导致的错误是可以通过重试进行恢复的。
治理的手段主要是采用 异常重试 来实现的，通过重试负载到健康实例上（实例越多重试成功率越高），降低用户感知到的故障频率。

执行过程说明

• 这边以示例服务 Svc-A 向 Svc-B 发起访问为例子。
• 第1次执行失败之后，根据策略，间隔25ms之后发起第2次请求。
• 会看到有两条日志，日志的trace_id 一致，说明他是同一个调用过程（1个调用过程，包含2次请求，首发1次与重试1次）
• 请求方为同一个实例 Svc-A-Instance1，说明请求发起方一致。
• 被请求方发生了变动，说明调度到新的实例（Svc-B-Instance1 到 Svc-B-Instance2）。
• 返回正常的 200 。

因为我们的负载均衡模式默认是RR，所以实例越多，实际上重试成功的概率会越高。比如有50个实例，其中一个实例出故障，导致执行返回5xx，那么第二次请求的时候一般来说会有 49/50 的成功概率。如下图：

 

 

 

 

www.51sole.com/xinxi/454872898.htm
www.51sole.com/xinxi/454872977.htm
www.51sole.com/xinxi/454873028.htm
www.51sole.com/xinxi/454873098.htm
www.51sole.com/xinxi/454873221.htm
www.51sole.com/xinxi/454873268.htm
www.51sole.com/xinxi/454873340.htm
www.51sole.com/xinxi/454873391.htm
www.51sole.com/xinxi/454873464.htm
www.51sole.com/xinxi/454873535.htm
www.51sole.com/xinxi/454873605.htm
www.51sole.com/xinxi/454873680.htm
www.51sole.com/xinxi/454873765.htm
www.51sole.com/xinxi/454873820.htm
www.51sole.com/xinxi/454873902.htm
www.51sole.com/xinxi/454873963.htm
www.51sole.com/xinxi/454874031.htm
www.51sole.com/xinxi/454874081.htm
www.51sole.com/xinxi/454881989.htm
www.51sole.com/xinxi/454882049.htm
www.51sole.com/xinxi/454882167.htm
www.51sole.com/xinxi/454882212.htm
www.51sole.com/xinxi/454882273.htm
www.51sole.com/xinxi/454882357.htm
www.51sole.com/xinxi/454882434.htm
www.51sole.com/xinxi/454882495.htm
www.51sole.com/xinxi/454882552.htm
www.51sole.com/xinxi/454882670.htm
www.51sole.com/xinxi/454882759.htm
www.51sole.com/xinxi/454882829.htm
www.51sole.com/xinxi/454882872.htm
www.51sole.com/xinxi/454882937.htm
www.51sole.com/xinxi/454882987.htm
www.51sole.com/xinxi/454883037.htm
www.51sole.com/xinxi/454883116.htm
www.51sole.com/xinxi/454883175.htm
www.51sole.com/xinxi/454883295.htm
www.51sole.com/xinxi/454883355.htm
www.51sole.com/xinxi/454883416.htm
www.51sole.com/xinxi/454883515.htm
www.51sole.com/xinxi/454883598.htm
www.51sole.com/xinxi/454883731.htm
www.51sole.com/xinxi/454883836.htm
www.51sole.com/xinxi/454883899.htm
www.51sole.com/xinxi/454883949.htm
www.51sole.com/xinxi/454884016.htm
www.51sole.com/xinxi/454884077.htm
www.51sole.com/xinxi/454884133.htm
www.51sole.com/xinxi/454884211.htm
www.51sole.com/xinxi/454884285.htm
www.51sole.com/xinxi/454884358.htm
www.51sole.com/xinxi/454884414.htm
www.51sole.com/xinxi/454884472.htm
www.51sole.com/xinxi/454884548.htm
www.51sole.com/xinxi/454884598.htm
www.51sole.com/xinxi/454884643.htm
www.51sole.com/xinxi/454884711.htm
www.51sole.com/xinxi/454884778.htm
www.51sole.com/xinxi/454892008.htm
www.51sole.com/xinxi/454892058.htm
www.51sole.com/xinxi/454892164.htm
www.51sole.com/xinxi/454892204.htm
www.51sole.com/xinxi/454892236.htm
www.51sole.com/xinxi/454892339.htm
www.51sole.com/xinxi/454892393.htm
www.51sole.com/xinxi/454892462.htm
www.51sole.com/xinxi/454892533.htm
www.51sole.com/xinxi/454892633.htm
www.51sole.com/xinxi/454892762.htm
www.51sole.com/xinxi/454892817.htm
www.51sole.com/xinxi/454892867.htm
www.51sole.com/xinxi/454892930.htm
www.51sole.com/xinxi/454892997.htm
www.51sole.com/xinxi/454893045.htm
www.51sole.com/xinxi/454893092.htm
www.51sole.com/xinxi/454893141.htm
www.51sole.com/xinxi/454893200.htm
www.51sole.com/xinxi/454893253.htm
www.51sole.com/xinxi/454893300.htm
www.51sole.com/xinxi/454893408.htm
www.51sole.com/xinxi/454893476.htm
www.51sole.com/xinxi/454893592.htm
www.51sole.com/xinxi/454893635.htm
www.51sole.com/xinxi/454893693.htm
www.51sole.com/xinxi/454893797.htm
www.51sole.com/xinxi/454893853.htm
www.51sole.com/xinxi/454893902.htm
www.51sole.com/xinxi/454893969.htm
www.51sole.com/xinxi/454894025.htm
www.51sole.com/xinxi/454894077.htm
www.51sole.com/xinxi/454894130.htm
www.51sole.com/xinxi/454894184.htm
www.51sole.com/xinxi/454894220.htm
www.51sole.com/xinxi/454894251.htm
www.51sole.com/xinxi/454894302.htm
www.51sole.com/xinxi/454894353.htm
www.51sole.com/xinxi/454894412.htm
www.51sole.com/xinxi/454894461.htm
www.51sole.com/xinxi/454898175.htm
www.51sole.com/xinxi/454898215.htm
www.51sole.com/xinxi/454898275.htm
www.51sole.com/xinxi/454898312.htm
www.51sole.com/xinxi/454898342.htm
www.51sole.com/xinxi/454898377.htm
www.51sole.com/xinxi/454898413.htm
www.51sole.com/xinxi/454898455.htm
www.51sole.com/xinxi/454898524.htm
www.51sole.com/xinxi/454898547.htm
www.51sole.com/xinxi/454898617.htm
www.51sole.com/xinxi/454898647.htm
www.51sole.com/xinxi/454898671.htm
www.51sole.com/xinxi/454898734.htm
www.51sole.com/xinxi/454898768.htm
www.51sole.com/xinxi/454898806.htm
www.51sole.com/xinxi/454898846.htm
www.51sole.com/xinxi/454898879.htm
www.51sole.com/xinxi/454898913.htm
www.51sole.com/xinxi/454898951.htm
www.51sole.com/xinxi/454899022.htm
www.51sole.com/xinxi/454899090.htm
www.51sole.com/xinxi/454899111.htm
www.51sole.com/xinxi/454899141.htm
www.51sole.com/xinxi/454899181.htm
www.51sole.com/xinxi/454899220.htm
www.51sole.com/xinxi/454899288.htm
www.51sole.com/xinxi/454899319.htm
www.51sole.com/xinxi/454899356.htm
www.51sole.com/xinxi/454899388.htm
www.51sole.com/xinxi/454899419.htm
www.51sole.com/xinxi/454899456.htm
www.51sole.com/xinxi/454899480.htm
www.51sole.com/xinxi/454899513.htm
www.51sole.com/xinxi/454899544.htm
www.51sole.com/xinxi/454899612.htm
www.51sole.com/xinxi/454899642.htm
www.51sole.com/xinxi/454899675.htm
www.51sole.com/xinxi/454899720.htm
www.51sole.com/xinxi/454899758.htm
www.51sole.com/xinxi/454904626.htm
www.51sole.com/xinxi/454904665.htm
www.51sole.com/xinxi/454904727.htm
www.51sole.com/xinxi/454904774.htm
www.51sole.com/xinxi/454904803.htm
www.51sole.com/xinxi/454904839.htm
www.51sole.com/xinxi/454904885.htm
www.51sole.com/xinxi/454904919.htm
www.51sole.com/xinxi/454905038.htm
www.51sole.com/xinxi/454905067.htm
www.51sole.com/xinxi/454905157.htm
www.51sole.com/xinxi/454905198.htm
www.51sole.com/xinxi/454905229.htm
www.51sole.com/xinxi/454905264.htm
www.51sole.com/xinxi/454905301.htm
www.51sole.com/xinxi/454905344.htm
www.51sole.com/xinxi/454905395.htm
www.51sole.com/xinxi/454905432.htm
www.51sole.com/xinxi/454905463.htm
www.51sole.com/xinxi/454905510.htm
www.51sole.com/xinxi/454905587.htm
www.51sole.com/xinxi/454905656.htm
www.51sole.com/xinxi/454905727.htm
www.51sole.com/xinxi/454905762.htm
www.51sole.com/xinxi/454905801.htm
www.51sole.com/xinxi/454905831.htm
www.51sole.com/xinxi/454905927.htm
www.51sole.com/xinxi/454906016.htm
www.51sole.com/xinxi/454906049.htm
www.51sole.com/xinxi/454906098.htm
www.51sole.com/xinxi/454906130.htm
www.51sole.com/xinxi/454906166.htm
www.51sole.com/xinxi/454906206.htm
www.51sole.com/xinxi/454906270.htm
www.51sole.com/xinxi/454906346.htm
www.51sole.com/xinxi/454906395.htm
www.51sole.com/xinxi/454906445.htm
www.51sole.com/xinxi/454906497.htm
www.51sole.com/xinxi/454906556.htm
www.51sole.com/xinxi/454906618.htm
www.51sole.com/xinxi/454912174.htm
www.51sole.com/xinxi/454912230.htm
www.51sole.com/xinxi/454912303.htm
www.51sole.com/xinxi/454912353.htm
www.51sole.com/xinxi/454912391.htm
www.51sole.com/xinxi/454912475.htm
www.51sole.com/xinxi/454912504.htm
www.51sole.com/xinxi/454912551.htm
www.51sole.com/xinxi/454912649.htm
www.51sole.com/xinxi/454912690.htm
www.51sole.com/xinxi/454912770.htm
www.51sole.com/xinxi/454912822.htm
www.51sole.com/xinxi/454912857.htm
www.51sole.com/xinxi/454912898.htm
www.51sole.com/xinxi/454912941.htm
www.51sole.com/xinxi/454912978.htm
www.51sole.com/xinxi/454913006.htm
www.51sole.com/xinxi/454913051.htm
www.51sole.com/xinxi/454913100.htm
www.51sole.com/xinxi/454913141.htm
www.51sole.com/xinxi/454913192.htm
www.51sole.com/xinxi/454913291.htm
www.51sole.com/xinxi/454913354.htm
www.51sole.com/xinxi/454913408.htm
www.51sole.com/xinxi/454913459.htm
www.51sole.com/xinxi/454913488.htm
www.51sole.com/xinxi/454913561.htm
www.51sole.com/xinxi/454913635.htm
www.51sole.com/xinxi/454913689.htm
www.51sole.com/xinxi/454913731.htm
www.51sole.com/xinxi/454913780.htm
www.51sole.com/xinxi/454913821.htm
www.51sole.com/xinxi/454913871.htm
www.51sole.com/xinxi/454913910.htm
www.51sole.com/xinxi/454913957.htm
www.51sole.com/xinxi/454913994.htm
www.51sole.com/xinxi/454914043.htm
www.51sole.com/xinxi/454914120.htm
www.51sole.com/xinxi/454914162.htm
www.51sole.com/xinxi/454914209.htm
www.51sole.com/xinxi/454918918.htm
www.51sole.com/xinxi/454918961.htm
www.51sole.com/xinxi/454919025.htm
www.51sole.com/xinxi/454919057.htm
www.51sole.com/xinxi/454919102.htm
www.51sole.com/xinxi/454919163.htm
www.51sole.com/xinxi/454919208.htm
www.51sole.com/xinxi/454919245.htm
www.51sole.com/xinxi/454919314.htm
www.51sole.com/xinxi/454919348.htm
www.51sole.com/xinxi/454919418.htm
www.51sole.com/xinxi/454919464.htm
www.51sole.com/xinxi/454919504.htm
www.51sole.com/xinxi/454919551.htm
www.51sole.com/xinxi/454919584.htm
www.51sole.com/xinxi/454919612.htm
www.51sole.com/xinxi/454919676.htm
www.51sole.com/xinxi/454919710.htm
www.51sole.com/xinxi/454919748.htm
www.51sole.com/xinxi/454919780.htm
www.51sole.com/xinxi/454919810.htm
www.51sole.com/xinxi/454919893.htm
www.51sole.com/xinxi/454919964.htm
www.51sole.com/xinxi/454919992.htm
www.51sole.com/xinxi/454920040.htm
www.51sole.com/xinxi/454920070.htm
www.51sole.com/xinxi/454920141.htm
www.51sole.com/xinxi/454920207.htm
www.51sole.com/xinxi/454920242.htm
www.51sole.com/xinxi/454920284.htm
www.51sole.com/xinxi/454920311.htm
www.51sole.com/xinxi/454920345.htm
www.51sole.com/xinxi/454920371.htm
www.51sole.com/xinxi/454920402.htm
www.51sole.com/xinxi/454920442.htm
www.51sole.com/xinxi/454920475.htm
www.51sole.com/xinxi/454920523.htm
www.51sole.com/xinxi/454920555.htm
www.51sole.com/xinxi/454920592.htm
www.51sole.com/xinxi/454920614.htm
www.51sole.com/xinxi/454924381.htm
www.51sole.com/xinxi/454924412.htm
www.51sole.com/xinxi/454924477.htm
www.51sole.com/xinxi/454924508.htm
www.51sole.com/xinxi/454924536.htm
www.51sole.com/xinxi/454924605.htm
www.51sole.com/xinxi/454924642.htm
www.51sole.com/xinxi/454924663.htm
www.51sole.com/xinxi/454924719.htm
www.51sole.com/xinxi/454924760.htm
www.51sole.com/xinxi/454924793.htm
www.51sole.com/xinxi/454924818.htm
www.51sole.com/xinxi/454924841.htm
www.51sole.com/xinxi/454924893.htm
www.51sole.com/xinxi/454924923.htm
www.51sole.com/xinxi/454924945.htm
www.51sole.com/xinxi/454924978.htm
www.51sole.com/xinxi/454925002.htm
www.51sole.com/xinxi/454925023.htm
www.51sole.com/xinxi/454925056.htm
www.51sole.com/xinxi/454925086.htm
www.51sole.com/xinxi/454925151.htm
www.51sole.com/xinxi/454925197.htm
www.51sole.com/xinxi/454925231.htm
www.51sole.com/xinxi/454925256.htm
www.51sole.com/xinxi/454925288.htm
www.51sole.com/xinxi/454925345.htm
www.51sole.com/xinxi/454925409.htm
www.51sole.com/xinxi/454925442.htm
www.51sole.com/xinxi/454925468.htm
www.51sole.com/xinxi/454925500.htm
www.51sole.com/xinxi/454925542.htm
www.51sole.com/xinxi/454925574.htm
www.51sole.com/xinxi/454925611.htm
www.51sole.com/xinxi/454925647.htm
www.51sole.com/xinxi/454925679.htm
www.51sole.com/xinxi/454925712.htm
www.51sole.com/xinxi/454925744.htm
www.51sole.com/xinxi/454925762.htm
www.51sole.com/xinxi/454925792.htm