[warm-reboot] Warm reboot failed on T0 due to pending tasks

[msosyak]

Description
warm-reboot command failed with following error:

admin@sonic:~$ sudo warm-reboot 
RESTARTCHECK failed
admin@sonic:~$ sudo cat /var/log/syslog 
* * * 
Jan 21 17:32:12.888204 cab18-2-dut INFO swss#orchagent_restart_check: :- subscribe: subscribed to RESTARTCHECKREPLY
Jan 21 17:32:12.888204 cab18-2-dut NOTICE swss#orchagent_restart_check: :- main: requested orchagent to do warm restart state check, retry count: 0
Jan 21 17:32:12.888366 cab18-2-dut NOTICE swss#orchagent: :- doTask: RESTARTCHECK notification for orchagent 
Jan 21 17:32:12.888366 cab18-2-dut NOTICE swss#orchagent: :- doTask: orchagent|NoFreeze:false|SkipPendingTaskCheck:false
Jan 21 17:32:12.888530 cab18-2-dut NOTICE swss#orchagent: :- warmRestartCheck: WarmRestart check found pending tasks: 
Jan 21 17:32:12.888530 cab18-2-dut NOTICE swss#orchagent: :- warmRestartCheck:     ROUTE_TABLE:10.1.0.1|SET|nexthop:0.0.0.0|ifname:Loopback0
Jan 21 17:32:12.888530 cab18-2-dut NOTICE swss#orchagent: :- warmRestartCheck: Restart check result: NOT_READY
Jan 21 17:32:12.888652 cab18-2-dut NOTICE swss#orchagent_restart_check: :- main: RESTARTCHECK failed, orchagent is not ready for warm restart with status NOT_READY
Jan 21 17:32:18.735294 cab18-2-dut INFO swss#orchagent: :- addRoute: Failed to get next hop 0.0.0.0@Loopback0 for 10.1.0.1/32
* * * 

Steps to reproduce the issue:

1. deploy t0 topology
2. run warm-reboot command

Describe the results you received:
The switch is not rebooted

Describe the results you expected:
The switch should do a warm reboot

Additional information you deem important (e.g. issue happens only occasionally):

Output of show version:

admin@sonic:~$ show version 

SONiC Software Version: SONiC.HEAD.543-dirty-20200121.042134
Distribution: Debian 9.11
Kernel: 4.9.0-9-2-amd64
Build commit: d0d1a559
Build date: Tue Jan 21 05:52:57 UTC 2020
Built by: johnar@jenkins-worker-12

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[msosyak]

Looks like some configuration for Loopback0 interface is not cleared during deploy topology.
Here is the DB dump and ip addr output before deploy topology:

"LOOPBACK_INTERFACE": {
        "Loopback0": {}, 
        "Loopback0|10.1.0.1/32": {}
    }, 

 Loopback0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ee:ea:e3:48:45:b0 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/32 scope global Loopback0
       valid_lft forever preferred_lft forever

after deploy topology command:

"LOOPBACK_INTERFACE": {
        "Loopback0": {}, 
        "Loopback0|10.1.0.32/32": {}, 
        "Loopback0|FC00:1::32/128": {}
},

Loopback0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ee:ea:e3:48:45:b0 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/32 scope global Loopback0
       valid_lft forever preferred_lft forever
    inet 10.1.0.32/32 scope global Loopback0
       valid_lft forever preferred_lft forever

and ip addr after reboot:

Loopback0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 3a:be:88:cf:cc:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.32/32 scope global Loopback0
       valid_lft forever preferred_lft forever

[xinliu-seattle]

@yxieca please help to triage this issue in the warm reboot subgroup.

[yxieca]

@msosyak can you give a bit more detail on:

• What is your platform / hardware sku (unlikely related, just want to try with a testbed similar to yours).
• What is the command you used to deploy? (I supposed that you used testbed-cli.sh to deploy, please give me the parameters you provided and the testbed definition in testbed.csv file).
• If the deployment includes uploading a minigraph and run config load_minigraph, please upload the minigraph too.

[msosyak]

Close this issue as it is not reproducible anymore on latest image.
I look like this sonic-net/sonic-swss#1141 PR fixed this issue.