Issue Description

Customer environment having 3 nodes deployment. When login to VMP, it always showing 2 of the nodes is down.

Most of the time user also facing the admin account could not login.

Error/Warning Information

When user login to VMP using cluster IP, it also showing error "It is synchronizing data among clustered nodes. Please try again later".

Handling Process

1. Perform ping test between 3 VMP node, found that the nodes management IP is pingable and able to SSH device back end using port 22.

2. Access to device backend, it shown that the cluster is not stable.

3. Perform CHPING to test the multicast connection between node. Download the script in the KB attachment and upload to "/sf" partition. The result show the multicast is working fine.

   Command: ./chping $hostname
   

4. Check for the log also not found any abnormal for VMP sides, suspect is caused by physical switch configuration

5. Request for network topology from user to have more understanding of the deployment of VDI, found that after VMP nodes, the netwrk flow will be going through NETGEAR switch and H3C core switch.p

6. User already disabled the IGMP snooping and STP features on physical switch (It’s advised to disable these 2 protocol to prevent cluster instability).

7. Try to check on user NETGEAR brand switch, found that it is using "dynamic" broadcase forwarding. This will cause the broadcast packet loss between node, and lead to abnormal cluster communication. It is recommended change to "Hardware" mode.
   

8. After changed the forwarding method to hardware, the nodes able to detect each other and the cluster status become stable and normal.
   

Root Cause

Dynamic mode will switch between software and hardware mode. When in software mode, the broadcast frames will be discarded if the rate is higher than the "software" mode can handle.

Solution

Change the NETGEAR switch broadcast forwarding method to hardware.