In this case, you can randomly disable the public IP address from the virtual machine---network interface---IP Configurations---ipconfig1---disassociate---save. Then you can add the desired VMs to backend pool again.
Do this for rest of the VM Servers wanted to be added in Load Balancer.