I have a similar issue. Could you check with tcpdump that you don't have ICMP address unreachable errors from br-lan:
tcpdump -i br-lan -n -v 'host internet-address or icmp'
If you have ICMP errors could you check the rules :
ip -4 rule show
ip -6 rule show
Look at rule 2062 you can retry the internet address access after deleting it.