I'd like to request for a little bit more detail regarding this issue. I've tested WG tunnel on my end using a large file (3GB+) and trying to transfer it from one end-point to another end-point via WG tunnel from one LAN device to another LAN device. I've also tried transferring the same data over the internet but I did not seem to run into any issues regardless of which interface was in use for the test - mobile or wired (Ethernet).
I've also done some testing with iperf3 and it seems like router can handle ~350-400 Mbps throughput over the tunnel when running in TCP mode without any bandwidth limitations and using wired WAN. It did struggle quite a bit with the UDP test, dropping packets consistently when transferring any sort of data faster than ~125-150 Mbps but after digging a little bit deeper into this problem it seems to be a separate issue with iperf3 package itself.
The mobile data transfer was (naturally) much lower than over the wire but it did not hang up during a real file transfer and when testing it with iperf3.
It is unlikely that it's an MTU-related issue in this case. In regards to MTU, the maximum permitted MTU value will be dictated mostly by the mobile operator that you've connected to and they may be dropping packets which are too large (possibly at their edge). Also, MTU being the issue doesn't explain why tunnel would work with an older firmware rather than a new one since in both firmware versions the default MTU value of mobile interface is the same.
This may be a very specific issue with the newest firmware and the server/client data exchange, with some external variables also possibly impacting performance of the tunnel. Could you try setting up an alternative wireguard tunnel somewhere else (can also do it online by using, for example, a VPS) and testing whether these issues occur with different devices when RUTX11 tries to transfer traffic to them via the wireguard tunnel? I'd like to get to the bottom of this issue but it sounds like this one may be quite difficult to catch in action consistently unless the environment would be setup in an identical manner just like the first time.
Any information or potential thoughts would be highly appreciated as well.