we've found the issue. for reference:
Our service provider for the 4G connection, KPN in the Netherlands, confirmed they are limiting the maximum number of connections to 512 when you use the automatic APN selection, or to be more precise: this limit are is place for any APN they broadcast other than the APN called "advancedinternet".
When you need a less restricted (in regards to maximum connections, port usage etc.) and without extra NAT 4G connection, you have to use the APN "advancedinternet".
Recommendations are to have good/better security in place for the connection, thinking of a firewall etc, because the less secure nature of the connection compared to the default 4G APN.
Setting that APN solved our issues.