There is no specific limit set to the number of available IPSec instances. As you mentioned, the hardware can be a limiting factor when it comes to handling multiple VPN connections. The CPU and memory of the device need to process and encrypt/decrypt the traffic, which can consume system resources. The amount and type of traffic can also affect the device's performance. If the data being sent and received is substantial, it can strain the resources of the device and impact its ability to handle multiple connections.
You can try using simpler encryption algorithms to reduce the load on the CPU and improve the performance. Basically, this would alleviate some of the load on the hardware and potentially accommodate a larger number of connections.
I would also suggest checking that there are no conflicts in the IPSec configurations. Subnet overlaps, as well as instances with identical identifiers or certificates, can also result in conflicts and disrupt the VPN connections.
If needed, you can check IPsec information with the following commands via CLI/SSH:
- logread | grep ipsec
- ipsec statusall