This is a known issue. The limit comes from WebUI validation side only. Currently, this limit is planned to be removed in 7.5 firmware version.
As a temporary workaround, configuration details can be modified by SSH, specifically targeting /etc/config/ipsec file.
First, I suggest to create a working configuration in the current WebUI by giving any name to the instance within the 8 character limit and edit name in the SSH afterwards. Also, make sure the tunnel establishes and configuration is correct, because, because, if instance name is longer than 8 characters, WebUI will not accept configuration changes.
Below are the instructions, on how to modify ipsec configuration via SSH. To login, use command line interface from router's WebUI in System -> CLI page or an SSH client such as PuTTy. Use root as username and router's password to login.
Execute the following command to open configuration file with a text editor:
Press letter i to enable editing and change every option, including instance name, set in the WebUI.
Once done, press Esc, then enter :wq and press Enter.
Then, execute the following command:
- /etc/init.d/ipsec restart
That should be enough to continue using ipsec.