Hello,
I have just tested SCP file transfer via IPSec tunnel from RUTX11 (Firmware version RUTX_R_00.02.6) to RUT955 and it worked as expected. I have established a tunnel between RUTX11 and RUT955, uploaded the file to RUTX11, and used the command:
scp file.exe root@192.168.2.1:/tmp
The file size was 65MB and was transferred successfully. Inform me if your IPSec configuration differs.
I would suggest to reinstall your firmware version to the router without keeping any settings and try again.
Regards.