I think I have solved this problem.
To prevent the modem using voice connections, I used the AT command
gsmctl -A 'AT+QCFG="servicedomain",1'
To be super sure that it runs every time, I added this line in /etc/rc.local
I also do a full reboot after 2 failed pings (every 15 minutes) to prevent getting stuck in a non-connected state. For some reason, just doing the modem reboot did not work as well.