Question

Hello,

We are facing a regression when updating our router RUTX11 from firmware version RUTX_R_00.02.06.1 to version RUTX_R_00.07.03.3.

A SIM card is inserted into the router (wwan0). On the LAN side, we have 6 nodes (clients) on the network 192.168.1.0/24.

We noticed a problem after the update. We could not ssh to the nodes via the WAN. The connection hanged.
After doing some research we found out that with a different ssh cipher (aes128-str) the ssh connection worked again. Finally, we found out that we no longer have to specify this cipher (kind of a hack) if we specified the MTU of the router (interface wwan0) to 1500 (which was unset in the router web UI, but using ifconfig on the router we saw 1460 as the defacto default value).

Problem solved, right? Nope, changing the MTU led to having problems on the nodes themselves. On the nodes, we have rclone (version 1.59.2) installed. We use it to upload files to AWS S3. The problem is that now that we have set the router's MTU to 1500, we cannot upload to S3 anymore. If we revert the MTU back to the default value, the upload works again, but no ssh. Also, it seems that very small files are uploaded but not bigger ones. Also, we know that this is not specific to S3, because we setup a webdav backend on rclone and the symptoms were the same.

In summary:

Old (working) firmware RUTX_R_00.02.06.1:

* SSH connection to a node via wwan0: OK
* rclone uploads: OK

New firmware RUTX_R_00.07.03.3 (with default (unset) MTU = 1460)

* SSH connection hangs: KO
* rclone uploads: OK

New firmware RUTX_R_00.07.03.3 (with MTU manually set to 1500)

* SSH connection to a node via wwan0: OK
* rclone cannot upload (bigger) files: KO

An other piece of information that can be useful: if we keep the default config but change the SIM card (i.e. use one from an other ISP), everything works with the new firmware. So, I am guessing it may be related to some kind of (MTU? other settings?) negotiation between the router and the ISP's network.

Finally, I tried to run tcpdump on both the nodes and the router to capture what is going on but my knowledge with regards to tcpdump/wireshark frames decoding is very limited. I do see black and red lines on Wireshark (which is not good :) ) but cannot explain what they mean. I can provide these if needed.

From my (poor) understanding, it feels like the new firmware (although we also did some tests with the latest firmware to date which resulted with the same conclusions) changes something with the default MTU or fails to negotiate a correct one with the SIM card provider (although it seems to work with an other ISP, don't know why).

Do you have any idea what could explain this issue? Do you have any hints to solve this problem?

I can run as many tests as needed (it is quite reproducible) and provide additional context if needed.

Thank you in advance.

Best regards,
Gary

Answer 1

Hello,

Could you update your router's firmware to RUTX_R_00.07.04.3? In 7.04 and above versions MTU value is by default set to 1500.

Best regards,

Comments

Hello,

Thanks for the answer. We already tried that one with no improvements sadly

From my (poor) understanding, it feels like the new firmware (although we also did some tests with the latest firmware to date which resulted with the same conclusions)

Any other hints?

---

For one you could login to the router via CLI/SSH and execute the following command:

• gsmctl -A 'AT+QMTUINFO'

It will give MTU value advertised by the mobile operator.

Otherwise, I would like you to attach a troubleshoot a couple of troubleshoot files from the device with different firmware versions to look for any differences, which might cause connectivity issues. To generate the files, access router's WebUI, go to System -> Administration -> Troubleshoot section and download troubleshoot file from there. 

Best regards,

Answer 2

Hi,

I'm a colleague of gary.erx, and as he's out of office I'll try to summarize our investigations so far on this case.

We did multiple tests, with three different operators, in two different locations (we have two offices), and with firmware 2.6.1, 7.3.3 and 7.4.3. We tried to reproduce systematically the issue, using two types of connections. We have a device runing Ubuntu attached to the Lan of the router, running Wireguard client, and also using rclone from this device to outside (rclone target doesn't matter, problem is the same with S3, Google Drive, ...).

As a conclusion, we observed that the Location is more important than any other parameters, as at one office's location everything is OK (for every operator and protocol), and at the other one, depending on the firmware version, either ssh through wireguard OR rclone is not working.

We struggle to understand how this is supposed to work, as obviously in addition of the operator the antenna seems to have an active role.

We would appreciate any help you may provide, thanks !

Update: After having a call to an engineer working for an GSM operator, I realized that the number of 4G frequencies available on each antenna is different, and therefore the bandwidth. The router should then negociate the right MTU using MSS, and this seems to be not well working on recent firmwares (our guess).

Is there any other "internal" commands we might use to diagnosticate this deeper ?

Comments

Hello,

MTU value is indeed not negotiated, at least in 7.4 firmware versions.

I would suggest to set it to 1420, as it is a proven value to work in many related cases.  

Alternatively, I refer to the output of the gsmctl -A 'AT+QMTUINFO' command. For wireguard tunnels, reduce this value by 80 (WG header size).

Automatic MTU negotiation is planned to be reimplemented in 7.5 firmware version, but at the moment this is still a matter of change.

Best regards,

Answer 3

Hello,

I confirm that pascal.roulin.erx is my colleague.

Thanks for the answer. We still have several questions with regards to this issue.

Automatic MTU negotiation is planned to be reimplemented in 7.5 firmware version

Why the this feature was removed? Was it a regression or was it done on purpose? If it's the latter, can you explain why?

Does you have an ETA for the version 7.5?

For wireguard tunnels, reduce this value by 80 (WG header size).

On some our routers deployed in Spain, we noticed that, as opposed to the many other routers we deployed in Europe which return a MTU of 1460, it returns a MTU of 1280. If we would configure wireguard to have a MTU of 1460-80=1380, it would mean that this value won't work for this machine in Spain. Did I understand that correctly? If so, should we query the MTU obtained by the router to dynamically configure our wireguard client's MTU? Or put a MTU so low it would work for any case?

Best regards,

Gary