Question

Dear all,

We have deployed 40 RUT240 units in the field using RUT2_R-00.07.03 firmware.

After an initial successful rollout, we noticed that we stopped receiving telemetry from our microcontrollers, connected to RUT240 with Wi-Fi.

Curiously, most transmission stopped shortly before midnight of a Monday.

This matches the reboot rule we set up in the Reboot schedule, where it is set up to reboot the router every Monday at midnight.

When we enquired those RUT240 with SMS commands, the units replied with messages like "Router name - RUT240; WAN IP - 10.5.45.211; Data Connection state - Connected; Connection type - LTE; Signal Strength - -46; New FW available - No;"

The data connection reports is connected, but no data is exchanged and RMS is offline.

We tried and succesfully replicated the issue with a test unit.

Unfortunately, powering off and on the router DOES NOT restablish internet connection. The router connectivity appears to be permanently inhibited.

The 4G connection is indicated as "Connected" however no data is exchanged. The internal ping utility says "Operation not permitted". Teltonika RMS reports a "Failure (other error)" connection state.

We have captured several screenshots. We can provide privately the configuration backup file we applied to all RUT240 units, and troubleshooting file of the test unit not working.

We applied the same configuration backup file to the test unit, and the connectivity is restored. We also downloaded a troubleshooting file of this unit in working state so you can compare and find any differences.

This appears to be a severe firmware bug, as it leads to loss of connectivity and loss of control through RMS.

As the units in the field are 1000 km away from our location, we kindly ask to pinpoint the root cause, in order to have a working solution to restore connectivity.

With regards

Niccolo Gallarati

Answer 1

Hello,

There are some know WiFi issues in the v7.3 firmware. I will ask you to update a few units to RUT2_R_00.07.04.2 and check if the behavior reappears.

Let me know how it goes!

Best regards,
DaumantasG

Comments

Hi,

So far I was able to remotely factory reset units via SMS, from there connectivity is re-established as well as RMS connection.

Then I'm able to access WebUI and set up required parameters.

This saves a 1000 km trip, the humble SMS message proved to be useful.

I'm avoiding restoring configuration through config file and will keep auto reboot disabled.

I will reset 10 units and apply 7.04.2 firmware on 5 units to see what is going on.

7.04.2 firmware is brand new and, while it solves many issues, it could potentially introduce new ones.

In the meantime I would kindly ask if your team can have a look at the logs as all units stopped working properly after the weekly auto reboot, and whether this bug has been solved by new firmware.

With regards

Nick

---

Hello,

  

The logs do indicate, that the ping reboot is causing constant restarts of the modem.

I have also noticed, that you are using an APN, which is not standard for TIM Italy, perhaps this is a private network without an access to the public internet?

If these SIM do have access to the public internet, it seems like the ISP provides DNS servers in the private IP range, which could be causing some issues. Try changing the DNS servers on the WAN and LAN zones manually to 1.1.1.1 + 8.8.8.8 in the Network → Interfaces → General nemu.

Let me know how it goes!

  

Best regards,
DaumantasG

---

Hi Daumantas,

Thank you for checking the logs.

Yes, I noticed that the ping reboot caused constant restarts of the modem as expected.

What is not expected is that there is an underlying issue that prevents Internet connectivity, therefore ping reboot is triggered as a consequence.

Yes, I'm using TIM M2M SIM card for this particular test, they do have access to the public internet.

When the router was having this connectivity issue, I tried to ping Google DNS address on 8.8.8.8 by using the diagnostic tool in the Administration menu. This shouldn't be affected by DNS issues. However, ping failed, moreover with a strange error "operation not permitted".

It looks like there is some WAN or firewall issue, which was triggered by the weekly auto reboot.

I haven't changed any WAN or firewall setting.

In fact, by simply restoring the router to the configuration file I sent you, the connectivity starts working again. Same SIM card, same APN and same DNS setting. But then, on Monday night it will stop working and powering off and on the unit won't restore connectivity.

The bricked routers deployed in the field are using Things Mobile SIM cards, they do not require a special APN, thus this rule out a network operator issue.

Hope this helps pinpointing the issue.

With regards,

Nick

---

Since as I understand the troubleshoot files were taken from the test units, could you disable the ping reboots, leave it for a few hours and generate another troubleshoot file? Currently, the ping reboot logs are spamming the entire system logs, so it's quite hard to see what is actually happening.

  

Best regards,
DaumantasG

---

Okay, I will try to reproduce the issue again on the test unit with the ping reboot rule disabled, and will report back to you

With regards,

Nick

---

Hi Daumantas,

the test unit without the ping reboot rule enabled was rebooted at midnight of May 1st, and is transmitting data without issues.

Therefore, one possible explanation could be a race condition of the ping reboot rule upon restart.

However the ping reboot rule was set up with plenty of time: Interval 5 mins, Interval count 3, Timeout 10 secs. So it has to fail 3 consecutive pings in 15 minutes in order to trigger the rule. The connection is usually established 2/3 minutes after reboot.

Moreover, additional 15 units which were stuck for 2/3 weeks started sending data from midnight, May 1st, when the weekly reboot rule has been triggered, despite having also the ping reboot rule enabled. So the race condition not always occur.

I will now disable both rules as they caused more harm than good; I can always reboot a router with SMS.

I will compare stability between 7.03 and 7.04.2 firmwares and let you know.

With regards

Nick

---

Hi Daumantas,

Here is an update on my fleet of 40 RUT240 devices.

Of 40 units, I was able to upgrade to 7.04.02 to 8 units. These units are not experiencing any connectivity issues.

Of the remaining 32 units, I was able to perform a remote factory reset on 31 units on May 2nd.

18 units stopped transmitting since then.

Please notice that after factory reset I haven't used any configuration backup, nor auto reboot rules. Just configured Wi-Fi network and set up network operator lock.

I conclude that FW 7.03 is fatally flawed and should be recalled.

I'm spending dozens of hours trying to rectifying the issue remotely as the units are 1000 km away and as such I'm rather disappointed.

I'm trying to factory reset unresponsive units and update them to latest firmware, however the process is unreliable.

I opened another ticket on this specific issue

https://community.teltonika-networks.com/65427/rut240-unresponsive-while-uploading-7-04-firmware

Please let me know if you want to try to update some of my units remotely, I can provide all the necessary details.

With regards

Nick

---

Hi,

I just found that the test unit I left powered on stopped sending data on Monday at midnight, after the weekly reboot.

Please have a look at "troubleshoot-Teltonika-RUT240.com-2023-05-17.tar.gz" file I uploaded.

This time ping auto reboot rule was disabled, hopefully it should be clearer to determine what causes the loss of connectivity with firmware 7.03.

With regards,

Nick

---

Hello,

  

Indeed, I can see that my colleague is helping you out with the firmware update.

Now regarding the mobile data issue, from the logs, the disconnection is caused either by the operator or by miscommunication between the carrier and the RUT240. In this case, it would be best to update all of the devices as soon as possible. However, this issue has not been noticed on any other operator, so it's hard to say what could be causing this sort of behavior. The firmware refusing to update may also be caused by something going wrong when downloading the firmware.

On RMS, you can schedule the update on all devices, and even if they are offline, they will be updated as soon as they come online.

As for the units that do not respond after a factory reset, have you tried sending an SMS message to check their status? If yes, what was it? Perhaps the APN has to be set?

  

Best regards,
DaumantasG

---

Hi Daumantas,

This morning I'm having a low success rate, and was able to update just 1 unit out of 10.

I'm initiating uploads from RMS as the WebUI doesn't appear to be more reliable, and it takes more steps.

I'm also racking up a sizable bill as the SIM cards have a 0.10€/MB plan and each failed update costs 1 euro.

When an update is initiated and fails, the unit become offline and will stay in this way unless it is rebooted or restored via SMS.

Do you confirm that firmware update disable some services and thus require a reboot in case it is not completed?

Perhaps is there a way to download the update and apply it from CLI?

Therefore if download fails I could quickly retry it, providing RMS connection is not lost.

Regarding your question "As for the units that do not respond after a factory reset, have you tried sending an SMS message to check their status? If yes, what was it? Perhaps the APN has to be set?", all units become online after a factory reset, fortunately Auto APN works with these SIM cards.

Nick

---

Hello,

  

It is possible to update the device via CLI. However, you will first need to upload the .bin file to the router itself.

This can be done by connecting to it using RMS Remote Access, by choosing SFTP as the protocol to use. Upload the .bin file into the /tmp (RAM) folder on the router.

Then login to the CLI of the router, and run the command sysupgrade /tmp/RUT2_R_00.07.04.2_WEBUI.bin

This will start the update procedure. To keep all of the configuration files, please add -c argument to the command.

Generally, the update procedure should kill the network connection only once the router has to reboot. I'm not sure why the devices might not connect. 

Let me know if this procedure works any better.

  

Best regards,
DaumantasG