Question

Hello,

On multiple RUT955 (firmware version: RUT9_R_00.07.02.7) I have frequently (multiple times a day) an issue with discovering the devices that are connected to the LAN (not WAN port) of the router through a gigabit switch (two devices with a static IP are connected to the swtich and one cable goes from the switch to the router).

It can take easily 15 minutes to see my devices after startup, I precise that I can connect to the WiFi of the router and also SSH on it but even when pinging directly from the router I don't see my devices, however, connecting directly to the gigabit switch I can clearly ping my devices but not the router !

I can also add the fact that I updated the firmware but still have the same issue.. Unfortunately I cannot stay on the last firmware version because of the issue with the Modbus (which I am still waiting for a fix btw). I actually have this issue on the latest routers I purchased which are on the new WebUI.. On previous routers and with the exact same switch I never have this issue.

Please advise on what could be wrong as I suspect some new configuration with the LAN Interface that makes my LAN discovery insanely slow from time to time..

Best,

Answer 1

Hello,

Since you have multiple RUT955, I would like to ask, if this behavior is present on all of the devices? 

Next, I would like you to attach a troubleshoot file to your question. Please, replicate the issue, then access router's WebUI, go to System -> Administration -> Troubleshoot section and download troubleshoot file from there. The logs in the file might provide more insight into the issue.

For comparison purposes, a file from the properly operating RUT device would be helpful as well.

For additional troubleshooting, could you download TCPdump from router's WebUI Services -> Package manager -> Packages page and enable TCP dump monitoring in System -> Administration -> Troubleshoot page specifying br-lan (lan) interface? Reboot the device, leave it running for some time until it sees all of the LAN devices. Download capture file from the same Troubleshoot page and attach it by editing your question.

Best regards,

Comments

Hello,

Thanks for the quick answer. This behavior is present only on two new routers.

Once the issue is replicated I will download the troubleshoot file and forward it.

Best,

Answer 2

Hello, 

Please find attached the troubleshooting file on a router where I currently have the issue (firmware version is RUT9_R_00.07.02.7): https://community.teltonika-networks.com/?qa=blob&qa_blobid=1628835914574971351

Also, I could not download the TCPdump (opkg update and opkg install tcpdump) from our router as it is not supposed to be connected to the internet.. I will have to do it next time and on another router as this one won't be available anymore.

I precise that it took 11 min after the router power ON to be able to ping my devices... This router is being installed in an industrial device therefore this delay is not acceptable at all. I hope that the troubleshooting file points to something.. 

Best, 

Comments

The file was generated 2 minutes, after the device boot:

[Uptime]
 11:39:49 up 2 min,

During this time, nothing, related to port events, happen. 

I have also replicated your interfaces configuration, but did not experience any issues, thus configuration can be ruled out.

By the way, switch logs do not show that the router has anything connected to any of its physical ports.

Was anything connected to the router, when the file was generated? Could there be a mismatch between router's and switch's duplex settings?

Have you tried to connect your devices directly to the router? 

What are those devices exactly?

Could you check, if configuring static leases from the router's WebUI Network -> Interfaces -> Static leases page gets them detected sooner?

Best regards, 

Answer 3

Hello,

Yes, once I realized that I had the issue, I directly downloaded the troubleshoot file and it was like you said 2min after the device boot. But, like I said, it took about 11min to be able to ping my devices..

The router has only one cable: one that is going from the gigabit switch to the router itself and yes, everything is always connected (it is inside a robot so there is no easy way to connect/disconnect things from the router or the switch).

"Could there be a mismatch between router's and switch's duplex settings?" Could you please explain this part I am not familiar with those terms.

Have you tried to connect your devices directly to the router? No I did not and I actually cannot: the router is only megabit and I actually need gigabit.

What are those devices exactly? There are exactly three devices on the gigabit switch and each one of them has a static IP: the main PC (192.168.1.102), the secondary PC (192.168.1.210), the router RUT955 (192.168.1.1).

"Could you check, if configuring static leases from the router's WebUI Network -> Interfaces -> Static leases page gets them detected sooner?"

So you are asking me to declare those devices directly right ? I am honestly more than puzzled as this issue is really not present on my other (older) RUT955 routers..

Best,

Comments

The issue itself is quite puzzling. There was a similar case here, where devices would not connect to specific RUT955 models. Though, your router is slightly different, it is possible that the issue is related to hardware.

It could simply be resorted to a replacement of the device.

Otherwise, for further troubleshooting, I would like another troubleshoot file, generated after the router is able to ping connected devices.

• "Could there be a mismatch between router's and switch's duplex settings?" Could you please explain this part I am not familiar with those terms.

Duplex settings define the speed and direction of how data is transferred between the devices. It can be either in one direction at a time or both simultaneously. By default these should be negotiated automatically, though at the moment it looks like devices are not even connected, as there are no indication of errors or packets in between. 

• So you are asking me to declare those devices directly right?

I do, though, I doubt that it would change anything. It is rather a step for additional testing and information about the issue collecting.

A TCP dump file downloaded after the connected devices are ping'able could also provide more insight.

To rule out firmware issues, I assume, your older router has a firmware version of 6.0x.x installed. Could you install the same version on the new RUT?

Best regards, 

Answer 4

Hello,

Thanks again for those fast answers. I went through the similar case and I agree it looks similar... I currently don't know the HW revision but its serial number is 1125835893, another router that also has the same issue is 1125835224. Can you find the HW revision with those ?

The additionnal troubleshooting file where I am able to ping all devices is here: https://community.teltonika-networks.com/?qa=blob&qa_blobid=2451525770890416505

Regarding the "Duplex settings", the gigabit switch is non manageable, I cannot change any settings regarding this. I will try to install TCP dump on the router but it's a bit complicated honestly..

"To rule out firmware issues, I assume, your older router has a firmware version of 6.0x.x installed. Could you install the same version on the new RUT?" -> Yes exactly, as an example I don't have any issue with the router 1120401161 which has the firmware version RUT9XX_R_00.06.08.6 or another router 1106813474 which has the firmware version RUT9XX_R_00.06.08.2... 

Also, on those new routers, is it possible to downgrade to RUT9XX_R_00.06.08.6 version ? I have a feeling that I already tried with no success.. 

I also insist on the fact that this behavior is not repeatable at each reboot of our router.. It can happen once or twice a day after a reboot but it can also be more or less than that..

EDIT: While trying to install tcpdump on the router I had to manage to connect it to an external WiFi access point with internet access. To do that, I go to Network>Wireless and click on SCAN and then join network. While this doesn't work, I don't manage to get an internet access (on fw 6.08 no issue with this either..), I see that I lost all my LAN devices..  In STATUS>LOGS>NETWORK EVENTS I see the following:

Also, while being on SSH on the router i tried the command "dmesg | grep duplex", output is:

[   66.009079] eth0: link up (1000Mbps/Full duplex)

On the router, output of "ethtool eth0.1" is:

Settings for eth0.1:

Supported ports: [ TP MII ]

Supported link modes:   1000baseT/Full 

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: Yes

Supported FEC modes: Not reported

Advertised link modes:  1000baseT/Full 

Advertised pause frame use: Symmetric Receive-only

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Speed: 1000Mb/s

Duplex: Full

Port: MII

PHYAD: 0

Transceiver: external

Auto-negotiation: on

Link detected: yes

On my main pc, output of "ethtool eth1" (eth1 is the interface on which the main PC is connected to the gigabit switc):

Settings for eth1:

Supported ports: [ TP MII ]

Supported link modes:   10baseT/Half 10baseT/Full 

                        100baseT/Half 100baseT/Full 

                        1000baseT/Full 

Supported pause frame use: Symmetric Receive-only

Supports auto-negotiation: Yes

Supported FEC modes: Not reported

Advertised link modes:  10baseT/Half 10baseT/Full 

                        100baseT/Half 100baseT/Full 

                        1000baseT/Full 

Advertised pause frame use: Symmetric Receive-only

Advertised auto-negotiation: Yes

Advertised FEC modes: Not reported

Link partner advertised link modes:  100baseT/Half 100baseT/Full 

Link partner advertised pause frame use: No

Link partner advertised auto-negotiation: Yes

Link partner advertised FEC modes: Not reported

Speed: 100Mb/s

Duplex: Full

Port: MII

PHYAD: 0

Transceiver: internal

Auto-negotiation: on

Cannot get wake-on-lan settings: Operation not permitted

Current message level: 0x00000033 (51)

       drv probe ifdown ifup

Link detected: yes

I hope that those details helps a bit..

Best,

Comments

Thank you.

The hardware revision does differ in your device, but the issue most likely applies to your case as well.

Device event logs got overwritten by mobile logs, however attached screenshot illustrates the situation.

The switch and the router can only negotiate 10 MB link, though it should be 100. Also there are frequent port state changes to up/down with occasional lengthy intervals of disconnection in between, which I assume, where not intentional cable plugging/unplugging 

Considering that the remaining devices can communicate properly, issue can be narrowed down to: 

• Bad cable. 
• Router's switch hardware issue.

I tend to believe it is the second option, thus advice to return the device for a repair or replacement.

Best regards,

Answer 5

Again, thanks for the quick answers.

"which I assume, where not intentional cable plugging/unplugging" you are right, I did not touch the cable and I actually have a laptop directly connected to the gigabit switch that can actually ping all my devices but the router (at least for the first minutes).

I highly doubt a cable issue on two separate robots with their own RUT955 router. 

I am actually trying to downgrade to RUT9XX_R_00.06.08.6 before swapping the router but when uploading the firmware, the webpage is stuck in "VERIFYING"... any idea how to force the downgrade ? Even using command line on ssh is fine for me... 

EDIT: nevermind, third try (refreshing the page) finally made it..!

I will keep you updated.

Answer 6

Hello again,

I just swapped the router with another new one: batch 123; S/N: 6000010635; FW:RUT9_R_00.07.02.7

It has been five minutes that the router is UP and again.. I still don't see my devices that are on my gigabit switch.

Here are the two troubleshooting files

- no device seen : https://community.teltonika-networks.com/?qa=blob&qa_blobid=623545231845265224

- all my LAN devices seen:  https://community.teltonika-networks.com/?qa=blob&qa_blobid=4321092617171482448

Thank you for any advice provided..

Comments

Just for convenience: update to 7.04 and regenerate the files. There have been improvements in the way logs are collected. Currently, with SIM not being inserted, it floods all the log memory, removing relevant messages.

Also, when generating file with all of the devices accessible, if possible, keep the device running for a longer period of time, for example, 30minutes.

Best regards,

---

Hello,

I cannot update to 7.04 because of an issue with the modbus collector output not working anymore.. sorry for that.

---

Could you them regenerate the second file after an extended monitoring period on 7.02.7?

I will take them to the developers to provide their insights.

Best regards,

---

Hello,

I went back to the original router (S/N: 1125835893) I had with the firmware updated on 7.02.7.

Here is the troubleshooting archive after 33 minutes: https://community.teltonika-networks.com/?qa=blob&qa_blobid=13302298591396510576 ; In this troubleshooting file the network connexion time to the devices behind the gigabit switch was rather fine. 

Did you already experience issues when the router was connected to a gigabit switch with all devices connected only to the router ?

I did the following test (not in the troubleshooting), two laptops where each one was connected via ethernet to either the router or the gigabit switch.

First laptop (connected to the router): I manage to ping the router 60s after bootup; I manage to ping the main PC (connected to the gigabit switch) after more than five minutes (it's a bit random.. it can take less than two minutes and up to 12 minutes..)

Second laptop (connected to the gigabit switch): I manage to ping the main PC after 13s but pinging the router takes about five minutes.

Best,