Question

Hello,

I have a weird network bug that does not manifest with 00.07.03.4 but does happen with 00.07.04.2.

Here attached you'll find the device config named "OK" with 00.07.03.4, then "KO" which is just a firmware upgrade of the same config to 00.07.04.2.

Basically the router has a SIM card that provides internet to devices plugged to the LAN port. When I plug my laptop I can surf internet just fine with both configs.

The bug happens with an AXIS cameras plugged to the router, in both cases it also resolves internet just fine (I can resolve HTTP requests when SSH'ing to the camera).

Where it differs is that the camera is unable to create an O3C tunnel.

In axis-log.txt you can see the following:

• At 15:06:35 with firmware 07.03.4 it gets internet after a router reboot and immediatly Axis O3C works ("Command channel successfully established to o3c.arqivis.com:8080").
• At 15:16:27 with firmware 07.04.2 it gets internet after firmware upgrade but Axis O3C never manages to connect. It looks like it initiates the connection properly but something is filtered when the packets are coming back and we get a timeout after 20 seconds.

Here is documentation about Axis O3C: https://cdn.axis.com/o3c_files/o3c-server-technical-reference-2.33.0-rev1.pdf

Kind regards,
Philippe

Answer 1

Hello,

  

In this case it would be interesting to see what kind of communication happens between the camera and the service.

I will ask you to navigate to Services → Package Manager → Packages and install a TCPdump package. 

Then, navigate to System → Administration → Troubleshoot. Enable the TCP dump option, select the interface as br-lan, and leave everything else on default settings. Press Save & Apply. Boot up the camera, and wait for it to not be able to connect. Then, after around 20 seconds, in the same troubleshoot menu press the Download button for the TCP dump file.

This file can be attached to the original post. I will also ask you to download a troubleshoot file from the same menu and attach it to the question as well, as backup files do not contain some necessary information.

I'll be awaiting the information!

  

Best regards,
DaumantasG

Comments

Thanks, here are two captures.

One interesting filter can be: "ip.addr == 10.13.37.20 && ip.dst == 167.235.254.153"

10.13.37.20 is the camera IP and 167.235.254.153 is the O3C server (o3c.arqivis.com) port 8080.

I see in the KO file that the filter shows lots of rejected packets but I have no clue how to interpret it.

---

Hi,

It seems that you have uploaded backup files and not troubleshoot files. Could you please upload the troubleshoot files as they provide more logs?

There are a few odd things in the config. Though I am not sure if those changes are intended or are caused by the firmware upgrade. Did you update the firmware with keep settings by any chance? If so, for testing purposes, could you try updating the firmware with 'keep settings' turned off? Or restore the device to factory defaults.

Also, could you try to execute the following command:

echo "net.netfilter.nf_conntrack_helper=1" >> /etc/sysctl.d/11-nf-conntrack.conf

Kind Regards,

Andzej

---

Hello,

I added the two troubleshoot file

07.03.04-OK-troubleshoot-Teltonika-RUT240.com-2023-05-04.tar.gz
07.04.02-KO-troubleshoot-RUT240-2023-05-04.tar.gz

For both of them all I did is: factory reset, change LAN to 10.13.37.0/24 range, export troubleshoot file.

Then I did modify /etc/sysctl.d/11-nf-conntrack.conf as suggested and rebooted but same issue.

Kind regards,
Philippe

---

Hello,

  

Would it be possible to set a static IP address on the camera?

It's hard to say, but from the camera logs it seems like it's having a hard time obtaining an IP address, so a static address might help.

If it does not, then try pinging the IP address of the camera service from the router to check if it's reachable and how long it takes to respond. Let me know the results.

  

Best regards,
DaumantasG

---

I originally used a static IP address, it's the same behavior.

I opted to test with DHCP thinking maybe it'd help with the automatic routes DNS/whatever provided by the DHCP.

The router can ping the camera just fine with low latency, but I'll try one more time and report.

To be honest I'm not sure why we are looking at the camera side when obviously the same camera works just fine with the older firmware.

Did you find anything in the TCPdump traces?

---

Hello,

i have exactly the same Problem with my RUT 360

from Firmware Versions 07.04.0 up it is not possible to connect to AXIS Cameras anymore!!!

when i downgrade to firmware 07.03.04 it works and i can connect to the Axis Cams - AXIS Companion

here is some BUG inside from Firmware 07.04.0 up in all Versions!!

it is not possible to connect to AXIS Cams with the Software AXIS Companion...

---

Hello,

  

@PhilippeVaucher, it's hard to determine the source of the issue, because, from the logs and the packet trace, it seems like the camera had some difficulties establishing a connection even on v7.3 of RutOS.

However, one difference that I can see, is that packets with MTU size over 1500 are being sent, which can cause all sorts of issues on LTE. To troubleshoot the issue, please navigate to Network → Interfaces → General, edit the mobile interface (mob1s1a1/mob1s2a1), and in the advanced settings, change the Override MTU option to 1200. If the connection establishes, try increasing it to 1300, 1400, 1450.

Let me know if this helps.

  

Best regards,
DaumantasG

---

@DaumantasG: setting MTU to 1450 (or lower) works, but implicitely or explicitely setting it 1500 fails.

Some more findings:

• 1499 fails
• 1475 fails
• 1450 works
• Using a MTU lower than 1450, letting the camera be accepted, setting it back to 1500, then telling the camera to disconnect from O3C, then reconnect, also works. I think this happens because the protocol has probably something like "oh it's you again, you authed recently so let's just skip the authentication procedure". But if you reboot the camera then it fails.

Can you explain why it work with lower-than-1500 MTUs? Did the firmware 00.07.03.4 had a low one?
Also I'm curious why 1450 works but not 1475.

Kind regards,
Philippe

---

Hello,

While this is more of a question for RnD, I believe that the way MTU is being set has changed, and the mechanism requires some more optimization. It has been improved and will be released with the v7.5 update.

For now, I'd recommend leaving the MTU on 1450.

Best regards,
DaumantasG

---

@DaumantasG: how does one ask this question to RnD? I'm a developer so I'm curious and would like to be informed about what bug was found about this and be notified when a newer firmware fixes it.

---

This is probably a fragmentaion / defragmentaion issue either at the kernel or openssl level, or in the network. This sort of issue is frenquent when using wireguard telnet / http to a remote server works but ssh / https fail during establishment. The solution is to decrease the MTU of the wg interface even below the recommended wan-mtu - 80.

Just my two cents.

---

@PhilippeVaucher The exact issue and the solution will be provided in the change log once the new firmware is released (around the beginning of July). Direct contact with the RnD team is not possible, unfortunately. But as mentioned in my previous message, it is caused by incorrectly determined MTU on the mobile connection side.

    

Best regards,
DaumantasG

---

Alright thanks guys, I'll wait for new firmware then.

Kind regards,
Philippe