FOR TIPS, gUIDES & TUTORIALS

subscribe to our Youtube

GO TO YOUTUBE

14455 questions

17168 answers

28195 comments

0 members

We are migrating to our new platform at https://community.teltonika.lt. Moving forward, you can continue discussions on this new platform. This current platform will be temporarily maintained for reference purposes.
0 votes
1,171 views 23 comments
by
Hi,

I have a problem with Modbus communication. This works for a while and suddenly it breaks off as if the TRB140 would go into standby! If I start this new everything works again for a certain time. The firmware is up to date.

1 Answer

0 votes
by anonymous

Hello,

Can you provide a little more information about your issue? Can you draw and provide a simple topology of your system, how everything is connected, via what interfaces, what IP addresses and gateways the devices has? Also, do you run TRB140 as Modbus TCP slave and you are monitoring TRB140 via Modbus, or you have configured TRB140 as Modbus TCP Master and TRB is pulling Modbus Data from slave device and here the issue is? TRB stops pulling Modbus data from slave device and the reboot helps to solve this issue?

Can you please PM me with the troubleshoot file (System -> Administration -> Troubleshoot) when issue persist?

Let me know.

by

Hello,

here is the drawing of the topology.

by
Hello,

how can I send private messages?

I now believe that the problem is not due to Modbus alone. The moment I no longer have a Modbus connection, I can no longer access the device via the browser. This problem is only resolved after the gateway has been restarted. Then the Modbus runs again and I can access it again via the browser.

Greetings Jochen
by anonymous
Hello,

To send private message, click on my avatar and there should be button to send private message.

About you issue, I need to get troubleshoot file and also if possible a steps how I can reproduce this issue on my side.

Please let me know.
by
Hello,

sorry but there is no button to send private messages.

I have now done some tests and can narrow the problem down.

Modbus slave activated + master activated = disconnect after some time

Modbus slave activated + master deactivated = disconnect after some time

Modbus slave deactivated + master activated = runs without problems

So I think the problem is somewhere with the Modbus slave.
by anonymous

Hello,

Attached TEST Firmware release is for this particular issue.

Please check if this specific function works properly now.

Once we receive positive feedback from you – these changes will be included into next Master Firmware release version.

Note: without final confirmation that the issue has been resolved these changes will not be included into Master Firmware version, hence shall not be included from the Factory.

FW: https://we.tl/t-KFVkl0aydt

by
Hello,

thanks for the beta firmware. I uploaded it and will now test it. I will then share my experience with you.
by anonymous
Hello,

Ok, will be waiting for the feedback.
by
Hello,

Unfortunately I have to tell you that the new firmware has not improved my problem. I still have the same problems. As soon as I use the TRB 140 as Modbus slave, I get a connection break after a while. What I also noticed is that the entire TRB freezes. He no longer reacts to SMS or can be reached. Only a restart will fix this problem again. Thanks for your effort and help. I will now leave the slave deactivated and probably query the outputs with relays because I have to install the device so slowly.

Greetings Jochen
by anonymous
Hello,

Can we arrange remote session when the issue persist?

For remote session, make sure:

  - TRB has internet connection
  - PC and TRB has separate internet sources
  - TRB is reachable from PC directly via IP 192.168.2.1
  - PC must have "putty" installed for SSH connection to the router

For remote session we can use "AnyDesk".

Let me know when it is best for you to arrange remote session.
by
any update on this? what was the issue?
by
Hi there

We are also experience the same issue on a TRB141 with FW TRB1_R_00.02.05.2. Was this issue ever resolved? We urgently need to resolve this issue, so if there is no known solution yet, it would be great to at least get a hint on how far you got with the debugging.
by anonymous
Hello,

It should have been solved with the latest available firmware version. To check and confirm it could you please share with me how do you manage to replicate this issue? As I understand TRB141 is reading MODBUS data from MODBUS TCP slave and after sometimes it stops to read data and the only solution is to reboot the device?

Correct?

Or I misunderstood something? Please let me know.
by

Hi!

Thank you for the quick response!

We are actually using the TRB141 as a MODBUS slave. So, something like:

libmodbus (master) <--- 4G ---> TRB141 (slave)

We are periodically reading from the TRB141, which seems to be working fine over time. However, just after a few writes the issue is triggered.

When the issue occurs, I have observed the following:

  • Modbus reads are no longer working
  • The WEB-UI is unavailable with a proxy error
  • SSH and OpenVPN is still working
  • I tried to manually restart the webserver from SSH, which gave a ubus communication error
  • After killing the ubusd (and a new ubusd was automatically started) and restarting the rpc daemon I was able to properly restart the webserver, and the WEB-UI is back.
  • Even after doing the same with the modbusd process, reading over MODBUS is not working
  • Reboot command (both from WEB-UI and SSH) is not working at this stage.
So far I haven't been able to test to manually trigger the issue (it has been used in production environment, so the operator has quickly power-cycled it when the issue occurs), but now we have a few days timeslot to lab on it.
by anonymous
Thank you for provided information.

Just to confirm, you are reading TRB141 parameters (IP, hostname and etc) via MODBUS protocol remotely via internet? Correct?

TRB141 has public IP address, or you are reaching it via VPN?

As far as I understand, you are reading TRB141 MODBUS data (for monitoring) remotely and after some time you are not able to read MODBUS data from TRB141, correct?
by
Yes, correct!

We are reading the status of some of the digital I/Os periodically over MODBUS, which works fine. Irregularly we change the output value of one of the digital I/Os by writing over MODBUS. After just a few of those writes the issue is triggered.

We connect to the the TRB141 over a routed OpenVPN connection.
by

Hi again

I now had the chance to test things on the device (although still remote, so I'm a bit limited in what I can do).

I have to take back some of my claims, I apologize for that!
It looks like you have to do a lot of writes before the issue occurs. And it looks like you have the same issue with readings as well. Every time the issue has occurred, it has been soon after a write, hence my assumption that the write was the bad guy. Maybe the timing was just a coincidence.

Anyhow, it looks both read and writes leaks memory. When the modbusd-process hits ~65MB of memory utilization, hell breaks loose.

Here is dump from /proc before and after doing 1200 writes:

root@Teltonika-TRB141:~# cat /proc/3336/status
Name:   modbusd
State:  S (sleeping)
Tgid:   3336
Ngid:   0
Pid:    3336
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 256
Groups:
VmPeak:     4332 kB
VmSize:     4332 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:       828 kB
VmRSS:       828 kB
VmData:     1712 kB
VmStk:       136 kB
VmExe:        24 kB
VmLib:      2332 kB
VmPTE:         8 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/1178
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000000014002
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
voluntary_ctxt_switches:        260
nonvoluntary_ctxt_switches:     9

root@Teltonika-TRB141:~# cat /proc/3336/status
Name:   modbusd
State:  S (sleeping)
Tgid:   3336
Ngid:   0
Pid:    3336
PPid:   1
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 1024
Groups:
VmPeak:    67044 kB
VmSize:    67044 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:      8704 kB
VmRSS:      8704 kB
VmData:    64424 kB
VmStk:       136 kB
VmExe:        24 kB
VmLib:      2332 kB
VmPTE:        70 kB
VmSwap:        0 kB
Threads:        1
SigQ:   0/1178
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000000014002
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
Cpus_allowed:   1
Cpus_allowed_list:      0
voluntary_ctxt_switches:        4980
nonvoluntary_ctxt_switches:     237
by

An additional reflection:
Looks like the number of open file descriptors to /var/run/ubus.sock increases with every MODBUS connection.

by anonymous

Hello,

Thank you for the information. I try to replicate the issue on my side but no luck, every time I was able to read and write MODBUS register to TRB141 via OpenVPN.

Could you please:

- Try to replicate the issue once more;

- After issue persist, download troubleshoot file (System -> Administration -> Troubleshoot)

And PM me that file so I could check what could be the potential issue. Also, can you try replicating this issue locally without use of OpenVPN?

by

Hi!

Did you observe the memory consumption and number of open file descriptors after each read/write, and it didn't increase?

I have another timeslot tonight when I will reproduce it. Unfortunately the device is still in use in a pump-house 350km away, so no local access at the moment. I will try to get hold of another device to test with.

I'm not sure I can get the troubleshoot file though, since the UI is half-broken even after killing and restarting some services, but I will try.

For reference, here is the small C application I have been using to reproduce it:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <modbus/modbus.h>

int main(int argc, char *argv[]) {
  modbus_t *mb;
  int reg_addr = 0;
  uint16_t reg_val = 0;
  int reg_count = 0;

  if (argc != 4) {
    printf("Usage: %s <hostname> <register> <read_count>\n", argv[0]);
    return -1;
  }

  reg_addr = atoi(argv[2]);
  reg_count = atoi(argv[3]);

  int i;
  for (i=0; i<reg_count; i++) {
      mb = modbus_new_tcp(argv[1], 502);
      modbus_set_slave(mb, 1);
      modbus_connect(mb);
      modbus_read_registers(mb, reg_addr, 1, &reg_val);
      modbus_close(mb);
      modbus_free(mb);
      sleep(1);
  }

  return 0;
}
by anonymous
Thank you for the information, I will try consulting with RnD to see what we cansuggest as a solution.
by anonymous

Attached TEST Firmware release is for this particular issue.
Please check if this specific function works properly now.
Since this is a TEST version of the Firmware, designed to address specific issues, use it only with a handful of devices; you should not use it for your whole fleet.
Once we receive positive feedback from you – these changes will be included into next Master Firmware release version.
Note: without final confirmation that the issue has been resolved these changes will not be included into Master Firmware version, hence shall not be included from the Factory.

by anonymous
Thank you for the quick fix!

The TEST firmware works well as far as I can tell. I tested with both read and writes without any increase in neither number of file descriptors or memory usage. Good work!

When is the next Master Firmware planned to be released?
by anonymous

Hello,

Thank you for the feedback, the master firmware is still undergoing development and testing phase so it is hard to tell when exactly it will be released. I recommend keeping an eye on here: https://wiki.teltonika-networks.com/view/TRB141_Firmware_Downloads

For the latest available firmware version.