10 meters is only general guideline, but 3dB is already a loss of half the power the router transmits. However, comparing the cables, 20 meters of LMR 400 does have lower attenuation than 10 meters of LMR 200 at 900MHz and 20m length of LMR 400 performs even better than LMR 200 at 1800/1900MHz frequencies. It is more expensive, but calculations show that is should work. If possible, I would suggest to try to reduce the distance between the router and the antenna as much as possible. Another way to improve the connection quality is to use a high gain antenna. Usually, directional antennas have better are better at providing higher gain but might not be suitable for moving objects such as a boat.
It would be really interesting if you could comment back with the results from your testing with the specific cable.