[Fixed in 1.10.2] Devices stuck in Discover or Backup

[SteveLamb]

We have several instance so Unimus running. it seems that when one instance has > 1000 devices that devices will fail to finish discovery or backup. once they are stuck in this state it is not able to try any further action on this device. restarting the unimus instance or deleting and readding the device seems to resolve the issue.

some of our backups may be very long as we have some switches and routers with significantly large vlan tables.

we are currently running version 1.10.1
below is from the error log when this appears to occur.

Code: Select all

[email protected]    | 2019-04-30 03:07:18.274  WARN 1 --- [  discovery-106] net.unimus.core.api.CoreImpl             : Error during discovery of 10.109.41.5
[email protected]    | 
[email protected]    | java.lang.IllegalStateException: Can't start StopWatch: it's already running
[email protected]    |   at org.springframework.util.StopWatch.start(StopWatch.java:127)
[email protected]    |   at org.springframework.util.StopWatch.start(StopWatch.java:116)
[email protected]    |   at net.unimus.core.util.metrics.JobDurationMetrics.startMeasuring(JobDurationMetrics.java:27)
[email protected]    |   at net.unimus.core.api.CoreImpl$DiscoveryExecutor.doRun(CoreImpl.java:348)
[email protected]    |   at net.unimus.core.api.CoreImpl$ErrorHandlingExecutor.run(CoreImpl.java:302)
[email protected]    |   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[email protected]    |   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[email protected]    |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[email protected]    |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[email protected]    |   at java.lang.Thread.run(Thread.java:745)

[Tomas]

Just an update on this - we are investigating this issue right now.

We have had 1 additional customer report this as well, so it is definitely something on our end.
It seems this issue is not wide-spread however, as we have had no other report other than these 2.

I will post an update as soon as we have any news.

[Tomas]

Update:

This issue should now be solved, and the fix will be available in 1.10.2.

Can you please test with the latest 1.10.2 Beta release and let us know if this fixes it for you?
viewtopic.php?p=2172#p2172

Thanks!

[SteveLamb]

this is working better but not totally fixed. after the first scheduled event we have 3 devices stuck in discovery. running version Version : 1.10.2-Beta2. is there information i can provide that will assist with this.

Thanks

[Tomas]

We are investigating - we have a report from another custom also that this is not yet fully fixed.

Will provide an update ASAP.

[Tomas]

We have found additional rare cases where jobs could get stuck.

Could you please try with the latest Beta build and let us know if you are still seeing issues?
viewtopic.php?p=2172#p2172

Thanks!

[SteveLamb]

good news. on the first run through of 1200 devices i had 0 failed to discover. I will let you know if i see others fail later in the week

this is on beta5

[SteveLamb]

I may have spoken too soon.

we don't have as much of an issue with discovery, after 2 days i am seeing 5 out of ~1200. but 75% of them are sitting stuck in backup.

let me know if i can provide any information that would help with this.

1.10.2-Beta5

[Tomas]

Could we schedule a Webex session to investigate this in detail?
Sent a PM to work out the details.