[Solved] Terminal codes & format deltas showing in backup

Unimus support forum
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Thu Mar 09, 2017 2:34 pm

Since upgrading, I've logged several instances where terminal codes are showing up in the backup:
unimus3.png
unimus3.png (6.61 KiB) Viewed 10729 times
unimus4.png
unimus4.png (5.8 KiB) Viewed 10729 times
So far, I've found these artifacts in HP Procurve switch backups. I didn't notice this happening under 0.2.x.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Thu Mar 09, 2017 3:50 pm

I was not able to reproduce this on the ProCurves we have here.

Is it doing it on all ProCurves you have, or only on specific model / FW version?
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Thu Mar 09, 2017 4:19 pm

It appears to be happening primarily on generation 2 & 3 ZL series switches; my gen 2 is running K16 (latest major) and gen 3 running K15 code. It seems to be random from day to day; however, the error has an affinity for switches with >1000 line configurations and those with long line (screen wrapped) output.

I'm stepping through other gear...

Another thought:
The regex --MORE-- match in this log:

Code: Select all

2017-03-03 15:46:49.436  WARN 7604 --- [      backup-45] n.u.c.drivers.hp.HpProcurveBackupDriver  : Unable to download backup - 'Expect operation fails (timeout: 20000 ms) for matcher: anyOf(regexp('\QMySwitchName\E# $'),regexp('-- MORE --'))'!
The Procurve command:

Code: Select all

no page
will prevent more "press-any-key" style interaction. This has helped me in the past when using plink/Rancid/powershell scripts to strip extraneous terminal codes (e.g. --MORE--) or anything that would normally require user interaction for continuous "show running-config" output.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Thu Mar 09, 2017 6:04 pm

VT100 sequences
For VT100 control sequences, we have a global filter that filters them before data from the SSH session even make it into any code that interacts with SSH. The filter implementation has not been changed since 0.1.3 when it was implemented.

So it is very strange that you are seeing issues after migration from 0.2 to 0.3.

Are the issues consistent across backups on a single device?
So if the control sequence is present on a device in one backup - is it also in another?
Or is it totally random, and they appear arbitrarily across devices and backups?

'- more -' and pagination
As for pagination and '- more -', we use pagination on purpose.
It is very handy as a synchronization point - the code actually expects it, because it signifies that device is outputting as expected.
(since its a well know sequence, which can be depended upon to appear every now and then)

Unimus actually uses a quite large terminal size (which means pagination doesnt appear very often) - but it it still very useful :).
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Thu Mar 09, 2017 9:26 pm

Are the issues consistent across backups on a single device?

No. The issues appear on multiple devices; however, the devices I'm seeing this happen on are HPE Procurves running two versions of code. I count the following gear:model:code-(date) with this manifestation:
E1:J9850A:KB.15.18.0006-(Mar6-7&7-8&8-9)
DCSw1:J8698A:K.16.02.0010-(Mar8-9)

So if the control sequence is present on a device in one backup - is it also in another?
Sometimes they appear in different places of the config - from day to day.

Or is it totally random, and they appear arbitrarily across devices and backups?
Sometimes in the middle of the config and the very end. It appears to be random areas toward the half-way point to the end of the config.

-----

I came across another issue while stepping through my backups (I wish there was a search function!). I have an HP j9726a that is being discovered correctly, but not being backed up. It's similar to the ASA issue described in a previous post.

Code: Select all

2017-03-09 14:16:15.038  INFO 7604 --- [nio-8085-exec-1] net.unimus.service.DeviceService         : Attempt to run 1 backup(s)
2017-03-09 14:16:15.051  INFO 7604 --- [nio-8085-exec-1] net.unimus.business.core.Core            : Performing backup on 1 device(s)
2017-03-09 14:16:37.929  WARN 7604 --- [      backup-29] n.u.c.drivers.hp.HpProcurveBackupDriver  : Unable to download backup - 'Expect operation fails (timeout: 20000 ms) for matcher: anyOf(regexp('\QSw6\E# $'),regexp('-- MORE --'))'!

2017-03-09 14:17:05.824  WARN 7604 --- [       backup-5] n.u.c.drivers.hp.HpProcurveBackupDriver  : Unable to download backup - 'Expect operation fails (timeout: 20000 ms) for matcher: anyOf(regexp('\QSw6\E# $'),regexp('-- MORE --'))'!
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Fri Mar 10, 2017 11:09 pm

I had more on last night's backup run:
unimus7.png
unimus7.png (7.68 KiB) Viewed 10700 times
unimus6.png
unimus6.png (3.65 KiB) Viewed 10700 times
unimus5.png
unimus5.png (15.47 KiB) Viewed 10700 times
This time it occurred on a once unaffected Procurve switch in addition to the previous two. Same ZL/YL series switches as before.
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Fri Mar 10, 2017 11:12 pm

I forgot to mention:
My previous Unimus 0.2.1 was running as a jar.
My current instance (with these issues) is running the Windows Portable release.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Tue Mar 14, 2017 4:05 pm

I have prepared a dev-build of 0.3.2 with full debugging enabled.
This will log A LOT of data into the log file (including the full device communication).
It can be downloaded here.

1) Can you please send me logs from the communication with the j9726a thats not being backed up correctly?
(please strip any sensitive data from the logs)

2) What search function you mentioned was missing?
We will gladly add it :)

3) This request is a more complicated - if you could catch one backup that includes the VT100 control characters while running the debug version and send me the logs, it would help greatly.

Feel free to PM me the logs, so they are not publicly available.
Itcoop
Posts: 32
Joined: Tue Aug 02, 2016 12:39 am

Tue Mar 14, 2017 7:53 pm

Thanks Thomas,

Unfortunately, I still cannot access your owncloud. I'm presented with this error:
unimus8.png
unimus8.png (11.31 KiB) Viewed 10676 times
1) 2017-03-14 13:50:38.122 INFO 7604 --- [nio-8085-exec-9] net.unimus.service.DeviceService : Attempt to run 1 backup(s)
2017-03-14 13:50:38.134 INFO 7604 --- [nio-8085-exec-9] net.unimus.business.core.Core : Performing backup on 1 device(s)
2017-03-14 13:51:00.969 WARN 7604 --- [ backup-29] n.u.c.drivers.hp.HpProcurveBackupDriver : Unable to download backup - 'Expect operation fails (timeout: 20000 ms) for matcher: anyOf(regexp('\QMySwitchNameHere\E# $'),regexp('-- MORE --'))'!

2) For a search function: Backups --> Search allows me to search my list of _devices_. Should I search the content of a backup selection for the word '[100;1H' (e.g. an escape code referenced this trouble), I would not have to step through every date iteration of backup-per-device. It would be a quick, google-esque, search.

3) Will do. Once I can download the debug build, I will run it in a sandbox and provide you with the logs via PM.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Wed Mar 15, 2017 12:11 am

I created a ticket for 2:
https://tracker.unimus.net/browse/UN-156

For 1 and 3, here is the build on Google Drive:
https://drive.google.com/open?id=0B6GBO ... jlXZFd5MzQ

Please let me know if you can get it and run it, and send me the logs for 1 and 3 from the debug build if so.

Thanks in advance!
Post Reply