[Solved] Unimus is 'chatty' in the I/O

Unimus support forum
Post Reply
JAz
Posts: 43
Joined: Thu Apr 26, 2018 11:06 pm

Tue May 29, 2018 10:51 pm

Hey now,

Have a small test VM running (Deb9 on Hyper-V 2012) on an iSCSI storage target with 5 routers in the backup rotation. There is one scheduled backup per day, around 3am and these routers do not change often. They are using HSQL backend.
When watching these iSCSI targets (looking at utilization and iops) often times this vm spikes higher than any of the other servers using this target (300-400 iops, utilization spiking or pinning at or near 100%), including Windows servers running things I'd expect to use much more I/O (Spiceworks, RMM)

To verify I set up a second VM (same config) with just ONE router. It also is very chatty, sometimes exceeding the Unimus VM with 5 routers.

At various times the VMs' webi become unresponisve to login. Not sure what these machines are doing and especially what all this I/O is about when they are supposedly not doing backups or anything I can discern. These are clean, default, fairly minimal Deb 9 installs.

Anyone else seeing high I/O or any thoughts on what goes on?
Is there a documented procedure to convert hsql to rdbms?

thx.
Last edited by JAz on Wed Jun 13, 2018 10:18 am, edited 1 time in total.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Tue May 29, 2018 11:09 pm

I dont think the problem here will be HSQL.

How much memory do these VMs have assigned?

By default Unimus is configured to use up to 768MB of RAM.
If the machine has less, it will claim all the RAM, and the VM will swap.
This would create the high IO you see.
JAz
Posts: 43
Joined: Thu Apr 26, 2018 11:06 pm

Tue May 29, 2018 11:22 pm

Tomas wrote:
Tue May 29, 2018 11:09 pm
I dont think the problem here will be HSQL.

How much memory do these VMs have assigned?

By default Unimus is configured to use up to 768MB of RAM.
If the machine has less, it will claim all the RAM, and the VM will swap.
This would create the high IO you see.
Ok Tomas. In fact these VMs have a bit less so swapping may be the answer. This HV is nearly max'd on RAM but we are adding a few sticks in the next week or so. Will report back after.

Thansk for fast reply. 8-)
JAz
Posts: 43
Joined: Thu Apr 26, 2018 11:06 pm

Wed May 30, 2018 12:31 am

Thinking about thsi a bit more - while you may very well be right about the ram/swap, I wonder what is Unimus doing during this time when it should be essentially idle, that is requiring any memory change?

I mean if no one is accessing the web ui and there are no scheduled jobs, then what is it swapping/doing? It should just be sitting quietly no?
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Wed May 30, 2018 10:01 am

In a case where the system starts to swap, it's not just Unimus that is casing such IO.
When RAM runs out, pretty much ALL Linux services use the swap for all their scheduled jobs / tasks.

Basically, you are seeing the IO that the entire Linux system needs from the RAM when idle (using the word "idle" very loosely here, since modern OSes are never really idle, there is that much going on).
This is because when the system is out of RAM (due to Unimus filling it up due to insufficient RAM allocation for the default config) every service / scheduled job in the Linux itself will write to swap.

You can also change the RAM allocation for Unimus:
edit "/etc/default/unimus"

More info here:
https://wiki.unimus.net/display/UNPUB/C ... mory+usage

However please note that setting the allocation too low can cause Unimus to crash with "out of memory" errors.
JAz
Posts: 43
Joined: Thu Apr 26, 2018 11:06 pm

Wed May 30, 2018 6:06 pm

Tomas wrote:
Wed May 30, 2018 10:01 am
In a case where the system starts to swap, it's not just Unimus that is casing such IO.
When RAM runs out, pretty much ALL Linux services use the swap for all their scheduled jobs / tasks.

Basically, you are seeing the IO that the entire Linux system needs from the RAM when idle (using the word "idle" very loosely here, since modern OSes are never really idle, there is that much going on).
This is because when the system is out of RAM (due to Unimus filling it up due to insufficient RAM allocation for the default config) every service / scheduled job in the Linux itself will write to swap.

You can also change the RAM allocation for Unimus:
edit "/etc/default/unimus"

More info here:
https://wiki.unimus.net/display/UNPUB/C ... mory+usage

However please note that setting the allocation too low can cause Unimus to crash with "out of memory" errors.
Thanks Tomas. I realize the OS is more than just the app(s) running on it but I have, for example, debian vms running with as little as 384mb (unifi controller is one) that don't exhibit this same i/o spikiness. Granted it is a deb8 but deb9 reqs don't look that different without a desktop.
In each case they are single-purpose - the deb8 has only unifi (which obv includes MongoDB - and not sure but also Java?) and Unimus has Unimus and Java (and no DB since it's HSQL)

The unimus machine started out as 384 and deb installed w/o complaint. I've upped it to 512 and still see the spikes. I wouldn't think deb9 in a base minimal should need that much ram w/ so little load on it. Maybe it's Java. Just trying to narrow down if it's a problem or is what it is.
My Windows Server 2012 VMs, one running both Spiceworks (inventory scanning) and Desktop Central (RMM), show less I/O spikes (typical is a bit higher but I think that's totally to be expected.)

Anyway, not trying to make mountains out of molehills just struck me as odd and maybe indicative of a [bigger] problem somewhere? We will be feeding a few more sticks into the hypervisor soon and up the unimus to 768 or 1024 and see if persists.

Thanks for quick answers. :)
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Wed May 30, 2018 8:23 pm

Please let me know how it ends up when there is enough memory on the VM.

If you keep the default configuration (which is 768MB for Unimus), I advise having 1GB of RAM on the VM.
(so when Unimus fills up its allowance, there is still something left for the system)

If the disk IO spikes are still present after, we will be happy to look at it :)
JAz
Posts: 43
Joined: Thu Apr 26, 2018 11:06 pm

Wed Jun 13, 2018 10:17 am

Tomas wrote:
Wed May 30, 2018 8:23 pm
Please let me know how it ends up when there is enough memory on the VM.

If you keep the default configuration (which is 768MB for Unimus), I advise having 1GB of RAM on the VM.
(so when Unimus fills up its allowance, there is still something left for the system)

If the disk IO spikes are still present after, we will be happy to look at it :)
Confirming that after upping the RAM on VM to 1GB that I/O spikes have calmed.
Also, rereading I see where you say that you have config'd Unimus to expect 768 for itself. I overlooked this initially and for our small testing env. I could easily have lowered that. Not sure how many devices that's meant to target but we've run (for example) Unifi server with 30 or 40 devices in 512MB or less.
Anyway, just filling in info for the next guy.
User avatar
Tomas
Posts: 1206
Joined: Sat Jun 25, 2016 12:33 pm

Wed Jun 13, 2018 10:39 am

JAz wrote:
Wed Jun 13, 2018 10:17 am
Not sure how many devices that's meant to target but we've run (for example) Unifi server with 30 or 40 devices in 512MB or less.
With the default config, you can do 200-300 devices using HSQL, or up to 1000 using an external database.

For a small test deploy (5-10 devices), even 256MB of RAM should be sufficient.
For around 100 devices (especially when using HSQL) 512MB of RAM will be appropriate.

Properly configuring that in "/etc/default/unimus" will of course be neccessary. As you have saw in your config, if you just assign that ammount of RAM to the VM and do not configure it properly, it will cause RAM depletion and swapping.
Post Reply