bloxHub

www.infoblox.com/community
20 posts / 0 new
Log in or register to post comments
topic-opened
Version 3 is SOOO slow
#1 Wed, 02/04/2009 - 06:32
Version 3 is SOOO slow

Anything to look at to try and speed up the presentation of data?

 I upgraded our system yesterday and was hoping that over night the database would settle down and indexes would be done doing their thing and performance would return. This is unbearable.

 Also, is there anyway to save the performance graphs now? When researching performance problems in the past, we would save the charts and send them off to our server team to show trending of interfaces.

 NetMRI> sh stats  
+------------+------------+------------+-------------+
| Day        | CpuAvgBusy | FreeMemAvg | DiskUsage   |
+------------+------------+------------+-------------+
| 2009-02-03 |    65.3710 |       NULL | 23.46303617 |
+------------+------------+------------+-------------+

 NetMRI> sh io
Linux 2.4.20-8smp (NetMRI)      02/04/09

Time: 16:29:56 
avg-cpu:  %user   %nice    %sys   %idle
          34.44   23.55    5.18   36.83

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0            8.09        22.26       622.18    6568006  183602956
dev8-1           45.48        52.31       546.79   15435850  161355632

Time: 16:30:01 
avg-cpu:  %user   %nice    %sys   %idle
          64.40   14.60    6.20   14.80

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           25.00         0.00       593.60          0       2968
dev8-1           23.80         0.00       240.00          0       1200

NetMRI> sh load
 16:30:35  up 3 days,  9:58,  1 user,  load average: 5.98, 6.39, 8.07

sh processes shows that java is my largest cpu hog along with MySQL.

root     23906 33.0 11.4 963064 412020 ?     S    16:52:46 07:48:59 schedu /tools/jdk/bin/java -verbose:gc -Dtomcat.sessionid.randomclass=java.util.Random -XX:+UseConcMarkSweepGC -Xms32m -Xmx650m -Dto
mcat.home=/tools/skipjack org.apache.tomcat.startup.Tomcat

+1
0
-1
Tags
Automation Change Manager
Top
Re: Version 3 is SOOO slow
Wed, 02/04/2009 - 06:35
Re: Version 3 is SOOO slow

 There appears to be at least one issue from looking at your output ( incorrect kernel). A Netcordia support engineer will contact you shortly

 

+1
0
-1
Top
Re: Version 3 is SOOO slow
Thu, 02/05/2009 - 12:22
Re: Version 3 is SOOO slow

[quote user="campbech"]

 Also, is there anyway to save the performance graphs now? When researching performance problems in the past, we would save the charts and send them off to our server team to show trending of interfaces.

[/quote] 

 I was also wondering what happened to the ability to save interface peformance graphs. This was very useful in past.

 

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #3) Thu, 02/05/2009 - 22:47
Re: Version 3 is SOOO slow

 Therecurrently is no way within NetMRI to save the graphs, you will have to use the "Print Screen" function of your PC or use a 3rd party screen capture utility. 

 

We will open a feature request asking for this in a future release

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #3) Fri, 02/06/2009 - 01:52
Re: Version 3 is SOOO slow

 A feature as been submitted for this requirement and will be considered for our next Major release which is due out in the middle part of 2009

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #5) Thu, 02/12/2009 - 05:03
Re: Version 3 is SOOO slow

There will be a patch provided shortly that allows for the ability to save the charts

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #3) Mon, 03/02/2009 - 06:21
Re: Version 3 is SOOO slow

 We've uploaded a patch in the downloads > Software > 3.0.1.94 Patch section to allow the exporting of charts. This is also availible via autoupdate if the netmri has public access.  Simply type au at the admin shell.

 After applying, right-click on any image and select the export option.

+1
0
-1
Top
Re: Version 3 is SOOO slow
Thu, 02/19/2009 - 04:02
Re: Version 3 is SOOO slow

I know that Netcordia has been working hard at finding out why our performance has been so bad, but the stats actually look worse now.

See the results above and the results now. The CpuAvgBusy has increased, and every other stat (including Blk_wrtn/s) has doubled or even more in some cases. Any more ideas? 

NetMRI> sh stats
+------------+------------+------------+-------------+
| Day        | CpuAvgBusy | FreeMemAvg | DiskUsage   |
+------------+------------+------------+-------------+
| 2009-02-18 |    82.9574 |       NULL | 24.95841146 |
+------------+------------+------------+-------------+
NetMRI> sh io
Linux 2.4.31-smp (NetMRI)       02/19/09

Time: 13:58:38 
avg-cpu:  %user   %nice    %sys   %idle
          37.38   31.77   10.71   20.15

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           25.38         5.59      1893.65    2081174  704449486
dev8-1           64.57        48.72       765.32   18124882  284705272

Time: 13:58:43 
avg-cpu:  %user   %nice    %sys   %idle
          36.00   51.90   12.00    0.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           23.60         0.00      3785.60          0      18928
dev8-1           14.60         0.00       163.20          0        816

NetMRI>

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #7) Mon, 03/02/2009 - 01:33
Re: Version 3 is SOOO slow

Our unit is also slow, figured another set of data might be of some help:

netmriu> show stats
+------------+------------+------------+-------------+
| Day        | CpuAvgBusy | FreeMemAvg | DiskUsage   |
+------------+------------+------------+-------------+
| 2009-03-01 |    99.6087 |       NULL | 31.76115811 |
+------------+------------+------------+-------------+
netmriu> show io
Linux 2.4.31-smp (netmri)     03/02/09

Time: 11:28:47 
avg-cpu:  %user   %nice    %sys   %idle
          29.63   56.24   13.62    0.52

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           45.18       187.36      2086.61   19090662  212606450
dev8-1          154.15       781.71      1349.64   79648868  137516394

Time: 11:28:52 
avg-cpu:  %user   %nice    %sys   %idle
           8.70   78.40   12.70    0.20

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0            1.60         0.00        83.20          0        416
dev8-1          556.00       249.60      4955.20       1248      24776
 

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #7) Mon, 03/02/2009 - 08:16
Re: Version 3 is SOOO slow

Here's a comparison of v2.4 to v3.0 running in parallel on the same large enterprise network:

    v2.4 CpuAvgBusy = 87.8%, v3.0 CpuAvgBusy = 99.5%

And the device details -> performance for both NetMRIs has daily and 7-day plots that match those snapshots.

The other thing I've noticed is that v3.0 really beats up on the client browser.  I run Sysinternals' Process Explorer in the systray because it graphically plots process and kernel mode CPU over ~10 seconds.  When just about any NetMRI 3.0 table "loading..." is in progress, there's 100% CPU, most of it in kernel mode, top process = Firefox.  I attribute this to the wonderful world of web 2.0.  [:'(]

+1
0
-1
Top
Re: Version 3 is SOOO slow
Thu, 03/05/2009 - 00:01
Re: Version 3 is SOOO slow

Our unit is also painfully slow:

netmri.engin.umich.edu> show stats
+------------+------------+------------+-------------+
| Day        | CpuAvgBusy | FreeMemAvg | DiskUsage   |
+------------+------------+------------+-------------+
| 2009-03-04 |    99.7644 |       NULL | 33.71588977 | 
+------------+------------+------------+-------------+
netmri.engin.umich.edu> show io
Linux 2.4.31-smp (netmri.engin.umich.edu) 	03/05/09

Time: 09:52:48  
avg-cpu:  %user   %nice    %sys   %idle
          37.66   52.42    9.54    0.37

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           39.92         9.83      2017.95    3385726  695058252
dev8-1           92.67       193.75       987.40   66735892  340095530

Time: 09:52:53  
avg-cpu:  %user   %nice    %sys   %idle
          24.20   66.10    9.60    0.10

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
dev8-0           15.00       441.60       198.40       2208        992
dev8-1           49.20        48.00       459.20        240       2296

netmri.engin.umich.edu> show load
 10:00:05  up 3 days, 23:47,  1 user,  load average: 13.95, 19.27, 19.34
+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #11) Wed, 03/25/2009 - 04:33
Re: Version 3 is SOOO slow

I'm in the same boat.  My NetMRIs are almost impossible to deal with now since the upgrade.  Things take forever to show up, I constant get grid and other types of error messages.  I've loaded all the patches available.  My applainces are 3 yrs old now and wonder if now it's a hardware issue.

BB

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #12) Wed, 03/25/2009 - 04:56
Re: Version 3 is SOOO slow

Bryan, are you saying that you have the previous generation platform -- something other than the small 1U half-rack-depth quad-core unit?  If so, I can believe it's going to be a dog.

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #13) Wed, 03/25/2009 - 05:01
Re: Version 3 is SOOO slow

My understanding is the new units are quad-core.  These units have been in place since the beginning which was Sept 2006.  I can't see them physically so I don't remember what theu look like.

BB

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #14) Wed, 03/25/2009 - 05:11
Re: Version 3 is SOOO slow

 I'm sure you'll hear back from the official channel, but the old units were EOL'd as of a year ago.  This came out in Oct 2007:

 "Netcordiais announcing that starting April 1, 2008 Netcordia will be no longer supportthe Dell 1850 appliance hardware with new versions of the NetMRI Software. AllNetMRI software versions released between now and April 1, 2008 will continueto support the Dell 1850 appliance hardware; although some new functionalitymay not be available."

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #15) Wed, 03/25/2009 - 06:06
Re: Version 3 is SOOO slow

My boxes are remote but I had someone look at them and they do not appear to be Dells.  They have the NetMRI logo on them with the blue lcd window.

BB

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #13) Thu, 04/02/2009 - 23:49
Re: Version 3 is SOOO slow

 [quote user="Marty Adkins"]

Bryan, are you saying that you have the previous generation platform -- something other than the small 1U half-rack-depth quad-core unit?  If so, I can believe it's going to be a dog.

[/quote]

Our unit is one of the 2U units that is about half depth. Is that what you mean by previous generation?

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #17) Fri, 04/03/2009 - 00:31
Re: Version 3 is SOOO slow

That sounds like an old unit.  The one we're running 3.0 MR1 on identifies itself as:

Platform       = NCS Quad Core
CPU Type       = Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
Usable Memory  = 3.4GB
Kernel Version = 2.4.31-qc-rm

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #18) Fri, 04/03/2009 - 01:04
Re: Version 3 is SOOO slow

[quote user="Marty Adkins"]

That sounds like an old unit.  The one we're running 3.0 MR1 on identifies itself as:

Platform       = NCS Quad Core
CPU Type       = Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz
Usable Memory  = 3.4GB
Kernel Version = 2.4.31-qc-rm

[/quote] 

 How did you find this? I looked around but did not find a command that would give me that information.

+1
0
-1
Top
Re: Version 3 is SOOO slow
(Reply to #19) Fri, 04/03/2009 - 03:49
Re: Version 3 is SOOO slow

 I did the upgrade to the new version and at the end of the script it shows the what I have :

 ---------------------------------------------------
Platform       = NCS Enterprise
CPU Type       = Intel(R) Xeon(TM) CPU 3.20GHz
Usable Memory  = 3.4GB
Kernel Version = 2.4.31-smp
---------------------------------------------------

So I have the old hardware.

+1
0
-1
Top