Extending HP network devices with CPU and Memory counters

Hi,

This blog I had prepared a long time ago but never published it. So now it’s the time to do it.  Since we all know that SCOM 2012 has build in network device monitoring we of course want to use it. This works perfectly , except I you are using devices that are not certified by SCOM. In that case you will not get the CPU and memory counter and Fan/PSU states. For example most of the Procurve network devices from HP are left in the dark. So..

The problem

We want to monitor also CPU and Memory usages from HP network devices.

The solution

I will keep it simple and clear. I will demonstrate the steps using VSAE and give you as bonus the MP at the end. I will use VSAE management pack templates just to illustrate the power of it.

The steps will be:

1. [Classes] create the memory and CPU classes

We just simple make use of on the memory and processor classes that are already present in the system center network library. So we create 2 classes based on that:

 

2. [Datasources] create the memory and CPU discovery datasources

Now we have to create the datasources for the discovery of the CPU and memory targets

The most important is that you set the device key correctly to the parent node. Otherwise the relationship between the new CPU / memory class and the network device node will not be set. And you will not see any CPU / Memory targets. This can sometime be a ‘error and retry’ process.

3. [Discovery’s] create discovery’s for the CPU and Memory targets

Now we have the discovery datasource we can create the 2 discovery rules for the CPU and Memory targets. We will use the VSAE discovery templates for this. Just simple add new ‘Discovery’ template and you add the 2 rules below.

Create the 2 new discovery’s and specify the correct datasources.

You specify the datasources we created above to the correct discovery rule. Fill in the correct OID to match the HP processor and Memory OIDs.

If you now import the MP will will see at a HP node the CPU and Memory classes created. So continue the the next step.

4) [Rules] create the memory and CPU collection rules

So now we can make the stuff we wanted to see. The Memory and CPU counters. So we just simple use the add new item ‘Rule (Performance Collection)’ template and make the 2 new rules. We are going to use the already build in performance collection rules for the network nodes.

 

The most important is to specify the correct OID for the Memory and CPU counters at the datasource configuration. See below:

CPU: .1.3.6.1.4.1.11.2.14.11.5.1.9.6.1.0
Memory: .1.3.6.1.4.1.11.2.14.11.5.1.1.2.2.1.1.7 and .1.3.6.1.4.1.11.2.14.11.5.1.1.2.2.1.1.5

I am not going to do a deep dive on this. You can just simple reference to the downloadable project source at the end of this post to check the details out.

The results

Now you import the MP and you will see you most wanted performance counters !! Open the Network Summary Dashboards and Node dashboard for the CPU Usage to check it out.

And the ‘free Memory (Percent)’ performance view for the memory usage.

 

And below the relation ship diagram.

The End

The next step could be to add monitors to alert on high Memory or CU usages. I am not going to give you this bonus because its better to get some VSAE practice you self’s… You can do this the same way you did with the collection rules. There is a template for monitors also.

Download VSAE project (for the diehards): http://sdrv.ms/11mkq9j

Download Management Pack example: http://sdrv.ms/11mku8S

Happy Scomming

Michel Kamp

Let SCOM check for Updated Management Packs

The challenge

Using the SCOM native console the import from the Microsoft Management Pack Catalog is a nice feature. I like also the feature to check and import updated MPs that you have already imported in your management group. But what I really miss and don’t understand : why did the product team removed the monitor that gives us a alert when a new MP version is in the MP catalog ?. This monitor was build in MOM 2005 but removed in the begin of SCOM 2007.

The solution

So since we are SCOM author diehards we are going to build our own MP update monitor. I am going to use VSAE to build it all. But wait even if you aren’t a SCOM author diehard it still worth reading this post because this time I will share the VSAE project and even the MP with you at the end!!!

Analyze

So I used my good old friend ‘Fiddler’ to backward-engineer what the scom console is doing when I press the ‘check for updated management packs’ button. It seems it sends a SOAP request to a webservice. The SOAP request contains a MP list of the MPs that you have already imported. The answer result of this request will be a MP list with the updated MP versions or an empty list if there aren’t any updates for you.

Building time

Below I’m going to give you a overview what I have done. You can look in the VSAE project for details on it. If you have any questions just let it know and I will help you out.

1) The datasource

So now we are going to make a datasource that runs a PowerShell script. This PowerShell script is simulating the webservice request.

Below a snippet of the code. (the full code is in the VSAE project). What I am doing here are 3 steps:

1) Build a SOAP request message that contains all my MP version meta data from all MPs that I have already imported in my management group.

2) I call the “ManagementPackCatalogWebService.asmx” and execute the method “FindManagementPacks”

3) as last step I check if there are any MPs returned and set the $Status flag according the result. And I return the scom property bag.

# step 1 $MPSoap = get_MP_List $ret = Do-SOAPRequest -SOAPRequest $MPSoap -URL $MPCatalogURL -SOAPAction $SOAPAction # step 2 ## show MPs that have a Update $MpList = $ret.Envelope.Body.FindManagementPacksResponse.FindManagementPacksResult.CatalogItem | where { $_.IsManagementPack -eq $true} | select-Object DisplayName # step 3 ## check MP returned if ( $MpList.Count -eq 0) { $Status=”UPTODATE” } else { $Status=”NOTUPTODATE” }   # Create the property bags $pb = $oAPI.CreatePropertyBag() $pb.AddValue(“Status”,$Status) $pb.AddValue(“MpList”,($MpList | Out-String)) $pb

The script above we are going to use in the datasource below

2) The Monitor

Now we are going to compose a 2 state UnitMonitorType that uses this datasource. The health state check is done with the “Status” value in the returned property bag.

Having this UnitMonitorType composed we can now use it in the real monitor KPI. See below for the KPI. The target is the Management server. I choose this target because I have only one MNG server in my test lab but if you have more it’s better to choose the RMS emulator target.

Now when the monitor is unhealthy it will generate an alert message constructed below:

 

The result

Building and importing the MP in your SCOM management group will show you the result below:

 

And of course a nice ALERT message also:

 

So now the part you are waiting for..

As promised I will share the VSAE project and the MP it self. Please notice that it is a show case alias prototype MP and so it is far from complete. For example not all display strings are applied and no knowledge is supplied. But that’s up to you to complete…. In my production version I have even build in a recovery/console task that also automatically imports the updated MPs.. Just a idea for you to work out…

MP download: http://sdrv.ms/XPl38e

VSAE project download: http://sdrv.ms/YDUn7T

The End

Feel free to comment or contact me if you have any questions.

Happy SCOMMING

Michel Kamp

Get a grip on the DWH aggregations

 

The problem

If you run availability reports or performance reports with a aggregation type of daily or hourly the reports are empty. This problem is described a lot on the web. And I have also written a couple of blog post how to fix this issue. But as you know we are using scom to monitor stuff , so why not monitor this aggregation processing and alert if a processing delay is occurring. ? That’s our mission today….

Analyze

Using SQL enterprise manager and a SQL query on the data warehouse DB we can read out the aggregation processing. This query looks like this:

Select AggregationTypeId, Datasetid, (Select SchemaName From StandardDataSet Where Datasetid = StandardDataSetAggregationHistory.Datasetid) ,  COUNT(*) as ‘Count’, MIN(AggregationDateTime) as ‘First’, MAX(AggregationDateTime) as ‘Last’ From StandardDataSetAggregationHistory Where LastAggregationDurationSeconds IS NULL group by AggregationTypeId , Datasetid

The output will show us how many aggregations there have still to be processed /aggreationtype  (20=hourly , 30 = daily).

So in this case we have no problem. But I have seen scom environments where the state aggregations where so far behind that it was almost not possible to fix it. This bring up a point: especially the state aggregations are the tricky ones. If you have many ‘flipping’ monitors there will be a lot of state changes and so a lot of aggregations data to process. This process takes a lot of SQL CPU power and also disk space. In most of this cases it was the tempdb data space free or transaction log that was the root cause of the failure.

Solution

In scom we have for every aggregation an target. This target is named ‘Standard data set’. You can find it here:

If you compare the screenshot with the results on your scom console you will notice that you don’t have the green healthy state… And that’s why you are reading this post. So lets add this state.

I wanted to give every dataset that has to be processed a health state on how many aggregation it has still to process. So we make a monitor that executes for every data set the query above and if a threshold is hit the health state is changed. Also we will add a rule so that this aggregation behind count is put in a trend graph.

I have used VSAE for this , and I will not share the code but only the idea. Why not ? I believe you have to know what you are doing and by copy & pasting you don’t learn from it if you don’t have done it once from start till end.

The real work

Open a new VSAE project and add a empty MP fragment and a PowerShell fragment.

Then you make a datasource that reads the aggregation count. This is done using PowerShell and the SQL snapin.

The PowerShell script has as input the GUID of the dataset (property of the target) and as output a property bag with the aggregations count (daily and hourly). I made the script somewhat intelligent by reading out the registry where the data warehouse is located.

Now we use this datasource in a monitor module type to create a 3 state monitor. And since we have created a datasource module we can create also a rule that collects the aggregation behind for the trend graph. Yes know know this is easier to type as to do…

Below a snap of the datasource module

And below a snap of the monitor module type

and the monitor. Create one for hourly(not shown) an one for daily.

At last for trending we have to create a collection rule.

Notice that the monitor and collection rule are having as target the “Microsoft.SystemCenter.DataWarehouse.DataSet” alias “standard dataset” and notice the runas profile.

The result

When you have constructed the MP and build/deployed it you will see 2 extra monitors on the standard dataset targets as show above. Open the health explorer to see if all is ok.

Above dataset has had a problem. To see some details, view the performance counters and you will see the aggregations trend.

In this case the state hourly aggregations where way behind. So I followed one of my own blog posts to solve this one. Where I manually executed in a loop the state aggregation process to speed up the processing.

The End.

Yes I know this post is a bit ‘çloudy’ and not something you can download and import. But I hope by sharing the idea I triggered you to try it your self.

Happy SCOMMING!

Michel Kamp

Authoring SCOM Reports in VS 2010

Hi,

Short post on how to get you dev environment ready for authoring scom reports.

Challenge:

You have installed SCOM 2012 on SQL 2008. You want to author a custom report using Visual studio 2010. When you open visual studio you will notice that NO BI project template is shown. Normally you selected this project template and selected the new report project to make your custom report. How now to continue ?

Solved:

Grab a SQL 2012 ISO (YES 2012) and startup the setup.

1) Select installation:

2) New sql or add features

3) Select SQL features Install

4) Now the important step. Select the 3 options here. Most important is the “SQL Server Data Tools”. This features contains the VS BI project template.

5) Step though the install windows.

And now open Visual studio 2010 and create a new project. And what do we see ?

Yes the BI template  

 

Now you can create the new SCOM reports. Notice also the NEW chart types !!!

 

Remember that if you use custom report code components you must copy the correct .dll assemble to the directory:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies

The End.

Happy Scomming

Michel Kamp

10 reasons NOT to use the Visual Studio Authoring Extensions VSAE

Hi this weekly post will be a short one all about the yet to be released VSAE.

Have you seen the title ?… Hmm what are you thinking now ? Not really something he would write … And you are correct its nonsense .. But I have your attention now , haven’t I…

Reason 1: SourceSafe integration

If you have read my previous post you know I really prio using SourceSafe for managing your management pack developments. And good news since the VSAE is a Visual studio project template is can be integrated out of the box with all VS supported SourceSafe systems. Just put the Solution to your SourceSafe system and every thing will be versioned. I use the good old VSS 2005 for this. But TSFS will also do the job.

Reason 2: MP object viewer

Yea! this feature is super. Normally you had to browse your MP by XML to view your project content. Or you use external tools for it. Now you will get it out of the box.In the View menu of the main VS window. Press Management Pack browser.

And you will see all the MPs in your solution/projects. Now you can browse it by type. You can even now browse SEALED mps at ease.

 

Also when double clicking on the type. For example the discovery rule. You will be jumping directly to it in your MP.
BE AWARE THAT IF YOU DO THIS AT A SEALED MP YOU WILL BE REDIRECTED TO A READONLY VERSION OF THE MP FILE. YOU CAN’T EDIT IN THIS WINDOW. Sounds logical since it’s a sealed MP. But I had this confusion situation one’s when I clicked on my own sealed MP that I use as reference in a other MP. And was wondering why I couldn’t edit it since it was checked out by VSS…

Reason 3: Find references

You can now find at ease where you MP object is used. Just use the find references find feature. Select the MP object and right click “Find all references”

And you will get:

And of course you can double click to go to MP object ‘code’. ; – )

If you don’t see the window. Just press in the VS main menu: View –> Find MP Element Reference Results.

Reason 4: Simulating your workflows

Of course you will never used this , because your MPs are faultless. ;- ) The mine aren’t so I use this a lot. The Workflow simulator is integrated in VS now. Just select in the Management Pack Browser the , for example discovery , and right click “MP Simulator”. And that all.!!

In a separate windows the already known workflow simulator will popup.

Reason 5: Jump to referenced MP objects.

Very powerful feature. When you are working in your MP project at a MP object and are using for example a MP TYPE you can easily jump to this type by right click on the name and select “Go to Definition” (F12).

Reason 6: intellisense

If you have read my post on how to enable intellisense for OM07 MPs in VS this isn’t so new. But now it even does intellisense on references also.. this really speeds up the development time. cool

Remember: This intellisense is not always automatically showing up. Most of the time you must press SHIFT+Space. Example above, press the keys at the ! sign and you will prompted a dropdown list with the possible choices.  Notice that only the valid modules of type datasource are displayed. Well done Microsoft product team( s ) !

Reason 7: Build and deploy.

At the end we all have to import the MP into our dev environment to test it. Normally you will do a MP verify and then import it. You can use tasks for this in VS. I already blogged about this earlier. However in VSAE the MP is build of MP fragments and has to be compiled and build. Now the great time safer is that you can now specify the OM MNG group were this MP has to be imported after a successful build. You can even configure that the MP has to be sealed before import. And yes the version number is also incremented at build.

Even when the build gives errors back you will see it in the Error list window. And you can click on it to jump to the MP object. Wow never have to use CTRL+G again ; – ) 
Notice that this jump does not apply for target typo’s . I hope this will be fixed in the RTM release.

Reason 8: Management objects templates

You can now create/insert at ease new KPIs. Just press NEW-> Add existing Item and you can choose several KPI to create. I already blogged about this. The nice part is that you can even automate this so that you can create 100x rule at ones. I have to amid that I don’t use this templates a lot. I prefer to type it manually but sometimes I use this to get a sort of snap in fragments. Just create the KPI using the template and the look at the generated code behind and copy it. Be aware that if you change the code behind you will loose it at next build… I personally would liked if this generated code was also reflexed back against the template.

Reason 9: Convert your ‘old’ OM07 MPs.

Using the VSAE you can convert your OM07 MPs to the new VSAE project types. It very easy just use the “Project from existing Management Pack” , select the om07 MP and the references and a new project will be created in your solution. If you select multiply om07 MPs there will be created a new project for every MP. Very handy. The down site is however that it will put all the MP objects into one big MP fragment. I personally would liked to have seen this split-up by type or maybe converted to templates.

Tip: If you converted MP has images it will extract this images to resources and put them in the root of the project directory. You can create a new folder and drag them into here. Be aware that there is a little bug: The resource MP objects haven’t the images file extensions. So a build will fail. Just add the file extensions .jpg ect.. to it and it will run.

 

Reason 10: It just a must have.

I could continue to write about the tons of great features that are build in the VSAE but over 30min the soccer Europe 2012 game Netherlands->Portugal is going to start so I will warp it up and say:
use it use it use it use it . its no question why , just use the VSAE !

Happy scomming and till next week.

Michel Kamp

Part 2 : Get more value using Visio drill trough

Hi Quick Update. In part 1 I promised when I had time I would make a prototype of a OM 12 Widget that would show a Visio document and update the health states on it.

So , not that I had time but the coolness factor was to high , so I offered some hours on Sunday to get this working… yes you have read it correctly. I have it working !

Mission

Make a OM 2012 widget and load a Visio document in it. Then refresh the shape data according to the OM target health states. Just the way the official Microsoft Visio SCOM plug in it does.

Accompliced

So far I have it working as a prototype. Yes its stable but as every prototype not fool proof. See screen shots below.

The prototype works based on Visio documents you have made with the official Visio SCOM plugin. Just create the Visio document and drag the OM targets on it and save it. Then load it in to the widget , for now with File –> open. There is one rule: You must have Visio installed on the machine running the OM console.

You don’t have to have the official Visio SCOM plugin installed when you only view the Visio sheets in the Widgets. I have made a refresh feature that connects to the OM group and gets the health state data and refreshes the Visio shapes based on the outcome. But when you have the official Visio SCOM plugin installed you get a bonus. It works also in the widget with official refreshing. See screenshot below:

After getting a try timeout it shows the login form. Better would be that it takes the credentials you are using running the OM console . But … Again prototype facts.

Conclusion

Sorry I really love being a nerd! (that’s what my wife says anyway) If you are honest you must admit this is really cool stuff!!. Since it will cost me for now to much time to finish a build I can share and because the VSAE is still under NDA and I maybe have plans with this. I will not share it for now. However I will try to convince the MS OM guys ( Dale , Marcin, Baelson hope you read this) to deliver this widget out of the box..

Happy Scomming and till next week!

Michel Kamp

Great resources on Visual Studio Authoring Extensions

Almost forgot to share. Don’t know if you authors know it already.

Remember you can also use this for making 2007R2 MPs!!

Visual Studio Authoring Extensions for System Center 2012 – Operations ManagerVisual Studio Authoring Extensions – Visual Studio Features
Visual Studio Authoring Extensions – Simple Templates
Visual Studio Authoring Extensions – Advanced Templates
Visual Studio Authoring Extensions – Example Management Pack

Don’t forget to look at my latest post on creating a MP with VSAE

Happy Scomming

Michel Kamp