Technical Blog Post
Abstract
Performance issues - Part 3 Collection of historical data.
Body
Performance issues - Part 3 Collection of historical data.
Once again the major point is to think about what is being done, what is the data being kept for and is it being used.
In one case recently it had been decided to collect every attribute for each agent type that they had just because they could.
That in itself should not have caused the full issue that it did, however all the collections were started at the same time without checking that the WPA's and the S&P agent were all working correctly.
Due to there being some agents that could not contact the WPA's, data built up at those agents, then once that was fixed, the amount of data that the warehouse received was large.
To compound the issue the S&P had not been running for a couple of days, the result was a S&P run that took days to run, meanwhile more data was being sent in, so some data had to be culled from the warehouse.
So to avoid the issues :
1) Add historical collection a little at a time, making sure that the data collected is getting to the warehouse and is being summarized correctly before adding the next data.
2) There is a spread sheet to help you estimate the size of database needed and how much data each attribute will collect and be obtained via this blog:
3) Make sure the WPA and S&P are tuned for the environment, the blogs to help with this are:
4) Make sure that you are not collecting the same type of data from more than one agent type, as some do collect the same information.
5) Make sure that all the historical data being collected is useful and is going to be used.
The first step is to think about what data you want to save and why you want to keep it.
It may be that you have TCR reports to run and these will state what data is needed to be collected for those reports.
again overtime other reports may be needed, but adding overtime is better than adding everything and then finding you don't need half the data you have.
At each stage of adding data check how long the S&P run is taking, this is reported on in the TEP .
The default configuration is to set a S&P run at 02:00 once each day, however depending on when summarized data is required the runs can be be every hour.
However if you do go down to every hour because you need summarized data, check that the run will finish in that time.
Each run will continue until finished, then the next run will not run until a hour after the finish time, at the time specified in the configuration.
If each time the run is longer than the time allocated then you are not going to see the summarized data expected at the time expected.
For planning you also need to think about how long the data needs to be kept for.
Again better to think about this at the beginning, not when there is a huge amount of data no longer needed which is filling up the warehouse database.
To change the pruning at a later stage once a large amount of data has been collected, means again longer S&P runs, which could result in newer information being delayed from being summarized.
In some cases the data has had to be pruned from the tables manually.
The last area to be aware of, is making sure that you check your WPA's and S&P agents are working and all the data is being collected from agents.
If the information is all being used then usually any issues come to light very quickly.
There are default situations to alert on a number of issues with the WPA and S&P, USE them and either fix the issues, or get help from Support as soon as you can.
The quicker the problem is dealt with the less amount of data the S&P agent has to catch up on.
Subscribe and follow us for all the latest information directly on your social feeds:
|
|
 |
|
| Check out all our other posts and updates: | |
| Academy Blogs: | https://goo.gl/U7cYYY |
| Academy Videos: | https://goo.gl/TLfMoF |
| Academy Google+: | https://goo.gl/HnTs0w |
| Academy Twitter : | https://goo.gl/HnTs0w |
UID
ibm11083759