new()
Initialise()
LoadCfgFile()
PrintDebug()
PrintTimestampedDebug()
IsInSubnet()
ApplyUserMethodFilter()
StdUpdateData()
StdGetInfo()
StdGetDeviceList()
StdGetDeviceInfo()
StdGetInventory()
StdGetEntities()
StdGetAssociatedAddresses()
StdGetMplsInterfaces()
StdGetConnections()
Collector::Collector - Base class to ease Collector development
See the Example Perl Collector section of the DESCRIPTION for an example too detailed for this SYNOPSIS section.
Tip: This document acts as the entry point into the Perl Collector Library documentation. It is recommended that the Collector Developer Guide document be read prior to this and the Collector::Store documentation
The Collector::Collector module is the base class of all Perl based Collectors. It provides a standard implementation of the Precision RPC methods using the data from the supplied Store object.
It is intended that the user derive their Collector class from this class, pass it their required Collector::Store derived storage object, and override any RPC methods needing processing above and beyond that which is automatically provided by this class.
ITNM's Perl based Collectors are shipped with a Collector support library, which eases development by taking care of XML generation and some ITNM facing tasks.
If you are writing a Collector in Perl, you may wish to consider using this library to reduce development and maintenance time. When using the Perl Collector library, the developer's responsibility changes from 'convert EMS data into XML' to 'convert EMS data for storing in the Store object'. At present, the Collector support library exists only for Perl based Collectors.
The Collector support library consists of the following modules:
This module provides general facilities of use to all Collectors, such as configuration file parsing. This module also provides a standard method for each ITNM supported XML-RPC method, which uses the Collector::Store data to automatically build an XML response. The user need only pass control to these standard functions - no manual XML generation is required.
This module provides a device data store, which allows automatic generation of valid XML via internal use of the Collector::DataValidation.
This module provides facilities to validate Collecter::Store data and to translate Collector::Store fields into XML tags for use with Collector::Store. From a Collector developers perspective, the Collector::DataValidation module is hidden by the Collector::Store module. The Collector::DataValidation module will not be covered in detail because the developer will not use it directly - there is no public API.
A Collector can use this library to;
The following table provides a quick reference showing which Collector::Store and Collector::Collector methods need to be called to satisfy a given RPC request.
+-------------------------+-----------------------+--------------------------+ | Enable automatic XML | ...by adding your | ...and then call this | | response generation | discovered data to | Collector::Collector | | for this ITNM XML-RPC | your Collector::Store | method from your XML-RPC | | call... | via... | call implementation... | +-------------------------+-----------------------+--------------------------+ | UpdateData() | N/A | StdGetUpdateData() | | GetInfo() | SetupDataSources() | StdGetInfo() | | GetDeviceList() | AddDeviceData() | StdGetDeviceList() | | GetDeviceInfo() | AddDeviceData() | StdGetDeviceInfo() | | GetAssociatedAddresses()| AddAssocAddress() | StdGetAssocAddresses() | | GetInventory() | AddInterface() | StdGetInventory() | | GetEntities() | AddEntity() | StdGetEntities() | | GetLayer2Vpns() | AddL2Vpn() | StdGetLayer2Vpns() | | GetLayer3Vpns() | AddL3Vpn(), | StdGetLayer3Vpns() | | | AddL3VpnInterface() | | | | AddL3VpnRouteTarget() | | | GetLayer1Connections() | AddLayer1Connection() | StdGetLayer1Connections()| | GetLayer2Connections() | AddLayer2Connection() | StdGetLayer2Connections()| | GetLayer3Connections() | AddLayer3Connection() | StdGetLayer3Connections()| +-------------------------+-----------------------+--------------------------+
For example, if the developer wishes to support the RPC method GetDeviceInfo()
then the Collector would have to populate the store via Collector::Store's AddDeviceData()
and in its implementation of GetDeviceInfo()
it would call Collector::Collector-> StdGetDeviceInfo()
to convert the stored data into a valid XML response.
See the Collector::Store documentation for information on the methods that can be used to populate the Store.
Any changes to the Perl Collector Library are backwards compatible meaning that Collectors written against an earlier release will work with the newer releases. However you may wish to update your Collector to take advantage of any library changes that have been made in more recent releases.
The following tables summarise changes since 4.1 GA that may be of particular interest to those considering migrating their custom Collectors. It is not a complete list of changes.
+-------------------------+--------------+------------------------------------------------------+ | Area | Version | Summary of Change | | | Introduced | | +-------------------------+--------------+------------------------------------------------------+ | Collector.pm | 3.9 IF1/FP4 | New method: CreateStdXmlRpcServer | | | 4.1.1 FP1 | Collectors can call to create a server | | | 4.2 GA | that has a configurable queue size with a higher | | | | default (40 vs 5). This is now of more importance as | | | | in these releases DISCO now sends multiple requests | +-------------------------+--------------+------------------------------------------------------+ | Store | 4.1.1 GA | New module, LteDataStore.pm, for LTE data | +-------------------------+--------------+------------------------------------------------------+
This section details an example Perl Collector that makes use of the Perl Collector support library to implement the pseudo code Collector described the separate Collector Developer Guide.
# Example Perl Collector # use 5.6.1; use strict; use warnings;
# Assumes the Perl Collector Library is in ../lib/ use lib "../lib/";
use MyCollector; # your class to handle EMS data
use XMLRPC::Transport::HTTP; # XML-RPC Server
# g_XmlRpcServer: For XML-RPC communication with ITNM. my $g_XmlRpcServer;
# g_Collector: Manages the communication with the EMS, and provides # methods for converting EMS data into Store data. # This class is derived from Collector::Collector. It will create and # use a Collector::Store my $g_Collector;
####################################################################### # Entry Point #######################################################################
# instantiate the Collector $g_Collector = MyCollector->new();
$g_Collector->LoadData();
# Set up the XML-RPC methods supported by this Collector. my @availableMethods = ( 'UpdateData', 'GetDeviceInfo'); my %methodCfg; # not supported in this example - all methods enabled
my @filteredMethods = $g_Collector->ApplyUserMethodFilter( \@availableMethods, %methodCfg);
# start the XML-RPC server (for ITNM communications) my $listenPort = 8086;
$g_XmlRpcServer=XMLRPC::Transport::HTTP::Daemon->new(LocalPort=>$listenPort, ReuseAddr=>1);
$g_XmlRpcServer->dispatch_to( @filteredMethods );
print "Listening for XML-RPC calls...\n";
# Enter Infinite loop listening for ITNM calls... $g_XmlRpcServer->handle();
####################################################################### # Collector-To-ITNM XML-RPC Method #######################################################################
## ####################################################################### # GetDeviceInfo() Implementation ####################################################################### ## sub GetDeviceInfo() { my $dataSourceId = $_[1]; my $deviceId = $_[2];
my $xmlResponse = $g_Collector->StdGetDeviceInfo($dataSourceId, $deviceId);
return $xmlResponse; }
## ####################################################################### # UpdateData() Implementation ####################################################################### ## sub UpdateData() { my $dataSourceId = $_[1]; my $requestType = $_[2]; my $address = $_[3]; my $mask = $_[4];
# here we would refresh our Store data, if necessary and then send # back a the standard response
my $xmlResponse = $g_Collector->StdUpdateData();
return $xmlResponse; }
# MyCollector # A simple example Perl Collector that only supports basic device data # package MyCollector; use lib "../lib/"; use Collector::Collector; use Collector::Store; @ISA = qw(Collector::Collector);
use 5.6.1; use strict; use warnings;
sub new() { # All Collectors define a data source. Usually just one - the EMS. my @sourceInfo = ( { Id=>1, Descr=>"Primary Data Source" } );
my $store = Collector::Store->new();
$store->SetupDataSources( \@sourceInfo );
$store->Initialise( "MyCollector", "An example Collector" ); my $self = Collector::Collector->new( $store ); $self->{Store} = $store;
bless( $self ); return $self; }
sub LoadData() { my $self = $_[0]; my $store = $self->{Store};
$store->DeleteData(1); # Assume only one data source (1)
# Get EMS device data in a Perl hash that meets the requirements # of Collector::Stores AddDeviceData() method # Here we fake it; my $node = { BaseName => "deviceA", SysDescr => "Example device", };
$store->AddDeviceData( 1, $node->{BaseName}, $node ); } return 1;
The above example Collector can be run on the command line as follows:
> ncp_perl main.pl Listening for XML-RPC calls...
The example Collector as it stands lacks support for the mandatory methods required by ITNM and so it is not possible to discover it. You can however use the ncp_query_collector tool to issue a request for the supported XML-RPC method.
This can be done as follows; (you can find the tool in $NCHOME/precision/scripts/perl/scripts)
> ncp_perl ncp_query_collector.pl -port 8086 Starting ncp_query_collector... Interactive Mode: Known Methods are;
GetMplsInterfaces(datasourceid deviceid) GetDeviceList(datasourceid requestType address mask) GetEntities(datasourceid deviceid) GetAssociatedAddresses(datasourceid deviceid) UpdateData(datasourceid requestType address mask) GetLayer2Connections(datasourceid deviceid) GetInventory(datasourceid deviceid) GetInfo() GetFunctionData(datasourceid deviceid functionName optionalFunctionArgs) GetLsps(datasourceid deviceid) GetDeviceInfo(datasourceid deviceid) GetLayer3Vpns(datasourceid deviceid) GetLayer3Connections(datasourceid deviceid) GetLayer2Vpns(datasourceid deviceid)
Note: requestType (where required) are; 0: Full Discovery 1: Single Device 2: Single Subnet
Toggle XML pretty printing on/off using 'prettyOn'/'prettyOff'
Display command history using 'hist', and execute from history using !<index>
exit quits
> GetDeviceInfo(1,deviceA) <deviceInfo> <name>deviceA</name> <descr>Example device</descr> <protocol>4</protocol> </deviceInfo>
> exit
TIP: Help is available for this tool by typing 'help' on the command line.
This section provides information on how Perl Collectors can use the NCP::XSV CSV parsing Perl module. For further detail of NCP::XSV specifics see the documentation for that module.
The parsing performed by NCP::XSV is controlled via a configuration file (.cfg) and associated driver file (.drv). The .cfg file is passed as parameters to the NCP::XSV module by the Collector during initialisation.
The NCP::XSV module works on data 'groups', which are user defined. The configuration file states which CSV file the data for a given group comes from while the driver file tells NCP::XSV how to parse the column data in the group. Collectors can then use the NCP::XSV module to load data groups.
The Configuration file defines the data groups, which .CSV files are associated with the groups and which Driver file to use during parsing.
The following top level configuration options are supported:
System level configuration options; for example, logging.
The name of the driver file (defines the column mappings for all data groups)
The .csv data file location for each data group. Each group must be represented even if it re-uses the .csv data file of an already listed group.
Allows configuration of the character used as the column delimeter. This is useful in rare cases where the standard delimeter is occurs in the data causing incorrect column dissection.
Where multiple data groups use the same .csv file (specified in 'file') there needs to be a way for NCP::XSV to distinguish between rows. This is achieved by allowing each data group to have a regular epression defined that identifies the row.
The Driver file (.drv) specifies the per-group mapping for CSV data to data group attribute. The following top level configuration options are supported:
Defines the form of the data; columMap or attributeMap.
For each data group provides a description and which columns are of interest.
For each data group provides the per column mapping to the Collector attribute
The NCP::XSV module works on data 'groups', which are user defined. The configuration file states which CSV file the data for a given group comes from while the driver file tells NCP_XSV how to parse the column data in the group. Collectors can then use the NCP::XSV module to load data groups.
It is usually convinient to configure data groups that relate to the XML-RPC methods that you intend to support.
For example, the following is a list of the data groups supported by the Generic CSV Collector and the XML-RPC methods that use the data from the group:
GetDeviceInfo()
GetInventory()
GetEntities()
Usually a data group will have its own associated data file (.csv) however where this is inconvienient it is possible to have multiple groups (or even all groups) use the same data file. In such cases an additional configuration must be made in the .cfg file (see 'lineMatch') to provide a regular expression that allows NCP::XSV to identify which rows in the data file belong to which data group.
Normally when defining the column-to-Collector attribute mapping in the Driver file (.drv) you are mapping a single column value to a single attribute, e.g.
'mappingGroupDefinitions' => { 'MainEntityData' => { 1 => { Name => 'keyIpAddress', Description => '', MapsTo => 'BaseName' },
which maps column 1 'keyIpAddress' in a .CSV file to the BaseName attribute. However with freeform data there are two options; either you can copy a single consolidated XML string into the extraInfo attribute, or you can specify each freeform attribute in its own column and map it to a sub-attribute of the extraInfo attribute,
For example if you need to add freeform fields for Temperature and FanSpeed then you can either do it as a single column like this;
'mappingGroupDefinitions' => { 'MainEntityData' => { ... 4 => { Name => 'extraInfo', Description => 'single XML string with both attributes', MapsTo => 'extraInfo' }, ...
where column 1 in the .CSV file looks like this;
...,"<Temperature>48 degrees</Temperature><FanSpeed>Low</FanSpeed>",...
Or you can split the freeform attributes into columns;
'mappingGroupDefinitions' => { 'MainEntityData' => { ... 4 => { Name => 'Temperature', Description => 'Degrees C', MapsTo => 'ExtraInfo->Temperature' }, 5 => { Name => 'FanSpeed', Description => 'High, Low', MapsTo => 'ExtraInfo->FanSpeed' }, ...
where column 1 in the .CSV file looks like this;
...,48 degrees,Low,...
Here are examples of a configuration and a driver for use with NCP::XSV:
( system => { logfile => 'example.log', verbose => 0, logging => 0 }, driver => { file => './myDriver.drv' }, file => { MainEntityData => './devices.csv', ...other supported data groups and CSV files... }, delimeter => { MainEntityData => ',', ...delimeters for the other data groups... } )
( 'type' => 'columnMap',
'mappingGroupDescriptions' => { 'MainEntityData' => { Description => 'One row per device', DelimetedBy => ',', UseColumns => '*', }, ...descriptions for other supported data groups... },
'mappingGroupDefinitions' => { 'MainEntityData' => { 1 => { Name => 'ManagementIpAddress', Description => '', MapsTo => 'ManagementIpAddress' }, 2 => { Name => 'BaseName', Description => '', MapsTo => 'BaseName' }, 3 => { Name => 'SysObjectId', Description => '', MapsTo => 'SysObjectId' }, },
...column parsing data for the other data groups... } )
# Create parser using the configuration information in myConfig.cfg $myParser = NCP::XSV->new( 'myConfig.cfg' );
# Load the data of data group 'MainEntityData'. # This will load each row of each column from the file specified # in 'myConfig.cfg' for data group 'MainEntityData'. # The resulting list holds one hash per row of data, where the # keys in each hash represent the column names as defined in # the driver file (myDriver.drv). # my @response = $myParser->load( 'MainEntityData' );
foreach( my $row (@response) ) { my $ipAddress = $row->{ManagementIpAddress}; my $name = $row->{BaseName}; my $sysOid = $row->{SysObjectId}; Print "Extracted $ipAddress, $name, $sysOid\n"; ... etc...
The methods in this section cover the creation and initialisation of the Collector::Collector object.
A users Collector will be implemented mostly as a class derived from Collector::Collector, and it is during the construction and initialisation of an object of this class that the Collector::Collector object is created and initialised.
Typically a users implementation of new()
will;
Then configuration options can be loaded by calling Collector::Collector->LoadCfgFile()
.
These configuration options can then be manipulated prior to use in Collector initialisation.
The user Collector would then call Initialise()
which;
Initialise()
on the Collector::Collector object (passing the debug level)
For example, in the MyCollector.pm example presented earlier notice how
MyCollector::new() calls Collector::Collector::new(). Unfortunately the example is too
simple to show the Initialise()
method being chained in the same way
(examine the GenericCsv Collector code for more complex example).
new()
Initialise()
LoadCfgFile()
One of the most useful features of the Collector::Collector module is that it can make use of the Collector::Store and Collector::DataValidation modules to provide valid XML responses to RPCs that are called upon.
To take advantage of this, the developer's implementation of the RPC methods should call the methods described in this section to automatically generate XML.
StdUpdateData()
StdGetInfo()
StdGetDeviceList()
StdGetDeviceInfo()
StdGetInventory()
StdGetEntities()
StdGetAssociatedAddresses()
StdGetLsps()
StdGetMplsInterfaces()
This section lists some helper methods.
The ApplyUserMethodFilter()
method is useful in registering the supported
XML-RPC methods with your XML-RPC server in a configurable way.
API methods are described below.
new()
Creates a new Collector::Collector object.
The following members are created;
- {Store}
Holds a reference to the Collector::Store derived object which is used by the Collector to hold discovered data and to translate it into XML
- {DebugLevel}
Holds the debug level; 0 = no debug, 1 = debug
- Arguments
class: The class name
store: Reference to a valid Collector::Store derived object
- Return Value
blessed $self
Initialise()
Initializes the Collector object. This method should be called before the object is used.
- Arguments
debuglevel (integer): controls the output of debug messages 0 (off) or 1 - 4 (on)
- Return Value
1 if initialised ok, else 0
LoadCfgFile()
Reads the Collector configuration options from the specified file and returns the options as a Hash.
Note this method can be called without the need to create a Collector object via Collector::Collector::LoadCfgFile( cfgName ).
- Arguments
cfgName(string): name of the file that holds Collector configuration
- Return Value
Configuration hash. Updated with options loaded from the configuration file
PrintDebug()
Prints the supplied text to the screen IF the debug level is not 0
- Arguments
message(string): Message to print if debug is enabled
PrintTimestampedDebug()
Prints the supplied text to the screen IF the debug level is not 0. Text has a timestamp added to the end.
- Arguments
message(string): Message to print if debug is enabled
IsInSubnet()
Checks to see if the supplied IP address lies in the range of addresses covered by the supplied subnet nad mask
- Arguments
ip(string): Ip address to test subnet(string): Subnet to check against mask(string): Subnet mask to check against
- Return Value
1 if ip falls withing the scope defined by subnet/mask, else 0
Create the server that will listen for Disco issued XML-RPC calls.
- Arguments
self(object): this pointer config(object): The configuration object
- Return Value
returns server object
ApplyUserMethodFilter()
Applies the user specified method configuration options to the supplied method list. The end result is a list of methods that were in the original method list AND that are enabled according to the user configuration.
A method is considered enabled if it exists in the methodList, and either is set to 1 in the methodCfg hash or doesn't exist in the methodCfg hash.
This method is used once the Data Acquisition options have been loaded from the Collector configuration file. It allows the Collector to pass its list of supported RPC methods along with a filter based on the users desired data set.
- Arguments
methodList(list): a list of all RPC method names supported by the Collector
methodCfg(hash): a hash list of methods, which the user wants the Collector to support. Of the form methodName=>0 or 1
- Returns
This method returns the list of RPC methods that the Collector should register based on those it supports and those the user has selected.
StdUpdateData()
The standard implementation of the
UpdateData()
RPC method.
- Arguments
dataSourceId (integer): data source to update
requestType (integer): affect 0=all devices, 1=single device, 2=all in subnet
address (string): address or subnet of device(s) to update
mask (string): mask for subnet (only used if requestType is 2)
- Return Value
Returns an XML string that is a valid response to the
UpdateData()
RPC call.
StdGetInfo()
The standard implementation of the
GetInfo()
RPC method.
- Arguments
none
- Return Value
Returns an XML string that is a valid response to the
GetInfo()
RPC call.
StdGetDeviceList()
The standard implementation of the
GetDeviceList()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
requestType (integer) : 0=all devices, 1=single device, 2=all in subnet
address (string): address or subnet returned device(s) should match
subnet (string): subnet mask (only used if requestType is 2)
- Return Value
Returns an XML string that is a valid response to the
GetDeviceList()
RPC call.
StdGetDeviceInfo()
The standard implementation of the
GetDeviceInfo()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the
GetDeviceInfo()
RPC call.
StdGetInventory()
The standard implementation of the
GetInventory()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the
GetInventory()
RPC call.
StdGetEntities()
The standard implementation of the
GetEntities()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the
GetEntities()
RPC call.
StdGetAssociatedAddresses()
The standard implementation of the
GetAssociatedAddresses()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the
GetAssociatedAddresses()
RPC call.
The standard implementation of the GetLayer3Vpns() RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the GetLayer3Vpns() RPC call.
The standard implementation of the GetLayer2Vpns() RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the GetLayer2Vpns() RPC call.
StdGetMplsInterfaces()
The standard implementation of the
GetMplsInterfaces()
RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the
GetMplsInterfaces()
RPC call.
The standard implementation of the GetLayer3Connections() RPC method.
This returns data to the CollectorLayer3 agent.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the GetLayer3Connections() RPC call.
The standard implementation of the GetLayer2Connections() RPC method.
This returns data to the CollectorLayer2 agent.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the GetLayer2Connections() RPC call.
The standard implementation of the GetLayer1Connections() RPC method.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
- Return Value
Returns an XML string that is a valid response to the GetLayer1Connections() RPC call.
StdGetConnections()
The standard implementation of the
GetConnections()
RPC method. This returns connections for a single layer, as specified by the topology argument.
- Arguments
dataSourceId (integer): data source for which to get the device list
deviceId (string): device id for which to get device info
topology (string): The topology type, as added using
AddConnection()
- Return Value
Returns an XML string that is a valid response to the
GetConnections()
RPC call.
Ian Felstead
Licensed Materials - Property of IBM
"Restricted Materials of IBM"
5724-S45
(C) Copyright IBM Corp. 1997, 2012
IBM Tivoli Network Manager for Optical