All Apps and Add-ons

Has anyone successfully extracted Cisco Prime logs with matching field names?

mikev
Path Finder

Thanks in advance.

We just installed Cisco Prime (well Cisco installed it), it is feeding a syslog-ng server. I have tried both sourcetype=syslog and sourcetype=cisco:ios, while I do get more fields extracted via cisco:ios, we are still not getting good field types and / or data extractions.

Has anybody successfully extracted Prime logs with matching field names? I have scoured the forums and while Mikael Bjerkeland has done quite a bit of work with the Cisco Networks Add-on for Splunk Enterprise: https://splunkbase.splunk.com/app/1467 , still not seeing any further items that will help pull in the data. As a side note, SNMP is not an option so it has to be the feed from syslog.

If anyone has common fields and or props / transforms they would like to share that would be great.

Again thanks for any assistance provided,

Mike

vladx
New Member

Another option is to use syslog-ng's extensive re-write capabilities to modify Prime's not-very-standard syslog. Here is my solution for the re-write statement in syslog-ng. It has modified the logs by putting text within the first [] in the program field and also removes extra timestamp causing troubles when processed as syslog message. After this Splunk can process some parts and I'm sure additional fields can be extracted if someone knows what is interesting and what is not.

rewrite r_cisco_prime {
  subst('\d*:\d*:\d*\.\d*.\w*',
  "",
  value("MESSAGE"),
  type(pcre)
  flags(ignore-case));

  subst('\[',
  "",
  value("MESSAGE"),
  type(pcre));

  subst("\w*",
  "",
  value("MESSAGE")
  type(pcre));

  set("${0}",value("PROGRAM"));

  subst('\]',
  "",
  value("PROGRAM")
  type(pcre));

};
0 Karma

mikev
Path Finder

Mikael,
I've included a sample of the various entries. One thing to note, Cisco truncates @ 1024, for those entries they end with ... and begin with ... on the following entry.
Hope that helps. I would be willing to get you a larger sample if it will make more sense.

May 11 03:31:12 192.168.122.8 05/11/17 02:30:05.471 INFO  [jobmanager] [seqtaskexecutor-10208] INFO: [TaskResultUpdateWorker: doPersistJobElement: created result for task with id=11333891].
    May 11 03:31:12 192.168.122.8 05/11/17 02:30:05.496 INFO  [logging] [seqtaskexecutor-10208] IFM-SAM-INFO:  Total number of clients processed for the time: 1494494704409  is processed::0 pushed:0 Removed:0.
    May 11 03:31:25 192.168.122.8 05/11/17 02:30:18.484 ERROR [PollTask] [xmppollerAsync-2] AesLogImpl.error:6598594_10.10.29.19Exception when query PollUnitIFAVAILABILITYDeviceUnreachableException: Device not reachable: 10.10.29.19/161, nested exception=java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
    May 11 03:31:25 192.168.122.8 05/11/17 02:30:18.490 ERROR [PollTask] [xmppollerAsync-2] AesLogImpl.throwing: class: PollUnit, method:pollDevice, Trace: DeviceUnreachableException: Device not reachable: 10.10.29.19/161, nested exception=java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
        at com.cisco.xmp.snmp.mediation.impl.MediationOperation.reportException(MediationOperation.java:133)
        at com.cisco.xmp.snmp.mediation.impl.MediationOperation.onTarget(MediationOperation.java:458)
        at com.cisco.xmp.snmp.mediation.impl.MediationOperation$2.onDone(MediationOperation.java:72)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.notifyListener(AbstractFuture.java:49)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.setCompletionStatus(AbstractFuture.java:72)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.reportException(AbstractFuture.java:88)
        at com.cisco.xmp.snmp.sessionmgr.async.TargetFuture.onDone(Ta...
    May 11 03:31:25 192.168.122.8 ...rgetFuture.java:84)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.notifyListener(AbstractFuture.java:49)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.setCompletionStatus(AbstractFuture.java:72)
        at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.reportException(AbstractFuture.java:88)
        at com.cisco.xmp.snmp.sessionmgr.async.EngineIDFuture.onDone(EngineIDFuture.java:121)
        at com.cisco.xmp.snmp.sessionmgr.async.ResponseFuture.notifyListener(ResponseFuture.java:120)
        at com.cisco.xmp.snmp.sessionmgr.async.ResponseFuture.onResponse(ResponseFuture.java:298)
        at org.snmp4j.Snmp$PendingRequest.run(Snmp.java:1637)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
    Caused by: java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
        at com.cisco.xmp.snmp.sessionmgr.async.EngineIDFuture.onDone(EngineIDFuture.java:119)
        ... 5 more
     : 
    May 11 03:31:25 192.168.122.8 DeviceUnreachableException: Device not reachable: 10.10.29.19/161, nested exception=java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
    May 11 03:31:25 192.168.122.8 DeviceUnreachableException: Device not reachable: 10.10.29.19/161, nested exception=java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.mediation.impl.MediationOperation.reportException(MediationOperation.java:133)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.mediation.impl.MediationOperation.onTarget(MediationOperation.java:458)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.mediation.impl.MediationOperation$2.onDone(MediationOperation.java:72)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.notifyListener(AbstractFuture.java:49)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.setCompletionStatus(AbstractFuture.java:72)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.reportException(AbstractFuture.java:88)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.async.TargetFuture.onDone(TargetFuture.java:84)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.notifyListener(AbstractFuture.java:49)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.setCompletionStatus(AbstractFuture.java:72)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.util.AbstractFuture.reportException(AbstractFuture.java:88)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.async.EngineIDFuture.onDone(EngineIDFuture.java:121)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.async.ResponseFuture.notifyListener(ResponseFuture.java:120)
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.async.ResponseFuture.onResponse(ResponseFuture.java:298)
    May 11 03:31:25 192.168.122.8 at org.snmp4j.Snmp$PendingRequest.run(Snmp.java:1637)
    May 11 03:31:25 192.168.122.8 at java.util.TimerThread.mainLoop(Timer.java:555)
    May 11 03:31:25 192.168.122.8 at java.util.TimerThread.run(Timer.java:505)
    May 11 03:31:25 192.168.122.8 Caused by: java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.19/161
    May 11 03:31:25 192.168.122.8 at com.cisco.xmp.snmp.sessionmgr.async.EngineIDFuture.onDone(EngineIDFuture.java:119)
    May 11 03:31:25 192.168.122.8 ... 5 more
    May 11 03:31:25 192.168.122.8 05/11/17 02:30:18.493 ERROR [PollTask] [xmppollerAsync-2] AesLogImpl.error:6598594_10.10.29.19Process is aborted. No data is being saved.
    May 11 03:31:25 192.168.122.8 05/11/17 02:30:18.493 ERROR [PollTask] [xmppollerAsync-3] AesLogImpl.error:6598593_10.10.29.18Exception when query PollUnitIFAVAILABILITYDeviceUnreachableException: Device not reachable: 10.10.29.18/161, nested exception=java.io.IOException: Cannot discover SNMPv3 engine ID 10.10.29.18/161
    May 11 03:31:37 192.168.122.8 05/11/17 02:30:30.525 INFO  [sam] [CDBPool-DVAVAILABILITY-thread-1] Thread Id : [1,265] : SAM_COLLECTION_INFO: [Truncated table is: DEVICEAVAILABILITY$ERR]
    May 11 03:35:08 192.168.122.8 05/11/17 02:34:01.233 ERROR [logging] [Thread-78] IFM-SAM-ERROR: ApplicationServiceDefinitionLoader - Though file is modified, file version tag value is same as in DB. Not updating taxonomy file content to PI..
    May 11 03:38:04 192.168.122.8 05/11/17 02:36:58.221 INFO  [apic] [pool-40-thread-1] Thread Id : [255] : IFM_APIC_INFO: [ApicPollingService: Polling APIC controller for device ]
    May 11 03:42:07 192.168.122.8 05/11/17 02:41:00.595 INFO  [IpepCacheTQImpl] [ipepTimerFactory] changeMapSize: 0
    May 11 03:45:09 192.168.122.8 05/11/17 02:44:02.861 INFO  [systemmonitoring] [seqtaskexecutor-2558] [1,186,640] Starting  getLocation
0 Karma

mikaelbje
Motivator

My question now is what you want to achieve by ingesting these logs? To me they look pretty much like garbage and they're saying more about the state of Cisco Prime Infrastructure, not your network devices. If you want to leverage the functionality of the Cisco Networks app you need to configure your network devices to send syslog to Splunk or your syslog daemon which in turn forwards the logs to Splunk.

0 Karma

mikev
Path Finder

Mikael,

The logs are sent to syslog-ng then ingested by Splunk, no problem. I was able to get in contact with someone who has access to the Prime environment and will be working with them next week. For the logs, most are status for the systems so I'm going to see if we can get them to narrow what they what to see. I have identified 45 different types of entries (those in the braces - [topology] for example) most of them I have been able to get useful information out of. The ones that are the real issue are the multi-line items.

For instance the entry - ERROR [PollTask] [xmppollerAsync-2] above with this timestamp - May 11 03:31:25 192.168.122.8 05/11/17 02:30:18.490 is different than the same error entry above it, and I believe that the following entries below it are all associated with this record as the timestamp is not complete - so supplied by syslog-ng ( May 11 03:31:25 192.168.122.8) although it does not follow the prime convention of end... begin... for those records over 1024 bytes of which most of them are not.

I know the customer will want anything with error, high, critical in it so what am I looking for? The ability to have this hodgepodge of "crap" broken down into something useful for at least the items containing issues.

sdemoss mentioned the Cisco Prime API, I will have the customer look into that as well, in the mean time I'm still trying to figure the best way to get all of the multi-line items joined, I am up for suggestions. I can supply the community with larger sample files it it helps.

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

This is a continuation of the log snippet above (this was originally an answer but it too long to be converted to a comment so I had to separate it).

May 11 04:12:08 192.168.122.8 05/11/17 03:11:03.180 INFO  [aesReports] [Task Scheduler Worker-4] insertByGroupString is: insert into AHETHERNETSWITCHINTFINFO(id,instance_version,owningentityid, ifindex, interfacename,avgportpoeutilization,maxportpoeutilization,minportpoeutilization,maxhealthscore,avgifindiscards,maxifindiscards,minifindiscards,avgifhcinoctets,maxifhcinoctets,minifhcinoctets,avgifhcoutoctets,maxifhcoutoctets,minifhcoutoctets,avgoutpututilization,maxoutpututilization,minoutpututilization,avgifouterrors,maxifouterrors,minifouterro...
    May 11 04:12:08 192.168.122.8 ...rs,avgifinerrors,maxifinerrors,minifinerrors,avginpututilization,maxinpututilization,mininpututilization,avgifoutdiscards,maxifoutdiscards,minifoutdiscards,minthreshold,totalcount,eventtime) select max(id) id,0 instance_version,owningentityid, ifindex, interfacename,avg(portpoeutilization) avgportpoeutilization,max(portpoeutilization) maxportpoeutilization,min(portpoeutilization) minportpoeutilization,max(healthscore) maxhealthscore,avg(ifindiscards) avgifindiscards,max(ifindiscards) maxifindiscards,min(ifindisc...
    May 11 04:12:08 192.168.122.8 ...ards) minifindiscards,avg(ifhcinoctets) avgifhcinoctets,max(ifhcinoctets) maxifhcinoctets,min(ifhcinoctets) minifhcinoctets,avg(ifhcoutoctets) avgifhcoutoctets,max(ifhcoutoctets) maxifhcoutoctets,min(ifhcoutoctets) minifhcoutoctets,avg(outpututilization) avgoutpututilization,max(outpututilization) maxoutpututilization,min(outpututilization) minoutpututilization,avg(ifouterrors) avgifouterrors,max(ifouterrors) maxifouterrors,min(ifouterrors) minifouterrors,avg(ifinerrors) avgifinerrors,max(ifinerrors) maxifinerro...
    May 11 04:12:08 192.168.122.8 ...rs,min(ifinerrors) minifinerrors,avg(inpututilization) avginpututilization,max(inpututilization) maxinpututilization,min(inpututilization) mininpututilization,avg(ifoutdiscards) avgifoutdiscards,max(ifoutdiscards) maxifoutdiscards,min(ifoutdiscards) minifoutdiscards,min(threshold) minthreshold,sum(1) totalcount,1494496799999 eventtime from ETHERNETSWITCHINTFINFO where owningentityid = '6598594_10.10.29.19' and eventtime >= 1494493200000 and eventtime < 1494496800000 group by owningentityid, ifindex, interfacename 
    May 11 04:12:09 192.168.122.8 05/11/17 03:11:03.934 INFO  [aesReports] [Task Scheduler Worker-4] insertByGroupString is: insert into AHDEVICEAVAILABILITY(id,instance_version,owningentityid,maxhealthscore,avgoperationalstatus,minthreshold,totalcount,eventtime) select max(id) id,0
    instance_version,owningentityid,max(healthscore) maxhealthscore,avg(operationalstatus) avgoperationalstatus,min(threshold) minthreshold,sum(1) totalcount,1494496799999 eventtime from DEVICEAVAILABILITY where owningentityid = '6598594_10.10.29.19' and eventtime >= 1494493200000 and eventtime < 1494496800000 group by owningentityid 
    May  4 12:03:44 192.168.122.8 05/04/17 10:58:41.554 ERROR [topology] [pool-7-thread-1] For alarm WiredWirelessAlarm[log=<Log4jLoggerAdapter>,causingAlarmId=0,srcObjectBusinessKey=8e57ebcd[00:6b:f1:1b:88:10],generatedBy=Controller,alarmCreationTime=2017-04-05 14:46:11.809,applicationSpecificAlarmID=UnifiedAp!00:6b:f1:1b:88:10,eventCount=1,isAcknowledged=false,lastModifiedTimestamp=2017-04-05 14:46:11.809,mayBeAutoCleared=false,previousSeverity=CRITICAL,subclassName=WiredWirelessAlarm,alarmDisplayable=true,applicationCategoryData=AP_DISASSOCIATED,category=AP(1),description=AP 'MAP' disassocia...
    May  4 12:03:44 192.168.122.8 ...ted.,displayName=MAP,00:6b:f1:1b:88:10,eventType=AP_DISASSOCIATED,isDeviceMaster=false,notificationDeliveryMechanism=SNMP_TRAP,notificationState=0,severity=CRITICAL,source=UnifiedAp!00:6b:f1:1b:88:10,sourceMacAddress=00:6b:f1:1b:88:10,srcObjectClassId=-1906840627,srcObjectId=6602604,transientNameValue={},instanceUuid=c40be018-792b-4ce3-b3f5-1cd7158a91bf,instanceId=8237230,authEntityId=6602604,authEntityClass=-1906840627,_orderedListOEIndex=<Integer>,_creationOrderIndex=<Integer>,_isBeingChanged=<Boolean>,instanceVersion=0] getReportingEntityAddress() is a null
0 Karma

shartwell
Explorer

mikev -

I used these commands in search against a single event. The first takes everything after INFO (like you had below) and the second removes beginning / end brackets. It needed a pipe (or) to get rid of all brackets - attempts w/out the pipe only got beginning or ending brackets, but not both.

| makeresults
| eval message="May 1 12:00:00 192.168.1.1 05/01/17 00:01:00.100 INFO [scott] [execute-1] ProgramA Status complete code 2"
| rex field=message "INFO\s+(?<message2>.*)$"
| rex mode=sed field=message2 "s/\[|\]//g"

I think it might be worth using SEDCMD in props.conf to clean it up a bit to make extraction easier?
Something like below to remove the brackets, then use REPORT for extractions.

[cisco:prime]
SEDCMD-remove-brackets = s/\[|\]//g
REPORT-blah-blah =

Hope this helps.

0 Karma

sdemoss
Explorer

mikev,

I completed a fairly large project aimed at onboarding Cisco Prime data into Splunk. There are a few options we discovered (some a lot better than others) and a few lessons we learned along the way, mostly related to the nature of the data that Cisco Prime sends out of the system. For example, one of the syslog-style feeds (maybe the only one) is for this normalized data type that Prime maintains called "Events" which as near as I could tell, were a combination of Prime alarms/alerts, regular syslog messages, and certain SNMP traps.

The focus of our project was mostly related to Cisco Wireless telementry to instrument performance, availability, fault tolerance, and end user experience/activity. In the end, we leveraged the Cisco Prime API, which I strongly encourage you to do as well (for at least part of your solution). The API allows quite a bit more flexibility and control on what you retrieve in addition to allowing for the opportunity to transform the output using a Splunk scripted input prior to indexing. This can be good for search optimization and/or controlling license consumption. The data format comes back as XML by default, but you can optionally request results in JSON.

This link is for the Cisco Prime API reference doc - just use the version that matches your Prime installation.
https://developer.cisco.com/site/prime-infrastructure/documents/api-reference/rest-api-v3-1/#

seegeekrun
Path Finder

@sdemoss - With the approach you took for Cisco Wireless telemetry, was it necessary to make one call to get a list of devices and then N number of calls per device returned where N is the number if metrics?

We're in the process of doing something that sounds similar and I'm sizing up the effort, but reading through the API documentation makes me think that I'm going to need thousands of calls to gather the metrics.

0 Karma

sdemoss
Explorer

The answer is, it kind of depends. In our experience, it mostly was making a single API call to the reporting endpoint of interest as most of these have an 'entity detail' view where it will give you one result per entity with all of its respective details. For example, "AP Details" (/webacs/api/v1/data/AccessPointDetails) returns one result per endpoint with a TON of details about the specific Access Point.

I would encourage you to simply login to Prime and enter some of the REST URL's in a browser. You can specify either XML or JSON format for results and it lets you quickly and easily check what data gets returned. We ended up doing some "pre-processing" of the results before indexing it into Splunk to optimize license consumption.

0 Karma

mikev
Path Finder

Thanks!, not my call on the API, I don't have access to the device and limited access to the folks that support it. I'll keep it in mind.

0 Karma

mikaelbje
Motivator

A few sample log entries would help 😉

0 Karma

mikev
Path Finder

Mikael,

If I can contact you out of band, I can send you a large file as it is all over, some of it WLC, some of it network, I think a larger sample would be best for you.

0 Karma

mikaelbje
Motivator

I only need to see about 15 log lines, not the whole file. Please post them inline using the code formatter to keep style.

0 Karma

mikev
Path Finder

Mikael,
I've included a sample of the various entries. One thing to note, Cisco truncates @ 1024, for those entries they end with ... and begin with ... on the following entry. Hope that helps. I would be willing to get you a larger sample if it will make more sense.

0 Karma

mikev
Path Finder

For some reason it won't let me submit when I use the "Code Sample" button. I also tried to paste directly into the window and it won't let me submit that way, both say I have characters left just the submit button goes grey. Other suggestions?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...