All Apps and Add-ons

How to generate a search for my mainframe SMF data for CPU Utilization?

szimmer661
Explorer

We are forwarding our mainframe statistical data from a purchased ISV product into Splunk.

The below information deals with our CPU utilization.

The values associated with the SMF70CIN_xxxx names are our different engine types.

The values associated with the SMF70PDT_nnnn names are processor dispatch times, cut at various times in the time interval.

The values associated with the SMF70CIX_nnnn names are index values to the engine type.

For instance, SMF70PDT entries 0001 thru 0004 refer to the SMF70CIX entries 0001 thru 0004 and point to SMF70CIN_0001, which indicates they are processor dispatch times for the CPs, our general purpose processors.

Likewise, SMF70PDT_0005 and 0006 refer to SMF70CIX_0005 and 0006 and indicate the entries are for the IIPs, our zIIP specialty engines.

I need to total the SMF70PDT entries for each engine type, labeling them with the values in the SMF70CIN_xxxx name/value pairs.

I've been searching the Splunk Answers for a solution, but likely don't know what keywords to use to properly describe my problem.

I hope my explanation and question make sense. Any help is greatly appreciated, even a "can't be done" answer.

 SMF70CIN_0001:  CP 
 SMF70CIN_0002: 
 SMF70CIN_0003:  IFA    
 SMF70CIN_0004:  IFL    
 SMF70CIN_0005:  ICF    
 SMF70CIN_0006:  IIP    

 SMF70CIX_0001:  0001   SMF70PDT_0001:   89783976  
 SMF70CIX_0002:  0001   SMF70PDT_0002:   1908676
 SMF70CIX_0003:  0001   SMF70PDT_0003:   65
 SMF70CIX_0004:  0001   SMF70PDT_0004:   40340832
 SMF70CIX_0005:  0006   SMF70PDT_0005:   2462040
 SMF70CIX_0006:  0006   SMF70PDT_0006:   23775
 SMF70CIX_0007:  0001   SMF70PDT_0007:   696702146
 SMF70CIX_0008:  0001   SMF70PDT_0008:   708156256
 SMF70CIX_0009:  0001   SMF70PDT_0009:   568186607
 SMF70CIX_0010:  0001   SMF70PDT_0010:   488755783
 SMF70CIX_0011:  0006   SMF70PDT_0011:   159362721
 SMF70CIX_0012:  0006   SMF70PDT_0012:   86652638
 SMF70CIX_0013:  0005   SMF70PDT_0013:   534089
 SMF70CIX_0014:  0001   SMF70PDT_0014:   74161460
 SMF70CIX_0015:  0001   SMF70PDT_0015:   141
 SMF70CIX_0016:  0001   SMF70PDT_0016:   85
 SMF70CIX_0017:  0001   SMF70PDT_0017:   19617663
 SMF70CIX_0018:  0006   SMF70PDT_0018:   6046543
 SMF70CIX_0019:  0006   SMF70PDT_0019:   27434
 SMF70CIX_0020:  0005   SMF70PDT_0020:   1410129
 SMF70CIX_0021:  0004   SMF70PDT_0021:   545731359
 SMF70CIX_0022:  0004   SMF70PDT_0022:   565817697
 SMF70CIX_0023:  0004   SMF70PDT_0023:   591954605
 SMF70CIX_0024:  0004   SMF70PDT_0024:   378145953
 SMF70CIX_0025:  0004   SMF70PDT_0025:   102482037
 SMF70CIX_0026:  0004   SMF70PDT_0026:   102639379
 SMF70CIX_0027:  0004   SMF70PDT_0027:   2659389
 SMF70CIX_0028:  0004   SMF70PDT_0028:   91
 SMF70CIX_0029:  0001   SMF70PDT_0029:   35572965
 SMF70CIX_0030:  0001   SMF70PDT_0030:   224
 SMF70CIX_0031:  0001   SMF70PDT_0031:   99
 SMF70CIX_0032:  0001   SMF70PDT_0032:   5273212
 SMF70CIX_0033:  0006   SMF70PDT_0033:   3508159
 SMF70CIX_0034:  0006   SMF70PDT_0034:   446519
 SMF70CIX_0035:  0001   SMF70PDT_0035:   2458490
 SMF70CIX_0036:  0001   SMF70PDT_0036:   1859409
 SMF70CIX_0037:  0001   SMF70PDT_0037:   2688390
 SMF70CIX_0038:  0001   SMF70PDT_0038:   5641697
 SMF70CIX_0039:  0005   SMF70PDT_0039:   3574955
 SMF70CIX_0040:  0004   SMF70PDT_0040:   2802845
 SMF70CIX_0041:  0004   SMF70PDT_0041:   2958058
 SMF70CIX_0042:  0004   SMF70PDT_0042:   6064567
 SMF70CIX_0043:  0004   SMF70PDT_0043:   5966167
 SMF70CIX_0044:  0006   SMF70PDT_0044:   543975
 SMF70CIX_0045:  0006   SMF70PDT_0045:   525916 
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Assuming all these events have same timestamp as they are generated at the same 15 min interval, give this a try

your base search | bucket span=15m _time
| rex "SMF70CIN_(?<EngineTypeIndex>\d+)\:\s*(?<EngineType>\w+)"
| rex "SMF70PDT_\d+\:\s*(?<ProcDispatchTime>\d+)"
| rex "SMF70CIX_\d+\:\s*(?<EngineTypeIndex_temp>\d+)"
| eval EngineTypeIndex=coalesce(EngineTypeIndex,EngineTypeIndex_temp)
| stats values(EngineType) as EngineType sum(ProcDispatchTime) as ProcDispatchTimetotal by _time EngineTypeIndex
| timechart sum(ProcDispatchTimetotal) by EngineType
0 Karma

szimmer661
Explorer

It tells me the base search returns a number of events, but then yields no results. Is there a trace facility in Splunk or some way to display intermediate results at stages in the search?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Run the query part by part. and see what caused it to fail. Start with line 1 and keep adding each line. Check if the extracted fields have values before stats.

your base search | table _time _raw | bucket span=15m _time
 | rex "SMF70CIN_(?<EngineTypeIndex>\d+)\:\s*(?<EngineType>\w+)"
 | rex "SMF70PDT_\d+\:\s*(?<ProcDispatchTime>\d+)"
 | rex "SMF70CIX_\d+\:\s*(?<EngineTypeIndex_temp>\d+)"
 | eval EngineTypeIndex=coalesce(EngineTypeIndex,EngineTypeIndex_temp)
 | stats values(EngineType) as EngineType sum(ProcDispatchTime) as ProcDispatchTimetotal by _time EngineTypeIndex
 | timechart sum(ProcDispatchTimetotal) by EngineType
0 Karma

szimmer661
Explorer

For a 60 minute time period, running the search:

index=ironstream MFSOURCETYPE=SMF070 SMF70STY=1 SMF70SID=TXFA
| table _time _raw

gave the expected 4 lines with _time and the raw data for the SMF70 records.

Changing the search to:

index=ironstream MFSOURCETYPE=SMF070 SMF70STY=1 SMF70SID=TXFA
| bucket span=15m time
| rex "SMF70CIN
(?\d+):\s*(?\w+)"
| table _time EngineTypeIndex EngineType

gave 4 lines with the _time entry, but nothing under EngineTypeIndex or EngineType column headers.

Did I do this as you'd envisioned?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

If you're getting 4 records for last 60 mins mean the data you posted in question is from single event, and not separate event. Let me see if I can modify the query to accommodate that.

0 Karma

szimmer661
Explorer

I'm confused. The Events Tab in Verbose Mode shows a counter of (4) and there are four records with different time stamps shown. What am I missing or not understanding properly?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You've multiline events (each SMF70CIN_* and other entries are appearing in separate lines. Each line is not an event and that's not what my answer was based on. Give this a try

your base search | bucket span=15m _time
| rex max_match=0 field=_raw "(?<temp>[^\n]+)"
| table _time temp | mvexpand temp | rename temp as _raw
 | rex "SMF70CIN_(?<EngineTypeIndex>\d+)\:\s*(?<EngineType>\w+)"
 | rex "SMF70PDT_\d+\:\s*(?<ProcDispatchTime>\d+)"
 | rex "SMF70CIX_\d+\:\s*(?<EngineTypeIndex_temp>\d+)"
 | eval EngineTypeIndex=coalesce(EngineTypeIndex,EngineTypeIndex_temp)
 | stats values(EngineType) as EngineType sum(ProcDispatchTime) as ProcDispatchTimetotal by _time EngineTypeIndex
 | timechart sum(ProcDispatchTimetotal) by EngineType
0 Karma

szimmer661
Explorer

The events tab shows a count of (4) for the 60 minute period, but still no results found.

I very much appreciate the help on this. I'm going to have to look in the manuals to figure out what you're doing with the first rex and table lines and the eval coalesce line. Up to now, the Splunk searches I've done with the mainframe data have been with less complicated records and have been more straightforward and simple.

0 Karma

szimmer661
Explorer

I don't know what happened above, the changed search should have been:

index=ironstream MFSOURCETYPE=SMF070 SMF70STY=1 SMF70SID=TXFA
| bucket span=15m time
| rex "SMF70CIN
(?\d+):\s*(?\w+)"
| table _time EngineTypeIndex EngineType

0 Karma

szimmer661
Explorer

When I submit the comment, something is removing the EngineTypeIndex and EngineType in the rex statement.

0 Karma

szimmer661
Explorer

The same type results , nothing under the column ProcDispatchTime, are produced with the search:

index=ironstream MFSOURCETYPE=SMF070 SMF70STY=1 SMF70SID=TXFA
| bucket span=15m time
| rex "SMF70PDT
\d+:\s*(?\d+)"
| table _time ProcDispatchTime

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You need to format the search query by clicking button 101010 or pressing Ctrl + k after selecting the query.

0 Karma

woodcock
Esteemed Legend

For the data above, what is the desired output? Is there any need to consider time or other sets of events or will every search return events that should be considered as 1 set (not multiple sets)?

0 Karma

szimmer661
Explorer

Thank you for the quick response.

We'll want to sum the processor dispatch time for each interval record created by engine type.

TimeStamp CP IFL ICF IIP (We don't have IFA engines)

Under each engine type and each time interval will be the sum of the SMF70PDT entries for that engine type. With there being 96 15-minute intervals per day, I'd expect a report with 96 entries per day. There will be further eval statements to compute CPU percentage based on the processor dispatch times, but for now I'd like to get past this giant hurdle (in my view) and see if I can figure out the rest.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

All the data that you've show is single event or each line is an event in Splunk? Is SMF70CIN_xxxx entries always same (for every 15 mins feed that you get)?

0 Karma

szimmer661
Explorer

I copied the entries from Splunk for one time interval record. Currently, we receive the same multiple lines, as shown, for each interval record forwarded to Splunk. I don't know what will happen to the number of entries (the 45) if we upgrade the mainframe to a different box or upgrade our operating system. For right now, the SMF70CIN_xxxx entries are the same and static, as these are the engine types IBM has available on their SystemZ boxes.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Is it a single event or multiple event? E.g. if your run your search and do | stats count do you get 1 or more than 1?

0 Karma

szimmer661
Explorer

It comes up greater than 1.

0 Karma

woodcock
Esteemed Legend

We need to know where the event boundaries are before we can begin with a solution. Your events do not look cut/pasted from the search results area.

0 Karma

szimmer661
Explorer

They are cut/pasted from the Events tab in Verbose Mode, if that helps.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...