Hi,
I have some logs that look like the sample below. If I use .net or java or SQL, I can solve it, but I really don't know how to figure out how to get the final results in Splunk.....
Deploy.joblist=A,B,C,D,E,F
Deploy.job.A.start=true
Deploy.job.E.start=true
Deploy.job.B.start=true
Deploy.job.A.status=Completed
Deploy.job.C.start=true
Deploy.job.B.status=Failed
Expected Result
Job. Status
A Completed
B Failed
C In Progress
D Not found
E In Progress
F Not found
Does your data in Splunk look like the dummy data generated by this?
| stats count | eval data = "Deploy.joblist=A,B,C,D,E,F;Deploy.job.A.start=true;Deploy.job.E.start=true;Deploy.job.B.start=true;Deploy.job.A.status=Completed;Deploy.job.C.start=true;Deploy.job.B.status=Failed" | makemv data delim=";" | mvexpand data | streamstats count | eval _time = now() + count | rename data as _raw | table _time _raw
_time _raw
2014-09-18 15:35:20 Deploy.joblist=A,B,C,D,E,F
2014-09-18 15:35:21 Deploy.job.A.start=true
2014-09-18 15:35:22 Deploy.job.E.start=true
2014-09-18 15:35:23 Deploy.job.B.start=true
2014-09-18 15:35:24 Deploy.job.A.status=Completed
2014-09-18 15:35:25 Deploy.job.C.start=true
2014-09-18 15:35:26 Deploy.job.B.status=Failed
If so, you can append this to calculate the result table you had in mind in the question:
... | rex "Deploy\\.joblist=(?<job>[\w,]+)" | rex "Deploy\\.job\\.(?<job>\w+)\.(?:status|start)=(?<status>\w+)" | replace true with "In Progress" in status | eval status = coalesce(status, "Not Found") | makemv job delim="," | stats latest(status) by job
job latest(status)
A Completed
B Failed
C In Progress
D Not Found
E In Progress
F Not Found
The two rex
es extract the job
and status
fields. The replace
beautifies the "true"
to read "In Progress"
instead. The eval
sets up the fallback "Not Found"
for all jobs listed in the first event. The makemv
splits the list of jobs into a multivalue field. The stats
computes the most recent status for each job according to _time
.
Does your data in Splunk look like the dummy data generated by this?
| stats count | eval data = "Deploy.joblist=A,B,C,D,E,F;Deploy.job.A.start=true;Deploy.job.E.start=true;Deploy.job.B.start=true;Deploy.job.A.status=Completed;Deploy.job.C.start=true;Deploy.job.B.status=Failed" | makemv data delim=";" | mvexpand data | streamstats count | eval _time = now() + count | rename data as _raw | table _time _raw
_time _raw
2014-09-18 15:35:20 Deploy.joblist=A,B,C,D,E,F
2014-09-18 15:35:21 Deploy.job.A.start=true
2014-09-18 15:35:22 Deploy.job.E.start=true
2014-09-18 15:35:23 Deploy.job.B.start=true
2014-09-18 15:35:24 Deploy.job.A.status=Completed
2014-09-18 15:35:25 Deploy.job.C.start=true
2014-09-18 15:35:26 Deploy.job.B.status=Failed
If so, you can append this to calculate the result table you had in mind in the question:
... | rex "Deploy\\.joblist=(?<job>[\w,]+)" | rex "Deploy\\.job\\.(?<job>\w+)\.(?:status|start)=(?<status>\w+)" | replace true with "In Progress" in status | eval status = coalesce(status, "Not Found") | makemv job delim="," | stats latest(status) by job
job latest(status)
A Completed
B Failed
C In Progress
D Not Found
E In Progress
F Not Found
The two rex
es extract the job
and status
fields. The replace
beautifies the "true"
to read "In Progress"
instead. The eval
sets up the fallback "Not Found"
for all jobs listed in the first event. The makemv
splits the list of jobs into a multivalue field. The stats
computes the most recent status for each job according to _time
.
Use can use rex if the data has been indexed already.
rex :
job.(?P<job>\w+)\.status=(?P<status>\S+)
You have to understand splunk is advanced in handle and present the clean and format log, but not easy and have a limitation for developer to do that.
For best practice
1. Make the log event clearly
2. If the log format can't be restructured, better to write the converter to handle.
3. Or like you said, create the web page using your known programming language to do.