Deployment Architecture

How to make a loop including a dataframe for each jobname

nsantiago17
Explorer

I have a dataset like this:

jobname     date                   avg      start             end
A              30/03/2019            84900  03:13:25           20:59:47
B              02/04/2019            60798  16:53:05           16:00:05
C              02/04/2019            60798  16:53:05           16:00:04
D              02/04/2019            79200  22:00:01           15:00:39

The length is 100k, and I need to do a loop including a dataframe for each jobname, for example:

df1 = A, 30/03/2019, 84900, 03:13:25,20:59:47
df2 = B, 02/04/2019,60798,16:53:05,16:00:05

So I can use the kmeans algorithm to cluster all of the data.

How can I do this?

0 Karma

evania
Splunk Employee
Splunk Employee

Hi @nsantiago17 ,

Did you have a chance to check out an answer? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

0 Karma

nsantiago17
Explorer

Hi, I replied Martyn but he didn't answer anymore, so I'm still without a solution or something that can help me to solve the question.

0 Karma

martynoconnor
Communicator

Hi There,

What's the end use case you're trying to achieve here? I have a similar sort of search that I use in a dashboard of my own. The data you have looks fairly simple in structure and so you could index the data, ensuring the fields are correctly extracted and then use a search something like the following:

index=<your_index>  
| kmeans k=<how_many_clusters> dt=<your_distype> <numeric_field_1> <numeric_field_2> cfield="Cluster number"
| stats mode(<field1>) mode(<field2>) by "Cluster number"
| rename mode(<field1>) AS "<Better Name>" mode(<field2>) AS "<Better Name>"

The search above is one I use to cluster patterns in airline traffic, grouping them by heading and altitude.

Hopefully that's enough to get going with - let me know if you have any more questions.

Regards,

Martyn

0 Karma

nsantiago17
Explorer

Hi Martyn,

I need to cluster my dataset in 4 groups: those who have outliers early in the month, 15th, 21th and last day of the month by the jobname, and I have to use the avg, date and one other numerical column.
So I can analyze each jobname already knowing their pattern. I'm trying to separate each jobname with their information and then start to figure how to cluster them. If you have some idea, please share with me.

Regards,

Nick

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...