Splunk Search

Create new field for incoming logs

swdowiarz
Path Finder

Hi

I have some issue with creating field for my logs. I have logs which contain number. I wan extract this number as session_id. First I've indexed data and then I've used "Extract new field", but new logs that are coming don't have this field ? Should I create this field before I extract this data ?

I would be grateful for some advice.

0 Karma
1 Solution

swdowiarz
Path Finder

Hi

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

View solution in original post

0 Karma

swdowiarz
Path Finder

Hi

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

0 Karma

micahkemp
Champion

Thanks for posting your answer, so others can see how you accomplished this. You should also accept the answer, so the question looks closed to others.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi swdowiarz,
at first verify you regex,

(?ms)\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+\s\w+\s+(?<session_io>\d+)

you can test it at https://regex101.com/r/ouA97r/1

After you can think to extract you fields at index time (see http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction )
Bye.
Giuseppe

0 Karma

nikita_p
Contributor

Hi @swdowiarz,
You can also create custom field in index time in transforms.conf.
You can check below splunk docs for this.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Configureindex-timefieldextraction

0 Karma

swdowiarz
Path Finder

Hi @nikita_p

I was thinking about it, but I've heard that is not good idea, that's why I'm asking about it.

0 Karma

nikita_p
Contributor

@swdowiarz, you can try it will help you.

0 Karma

Kwip
Contributor

Did your extraction matched all the session_id pattern you are looking for?

Could you post some sample logs to see the format of your session_ID?

0 Karma

kunalmao
Communicator

when you use automatic field extractor there are multiple example of logs, try scrolling down to select the one which you need, else its always easier to use rex command. Please paste the sample log i can help you with the regex required for the rex command.

0 Karma

swdowiarz
Path Finder

so I can use rex command in search instead of extracting new field ? That's good point !

0 Karma

swdowiarz
Path Finder
12/27/17
8:24:49.599 AM  
1514363089599   Groupon   I will do my best to help you, please tell me more about the issue you are having.   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   user   i need help   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   Hello! Thank you for contacting Groupon   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

These are examples of my logs and as you can see there is a number the beginning, so I want this number to be my session_id field.

0 Karma

nickhills
Ultra Champion

I would think:
(?<session_id>\d+)\s+\w\w\w
would be a good place to start for your regex. Please give that a spin, and let me know how it goes.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...