Splunk Enterprise Security

Help with creating field extractions for map

vikram1583
Explorer

Can you help map creating field extractions Please use the ES CIM model where possible for field names:

There are some variations in the log files so I included these two that we’re looking at:

2019-09-17 **:**:**.**** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| "AuthenticationLevel":"1","WebUserToken":"354643"}"][TimeTaken:][StatusCode:Created(201)]
2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, "Code":30116267

Below are the fields need to be extracted:

Accept-Language
User-Agent
X-GDC-DeviceID
X-GDC-SessionToken
X-GDC-Method
X-GDC-ApplicationID
X-Forwarded-For
X-Original-URL
AuthenticationLevel
WebUserToken
StatusCode
ErrorCode
ErrorDescription
Code

For X-Forwarded-For, please only capture the first IP address.

0 Karma

jacobpevans
Motivator

Greetings @vikram1583,

Here's a run-anywhere search to extract Accept-Language as AcceptLanguage. Try playing with that to get the rest of the fields. They're all nearly identical.

| makeresults
| eval _raw = "2019-09-17 ::.** [Level: INFO][Server: **********][ServerIP: ::1][ThreadId: 141][RequesterIP:**.***.1.1][Verb:POST][RequestUri:https://***svcv3/authenticationgateway/profile/******/login][Headers:[Connection:Keep-Alive|Content-Length:118|Content-Type:application/json|Accept:*/*|Accept-Language:en-us |User-Agent:iOS **Bank (Version 2.18.0 Build 80329; 12.4.1; en-US; iPhone(iPhone11,2); |X-GDC-DeviceID:BA8BB0C7-2FF8-4C37-B17B-A5F01148D38E|X-GDC-Digest:l2RLaisPFvk6libgtBFQb85Sh17kM5moYGp6ipQ2Su0=|X-GDC-SessionToken:fe9bc5d5-259d-402b-aa35-861e0d260068|X-GDC-Method:2|X-GDC-Timestamp:2019-09-17T22:41:10.009|Originator:FlexClient|X-GDC-Version:1.001|X-GDC-ApplicationID:10043|X-GDC-MessageID:BABBFB13-F781-4FF6-B777-894BAF5CBD8A|RequestId:AEABFB13-F781-4FF6-B777-894BAF5CBD8A|X-Forwarded-For:108.**.233.***, 127.**.242.145, 10.126.**.250|X-Original-URL:/***/auth/**/profile/tokens/login| \"AuthenticationLevel\":\"1\",\"WebUserToken\":\"354643\"}\"][TimeTaken:][StatusCode:Created(201)]"
| append 
    [ | makeresults
      | eval _raw = "2019-09-13 23:**:51.3120 [Level: INFO][Server: *****SVC04][ServerIP: ::1][ThreadId: 58][Response:{ ErrorCode = 10003, ErrorDescription = Unable to process the login request, \"Code\":30116267" ]

| rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"
Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

vikram1583
Explorer

hey thanks for the reply its working and i am poor at regex can you send Rex for remaining fields please?

0 Karma

jacobpevans
Motivator

I'll help you out a little more.

rex "Accept-Language:(?<AcceptLanguage>[^\|]+)"

See here for more info regarding the rex command: https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex. By default, we're performing rex on the full _raw value (the two lines in your question) which is what you want. From there, we have "Accept-Language:(?<AcceptLanguage>[^\|]+)"

The basic structure is "(?<FieldName>[field extraction regex])". In your case, the Accept-Language field always starts with Accept-Language: which is why I put that before the parend. (? to signify that.

[^\|]+ - This is the magic extraction. [ and ] defines multiple possible matches. Inside that, we have ^ which just means NOT. After that, we have \| which is just the | character with the escape character \. I did this because your Accept-Language field ends with | in your sample data.

Add that all up, and we're grabbing everything between "Accept-Language:" and "|" in your sample data.

See here for a nice introduction tutorial: https://medium.com/factory-mind/regex-tutorial-a-simple-cheatsheet-by-examples-649dc1c3f285

See here for a fantastic online regex tester where you can practice using regular expressions (you can even use your data): https://regex101.com. You can test your skills here: https://regex101.com/quiz.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

vikram1583
Explorer

will the same Regex work for indexing operations?

0 Karma

jacobpevans
Motivator

I don't understand the question

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...