Getting Data In

Json huge data - issue with breaking the individual events

Nadhiya_Dubai
Explorer
"Global Users":[
    {
         "AP name":"T2-GF-WDN-ISP-079", 
         "Auth":null, 
         "Bssid":"94:b4:0f:04:51:f1", 
         "Current switch":"172.30.97.41", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.11.0.23", 
         "MAC":"68:e7:c2:5d:a1:ad", 
         "Name":null, 
         "Phy":"a-HT", 
         "Profile":"FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD07-WDN-OSP-109", 
         "Auth":null, 
         "Bssid":"40:e3:d6:23:3b:21", 
         "Current switch":"172.30.97.111", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.234.0.213", 
         "MAC":"fc:aa:b6:17:1a:a3", 
         "Name":null, 
         "Phy":"g-HT", 
         "Profile":"T3-FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD12-WDN-ISP-020", 
         "Auth":"802.1x", 
         "Bssid":"b4:5d:50:f8:57:e2", 
         "Current switch":"172.30.97.112", 
         "Essid":"tenantauth", 
         "IP":"10.235.197.85", 
         "MAC":"d4:e6:b7:94:39:95", 
         "Name":"torydxb@tenant", 
         "Phy":"g-HT", 
         "Profile":"TENANTAUTH-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"TENANTAUTH-user-Role", 
         "Type":"Android", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"CB-GF-FD07-WDN-OSP-050", 
         "Auth":"802.1x", 
         "Bssid":"20:a6:cd:30:9a:22", 
         "Current switch":"172.30.97.112", 
         "Essid":"ahlan", 
         "IP":"10.211.2.144", 
         "MAC":"48:9d:d1:6d:8d:e9", 
         "Name":"GNSCDWC02", 
         "Phy":"g-HT", 
         "Profile":"T3-CB-Ahlan-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Ahlan-User-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    }
],
"_data":[
    "Total entries = 14995" 
],
"_meta":[
    "IP", 
    "MAC", 
    "Name", 
    "Current switch", 
    "Role", 
    "Auth", 
    "AP name", 
    "Roaming", 
    "Essid", 
    "Bssid", 
    "Phy", 
    "Profile", 
    "Type", 
    "User Type" 
]

}"

Above is my json data . Well i have trimmmed the events ,its so huge lines in millions for a single event .
I tried giving the sourcetype as _json but its not breaking my events .Kindly help .I always have trouble while the data is in json format . Looking for the right solution and the explanation . Kindly help

Tags (1)
0 Karma

FrankVl
Ultra Champion

Maybe start with explaining what the desired behavior would be, because that is not very clear from your question. Do you want each { "AP name"... } section in a separate event?

In general, I would concur with the answer from @starcher that this looks like something you want to pre-process and then send into splunk as individual events, rather than massive json structs.

0 Karma

to4kawa
Ultra Champion

"Total entries = 14995"

props.conf
LINE_BREAKER in single line printed JSON doc

I hope this can be done well.

0 Karma

to4kawa
Ultra Champion
| makeresults 
 | eval _raw="{\"Global Users\":[{\"AP name\":\"T2-GF-WDN-ISP-079\",\"Auth\":null,\"Bssid\":\"94:b4:0f:04:51:f1\",\"Current switch\":\"172.30.97.41\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.11.0.23\",\"MAC\":\"68:e7:c2:5d:a1:ad\",\"Name\":null,\"Phy\":\"a-HT\",\"Profile\":\"FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD07-WDN-OSP-109\",\"Auth\":null,\"Bssid\":\"40:e3:d6:23:3b:21\",\"Current switch\":\"172.30.97.111\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.234.0.213\",\"MAC\":\"fc:aa:b6:17:1a:a3\",\"Name\":null,\"Phy\":\"g-HT\",\"Profile\":\"T3-FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD12-WDN-ISP-020\",\"Auth\":\"802.1x\",\"Bssid\":\"b4:5d:50:f8:57:e2\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"tenantauth\",\"IP\":\"10.235.197.85\",\"MAC\":\"d4:e6:b7:94:39:95\",\"Name\":\"torydxb@tenant\",\"Phy\":\"g-HT\",\"Profile\":\"TENANTAUTH-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"TENANTAUTH-user-Role\",\"Type\":\"Android\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"CB-GF-FD07-WDN-OSP-050\",\"Auth\":\"802.1x\",\"Bssid\":\"20:a6:cd:30:9a:22\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"ahlan\",\"IP\":\"10.211.2.144\",\"MAC\":\"48:9d:d1:6d:8d:e9\",\"Name\":\"GNSCDWC02\",\"Phy\":\"g-HT\",\"Profile\":\"T3-CB-Ahlan-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Ahlan-User-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"}],\"_data\":[\"Total entries = 14995\"],\"_meta\":[\"IP\",\"MAC\",\"Name\",\"Current switch\",\"Role\",\"Auth\",\"AP name\",\"Roaming\",\"Essid\",\"Bssid\",\"Phy\",\"Profile\",\"Type\",\"User Type\"]}"
 | spath  
 | fields - _*
 | rename "Global Users"{}.* as *
 | rename data{} as _data, meta{} as _meta
 | mvexpand IP
 | rename IP as _IP
 | streamstats count
 | foreach *
    [eval <<FIELD>> = mvindex('<<FIELD>>', count - 1)]
| rename _IP as IP, _data as data, _meta as meta
| table IP MAC Name "Current switch" Role Auth  "AP name" Roaming Essid Bssid Phy Profile Type "User Type" data

At this level, you can normally spath .

0 Karma

starcher
SplunkTrust
SplunkTrust

If JSON is that big you should use code outside of Splunk to parse it into reasonable events and send those in. Also when sending in properly formed JSON use kv_mode = JSON on your sourcetype definition in props.

bowesmana
SplunkTrust
SplunkTrust

JSON auto extraction will only extract, I believe, the first 5000 bytes. You need to use spath on the elements of the data you need.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...