Getting Data In

Json huge data - issue with breaking the individual events

Nadhiya_Dubai
Explorer
"Global Users":[
    {
         "AP name":"T2-GF-WDN-ISP-079", 
         "Auth":null, 
         "Bssid":"94:b4:0f:04:51:f1", 
         "Current switch":"172.30.97.41", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.11.0.23", 
         "MAC":"68:e7:c2:5d:a1:ad", 
         "Name":null, 
         "Phy":"a-HT", 
         "Profile":"FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD07-WDN-OSP-109", 
         "Auth":null, 
         "Bssid":"40:e3:d6:23:3b:21", 
         "Current switch":"172.30.97.111", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.234.0.213", 
         "MAC":"fc:aa:b6:17:1a:a3", 
         "Name":null, 
         "Phy":"g-HT", 
         "Profile":"T3-FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD12-WDN-ISP-020", 
         "Auth":"802.1x", 
         "Bssid":"b4:5d:50:f8:57:e2", 
         "Current switch":"172.30.97.112", 
         "Essid":"tenantauth", 
         "IP":"10.235.197.85", 
         "MAC":"d4:e6:b7:94:39:95", 
         "Name":"torydxb@tenant", 
         "Phy":"g-HT", 
         "Profile":"TENANTAUTH-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"TENANTAUTH-user-Role", 
         "Type":"Android", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"CB-GF-FD07-WDN-OSP-050", 
         "Auth":"802.1x", 
         "Bssid":"20:a6:cd:30:9a:22", 
         "Current switch":"172.30.97.112", 
         "Essid":"ahlan", 
         "IP":"10.211.2.144", 
         "MAC":"48:9d:d1:6d:8d:e9", 
         "Name":"GNSCDWC02", 
         "Phy":"g-HT", 
         "Profile":"T3-CB-Ahlan-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Ahlan-User-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    }
],
"_data":[
    "Total entries = 14995" 
],
"_meta":[
    "IP", 
    "MAC", 
    "Name", 
    "Current switch", 
    "Role", 
    "Auth", 
    "AP name", 
    "Roaming", 
    "Essid", 
    "Bssid", 
    "Phy", 
    "Profile", 
    "Type", 
    "User Type" 
]

}"

Above is my json data . Well i have trimmmed the events ,its so huge lines in millions for a single event .
I tried giving the sourcetype as _json but its not breaking my events .Kindly help .I always have trouble while the data is in json format . Looking for the right solution and the explanation . Kindly help

Tags (1)
0 Karma

FrankVl
Ultra Champion

Maybe start with explaining what the desired behavior would be, because that is not very clear from your question. Do you want each { "AP name"... } section in a separate event?

In general, I would concur with the answer from @starcher that this looks like something you want to pre-process and then send into splunk as individual events, rather than massive json structs.

0 Karma

to4kawa
Ultra Champion

"Total entries = 14995"

props.conf
LINE_BREAKER in single line printed JSON doc

I hope this can be done well.

0 Karma

to4kawa
Ultra Champion
| makeresults 
 | eval _raw="{\"Global Users\":[{\"AP name\":\"T2-GF-WDN-ISP-079\",\"Auth\":null,\"Bssid\":\"94:b4:0f:04:51:f1\",\"Current switch\":\"172.30.97.41\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.11.0.23\",\"MAC\":\"68:e7:c2:5d:a1:ad\",\"Name\":null,\"Phy\":\"a-HT\",\"Profile\":\"FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD07-WDN-OSP-109\",\"Auth\":null,\"Bssid\":\"40:e3:d6:23:3b:21\",\"Current switch\":\"172.30.97.111\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.234.0.213\",\"MAC\":\"fc:aa:b6:17:1a:a3\",\"Name\":null,\"Phy\":\"g-HT\",\"Profile\":\"T3-FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD12-WDN-ISP-020\",\"Auth\":\"802.1x\",\"Bssid\":\"b4:5d:50:f8:57:e2\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"tenantauth\",\"IP\":\"10.235.197.85\",\"MAC\":\"d4:e6:b7:94:39:95\",\"Name\":\"torydxb@tenant\",\"Phy\":\"g-HT\",\"Profile\":\"TENANTAUTH-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"TENANTAUTH-user-Role\",\"Type\":\"Android\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"CB-GF-FD07-WDN-OSP-050\",\"Auth\":\"802.1x\",\"Bssid\":\"20:a6:cd:30:9a:22\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"ahlan\",\"IP\":\"10.211.2.144\",\"MAC\":\"48:9d:d1:6d:8d:e9\",\"Name\":\"GNSCDWC02\",\"Phy\":\"g-HT\",\"Profile\":\"T3-CB-Ahlan-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Ahlan-User-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"}],\"_data\":[\"Total entries = 14995\"],\"_meta\":[\"IP\",\"MAC\",\"Name\",\"Current switch\",\"Role\",\"Auth\",\"AP name\",\"Roaming\",\"Essid\",\"Bssid\",\"Phy\",\"Profile\",\"Type\",\"User Type\"]}"
 | spath  
 | fields - _*
 | rename "Global Users"{}.* as *
 | rename data{} as _data, meta{} as _meta
 | mvexpand IP
 | rename IP as _IP
 | streamstats count
 | foreach *
    [eval <<FIELD>> = mvindex('<<FIELD>>', count - 1)]
| rename _IP as IP, _data as data, _meta as meta
| table IP MAC Name "Current switch" Role Auth  "AP name" Roaming Essid Bssid Phy Profile Type "User Type" data

At this level, you can normally spath .

0 Karma

starcher
Influencer

If JSON is that big you should use code outside of Splunk to parse it into reasonable events and send those in. Also when sending in properly formed JSON use kv_mode = JSON on your sourcetype definition in props.

bowesmana
SplunkTrust
SplunkTrust

JSON auto extraction will only extract, I believe, the first 5000 bytes. You need to use spath on the elements of the data you need.

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...