Getting Data In

Json huge data - issue with breaking the individual events

Nadhiya_Dubai
Explorer
"Global Users":[
    {
         "AP name":"T2-GF-WDN-ISP-079", 
         "Auth":null, 
         "Bssid":"94:b4:0f:04:51:f1", 
         "Current switch":"172.30.97.41", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.11.0.23", 
         "MAC":"68:e7:c2:5d:a1:ad", 
         "Name":null, 
         "Phy":"a-HT", 
         "Profile":"FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD07-WDN-OSP-109", 
         "Auth":null, 
         "Bssid":"40:e3:d6:23:3b:21", 
         "Current switch":"172.30.97.111", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.234.0.213", 
         "MAC":"fc:aa:b6:17:1a:a3", 
         "Name":null, 
         "Phy":"g-HT", 
         "Profile":"T3-FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD12-WDN-ISP-020", 
         "Auth":"802.1x", 
         "Bssid":"b4:5d:50:f8:57:e2", 
         "Current switch":"172.30.97.112", 
         "Essid":"tenantauth", 
         "IP":"10.235.197.85", 
         "MAC":"d4:e6:b7:94:39:95", 
         "Name":"torydxb@tenant", 
         "Phy":"g-HT", 
         "Profile":"TENANTAUTH-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"TENANTAUTH-user-Role", 
         "Type":"Android", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"CB-GF-FD07-WDN-OSP-050", 
         "Auth":"802.1x", 
         "Bssid":"20:a6:cd:30:9a:22", 
         "Current switch":"172.30.97.112", 
         "Essid":"ahlan", 
         "IP":"10.211.2.144", 
         "MAC":"48:9d:d1:6d:8d:e9", 
         "Name":"GNSCDWC02", 
         "Phy":"g-HT", 
         "Profile":"T3-CB-Ahlan-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Ahlan-User-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    }
],
"_data":[
    "Total entries = 14995" 
],
"_meta":[
    "IP", 
    "MAC", 
    "Name", 
    "Current switch", 
    "Role", 
    "Auth", 
    "AP name", 
    "Roaming", 
    "Essid", 
    "Bssid", 
    "Phy", 
    "Profile", 
    "Type", 
    "User Type" 
]

}"

Above is my json data . Well i have trimmmed the events ,its so huge lines in millions for a single event .
I tried giving the sourcetype as _json but its not breaking my events .Kindly help .I always have trouble while the data is in json format . Looking for the right solution and the explanation . Kindly help

Tags (1)
0 Karma

FrankVl
Ultra Champion

Maybe start with explaining what the desired behavior would be, because that is not very clear from your question. Do you want each { "AP name"... } section in a separate event?

In general, I would concur with the answer from @starcher that this looks like something you want to pre-process and then send into splunk as individual events, rather than massive json structs.

0 Karma

to4kawa
Ultra Champion

"Total entries = 14995"

props.conf
LINE_BREAKER in single line printed JSON doc

I hope this can be done well.

0 Karma

to4kawa
Ultra Champion
| makeresults 
 | eval _raw="{\"Global Users\":[{\"AP name\":\"T2-GF-WDN-ISP-079\",\"Auth\":null,\"Bssid\":\"94:b4:0f:04:51:f1\",\"Current switch\":\"172.30.97.41\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.11.0.23\",\"MAC\":\"68:e7:c2:5d:a1:ad\",\"Name\":null,\"Phy\":\"a-HT\",\"Profile\":\"FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD07-WDN-OSP-109\",\"Auth\":null,\"Bssid\":\"40:e3:d6:23:3b:21\",\"Current switch\":\"172.30.97.111\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.234.0.213\",\"MAC\":\"fc:aa:b6:17:1a:a3\",\"Name\":null,\"Phy\":\"g-HT\",\"Profile\":\"T3-FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD12-WDN-ISP-020\",\"Auth\":\"802.1x\",\"Bssid\":\"b4:5d:50:f8:57:e2\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"tenantauth\",\"IP\":\"10.235.197.85\",\"MAC\":\"d4:e6:b7:94:39:95\",\"Name\":\"torydxb@tenant\",\"Phy\":\"g-HT\",\"Profile\":\"TENANTAUTH-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"TENANTAUTH-user-Role\",\"Type\":\"Android\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"CB-GF-FD07-WDN-OSP-050\",\"Auth\":\"802.1x\",\"Bssid\":\"20:a6:cd:30:9a:22\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"ahlan\",\"IP\":\"10.211.2.144\",\"MAC\":\"48:9d:d1:6d:8d:e9\",\"Name\":\"GNSCDWC02\",\"Phy\":\"g-HT\",\"Profile\":\"T3-CB-Ahlan-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Ahlan-User-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"}],\"_data\":[\"Total entries = 14995\"],\"_meta\":[\"IP\",\"MAC\",\"Name\",\"Current switch\",\"Role\",\"Auth\",\"AP name\",\"Roaming\",\"Essid\",\"Bssid\",\"Phy\",\"Profile\",\"Type\",\"User Type\"]}"
 | spath  
 | fields - _*
 | rename "Global Users"{}.* as *
 | rename data{} as _data, meta{} as _meta
 | mvexpand IP
 | rename IP as _IP
 | streamstats count
 | foreach *
    [eval <<FIELD>> = mvindex('<<FIELD>>', count - 1)]
| rename _IP as IP, _data as data, _meta as meta
| table IP MAC Name "Current switch" Role Auth  "AP name" Roaming Essid Bssid Phy Profile Type "User Type" data

At this level, you can normally spath .

0 Karma

starcher
SplunkTrust
SplunkTrust

If JSON is that big you should use code outside of Splunk to parse it into reasonable events and send those in. Also when sending in properly formed JSON use kv_mode = JSON on your sourcetype definition in props.

bowesmana
SplunkTrust
SplunkTrust

JSON auto extraction will only extract, I believe, the first 5000 bytes. You need to use spath on the elements of the data you need.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...