Getting Data In

Json huge data - issue with breaking the individual events

Nadhiya_Dubai
Explorer
"Global Users":[
    {
         "AP name":"T2-GF-WDN-ISP-079", 
         "Auth":null, 
         "Bssid":"94:b4:0f:04:51:f1", 
         "Current switch":"172.30.97.41", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.11.0.23", 
         "MAC":"68:e7:c2:5d:a1:ad", 
         "Name":null, 
         "Phy":"a-HT", 
         "Profile":"FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD07-WDN-OSP-109", 
         "Auth":null, 
         "Bssid":"40:e3:d6:23:3b:21", 
         "Current switch":"172.30.97.111", 
         "Essid":"#DXB Free WiFi", 
         "IP":"10.234.0.213", 
         "MAC":"fc:aa:b6:17:1a:a3", 
         "Name":null, 
         "Phy":"g-HT", 
         "Profile":"T3-FreeWifi-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Free-Wifi-user-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"T3-L2-FD12-WDN-ISP-020", 
         "Auth":"802.1x", 
         "Bssid":"b4:5d:50:f8:57:e2", 
         "Current switch":"172.30.97.112", 
         "Essid":"tenantauth", 
         "IP":"10.235.197.85", 
         "MAC":"d4:e6:b7:94:39:95", 
         "Name":"torydxb@tenant", 
         "Phy":"g-HT", 
         "Profile":"TENANTAUTH-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"TENANTAUTH-user-Role", 
         "Type":"Android", 
         "User Type":"WIRELESS" 
    },
    {
         "AP name":"CB-GF-FD07-WDN-OSP-050", 
         "Auth":"802.1x", 
         "Bssid":"20:a6:cd:30:9a:22", 
         "Current switch":"172.30.97.112", 
         "Essid":"ahlan", 
         "IP":"10.211.2.144", 
         "MAC":"48:9d:d1:6d:8d:e9", 
         "Name":"GNSCDWC02", 
         "Phy":"g-HT", 
         "Profile":"T3-CB-Ahlan-AAA-Profile", 
         "Roaming":"Wireless", 
         "Role":"Ahlan-User-Role", 
         "Type":"Linux", 
         "User Type":"WIRELESS" 
    }
],
"_data":[
    "Total entries = 14995" 
],
"_meta":[
    "IP", 
    "MAC", 
    "Name", 
    "Current switch", 
    "Role", 
    "Auth", 
    "AP name", 
    "Roaming", 
    "Essid", 
    "Bssid", 
    "Phy", 
    "Profile", 
    "Type", 
    "User Type" 
]

}"

Above is my json data . Well i have trimmmed the events ,its so huge lines in millions for a single event .
I tried giving the sourcetype as _json but its not breaking my events .Kindly help .I always have trouble while the data is in json format . Looking for the right solution and the explanation . Kindly help

Tags (1)
0 Karma

FrankVl
Ultra Champion

Maybe start with explaining what the desired behavior would be, because that is not very clear from your question. Do you want each { "AP name"... } section in a separate event?

In general, I would concur with the answer from @starcher that this looks like something you want to pre-process and then send into splunk as individual events, rather than massive json structs.

0 Karma

to4kawa
Ultra Champion

"Total entries = 14995"

props.conf
LINE_BREAKER in single line printed JSON doc

I hope this can be done well.

0 Karma

to4kawa
Ultra Champion
| makeresults 
 | eval _raw="{\"Global Users\":[{\"AP name\":\"T2-GF-WDN-ISP-079\",\"Auth\":null,\"Bssid\":\"94:b4:0f:04:51:f1\",\"Current switch\":\"172.30.97.41\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.11.0.23\",\"MAC\":\"68:e7:c2:5d:a1:ad\",\"Name\":null,\"Phy\":\"a-HT\",\"Profile\":\"FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD07-WDN-OSP-109\",\"Auth\":null,\"Bssid\":\"40:e3:d6:23:3b:21\",\"Current switch\":\"172.30.97.111\",\"Essid\":\"#DXB Free WiFi\",\"IP\":\"10.234.0.213\",\"MAC\":\"fc:aa:b6:17:1a:a3\",\"Name\":null,\"Phy\":\"g-HT\",\"Profile\":\"T3-FreeWifi-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Free-Wifi-user-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"T3-L2-FD12-WDN-ISP-020\",\"Auth\":\"802.1x\",\"Bssid\":\"b4:5d:50:f8:57:e2\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"tenantauth\",\"IP\":\"10.235.197.85\",\"MAC\":\"d4:e6:b7:94:39:95\",\"Name\":\"torydxb@tenant\",\"Phy\":\"g-HT\",\"Profile\":\"TENANTAUTH-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"TENANTAUTH-user-Role\",\"Type\":\"Android\",\"User Type\":\"WIRELESS\"},{\"AP name\":\"CB-GF-FD07-WDN-OSP-050\",\"Auth\":\"802.1x\",\"Bssid\":\"20:a6:cd:30:9a:22\",\"Current switch\":\"172.30.97.112\",\"Essid\":\"ahlan\",\"IP\":\"10.211.2.144\",\"MAC\":\"48:9d:d1:6d:8d:e9\",\"Name\":\"GNSCDWC02\",\"Phy\":\"g-HT\",\"Profile\":\"T3-CB-Ahlan-AAA-Profile\",\"Roaming\":\"Wireless\",\"Role\":\"Ahlan-User-Role\",\"Type\":\"Linux\",\"User Type\":\"WIRELESS\"}],\"_data\":[\"Total entries = 14995\"],\"_meta\":[\"IP\",\"MAC\",\"Name\",\"Current switch\",\"Role\",\"Auth\",\"AP name\",\"Roaming\",\"Essid\",\"Bssid\",\"Phy\",\"Profile\",\"Type\",\"User Type\"]}"
 | spath  
 | fields - _*
 | rename "Global Users"{}.* as *
 | rename data{} as _data, meta{} as _meta
 | mvexpand IP
 | rename IP as _IP
 | streamstats count
 | foreach *
    [eval <<FIELD>> = mvindex('<<FIELD>>', count - 1)]
| rename _IP as IP, _data as data, _meta as meta
| table IP MAC Name "Current switch" Role Auth  "AP name" Roaming Essid Bssid Phy Profile Type "User Type" data

At this level, you can normally spath .

0 Karma

starcher
Influencer

If JSON is that big you should use code outside of Splunk to parse it into reasonable events and send those in. Also when sending in properly formed JSON use kv_mode = JSON on your sourcetype definition in props.

bowesmana
SplunkTrust
SplunkTrust

JSON auto extraction will only extract, I believe, the first 5000 bytes. You need to use spath on the elements of the data you need.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

REGISTER NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If ...

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...