Splunk Enterprise

Switching from ELK to Splunk: Logstash data to TCP listener

DigiAngel
New Member

So I have several things that logstash is currently reading, parsing, and sending to an elasticsearch instance. When creating a tcp listener (_json, no_timestamp_json, or custom type) I get large..."blobs" of data. It appears that the data within these large blobs have several events, all in one line. I can see that each event within the blob starts with:

{"unixtime":"

I've added a BREAK_ONLY_BEFORE with unixtime, but I continue to see the large blobs of data and not getting broken up. I'm just trying to do this in the interim as we transition from ELK to Splunk and thought this might be an easy way to go. Thank you.

Tags (1)
0 Karma

DigiAngel
New Member

So ok....recreated the setup...here's what I got...hopefully sanitized enough. Logstash has an output directive which I've set as tcp:

        tcp {
                host => "x.x.x.x"
                port => "10000"
        }

What's really odd is the fact that the gui shows different then cli:

alt text alt text

in any case the changes made no different....splunk still isn't able to parse the info it seems.

the below is the raw info:

{"unixtime":"1527789306.404820","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.001029","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CqoZKB2VQhBTXbjEQl","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789306.404820\tCqoZKB2VQhBTXbjEQl\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.001029\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.512Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789313.325500","resp_packts":"0","orig_packts":"3","type":"connlog","dst_ip":"52.43.121.255","src_ip":"192.168.1.7","duration":"3.000128","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CGHm9R2xM910MzWa9b","dst_geoip":{"timezone":"America/Los_Angeles","ip":"52.43.121.255","latitude":45.8696,"continent_code":"NA","city_name":"Boardman","country_code2":"US","country_name":"United States","dma_code":810,"country_code3":"US","region_name":"Oregon","location":[-119.688,45.8696],"postal_code":"97818","longitude":-119.688,"region_code":"OR"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"T","orig_ip_bytes":132,"missed_bytes":"0","history":"S","message":"1527789313.325500\tCGHm9R2xM910MzWa9b\t192.168.1.7\t2258\t52.43.121.255\t10001\ttcp\t-\t3.000128\t0\t0\tS0\tT\tF\t0\tS\t3\t132\t0\t0\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":2258,"@timestamp":"2018-05-31T17:56:39.530Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":10001}{"unixtime":"1527789313.323127","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"192.168.1.253","src_ip":"192.168.1.7","duration":"0.001954","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CjLnl34b612eHlZ5ij","dst_geoip":{},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":283,"orig_bytes":38,"local_orig":"T","orig_ip_bytes":66,"missed_bytes":"0","history":"Dd","message":"1527789313.323127\tCjLnl34b612eHlZ5ij\t192.168.1.7\t4365\t192.168.1.253\t53\tudp\tdns\t0.001954\t38\t255\tSF\tT\tT\t0\tDd\t1\t66\t1\t283\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":4365,"@timestamp":"2018-05-31T17:56:39.538Z","resp_bytes":255,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789318.967153","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000966","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CVGgB83Uq7olftRsAl","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789318.967153\tCVGgB83Uq7olftRsAl\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.000966\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.538Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789326.745947","resp_packts":"0","orig_packts":"1","type":"connlog","dst_ip":"x.x.x.x","src_ip":"181.214.87.34","duration":"-","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CnZgr04QJZFkkdpJh","dst_geoip":{"timezone":"America/Bleh","ip":"x.x.x.x","latitude":bleh,"continent_code":"NA","city_name":"Bleh","country_code2":"US","country_name":"United States","dma_code":757,"country_code3":"US","region_name":"Bleh","location":[-116.2516,bleh],"postal_code":"83703","longitude":-116.2516,"region_code":"ID"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{"timezone":"America/Los_Angeles","ip":"181.214.87.34","latitude":36.175,"continent_code":"NA","city_name":"Las Vegas","country_code2":"US","country_name":"United States","dma_code":839,"country_code3":"US","region_name":"Nevada","location":[-115.1372,36.175],"postal_code":"89101","longitude":-115.1372,"region_code":"NV"},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"F","orig_ip_bytes":40,"missed_bytes":"0","history":"S","message":"1527789326.745947\tCnZgr04QJZFkkdpJh\t181.214.87.34\t44625\tx.x.x.x\t4025\ttcp\t-\t-\t-\t-\tS0\tF\tT\t0\tS\t1\t40\t0\t0\t(empty)\t-","src_port":44625,"@timestamp":"2018-05-31T17:56:39.546Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":4025}{"date":"May 31 11:55:26","kernel":"kernel","flags":"SYN","message":"May 31 11:55:26 hostname kernel: [512087.895915] IN=ppp0 OUT= MAC= SRC=181.214.87.34 DST=x.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=61290 PROTO=TCP SPT=44625 DPT=4025 WINDOW=1024 RES=0x00 SYN URGP=0 ","type":"log","dst_ip":"x.x.x.x","tags":["kernel"],"src_ip":"181.214.87.34","src_port":44625,"path":"/var/log/messages","in_int":"ppp0","dst_geoip":{"timezone":"America/Bleh","ip":"x.x.x.x","latitude":bleh,"continent_code":"NA","city_name":"Bleh","country_code2":"US","country_name":"United States","dma_code":757,"country_code3":"US","region_name":"Bleh","location":[-116.2516,bleh],"postal_code":"83703","longitude":-116.2516,"region_code":"ID"},"@timestamp":"2018-05-31T17:56:39.555Z","len":"40","proto":"TCP","@version":"1","host":"hostname","dst_port":4025,"src_geoip":{"timezone":"America/Los_Angeles","ip":"181.214.87.34","latitude":36.175,"continent_code":"NA","city_name":"Las Vegas","country_code2":"US","country_name":"United States","dma_code":839,"country_code3":"US","region_name":"Nevada","location":[-115.1372,36.175],"postal_code":"89101","longitude":-115.1372,"region_code":"NV"},"device":"hostname"}{"unixtime":"1527789332.198834","resp_packts":"0","orig_packts":"3","type":"connlog","dst_ip":"52.43.121.255","src_ip":"192.168.1.7","duration":"3.000137","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CWEJ6zf7yppjEX8Lk","dst_geoip":{"timezone":"America/Los_Angeles","ip":"52.43.121.255","latitude":45.8696,"continent_code":"NA","city_name":"Boardman","country_code2":"US","country_name":"United States","dma_code":810,"country_code3":"US","region_name":"Oregon","location":[-119.688,45.8696],"postal_code":"97818","longitude":-119.688,"region_code":"OR"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"T","orig_ip_bytes":132,"missed_bytes":"0","history":"S","message":"1527789332.198834\tCWEJ6zf7yppjEX8Lk\t192.168.1.7\t2259\t52.43.121.255\t10001\ttcp\t-\t3.000137\t0\t0\tS0\tT\tF\t0\tS\t3\t132\t0\t0\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":2259,"@timestamp":"2018-05-31T17:56:39.555Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":10001}{"unixtime":"1527789331.568125","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000977","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CLQX9F1XZDxbNvyRLj","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789331.568125\tCLQX9F1XZDxbNvyRLj\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.000977\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.560Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789332.197317","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"192.168.1.253","src_ip":"192.168.1.7","duration":"0.001155","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CSK5Kw8h9CLb0YAm2","dst_geoip":{},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":283,"orig_bytes":38,"local_orig":"T","orig_ip_bytes":66,"missed_bytes":"0","history":"Dd","message":"1527789332.197317\tCSK5Kw8h9CLb0YAm2\t192.168.1.7\t4366\t192.168.1.253\t53\tudp\tdns\t0.001155\t38\t255\tSF\tT\tT\t0\tDd\t1\t66\t1\t283\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":4366,"@timestamp":"2018-05-31T17:56:39.565Z","resp_bytes":255,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789344.063242","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000178","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"C4RlGO3fHYyrbRXD9k","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Based on the limited info in the question I'm guessing you want this props.conf:

[your_sourcetype_here]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\{"unixtime":
...more stuff here, e.g. indexed extractions, timestamping, etc.

That assumes there's some form of newline/carriage return combo in front of each json object, gobbles up that combo and breaks events there. No further merging based on the default "merge until before you find a timestamp" or BREAK_ONLY_BEFORE and its comrades will need to happen.
If there's something else between the json objects you'll need to put that into the capturing group for LINE_BREAKER - empty strings are fine too, ie ()\{"unixtime": if there's nothing between the objects. The capturing group is still necessary, to explicitly tell splunk "don't eat any chars between events".

0 Karma

DigiAngel
New Member

Been a crazy couple weeks...sorry about that. Unfortunately as this is a dev setup I have already removed the data. I'll make the changes suggested and report my findings thanks.

0 Karma

xpac
SplunkTrust
SplunkTrust

Could you please show a broader example of your data (especially one event where multiple events are treated as one), and the props/transforms that relate to that sourcetype?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...