So I have several things that logstash is currently reading, parsing, and sending to an elasticsearch instance. When creating a tcp listener (_json, no_timestamp_json, or custom type) I get large..."blobs" of data. It appears that the data within these large blobs have several events, all in one line. I can see that each event within the blob starts with:
{"unixtime":"
I've added a BREAK_ONLY_BEFORE with unixtime, but I continue to see the large blobs of data and not getting broken up. I'm just trying to do this in the interim as we transition from ELK to Splunk and thought this might be an easy way to go. Thank you.
So ok....recreated the setup...here's what I got...hopefully sanitized enough. Logstash has an output directive which I've set as tcp:
tcp {
host => "x.x.x.x"
port => "10000"
}
What's really odd is the fact that the gui shows different then cli:
in any case the changes made no different....splunk still isn't able to parse the info it seems.
the below is the raw info:
{"unixtime":"1527789306.404820","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.001029","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CqoZKB2VQhBTXbjEQl","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789306.404820\tCqoZKB2VQhBTXbjEQl\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.001029\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.512Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789313.325500","resp_packts":"0","orig_packts":"3","type":"connlog","dst_ip":"52.43.121.255","src_ip":"192.168.1.7","duration":"3.000128","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CGHm9R2xM910MzWa9b","dst_geoip":{"timezone":"America/Los_Angeles","ip":"52.43.121.255","latitude":45.8696,"continent_code":"NA","city_name":"Boardman","country_code2":"US","country_name":"United States","dma_code":810,"country_code3":"US","region_name":"Oregon","location":[-119.688,45.8696],"postal_code":"97818","longitude":-119.688,"region_code":"OR"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"T","orig_ip_bytes":132,"missed_bytes":"0","history":"S","message":"1527789313.325500\tCGHm9R2xM910MzWa9b\t192.168.1.7\t2258\t52.43.121.255\t10001\ttcp\t-\t3.000128\t0\t0\tS0\tT\tF\t0\tS\t3\t132\t0\t0\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":2258,"@timestamp":"2018-05-31T17:56:39.530Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":10001}{"unixtime":"1527789313.323127","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"192.168.1.253","src_ip":"192.168.1.7","duration":"0.001954","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CjLnl34b612eHlZ5ij","dst_geoip":{},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":283,"orig_bytes":38,"local_orig":"T","orig_ip_bytes":66,"missed_bytes":"0","history":"Dd","message":"1527789313.323127\tCjLnl34b612eHlZ5ij\t192.168.1.7\t4365\t192.168.1.253\t53\tudp\tdns\t0.001954\t38\t255\tSF\tT\tT\t0\tDd\t1\t66\t1\t283\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":4365,"@timestamp":"2018-05-31T17:56:39.538Z","resp_bytes":255,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789318.967153","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000966","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CVGgB83Uq7olftRsAl","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789318.967153\tCVGgB83Uq7olftRsAl\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.000966\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.538Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789326.745947","resp_packts":"0","orig_packts":"1","type":"connlog","dst_ip":"x.x.x.x","src_ip":"181.214.87.34","duration":"-","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CnZgr04QJZFkkdpJh","dst_geoip":{"timezone":"America/Bleh","ip":"x.x.x.x","latitude":bleh,"continent_code":"NA","city_name":"Bleh","country_code2":"US","country_name":"United States","dma_code":757,"country_code3":"US","region_name":"Bleh","location":[-116.2516,bleh],"postal_code":"83703","longitude":-116.2516,"region_code":"ID"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{"timezone":"America/Los_Angeles","ip":"181.214.87.34","latitude":36.175,"continent_code":"NA","city_name":"Las Vegas","country_code2":"US","country_name":"United States","dma_code":839,"country_code3":"US","region_name":"Nevada","location":[-115.1372,36.175],"postal_code":"89101","longitude":-115.1372,"region_code":"NV"},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"F","orig_ip_bytes":40,"missed_bytes":"0","history":"S","message":"1527789326.745947\tCnZgr04QJZFkkdpJh\t181.214.87.34\t44625\tx.x.x.x\t4025\ttcp\t-\t-\t-\t-\tS0\tF\tT\t0\tS\t1\t40\t0\t0\t(empty)\t-","src_port":44625,"@timestamp":"2018-05-31T17:56:39.546Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":4025}{"date":"May 31 11:55:26","kernel":"kernel","flags":"SYN","message":"May 31 11:55:26 hostname kernel: [512087.895915] IN=ppp0 OUT= MAC= SRC=181.214.87.34 DST=x.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=243 ID=61290 PROTO=TCP SPT=44625 DPT=4025 WINDOW=1024 RES=0x00 SYN URGP=0 ","type":"log","dst_ip":"x.x.x.x","tags":["kernel"],"src_ip":"181.214.87.34","src_port":44625,"path":"/var/log/messages","in_int":"ppp0","dst_geoip":{"timezone":"America/Bleh","ip":"x.x.x.x","latitude":bleh,"continent_code":"NA","city_name":"Bleh","country_code2":"US","country_name":"United States","dma_code":757,"country_code3":"US","region_name":"Bleh","location":[-116.2516,bleh],"postal_code":"83703","longitude":-116.2516,"region_code":"ID"},"@timestamp":"2018-05-31T17:56:39.555Z","len":"40","proto":"TCP","@version":"1","host":"hostname","dst_port":4025,"src_geoip":{"timezone":"America/Los_Angeles","ip":"181.214.87.34","latitude":36.175,"continent_code":"NA","city_name":"Las Vegas","country_code2":"US","country_name":"United States","dma_code":839,"country_code3":"US","region_name":"Nevada","location":[-115.1372,36.175],"postal_code":"89101","longitude":-115.1372,"region_code":"NV"},"device":"hostname"}{"unixtime":"1527789332.198834","resp_packts":"0","orig_packts":"3","type":"connlog","dst_ip":"52.43.121.255","src_ip":"192.168.1.7","duration":"3.000137","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CWEJ6zf7yppjEX8Lk","dst_geoip":{"timezone":"America/Los_Angeles","ip":"52.43.121.255","latitude":45.8696,"continent_code":"NA","city_name":"Boardman","country_code2":"US","country_name":"United States","dma_code":810,"country_code3":"US","region_name":"Oregon","location":[-119.688,45.8696],"postal_code":"97818","longitude":-119.688,"region_code":"OR"},"conn_state":"S0","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":0,"orig_bytes":0,"local_orig":"T","orig_ip_bytes":132,"missed_bytes":"0","history":"S","message":"1527789332.198834\tCWEJ6zf7yppjEX8Lk\t192.168.1.7\t2259\t52.43.121.255\t10001\ttcp\t-\t3.000137\t0\t0\tS0\tT\tF\t0\tS\t3\t132\t0\t0\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":2259,"@timestamp":"2018-05-31T17:56:39.555Z","resp_bytes":0,"service":"-,-","tun_parent":"(empty)","proto":"TCP","dst_port":10001}{"unixtime":"1527789331.568125","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000977","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"CLQX9F1XZDxbNvyRLj","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-122.3933,37.7697],"postal_code":"94107","longitude":-122.3933,"region_code":"CA"},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":85,"orig_bytes":41,"local_orig":"T","orig_ip_bytes":69,"missed_bytes":"0","history":"Dd","message":"1527789331.568125\tCLQX9F1XZDxbNvyRLj\t192.168.1.100\t58364\t208.67.220.220\t53\tudp\tdns\t0.000977\t41\t57\tSF\tT\tF\t0\tDd\t1\t69\t1\t85\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":58364,"@timestamp":"2018-05-31T17:56:39.560Z","resp_bytes":57,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789332.197317","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"192.168.1.253","src_ip":"192.168.1.7","duration":"0.001155","local_resp":"T","path":"/usr/local/bro/spool/bro/conn.log","uid":"CSK5Kw8h9CLb0YAm2","dst_geoip":{},"conn_state":"SF","@version":"1","host":"hostname","src_geoip":{},"resp_ip_bytes":283,"orig_bytes":38,"local_orig":"T","orig_ip_bytes":66,"missed_bytes":"0","history":"Dd","message":"1527789332.197317\tCSK5Kw8h9CLb0YAm2\t192.168.1.7\t4366\t192.168.1.253\t53\tudp\tdns\t0.001155\t38\t255\tSF\tT\tT\t0\tDd\t1\t66\t1\t283\t(empty)\t-","tags":["_geoip_lookup_failure"],"src_port":4366,"@timestamp":"2018-05-31T17:56:39.565Z","resp_bytes":255,"service":"dns,-","tun_parent":"(empty)","proto":"UDP","dst_port":53}{"unixtime":"1527789344.063242","resp_packts":"1","orig_packts":"1","type":"connlog","dst_ip":"208.67.220.220","src_ip":"192.168.1.100","duration":"0.000178","local_resp":"F","path":"/usr/local/bro/spool/bro/conn.log","uid":"C4RlGO3fHYyrbRXD9k","dst_geoip":{"timezone":"America/Los_Angeles","ip":"208.67.220.220","latitude":37.7697,"continent_code":"NA","city_name":"San Francisco","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region
Based on the limited info in the question I'm guessing you want this props.conf:
[your_sourcetype_here]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\{"unixtime":
...more stuff here, e.g. indexed extractions, timestamping, etc.
That assumes there's some form of newline/carriage return combo in front of each json object, gobbles up that combo and breaks events there. No further merging based on the default "merge until before you find a timestamp" or BREAK_ONLY_BEFORE and its comrades will need to happen.
If there's something else between the json objects you'll need to put that into the capturing group for LINE_BREAKER - empty strings are fine too, ie ()\{"unixtime":
if there's nothing between the objects. The capturing group is still necessary, to explicitly tell splunk "don't eat any chars between events".
Been a crazy couple weeks...sorry about that. Unfortunately as this is a dev setup I have already removed the data. I'll make the changes suggested and report my findings thanks.
Could you please show a broader example of your data (especially one event where multiple events are treated as one), and the props/transforms that relate to that sourcetype?