Getting Data In

How do I set a source type for this data

maxd
Engager

I have a script that pulls the data at the bottom into a file and then splunk pull the files from the corresponding directory
However one of the 3 hosts works properly and 2 hosts split into 2 events, that are different sizes. All the files should be the same in terms of structure. I cannot seem to solve the source type to fix it.

I am still fairly new to splunk, Any advice?

alt text

Script to retrieve log data

#!/bin/bash
cd /home/max    
hosts=$(pcregrep -M '^192.*-.*\n' /etc/hosts | sed  -e 's/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/\n/g' | tail -n +2 | tr -d '[:blank:]')    
rm -rf logs/*
rm -rf logs2/*    
for host in $hosts
do
        if ping -c 1 -W 1 $host; then
                if [[ $host == *"SW"* ]]; then
                        mkdir -p logs2/$host
                        ssh  -oIdentitiesOnly=yes -i /home/max/.ssh/id_rsa_2 marq-net@$host -q 'ubntbox mca-status'  | sed 's/,/\n/g' | sed '/^[[:space:]]*$/d' > logs2/$host/$host.log
                else
                        mkdir -p logs/$host
                        ssh  -oIdentitiesOnly=yes -i /home/max/.ssh/id_rsa_2 marq-net@$host -q 'ubntbox mca-status'  | sed 's/,/\n/g' | sed '/^[[:space:]]*$/d' > logs/$host/$host.log
                fi
        fi
done

echo $(TZ='America/Chicago' date)
echo $(date)

They should be the exact same structure

max@splunk:~$ cat logs/Silo-Link02/Silo-Link02.log| wc -l
67
max@splunk:~$ cat logs/Marq-Link01/Marq-Link01.log| wc -l
67

Here is the log from Marq-Link01

deviceName=Marq-Link01
deviceId=XX:CC:XX:CC:XX:CC
firmwareVersion=WA.ar934x.v8.6.2.41239.190822.1633
platform=LiteBeam 5AC Gen2
deviceIp=192.168.0.50
apMac=XX:CC:XX:CC:XX:CC
wlanOpmode=ap-ptp-ac
wlanConnections=1
wlanUptime=235026
essid=DataLink
security=WPA2
freq=5520
centerFreq=5550
txPower=24
chanbw=80
signal=-70
chain0Signal=-72
chain1Signal=-74
noise=-88
cinr=23
evm=24
uptime=497206
airTime=0.1
cpuUsage=33.3
loadavg=7
memTotal=61952
memFree=13764
memBuffers=3376
distance=3605
netrole=bridge
lanIpAddress=0.0.0.0
wlanIpAddress=0.0.0.0
wlanTxRate=351.0
wlanRxRate=468.0
txModRate=4x
rxModRate=6x
wlanTxLatency=0
wlanPolling=1
wlanScanStatus=0
wlanDownlinkCapacity=263250
wlanUplinkCapacity=238680
lanRxBytes=3192091802
lanRxPackets=17133341
lanRxErrors=0
lanTxBytes=11338063946
lanTxPackets=10570286
lanTxErrors=0
lanPlugged=1
lanSpeed=1000Mbps-Full
cableLen=29
wlanRxBytes=8409699951
wlanRxPackets=7949519
wlanRxErrors=0
wlanTxBytes=2920142565
wlanTxPackets=14584597
wlanTxErrors=0
wlanRxErrNwid=19662
wlanRxErrCrypt=0
wlanRxErrFrag=0
wlanRxErrRetries=0
wlanRxErrBmiss=0
wlanRxErrOther=0
latitude=42
longitude=-97
boardCrc=e06fb32b
cfgCrc=39387dfa
status_flags=4

Here is the log that works properly from Silo-Link2

deviceName=Silo - link02
deviceId=B4:FB:E4:B8:8E:85
firmwareVersion=WA.ar934x.v8.6.2.41239.190822.1633
platform=LiteAP AC
deviceIp=192.168.0.52
apMac=B4:FB:E4:B8:8E:85
wlanOpmode=ap-ptmp-ac
wlanConnections=0
wlanUptime=0
essid=Marq-DataLink2
security=WPA2
freq=5680
centerFreq=5680
txPower=24
chanbw=10
signal=0
chain0Signal=0
chain1Signal=0
noise=0
cinr=0
evm=0
uptime=93652
airTime=1.4
cpuUsage=65.0
loadavg=0
memTotal=61952
memFree=17580
memBuffers=3264
distance=100000
netrole=bridge
lanIpAddress=0.0.0.0
wlanIpAddress=0.0.0.0
wlanTxRate=0.0
wlanRxRate=0.0
txModRate=1x
rxModRate=1x
wlanTxLatency=0
wlanPolling=1
wlanScanStatus=0
wlanDownlinkCapacity=0
wlanUplinkCapacity=0
lanRxBytes=262240342
lanRxPackets=2100983
lanRxErrors=0
lanTxBytes=443438977
lanTxPackets=448666
lanTxErrors=0
lanPlugged=1
lanSpeed=1000Mbps-Full
cableLen=41
wlanRxBytes=0
wlanRxPackets=0
wlanRxErrors=0
wlanTxBytes=174479589
wlanTxPackets=1408700
wlanTxErrors=0
wlanRxErrNwid=0
wlanRxErrCrypt=0
wlanRxErrFrag=0
wlanRxErrRetries=0
wlanRxErrBmiss=0
wlanRxErrOther=0
latitude=42
longitude=-97
boardCrc=ef03416f
cfgCrc=d59a5d30
status_flags=4
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

It's likely not a script problem, but a props.conf problem. In the inputs.conf file that defines the scripted input, add sourceype = foo. Then in the corresponding props.conf file add:

[foo]
DATE_TIME_CONFIG = current
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)deviceName
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

It's likely not a script problem, but a props.conf problem. In the inputs.conf file that defines the scripted input, add sourceype = foo. Then in the corresponding props.conf file add:

[foo]
DATE_TIME_CONFIG = current
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)deviceName
---
If this reply helps you, Karma would be appreciated.

maxd
Engager

I apologize, I'm not sure where those files are.
my splunk is at /opt/splunk which file is the one I want to edit

max@splunk:~$ sudo find /opt/splunk | grep inputs.conf
/opt/splunk/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunk/etc/apps/splunk_app_infrastructure/README/inputs.conf.spec
/opt/splunk/etc/apps/splunk_app_infrastructure/local/inputs.conf
/opt/splunk/etc/apps/splunk_app_infrastructure/default/inputs.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/inputs.conf
/opt/splunk/etc/apps/splunk_instrumentation/default/inputs.conf
/opt/splunk/etc/apps/sample_app/default/inputs.conf
/opt/splunk/etc/apps/snmp_ta/README/inputs.conf.spec
/opt/splunk/etc/apps/splunk_httpinput/default/inputs.conf
/opt/splunk/etc/apps/splunk_monitoring_console/default/inputs.conf
/opt/splunk/etc/apps/search/local/inputs.conf
/opt/splunk/etc/system/README/inputs.conf.spec
/opt/splunk/etc/system/README/inputs.conf.example
/opt/splunk/etc/system/local/inputs.conf
/opt/splunk/etc/system/default/inputs.conf
/opt/splunk/etc/modules/distributedDeployment/classes/deployable/inputs.conf
/opt/splunk/lib/python2.7/site-packages/slim/config/conf-specs/inputs.conf.spec

max@splunk:~$ sudo find /opt/splunk | grep props.conf
/opt/splunk/etc/apps/splunk_app_infrastructure/default/props.conf
/opt/splunk/etc/apps/learned/local/props.conf
/opt/splunk/etc/apps/splunk_archiver/default/props.conf
/opt/splunk/etc/apps/SplunkLightForwarder/default/props.conf
/opt/splunk/etc/apps/legacy/default/props.conf
/opt/splunk/etc/apps/splunk_instrumentation/default/props.conf
/opt/splunk/etc/apps/sample_app/default/props.conf
/opt/splunk/etc/apps/snmp_ta/default/props.conf
/opt/splunk/etc/apps/splunk_monitoring_console/default/props.conf
/opt/splunk/etc/apps/search/local/props.conf
/opt/splunk/etc/apps/search/default/props.conf
/opt/splunk/etc/system/README/props.conf.spec
/opt/splunk/etc/system/README/props.conf.example
/opt/splunk/etc/system/default/props.conf
/opt/splunk/lib/python2.7/site-packages/slim/config/conf-specs/props.conf.spec
max@splunk:~$
0 Karma

maxd
Engager

That did the trick Sir. Thank you so much! If you dont mind explaining it a bit it would be much appreciated.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

First, do NOT edit any file in a 'default' directory or your changes will be lost the next time you upgrade Splunk. See the note at the top of $SPLUNK_HOME/etc/system/default.props.conf. It's better to create a custom apps, myorg_foo_inputs and myorg_foo_props, to hold the config files then install those apps in the appropriate places (search head, indexer, forwarder).

The sourcetype attribute in inputs.conf answers the original question: how to set a source type for a data source. Every unique source type (format) should have a sourcetype attribute and a corresponding stanza in props.conf.

The props.conf stanza tells Splunk how to parse the data of that type. Splunk is pretty good at guessing, but, as you've seen, doesn't always get it right. Use props to prevent Splunk from guessing wrong.

DATE_TIME_CONFIG = current says to use the current time for events since the data does not contain a timestamp.
SHOULD_LINEMERGE = false says not to join lines together.
'LINE_BREAKER = ([\r\n]+)deviceName' tells Splunk events begin when "deviceName" is seen immediately after a newline.

All of these settings are in the docs and in $SPLUNK_HOME/etc/system/README/props.conf.spec.

---
If this reply helps you, Karma would be appreciated.
0 Karma

maxd
Engager

Nevermind got it 🙂

For anyone like me
/opt/splunk/etc/system/default/props.conf

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...