Hello, I have a win2008 fwd not sending data to a custom index
we have several indexers + 1 searchhead (all centos) and we were getting the data from this fwd up until a week ago, around the time the winbox was rebooted for updates. After that data stopped coming in.
I checked:
network - all ports are open and I can confirm data from this fwd is coming in to other indexes, just not this index
watched file is updated every few min, so new data is coming in
ran btool for syntax check, looks ok
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf [monitor://C:\Windows\System32\LogFiles]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf alwaysOpenFile = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf baseline = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf crcSalt =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = BICS03
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf ignoreOlderThan = 1h
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf index = sec-radius
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf recursive = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf sourcetype = RADIUS
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf whitelist = IN.+.log
the watched file is here, just a log looks like this
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:21:33,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A5,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:07,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,2,41,0,42,1612796,43,2380880,44,DEE001A5,45,1,46,1414,47,4545,48,4392,49,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,5,2109440,30,111.222.33312,31,111.222.333.98,61,5,66,111.222.333.98,4,162.xxx..250.1,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,5000,ip:source-ip=111.222.333.98,4154,CGY: CGY VPN CR,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4136,1,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4130,COMP\xxxxx,4129,COMP\xxxxx,4155,1,4154,CGY: CGY VPN CR,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:45:44,IAS,COMDOM03,5,2109440,6,2,7,1,8,162.xxx..238.21,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A6,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
also the index is present in the indexer,
[root@splunk01 /opt/splunk/bin]# cat /opt/splunk/etc/apps/search//local/indexes.conf
[sec-radius]
coldPath = $SPLUNK_DB/sec-radius/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/sec-radius/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/sec-radius/thaweddb
Preview [Hide]
Hello, I have a win2008 fwd not sending data to a custom index
we have several indexers + 1 searchhead (all centos) and we were getting the data from this fwd up until a week ago, around the time the winbox was rebooted for updates. After that data stopped coming in.
I checked:
network - all ports are open and I can confirm data from this fwd is coming in to other indexes, just not this index
watched file is updated every few min, so new data is coming in
ran btool for syntax check, looks ok
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf [monitor://C:\Windows\System32\LogFiles]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf alwaysOpenFile = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf baseline = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf crcSalt = C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf disabled = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name = C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name = C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0 C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = BICS03 C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf ignoreOlderThan = 1h C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf index = sec-radius C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60 C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf recursive = false C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf sourcetype = RADIUS C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf whitelist = IN.+.log
the watched file is here, just a log looks like this
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:09,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 8,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:31,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 9,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:20:36,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 10,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,36
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:24,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 11,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:21:28,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 12,4127,1,4130,COMP\xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:21:33,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A5,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:07,IAS,COMDOM03,5,2101248,6,2,7,1,8,162.xxx..238.19,30,111.222.33312,31,111.222.333.98,40,2,41,0,42,1612796,43,2380880,44,DEE001A5,45,1,46,1414,47,4545,48,4392,49,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,4,162.xxx..5.29,31,111.222.333.98,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8136,1,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4136,1,4142,0
162.xxx..5.29,xxxxx,04/28/2017,14:45:37,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 13,8153,0,8111,0,4130,COMP.local/COM-Users/Employees/Calgary/Greg xxxxx,4108,162.xxx..5.29,4116,0,4128,CGY: DDD DOM03,4154,CGY: CGY DDD CRP,4155,1,4129,COMP\xxxxx,4127,1,4149,CGY: CGY VPN DUO NP,8136,1,7,1,6,2,4294967210,50,4294967209,120,4136,2,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,5,2109440,30,111.222.33312,31,111.222.333.98,61,5,66,111.222.333.98,4,162.xxx..250.1,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,5000,ip:source-ip=111.222.333.98,4154,CGY: CGY VPN CR,4155,1,4129,COMP\xxxxx,4130,COMP\xxxxx,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4136,1,4142,0
162.xxx..250.1,xxxxx,04/28/2017,14:45:40,IAS,COMDOM03,25,311 1 162.xxx..5.29 04/28/2017 20:02:29 14,4127,1,4130,COMP\xxxxx,4129,COMP\xxxxx,4155,1,4154,CGY: CGY VPN CR,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4136,3,4142,16
162.xxx..250.1,xxxxx,04/28/2017,14:45:44,IAS,COMDOM03,5,2109440,6,2,7,1,8,162.xxx..238.21,30,111.222.33312,31,111.222.333.98,40,1,41,0,44,DEE001A6,45,1,61,5,66,111.222.333.98,26,0x00000C04920A4461726B2D44756F,26,0x00000C04960600000002,26,0x00000C04970600000001,26,0x00000C04980600000003,4,162.xxx..250.1,4108,162.xxx..250.1,4116,0,4128,CGY: ASA CGY,4154,CGY: CGY VPN CR,4136,4,4142,0
also the index is present in the indexer,
[root@splunk01 /opt/splunk/bin]# cat /opt/splunk/etc/apps/search//local/indexes.conf
[sec-radius]
coldPath = $SPLUNK_DB/sec-radius/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/sec-radius/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/sec-radius/thaweddb
... View more