Getting Data In

indexed only 1 CSV file

thinguy
New Member

Trying to index some radius accounting (.act) files that are really CSV files with a header

"Date","Time","RAS-Client","Record-Type","Full-Name","Auth-Type","User-Name","NAS-IP-Address","NAS-Port","Service-Type","Framed-Protocol","Framed-IP-Address","Framed-IP-Netmask","Framed-Routing","Filter-ID","Framed-MTU","Framed-Compression","Login-IP-Host","Login-Service","Login-TCP-Port","Callback-Number","Callback-ID","Framed-Route","Framed-IPX-Network","Class","Session-Timeout","Idle-Timeout","Termination-Action","Called-Station-ID","Calling-Station-ID","NAS-Identifier","Proxy-State","Acct-Status-Type","Acct-Delay-Time","Acct-Input-Octets","Acct-Output-Octets","Acct-Session-Id","Acct-Authentic","Acct-Session-Time","Acct-Input-Packets","Acct-Output-Packets","Acct-Termination-Cause","Acct-Multi-Session-Id","Acct-Link-Count","NAS-Port-Type","Port-Limit","Tunnel-Type","Tunnel-Medium-Type","Tunnel-Client-Endpoint","Tunnel-Server-Endpoint","Acct-Tunnel-Connection","Tunnel-Private-Group-ID","Tunnel-Assignment-ID","Acct-Tunnel-Packets-Lost","Acct-Input-Gigawords","Acct-Output-Gigawords","Connect-Info","MS-Acct-Auth-Type","MS-Acct-EAP-Type","Event-Timestamp","NAS-Port-ID","ACC-Err-Message","Annex-Product-Name","Annex-SW-Version","Annex-System-Disc-Reason","Annex-Modem-Disc-Reason","Annex-Disconnect-Reason","Annex-Transmit-Speed","Annex-Receive-Speed","Ascend-Modem-Port-Number","Ascend-Modem-Slot-Number","Ascend-Modem-Shelf-Number","Ascend-Xmit-Rate","Nautica-Acct-SessionId","Nautica-Acct-Direction","Nautica-Acct-CauseProtocol","Nautica-Acct-CauseSource","Telebit-Accounting-Info","Last-Number-Dialed-Out","Last-Number-Dialed-In-DNIS","Last-Callers-Number-ANI","Channel","Event-Id","Event-Date-Time","Call-Start-Date-Time","Call-End-Date-Time","Default-DTE-Data-Rate","Initial-Rx-Link-Data-Rate","Final-Rx-Link-Data-Rate","Initial-Tx-Link-Data-Rate","Final-Tx-Link-Data-Rate","Sync-Async-Mode","Originate-Answer-Mode","Modulation-Type","Equalization-Type","Fallback-Enabled","Characters-Sent","Characters-Received","Blocks-Sent","Blocks-Received","Blocks-Resent","Retrains-Requested","Retrains-Granted","Line-Reversals","Number-Of-Characters-Lost","Number-of-Blers","Number-of-Link-Timeouts","Number-of-Fallbacks","Number-of-Upshifts","Number-of-Link-NAKs","Back-Channel-Data-Rate","Simplified-MNP-Levels","Simplified-V42bis-Usage","PW_VPN_ID"
"06/10/2009","08:36:13","CISCO 3000 VPN","Start","jsmith","200","jsmith","10.12.44.33","1922","2","1","10.19.12.13",,,,,,,,,,,,,"0x53425232434ce3d796b1dadd9dd5b98011802501800481998c868002800781b0d8cdc68b8dd612800e81e3d796b1dadd9dd5b98082edaa98",,,,,"74.133.61.240","CISCO 3000 VPN",,"1","0",,,"BF70ACEA","1",,,,,,,"5",,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

No matter how many files I put in my directory only the first file is indexed. I noticed that if I delete the header from another file it will get indexed but it shows as a second sourcetype. If I add another file with a deleted header it will appear in the second sourcetype.

I've tried setting as automatic and as CSV. Doing fresh installs on each test.

How do I index all files without having to delete the header? And how do I get the header fields recognized?

Thanks for any help you can throw my way.

Tags (2)
0 Karma
1 Solution

ziegfried
Influencer

Seems to be the same problem as here: http://answers.splunk.com/questions/4629/splunks-mechanism-to-detect-files-with-the-same-content

You can work around that by specifying a crcSalt in your monitor configuration:

inputs.conf

[monitor:///path/to/directory]
crcSalt = <SOURCE>
host = your_host
index = your_index
sourcetype = your_sourcetype

View solution in original post

0 Karma

ziegfried
Influencer

Seems to be the same problem as here: http://answers.splunk.com/questions/4629/splunks-mechanism-to-detect-files-with-the-same-content

You can work around that by specifying a crcSalt in your monitor configuration:

inputs.conf

[monitor:///path/to/directory]
crcSalt = <SOURCE>
host = your_host
index = your_index
sourcetype = your_sourcetype
0 Karma

thinguy
New Member

Thanks that did it.

0 Karma

ziegfried
Influencer

Obviously it doesn't look at the last 256 bytes. Have you added your new index (sbrras) to the default indexes of one of your roles? If not, you won't see it on the summary page.

0 Karma

thinguy
New Member

Also wondering like the link you posted.
If Splunk uses the first AND last 256bytes of the file, it should be seeing my files as unique. Since only the first part is duplicated.

0 Karma

thinguy
New Member

Thanks for the fast response. I've done something wrong.
I did a fresh install created an index "sbrras" a data input of csv pointing to my new index
and updated my f:\splunks\etc\apps\search\local\inputs.conf file with the info below

I added one file into my Dir and nothing shows up.
I added 2 more files and still now, nothing appears on my search page under Source|Sourcetype|Hosts

[monitor://F:\Splunk\var\raslogs]
disabled = false
followTail = 0
host = RAS
crcSalt =
index = sbrras
sourcetype = csv

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...