What are the differences between monitoring a folder on a UF and monitor a folder local on the indexer?

[max8006]

Hi

I have the problem that I have different parsing results when I monitor a file (csv) on a universal forwarder or local on the SH+Indexer.

Local on the SH+Indexer the parsing with the field name assignment/extraction works.
When the same file was read by the UF and sent to the SAME Sourcetype the result is different - no fieldnames are applied. I think there is no parsing happening?

I think the config problem is on the UF. I only put there a inputs.conf with (monitor --> sourcetype + host and index)

Maybe someone can help me out?

[max8006 · Accepted solution]

IDX-props.conf

C:\Program Files\Splunk\bin>splunk btool props list --debug tv:star2
C:\Program Files\Splunk\etc\apps\search\local\props.conf [tv:star2]
C:\Program Files\Splunk\etc\system\default\props.conf    ADD_EXTRA_TIME_FIELDS = True
C:\Program Files\Splunk\etc\system\default\props.conf    ANNOTATE_PUNCT = True
C:\Program Files\Splunk\etc\system\default\props.conf    AUTO_KV_JSON = true
C:\Program Files\Splunk\etc\system\default\props.conf    BREAK_ONLY_BEFORE =
C:\Program Files\Splunk\etc\system\default\props.conf    BREAK_ONLY_BEFORE_DATE = True
C:\Program Files\Splunk\etc\system\default\props.conf    CHARSET = AUTO
C:\Program Files\Splunk\etc\apps\search\local\props.conf DATETIME_CONFIG =
C:\Program Files\Splunk\etc\system\default\props.conf    DEPTH_LIMIT = 1000
C:\Program Files\Splunk\etc\apps\search\local\props.conf FIELD_DELIMITER = ,
C:\Program Files\Splunk\etc\apps\search\local\props.conf FIELD_NAMES = id,detection_time,log,parameter_list,text_english,error,sequence,roc_name,file_name,error_position,error_host,ack_required,acknowledged,ack_time,based_on_id,ack_by_user,job_id,recipient,destination,sender,subject,computed_size,spool
C:\Program Files\Splunk\etc\apps\search\local\props.conf FIELD_QUOTE = '
C:\Program Files\Splunk\etc\system\default\props.conf    HEADER_MODE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf INDEXED_EXTRACTIONS = csv
C:\Program Files\Splunk\etc\apps\search\local\props.conf KV_MODE = none
C:\Program Files\Splunk\etc\system\default\props.conf    LEARN_MODEL = true
C:\Program Files\Splunk\etc\system\default\props.conf    LEARN_SOURCETYPE = true
C:\Program Files\Splunk\etc\system\default\props.conf    LINE_BREAKER_LOOKBEHIND = 100
C:\Program Files\Splunk\etc\system\default\props.conf    MATCH_LIMIT = 100000
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DAYS_AGO = 2000
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DAYS_HENCE = 2
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DIFF_SECS_AGO = 3600
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_DIFF_SECS_HENCE = 604800
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_EVENTS = 256
C:\Program Files\Splunk\etc\system\default\props.conf    MAX_TIMESTAMP_LOOKAHEAD = 128
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_NOT_BREAK_AFTER =
C:\Program Files\Splunk\etc\system\default\props.conf    MUST_NOT_BREAK_BEFORE =
C:\Program Files\Splunk\etc\apps\search\local\props.conf NO_BINARY_CHECK = true
C:\Program Files\Splunk\etc\apps\search\local\props.conf REPORT-star2_2 = REPORT-star2_2
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION = indexing
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-all = full
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-inner = inner
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-outer = outer
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-raw = none
C:\Program Files\Splunk\etc\system\default\props.conf    SEGMENTATION-standard = standard
C:\Program Files\Splunk\etc\apps\search\local\props.conf SHOULD_LINEMERGE = False
C:\Program Files\Splunk\etc\apps\search\local\props.conf TIMESTAMP_FIELDS = detection_time
C:\Program Files\Splunk\etc\apps\search\local\props.conf TIME_FORMAT = %Y-%m-%d%H:%M:%S.%3Q
C:\Program Files\Splunk\etc\system\default\props.conf    TRANSFORMS =
C:\Program Files\Splunk\etc\system\default\props.conf    TRUNCATE = 10000
C:\Program Files\Splunk\etc\apps\search\local\props.conf category = Custom
C:\Program Files\Splunk\etc\apps\search\local\props.conf description = Comma-separated value format. star2 Eventlog
C:\Program Files\Splunk\etc\system\default\props.conf    detect_trailing_nulls = auto
C:\Program Files\Splunk\etc\apps\search\local\props.conf disabled = false
C:\Program Files\Splunk\etc\system\default\props.conf    maxDist = 100
C:\Program Files\Splunk\etc\system\default\props.conf    priority =
C:\Program Files\Splunk\etc\apps\search\local\props.conf pulldown_type = true
C:\Program Files\Splunk\etc\system\default\props.conf    sourcetype =

SH - inputs.conf

C:\Program Files\SplunkUniversalForwarder\etc\apps\uf_star2_srv\local\inputs.conf               [monitor://C:\TV\star2_LOG\*_TV.csv]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf                        _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\uf_star2_srv\local\inputs.conf               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf                        evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf                        evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf                        evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\uf_star2_srv\local\inputs.conf               host = TV_TEST
C:\Program Files\SplunkUniversalForwarder\etc\apps\uf_star2_srv\local\inputs.conf               index = tv
C:\Program Files\SplunkUniversalForwarder\etc\apps\uf_star2_srv\local\inputs.conf               sourcetype = tv:star2

[max8006]

Hi Valiquet!

It works - BUT i do not understand this. I always thought the UF only makes the input and has nothing to do with parsing, this is the job of the indexer or heavy forwarder.

Do you know which of the props parameters are the ones which make this work? - Is the full parsing made on the UF? -no?

Maybe you can explain it to me so that I will know it better - I start splunking one month ago. I like Splunk, but this problem takes me now 4 day - and was very frustrating.
Thanks a lot for helping!!!
Max

[valiquet]

Parsing don't always occurs at the same place depending of your architecture. If you have UF,HF,IDX and SHC it is more spread. You were missing the CSV settings it's only preparsing. Most of the other settings (date, line breaking, etc) are on the IDX. It's a bit of error and trial.
You can follow this wiki, it is mostly accurate. https://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

It doesn't hurt to have the same props settings everywhere. Just don't do that with indexes.conf, server, etc. Also, _internal is a great source to TB shoot that.

Cheers

[gjanders]

Not quite the wording on the documentation is;

Caveats to extracting fields from
structured data files Splunk software
does not parse structured data that
has been forwarded to an indexer

Refer to this page or for the more technical details refer to the wiki page you are effectively doing header extraction/structured parsing and will go down the structured parsing queue on the universal forwarder

[gjanders]

Also refer to https://www.aplura.com/wp-content/uploads/where_to_put_props.pdf

[max8006]

Very cool -> you can see that the indexer is actually the last place to place the props.conf.

[valiquet]

I would copy everything from local\props.conf to local\props.conf on UF! If it doesn't work, search for

index=_internal sourcetype=splunkd "tv:star2"

[max8006]

btool output props.conf on IDX

[tv:star2]
ADD_EXTRA_TIME_FIELDS = True
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
CHARSET = AUTO
DATETIME_CONFIG =
DEPTH_LIMIT = 1000
FIELD_DELIMITER = ,
FIELD_NAMES = id,detection_time,log,parameter_list,text_english,error,sequence,roc_name,file_name,error_position,error_host,ack_required,acknowledged,ack_time,based_on_id,ack_by_user,job_id,recipient,destination,sender,subject,computed_size,spool
FIELD_QUOTE = '
HEADER_MODE =
INDEXED_EXTRACTIONS = csv
KV_MODE = none
LEARN_MODEL = true
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MATCH_LIMIT = 100000
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
REPORT-star2_2 = REPORT-star2_2
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = False
TIMESTAMP_FIELDS = detection_time
TIME_FORMAT = %Y-%m-%d%H:%M:%S.%3Q
TRANSFORMS =
TRUNCATE = 10000
category = Custom
description = Comma-separated value format. star2 log
detect_trailing_nulls = auto
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =

btool output inputs.conf on UF

[monitor://C:\TV\star2_LOG\*_TV.csv]
_rcvbuf = 1572864
disabled = false
evt_dc_name =
evt_dns_name =
evt_resolve_ad_obj = 0
host = TV_TEST
index = tv
sourcetype = tv:star2

[valiquet]

Can you --debug btool ?

[valiquet]

It would be easier to compare. It looks like your indexer is configure to ingest CSV but not your forwarder. Use debug and Ill tell you what to move

[valiquet]

Can you show us inputs props on FW and IDX?