After creating defining a new sourcetype in Splunk Cloud, why are the specified fields not being extracted for a PostgreSQL CSV log type?

[akurilin]

Splunk Cloud user here. I have defined a new sourcetype with the following properties:

Name: postgresql_csv
Destination app: Distributed Management Console
Category: Database
Description: PostgreSQL CSV log
Indexed Extractions: csv
Timestamp:
  Extraction: Auto
Delimited settings:
  Field delimiter: ,
  Quote character: "
  File preamble: (empty)
  Field names: Custom
    Comma separated field names: log_time,user_name,database_name,process_id,connection_from,session_id,session_line_num,command_tag,session_start_time,virtual_transaction_id,transaction_id,error_severity,sql_state_code,message,detail,hint,internal_query,internal_query_pos,con,query,query_pos,location,application_name

Then I conduct a search for sourcetype=postgresql_csv, but it doesn't look like any of the fields I defined above appear in the fields bar on the left. After a bit of searching, 40+ new fields pop up, all with approximately 0% coverage, fields such as "database" or "average" or "AND_classid", which are AFAICT just guessed extractions from the actual log text, but aren't particularly useful.

Sample CSV below:

2015-08-30 00:50:31.539 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,2,"authentication",2015-08-30 00:50:31 UTC,2/1713,0,LOG,00000,"connection authorized: user=classroom_api database=classroom_dev",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,3,"PARSE",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.106 ms  parse <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,4,"BIND",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms  bind <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,5,"SET",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms  execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2

What am I doing wrong here? I read in a couple of posts that it's not possible to disable CSV auto-learning without digging into props.conf, which I don't have access to in Splunk Cloud. Please advise.

[yannK · Accepted solution]

INDEXED_EXTRACTIONS only works when ingesting local files.
So you want to setup your props.conf with the sourcetype definition on the forwarder.

[akurilin]

Looks like your suggestion works, I added a props.conf with a custom source type to system/local.

On that note, if I want to add a few custom field extractions to this type that only exists in props.conf at this point, do I create a sourcetype with matching name on my Splunk Cloud instance and add those extractions there?

[richgalloway]

Does your CSV have a header line? CSV parsing works much better if there is a header.

[akurilin]

I don't think postgres ever leaves the header in the file itself, so that wouldn't be a possibility. I thought the system was able to use column ordering for identifying which column it's working with.

[Richfez]

Could you add a line or two of the log file you are applying this to? Thanks.

[akurilin]

Sure thing, done!