Goodday,
I am a Newbie.
Am trying to ingest a stock price file into Splunk,
I open Splunk by using http://localhost:8000/en-US/...
This is the format of the stock price file:
01/08/2001,15:45,1255.50,1257.00,1251.50,1255.25,2099,0
01/08/2001,16:00,1255.25,1256.50,1248.25,1253.25,2227,0
01/08/2001,16:15,1253.25,1259.00,1248.25,1250.00,2642,0
01/08/2001,16:30,1249.75,1253.25,1248.25,1251.25,1791,0
01/08/2001,16:45,1251.50,1258.75,1251.25,1255.50,1726,0
Date format is mm/dd/yyyy and time format is HH:mm
So I tried Timestamp format: %m/%d/%Y,%H:%M (is this the correct approach?)
The other fields are: opening price, highest price of the day, lowest price of the day, closing price, volume, open interest
Now I receive a message "timestamp outside acceptable".
So I suspect I need to increase the MAX_DAYS_AGO, because this value is too small.
But which props.conf needs to be updated? I can see several props.conf files.
Thanks for your helps, guys,
Kind regards,
Aad
I just found the solution. Maybe interesting for other newbies.
WIndows 10 laptop:
Location: $SPLUNK_HOME\etc\system\local ( for example: C:\Program Files\Splunk\etc\system\local )
File: props.conf
Add to props.conf:
[sourcetype]
max_days_ago =1000
for example:
[import_stockfile]
max_days_ago =1000
Restart Splunk. Importing a stock price file should work now.
@aadbosma, If your problem is resolved please accept an answer to help future readers.
Hi aadbosma,
try with this props.conf
[my_sourcetype]
DATETIME_CONFIG =
FIELD_NAMES = date, time, opening_price, highest_day_price, lowest_day_price, closing_price, volume, open_interest
HEADER_FIELD_DELIMITER = \s
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = date,time
TIME_FORMAT = %m/%d/%Y,%H:%M
category = Structured
description = Comma-separated value format
disabled = false
pulldown_type = true
Bye.
Giuseppe
Thanks guys.
Just created my first app and called it "Importing Tradestation Price Files"
And took Guiseppe's props.conf file, added max_days_ago =10000 and put it in
C:\Program Files\Splunk\etc\apps\ImportingTradestationPriceFiles\local
But how can I configure the app itself?
How do I tell the app the location of the props.conf file?
How do I tell the app the location of the price file?
And how do I tell the app what Splunk index to use?
I am asking this because when I use "add data" and "upload" I don't see the app I've just created.
Or should I bypass the "add data" and "upload" and only use the app?
Thanks again!
Aad
Hi aadbosma,
answering to your questions:
But how can I configure the app itself?
if you modify your app by gui, you don't need to restart Splunk,
if instead you modify config files (probably you modified props.conf) you need to restart Splunk.
How do I tell the app the location of the props.conf file?
props.conf file location is $SPLUNK_HOME/etc/apps/my_app/local
How do I tell the app the location of the price file?
if price file is on the same server where is Splunk you can put inputs.conf file in the same directory of props.conf, if instead is in another server you have to use a Universal Forwarder on this server.
If price file is on the same server, you can do this by gui.
And how do I tell the app what Splunk index to use?
you choose index in inputs.conf stanza
If price file is on the same server, you can do this by gui.
Bye.
Giuseppe
After you edit a config file you should restart Splunk so it will recognize the new file. Then the app will find its props.conf file (it looks in its own local and default directories).
If the file will be monitored for changes, tell the app where the input file is via the inputs.conf file.
[monitor://C:\foo\bar\prices.csv]
index = something
sourcetype = my_sourcetype
If the file will not be monitored, you can use the Add Data wizard to read the file. The wizard should offer your app as a choice once you've restarted Splunk.
You seem to be taking the right approach. The props.conf file to update is the one for the app the data will belong to. The default app is search, but you'll want to create your own app to hold your own knowledge objects. Create your own app by opening the app manager and clicking the "Create app" button.
When you edit props.conf, be sure to change the one in the 'local' directory. Never change a file in the 'default' directory.
Thanks again for your help!
Tried to create an app and configuring it, but didn't succeed.
Because I am really a newbie in this line of business, I decided to start very simple and trying to build from there
It was possible to add the data to Splunk by creating a custom sourcetype, but still find the max_days_ago to be a problem.
Ingesting data after 2013 was not problem, but ingesting data before June 2012 still is.
I tried modifying several props.conf files and added MAX_DAYS_AGO=10000.
And added a props.conf file to:
C:\Program Files\Splunk\etc\system\local\
with content:
MAX_DAYS_AGO=10000
This didn't do the trick either.
So my question is: what props.conf file in which directory should I modify and add MAX_DAYS_AGO=10000 ?
This is what I see on my Splunk. Five datapoints. Points 3, 4 and 5 are correct. Points 1 and 2 are older than 2000 days ago and there is an error message and no timestamp
1
The TIME_FORMAT specified is matching timestamps (Mon Jan 8 15:45:00 2001) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.
Failed to parse timestamp. Defaulting to file modtime.
11/24/17 5:21:56.000 PM
01/08/2001,15:45,1255.50,1257.00,1251.50,1255.25,2099,0
timestamp = none
2
The TIME_FORMAT specified is matching timestamps (Mon Aug 22 22:00:00 2011) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.
Failed to parse timestamp. Defaulting to file modtime.
11/24/17 5:21:56.000 PM
08/22/2011,22:00,968.50,969.00,959.50,963.00,99994,124855
timestamp = none
3
8/28/12 9:00:00.000 PM
08/28/2012,21:00,1274.75,1275.00,1273.25,1274.00,5257,8613
4
12/7/15 4:15:00.000 PM
12/07/2015,16:15,2034.50,2035.00,2030.50,2032.75,38813,45919
5
11/17/17 9:00:00.000 PM
11/17/2017,21:00,2578.00,2579.50,2576.25,2577.75,58786,75719
Thanks again and have a good weekend.
Aad