Getting Data In

Trying to ingest a stock price file into Splunk

aadbosma
New Member

Goodday,

I am a Newbie.
Am trying to ingest a stock price file into Splunk,
I open Splunk by using http://localhost:8000/en-US/...

This is the format of the stock price file:
01/08/2001,15:45,1255.50,1257.00,1251.50,1255.25,2099,0
01/08/2001,16:00,1255.25,1256.50,1248.25,1253.25,2227,0
01/08/2001,16:15,1253.25,1259.00,1248.25,1250.00,2642,0
01/08/2001,16:30,1249.75,1253.25,1248.25,1251.25,1791,0
01/08/2001,16:45,1251.50,1258.75,1251.25,1255.50,1726,0

Date format is mm/dd/yyyy and time format is HH:mm
So I tried Timestamp format: %m/%d/%Y,%H:%M (is this the correct approach?)
The other fields are: opening price, highest price of the day, lowest price of the day, closing price, volume, open interest

Now I receive a message "timestamp outside acceptable".
So I suspect I need to increase the MAX_DAYS_AGO, because this value is too small.
But which props.conf needs to be updated? I can see several props.conf files.

Thanks for your helps, guys,

Kind regards,
Aad

0 Karma

aadbosma
New Member

I just found the solution. Maybe interesting for other newbies.

WIndows 10 laptop:

Location: $SPLUNK_HOME\etc\system\local ( for example: C:\Program Files\Splunk\etc\system\local )

File: props.conf

Add to props.conf:
[sourcetype]
max_days_ago =1000

for example:
[import_stockfile]
max_days_ago =1000

Restart Splunk. Importing a stock price file should work now.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

@aadbosma, If your problem is resolved please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aadbosma,
try with this props.conf

[my_sourcetype]
DATETIME_CONFIG = 
FIELD_NAMES = date, time, opening_price, highest_day_price, lowest_day_price, closing_price, volume, open_interest
HEADER_FIELD_DELIMITER = \s
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = date,time
TIME_FORMAT = %m/%d/%Y,%H:%M
category = Structured
description = Comma-separated value format
disabled = false
pulldown_type = true

Bye.
Giuseppe

0 Karma

aadbosma
New Member

Thanks guys.

Just created my first app and called it "Importing Tradestation Price Files"

And took Guiseppe's props.conf file, added max_days_ago =10000 and put it in
C:\Program Files\Splunk\etc\apps\ImportingTradestationPriceFiles\local

But how can I configure the app itself?
How do I tell the app the location of the props.conf file?
How do I tell the app the location of the price file?
And how do I tell the app what Splunk index to use?

I am asking this because when I use "add data" and "upload" I don't see the app I've just created.
Or should I bypass the "add data" and "upload" and only use the app?

Thanks again!
Aad

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi aadbosma,
answering to your questions:
But how can I configure the app itself?
if you modify your app by gui, you don't need to restart Splunk,
if instead you modify config files (probably you modified props.conf) you need to restart Splunk.

How do I tell the app the location of the props.conf file?
props.conf file location is $SPLUNK_HOME/etc/apps/my_app/local

How do I tell the app the location of the price file?
if price file is on the same server where is Splunk you can put inputs.conf file in the same directory of props.conf, if instead is in another server you have to use a Universal Forwarder on this server.
If price file is on the same server, you can do this by gui.

And how do I tell the app what Splunk index to use?
you choose index in inputs.conf stanza
If price file is on the same server, you can do this by gui.

Bye.
Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

After you edit a config file you should restart Splunk so it will recognize the new file. Then the app will find its props.conf file (it looks in its own local and default directories).

If the file will be monitored for changes, tell the app where the input file is via the inputs.conf file.

[monitor://C:\foo\bar\prices.csv]
index = something
sourcetype = my_sourcetype

If the file will not be monitored, you can use the Add Data wizard to read the file. The wizard should offer your app as a choice once you've restarted Splunk.

---
If this reply helps you, Karma would be appreciated.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

You seem to be taking the right approach. The props.conf file to update is the one for the app the data will belong to. The default app is search, but you'll want to create your own app to hold your own knowledge objects. Create your own app by opening the app manager and clicking the "Create app" button.

When you edit props.conf, be sure to change the one in the 'local' directory. Never change a file in the 'default' directory.

---
If this reply helps you, Karma would be appreciated.
0 Karma

aadbosma
New Member

Thanks again for your help!

Tried to create an app and configuring it, but didn't succeed.
Because I am really a newbie in this line of business, I decided to start very simple and trying to build from there

It was possible to add the data to Splunk by creating a custom sourcetype, but still find the max_days_ago to be a problem.
Ingesting data after 2013 was not problem, but ingesting data before June 2012 still is.

I tried modifying several props.conf files and added MAX_DAYS_AGO=10000.

And added a props.conf file to:
C:\Program Files\Splunk\etc\system\local\
with content:
MAX_DAYS_AGO=10000

This didn't do the trick either.

So my question is: what props.conf file in which directory should I modify and add MAX_DAYS_AGO=10000 ?

This is what I see on my Splunk. Five datapoints. Points 3, 4 and 5 are correct. Points 1 and 2 are older than 2000 days ago and there is an error message and no timestamp

1
The TIME_FORMAT specified is matching timestamps (Mon Jan 8 15:45:00 2001) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.
Failed to parse timestamp. Defaulting to file modtime.

11/24/17 5:21:56.000 PM
01/08/2001,15:45,1255.50,1257.00,1251.50,1255.25,2099,0
timestamp = none

2
The TIME_FORMAT specified is matching timestamps (Mon Aug 22 22:00:00 2011) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.
Failed to parse timestamp. Defaulting to file modtime.

11/24/17 5:21:56.000 PM
08/22/2011,22:00,968.50,969.00,959.50,963.00,99994,124855
timestamp = none

3
8/28/12 9:00:00.000 PM

08/28/2012,21:00,1274.75,1275.00,1273.25,1274.00,5257,8613

4
12/7/15 4:15:00.000 PM

12/07/2015,16:15,2034.50,2035.00,2030.50,2032.75,38813,45919
5
11/17/17 9:00:00.000 PM
11/17/2017,21:00,2578.00,2579.50,2576.25,2577.75,58786,75719

Thanks again and have a good weekend.
Aad

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...