About richardAtOmni

richardAtOmni · ‎03-09-2021

Unfortunately, we never did find a solution to this.

richardAtOmni · ‎11-20-2020

I would also appreciate more information on using summary indexes to store "transformed" data. The documentation on summary indexes all describe using it to store summary information. Is there any difference?

richardAtOmni · ‎05-31-2018

Thanks! Appreciate the response. Our team is in the process of verifying that this will work for us. I'll update once I hear back.

richardAtOmni · ‎05-30-2018

Hello, We have built a data model, and have defined a pivot search against it. It is working great through the search API, and we have built a dashboard using it as well. When in the search interface or the dashboard, we have a time-range picker that can limit the time range. This works as expected. Now, we need to run the same query through the REST API. For a regular query, we would use earliest or latest to specify the time range without the timerange picker control. But these don't work with the pivot search command. We need to know how to specify the time range using the query language only, so that it applies to the pivot command. Can anyone help? Thanks! Richard

richardAtOmni · ‎02-16-2018

We recreated it with a different transform name and also changed some other stuff, and it showed up. Not sure what fixed it. Thanks for your help!

richardAtOmni · ‎02-15-2018

Thanks. That does show me all the lines and the source files. But I'm still missing the transform that's associcated with my "delimited field extraction". The extraction is named REPORT-delimited-RivaActCp4, but I can't find the transforms that this uses anywhere. It's supposed to define the delimiter character and each of the field names. I can see that the extraction is working because all the fields are available in Splunk. But I need to extract the actual transform from the file system. This is the output to the command with the double dashes: C:\Program Files\Splunk\etc\apps\search\local\props.conf [RivaActCpt4] C:\Program Files\Splunk\etc\system\default\props.conf ADD_EXTRA_TIME_FIELDS = True C:\Program Files\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True C:\Program Files\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE = C:\Program Files\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True C:\Program Files\Splunk\etc\system\default\props.conf CHARSET = AUTO C:\Program Files\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\ datetime.xml C:\Program Files\Splunk\etc\system\default\props.conf DEPTH_LIMIT = 1000 C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-crmType = ^(?:[ ^|]*\|){2}.*?\[(?P<crmType>\w+) C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-cycleId = ^[^\. \n]*\.\w+\|(?P<cycleId>[^\|]+) C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-mailType = ^(?: [^|]*\|){2}.*?\[\w+\+(?P<mailType>\w+) C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-node = ^[^\)\n] *\)\s+\[(?P<node>[^\]]+) C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-pod = ^[^ \n]* $(?P<pod>[^$]+) C:\Program Files\Splunk\etc\apps\search\local\props.conf EXTRACT-version,runtime = ^(?:[^|]*\|){6}1\|17\|Agent running - version:(?P<version>[^,]+), total runti me:\s+(?P<runtime>[^\|]+) C:\Program Files\Splunk\etc\apps\search\local\props.conf FIELDALIAS-RivaActCpt4 cycleId to referenceId = cycleId AS referenceId C:\Program Files\Splunk\etc\system\default\props.conf HEADER_MODE = C:\Program Files\Splunk\etc\system\default\props.conf LEARN_MODEL = true C:\Program Files\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true C:\Program Files\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100 C:\Program Files\Splunk\etc\apps\search\local\props.conf LOOKUP-auto_crmobjectty pelookup = CrmObjectTypeLookup crmObjectTypeId AS crmObjectTypeId OUTPUTNEW crmO bjectType AS crmObjectType C:\Program Files\Splunk\etc\apps\search\local\props.conf LOOKUP-auto_eventsubtyp elookup = EventSubTypeLookup eventSubTypeId AS eventSubTypeId OUTPUTNEW eventSub Type AS eventSubType C:\Program Files\Splunk\etc\apps\search\local\props.conf LOOKUP-auto_eventtypell ookup = EventTypeLookup eventTypeId AS eventTypeId OUTPUTNEW eventType AS eventT ype C:\Program Files\Splunk\etc\system\default\props.conf MATCH_LIMIT = 100000 C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_AGO = 2000 C:\Program Files\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2 C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 360 0 C:\Program Files\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 6 04800 C:\Program Files\Splunk\etc\system\default\props.conf MAX_EVENTS = 256 C:\Program Files\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128 C:\Program Files\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER = C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER = C:\Program Files\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE = C:\Program Files\Splunk\etc\apps\search\local\props.conf REPORT-delimited-RivaAc tCpt4 = REPORT-delimited-RivaActCpt4 C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION = indexing C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-all = full C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-inner = in ner C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-outer = ou ter C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none C:\Program Files\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard C:\Program Files\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE = True C:\Program Files\Splunk\etc\system\default\props.conf TRANSFORMS = C:\Program Files\Splunk\etc\system\default\props.conf TRUNCATE = 0 C:\Program Files\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto C:\Program Files\Splunk\etc\system\default\props.conf maxDist = 100 C:\Program Files\Splunk\etc\system\default\props.conf priority = C:\Program Files\Splunk\etc\system\default\props.conf sourcetype = Any other ideas?

richardAtOmni · ‎02-15-2018

I checked etc/system/local but there isn't even a transforms.conf file there. I ran the first command you specified and it lists out the props - basically everything I can see in the props.conf file. Looks something like this: [RivaActCpt4] ADD_EXTRA_TIME_FIELDS = True ANNOTATE_PUNCT = True AUTO_KV_JSON = true BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = True CHARSET = AUTO DATETIME_CONFIG = \etc\datetime.xml DEPTH_LIMIT = 1000 EXTRACT-crmType = ^(?:[^|]*\|){2}.*?\[(?P<crmType>\w+) EXTRACT-cycleId = ^[^\.\n]*\.\w+\|(?P<cycleId>[^\|]+) EXTRACT-mailType = ^(?:[^|]*\|){2}.*?\[\w+\+(?P<mailType>\w+) EXTRACT-node = ^[^\)\n]*\)\s+\[(?P<node>[^\]]+) EXTRACT-pod = ^[^ \n]* $(?P<pod>[^$]+) ion>[^,]+), total runtime:\s+(?P<runtime>[^\|]+) FIELDALIAS-RivaActCpt4 cycleId to referenceId = cycleId AS referenceId HEADER_MODE = LEARN_MODEL = true LEARN_SOURCETYPE = true LINE_BREAKER_LOOKBEHIND = 100 LOOKUP-auto_crmobjecttypelookup = CrmObjectTypeLookup crmObjectTypeId AS crmObj ctTypeId OUTPUTNEW crmObjectType AS crmObjectType LOOKUP-auto_eventsubtypelookup = EventSubTypeLookup eventSubTypeId AS eventSubT peId OUTPUTNEW eventSubType AS eventSubType LOOKUP-auto_eventtypellookup = EventTypeLookup eventTypeId AS eventTypeId OUTPU NEW eventType AS eventType MATCH_LIMIT = 100000 MAX_DAYS_AGO = 2000 MAX_DAYS_HENCE = 2 MAX_DIFF_SECS_AGO = 3600 MAX_DIFF_SECS_HENCE = 604800 MAX_EVENTS = 256 MAX_TIMESTAMP_LOOKAHEAD = 128 MUST_BREAK_AFTER = MUST_NOT_BREAK_AFTER = MUST_NOT_BREAK_BEFORE = REPORT-delimited-RivaActCpt4 = REPORT-delimited-RivaActCpt4 SEGMENTATION = indexing SEGMENTATION-all = full SEGMENTATION-inner = inner SEGMENTATION-outer = outer SEGMENTATION-raw = none SEGMENTATION-standard = standard SHOULD_LINEMERGE = True TRANSFORMS = TRUNCATE = 0 detect_trailing_nulls = auto maxDist = 100 priority = sourcetype = Running the second command just outputs a blank line. Any other suggestions?

richardAtOmni · ‎02-14-2018

Hi, Can someone please point me to where the delimiter based field extraction definitions are now stored in Splunk configuration files? Previously, we could see that these were stored in transforms.conf. I can see that the extractions we built before splunk was upgraded are there. (This is located in C:\Program Files\Splunk\etc\apps\search\local". After upgrading, we defined new field extractions for a new source type. They are working, but they don't appear in transforms.conf. I've done a text search on the whole machine and can't seem to find where the new definition is stored. I need to get these extractions so that I can export them to our production Splunk Cloud instance. Any assistance would be greatly appreciated. Thanks, Richard

richardAtOmni · ‎12-20-2017

Hello, I would like to be able to calculate the time difference between the last time parameter of the time range of the search and the _time value of the record. If the query is run from the current time, I would do something like this: eval runtime_raw=(now()-_time) But if the search time range end time is not now(), I want to replace now() in the above expression with something representing the end of the search time range. Does such a function exist? Or is there a way to calculate this? Thanks for your help, Richard

richardAtOmni · ‎06-05-2017

Thanks for your suggestion, but I don't think this quite addresses our scenario. I think your suggestion is to edit the regex used to extract the field to either include or exclude the slashes. Unfortunately, in this case, we are not using the regex extraction here. We are using a delimited extraction, with pipes as the delimiter. The field is just a text string with a log message, and sometimes we will output a UNC path in the log message. We want the UNC path to show up correctly in the resulting "message" field. And we don't necessarily want to extract a separate field just for these scenarios. So we are looking for suggestions to make the delimited extraction extract properly with respect to these slashes.

richardAtOmni · ‎05-31-2017

We use the HttpEventListener to input data into splunk. Our data is pipe ('|') delimited and we have setup field extractors using the "delimited" method. This works perfectly so long as there are no backslashes or pipes in our data. If there are backslashes or pipes, we've observed some odd behavior in Splunk. If there is a field that contains a double backslash, just as part of the data, such as could occur if we were to log a windows UNC path (i.e. "The network path path is: \serverA\folder1\subfolder"), in the "_raw" view, the data will look correct, but in the extracted field, Splunk will collapse the double backslash into a single backslash. My example message above would look like this: "The network path path is: \serverA\folder1\subfolder". This only happens in the extracted field however. In the _raw view, all the extra slashes are retained, which isn't ideal for our purposes. Is there a way to have both the raw view and the extracted field match showing the intended number of slashes? If a particular field ends in a backslash, it causes the following separator pipe character to become escaped and included in the field instead of treated as a separator, throwing off all the delimited field mappings. However if a field has a pipe character in it's actual data we can use the backslash to escape it to prevent it from being treated as a delimiter. This is great. However, the problem here is that in the extracted field, splunk still shows us the backslash character as though it was part of the data. For example, our _raw could be abc|def|12345 representing 2 fields, with values "abc|def" and "12345". But if we look at the extracted field value for field 1, we don't see "abc|def" we see "abc|def". The slash is used as an escape, but it is not removed from the result. This is inconsistent with what happens when the slash escapes another slash. This behavior has us scratching our heads as to how to properly handle special characters. Any advice would be greatly appreciated.

richardAtOmni · ‎02-02-2017

In case this helps the Splunk team find the problem, I was able to reproduce the problem this way: create the input as a "text" field, save. Edit the dashboard, and change the input type to multi-select. In the multi-select, define the different static values. Save. Edit the dashboard again, and change the input type to drop down. Save. Now, when I select one of the values, it gets added to the search with the extra spaces. Wierd! Hope this helps!

richardAtOmni · ‎02-02-2017

Thanks for your suggestions. If you haven't seen my other reply yet, the issue went away as soon as I left the splunk "source edit" mode. I have no idea why it happened, but it is resolved for me now. Thank you for your attention anyway!

richardAtOmni · ‎02-02-2017

Hi, the splunk version is: Splunk Version 6.5.1 Splunk Build f74036626f0c And yes, the XML was straight from the web editor.

richardAtOmni · ‎02-02-2017

In order to post this question, I opened my dashboard in "Edit Source" mode, so that I could paste the XML into this ticket. After posting, I exited "Edit Source" mode, and to my surprise, the inputs started working as expected, without the additional spaces! Maybe this is a bug in Splunk? But if anyone else encounters this, try entering "Edit Source" mode, then exiting. That seemed to have done the trick for me. Thanks!

richardAtOmni · ‎02-02-2017

Hi I'm trying to create a simple search input as a drop-down. I've defined static values for the drop-down. The problem I'm having is that each character in my value is being separated by a space when inserted into the search. Here's my entire search dashboard definition: <form> <label>Riva Cloud Diagnostics By Time</label> <fieldset submitButton="false"> <input type="time" token="time_tok"> <label></label> <default> <earliest>-1m</earliest> <latest>now</latest> </default> </input> <input type="dropdown" token="pod_tok" searchWhenChanged="true"> <label>pod</label> <default>*</default> <choice value="*">Any</choice> <choice value="na5">na5</choice> <choice value="xi5">xi5</choice> <choice value="yi5">yi5</choice> </input> </fieldset> <row> <panel> <table> <search> <query>index="rc-diagnostics" pod="$pod_tok$" sourcetype=RivaDiagnostics* | table "_time", "pod", "message", "referenceId", "exception", "errorDetails", "thread","processId", "levelInt", "userName" | sort -_time | head 500</query> <earliest>$time_tok.earliest$</earliest> <latest>$time_tok.latest$</latest> </search> <option name="count">30</option> <option name="wrap">true</option> </table> </panel> </row> </form> When one of my "pod_tok" values are selected, in the search the values are being added with extra spaces. For example, if I select 'na5' the resulting search looks like this: index="rc-diagnostics" pod="n a 5" sourcetype=RivaDiagnostics* | table "_time", "pod", "message", "referenceId", "exception", "errorDetails", "thread","processId", "levelInt", "userName" | sort -_time | head 500 Which of course is incorrect. How can I define my drop-down input properly so that the values are entered into my search without the extraneous spaces?

richardAtOmni · ‎02-02-2017

The delimited fields worked as expected after we changes our input format to be just a straight string, instead of a nested JSON object. So instead of this: {"data":"Omni.Riva.CrmAgentEx.exe (ci15)|NFRwlv #6218|3940170112032121|RC-SRV3||40000|14A21XFJS85PQ|NT AUTHORITY\SYSTEM|No sync required between CA and XA||Logger"} We have this: "Omni.Riva.CrmAgentEx.exe (ci15)|NFRwlv #6218|3940170112032121|RC-SRV3||40000|14A21XFJS85PQ|NT AUTHORITY\SYSTEM|No sync required between CA and XA||Logger" I'm not sure, but there seems to have been something about the nested JSON which prevented the parsing from working as expected? The original question of why this didn't work isn't quite answered, but we're good for our use cases to proceed, so I'll mark it closed. Thanks!

richardAtOmni · ‎02-02-2017

I checked these files, and the field extractor tool I used to define the delimited fields pretty much generated the same config that you suggest. However, it still doesn't work for some reason.

richardAtOmni · ‎01-31-2017

Hello, we are inputting data via the HTTP Event collector. The "event" member has this format, which we are trying to split into fields with the pipe delimiter: {"data":"Omni.Riva.CrmAgentEx.exe (ci15)|NFRwlv #6218|3940170112032121|RC-SRV3||40000|14A21XFJS85PQ|NT AUTHORITY\\SYSTEM|No sync required between CA and XA||Logger"} As I work through the field extraction definition tool, the delimiter properly splits out the fields. I work through and I rename each field the way I want. Then, I save the field extraction. I get a message saying that this was successful. Then I click on the link "explore the fields that I just extracted" (I'm paraphrasing from memory), then it takes me to a search with a filter on the sourcetype that I just defined the field extraction for. The problem is the search results do not show the new fields that I just defined. It only shows the first one. And as it's value it has the entire row as the value, as though none of the delimiters were recognized at all. For example, on the row above, if my first field was named "start", then it would have a value of: {"data":"Omni.Riva.CrmAgentEx.exe (ci15)|NFRwlv #6218|3940170112032121|RC-SRV3||40000|14A21XFJS85PQ|NT AUTHORITY\\SYSTEM|No sync required between CA and XA||Logger"} What am I missing? Thanks for any insight you can provide.

richardAtOmni · ‎11-28-2016

I figured out the problem. I had my test running through Fiddler, which was forcing it to renegotiate the SSL handshake for every request. Once I routed the test directly the response times were much better – 2ms or better. Thanks, Richard

richardAtOmni · ‎11-24-2016

Just an additional note: I measured the time each call takes to send and receive a response, and they seem to take approx. 1.35 seconds each. This is much slower than I would have expected. Is this normal?

richardAtOmni · ‎11-23-2016

I wrote a simple test to push logs to Splunk through the Http Event Collector, directly using .NET webclient. The code simply loops and pushes a web request to the collector on the Splunk instance running on my local machine. Running this test, I'm finding that I cannot get logs into Splunk any faster than a single log line per second. The code is very simple, so I'm sure that there aren't any delays on the client side. What I'm seeing is that it usually takes up to a full second for Splunk to respond with the "OK" response. I'm concerned this will become a performance bottleneck for my application. I'm wondering if this is the approximate expected response time per request. It almost seems like this is being intentionally throttled. If it is, is this configurable? I understand higher performance can be achieved by batching requests, or by spawning mutliple clients and issuing these requests in parallel. I want to know however what is the throughput limit that can be achieved on a per thread basis without batching. Any insight that can be provided would be appreciated. Thanks! Richard

Posts	22
Solutions	3
Karma Given	6
Karma Received	2
Member Since	‎11-23-2016

Online Status	Offline
Date Last Visited	‎03-09-2021 07:47 PM

How can we specify the timerange on a pivot based ...

What is the location of the delimiter based field ...

Calculate duration between end of search time rang...

How to properly handle backslashes in data

How can I define my drop-down input properly so th...

Why are delimited field extractions not working?

Is it possible to configure HTTP Event Collector f...

Re: How to properly handle backslashes in data

Re: As an already indexed field can't be removed, ...

Re: How can we specify the timerange on a pivot ba...

How can we specify the timerange on a pivot based ...

Re: What is the location of the delimiter based fi...

Re: What is the location of the delimiter based fi...

Re: What is the location of the delimiter based fi...

What is the location of the delimiter based field ...

Calculate duration between end of search time rang...

Re: How to properly handle backslashes in data

How to properly handle backslashes in data

Re: How can I define my drop-down input properly s...

Re: How can I define my drop-down input properly s...

Re: How can I define my drop-down input properly s...

Re: How can I define my drop-down input properly s...

How can I define my drop-down input properly so th...

Re: Why are delimited field extractions not workin...

Re: Why are delimited field extractions not workin...

Why are delimited field extractions not working?

Re: Is it possible to configure HTTP Event Collect...

Re: Is it possible to configure HTTP Event Collect...

Is it possible to configure HTTP Event Collector f...