Splunk Search

Splunk CLI and UI give different search results

psychogyiokosta
New Member

I index manually through UI the log file i wish to index (Data Inputs > Add new > Index Once) and select all the configurations (source, index, sourcetype, host) i complete the indexing and then run the following search in Splunk UI:

source= host= index= sourcetype=
| eval rex_template=
| cluster t= labelonly=true labelfield=Template match=termlist field=Content
| outputcsv

and then i do the exact same process using Splunk CLI:

For indexing i navigate to /bin directory of Splunk and run:

./splunk add oneshot -index -sourcetype -host -app

Then i run the same search:

./splunk search source= host= index= sourcetype=
| eval rex_template=
| cluster t= labelonly=true labelfield=Template match=termlist field=Content
| outputcsv

The raw log file contains 1 million lines. The structured file generated from the UI approach contains 1 million events too.
When i run the CLI approach the structured file will contain different number of events (e.g. 993,453 events instead of 1 million)

I have tried to add earliest/latest, or all time in my search, even configure [all_time] stanza from times.conf and add it in the local directory of the app i use to run my search, but nothing helped me solve the issue so far.

Any ideas what i'm doing wrong? Thank you.

Tags (1)
0 Karma

jeffrey_berry
Path Finder

@psychogyiokostas Based on your responses and other answers here, I would guess that both methods for ingesting the data (CLI and UI) is indexing all of the data, but during the indexing process, the data is ingested with the event break configuration for the sourcetype being different due to the process performing the ingestion using a different context. The props.conf file(s) controls the event breaks, but there are (can be) several props.conf files in a Splunk environment. See https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Wheretofindtheconfigurationfiles which explains the configuration (conf) file precedence. Also, the existence of multiple contexts (i.e global and app contexts) which follow different configuration (conf) file precedence does not make it easier.

At the CLI, you can execute the following splunk command to view the props.conf settings being used for the context at command prompt, but you will need to know which stanza applies for ingesting your data. Unfortunately, the btool command is not supported in the UI, but Splunkbase contains some "btool" apps that may be helpful. A Splunk feature request could be to ask for better tools to view configuration (conf) file precedence based on an user chosen context.

./splunk btool props list --debug

P.S. The Splunk Answers tag "CLI Auto for Splunk" is for an app that I developed and shared on Splunkbase. While I appreciate the visibility for the app, your question does not seem related to this app. The "CLI Auto for Splunk" app is an automation tool for executing CLI commands on any network device with a CLI (ssh) interface. You might try searching for other Splunk Answers tags that might be more related to your question.

0 Karma

psychogyiokosta
New Member

hello @jeffrey_berry
Thank you for the detailed response. I will test based on your suggestion and will update this thread. I changed the tab since "CLI Auto for Splunk" was not related to my query, thank you for enlightening me!

0 Karma

woodcock
Esteemed Legend

Use 4-leading spaces and a blank line on the top to mark your code as code; otherwise it gets treated as markup and things like asterisks get parsed away. The reason that you are getting different results is either:

1: You are running with a different timepicker (you are not using `earliest=x latest=y` in your SPL)
2: You are running as a different user
2a: You are not specifying "index=" and the "Indexes searched by default" is different between the 2 users.
2b: You are specifying "index=" but one user does not have permissions to see all/same values.
3: You are searching on difference servers: for example, logging onto an indexer for the CLI but the Search Head for the GUI.  To test this add `| stats count by splunk_server`

My bet is on #1.

0 Karma

psychogyiokosta
New Member

hello, thanks for the advice. I tried with earliest/latest too but is the same issue. i always log in Splunk as admin both when using UI or CLI and always specify same index values. For the CLI search if i rerun the events found are different everytime but never 1million as they should be.

0 Karma

woodcock
Esteemed Legend

I thought of another one, see my new #3 in my updated answer.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

When using the UI to search via your app, the search context is /opt/splunk/etc/apps/your_app_name/
When you use the CLI to search, the context is /opt/splunk/etc/apps/search/

The point is, when using the CLI to search you'll need to make sure that your sourcetype is available to the search context.
Verify that props.conf (and/or transforms.conf ) containing your sourcetype is available at /opt/splunk/etc/apps/search/local/

My guess is that your search is not being parsed correctly, causing the difference in result count.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

psychogyiokosta
New Member

hello, thanks for your reply! i used exactly the same conf files in /opt/splunk/etc/apps/search/ with the ones i use in /opt/splunk/etc/apps/your_app_name/ so that there is an "agreement" in terms of configuration both for CLI and UI, but problem persists. Something i noticed: In server.conf in path /opt/splunk/etc/apps/your_app_name/ i edit

cliLoginBanner = "custom message"

. When running via CLI i am asked to enter my credentials and i have "custom message" in my terminal screen as well. This means that CLI 's search context should be /opt/splunk/etc/apps/your_app_name/, except if server.conf is working in a different way than i think it is.

Also:

props.conf:

[hdfs_log] BREAK_ONLY_BEFORE =
MAX_DAYS_AGO = 100000 SHOULD_LINEMERGE
= true TRUNCATE = 0 DATETIME_CONFIG =

times.conf:

[all_time] label = All time
header_label = over all time
earliest_time = 0 order = 500

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...