Splunk Dev

Python SDK: search from job does not use all fields

ksander14
Engager

I have some source named "source1" with field named "field1". This field is not present in all events (field is filled only in 3 percent of events). So, I try to get events with this field using Splunk Python SDK using simple construction from documentation.

search_text = "source=source1 field1=* earliest=-1d@d latest=@d"
job = jobs.create(search_text)
while True:
while not job.is_ready():
pass
stats = {"isDone": job["isDone"],
"doneProgress": float(job["doneProgress"])*100,
"scanCount": int(job["scanCount"]),
"eventCount": int(job["eventCount"]),
"resultCount": int(job["resultCount"])}
status = ("\r%(doneProgress)03.1f%% %(scanCount)d scanned "
"%(eventCount)d matched %(resultCount)d results") % stats
sys.stdout.write(status)
sys.stdout.flush()
if stats["isDone"] == "1":
sys.stdout.write("\n\nDone!\n\n")
break
sleep(2)

But this search, run using Splunk Python SDK, return me 0 results. If I run this search in Splunk, I get right results - events with filled "field1". So, as I understand this situation, search in Splunk Python SDK doesn't see "field1" in my source.

How can I run search with Spunk Python SDK that see all fields in my source?

P.S. For example - if I run "search source=source1 earliest=-1d@d latest=@d | fieldsummary" in Splunk, I get information about 84 fields, in Splunk Python SDK - 81 fields

0 Karma

micahkemp
Champion

How is field1 defined? Is it a custom extraction you put in place? Is it shared globally?

Is the account you're using to log in to SplunkWeb the same account you're using for the python script?

I put your code into a python script and it seems to work just fine (I removed the timeframe from the search string for mine):

search_text = "search source=source1 field1=*"
job = service.jobs.create(search_text)
while True:
    while not job.is_ready():
            pass
    stats = {"isDone": job["isDone"],
        "doneProgress": float(job["doneProgress"])*100,
        "scanCount": int(job["scanCount"]),
        "eventCount": int(job["eventCount"]),
        "resultCount": int(job["resultCount"])}
    status = ("\r%(doneProgress)03.1f%% %(scanCount)d scanned "
        "%(eventCount)d matched %(resultCount)d results") % stats
    sys.stdout.write(status)
    sys.stdout.flush()
    if stats["isDone"] == "1":
        sys.stdout.write("\n\nDone!\n\n")
        break
    sleep(2)

burtica
New Member

Hi, I have the same problem.
I have also added the fields in the rf field of the search_kwargs as below, but still I am not returned any data, although there is data. The fields field1 and field2 are actually not returned.

search_kwargs = {
            "exec_mode": "normal",
            "earliest_time": "-8d",
            "latest_time": "now",
            "search_mode": "normal",
            "rf": ["field1", "field2"],
}
search = "search index=alfa | fields field1, field2"
job = jobs.create(search, **search_kwargs)
# Then pool for job completion, and list results....
0 Karma

burtica
New Member

Hi, I have the same problem.
I've updated the search to be search index=alfa | fields field1, field2. Also tried adding the rf field in search_kwargs, but still these fields are not returned, so my search returns no results (although there are of course results).

The code looks like this:

search_kwargs = {
            "exec_mode": "normal",
            "earliest_time": "-8d",
            "latest_time": "now",
            "search_mode": "normal",
            "rf": ["field1", "field2"],
}
search = "search index=alfa | fields field1, field2"
job = jobs.create(search, **search_kwargs)
# Then pool for job completion, and list results....
0 Karma

micahkemp
Champion

Change your search string in the script to search source=source1 field1=* earliest=-1d@d latest=@d.

The REST API, which the SDK uses, does not add the implied search command to the front of your search strings like the UI does.

Also, add | fields <field1> <field2>. This should ensure that Splunk grabs the fields you care about, even if running in fast mode (which may be what is happening here).

0 Karma

ksander14
Engager

Unfortunately, this did not help

0 Karma

micahkemp
Champion

Updated answer to include fields.

0 Karma

ksander14
Engager

Thanks for the advice, but it did not help 😞

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...