Getting Data In

C# API Query TImeOut

michaudel
Explorer

Hello,
I edited the c# splunk API search example and made it into the method below. It basically creates a connection then does a search and puts all the results into a list of csv. Which i use later.

the problem i am having is if the search takes a while to run, the connection gets closed so when i to too read each line from the stream i get an error saying "The request was aborted".

is there a way to set the timeout value?

or can i create the search, wait for it to finish, then re-connect to pull down the results.

   public static List<string> SplunkSearch(string strQuery)
    {
        var cli = Command.Splunk("search");
        cli.AddRule("search", typeof(string), "search string");

        cli.Opts.Add("host", "host");
        cli.Opts.Add("port", "8089");
        cli.Opts.Add("scheme", "https");
        cli.Opts.Add("username", "uname");
        cli.Opts.Add("password", "pwd");
        cli.Opts.Add("search",strQuery);
        cli.Opts.Add("Timeout", "1000");

        var service = Service.Connect(cli.Opts);

        var jobs = service.GetJobs();
        var job = jobs.Create((string)cli.Opts["search"]);

        while (!job.IsDone)
        {
            Thread.Sleep(1000);
        }



        var outArgs = new Args
        {
            { "output_mode", "csv" },

            // Return all entries.
            { "count", "0" }
        };
        int intRun = 0;
        string columnnames = "";
        List<string> lstResults = new List<string>();
        using (var stream = job.Results(outArgs))
        {
            using (StreamReader reader = new StreamReader(stream))
            {
                while (!reader.EndOfStream)
                {
                    lstResults.Add(reader.ReadLine());
                }
            }
        }

        return lstResults;
    }
Tags (2)
1 Solution

ywu
Splunk Employee
Splunk Employee

In your code, at the time the stream is read, search has finished already. It is a separate httpwebrequest to get the result. It is essentially what you said -- "wait for it to finish, then re-connect to pull down the results."

Do you have to more detailed info to share?

What many lines are there in the search result?

View solution in original post

0 Karma

ywu
Splunk Employee
Splunk Employee
0 Karma

ywu
Splunk Employee
Splunk Employee

In your code, at the time the stream is read, search has finished already. It is a separate httpwebrequest to get the result. It is essentially what you said -- "wait for it to finish, then re-connect to pull down the results."

Do you have to more detailed info to share?

What many lines are there in the search result?

0 Karma

ywu
Splunk Employee
Splunk Employee

Chunking the result set should be a good approach. There's first class support for this in Splunk REST API. Refer to 'count' and 'offset' parameters of

GET search/jobs/{search_id}/results

on

http://docs.splunk.com/Documentation/Splunk/5.0.2/RESTAPI/RESTsearch#search.2Fjobs.2F.7Bsearch_id.7D...

Using C# SDK, you can supply the two parameters using Args object which is a dictionary of name and value pairs. Let me know if you have any questions on this.

Btw, I am interested in exactly where it fails in your case now. So please send me any other info about the error if any.

0 Karma

michaudel
Explorer

Thank you for the explanation on the http request part. I think you are on to something, i think it is the size of result that is getting me. When i run a long query, but bucket into hour of day (so 24 results) it works even though the query took 5 mins. If I run another query which returns about 10K results,it aborts the connection. I am guessing i need to chunk my result set. Ask for 1K at a time or something?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...