C# SDK, Service.Connect(), System.Net.WebException: The operation has timed out

[afd0174]

Hi,

I am observing an intermittent exception "System.Net.WebException - The operation has timed out" occurring within a C# Splunk SDK program I have deployed as a CLR stored procedure within SQL Server 2008 R2. For background, I have created a class with two methods:

1. InitializeSplunkConnection (patterned after the examples/authenticate solution)
   
2. SubmitEventToSplunk (patterned after the examples/submit solution)

The exception can occur within either method. The class is included below. I am providing credentials via a .splunkrc file. I've verified that these credentials can be used to authenticate against the splunkd admin interface on 8089. This code had been working reliably for some time during testing and only recently began throwing the exception. I have verified that nothing has changed in either the Splunk instance or the machine running SQL Server. Restarting SQL Server will generally fix the problem temporarily, perhaps allowing for a successful invocation or two of either method, but then the timeout problem returns.

As I stated, I'm deploying each of the methods in this class as a stored procedure (via the SQL statements required to first create all necessary assemblies and then the procedures):

1. InitializeSplunkConnection -> sp_InitializeSplunkConnection
2. SubmitEventToSplunk -> sp_SubmitEventToSplunk

A third stored procedure is then calling these stored procedures roughly as:

-- INITIALIZE CONNECTION

EXEC sp_InititalizeSplunkConnection(@pathToSplunkRCFile);

...

-- LOOP OVER RECORDS IN TABLE AND SUBMIT EACH ONE TO SPLUNK INDEX

...

EXEC sp_SubmitEventToSplunk(@source, @sourceType, @index, @record);

I'm currently using Splunk SDK version 1, downloaded ~July 2013. I see there is a 1.0.1 SDK version now. I'm wondering if there have been any fixes implemented there that might benefit me. Thanks for any assistance.

Best regards,

-Andy

[3/10/2014 Edit]:

Some additional information about the problem I am observing. After restarting my MSSQL service (presumably resetting some aspect of my network connection to Splunk?), I am able to make multiple manual invocations the initialize connection procedure as:

EXEC sp_InititalizeSplunkConnection(@pathToSplunkRCFile);

Each manual invocation above behaves normally (e.g. returning almost instantaneously and yielding the message "Splunk instance initialized for instance specified in "+pathToDotSplunkRCFile). If I then invoke the third stored procedure referenced above, seeking to do a bulk record-at-a-time insert of data to a splunk index, the procedure will end with a timeout on what appears to consistently be the 101st invocation of the procedure sp_SubmitEventToSplunk (i.e. I observer 100 records successfully added to the index before the procedure ends with the timeout error). I've repeated this experiment a number of times, and each time I observe 100 records added before the timeout. I'm wondering if this behavior might be resulting from a Splunk configuration parameter.

namespace Splunk.Examples.Plugin
{
     using System;
     using System.Threading;
     using Splunk;
     using SplunkSDKHelper;
     using System.Data;
     using System.Data.SqlClient;
     using System.Data.SqlTypes;
     using Microsoft.SqlServer.Server;

     public class Program
     {
        public static Command cs_cli;

        [SqlProcedure()]
        public static void InitializeSplunkConnection(String parameterFilePath)
        {
            s_cli = new Command("plugin").Init();
            s_cli.Load(parameterFilePath);

            // Test connection
            Service.Connect(s_cli.Opts);
            SqlContext.Pipe.Send("Splunk instance initialized for instance specified in "+parameterFilePath);
        }

        [SqlProcedure()]
        public static void SubmitEventToSplunk(String splunkIndex, String splunkSource, String splunkSourceType, String eventMessage)
        {
           if (s_cli == null)
           {
               throw new Exception("Splunk connection must be initialized via InitializeSplunkConnection().");
           }

           var service = Service.Connect(s_cli.Opts);
           var args = new ReceiverSubmitArgs
           {
              Source = splunkSource,
              SourceType = splunkSourceType,
              Index = splunkIndex,
           };

           var receiver = new Receiver(service);

           // Submit to specified index
           receiver.Submit(args, eventMessage);

        }

}

[ConnorG]

I also am having the same issue and noticed that the receiver always times out after submitting 100 events.

Have you figured out how to fix this issue?

[David_Noble_at_]

Andy,

Would you also submit a GitHub issue that we can use for tracking your problem? Thanks.

Best,
David Noble

[David_Noble_at_]

Andy,

Your issue doesn't seem to be related to Splunk configuration parameters. Would it be possible for you to reproduce the problem outside of SQL server? If you send the program and some test data I would be happy to look at it.

Thanks for using Splunk.

Best,

David Noble