Getting Data In

What's the fastest way to inject data to HTTP Event Collector, from a PERL script?

mdonnelly_splun
Splunk Employee
Splunk Employee

I've been searching for a generic example of how to bring data from a perl script, into Splunk using HEC, including HTTPS.

Use dedicated HEC libraries for PERL?

Have the perl script call the curl commands that are found in the HEC docs?

Or use something perl-native, really lightweight and readily available?

0 Karma
1 Solution

mdonnelly_splun
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

After testing functionality and performance - one good option is REST::Client

The example file below is useful for the use case of, testing performance of injection from perl. On my test VM, the code below inserted 20,000 events per minute into Splunk 7.3.1.

Of course, this example's real purpose is to show you how to establish the connection, disable SSL verification if needed, and how to inject using POST routines. In a real script, you'd set the JSON content in the $client->POST commands. (Fancy coders might even use additional perl libraries to build up the JSON.)

#!/usr/bin/perl -w
use strict;
use v5.16.3;
use REST::Client;
my $token = '7a49c676-22b8-40e3-8e37-9dad19a9bdb2';
my $hec_uri = 'https://127.0.0.1:8088';
my $n=3600;  # 3600 is one insert every 1 second for 1 hour

my $client = REST::Client->new();
$client->getUseragent()->ssl_opts(verify_hostname => 0);
$client->getUseragent()->ssl_opts(SSL_verify_mode => 'SSL_VERIFY_NONE');
$client->setHost($hec_uri);
$client->addHeader("Authorization", "Splunk $token");

$client->POST('/services/collector/event', '{"sourcetype": "mysourcetype", "event":"my payload here"}');  ## initial test
print $client->responseContent();

print "\n==========================";
print "\nTiming $n runs using direct perl libs: ";
my $start=time();
my $i=0;
while ($i++ <=$n)
{
        $client->POST('/services/collector', "{\"sourcetype\": \"mysourcetype\", \"event\":\"testrun=$$ speed TEST #$i\"}");  
}
my $end=time();
my $elapsed=$end-$start;
print "took $elapsed seconds\n";

View solution in original post

mdonnelly_splun
Splunk Employee
Splunk Employee

The Splunk Product Best Practices team provided this response. Read more about How Crowdsourcing is Shaping the Future of Splunk Best Practices.

After testing functionality and performance - one good option is REST::Client

The example file below is useful for the use case of, testing performance of injection from perl. On my test VM, the code below inserted 20,000 events per minute into Splunk 7.3.1.

Of course, this example's real purpose is to show you how to establish the connection, disable SSL verification if needed, and how to inject using POST routines. In a real script, you'd set the JSON content in the $client->POST commands. (Fancy coders might even use additional perl libraries to build up the JSON.)

#!/usr/bin/perl -w
use strict;
use v5.16.3;
use REST::Client;
my $token = '7a49c676-22b8-40e3-8e37-9dad19a9bdb2';
my $hec_uri = 'https://127.0.0.1:8088';
my $n=3600;  # 3600 is one insert every 1 second for 1 hour

my $client = REST::Client->new();
$client->getUseragent()->ssl_opts(verify_hostname => 0);
$client->getUseragent()->ssl_opts(SSL_verify_mode => 'SSL_VERIFY_NONE');
$client->setHost($hec_uri);
$client->addHeader("Authorization", "Splunk $token");

$client->POST('/services/collector/event', '{"sourcetype": "mysourcetype", "event":"my payload here"}');  ## initial test
print $client->responseContent();

print "\n==========================";
print "\nTiming $n runs using direct perl libs: ";
my $start=time();
my $i=0;
while ($i++ <=$n)
{
        $client->POST('/services/collector', "{\"sourcetype\": \"mysourcetype\", \"event\":\"testrun=$$ speed TEST #$i\"}");  
}
my $end=time();
my $elapsed=$end-$start;
print "took $elapsed seconds\n";
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...