Getting Data In

How to send back to old data to new server in forwarder

chrismok
Path Finder

Hi All,

I created the new splunk server and found that the forwarder is only send the latest log to the new server.

Here's the inputs.conf setting in forwarder, and the host "new-splunkserver" is just created.

Have anyone know how to send back the old data to new splunk server?

[default]
host = forwarder-host

[monitor://D:\logs\...\*build.log]
disabled = false
followTail = 0
host = old-splunkserver
index = deploy_db
sourcetype = deploy_log

[monitor://D:\logs\...\*build.log]
disabled = false
followTail = 0
host = new-splunkserver
index = deploy_db
sourcetype = deploy_log
0 Karma
1 Solution

jrodman
Splunk Employee
Splunk Employee

FIRST:
In Splunk, you cannot meaningfully have two identically named configuration stanzas. When you have two stanzas with the same name, Splunk will combine them into one stanza.

So if you have

[my_sourcetype]
NO_BINARY_CHECK=True

[my_sourcetype]
SHOULD_LINEMERGE=False

Then you've just created one sourcetype with two settings, equivalent to this:

[my_sourcetype]
NO_BINARY_CHECK=True
SHOULD_LINEMERGE=False

but in a slightly harder to read manner.

Similarly, you have created one input definition, where the host is either new-splunkserver or old-splunkserver (I don't know which will win, you could check btool but it's not very interesting). You should have only one of these two stanzas.

SECOND:
Choosing a value for 'host=' in an input stanza does not control where data is sent. Instead, it controls how the data from that stanza is labelled. Change host here will claim that that data comes from new-splunkserver or old-splunkserver, when it clearly comes from forwarder-host.

In general, assigning per-stanza values for host is not appropriate unless you use log-aggregation and have data from many hosts stored in per-hostname directories or similar.

THIRD:

Splunk does not re-send data it has already sent (or your Splunk indexers would quickly fill the disks as we continuously resent the same things infinite numbers of times forever).

If you want to cause a forwarder to send data a second time that it has already sent, you have a few choices.

On a non-indexing forwarder (the normal case for a forwarder) you can cause everything to be resent once, by running the command splunk clean eventdata from the command line, which will cause the forwarder to forget all of its bookmarking state on the input files, triggering all data to be sent according to the current configuration that is available at the monitored locations.

If you want to resend specific files, you can add the data as a oneshot, for example from the commandline splunk add oneshot d:\logs\path\to\filename.log The oneshot command does not appear to offer a choice of sourcetypes, so you would need to have pattern rules in props.conf to ensure your deploy_log sourcetype is used.

A third option, which is a little dirty, is to modify the data. If you alter one of the bytes within the first 256 bytes of the file, Splunk will not recognize the file and send it again. This can be useful while troubleshooting, but is a little awkward for production data, in my opinion.

View solution in original post

jrodman
Splunk Employee
Splunk Employee

FIRST:
In Splunk, you cannot meaningfully have two identically named configuration stanzas. When you have two stanzas with the same name, Splunk will combine them into one stanza.

So if you have

[my_sourcetype]
NO_BINARY_CHECK=True

[my_sourcetype]
SHOULD_LINEMERGE=False

Then you've just created one sourcetype with two settings, equivalent to this:

[my_sourcetype]
NO_BINARY_CHECK=True
SHOULD_LINEMERGE=False

but in a slightly harder to read manner.

Similarly, you have created one input definition, where the host is either new-splunkserver or old-splunkserver (I don't know which will win, you could check btool but it's not very interesting). You should have only one of these two stanzas.

SECOND:
Choosing a value for 'host=' in an input stanza does not control where data is sent. Instead, it controls how the data from that stanza is labelled. Change host here will claim that that data comes from new-splunkserver or old-splunkserver, when it clearly comes from forwarder-host.

In general, assigning per-stanza values for host is not appropriate unless you use log-aggregation and have data from many hosts stored in per-hostname directories or similar.

THIRD:

Splunk does not re-send data it has already sent (or your Splunk indexers would quickly fill the disks as we continuously resent the same things infinite numbers of times forever).

If you want to cause a forwarder to send data a second time that it has already sent, you have a few choices.

On a non-indexing forwarder (the normal case for a forwarder) you can cause everything to be resent once, by running the command splunk clean eventdata from the command line, which will cause the forwarder to forget all of its bookmarking state on the input files, triggering all data to be sent according to the current configuration that is available at the monitored locations.

If you want to resend specific files, you can add the data as a oneshot, for example from the commandline splunk add oneshot d:\logs\path\to\filename.log The oneshot command does not appear to offer a choice of sourcetypes, so you would need to have pattern rules in props.conf to ensure your deploy_log sourcetype is used.

A third option, which is a little dirty, is to modify the data. If you alter one of the bytes within the first 256 bytes of the file, Splunk will not recognize the file and send it again. This can be useful while troubleshooting, but is a little awkward for production data, in my opinion.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...