Getting Data In

How to send events to the nullQueue on indexer?

dingesbr
Explorer

The strange thing is that I can send events to the nullQueue on my Local installation of Enterprise Splunk (6.2.2.5). Using the same app I created on the indexer for my Local Installation, I tried to use on our Production installation of Enterprise Splunk (6.2.2.2), and it did not work. The source type is a custom one we create on the forwarder, oracle:oag:trc, for the trace files.

Am I allowed to create an app on the Splunk Indexer that will send the events to the nullQueue? If, not, where is the correct place to put the "props.conf" and "transforms.conf" files so that the indexer will pick up the transforms during Indexing/Parsing?

My Local Topology is: Universal Forwarder => Splunk Instance

My Production Topology is: Universal Forwarder => Splunk Indexer/Deployer => Search Head

props.conf

[oracle:oag:trc]
TRANSFORMS-nullqueue-oag=nullqueue-oag-filter

transforms.conf

[nullqueue-oag-filter]
REGEX=^ERROR[\s\S]{0,100}(\bCardinality violation\b|\berror handling connection: peer disconnected unexpectedly\b)
DEST_KEY=queue
FORMAT=nullQueue
0 Karma
1 Solution

dingesbr
Explorer

Turns out a buggy version of the REGEX was deployey for the transform. We're noobs! Thanks everyone for their support!

We corrected our transforms in "/etc/apps/myapp/transform.conf"

We ran "https://myindexer:myport/en-US/debug/refresh" to get it loaded and verify via command line "splunk cmd btool transforms list --debug".

That fixed the issue for us.

View solution in original post

0 Karma

dingesbr
Explorer

Turns out a buggy version of the REGEX was deployey for the transform. We're noobs! Thanks everyone for their support!

We corrected our transforms in "/etc/apps/myapp/transform.conf"

We ran "https://myindexer:myport/en-US/debug/refresh" to get it loaded and verify via command line "splunk cmd btool transforms list --debug".

That fixed the issue for us.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The REGEX is definitely correct. And since you're using Universal forwarder, the correct location for props/transforms is on Indexer. Now, let's check the event breaking is correct OR not as the event filter works at invidual event level. Do you have other attribute defined on the props.conf for the sourcetype?

0 Karma

dingesbr
Explorer

This what was done on Production. On my Local Installation, previously the forwarder had all the details except the transform, but I changed it to the below but that still worked Locally.

The input in coming from TCP (from the forwarder).

For Production, the below is applied to both the indexer and the forwarder:

[oracle:oag:trc]
TIME_PREFIX=^(ERROR|DEBUG|INFO|DATA|FATAL)\s+
TIME_FORMAT=%d/%b/%Y:%H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD=35
SHOULD_LINEMERGE=True
BREAK_ONLY_BEFORE=^(ERROR|DEBUG|INFO|DATA|FATAL)
TRUNCATE=999999
TRANSFORMS-nullqueue-oag=nullqueue-oag-filter

0 Karma

somesoni2
SplunkTrust
SplunkTrust

We if you're using Universal forwarders, they can't do event parsing so props/transform their is dead code. They have to be there on Indexer OR heavy forwarders. Which location you're putting your props/transforms etc/system OR etc/apps/AppName?

0 Karma

dingesbr
Explorer

somesoni, I appreciate the time you spent on this. Can you confirm that "/etc/apps/AppName" is a correct location for a transform.conf? Some of the documentation doesn't mention "/etc/app/AppName" for transforms during index time? I am not real clear on what gets dedicated to search time versus index time. Again, I appreciate your assistance!

0 Karma

dingesbr
Explorer

+1

"/etc/apps/AppName" is where the props.conf and transforms.conf is being placed on the indexer.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you provide some sample events which you want to filter (to ensure the REGEX is correct)?

0 Karma

dingesbr
Explorer

+somesoni2
Sure, thanks.

Event 1:
ERROR 26/Aug/2015:23:59:59.655 [b2db7700] java exception:
com.vordel.es.EntityStoreException: Cardinality violation - field [timezoneAware] should have 1 values
at com.vordel.es.Field.checkCardinality(Field.java:232)
at com.vordel.es.Field.setValues(Field.java:147)
at com.vordel.store.util.env.EnvExpandEntity.expandField(EnvExpandEntity.java:112)
at com.vordel.store.util.env.EnvExpandEntity.(EnvExpandEntity.java:39)
at com.vordel.store.util.env.EnvExpandEntityStore.getEntity(EnvExpandEntityStore.java:119)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.end(ShorthandKeyFinder.java:155)
at com.vordel.es.util.ShorthandKeyFinder$PartFinder.process(ShorthandKeyFinder.java:91)
at com.vordel.es.util.ShorthandKeyFinder$SingleEntityPF.process(ShorthandKeyFinder.java:119)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:262)
at com.vordel.es.util.ShorthandKeyFinder.getEntity(ShorthandKeyFinder.java:248)
at com.vordel.common.db.DbConnectionCache.getConnection(DbConnectionCache.java:49)
at com.vordel.common.db.JDBCDataSource.getConnection(JDBCDataSource.java:50)
at org.apache.openjpa.lib.jdbc.DelegatingDataSource.getConnection(DelegatingDataSource.java:110)
at org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:87)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:982)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:967)
at org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:240)
at org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:104)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1005)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:863)
at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:794)
at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:286)
at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
at com.vordel.circuit.oauth.persistence.JpaAccessTokenStore.removeExpiredTokens(JpaAccessTokenStore.java:621)
at com.vordel.circuit.oauth.persistence.SynchronizedTokenStore.removeExpiredTokens(SynchronizedTokenStore.java:173)
at com.vordel.circuit.oauth.persistence.OauthLoadableCache$2.run(OauthLoadableCache.java:84)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Event 2:
ERROR 25/Aug/2015:18:02:33.449 [94754700] error handling connection: peer disconnected unexpectedly

0 Karma

woodcock
Esteemed Legend

You need to bounce the Splunk instances on all the Indexers and then it should work.

0 Karma

dingesbr
Explorer

+Woodcock

Thanks for the reply. The indexer(s) were restarted but did not resolve the issue.

We ran the following command to verify that the transform was loaded:
splunk cmd btool transforms list --debug

It showed up in the list of transforms.

Any other ideas come to mind? I appreciate your help!

0 Karma

neelamssantosh
Contributor

Same issue.

Kindly provide the steps, if it has been resolved

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...