Getting Data In

Filtering events from Hadoop unstructured data

sansri7680
Path Finder

I am trying to read log files from Hadoop cluster. These are unstructured files which otherwise can be filtered after indexing using Regex searches. But my input data is huge and the throughput requirement is also very high. The result is only a small portion of the input. Hence is it possible to filter the input data before being indexed by Hunk so that I can avoid searching unnecessary data

Tags (3)
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Currently Hunk optimizes data access if the data is partitioned and Hunk is properly configured to recognize those partitions. Two types of partitioning exist: (a) time based, this is when the data is structured hierarchically using some time partitioning and (b) field based partitioning.

For example if your data is organized as follows

/some/path/20140108/server1/...
/some/path/20140108/server2/...
/some/path/20140109/server1/...
/some/path/20140109/server2/...

You can configure Hunk to recognize the third segment in the path as the data and the fourth segment as the server field. You can look at the details of how to do that here

Currently Hunk does not have the ability to optimize data access based on the file content, because we don't create an index - we just access/process the data in it's raw form.

Does this help?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...