How do I save job search results in Hunk?

[andreacorrie]

I am wondering how to save job search results in Hunk over the long term. I can see where to save a job but there seems to be a defined lifetime. I want to be able to run monthly jobs and save the results-indefinitely-in a directory on hdfs.

[Ledion_Bitincka]

I know accelerated search results are
saved in hdfs but they expire
Report acceleration contents stored in HDFS expire only based on the acceleration settings.

From my understanding of your question it seems like you want to store the final results of a search in HDFS. If that is correct, then out of the box you cannot do that in Hunk. However, you should be able to write an external search command (similar to, say, sendemail command) that can store the results in whatever format you need in HDFS or any other external system.

[martin_mueller]

Doesn't Hadoop Connect already come with writing search results to HDFS? Would be worth a shot to try that before building some custom solution... provided you can use regular Splunk apps on Hunk, of course.

[Ledion_Bitincka]

Yes, Hadoop Connect app is a very good starting and it works with Hunk - the exporthdfs search command supports a limited set of data formats for writing data out into HDFS. However, there are a couple of caveats, a) it is not tightly integrated with Hunk and therefore the Hadoop cluster info needs to be specified again, b) it is primarily intended for exporting continuous streams of data and thus it chunks the data up into separate files (the size of which can be controlled)

[andreacorrie]

Thanks. Sounds like a feature request is in order!

[Ledion_Bitincka]

Right, can you elaborate on the use case a bit more. What exactly are you trying to achieve?

[jitzpop]

I also have a similar requirement where the result of a search need to be stored in a csv file in the HDFS itself. I tried the summary index and it works well but I needed the results in HDFS. This could be something similar as "outputcsv" now. To extend the use case it can be used to store raw event, text as well. My use case is to have the HUNK query which generates a smaller subset of result which can be later used for performing faster queries on a larger time range.

[Ledion_Bitincka]

jitzpop - have you considered using Hunk's Report Acceleration feature?

[rdagan_splunk]

With Hunk you have 4 options when you create a search:
** report -> acceleration = The results are kept in HDFS under /user/hunk_user/working_directory/cache
** report -> schedule = The results are kept on the disk of the Search Head (Hadoop client) for twice the amount time of the schedule
** report -> summary indexing = Also kept on the disk of the Search Head, but in a different location as a schedule search
** Normal search = Kept for 10 minutes on the Search Head disk

[barakreeves]

Have you tried Report Acceleration? This feature will cache results in HDFS and run a cron job on the back end to update the data.

[martin_mueller]

Hmm... what happens when you use regular summary indexing? Do those results get stored in a regular Splunk index or are they sent to HDFS? If the latter then they should persist indefinitely and you would indeed not need any add-on. If the former then the add-on I linked to should provide just that ability, to write summary-index-style searches that pipe their results to HDFS.

[andreacorrie]

I'm not sure I understand. Hunk is a bi-directional hadoop connector by design so I don't believe I need an add-on. We don't have a splunk instance, only Hunk. I know accelerated search results are saved in hdfs but they expire. I want to be able to save search results over the long term.

[martin_mueller]

Splunk Hadoop Connect should give you the ability to export search results to Hadoop: http://apps.splunk.com/app/1180/