Dashboard performance with "large" data sets

[responsys_cm]

We're building an app to manage our Nessus vulnerability results. We would like to create a dashboard for people to search for various criteria in Nessus plugins and display a series of tables with tabs. We're using Sideview Utils 2.x for the app.

Currently, the dashboard is setup to do a search that does | inputlookup append=t nessus_plugin_database. That lookup file is about 30 MB in size. Each tab searches that data for results with CVE identifiers, Bugtraq or OSVDB IDs, etc. Even though I have defined a maximum search time of three minutes, the tabs that return the most amount of data are timing out with a message about how the search expired or was cancelled.

I have no idea what the relationship is between the client and server as far as where data is stored (locally or server), where it executes, etc.

What are the best practices for creating dashboards that need to input a large amount of data. Are there differences between doing an inputlookup vs. searching an index for the same raw data?

Thx.

Craig

[sideview]

From other questions I think I've seen this view and the searches and postprocess searches being used against this lookup. And I think that the problem is that you're using a base search to load the 30MB worth of inputlookup rows, and then you're using somewhat complex postprocess searches to then process those rows.

I would try just doing it all in the Search. The argument to involve postprocess usually centers around the desire to avoid pulling many GB of events off of disk. A 30MB lookup on the other hand is relatively tiny compared to 1GB or 50GB of raw events. So doing it all in Search isn't going to be any less efficient really. And on top of that the postprocess architecture can start to get slow when there are lots and lots of rows to filter and analyze and lots of rows to return - possibly you're hitting some of that.

Anyway, that's what I'd try.

instead of having a search like

| inputlookup nessus_plugin_reference_lookup append=t | inputlookup open_vulnerabilities_lookup append=t

and then running various long postProcess searches with

<module name="PostProcess">
  <param name="search">$selectedTab$</param>
</module>

try just doing a Search module down where you had that PostProcess module:

<module name="Search">
  <param name="search">| inputlookup nessus_plugin_reference_lookup | inputlookup open_vulnerabilities_lookup append=t | $selectedTab$</param>