JuniperSA not showing some roles from the log files

[dglinder]

My engineer reported that our JuniperSA reports were starting to look 'sparse' and seemed to be missing data. The syslog data is still being collected, no changes to the logging were knowingly made, and the raw data within Splunk matched what was being sent. Still, the reports were missing some role names and other data.

Here is some sample log entries:

Feb  4 08:41:49 A.A.A.A Juniper: 2014-02-04 02:41:49 - NYC - [X.X.X.X] username1(Security Group Alpha)[Role name one] - Login succeeded for username1/Security Group Alpha from X.X.X.X.
Feb  4 09:03:38 B.B.B.B Juniper: 2014-02-04 03:03:38 - ive - [Y.Y.Y.Y] username2(Security Group Beta)[Role-name two] - Login succeeded for username2/Security Group Beta from Y.Y.Y.Y.
Feb  4 08:49:32 C.C.C.C Juniper: 2014-02-04 02:49:32 - ive - [Z.Z.Z.Z] username3(Security Group Gamma)[Role-name three] - Login succeeded for username3/Security Group Gamma from Z.Z.Z.Z.

And here is the transforms.conf file that should be matching the role (the text within the square brackets):

[junipersa-role-info]
REGEX = Juniper\:\s[^\s]+\s[^\s]+\s-\sive\s-\s\[\d+.\d+.\d+.\d+\]\s[^\(\)\s]+\([\s\d\w]+\)\[([\s\d\w\,]+)\]\s-
FORMAT = role::$1

[dglinder · Accepted solution]

After a bit of work we determined that the REGEX fields were being too specific and were not accounting for non-HA nodes. I'm not a Juniper guy, but apparently the "- ive -" means it's in an HA arrangement, and the "- NYC -" (or other customer specified string) are present in non-HA nodes.

We also noticed that the role name (the original reason for the call) was being missed on otherwise good log entries because the role name contained a hyphen ("-"). We updated the transforms.conf file to account for these and the roles started populating.

Looking around at the rest of the REGEX in that file we noticed that the same fields were used in other lines so they were updated accordingly also.

The basic REGEX changes were:
1. Change the search for "\sive\s" to "\s\S+\s" - a string beginning with a single space, then 1 or more non-space characters, terminated by a space.
2. Adding the hyphen character as an acceptable character when extracting the role field: "([\s\d\w\,-]+)".

Here is the final transforms.conf file we ended up with:
[junipersa-host]
DEST_KEY = MetaData:Host
REGEX = (\d+.\d+.\d+.\d+)\sJuniper:\s
FORMAT = host::$1

[junipersa-client-info]
REGEX = Juniper\:\s[^\s]+\s[^\s]+\s-\s\S+\s-\s\[(\d+.\d+.\d+.\d+)\]\s([^\(\)\s]+)
FORMAT = clientip::$1 user::$2 

[junipersa-realm-info]
REGEX = Juniper\:\s[^\s]+\s[^\s]+\s-\s\S+\s-\s\[\d+.\d+.\d+.\d+\]\s[^\(\)\s]+\(([\s\d\w]+)\)
FORMAT = realm::$1 

[junipersa-role-info]
REGEX = Juniper\:\s[^\s]+\s[^\s]+\s-\s\S+\s-\s\[\d+.\d+.\d+.\d+\]\s[^\(\)\s]+\([\s\d\w]+\)\[([\s\d\w\,-]+)\]\s-
FORMAT = role::$1

[junipersa-session-info]
REGEX = session:([^\\)]+)
FORMAT = sessionid::$1

[junipersa-secure-meeting-info]
REGEX = Created\smeeting\s\'([\s\d\w]+)\s\(([0-9]+)\)\'
FORMAT = meeting_name::$1 meeting_id::$2

[sa_sourcetyper]
DEST_KEY = MetaData:Sourcetype
REGEX = Juniper\:\s[^\s]+\s[^\s]+\s-\s\S+
FORMAT = sourcetype::juniper_sa_log

Hope this helps other users.