Does anyone use Incapsula API to integrate Incapsula security events into Splunk?

[davisr7]

I am looking for a way to integrate Incapsula security events into Splunk. The events API response does not appear to adhere to standard json responses so it looks like I need a custom parser to make it Splunk worthy. Thoughts?

[kdick · Accepted solution]

Hi Davis,

It sounds like you already have an api script going and just need a parser. If that is the case I have a function you can use.

I have written a function in python that can parse the incapsula event data. Pass this function the event data from the api as a string and it will return a tuple with the first element being a list of ordered dicts(each ordereddict is an event) and the max timestamp in epoch as an integer. The fieldnames are normalized to the Splunk common information model where possible.

The reason I am returning the max timestamp in addition to the events is to use it to handle the APIs pagination. You can only get 100 results at a time so you need to pass the highest timestamp to the next api call or your checkpointing logic.

This parser creates one event for each "attack vector" stanza, with the fields from the enclosing stanza placed into the event.

Take a look at this and see if it suits your needs. You may want to just wait, however, as Imperva has indicated they will be rolling out simpler SIEM integration in the coming months.

EDIT: Forgot to mention, you need to import re (pythons regular expression module) for this function to work.

## Returns a tuple containing an array of ordered dictionaries (the events) and the max timestamp. ([event1,event2,...],max_ts) 
def parse(data):


    def extractFields(fieldsString):
        fields={}
        i=0
        while i<len(fieldsString):
            if fieldsString[i] == '[':
                value=""
                name=""
                i+=1
                while fieldsString[i]!='=' and fieldsString[i]!="]":
                    name+=fieldsString[i]
                    i+=1
                if fieldsString[i] =="]":
                    value=name
                    name="signature"
                    i-=1
                i+=1
                while fieldsString[i]!=']':
                    value+=fieldsString[i]
                    i+=1
                fields[name]=value
            i+=1
        return fields

    def normalizeFieldsToMapping(unNormalizedEvent):
        FIELD_MAPPING = [('Timestamp' , 'timestamp'),
        ('AccountID', 'account_id'),
        ('AccountName' , 'account_name'),
        ('SiteId', 'dest_id'),
        ('SiteName', 'dest_name'),
        ('EventID' , 'event_id'),
        ('EventTimestamp', 'event_timestamp'),
        ('EventType', 'event_type'),
        ('ClientIP' , 'src_ip'),
        ('ClientApp' , 'src_app'),
        ('VisitID', 'visit_id'),
        ('StartTime' , 'visit_start_date'),
        ('ClientApplication', 'src_application'),
        ('ClientType' , 'client_type'),
        ('UserAgent' , 'http_user_agent'),
        ('SupportsCookies' , 'supports_cookies'),
        ('SupportsJavaScript' , 'supports_javascript'),
        ('Country' , 'src_country'),
        ('ServedVia' , 'served_via'),
        ('NumberOfHitsOnVisit' , 'hit_count'),
        ('NumberOfPageViewsOnVisit' , 'page_view_count'),
        ('EntryReferer' , 'http_referrer'),
        ('EntryPage' , 'url'),
        ('URL' , 'uri'),
        ('ResponseCode' , 'response_code'),
        ('RequestResult' , 'request_result'),
        ('NumRequests' , 'request_count'),
        ('RequestsIndexOnVisit' , 'requests_index_on_visit'),
        ('QueryString', 'query_string'),
        ('PostData', 'post_data'),
        ('Referer', 'visit_http_referrer'),
        ('IncidentID', 'incident_id'),
        ('Rid','rid'),
        ('RuleName', 'signature'),
        ('ActionTaken', 'action'),
        ('AttemptedOn', 'attempted_on'),
        ('ThreatPattern', 'threat_pattern'),
        ('AttackInternalCode', 'attack_internal_code')]
        normalizedEvent = OrderedDict()
        for field in FIELD_MAPPING:
            if field[0] in unNormalizedEvent:
                normalizedEvent[field[1]] = unNormalizedEvent[field[0]]
            else:
                normalizedEvent[field[1]] = ''
        return normalizedEvent  


    events = data.split("==================================================")
    eventList = []
    max_ts = 0 
    for event in events:

        if "max-ts:" in event:
            max_ts = int(re.search('max-ts:\s(\d*)',event).group(1))


        visitsSplit = event.split("---- VISITS ----")
        eventInfo=visitsSplit[0]
        eventFields=extractFields(eventInfo)
        ## Parse the events. 
        if len(visitsSplit)>1:
            visits=visitsSplit[1].split("---- VISIT ----")      
            for visit in visits:
                requests=iter(visit.split("-- Request"))
                visitFields=eventFields.copy()
                visitFields.update(extractFields(next(requests)))
                for request in requests:
                    attacks_list=request.split("-- Attack Info:")
                    requestFields=visitFields.copy()
                    attacks = iter(attacks_list)
                    requestFields.update(extractFields(next(attacks)))
                    if  len(attacks_list) == 1 :
                        eventList.append(normalizeFieldsToMapping(requestFields))
                    for attack in attacks:

                        attack_vectors_list = request.split("-- Attack Vector:")
                        attackFields = requestFields.copy() 
                        attack_vectors = iter(attack_vectors_list)
                        attackFields.update(extractFields(next(attack_vectors)))
                        if len(attack_vectors_list) ==1:
                            eventList.append(normalizeFieldsToMapping(attackFields))
                        for attack_vector in attack_vectors:
                            subEventFields = attackFields.copy()
                            subEventFields.update(extractFields(attack_vector))
                            eventList.append(normalizeFieldsToMapping(subEventFields))

    return eventList,max_ts

[kdick]

An update for anyone looking to integrate Imperva with Splunk. Incapsula has released an upgraded SIEM API with support for standard CEF and W3C formats and realtime reporting. This answer relates to the old API, you are better off looking into the new SIEM integration options: https://www.incapsula.com/blog/turnkey-siem-integration.html

[davisr7]

Perfect! Thanks for your help on this!

[kdick]

great hope it helped!

Just wanted to update, I ended up changing the parser to not include events for requests that have no attack pattern attached. This gives less verbose data, but it ends up working out better for my needs at least.

If you want to change this also, just comment out lines 100 and 101 from the answer. Here is the example:

        requestFields.update(extractFields(next(attacks)))
                # Uncomment if you would like to make events for requests that do not have an attack stanza 
                #   if  len(attacks_list) == 1 :
                #       eventList.append(normalizeFieldsToMapping(requestFields))
                    for attack in attacks:
                        attack_vectors_list = request.split("-- Attack Vector:")