Splunk Search

Why is my external lookup python script unable to create file table.csv when using the lookup command in a search?

vincenteous
Communicator

Hi Everyone,

I have created a python script which will get data from a web service as an external lookup. Within my script, I create a csv file as a temporary table to prevent the lookup from getting data from the web service again for the same key. Below is the script which is located in $SPLUNK_HOME/etc/system/bin:

#!usr/bin/env python

import sys
import csv
import urllib
import os.path

def getmsisdn(IMSI):
        try:
                msisdnlist = urllib.urlopen("http://xx.xx.xx.xx:xxxx/?oprid=lstisdn&show=ISDN&imsi=" + IMSI + "&trx_id=SPLK############")
                lines = msisdnlist.readlines()
                temp = lines[0]
                msisdn = temp[3:]

                with open('/apps/splunk/etc/system/bin/table.csv', 'a') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
                        writer.writerow({'IMSI': IMSI, 'MSISDN': msisdn})

                return msisdn
        except:
                return ''

def lookup(IMSI):
        if os.path.isfile('/apps/splunk/etc/system/bin/table.csv'):
                found = True
                with open('/apps/splunk/etc/system/bin/table.csv', 'r') as csvfile:
                        fieldnames = ['IMSI', 'MSISDN']
                        reader = csv.DictReader(csvfile, fieldnames=fieldnames)

                        for row in reader:
                                if (row['IMSI'] == IMSI) and (row['MSISDN'] != ''):
                                        return row['MSISDN']
                                else: found = False

                if not found: return getmsisdn(IMSI)
        else:
                return getmsisdn(IMSI)

def main():
        if len(sys.argv) != 3:
                print "Usage: python external_lookup.py [IMSI field] [MSISDN field]"
                sys.exit(0)

        IMSIf = sys.argv[1]
        MSISDNf = sys.argv[2]

        r = csv.reader(sys.stdin)

        w = ''
        header = []
        first = True


        for line in r:
                if first:
                        header = line
                        if MSISDNf not in header or IMSIf not in header:
                                print "IMSI and MSISDN fields must exist in CSV data"
                                sys.exit(0)

                        w = csv.DictWriter(sys.stdout,header)
                        first = False

                result = {}
                i = 0

                while i < len(header):
                        if i < len(line):
                                result[header[i]] = line[i]
                        else:
                                result[header[i]] = ''
                        i += 1

                if len(result[IMSIf]) and len(result[MSISDNf]):
                        w.writerow(result)
                elif len(result[IMSIf]):
                        result[MSISDNf] = lookup(result[IMSIf])
                        if result[MSISDNf]:
                                w.writerow(result)

main()

Here is the configuration I have put within transforms.conf:

[MSISDNlookup]
external_cmd = external_lookup3.py IMSIf MSISDNf
external_type = python
fields_list = IMSIf,MSISDNf

When I ran the script from the command line, it worked just fine. Tables.csv was properly created. Unfortunately, when I used this using the lookup command in a Splunk search, only the lookup to the web service succeeded, but the table.csv was not created at all.

Is this a bug? I am using Splunk 6.1.3 and it is a production environment, so if I were forced to upgrade, it would be hard.

Thanks in advance.

Best Regards,

Vincent

0 Karma
1 Solution

vincenteous
Communicator

After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.

View solution in original post

vincenteous
Communicator

After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...