Hi Everyone,
I have created a python script which will get data from a web service as an external lookup. Within my script, I create a csv file as a temporary table to prevent the lookup from getting data from the web service again for the same key. Below is the script which is located in $SPLUNK_HOME/etc/system/bin:
#!usr/bin/env python
import sys
import csv
import urllib
import os.path
def getmsisdn(IMSI):
try:
msisdnlist = urllib.urlopen("http://xx.xx.xx.xx:xxxx/?oprid=lstisdn&show=ISDN&imsi=" + IMSI + "&trx_id=SPLK############")
lines = msisdnlist.readlines()
temp = lines[0]
msisdn = temp[3:]
with open('/apps/splunk/etc/system/bin/table.csv', 'a') as csvfile:
fieldnames = ['IMSI', 'MSISDN']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writerow({'IMSI': IMSI, 'MSISDN': msisdn})
return msisdn
except:
return ''
def lookup(IMSI):
if os.path.isfile('/apps/splunk/etc/system/bin/table.csv'):
found = True
with open('/apps/splunk/etc/system/bin/table.csv', 'r') as csvfile:
fieldnames = ['IMSI', 'MSISDN']
reader = csv.DictReader(csvfile, fieldnames=fieldnames)
for row in reader:
if (row['IMSI'] == IMSI) and (row['MSISDN'] != ''):
return row['MSISDN']
else: found = False
if not found: return getmsisdn(IMSI)
else:
return getmsisdn(IMSI)
def main():
if len(sys.argv) != 3:
print "Usage: python external_lookup.py [IMSI field] [MSISDN field]"
sys.exit(0)
IMSIf = sys.argv[1]
MSISDNf = sys.argv[2]
r = csv.reader(sys.stdin)
w = ''
header = []
first = True
for line in r:
if first:
header = line
if MSISDNf not in header or IMSIf not in header:
print "IMSI and MSISDN fields must exist in CSV data"
sys.exit(0)
w = csv.DictWriter(sys.stdout,header)
first = False
result = {}
i = 0
while i < len(header):
if i < len(line):
result[header[i]] = line[i]
else:
result[header[i]] = ''
i += 1
if len(result[IMSIf]) and len(result[MSISDNf]):
w.writerow(result)
elif len(result[IMSIf]):
result[MSISDNf] = lookup(result[IMSIf])
if result[MSISDNf]:
w.writerow(result)
main()
Here is the configuration I have put within transforms.conf:
[MSISDNlookup]
external_cmd = external_lookup3.py IMSIf MSISDNf
external_type = python
fields_list = IMSIf,MSISDNf
When I ran the script from the command line, it worked just fine. Tables.csv was properly created. Unfortunately, when I used this using the lookup command in a Splunk search, only the lookup to the web service succeeded, but the table.csv was not created at all.
Is this a bug? I am using Splunk 6.1.3 and it is a production environment, so if I were forced to upgrade, it would be hard.
Thanks in advance.
Best Regards,
Vincent
After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.
After a long struggle, I've finally found that csv file. It's not that the file wasn't created, but it's created in an entirely different place with the same specified directory. I put the script inside my search head and table.csv was created in my indexer with the same directory as I've specified within my script.