About jwhughes58

jwhughes58

Thanks. Since transforms.conf doesn't have the limitations of EXTRACT, I finally got it working.

jwhughes58

I'm working with a field named Match_Details.match.properties.user. It contains domain\user information that I'm trying to split into domain and user. I can't use EXTRACT in props.conf because of this restriction. EXTRACT-<class> = [<regex>|<regex> in <src_field>] NOTE: <src_field> has the following restrictions: * It can only contain alphanumeric characters and underscore (a-z, A-Z, 0-9, and _). Is this also true with REPORT in transforms.conf? I can't find any documentation that tells me. TIA, Joe

jwhughes58

Thanks. I hadn't thought of that. Since I posted the question, NetSkope came back with a solution. I was sent this conf_file_stanzas = conf_file_object.get_all() replace the above line with following: conf_file_stanzas = conf_file_object.get_all(only_current_app=True) With that the issue was resolved. The code was trying to get information from another TA.

jwhughes58 · ‎03-11-2024

I'm getting this error message in the log file, solnlib.credentials.CredentialNotExistException: Failed to get password of realm=. According to this page, https://splunk.github.io/addonfactory-solutions-library-python/credentials/#solnlib.credentials.CredentialNotExistException , this is due to the username not being valid. I'm trying to work out how to get what is passed to credentials.py since the information in the username doesn't make sense to me. Is there anyway of debugging credentials.py, I tried to put print statements in, but the TA UI didn't like it. I had to remove the print statements to get the UI working again. I've tried debugging via command line but always get stuck at this point, session_key = sys.stdin.readline().strip(). I can't work out what I need to do to see where the user information is coming from. Any help on how I can debug this? TIA, Joe

jwhughes58 · ‎02-28-2024

Hi All, I'm trying to debug netskope_email_notification.py from the TA-NetSkopeAppForSplunk by running this command. splunk cmd python -m pdb netskope_email_notification.py It runs until it hits this line session_key = sys.stdin.readline().strip() How do I get past this? Maybe something like this, but with a session key. splunk cmd python -m pdb netskope_email_notification.py < session_key If so, how do you create an external session key? TIA, Joe

jwhughes58 · ‎02-27-2024

@PickleRickYou are correct. It is poorly written. I have already made three suggestions to them of which one is to split it into an ingest piece and a search piece.

jwhughes58 · ‎02-27-2024

Hi @gcusello , The kv-store isn't usable in the SHs. The original ask was to take the items in the kv-store and create an alert for each item. Since the kv-store gets recreated every 4 hours, this would cause alert duplication which we wanted to avoid. I added another kv-store on the HF that contains a hash of the values in the items and then check to see if an item is new or already exists in the kv-store. If new, an alert. If not, it gets dropped. The analysts decided they wanted to have the kv-store copied to the SHs they use. Thus the original question since they don't have access to the HF. Currently the best suggestion is to output the kv-store as a csv, scp it from the HF to the SHs, and load it into a local kv-store. Regards, Joe

jwhughes58 · ‎02-27-2024

Hi Giuseppe, Greetings from another Joseph. We have a distributed environment with multiple SHs, 8 I think in 2 different groups. For that reason any app/TA that have API calls or binaries go onto an HF that we use for this purpose. We actually don't care about the UI for the app since all we want to do is pull in the events using the binaries. I tried splitting the App into an input and a search piece but that didn't work. We have suggested to Bloodhound that they do this. For now the app runs on the HF which is where the kvstores are created. The teams use the SHs to look at the events in the Search app. I've had a request to copy the kvstores to one of the SH groups so they can be examined instead of using the alerts that get created from the kvstores. Regards, Joe

jwhughes58 · ‎02-26-2024

The Bloodhound Enterprise TA is run on the HF and generates an updated KV file every 4 hours. I wrote a script that runs and turns the kvstore entries into alerts. Due to some weirdness in the data, the question has come up, can the kvstore on the HF be copied to a SH. I haven't found a suggestion that I think will work. We are running Splunk Enterprise 9.1.1 on prem on servers running RHEL 8.8. TIA, Joe

jwhughes58 · ‎02-21-2024

Hi All, I found this https://community.splunk.com/t5/Dashboards-Visualizations/9-0-5-ui-prefs-conf-Why-my-default-search-mode-in-search-page-on/m-p/652793 and in there is this. SplunkWeb users may experience different behaviors for the UI preferences that used to persist and show latest preferences by updating ui-prefs.conf on the fly. Now after upgrade to 9.0.5+ or 9.1.0+ its behavior changed and no longer uses ui-prefs.conf to remember the user's UI level preferences, but instead, uses the url in the request or localStorage/Web Storage. In Firefox I found this webappsstore.sqlite in my ../Library/Application Support/Firefox/Profiles/e0fxb1hs.default-release which is similar to the above. Is this where the ui-prefs.conf information was moved to? I've had a request from a user that wants to set the 'Selected fields', but after the upgrade to 9.1.2 the changes would be stored in a sqlite DB. Is this correct? Is there any way of changing the 'Selected fields' other than using the backend? Does this work for other apps beyond Search? TIA, Joe

jwhughes58 · ‎02-13-2024

I worked with Sahil Sharma of Technical Support on this. The answer was to update the add-on from 4.0.1 to 4.0.2. That fixed the problem.

jwhughes58 · ‎01-24-2024

Screenshot of error message

jwhughes58 · ‎01-23-2024

We are using 'Splunk App for Lookup File Editing' version 4.0.1. There are two issues that bother me. First is the continuous popup about 'Save Backup'. I need a way of turning this off since most of the time my edits are of adhoc lookups and I don't want a backup. Second is the can't save error message I get in Firefox. I have to quit Firefox, restart, and then I can save. After a number of adhoc lookups I get the error message again. Any work arounds beyond restarting the browser?

jwhughes58 · ‎01-05-2024

I just changed the cron job. I was just running it from the UI. Once I did that, I started getting alerts. I need to do some more cleanup, but the problem is solved.

jwhughes58 · ‎01-05-2024

I've tried that and I didn't see anything. I tried it again and I still don't see the alert firing.

jwhughes58 · ‎01-05-2024

Cool. Not quite as fast as the original method, but the difference is minuscule. I do like the fact that I don't have to repeat the same command. This is nice to know.

jwhughes58 · ‎01-05-2024

The outputlookup should have been inputlookup. My brain slipped a gear when I was entering the Subject. I have corrected it. Here is what I have in the alert. I should give the foreach a try.

jwhughes58 · ‎01-05-2024

Hi All, The Bloodhound TA creates a KV store lookup. I've been asked to take the entries in the KV store and turn them into events. I've setup an alert, but I'm not seeing the alert fire. The SPL looks like this | inputlookup path_principals_lookup | eval domain_id=if(isnull(domain_id), "NULL_domain_id", domain_id) | eval domain_name=if(isnull(domain_name), "NULL_domain_name", domain_name) | eval group=if(isnull(group), "NULL_Group", group) | eval non_tier_zero_principal=if(isnull(non_tier_zero_principal), "NULL_non_tier_zero_principal", non_tier_zero_principal) | eval path_id=if(isnull(path_id), "NULL_path_id", path_id) | eval path_title=if(isnull(path_title), "NULL_path_title", path_title) | eval principal=if(isnull(principal), "NULL_principal", principal) | eval tier_zero_principal=if(isnull(tier_zero_principal), "NULL_tier_zero_principal", tier_zero_principal) | eval user=if(isnull(user), "NULL_user", user) | dedup domain_id, domain_name, group, non_tier_zero_principal, path_id, path_title, principal, tier_zero_principal, user I see statistics, but that doesn't fire the alert. Is there something I'm missing to turn the values in the kvstore into events to be alerted on? TIA, Joe

jwhughes58 · ‎12-07-2023

I've got this search index=main sourcetype="bigfix" | eval raw=_raw | rex mode=sed field=raw "s/\n/ /g" | rex field=raw "At \d+:\d+:\d+\s+-0800\s+-(?<message>.*)" | rex field=message "^(?<message_type>[^:]+):\s" | eval message_type_ns=replace(message_type, " ", "") | eval x_message_type=if(message_type == message_type_ns, message_type, "No message type") | stats count by message_type, message_type_ns, x_message_type That doesn't appear to be working correctly. I'm always getting either all true or all false. This is the output. "message_type","message_type_ns","x_message_type",count " ActionLogMessage",ActionLogMessage,"No message type",240 " ActiveDirectory",ActiveDirectory,"No message type",128 " Client has an AuthenticationCertificate Relay selected",ClienthasanAuthenticationCertificateRelayselected,"No message type",2 " Client shutdown (Service manager shutdown request) ******************************************** Current Date","Clientshutdown(Servicemanagershutdownrequest)********************************************CurrentDate","No message type",3 " Encryption",Encryption,"No message type",11 " Initializing Site",InitializingSite,"No message type",43 " PollForCommands",PollForCommands,"No message type",13 " Processing fixlet site. ******************************************** Current Date","Processingfixletsite.********************************************CurrentDate","No message type",1 " RegisterOnce",RegisterOnce,"No message type",149 " Report posted successfully ******************************************** Current Date","Reportpostedsuccessfully********************************************CurrentDate","No message type",1 " Restricted mode Initializing Site",RestrictedmodeInitializingSite,"No message type",3 " User interface process disabled for user 'user' ActiveDirectory","Userinterfaceprocessdisabledforuser'user'ActiveDirectory","No message type",1 " User interface process disabled for user 'user' ActiveDirectory","Userinterfaceprocessdisabledforuser'user'ActiveDirectory","No message type",1 " User interface session ended for user 'user' User interface session ended for user 'user' ******************************************** Current Date","Userinterfacesessionendedforuser'user'Userinterfacesessionendedforuser'user'********************************************CurrentDate","No message type",1 " User interface session ended for user 'user' ActiveDirectory","Userinterfacesessionendedforuser'user'ActiveDirectory","No message type",1 " User interface session ended for user 'user' ******************************************** Current Date","Userinterfacesessionendedforuser'user'********************************************CurrentDate","No message type",1 When I try this simple case, it works. | makeresults | eval string_a="Client shutdown (Service manager shutdown request) ******************************************** Current Date" | eval string_b="Client_shutdown_(Service_manager_shutdown_request)_********************************************_Current_Date" | eval my_string=if(string_a == string_b, string_a, string_b) And the output _time my_string string_a string_b 2023-12-07 10:14:17 Client_shutdown_(Service_manager_shutdown_request)_********************************************_Current_Date Client shutdown (Service manager shutdown request) ******************************************** Current Date Client_shutdown_(Service_manager_shutdown_request)_********************************************_Current_Date What I'm trying to do is find these At 09:01:45 -0800 - Encryption: optional encryption with no certificate; reports in cleartext The above would have message_type=Encryption. This example At 09:00:39 -0800 - Starting client version xx.yy.zz.aa FIPS mode disabled by default. Cryptographic module initialized successfully. Using crypto library libBEScrypto - OpenSSL would have message_type="No message type". I've tried using colon (:), but there are messages with embedded colons. Any thoughts on how to solve this are appreciated. TIA, Joe

jwhughes58 · ‎11-07-2023

I have two lookups. One consists of the allowed URLs. The other consists of the URLs from a firewall. For example in the first google.com dummy.com In the second site1.google.com site2.google.com The first lookup is ingested from a file sent by the FW team. I create the second lookup with this search index=my_firewall sourcetype=my_sourcetype (rule=rule_1 OR rule=rule_2 OR rule=rule_3) [ | inputlookup external_url.csv ] | fields url | dedup url | table url | outputlookup external_results.csv This gives me the sites that have been reached over the time period. Next I use this search | inputlookup external_url.csv | lookup external_results.csv url OUTPUTNEW url as isFound I think this is giving me what I want, but I can't view the output the way I want. I would like to see allowed_url fw_url isFound Using the sample data google.com site_1.google.com true google.com site_2.google.com true dummy.com false TIA, Joe

jwhughes58 · ‎10-11-2023

@yuanliuThanks. I would have never figured out the mvjoin(mvindex. That is something I don't use. You gave me enough help that I was able to work out something I can give to another team. Karma point awarded.

jwhughes58 · ‎10-11-2023

@yuanliuYeah, it is a pain of a search. Here is the issue. A firewall device generates an event with URL when certain policies are triggered by contractors. That is the initial search. The firewall team has a list of the URLs the contractors have access to which is the csv file. The firewall team wants to remove any URLs that aren't used in a period of time. Thus, I have to compare the firewall URLs to the csv URLs and output any csv URLs that aren't used in the time frame. The search finds the firewall events. index=my_index sourcetype=my_sourcetype (rule=policy_1 OR rule=policy_2 OR rule=policy_3) [ | inputlookup my_list_of_urls.csv ] My issue is the firewall events use the long URL and not the short one. From the firewall a478076af2deaf28abcbe5ceb8bdb648.fp.measure.office.com/ aad.cs.dds.microsoft.com/ From the csv file *.microsoft.com/ microsoft.com/ *.office.com/ office.com/ The two events from the firewall mean that the two listed in the csv file are still good and don't need to be removed. I try to think of this as two sets, one the firewall results and the other the csv file, but I can't figure out how to search the firewall results with what is in the csv file. This make sense? TIA, Joe

jwhughes58 · ‎10-10-2023

Hi @ITWhisperer , If the urls are consistent, that is a great idea. Unfortunately, the urls have between 1 and 8 parts between . and I don't know where to start. Alternatively, if there is a way of adding what is in the csv to the results, that would work. Regards, Joe

jwhughes58 · ‎10-10-2023

Hi @richgalloway Unfortunately the csv file has 1156 entries so the use case would be huge. The other issue is that the csv file gets updated daily and urls are added and removed. I was hoping to use the csv file to say get only this part out of the result urls. Alternatively some way of joining the results url with the csv file urls. Regards, Joe

jwhughes58 · ‎10-10-2023

I'm working with data from this search index=my_index sourcetype=my_sourcetype (rule=policy_1 OR rule=policy_2 OR rule=policy_3) [ | inputlookup my_list_of_urls.csv ] | rename url AS my_url | stats count by my_url | table my_url The events look like this 02ef65dc96524dabba54a950da7cb0d8.fp.measure.office.com/ 0434c399ca884247875a286a10c969f4.fp.measure.office.com/ 14f8c4d0e9b7be86933be5d3c9fb91d7.fp.measure.office.com/ 3d8e055913534ff7b3c23101fd1f3ca6.fp.measure.office.com/ 4334ede7832f44c5badcfd5a6459a1a2.fp.measure.office.com/ 5d44dec60c9b4788fb26426c1e151f46.fp.measure.office.com/ 5f021e1b8d3646398fab8ce59f8a6bbd.fp.measure.office.com/ 6f6c23c1671f72c36d6179fdeabd1f56.fp.measure.office.com/ 7106ea87c1e2ed0aebc9baca86f9af34.fp.measure.office.com/ 88c88084fe454cbc8629332c6422e8a4.fp.measure.office.com/ 982db5012df7494a88c242d426e07be6.fp.measure.office.com/ a478076af2deaf28abcbe5ceb8bdb648.fp.measure.office.com/ aad.cs.dds.microsoft.com/ In the my_list_of_urls.csv there are these entries *.microsoft.com/ microsoft.com/ *.office.com/ office.com/ What I'm trying to do is get the microsoft.com and office.com from the results instead of the full url. I'm stumped on how to do it. Any help is appreciated. TIA, Joe

Posts	130
Solutions	8
Karma Given	25
Karma Received	6
Member Since	‎02-11-2015

Online Status	Offline
Date Last Visited	Tuesday

Does SOURCE_KEY have the same limitations as EXTRA...

aob_py3/solnlib/credentials.py question

Debugging python from the command line that needs ...

Copying a kvstore between an HF and a SH

What replaced ui-prefs.conf

Lookup Editor Issues

Change outputlookup to events

String comparison

Using a lookup to search another lookup

Extract domain from full url

Re: Does SOURCE_KEY have the same limitations as E...

Does SOURCE_KEY have the same limitations as EXTRA...

Re: Debugging python from the command line that ne...

aob_py3/solnlib/credentials.py question

Debugging python from the command line that needs ...

Re: Copying a kvstore between an HF and a SH

Re: Copying a kvstore between an HF and a SH

Re: Copying a kvstore between an HF and a SH

Copying a kvstore between an HF and a SH

What replaced ui-prefs.conf

Re: Lookup Editor Issues

Re: Lookup Editor Issues

Lookup Editor Issues

Re: Change inputlookup to events

Re: Change inputlookup to events

Re: Change outputlookup to events

Re: Change inputlookup to events

Change outputlookup to events

String comparison

Using a lookup to search another lookup

Re: Extract domain from full url

Re: Extract domain from full url

Re: Extract domain from full url

Re: Extract domain from full url

Extract domain from full url