@ITWhisperer  Thank you for your assistance. I've successfully tested the query you provided and managed to retrieve the expected success and failure counts. However, in addition to this, I also need to retrieve the step count for each step from Query 1. For instance, when I ran Query 1 individually for the last 30 days, I observed that 20 steps were extracted as a field along with the count of each step. Similarly, when I ran Query 2 for the same period, approximately 10 successes were extracted as a field with their respective counts. Likewise, when I ran Query 3, around 18 failures were extracted as a field with their counts. So, with the combined search you provided, I'm able to obtain a total of 18 fields comprising both successes and failures. This is because if any of the step fields have either a success or a failure, it reflects in the query output. However, the other two step fields don't have successes or failures, and their step information is not present in the output of the provided query. Therefore, we need to include the output of the first query, which includes the step field along with its count, in addition to the success and failure counts obtained from Queries 2 and 3. Since Query 1 captures all events, Query 2 captures successes, and Query 3 captures failures, we need to ensure that the first query's output is also included in the combined search. Could you please review and update accordingly? Thank you for your attention to this matter. ... View more

@ITWhisperer , Query1 has a field extracted as Step. So in this Step field which we have extracted as the information "Validation, Creation, Compliance Portal Report etc., with count So the same information Validation, Creation and Compliance Portal Report with Success count needs to be pulled using the second query and the failure ones needs to be extracted using the 3rd query. The output should be something like this(Combining all 3 queries): Step (Count)                                                      Success(Count)     Failure (Count) Validation                                                                     3                                           2 Creation                                                                       2                                          2 Compliance Report Portal                                  2                                          2\ So kindly help with the query.       ... View more

@gcusello @PickleRick @ITWhisperer  Can you kindly help to check and update on the same. ... View more

@gcusello , I have shared the sample events as well for all the 3 queries and for each Step field i want to get the Success and Failure information so kindly help to achieve the same.   The query you have provided pulls the total count of success and failure but i need a split of each "Step" field and their corresponding "Success" and "Failure" information. So kindly help to check and update on the same. ... View more

@ITWhisperer @PickleRick , Thank you for your response. Here are my updates as requested. ================================================================== Query1: index="abc" ("Restart transaction item" NOT "Pending : transaction item:") | rex field=_raw "Restart transaction item: (?<Step>.*?) \(WorkId:"| table Step |stats Count by Step Sample Events: 2024-04-21 03:00:02.6106|INFO|Transaction.Overflow.card.Command.Control|Restart transaction item: Validation (WorkId: 1234567) for RUNTIME: 987654| 2024-04-21 02:00:03.5437|INFO|Transaction.Overflow.card.Command.Control|Restart transaction item: Creation (WorkId: 1234567) for RUNTIME: 987654| 2024-04-18 09:00:10.9426|INFO|Transaction.Overflow.card.Command.Control|Restart transaction item: Compliance Portal Report (WorkId: 1234567) for RUNTIME: 987654| Output in Table Format: Step                                                    Count Validation                                              1 Creation                                                1 Compliance Portal Report            1 Query 2: index="abc" ("Error restart workflow item:") | rex field=_raw "Error restart workflow item: (?<Success>.*?) \(WorkId:"| table Success |stats Count by Success For the 1st and 2nd event it contains 30+ Lines of sample event hence I have took a small portion of it. While the 3rd and 4th event contains 10+ Lines and i have extracted a small amount of data. Sample Events: 2024-04-14 02:00:07.8759|ERROR|Transaction.Overflow.card.Command.Control|Error restart workflow item: Validation (WorkId: 1234567) for RUNTIME: 987654|System.Info.Entra.Solution.UpdateExecution: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.Entity.Core.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Transaction (Process ID 12) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. 2024-03-26 15:00:05.9123|ERROR|Transaction.Overflow.card.Command.Control|Error restart workflow item: Validation (WorkId: 1234567) for RUNTIME: 987654|System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.Entity.Core.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlClient.SqlException: Transaction (Process ID 12) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. 2024-03-27 03:00:15.3116|ERROR|Transaction.Overflow.card.Command.Control|Error restart workflow item: Creation (WorkId: 1234567) for RUNTIME: 987654|System.NullReferenceException: Object reference not set to an instance of an object. 2024-03-27 01:00:16.3231|ERROR|Transaction.Overflow.card.Command.Control|Error restart workflow item: Compliance Portal Report (WorkId: 1234567) for RUNTIME: 987654|System.NullReferenceException: Object reference not set to an instance of an object. Output in Table Format: Success                                         Count Validation                                           2 Creation                                             1 Compliance Portal Report         1   Query 3: index="abc" "Restart Pending event from command," | rex field=_raw "Restart Pending event from command, (?<Failure>.*?) \Workid"| table Failure |stats Count by Failure =============================================================================================================================================================================================== Sample Events: 2024-04-21 03:01:14.7929|INFO|Transaction.Overflow.card.Command.ValidationCommand|Pending: Restart Pending event from command, Validation Workid (WorkId: 1234567) for RUNTIME: 987654.| 2024-04-18 09:00:11.8332|INFO|Transaction.Overflow.card.Command.CreationCommand|Pending: Restart Pending event from command, Creation Workid (WorkId: 1234567) for RUNTIME: 987654.| 2024-04-17 06:51:16.7544|INFO|Transaction.Overflow.card.Command.CompliancePortalReportCommand|Pending: Restart Pending event from command, Compliance Portal Report Workid (WorkId: 1234567) for RUNTIME: 987654.| 2024-04-16 13:00:34.6238|INFO|Transaction.Overflow.card.Command.PageCountsCommand|Pending: Restart Pending event from command, Page Counts Workid (WorkId: 1234567) for RUNTIME: 987654.| Output in Table Format: Failure                                               Count Validation                                             1 Creation                                                1 Compliance Portal Report            1 Page Counts                                       1  So I need to combine all the 3 queries, i.e. Example For Step "Validation" i need to get how many Step are present for last 24 hours and in which how many success and how many failure.   Hence kindly help to check and update on the same please. ... View more

@KothariSurbhi  Yes I have developed an app and placed them in the default ui[prefs.conf and after app vetting process also it didnt worked. Need your inputs on the same please. I have also restarted the Splunk cloud search head instance but still the same.   ... View more

@gcusello , Any inputs from your end since still i can see the events are getting ingested with the password information present in it.     ... View more

@KothariSurbhi , Thank you for your prompt response. But actually it needs to be updated for each and every search and  all users want to have the default as 20 instead of 5. So our Search head is hosted in Cloud and I have tried to create an app with ui-prefs.conf but most of the time i got an error during app vetting process. But at some point of time the app has been deployed successfully and we have restarted the Search head and once again when we navigate and checked the max lines its still the same.  display.events.maxLines = 20 I can able to do it in the default directory whereas when i do from local its getting error. So kindly let me know how to achieve it. ... View more

@gcusello  Indeed, I have applied the correct sourcetype there to ensure that events are appropriately divided. Nonetheless, the masking of passwords is not taking place as intended. ... View more

@gcusello , This is the exact and correct sourcetype and I have created a custom app and uploaded the App in our Search head. Since our Search head is hosted in Splunk Cloud managed by Support. So I have uploaded the app in the upload app section and post vetting process completed i have installed the custom app into the Search head. This is the custom app i have created "abc_app" Under abc_app I have placed two folders "default" and "metadata" Under default I have created the app.conf and props.conf And under metadata I have created the default.metadata  Refer screenshots for reference.   So kindly let me know where i am missing since the lines are getting segregated as separate events whereas password masking is not getting applied to the events. Hence kindly help on the same.       ... View more

@gcusello  We had two requirements for the same sourcetype. One involved line breaks, and the other required password masking during ingestion. As our Search heads are managed by Splunk Support and hosted in the Cloud, we created a custom app and deployed the props.conf in the default directory. After uploading the apps for the cloud vetting process, they were successfully installed. However, I've noticed that the logs are now being separated into individual events, which is acceptable, but the passwords are still visible and haven't been masked according to our requirement. I'm unsure about where exactly I may have missed it.   This is the props.conf file for reference.  [sourcetype] SHOULD_LINEMERGE = false SEDCMD = s/password: ([^;]+);cpassword: ([^;]+);/password: (####);cpassword: (####);/gm   Sample log for reference:  [2024-03-01_06:32:08] INFO : REQUEST: User:abc CreateUser POST: name: xyz;email: abc@gmail.com;password: xyz@123;cpassword: xyz@123;role: Administrator; So kindly help on this requirement. ... View more

@gcuselloAs previously stated, I implemented the setting SHOULD_LINEMERGE = false in Splunk Cloud SH, which successfully resolved the issue. However, the logs contain HTML events, which are now being treated as individual events, resulting in difficulties extracting the desired fields. Could you please advise on how we can address this? ... View more

@gcusello Yes i have updated the props.conf in the UF of the server. Since I don't have access to the Indexers it didnt worked. Since our Search head are hosted in Cloud and managed by Splunk Support. So what should i need to do if i need to apply to Indexers directly. ... View more

  @lawrence_magpoc , I am running with Splunk Universal Forwarder 9.0.2 in one of my Linux client machine and recently for the past couple of days i am getting this events in the internal logs and it seems like its getting crashed and once again the service is getting started automatically. [build 17e00c557dc1] 2024-02-08 05:26:15 Received fatal signal 6 (Aborted) on PID 1908113. Cause: Signal sent by PID 1908113 running under UID 9991. Crashing thread: TcpOutEloop Registers: RIP: [0x00007F65EB39AACF] gsignal + 271 (libc.so.6 + 0x4EACF)   ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=30 ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=29   So how to fix this issue and also in which config file we need to add in the client machine where UF is running. autoBatch=false   ... View more

@lawrence_magpoc , I am running with Splunk Universal Forwarder 9.0.2 in one of my Linux client machine and recently for the past couple of days i am getting this events in the internal logs and it seems like its getting crashed and once again the service is getting started automatically. [build 17e00c557dc1] 2024-02-08 05:26:15 Received fatal signal 6 (Aborted) on PID 1908113. Cause: Signal sent by PID 1908113 running under UID 9991. Crashing thread: TcpOutEloop Registers: RIP: [0x00007F65EB39AACF] gsignal + 271 (libc.so.6 + 0x4EACF)   ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=30 ERROR TcpOutputQ [1908232 TcpOutEloop] - Unexpected event id=29   So how to fix this issue and also in which config file we need to add in the client machine where UF is running. autoBatch=false   ... View more