This is actually mentioned in the (Default) server.conf since around ver. 6.6.x, in that Mongo DB aka. kvstore (not Splunk) does not support Forward-Secrecy ciphers i.e. Mongo needs to have RSA ones to work. # The following non-forward-secrecy ciphers were added to support the kv store: # AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256. ... View more

Hi @ahmemohs03, So after the other comments think I know what part of the issue is... Splunk by default does not have the listening ports enabled i.e. 9997. So you will need to edit the inputs.conf file and insert then restart splunkd: [splunktcp://9997] connection_host = dns Note: By default Splunk will use "connection_host = ip" meaning that the "host" field will come up as the IP address and not use the DNS name... Note 2: You could also use the command line to enable the listening port: /opt/splunk/bin/splunk enable listen 9997 Confirm the configuration by the command line via: /opt/splunk/bin/splunk display listen Note: You will have to login but you should see the following line (or whatever port you enabled) Receiving is enabled on port 9997. The "X509Verify" message is more of a hint/tip that for enterprises or for production use, you should use a proper PKI solution (Certificates) either from a 3rd-party or if you have your own Certificate Service then use that. As the Splunk CA is one that gets shipped out in every download and as such the communications aren't as secure... For in the logs on your Linux A system (Splunk Enterprise) you should see something similar to the below lines for the various port inputs usually after all of the "HotDBManager" & "IndexWriter" messages... 07-22-2018 12:49:57.748 +1000 INFO TcpInputConfig - IPv4 port 517 is reserved for raw input 07-22-2018 12:49:57.748 +1000 INFO TcpInputConfig - IPv4 port 517 will negotiate s2s protocol level 4 07-22-2018 12:49:57.748 +1000 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk 07-22-2018 12:49:57.748 +1000 INFO TcpInputConfig - IPv4 port 9997 will negotiate s2s protocol level 4 07-22-2018 12:49:57.754 +1000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 514 with Non-SSL 07-22-2018 12:49:57.755 +1000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 515 with Non-SSL 07-22-2018 12:49:57.768 +1000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 516 with Non-SSL 07-22-2018 12:49:57.768 +1000 INFO TcpInputProc - Creating raw Acceptor for IPv4 port 517 with Non-SSL 07-22-2018 12:49:57.768 +1000 INFO TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with Non-SSL Regarding the web GUI on Linux A, any configuration on Linux B will have zero effect upon this... What I would say is confirm that the user that which Splunk is running under has permissions to open ports, or has ownership of the entire "/opt/splunk/" folder structure. For this would you mind sending in what your web.conf looks like, also check your splunkd.log for any errors, I believe if you look after the "TailReader" / "TailingProocessor" entries there should be couple items from "loader" talking about REST HTTP server, then two X509Verify entries... If there are issues this is where it probably will mention what is causing the web gui from working. Could possibly need to look at your mongod.log file as well to make sure that it starting correctly (esp. if you have changed the sslPassword setting within server.conf): ... 07-22-2018 12:50:05.910 +1000 INFO TailingProcessor - Adding watch on path: /var/adm. 07-22-2018 12:50:05.910 +1000 INFO TailingProcessor - Adding watch on path: /var/log. 07-22-2018 12:50:05.917 +1000 INFO loader - Limiting REST HTTP server to 1365 sockets 07-22-2018 12:50:05.917 +1000 INFO loader - Limiting REST HTTP server to 303 threads 07-22-2018 12:50:05.917 +1000 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts y our Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself- signcertificates> 07-22-2018 12:50:06.291 +1000 WARN X509Verify - X509 certificate (O=SplunkUser,CN=splunk) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instan ce at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see: <http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates> When you restart Splunk from the command line, I would recommend using the splunk binary instead of "systemctl" if that is what you are using as the "splunk" binary will provide some basic output on each stage... e.g. Stopping splunk helpers... [ OK ] Done. Splunk> Like an F-18, bro. Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbucket add_on_builder_index car_data cim_modactions cim_summary firedalerts history main os os_metrics perfmon pos_pu sophos summary synology syslog unifi windows wineventlog Done Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from '/opt/splunk/splunk-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Done [ OK ] Waiting for web server at https://127.0.0.1:8000 to be available......................... Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at https://splunk:8000 ... View more

No, it will show all data that the user(s) have access to for that time range But note that if the user is in Aisa they are roughly 6-10 hours ahead of Europe, so last 4 hours won't show that data because that would be "in the future" for Europe... I believe for the Europe user if selecting "last 4 hours", then it will show data that is current for Europe but because of timezones will be several hours old of the Asia systems... The time for the events (if on "list" view in search), will show it in the user's configured time zone. Whereas the _raw event will show the original time no matter what is set in Splunk. This is why most places that have international presence will just set all systems to UTC, then Splunk, Windows, etc... will convert the time to the user's configured timezone but the data coming in will be in the one timezone... ... View more

Hi ahmemohs03, What does the inputs.conf look like on your Linux A to receive these connections? Also what do you mean about weburl not coming up? Is this actually about two issues: One the data from the UF (Linux B) -> Splunk Enterprise (Linux A)? Two the issue with Splunk Enterprise Web not coming up? ... View more

Is Splunk running as SYSTEM or as a user and if as a user does it have the required permissions to listen to ports on Windows? Have you checked splunkd.log when it starts up for "TcpInput" type components and if there are any issues it is reporting? ... View more

Hi premranjithj, So as long as the data incoming from the various systems are configured with the timezone of their source. i.e. If a server from Asia has in its props.conf the "TZ = ..." set to its local timezone. Then Splunk will automatically convert this in its backend to UTC, then it will show the event time in the user's configured timezone that was/is set within their account preferences... No matter what timezone it is actually coming from. If you don't tell Splunk that the data coming in is in a different timezone (non-UF data) then it will presume it is UTC. If it is via a UF agent then I believe the UF will recognise the system time of where it is installed... Hopefully that helps you ... View more

Does the LINE_BREAKER Regex require full regex? Can't remember or not, as if so you might need to change the spaces to "\s" instead. Also the brackets around the "Seconds" if not a capture group will need to be escaped "\". Forward slash isn't a special character as such doesn't need to be escaped: "\d{2}\/\d{2}" So if it does require full regex then the below is what it should look like to a degree. If you read the props.conf on LINE_BREAKER you will notice that the Capture Group "()" is what it breaks upon and doesn't ingest, i.e. you shouldn't see the "User Time..." coming up in any of the events LINE_BREAKER = User\s+Time\s+\(Seconds\)\s+:\s+\d+\s+=\s+\w{3}\s+\d{2}/\d{2}/\d{2}\s+\d{2}:\d{2}:\d{2}\s+[A-Z]+([\r\n]+) The contents of the first capturing group are discarded, and will not be present in any event. You are telling Splunk that this text comes between lines ... View more

I would probably suggest not using both LINE_BREAKER and BREAK_ONLY_BEFORE in the same props stanza. As they are to do the same job to a degree (Performance wise use LINE_BREAKER). Pick one of these as LINE_BREAKER happens within the Parsing Pipeline and BREAK_ONLY_BEFORE (and the other similar ones) happens within the Merging Pipeline. Then SEDCMD happens in the Typing queue.. https://wiki.splunk.com/File:Splunk_EventProcessing_v19_0_standalone.png ... View more

Just to clarify: TRUNCATE (Default 10000) is for the size of an event in Bytes. MAX_EVENTS (Default 256) is for max number of lines per event ... View more

Look within the _internal index for the answers and to get at the issue faster use: These errors are the ones related to TIME_FORMAT or LINE_BREAKER errors: index=_internal source=*splunkd.log component=DataParserVerbose WARN OR ERROR For some related to Line Breaking issues: index=_internal source=*splunkd.log component=LineBreakingProcessor WARN OR ERROR These are the ones that will be related to MAX_EVENTS (256 Lines by default) & TRUNCATE (10,000 bytes by default) which are some of the top two causes but there are many others... A good 2016 Splunk .Conf preso (also one in '13 & '14) is the "Jiffy lube quick tune up for you Splunk environment": https://conf.splunk.com/files/2016/slides/jiffy-lube-quick-tune-up-for-your-splunk-environment.pdf ... View more

Hi Kozanic, Not sure why it only returns some results if doing just a basic search but if the attribute is in the LDAP schema then ldapsearch will pick it up, you just need to place the extra fields into a table output e.g. | table sAMAccountName, personalTitle, displayName, ..., pwdLastSet, badPasswordTime, badPwdCount, logonCount, etc.... ... View more

Hi, Believe this is what you are looking for: | inputlookup users.csv | fields identity, pwdLastSet | eval dateLastSet=strptime(pwdLastSet, "%Y-%m-%dT%H:%M:%S.%6NZ") | eval epochNextSet=(dateLastSet+7776000) | eval diff=epochNextSet-now() | eval days=diff/86400 | eval dateNextSet=strftime(epochNextSet, "%Y-%m-%d %H:%M:%S") The last line is optional as it will just provide the exact date on which they will need to change their password, also here is a screenshot of what this shows. Just round the days value to whatever you want... And then add in the fields you want on the table as well. ... View more

Might be worth looking at the opseclea_connection.conf file in the ../local/ folder and seeing if the settings match what you have configured in Check Point. Also remember they are case sensitive; password cannot contain certain special characters; reapply the password in Check Point after each failed attempt incase after the first failed try it blocks it out; and that all the other settings in the file match your environment as well. https://docs.splunk.com/Documentation/AddOns/released/OPSEC-LEA/Configureinputs Edit: Oh and on the end of the cert script it is a number one (1) right not an l (L) that you are running?? ... View more

Hi, Looking at the doco Mgt Server IP isn't that of the Splunk server but of the Check Point Mgt Server, if it is a standalone environment (#6.2 on doco page). I would suggest confirming all of the steps in the doco setup, then if it still isn't working provide here: Which step you believe it failed upon The stderr lines within the splunkd.log referenced in the error message Check the ../certs/ folder within the app - and the permissions for the folder/files Output from the web_service.log reference the troubleshoot section of the linked doco page ... View more

Hi, Could you expand on what you are trying to do and provide an example of what is currently shown and what you want it to show? ... View more

Hi, All of the windows inputs are disabled by default, on both the forwarders and the Windows TA unless you enabled them when installing the UF? Also what are you trying to archive 1) Stop splunkd from creating sub-processes altogether; or 2) Reduce the amount of 4688 events being ingested? For 1) yes the only way to stop it from generating new sub-processes is to stop it monitoring any events... Splunk along with applications like Chrome believe that creating sub processes for each task/tab is better than having one very large process i.e MS IE... So they have one main process that then spawns the sub-processes to monitor what you have configured. If you want to just reduce the amount of events being ingested back into Splunk then you can use a blacklist filter... Also making sure that you restart splunkd on the forwarders to make sure the changes have taken effect. From the default inputs.conf file: ... To set custom configurations, place an inputs.conf in $SPLUNK_HOME/etc/system/local/. For examples, see inputs.conf.example. You must restart Splunk to enable new configurations. An Example: [WinEventLog://Security] disabled = 0 blacklist.0 = EventCode="4688" Message=”.*New\sProcess\sName:\s+C:\\Program\sFiles\\Splunk\\bin\\splunk\-.*” This will blacklist most of splunks sub-processes, there are a few like mongod, python, powershell that the above regex won't pick up but you can always just keep adding those as new blacklists (blacklist.1 etc...). ... View more

Hi, So I have seen this issue before and actually ran into it again on the current customer site I am working upon. It appears when running Splunk Enterprise 6.5.x and any of the DBX 3.x versions so it isn't you and even the latest package (3.1.1) doesn't have a checklist.conf.spec file. The only one that comes with the system that I can see is within the "splunk_monitoring_console" app, but even coping this over to the DBX's README folder it still throws up the errors. I presume it is a limitation within Splunk 6.5.x and the best course would be to locally edit the "default" checklist.conf files to comment out these lines until you upgrade past Enterprise 6.5.x as with Enterprise 6.6.x and 7.0.x this doesn't seem to be an issue at all. ... View more

From a few searches I have done based upon any type of standard that Statsd or Collectd have, there are a few that make sense: schema.application_name.namespace.metric_name.metric_type So for something as simple as OS Metrics l believe for the "metric_name" field going along the lines of: os.<type>.<sub_type>.<metric> e.g: - os.cpu.percent - os.cpu.interrupts_sec - os.mem.swap.percent - os.mem.pages_sec - os.system.threads - os.process.mem.used - os.disk.free.percent etc... Refs: https://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/ https://collectd.org/wiki/index.php/Naming_schema https://www.slideshare.net/itnig/collecting-metrics-with-graphite-and-statsd ... View more

While it hasn't been updated since 2013 there haven't been too many changes to the Windows event logs to make it significant enough to be outdated but this NSA document does help a lot: (Page 8 for Overall list; Page 24-34 for in depth info in each category) https://www.nsa.gov/ia/_files/app/spotting_the_adversary_with_windows_event_log_monitoring.pdf Then with the various types of Logon Types for a login event; e.g. Logon Type 7 is Unlock, 10 Interactive, etc... Try this SANS white paper: https://www.sans.org/reading-room/whitepapers/forensics/windows-logon-forensics-34132 ... View more