About mguhad

mguhad · ‎03-26-2021

I basically i want to get a timestamp of when an index created in Splunk...I am aware that the timestamp of when each index was created is placed under "$SPLUNK_HOME/var/lib/splunk/<index_name>/db/CreationTime" Similarly, how do i get when a Search Head cluster app/bundle was created similar to above?

mguhad · ‎06-13-2020

Hi, Since you are already leaning toward IMF layer setup, it is simply much better to install a universal forwarder on each syslog server, then have those UFs forward their data to the HFs for parsing and then HFs pass on to indexers to be written to disk. With that in mind, below are few resources that can steer you best: https://docs.splunk.com/Documentation/Splunk/8.0.4/Forwarding/Forwarderdeploymenttopologies https://www.splunk.com/en_us/blog/tips-and-tricks/using-syslog-ng-with-splunk.html 1. Depends on your use-case. Running both concurrently can land you in resource consumption situation especially if a lot of parsing is expected on that one HF (poor LB etc.). Instead, if possible, seperate them as explained above. Have a UF on each syslog server then have them send to HFs for parsing and those HFs can parse data from other UFs etc. Or the other way round which ever fits your use-case. 2. Please have a look here: https://docs.splunk.com/Documentation/Splunk/8.0.4/Capacity/Referencehardware We have it layered whereby load is spread across quite a few HFs that do all the parsing before we index any data. so 2TB is not much at all if you're distributing that load and can thus get away with Mid-range or standard spec servers that your IMF send to. However, if you are beaming all that data to 1 or 2 (depending on the UF<>HF ratio) you would definietly need to consider much beefier, high spec servers. Hope this helps!

mguhad · ‎05-29-2020

@aamer86 , Yes well then in that case you will need to keep an eye on the pass4symkey (it basically MASKS plaintext passwords (i.e from -secret key). Once you have new members joining the cluster/env...simplly comment out their current pass4symkey/sslpassword value, then push the new value out ina app - you will notice splunk will automatically regenerate that value and encrypt that plain text password) . Pass4Symkey and sslpassword are two completely different things to keep an eye on when using custom SSL certs & clustering. Just ensure that all nodes can have access to the correct SSLpassword parameter (this is also generated automatically by splunk using its default certs, if you custom ones, simple comment this out, apply your custom certs and splunk will generate new sslpassword based on what you have just configured like above for pas4symkey.) More on that can be found here: https://docs.splunk.com/Documentation/Splunk/8.0.4/Security/WhatyoucansecurewithSplunk

mguhad · ‎05-28-2020

@aamer86 Yes, the defaults should work fine. You generally wouldnt want to fiddle with parameter that unless you are doing clustering (only applicable to indexers adn Cluster master) or SSL encryption (applicable to UFs too). It won't affect your clusters indexing/DS unless you have configured SSL and have set "requireClientCert = true" on the indexers etc. The values will always be differnt because of how splunk Hashes the plaintext password - passwords from sslpassword, -secret from clustering etc. But if you are not configuring ssl, you just need to configure outputs.conf on the UFs to send data to the indexers' recieving port (inputs.conf on indexers.) Please accept this answer if it has helped so others can benefit. Thanks.

mguhad · ‎05-21-2020

It will impact the license if they are sharing the same LM. If the local instance is of a differnt LM, then theortically no. But if they are sharing the same License, then yes. Remember, those HFs (that are usually not configured for indexing data) will NEED enterprise license to have the capability of indexing data before forwarding meaning this data will have to pass through the license meter before being written to disk thus innevetibly consuming your license. On that manual you shared, it states certain caveats that you should be aware of if you are utalising HFs such as: https://docs.splunk.com/Documentation/Splunk/8.0.2/Forwarding/Routeandfilterdatad#Perform_selective_indexing_and_forwarding *If you set indexAndForward to "true" and then filter out some data through forwardedindex blacklist attributes, the forwarder does not forward the blacklisted data, but it does still index the data. I hope this helps.

mguhad · ‎05-21-2020

To verify, please run this search on the SH (if all nodes are sending their internal logs to the indexing layer) : index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table _time hostname version sourceIp destPort ssl alternatively you can check manually verify the port using the openssl suite: /opt/splunk/bin/splunk cmd openssl s_client -connect : https://docs.splunk.com/Documentation/Splunk/7.3.1/Security/Validateyourconfiguration Hope this helps!

mguhad · ‎05-21-2020

The pass4SymKey is splunk's special key to let your indexers or other relevent instances (SHs, CM<>IDX) communite and authenticate with each other. In short, pass4Symkey controls the authentication between splunk instances and does not control user access. see here for more infro : https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/Aboutsecuringclusters. SSL is related to the encryption of comms/data between splunk instances and is unrelated to the functionality of the pass4Symkey as you can configure indexing of data without SSL (its optional, but highly recommended step for hardenening your env against casual snoopers etc..). For indexing a cluster peer communications, SSL is optional. But you MUST use pass4symkey otherwise the cluster peers will not be able to join the cluster and authenticate against the cluster master. You do this by setting the security key* ( -secret ) on the cluster peers which is then converted to pass4Symkey (server.conf). Simply configure the peers to have same -secret as Cluster master and then open the 9997 port on the indexers (inputs.conf as an app from CM) and you should be good to go in terms of indexing data! *Security key - This is the key that authenticates communication between the master and the peers and search heads. The key must be the same across all cluster nodes. Set the same value here that you previously set on the master node. More reading here: https://docs.splunk.com/Documentation/Splunk/8.0.3/Indexer/Enablethepeernodes Hope this helps!

mguhad · ‎04-15-2020

Hi , This is a very common issue, that is captured in the documentation as well. please visit this page of the documentation as they walk you through possible solutions/clues: https://docs.splunk.com/Documentation/Splunk/8.0.3/Troubleshooting/Cantfinddata

mguhad · ‎04-15-2020

Oh yeah then you will need to visit the Oracle docs. However,looking at the system requirements, the instance type will become obvious just a different name as these requirements are dictated by system needs and not product (instance type) names etc. Have a look at the min specs and simply pick an oracle VM that corresponds to that?

mguhad · ‎04-04-2020

Hi, If it is simply a universal forwarder: 1.Ensure that the new db host has the same source paths for the files it is monitoring as the old one 2.Ensure the deployment server is up to date with the client (that way, once the UF phones home under new db host, it can simply push the same app out again to resume service as usual....the caveat being the new db host has the same exact input.conf stanzas as the old db you are moving from) 3. Install the forwarder on the new db host 4. Point the forwarder to the deployment server : ./splunk set deploy-poll <deploymentserver>:8089 5. Once the forwarder is phoning home from the new db host, attach the same apps & serverclass as previous db forwarder and push out to the client - ./splunk reload deploy-server Once the new forwarder begins to send data to the indexers, the dashboards and other knowledge objects should begin populating as per usual witht the previous setup.

mguhad · ‎04-03-2020

Hi, i have seen this happen before where the deploymentclient.conf is initially place under /etc/apps/local/deploymentclient.conf and then once the client has established connection with deployment server (phones home), the same app is configured on the deployment server and pushed out to client which overrides the initial app () and this is what causes the breaking as splunk warns against this in below documentation snippet: https://docs.splunk.com/Documentation/Splunk/7.2.6/Updating/Configuredeploymentclients Important: Do not use the deployment server to push deploymentclient.conf updates to the deployment clients. Doing so can potentially lead to situations where the deployment clients lose their ability to contact the deployment server. You should also check for this kind of error as splunk tries to stack the app containing the deploymentclient.conf configs on top of each other due to the app being wrongly pushed out via deployment sever as warned against above: ERROR DeployedApplication - untarToDest, failed to mkdir opt/splunkforwarder/etc/apps/<my_app>/<my_app>/<my_app>/<my_app>/<my_app>/local: Permission denied To fix this issue, What we did we removed the app with deploymentclient.conf configs from the deployment server and re-configured the deployment client app again on the remote server or ran ./splunk set deploy-poll <IP_address/hostname>:<management_port>

mguhad · ‎04-02-2020

@iiooiiooiioo cheeck if your forwarder (myhost) are actually sending data at all to the _internal index. index=_internal host=myhost ALternatively, check to see if the main index has ANY data : | eventcount summarize=false index=* OR index=_*

mguhad · ‎04-01-2020

Hi, Splunk does not distinguish between the various cloud vendors as long as the VMs provided adhere to Splunk's hardware requirements & recommendations found below: With that, irrespective of the cloud provider, it should work fine. We have used the above documentaiton when we were moving our on-prem clusters to IBM's cloud platform. Cheers

mguhad · ‎04-01-2020

Hi, Since you didnt specify an index, splunk will, by default, place your data in the 'main' index. The server would check in under its hostname/ip address so you could this in your host= parameter. So you could try index=main host=<myhost> or <ipdress> If you want to find out the hsotname of the forwarder: ./splunk show default-hostname then pass this hostname in your search: index=main host=<output_of_above_command> Best practice is to simply create an inputs.conf file either under /system/local or /etc/apps//local and monitor files that way assuming you have configured the outputs.conf to send data to the indexers (unless standalone-all-in-one box).

mguhad · ‎02-11-2020

Just an update: The issue was NOT splunk issue but instead turned out to be firewalld that was accidentally configured by unix maintenance team after patching which disabled ALL ports on the server. This affected servers individually hence why some indexers worked and some didn't. After firewalld was disabled on all the indexers/CM, all worked fine again and the All indexers were visible on CM's dashboard again. Deployment server was also now reached from GUI and it was able to send internal logs to indexers + 8089 on DS began working fine as expected. solution : turn off firewalld on each server thanks again @nickhillscpl for your input which has led to me finding out the issue.

mguhad · ‎02-11-2020

You can eliminate unwanted data by routing it to the nullQueue and it is removed before indexing with a combination of props and tranforms .conf like below: props.conf: [web_logs] TRANSFORMS-removeIP= exclude transforms.conf: [exclude] REGEX = src_ip=\d{2}\.\d{1}\.\d{1}\.\d{1}\/8$ DEST_KEY = queue FORMAT = nullQueue For methods to keep segments of the data and discard the rest, please the docs here:https://docs.splunk.com/Documentation/Splunk/8.0.1/Forwarding/Routeandfilterdatad

mguhad · ‎02-10-2020

We are using ip addresses rather than DNS so that would not work. But when i ping indexers from CM, no connection at 8089. PING works fine, just port specific becomes an issue. But when the idx are the origin, they can connect to CM fine just not the other way round. That might explain the "CMRepJob" errors on the CM as it can't connect to the indexers? Simply trying to rule out whether it is a splunk issue or OS/server routing issue. Very Odd behaviour. I have enabled maintenence mode now on the CM. Will chase this up with networks and raise a ticket if all fails.

mguhad · ‎02-10-2020

No nothing fixed itself only 1/6 indexers is showing on my CM dashboard, so the cluster is in an unhealthy loop as it can't meet RF/SF. BUT... wheni ran the above command from CM <>indexers rather than idx<>idx and i am not able to establish connection at all. Only able to establish connection with the only working indexer. so does this mean cm server is at fault here when its originating.

mguhad · ‎02-10-2020

Oh ok got it! thanks. So it gets odd from here. I am able to connect (get XML response) to all but 2 of the indexers. i am also able to connect to the CM with that command! So if i can connect to the CM, why isnt this indexer actually connecting to the CM at 8089 and showup on the CM's dashboard? This has confused me more now.

mguhad · ‎02-10-2020

hmm. What does that command prove/verify? and yes great, i am able to get an XML response. So does this conclude the issue is indeed not Splunk as an application and more to do with OS/networking layers/routing issue of the server ?

mguhad · ‎02-10-2020

Yes but splunkd remain running. general PING without specifying works. However running a telnet between the indexers at port 8089 fails. Trying myidx no route to host Failed to connect to myidx:8089; No route to host closing connection 0 curl:(7) Failed connect to myidx:8089;No route to host Thats on all indexer<>Indexer comms except the only one that is working for some reason which connects at 8089. This is also the case for DS trying to connect to indexers at 9996...same above error. From what i can see and the troubleshooting i have done, it all seems beyond splunk and OS/network related or am i missing something.

mguhad · ‎02-10-2020

We are using IP addresses so no DNs at all. Although i havent asked about what ACL mode is being used

mguhad · ‎02-10-2020

Well understood. The networking team have discounted it as application specifc issue not firewall which i disagree with for reasons you mention above. But i want to clarify is, those issues are NOT being caused by splunk (or any of its configrations) but are instead being cuased by the routing/networking configurations on the servers/OS right?

mguhad · ‎02-10-2020

My indexer cluster is down except for 1 out of 6. 8089 is suddenly not working for indexers and CM<>indexer comms and i get the below error messages. Its a multi site indexer cluste. I have ran telnet and curl commands on 8089 & indexers but still unable to connect to all but 1/6 indexers. Also, deployment server is not accessible. CM is unable to connect to 8089 for the indexers, the indexers cannot talk to each other on port 8089 either and the DS is not able to connect to my indexers at 9996. FYI custom SSL is enabled at 8089 but i don't see as the cause for this connectivity issue. I have checked with networking team who are saying its an application issue and not iptables/routing issue on the server like i suspected. Please help. IDX: 02-10-2020 03:19:20.324 +0000 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=myCM:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=myidx mgmtport=8089 (reason: http client error=No route to host, while trying to reach https://myidx:8089/services/cluster/config). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=EF3B7708025567663732F8D6B146A83 add_type=Clear-Masks-And-ReAdd base_generation_id=2063 batch_serialno=1 batch_size=1 forwarderdata_rcv_port=9996 forwarderdata_use_ssl=1 last_complete_generation_id=0 latest_bundle_id=EF3B77080255676637732F8D6B146A83 mgmt_port=8089 name=EEC311D7-7778-44FA-B31D-E66672C1D568 register_forwarder_address= register_replication_address= register_search_address= replication_port=9100 replication_use_ssl=0 replications= server_name=myidx site=site3 splunk_version=7.2.6 splunkd_build_number=c0bf0f679ce9 status=Up } ]. CM: 02-07-2020 18:00:41.497 +0000 WARN CMMaster - event=heartbeat guid=BDD6A029-2082-48ED-96F3-21BD624D94CD msg='signaling Clear-Masks-And-ReAdd' (unknown peer and master initialized=1 02-07-2020 18:00:41.911 +0000 WARN TcpOutputFd - Connect to myidx:9996 failed. No route to host 02-07-2020 18:00:41.912 +0000 WARN TcpOutputProc - Applying quarantine to ip=myidx port=9996 _numberOfFailures=2 02-07-2020 18:00:42.013 +0000 WARN TcpOutputFd - Connect to myidx:9996 failed. No route to host 02-07-2020 18:00:42.323 +0000 WARN CMMaster - event=heartbeat guid=44AF1666-AB56-4CC1-8F01-842AD327CF79 msg='signaling Clear-Masks-And-ReAdd' (unknown peer and master initialized=1 02-07-2020 10:36:54.650 +0000 WARN CMRepJob - _rc=0 statusCode=502 transErr="No route to host" peerErr="" 02-07-2020 10:36:54.650 +0000 WARN CMRepJob - _rc=0 statusCode=502 transErr="No route to host" peerErr="" DS trying to connect to indexers: 02-07-2020 11:56:12.097 +0000 WARN TcpOutputFd - Connect to idx2:9996 failed. No route to host 02-07-2020 11:56:12.098 +0000 WARN TcpOutputFd - Connect to idx3:9996 failed. No route to host 02-07-2020 11:56:13.804 +0000 WARN TcpOutputFd - Connect to idx1:9996 failed. No route to host

mguhad · ‎02-03-2020

This is true, i kept on getting this error and when i looked at the command i was trying to run more closely, i found there was indeed a syntax error! Thank you !!!

Posts	42
Solutions	5
Karma Given	43
Karma Received	4
Member Since	‎07-11-2018

Online Status	Offline
Date Last Visited	‎07-13-2023 06:42 PM

How do i get tiemstamp of when an index was create...

No route to host at 8089 cluster

Which Deployer Push Mode

How do i get tiemstamp of when an index was create...

Re: Uf + HF on on server

Re: Difference between pass4SymmKey and SSL in a d...

Re: Difference between pass4SymmKey and SSL in a d...

Re: Index one input locally and then forward all i...

Re: Verifying Secure Communication between forward...

Re: Difference between pass4SymmKey and SSL in a d...

Re: no data being ingested

Re: Deploying Splunk to Oracle Cloud Platform

Re: steps to move universal forwarder from oracle ...

Re: Why is the Splunk universal forwarder dropping...

Re: Why is host=myhost giving no results?

Re: Deploying Splunk to Oracle Cloud Platform

Re: Why is host=myhost giving no results?

Re: No route to host at 8089 cluster

Re: how to send events to nullque?

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

Re: No route to host at 8089 cluster

No route to host at 8089 cluster

Re: Cannot perform action POST without a target na...