"This website is not affiliated with Splunk, Inc. and is not an authorized seller of Splunk products or services."
  • Home - Splunk Tutorial
  • Splunk training videos
  • Splunk interview questions
  • Contact US
  • About Us
  • Privacy Policy
  • Splunk Jobs

                                                                                                      <  Back |Home| Next  >


splunk common issues and solutions- 

 Below we have listed some of the known splunk issues and solutions :

1. splunk/splunkd service is not starting or splunk unable to start error

sample error in splunkd log
  1. 01-30-2013 18:29:05.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
  2. 01-30-2013 18:29:15.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
  3. 01-30-2013 18:29:25.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
  4. 01-30-2013 18:29:35.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.

Solution:-

 It's probably a locked file; to remove it:
  1. Stop splunk
  2. Remove any locked files
  3. Use this command ./splunk clean locks
  4. Or manually remove the file (`*.pid` or *.lock) in $SPLUNK_HOME\var\run\splunk\
  5. Start splunk and verify in the splunkd.log


2. Splunk forwarder not forwarding data to splunk


Solution:  Refer http://www.learnsplunk.com/splunk-forwarder-not-sending-data.html



3. Too many search jobs found in the dispatch directory error

 sample error message

Too many search jobs found in the dispatch directory (found=3692, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs 


Solution:

 
If you see this error you can manually clear out any jobs in the dispatch folder.The number of directories relating to the search artifacts in the Dispatch directory can potentially affect search performance since we have to scan each of the directories to determine if the artifacts are present or not.We prefer deleting old search jobs first :)


location of dispatch directory:

$SPLUNK_HOME/var/run/splunk/dispatch

you can manually delete old searches or run following command run following command to delete old searches:

/opt/splunk/bin/splunk cmd splunkd clean-dispatch /temp -1d -1d | tee -a /opt/splunk/var/log/splunk/internalscripts.log

for more info on dispatch directory refer below link:
http://blogs.splunk.com/2012/09/10/a-quick-tour-of-a-dispatch-directory/


4. Too many license violations error

Sample error message:

<code>"Error in 'UnifiedSearch': Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting <a href="http://www.splunk.com/store">www.splunk.com/store</a> or calling 866.GET.SPLUNK. </code>



Solution:

1) You're seeing this message because you indexed more data than your license allows, 5 times or more in the last 30 days. If the increase in volume is unexpected, use the searches on this page to help you figure out where this extra data came from. Searches on the '_internal' index should not be blocked

2) If you are an enterprise customer with a valid support agreement, in order to remove the error message and get all of your searches working again, you should file a case with Splunk Support and request a reset key. Make sure you have found and addressed the cause of the violations before you do this, because it's only going to happen again if you don't. If you are using a free license you will need to have 14 days of license compliance before searching is restored

3) You can also set up a search to run once daily and check on the license violation count, and alert you if it increases. The below search will only return results if the quota has incremented:

  1. index=_internal source=*license_audit.log LicenseManager-Audit | delta quotaExceededCount as quotadiff | stats first(quotadiff) as quotadiff | search quotadiff>0
 
You can use the instructions in the docs here to set this up










Comment Box is loading comments...
Powered by Create your own unique website with customizable templates.