< Back |Home| Next >
splunk common issues and solutions-
Below we have listed some of the known splunk issues and solutions :
1. splunk/splunkd service is not starting or splunk unable to start error
sample error in splunkd log
It's probably a locked file; to remove it:
2. Splunk forwarder not forwarding data to splunk
Solution: Refer http://www.learnsplunk.com/splunk-forwarder-not-sending-data.html
3. Too many search jobs found in the dispatch directory error
sample error message
Too many search jobs found in the dispatch directory (found=3692, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs
Solution:
If you see this error you can manually clear out any jobs in the dispatch folder.The number of directories relating to the search artifacts in the Dispatch directory can potentially affect search performance since we have to scan each of the directories to determine if the artifacts are present or not.We prefer deleting old search jobs first :)
location of dispatch directory:
$SPLUNK_HOME/var/run/splunk/dispatch
you can manually delete old searches or run following command run following command to delete old searches:
/opt/splunk/bin/splunk cmd splunkd clean-dispatch /temp -1d -1d | tee -a /opt/splunk/var/log/splunk/internalscripts.log
for more info on dispatch directory refer below link:
http://blogs.splunk.com/2012/09/10/a-quick-tour-of-a-dispatch-directory/
4. Too many license violations error
Sample error message:
<code>"Error in 'UnifiedSearch': Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting <a href="http://www.splunk.com/store">www.splunk.com/store</a> or calling 866.GET.SPLUNK. </code>
Solution:
1) You're seeing this message because you indexed more data than your license allows, 5 times or more in the last 30 days. If the increase in volume is unexpected, use the searches on this page to help you figure out where this extra data came from. Searches on the '_internal' index should not be blocked
2) If you are an enterprise customer with a valid support agreement, in order to remove the error message and get all of your searches working again, you should file a case with Splunk Support and request a reset key. Make sure you have found and addressed the cause of the violations before you do this, because it's only going to happen again if you don't. If you are using a free license you will need to have 14 days of license compliance before searching is restored
3) You can also set up a search to run once daily and check on the license violation count, and alert you if it increases. The below search will only return results if the quota has incremented:
You can use the instructions in the docs here to set this up
sample error in splunkd log
- 01-30-2013 18:29:05.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
- 01-30-2013 18:29:15.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
- 01-30-2013 18:29:25.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
- 01-30-2013 18:29:35.094 +0000 WARN loader - Sleep 10 sec, waiting for config lock.
It's probably a locked file; to remove it:
- Stop splunk
- Remove any locked files
- Use this command ./splunk clean locks
- Or manually remove the file (`*.pid` or *.lock) in $SPLUNK_HOME\var\run\splunk\
- Start splunk and verify in the splunkd.log
2. Splunk forwarder not forwarding data to splunk
Solution: Refer http://www.learnsplunk.com/splunk-forwarder-not-sending-data.html
3. Too many search jobs found in the dispatch directory error
sample error message
Too many search jobs found in the dispatch directory (found=3692, warning level=2000). This could negatively impact Splunk's performance, consider removing some of the old search jobs
Solution:
If you see this error you can manually clear out any jobs in the dispatch folder.The number of directories relating to the search artifacts in the Dispatch directory can potentially affect search performance since we have to scan each of the directories to determine if the artifacts are present or not.We prefer deleting old search jobs first :)
location of dispatch directory:
$SPLUNK_HOME/var/run/splunk/dispatch
you can manually delete old searches or run following command run following command to delete old searches:
/opt/splunk/bin/splunk cmd splunkd clean-dispatch /temp -1d -1d | tee -a /opt/splunk/var/log/splunk/internalscripts.log
for more info on dispatch directory refer below link:
http://blogs.splunk.com/2012/09/10/a-quick-tour-of-a-dispatch-directory/
4. Too many license violations error
Sample error message:
<code>"Error in 'UnifiedSearch': Your Splunk license expired or you have exceeded your license limit too many times. Renew your Splunk license by visiting <a href="http://www.splunk.com/store">www.splunk.com/store</a> or calling 866.GET.SPLUNK. </code>
Solution:
1) You're seeing this message because you indexed more data than your license allows, 5 times or more in the last 30 days. If the increase in volume is unexpected, use the searches on this page to help you figure out where this extra data came from. Searches on the '_internal' index should not be blocked
2) If you are an enterprise customer with a valid support agreement, in order to remove the error message and get all of your searches working again, you should file a case with Splunk Support and request a reset key. Make sure you have found and addressed the cause of the violations before you do this, because it's only going to happen again if you don't. If you are using a free license you will need to have 14 days of license compliance before searching is restored
3) You can also set up a search to run once daily and check on the license violation count, and alert you if it increases. The below search will only return results if the quota has incremented:
- index=_internal source=*license_audit.log LicenseManager-Audit | delta quotaExceededCount as quotadiff | stats first(quotadiff) as quotadiff | search quotadiff>0
You can use the instructions in the docs here to set this up
Comment Box is loading comments...
