Troubleshooting license violation error in splunk:
Handling lice violation is very common issue faced by Splunk admins.Sometimes it may take longer time to identify the root cause of violation if environment is large.Below we have collected most useful queries which will be helpful to identify the source of violation.
What happens when I exceed my Enterprise license limit?
"If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have 5 or more violations on an Enterprise license or 3 violations on a Free license in a rolling 30-day period, search will be disabled. Search capabilities return when you have fewer than 5 (Enterprise) or 3 (Free) violations in the previous 30 days or when you apply a new license with a larger volume limit.
Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks access while you exceed your license.
Note: Searches to the _internal index are not disabled even during a licensing-enforcement period, so you can still access the Indexing Status dashboard, or run searches against _internal to diagnose the licensing problem. "
so: you can exceed your Enterprise license 4 times within 30 days--the 5th time, search will be disabled. You can exceed your Free licenses 2 times, and the 3rd time, search will be disabled.
What happens when I exceed my Enterprise license limit?
"If you exceed your licensed daily volume on any one calendar day, you will get a violation warning. The message persists for 14 days. If you have 5 or more violations on an Enterprise license or 3 violations on a Free license in a rolling 30-day period, search will be disabled. Search capabilities return when you have fewer than 5 (Enterprise) or 3 (Free) violations in the previous 30 days or when you apply a new license with a larger volume limit.
Note: During a license violation period, Splunk does not stop indexing your data. Splunk only blocks access while you exceed your license.
Note: Searches to the _internal index are not disabled even during a licensing-enforcement period, so you can still access the Indexing Status dashboard, or run searches against _internal to diagnose the licensing problem. "
so: you can exceed your Enterprise license 4 times within 30 days--the 5th time, search will be disabled. You can exceed your Free licenses 2 times, and the 3rd time, search will be disabled.
For the first option here are the steps :
1 - Analyze you data, to identify where the volume it is coming from.
use those searches on the license-master
see http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume
if you prefer detail, you can add details on the source "s", host "h", sourcetype "st", indexer "i".
total per pool index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by pool
detail per sourcetype
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by st useother=false
detail per source
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by s useother=false
detail per host
index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by h useother=false
2- If some forwarders are not necessary, turn splunk forwarder off on those boxes.
Why did you deployed a forwarder on every single box in the first place !!!
3- If some useless files are being indexed, be more selective.
Disable the inputs, or use whitelist/blacklists to limit the scope
example to drop the core files, or to index only *.log files:
[montitor:///var/log] blacklist=\.core$ [monitor:///mypath/*\.log]
4 - If some servers are sending to much data (syslog by example)
disable the routing to splunk, or select the components to send.
example on syslog.conf (send only critical and errors, and every event from my application)
*.CRITICAL splunk.mydomain.com *.ERROR splunk.mydomain.com myapplication.* splunk.mydomain.com
5 - If some log files contains too much data, change the verbosity level of your applications (by example, avoid the DEBUG mode)
6- Search for duplicates events in the logs, please check they exists in the original logs, or if the same log file is being indexed several times (some log rotation may cause that)
here are searches to find duplicates in splunk :
* | eval raw=_raw | convert ctime(_indextime) as indextime | stats count first(indextime) as first last(indextime) as last by raw | where count > 1 | table count first last raw
Then drilldown to the source to figure.
7 - If your cannot disable an input but don't need all the events, you can setup a NULLQUEUE filtering of the events.
This has to be setup on the indexers (or heavy forwarders)
(with windows eventlogs, we usually use filtering on the eventcode)
see exampleshttp://docs.splunk.com/Documentation/Splunk/4.3/Deploy/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest:
- Discard specific events and keep the rest
- or Keep specific events and discard the rest
Comment Box is loading comments...