< Back |Home| Next >
Splunk forwarder troubleshoot - splunk forwarder not sending data?
Splunk forwarder basically acts as agent for log collection from remote machines .Splunk forwarder collects logs from remote machines and forward s them to indexer (Splunk database) for further processing and storage. Unlike other traditional monitoring tool agents splunk forwarder consumes very less cpu -1-2% only. Splunk universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk Enterprise for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data with minimal impact on performance. But when splunk forwarder stops sending data then splunk indexer will not able to store/index them.Troubleshooting forwarder issues can be tricky specially in large cluster environments.Below we have listed few most common checks which will help you in identifying the problem and resolving it efficiently.We recommend understanding splunk forwarder working and configuration here .
Splunk forwarder Tshoot step by step :
Initially we will start by simple basic checks
1. check if splunk process is running on splunk forwarder
For windows check in services | for linux use below command
ps -ef |grep splunkd
Or
cd $SPLUNK HOME/bin
./splunk status
2. check if splunk forwarder forwarding port is open by using below command
netstat -an | grep 9997
If output of above command is blank then you port is not open.You need to open it.
3.Check on indexer if receiving is enabled on port 997 and port 997 is open on indexer
check if receiving is configured :On indexer go to setting>>forwarding and receiving >> check if receiving is enabled on port 9997.If not enable it
4. check if you are able to ping indexer from forwarder host
ping indexer name
If you are not able to ping to the server ,then check network issue
5.Confirm on indexer if your file is already indexed or not by using below search query
In the Splunk UI, run the following search - index=_internal "FileInputTracker" *<path_to_file>*
as output of the search query you will get list of log files indexed.
6.Check if forwarder has completed processing log file i.e. tailing process by using below url
https://splunk forwarder server name:8089/services/admin/inputstatus/TailingProcessor:FileStatus
in tailing process output you can check if forwarder is having any issue for processing file
7. Check out log file permissions which you are sending to splunk.verify if splunk user has access to log file
8.Checkout filestam for last modification and verify if forwarder is monitoring it
9.Verify inputs.conf and outputs.conf gor proper configuration
Below are sample configuration files for comparison:
Inputs.conf example:
----------------------------------------------------------
[monitor:///var/log/secure]
disabled = false
sourcetype = linux_secure
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
-------------------------------------------------------------
Outputs.conf example:
-----------------------------------------------------------------------
outputs.conf example:
[tcpout:express_cityA]
server = cityAserver01.domain.local:9997,cityAserver02.domain.local:9997
autoLB = true
[tcpout]
defaultGroup = express_cityB
[tcpout:express_cityB]
server = cityBserver01.domain.local:9997,cityBserver02.domain.local:9997
autoLB = true
--------------------------------------------------------------------------
10. Checkout disk space availability on indexer
11. check splunkd.log on forwarder at location $SPLUNK_HOME/var/log/splunk for any errors.like for messages that are from 'TcpOutputProc', they should give you an indication as to what is occurring when the forwarder tries to connect to the indexer
12. tcpdump port 997 data for any errors
tcpdump -i etho port 997
13. check out ulimit if you have installed forwarder on linux. and set it to unlimites or max
ulimit is limit set by default in linux is limit for number files opened by a process
check ulimit command:
ulimit -n
set ulimit command:
ulimit -n expected size
14. Finallytry restarting splunk on forwarder :)
Splunk forwarder Tshoot step by step :
Initially we will start by simple basic checks
1. check if splunk process is running on splunk forwarder
For windows check in services | for linux use below command
ps -ef |grep splunkd
Or
cd $SPLUNK HOME/bin
./splunk status
2. check if splunk forwarder forwarding port is open by using below command
netstat -an | grep 9997
If output of above command is blank then you port is not open.You need to open it.
3.Check on indexer if receiving is enabled on port 997 and port 997 is open on indexer
check if receiving is configured :On indexer go to setting>>forwarding and receiving >> check if receiving is enabled on port 9997.If not enable it
4. check if you are able to ping indexer from forwarder host
ping indexer name
If you are not able to ping to the server ,then check network issue
5.Confirm on indexer if your file is already indexed or not by using below search query
In the Splunk UI, run the following search - index=_internal "FileInputTracker" *<path_to_file>*
as output of the search query you will get list of log files indexed.
6.Check if forwarder has completed processing log file i.e. tailing process by using below url
https://splunk forwarder server name:8089/services/admin/inputstatus/TailingProcessor:FileStatus
in tailing process output you can check if forwarder is having any issue for processing file
7. Check out log file permissions which you are sending to splunk.verify if splunk user has access to log file
8.Checkout filestam for last modification and verify if forwarder is monitoring it
9.Verify inputs.conf and outputs.conf gor proper configuration
Below are sample configuration files for comparison:
Inputs.conf example:
----------------------------------------------------------
[monitor:///var/log/secure]
disabled = false
sourcetype = linux_secure
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
-------------------------------------------------------------
Outputs.conf example:
-----------------------------------------------------------------------
outputs.conf example:
[tcpout:express_cityA]
server = cityAserver01.domain.local:9997,cityAserver02.domain.local:9997
autoLB = true
[tcpout]
defaultGroup = express_cityB
[tcpout:express_cityB]
server = cityBserver01.domain.local:9997,cityBserver02.domain.local:9997
autoLB = true
--------------------------------------------------------------------------
10. Checkout disk space availability on indexer
11. check splunkd.log on forwarder at location $SPLUNK_HOME/var/log/splunk for any errors.like for messages that are from 'TcpOutputProc', they should give you an indication as to what is occurring when the forwarder tries to connect to the indexer
12. tcpdump port 997 data for any errors
tcpdump -i etho port 997
13. check out ulimit if you have installed forwarder on linux. and set it to unlimites or max
ulimit is limit set by default in linux is limit for number files opened by a process
check ulimit command:
ulimit -n
set ulimit command:
ulimit -n expected size
14. Finallytry restarting splunk on forwarder :)
Comment Box is loading comments...