"This website is not affiliated with Splunk, Inc. and is not an authorized seller of Splunk products or services."
  • Home - Splunk Tutorial
  • Splunk training videos
  • Splunk interview questions
  • Contact US
  • About Us
  • Privacy Policy
  • Splunk Jobs

                                                                                                      <  Back |Home| Next  >

Splunk forwarder troubleshoot - splunk forwarder not sending data?

Picture
     Splunk forwarder basically acts as agent for log collection from remote machines .Splunk forwarder collects logs from remote machines and forward s them to indexer (Splunk database) for further processing and storage. Unlike other traditional monitoring tool agents splunk forwarder consumes very less cpu -1-2% only. Splunk universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk Enterprise for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data with minimal impact on performance. But when splunk forwarder stops sending data then splunk indexer will not able to store/index them.Troubleshooting forwarder issues can be tricky specially in large cluster environments.Below we have listed few most common checks which will help you  in identifying the problem and resolving it efficiently.We recommend understanding  splunk forwarder working  and configuration  here .

Splunk forwarder Tshoot step by step  :


Initially we will start by simple basic checks

1. check if splunk process is running on splunk forwarder

For windows check in services | for linux use below command

ps -ef |grep splunkd
 Or
cd $SPLUNK HOME/bin
./splunk status


2. check if splunk forwarder forwarding port is open by using below command

netstat -an | grep 9997

If output of above command is blank then you port is not open.You need to open it.

3.Check on indexer if receiving is enabled on port 997 and port 997 is open on indexer

check if receiving is configured :
On indexer go to setting>>forwarding and receiving >> check if receiving is enabled on port 9997.If not enable it

4. check if you are able to ping indexer  from forwarder host

ping indexer name

If you are not able to ping to the server ,then check network issue

5.Confirm on indexer if your file is already indexed or not by using below search query

In the Splunk UI, run the following search - index=_internal "FileInputTracker" *<path_to_file>*

as output of the search query you will get list of log files indexed.

6.Check if forwarder has completed processing log file i.e. tailing process by using below url

https://splunk forwarder server name:8089/services/admin/inputstatus/TailingProcessor:FileStatus

in tailing process output you can check if forwarder is having any issue for processing file

7. Check out log file permissions which you are sending to splunk.verify if splunk user has access to log file

8.Checkout filestam for last modification and verify if forwarder is monitoring it


9.Verify inputs.conf and outputs.conf gor proper configuration

Below are sample configuration files for comparison:

Inputs.conf example:

----------------------------------------------------------
[monitor:///var/log/secure]
 disabled = false
 sourcetype = linux_secure
 
 [monitor:///var/log/messages]
 disabled = false
 sourcetype = syslog
-------------------------------------------------------------

Outputs.conf example:

-----------------------------------------------------------------------
outputs.conf example:
[tcpout:express_cityA]
server = cityAserver01.domain.local:9997,cityAserver02.domain.local:9997
autoLB = true
[tcpout]
defaultGroup = express_cityB
[tcpout:express_cityB]
server = cityBserver01.domain.local:9997,cityBserver02.domain.local:9997
autoLB = true

--------------------------------------------------------------------------


10. Checkout disk space availability on indexer

11. check splunkd.log on forwarder at location
$SPLUNK_HOME/var/log/splunk for any errors.like
for messages that are from 'TcpOutputProc', they should give you an indication as to what is occurring when the forwarder tries to connect to the indexer

12. tcpdump port 997 data for any errors

tcpdump -i etho port 997

13. check out ulimit if you have installed forwarder on linux. and set it to unlimites or max

ulimit is limit set by default in linux is limit for number files opened by a process

check ulimit command:
ulimit -n
set ulimit command:
ulimit -n expected size


14. Finallytry restarting splunk on forwarder :)





Comment Box is loading comments...
Powered by Create your own unique website with customizable templates.