Understanding splunk Forwader | How to configure splunk forwarder?
Splunk forwarder is one of the components of splunk infrastructure. Splunk forwarder basically acts as agent for log collection from remote machines .Splunk forwarder collects logs from remote machines and forward s them to indexer (Splunk database) for further processing and storage. Unlike other traditional monitoring tool agents splunk forwarder consumes very less cpu -1-2% only. Splunk universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk Enterprise for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data with minimal impact on performance.
There are two types of splunk forwarder as below
a) universal forwarder(UF) -Splunk agent installed on non-Splunk system to gather data locally, can’t parse or index data
b) Heavy weight forwarder(HWF) - full instance of splunk with advance functionality as below
- Generally works as a remote collector, intermediate forwarder, and possible data filter because they parse data, they are not recommended for production systems
Forwarder functionalities:
> Tagging of metadata (source, sourcetype and host)
>Configurable throttling and buffering
>Data compression
>SSL security
>Transport over any available network ports
>Local scripted inputs
> Centralized management
How to install splunk forwarder ?
Linux installation steps click here
Windows installation steps click here
How to configure splunk forwarder to collect and forward logs?
We can add data to forwarder by directly clicking on settings>>add data and providing location of log file on local or remote server.But whatif you vae to monitor hundreds of server logs then its not practical each time to use GUI.In this case we can use splunk configuration files to collect logs froom multiple servers and locations.The configuration file for collecting/monitoring logs on local/Remote machine is inputs.conf.Again there are multiple inputs.conf files available on splunk server.We will be editing inputs.conf in $splunk_home/etc/system/local directory.Below are steps to add logs to forwarder in linux
Splunk forwarder configuration step by step
1.Login splunk forwarder server
2. Create and populate the app directory
First create a folder for your “app”. An app is a directory of scripts and configuration files. By creating your own app directory you can control the behavior of its contents
mkdir /Applications/splunkforwarder/etc/app/yourappname /
Inside your app folder create two more folders called bin and local:
mkdir /Applications/splunkforwarder/etc/app/yourappname/bin
mkdir /Applications/splunkforwarder/etc/app/yourappname/local
The bin folder is a Splunk security requirement. Any executable, such as a script, must reside in this folder.The local folder will contain two plain text configuration (.conf) files:
inputs.conf outputs.conf
Put simply, inputs.conf is the configuration file that controls executing the script and getting its data into the Splunk Forwarder. And outputs.conf is the configuration file that controls sending the data out to the indexing server or “Splunk Receiver”. These files can be very simple or very complex depending on the needs
3.Edit inputs.conf at $splunk_home/etc/app/yourappname to monitor logs like in below example
4.Add a stanza like below with sourcetype i.e. type of logs like syslog or other and index name if you wish to send data to other indexer.
In front of monitor specify remote log file location.
5.open port 514 to listen data from source machines-server generating logs
Below are few inputs.conf and outputs.conf sample configuration
Inputs.conf example:
outputs.conf
----------------------------------------------------------
[monitor:///var/log/secure.log]
disabled = false
index=gis_network
sourcetype = router
[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
-------------------------------------------------------------
6.After collecting logs from server we have to forward logs to indexers.Splunk forwarder uses port number 9997 to forward collected logs to indexer.We can configure these setting in outputs.conf file.
outputs.conf example:
----------------------------------------------------------
## outputs.conf [tcpout] #mention type of traffic like tcp/udp disabled=false defaultGroup=indexCluster #name of index sever 6 to which we want to forward data
---------------------------------------------------------------------------
7.Enable receiving on the indexer on port port 9997.On indexer go to setting>>forwarding and receiving >> enable receiving
8.verify on the splunk if your data is indexed by searching for logs or hostname through splunk search Gui.
How to get list of all forwarders installed in your environment?
By using below search query you can directly list out available forwarder in your environment:
index=_internal source=*metrics.log group=tcpin_connections
| eval sourceHost=if(isnull(hostname), sourceHost,hostname)
| rename connectionType as Type
| eval (fwd="uf","Universal Forwarder", fwd="lwf", "lf",fwd="full", "Heavy Forwarder", connect="cooked" or connect="cookedSSL","Splunk Forwarder", connect="raw" or connect="rawSSL","Legacy")
| rename version AS "Version", sourceIp AS "Source IP", sourceHost AS "Host", destPort AS "Port"
| fields Type, "Source IP", Host, Port, kb, tcp_eps, tcp_Kprocessed, tcp_KBps, splunk_server, Version
| eval Hour=relative_time(_time,"@h")
| stats avg(tcp_KBps), sum(tcp_eps), sum(tcp_Kprocessed), sum(kb), BY Hour, Type, "Source IP", Host, Port, Version
| fieldformat Hour=strftime(Hour,"%x %Hh")
Sample Output:
