< Back |Home| Next >
Understanding props.conf in splunk
props.conf in splunk
Splunk configuration files are the main brains behind splunk working. SPlunk .conf files controls behaviour of splunk.These files are available on splunk server and easily redable and editable if you have appropriate access.Whatever changes we make through GUI seatss in .conf files.Most of the time GUI does not offer full functionalities in that case we can achieve them through .conf files. Below we will go through and understand props.conf file.
Props.conf if one of the most important .conf file used by splunk while indexing the data and later.Splunk uses configuration in props.conf while indexing logs to indexer and for later processing.Below is the location where we can find props.conf.It's not necessary to add/configure all parameters in props.conf.Adding only one or more parameter explained below
Location of props.conf:
/opt/splunk/etc/system/default/props.conf --->never edit this file as its conatians default configuration
/opt/splunk/etc/system/local/props.conf ----->We can edit this file for configurations
We can create different folders for specific apps in $splunk default location/bin/apps and create props.conf inside default directory for that app.Splunk will automatically pick up configuration from that file and process them as per precedence if multiple pros.conf are available.
How to configure props.conf in splunk?
There happens to be seven attributes you want to set in props.conf every time you bring data into Splunk as below
· TIME_PREFIX. - This is the first attribute is used to tell Splunk where to start to look for the timestamp in your event.
· MAX_TIMESTAMP_LOOKAHEAD - Setting this attribute makes Splunk happy and run more efficiently because it will not have to spend any extra time and resources to find the time stamp. You can tell Splunk that your timestamp is 20 characters into your event so Splunk will not waste any time looking through the entire event.
· TIME_FORMAT - Many people "sleep" on this attribute and shouldn't. It is very important to help Splunk interpret your data. With this attribute you are telling Splunk the format your time stamp is in using strptime Splunk will not have to try to figure out if 10/2/12 is October 2, 2012, February 10, 2012, or even something weird like December 2, 2010.
· SHOULD_LINEMERGE - this attribute is the leader and decision maker of the group. Depending on the value of this attribute, other attributes are required. This setting should be set to "false" and used along with the LINE_BREAKER attribute, which can greatly increase processing speed.
· LINE_BREAKER -This attribute looks the toughest and most intimidating. It identifies how your events should be broken apart. This is important because if this is not set correctly you can have data that is spread across multiple events. Using regular expressions, Splunk will look for that specific pattern and break up your events accordingly, and as mentioned before, this should be used in conjunction with SHOULD_LINEMERGE = false.
· TRUNCATE.- This attribute is nothing to sneeze at. Measured in bytes, this attribute limits the length of an event (line of data), and will break when the limit is met. The default value is 10000 so set this to 999999 bytes or more depending on the size of your event. Because your data is not being broken up into multiline events (given SHOULD_LINEMERGE = false) you will want to ensure your events do not get broken up incorrectly.
· TZ - the time zone attribute is probably the most forgotten of the group. You will want to set the time zone for each host in your Splunk environment. This will ensure the time is displayed correctly on your search head.
Props.conf is used to define following configurations in splunk:
Format of props.conf file:
- Below is the sample format of props.conf generally used
[sourcetype name]
timestamp recognition configuration ----------------------->( this tells splunkwhere to look for date and time in logs)
linebreaking configuration -------------------------> (this tells splunk where log ends for perticular event)
binary check configuration
Fieldalias configuration --------------------------->Rrnaming filedname for autodetected fileds
regex1 for filed extarction -------------------------------->Regex for manual filed extarction
regex2 for filed extarction
Lookup table name reference --------------------------------->Lookup table name to check lookup table entries
[sourcetype name2]
timestamp recognition configuration
linebreaking configuration
binary check configuration
Fieldalias configuration
regex1 for filed extarction
regex2 for filed extarction
Lookup table name reference
Props.conf examples
-Bleow are few props.conf real time examples
example 1.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[mactime]
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %a %b %d %Y %H:%M:%S
MAX_DAYS_AGO = 36500
MAX_DAYS_HENCE = 36500
TRANSFORMS-mactime = mactime
# the following are imported from the csv stanza of the default props.conf
SHOULD_LINEMERGE = False
pulldown_type = true
CHECK_FOR_HEADER = true
LOOKUP-rhds_status = rhds_status status AS err OUTPUTNEW status_description AS status_description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
example 2.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[udp:54321]
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_PREFIX={"_timestamp":
LOOKUP-dmca_lookup = dmca_lookup src_ip AS ip OUTPUTNEW nat_ip AS src_translated_ip
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
example 3.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[perforce]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n]+)(Perforce server info:|Perforce server error:)
EXTRACT-user = pid (?<pid>\d+) (?<user>\w+@[^\s]+) (?<IP_addr>(?:[0-9]{1,3}\.){3}[0-9]{1,3})
EXTRACT-command = \[(?<client_version>[^\[\]]+)\] '(?<command>[^']+)'
EXTRACT-lapse = lapse (?<lapse>\d*\.?\d+)s
EXTRACT-usage = --- usage (?<usage_user>\d+)\+(?<usage_system>\d+)us (?<io_read>\d+)\+(?<io_write>\d+)io (?<IPC_received>\d+)\+(?<IPC_sent>\d+)net (?<memory>\d+)k (?<page_faults>\d+)
EXTRACT-rpc = --- rpc msgs\/size in\+out (?<RPC_received>\d+)\+(?<RPC_sent>\d+)\/(?<RPC_received_size>\d+)mb\+(?<RPC_sent_size>\d+)mb himarks (?<RPC_send_window>\d+)\/(?<RPC_rcv_window>\d+) snd\/rcv (?<RPC_send_time>\d*\.?\d+)s\/(?<RPC_rcv_time>\d*\.?\d+)s
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We will create our own props.conf for our own app in create your own app section.
For more information about each of these props.conf attributes, take a look at the following link: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
Please comment in case of any doubt :)
Props.conf if one of the most important .conf file used by splunk while indexing the data and later.Splunk uses configuration in props.conf while indexing logs to indexer and for later processing.Below is the location where we can find props.conf.It's not necessary to add/configure all parameters in props.conf.Adding only one or more parameter explained below
Location of props.conf:
/opt/splunk/etc/system/default/props.conf --->never edit this file as its conatians default configuration
/opt/splunk/etc/system/local/props.conf ----->We can edit this file for configurations
We can create different folders for specific apps in $splunk default location/bin/apps and create props.conf inside default directory for that app.Splunk will automatically pick up configuration from that file and process them as per precedence if multiple pros.conf are available.
How to configure props.conf in splunk?
There happens to be seven attributes you want to set in props.conf every time you bring data into Splunk as below
· TIME_PREFIX. - This is the first attribute is used to tell Splunk where to start to look for the timestamp in your event.
· MAX_TIMESTAMP_LOOKAHEAD - Setting this attribute makes Splunk happy and run more efficiently because it will not have to spend any extra time and resources to find the time stamp. You can tell Splunk that your timestamp is 20 characters into your event so Splunk will not waste any time looking through the entire event.
· TIME_FORMAT - Many people "sleep" on this attribute and shouldn't. It is very important to help Splunk interpret your data. With this attribute you are telling Splunk the format your time stamp is in using strptime Splunk will not have to try to figure out if 10/2/12 is October 2, 2012, February 10, 2012, or even something weird like December 2, 2010.
· SHOULD_LINEMERGE - this attribute is the leader and decision maker of the group. Depending on the value of this attribute, other attributes are required. This setting should be set to "false" and used along with the LINE_BREAKER attribute, which can greatly increase processing speed.
· LINE_BREAKER -This attribute looks the toughest and most intimidating. It identifies how your events should be broken apart. This is important because if this is not set correctly you can have data that is spread across multiple events. Using regular expressions, Splunk will look for that specific pattern and break up your events accordingly, and as mentioned before, this should be used in conjunction with SHOULD_LINEMERGE = false.
· TRUNCATE.- This attribute is nothing to sneeze at. Measured in bytes, this attribute limits the length of an event (line of data), and will break when the limit is met. The default value is 10000 so set this to 999999 bytes or more depending on the size of your event. Because your data is not being broken up into multiline events (given SHOULD_LINEMERGE = false) you will want to ensure your events do not get broken up incorrectly.
· TZ - the time zone attribute is probably the most forgotten of the group. You will want to set the time zone for each host in your Splunk environment. This will ensure the time is displayed correctly on your search head.
Props.conf is used to define following configurations in splunk:
- Configuring timestamp recognition
- Convertig timeformat to our default timeformat
- Configuring linebreaking for multiline events.
- Setting up character set encoding
- Defining manual filed extarction regex
- Allowing processing of binary files.
- Configuring event segmentation.
- Overriding Splunk's automated host and source type matching
- Defining where to lookup for lookup table etc
Format of props.conf file:
- Below is the sample format of props.conf generally used
[sourcetype name]
timestamp recognition configuration ----------------------->( this tells splunkwhere to look for date and time in logs)
linebreaking configuration -------------------------> (this tells splunk where log ends for perticular event)
binary check configuration
Fieldalias configuration --------------------------->Rrnaming filedname for autodetected fileds
regex1 for filed extarction -------------------------------->Regex for manual filed extarction
regex2 for filed extarction
Lookup table name reference --------------------------------->Lookup table name to check lookup table entries
[sourcetype name2]
timestamp recognition configuration
linebreaking configuration
binary check configuration
Fieldalias configuration
regex1 for filed extarction
regex2 for filed extarction
Lookup table name reference
Props.conf examples
-Bleow are few props.conf real time examples
example 1.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[mactime]
MAX_TIMESTAMP_LOOKAHEAD = 25
TIME_FORMAT = %a %b %d %Y %H:%M:%S
MAX_DAYS_AGO = 36500
MAX_DAYS_HENCE = 36500
TRANSFORMS-mactime = mactime
# the following are imported from the csv stanza of the default props.conf
SHOULD_LINEMERGE = False
pulldown_type = true
CHECK_FOR_HEADER = true
LOOKUP-rhds_status = rhds_status status AS err OUTPUTNEW status_description AS status_description
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
example 2.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[udp:54321]
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_PREFIX={"_timestamp":
LOOKUP-dmca_lookup = dmca_lookup src_ip AS ip OUTPUTNEW nat_ip AS src_translated_ip
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
example 3.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[perforce]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n]+)(Perforce server info:|Perforce server error:)
EXTRACT-user = pid (?<pid>\d+) (?<user>\w+@[^\s]+) (?<IP_addr>(?:[0-9]{1,3}\.){3}[0-9]{1,3})
EXTRACT-command = \[(?<client_version>[^\[\]]+)\] '(?<command>[^']+)'
EXTRACT-lapse = lapse (?<lapse>\d*\.?\d+)s
EXTRACT-usage = --- usage (?<usage_user>\d+)\+(?<usage_system>\d+)us (?<io_read>\d+)\+(?<io_write>\d+)io (?<IPC_received>\d+)\+(?<IPC_sent>\d+)net (?<memory>\d+)k (?<page_faults>\d+)
EXTRACT-rpc = --- rpc msgs\/size in\+out (?<RPC_received>\d+)\+(?<RPC_sent>\d+)\/(?<RPC_received_size>\d+)mb\+(?<RPC_sent_size>\d+)mb himarks (?<RPC_send_window>\d+)\/(?<RPC_rcv_window>\d+) snd\/rcv (?<RPC_send_time>\d*\.?\d+)s\/(?<RPC_rcv_time>\d*\.?\d+)s
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We will create our own props.conf for our own app in create your own app section.
For more information about each of these props.conf attributes, take a look at the following link: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
Please comment in case of any doubt :)
< Back |Home| Next >
Comment Box is loading comments...
