<  Back |Home| Next  >

Splunk regex tutorial | field extraction using regex

                     Regular expressions are extremely useful in extracting information from text such as code, log files, spreadsheets, or even documents.Regular expressions or regex is a specialized language for defining pattern matching rules .Regular expressions match patterns of characters in text. They have their own grammar and syntax rules.splunk uses regex for identifying interesting fields in logs like username,credit card number,ip address etc.By default splunk automatically extracts interesting fields and display them at left column is search result -only condition is log must contain key value pairs which means logs should contains field name and its value - like for username it should appear in log like usename=x or user:x.Extracted fields can be used later for sorting data,making specialized reports,creating valueable dashboards.But if logs do not contain field name in key value pair- like username or other fileds appears in log at random place then splunk will not detect the username automatically.In this condition regex comes for your help.You have to teach splunk to extract the field using regex.

So basically regex used to identify fields and list them in proper manner which later can be used for reporting,sorting and dashboard.Below image will help you in understanding the scenario

---

so the  next question popping in your mind is should i need to lean regex for using splunk..then answer is it depends.For using and operating splunk you do not need to learn regex in detail - basic knowledge will be ok.But if you want to become a skilled splunk admin then learning regex is necessary.


Why Regex?

Regex is helpful in transforming your horrible looking machine  logs into beautiful human understandable reports and dashboard -Easy to understand and use.Shown as below.

How to regex?

Splunk automatically identifies any fields that match its key/value pair intelligence, which can be found to the left of the search results as below. This can often allow you to start putting together useful data visualizations right out of the box.In below screenshot splunk has automatically extracted host,timestamp etc values.We can use these values for reporting,statistical analysis and creating dashboards.Splunk has inbuilt regex extractor called IFX (Interactive field extractor).By using IFX splunk autodetects useful fields and list them at left side.Splunk IFX can extract fields automatically which are in standard key value pair format i.e. key=value format like username=john etc.But if logs are not in key value pair format then you have to teach splunk to extract fields which you wan using regex.

We’re going to extract data that Splunk doesn’t recognize right away. There are a few of ways to do this, including using Splunk’s Interactive Field Extractor (IFX), or you can write your own regex (which I prefer)


How to extract fields using regex?

• Good regex sites to help with Splunk

• https://regex101.com/ - Great for general regex stuff and capture groups.
• http://www.regexe.com/ - Great for dealing with capture groups in the way that Splunk likes them for anonymising data.
• http://regexr.com/ - Classic website for quick PoC regexs.

Below are few commonly used regex notations while using extracting keywords using regex manually:

regex tutorial

Use of special notations in regex:

regex examples

Regex  usage  example

regex syntax with example

Comment Box is loading comments...