What is a Splunk Index? How to Create a New Splunk Index via the Web UI and CLI?

In this post, we are diving into one of the most crucial components of Splunk: the 'Indexer', where the magic of event indexing happens. If you recall the Data Input, Data Store, and Data Search phases of the data pipeline, or the Event Processing Tiers of Splunk, we will be focusing on the Data Storage phase, where parsing and indexing of input data take place.

Throughout this article, we will answer fundamental questions about the indexing process, such as:

By the end of this article, you will have a solid grasp of Splunk indexes and how to manage them effectively to optimize your Splunk environment. So, let's get started on this exciting journey of exploring Splunk indexes!

What is an Index?

In Splunk, an index is a repository for data that is going to be stored in an indexer. The Splunk instance is configured to index local and remote data, which can then be searched through a search app. Indexes live under the var/lib/splunk directory by default.

Splunk comes with multiple preconfigured indexes. The "main" index is the first index you should know about, because it is the default index Splunk stores the incoming data. However, you have the flexibility to create and specify other indexes for different data inputs, allowing you to organize your data more effectively. Scroll down if you want to know how to create a custom index. We have covered both Web UI and CLI procedures.

There are a few more indexes comes as preconfigured along with the index "main". Here you see:

You might wonder why we would want to create many indexes instead of storing all data in a single index. The answer lies in the following advantages:

How Does Splunk Store Indexes?

Now that you know what indexes are, let's dive into how Splunk actually stores these indexes. When data is fed into Splunk, the indexer processes it and stores it in a series of directories and files, collectively known as an Indexes.

Under the hood, each index is comprised of a set of directories called "buckets." These buckets are organized by age, with the most recently indexed data stored in the "hot" bucket, slightly older data in the "warm" bucket, and the oldest data in the "cold" bucket. This age-based organization helps Splunk efficiently manage the data lifecycle and optimize performance.

Here's a quick breakdown of the bucket types:

Hot Buckets: When new data arrives, it is first written to hot buckets. These buckets are always open for writing and are optimized for fast data ingestion. Hot buckets have the naming format hot_v1_<GUID>, where <GUID> is a unique identifier. Well, hot buckets are stored in the $SPLUNK_HOME/var/lib/splunk/[index_name]/db/ directory.

Warm Buckets: As hot buckets reach certain thresholds (e.g., maximum size, age, or number of buckets), data is rolled over to warm buckets. Warm buckets are read-only and are optimized for searching. The naming format for warm buckets is db_<timestamp>_<GUID>, where <timestamp> represents the earliest and latest timestamps of the events within the bucket. Worm buckets also share the same location as hot buckets in the directory hierarchy.

Cold Buckets: When warm buckets reach their respective thresholds, data is moved to cold buckets. Cold buckets are also read-only and are typically stored on slower, less expensive storage media. The naming format for cold buckets is the same as warm buckets, but they are stored in a separate directory called colddb. Path: $SPLUNK_HOME/var/lib/splunk/[index_name]/colddb

Thawed Buckets: When data needs to be restored from an archive, it is moved back into thawed buckets.

Within each bucket, Splunk stores the indexed data in a compressed format, along with associated metadata files that help facilitate fast searching and retrieval. These files include:

By default, Splunk will automatically manage the movement of data through the bucket lifecycle based on configurable policies. However, you can also manually manage buckets in the indexes.conf file.

Let's see the key factors, criteria and thresholds that influence data movement:

The specific thresholds and criteria for data movement are defined using the following settings in indexes.conf:

How to Create and Delete Indexes Using the Web UI?

Now that you have a solid understanding of what indexes are and how Splunk stores them, let's explore how you can create and delete indexes using the Splunk Web UI. The Web UI provides a user-friendly interface for managing indexes, making it easy for you to organize your data and keep your Splunk environment tidy.

Creating an Index Using the Web UI

4. (Optional) Specify the "Home Path", "Cold Path", and "Thawed Path" if you want to store the index data in custom locations.

5. Click on the "Save" button to create the new index.

Your new index is now created and ready to receive data. You can start forwarding data to this index by configuring data inputs or forwarders.

Note: If in case you don't see your index in the list, got to https://splunkhost:8000/en-US/debug/refreshreload the configuration.

Deleting an Index Using the Web UI

Please note that deleting an index will permanently remove all data associated with that index. Make sure you have backed up any important data before proceeding with the deletion.

How to Create and Delete Indexes Using CLI in Linux and Mac?

While the Splunk Web UI provides a convenient way to manage indexes, sometimes you may need to work with indexes from the command line. Whether you're automating index management tasks or simply prefer the flexibility of the CLI, Splunk has got you covered. In this section, we'll walk through the steps to create and delete indexes using the Splunk CLI on Linux and Mac operating systems.

Creating an Index

1. Connect to your Splunk instance using SSH or a terminal.

2. Navigate to the $SPLUNK_HOME/etc/apps/<app_name>/local/ directory where the indexes.conf file exist. The default, $SPLUNK_HOME in Linux is /opt/splunk, and in Mac is /Applications/splunk.

3. Navigate to the application underneath you need to create an index. Applications are stores underneath $SPLUNK_HOME/etc/apps/. We are going to create an index inside the "search" app.

cd /Applications/splunk/etc/apps/search/local/

If in case you want to create an index underneath a new application. You should create a new directory SPLUNK_HOME/etc/apps/ and inside the directory again create local directory.

Note: Read How to Create Apps and Add-Ons in Splunk in this article.

The content of indexes.conf something looks like in the above picture. Each stanza represents an index. We have two indexes macbookpro and security_logo in the indexes.conf file. Let's create another one.

[audit_logs]   
coldPath = $SPLUNK_DB/audit_logs/colddb
homePath = $SPLUNK_DB/audit_logs/db
maxTotalDataSizeMB = 51200
thawedPath = $SPLUNK_DB/audit_logs/thaweddb

We created another index named "audit_logs" underneath "search" application. Your new index is now created and ready to receive data. You can start forwarding data to this index by configuring data inputs or forwarders.

Deleting an Index from CLI

The index and its associated data are now permanently deleted from your Splunk instance.

How to Add and Remove Data from Indexes?

Splunk provides various methods to add and remove data from indexes. In this section, we will explore how to add data to an index using the Web UI and remove data from an index using the CLI.

Adding Data to an Index Using the Web UI

Splunk will now process and index the uploaded data, making it available for searching and analysis.

Removing Data from an Index Using the CLI

1. Connect to your Splunk instance using SSH or a terminal.

2. To remove data from an index, you can use the splunk remove command followed by the index name and the specific data you want to remove.

3. For example, to remove data from the "my_index" index based on a search query: This command will remove all data from the "my_index" index where the source matches "/path/to/file.log".

./splunk remove index="my_index" source="/path/to/file.log"

This command will remove all data from the "my_index" index where the source matches "/path/to/file.log".

./splunk remove index="my_index" earliest=-30d@d latest=now

This command will remove data from the "my_index" index that is older than 30 days.

1. After issuing the splunk remove command, Splunk will display a summary of the data to be removed and prompt for confirmation.

2. Review the summary and, if satisfied, type "yes" to proceed with the data removal.

Please note that removing data from an index is a permanent action and cannot be undone. Make sure to carefully review the data being removed before confirming the operation.

Removing Data from an Index using the Web UI

In this demo, we have logs from two hosts: Linux & Mac. Let's delete the events of Linux by filtering the Linux logs.

This method is suitable for removing smaller subsets of data based on specific search criteria.

It's important to exercise caution when removing data from indexes, as it can impact search results and any associated reports or dashboards. Always ensure that you have appropriate backups and consider the implications of data removal before proceeding.

What If You Can't Delete?

By default any account including admin doesn't have "can_delete" permission. You should add "can_delete" Role to the user to delete events from the search app.

Go to Settings -> Users -> Edit -> Assign Role -> can_delete

How to Take the Backup of Indexes?

Taking regular backups of your Splunk indexes is crucial to ensure data protection and recoverability in case of system failures, data corruption, or accidental deletions. In this section, we will walk you through the step-by-step process of taking backups of your Splunk indexes.

Step 1: Identify the Indexes to Backup

Step 2: Stop the Splunk Instance

1. Connect to your Splunk instance using SSH or a terminal.

2. Navigate to the $SPLUNK_HOME/bin directory:

3. Stop the Splunk instance: Stopping the Splunk instance ensures that no data is being actively written to the indexes during the backup process.

./splunk stop

Step 3: Back Up the Index Directories

1. Use the cp command to copy the index directories to a backup location: Replace index_name with the actual name of the index you want to back up and /backup/location/ with the path where you want to store the backup.

2. Repeat the above step for each index you want to back up.

cp -R ../var/lib/splunk/index_name /backup/location/

Step 4: Start the Splunk Instance

./splunk start

Step 5: Verify the Backup

We recommend to automate the backup process by scheduling using cron or at schedulers.

Wrap Up

We hope this article helps understanding the fundamentals of Splunk indexes, including what they are, how they store data, and how to manage them effectively. We covered creating and deleting indexes using both the Web UI and CLI, adding and removing data from indexes, and taking backups to ensure data protection. By understanding and implementing these concepts, you can optimize your Splunk deployment, ensure data accessibility, and maintain a robust data management strategy. Happy Splunking!

We are going to end this article for now, we will cover more information about the Splunk in the up coming articles. Please keep visiting thesecmaster.com for more such technical information. Visit our social media page on Facebook, Instagram,  LinkedIn, Twitter, Telegram, Tumblr, & Medium and subscribe to receive information like this. 

You may also like these articles: