Splunk Admin Manual

426 Pages·2011·1.65 MB·English

Splunk Admin Manual
Version: 4.2.3
Generated: 10/06/2011 03:45 pm
Copyright Splunk, Inc. All Rights Reserved
Table of Contents
Welcome to Splunk administration....................................................................................................1
What's in this manual.................................................................................................................1
What is Splunk?.........................................................................................................................2
What to do first.....................................................................................................................................3
Start Splunk................................................................................................................................3
Configure Splunk to start at boot time........................................................................................5
Find Splunk Manager in Splunk Web.........................................................................................6
Install your license......................................................................................................................6
Change default values...............................................................................................................7
Bind Splunk to an IP.................................................................................................................10
Specify a proxy server..............................................................................................................11
Functional differences between *nix and Windows in Splunk operations................................11
Getting started for Windows admins...............................................................................................13
Introduction for Windows admins.............................................................................................13
Try Splunk out..........................................................................................................................14
Learn what Splunk does...........................................................................................................15
Integrate Splunk into your enterprise.......................................................................................16
Get the most out of Splunk.......................................................................................................16
Optimize Splunk for peak performance....................................................................................16
Put Splunk onto system images...............................................................................................17
Integrate a universal forwarder onto a system image..............................................................19
Integrate full Splunk onto a system image...............................................................................20
Deploying Splunk on Windows.................................................................................................21
Manage Splunk licenses...................................................................................................................25
How Splunk licensing works.....................................................................................................25
Types of Splunk licenses.........................................................................................................26
Groups, stacks, pools, and other terminology..........................................................................27
Install a license.........................................................................................................................29
Configure a license master......................................................................................................30
Configure a license slave.........................................................................................................31
Create or edit a license pool....................................................................................................31
Add an indexer to a license pool..............................................................................................34
Manage your licenses..............................................................................................................34
About license violations............................................................................................................38
Migrate to the new Splunk licenser..........................................................................................41
More about Splunk Free...........................................................................................................43
Manage licenses from the CLI.................................................................................................45
Meet Splunk Web and Splunk apps..................................................................................................48
What's Splunk Web?................................................................................................................48
What are apps and add-ons?....................................................................................................49
Where to get more apps and add-ons......................................................................................52
i
Table of Contents
Meet Splunk Web and Splunk apps
App architecture and object ownership....................................................................................53
Manage app and add-on objects..............................................................................................55
How to configure Splunk...................................................................................................................57
Splunk configuration methods..................................................................................................57
About Splunk Manager..............................................................................................................58
About configuration files...........................................................................................................60
Configuration file precedence..................................................................................................64
Attribute precedence within a single props.conf file.................................................................68
Configuration parameters and the data pipeline......................................................................70
Indexing with Splunk.........................................................................................................................73
What's a Splunk index?............................................................................................................73
How indexing works.................................................................................................................73
Index time versus search time..................................................................................................76
Advanced indexing strategy......................................................................................................77
Add and manage users.....................................................................................................................79
About users and roles..............................................................................................................79
Set up user authentication.......................................................................................................80
Set up user authentication with Splunk's built-in system..........................................................80
Set up user authentication with LDAP......................................................................................82
Set up user authentication with external systems....................................................................90
Use single sign-on (SSO) with Splunk.....................................................................................95
Delete user accounts using the CLI.........................................................................................99
User language and locale........................................................................................................99
Configure user session timeouts............................................................................................100
Add and edit roles..................................................................................................................102
Manage indexes...............................................................................................................................107
About managing indexes........................................................................................................107
Set up multiple indexes..........................................................................................................107
Set limits on disk usage.........................................................................................................113
How Splunk stores indexes....................................................................................................114
Configure segmentation to manage disk usage.....................................................................125
Configure custom segmentation for a host, source, or source type.......................................127
Move the index database.......................................................................................................128
Remove indexed data from Splunk........................................................................................130
Optimize indexes....................................................................................................................133
Define alerts.....................................................................................................................................134
How does alerting work in Splunk..........................................................................................134
Set up alerts in savedsearches.conf......................................................................................135
Configure scripted alerts........................................................................................................142
ii
Table of Contents
Define alerts
Send SNMP traps to other systems.......................................................................................144
Set up backups and retention policies..........................................................................................147
What you can back up............................................................................................................147
How much space you will need..............................................................................................147
Back up indexed data.............................................................................................................148
Back up configuration information..........................................................................................151
Set a retirement and archiving policy.....................................................................................151
Archive indexed data..............................................................................................................153
Restore archived indexed data..............................................................................................156
Configure data security...................................................................................................................161
What you can secure with Splunk..........................................................................................161
Use SSL (HTTPS) for secure access to Splunk Web............................................................163
Use SSL for secure intra-Splunk communication...................................................................165
Use SSL to encrypt and authenticate data from forwarders...................................................172
Configure archive signing.......................................................................................................174
Configure IT data block signing..............................................................................................175
Cryptographically sign audit events........................................................................................178
Audit Splunk activity...............................................................................................................180
Configure event hashing........................................................................................................182
Hardening standards..............................................................................................................185
Manage search jobs.........................................................................................................................188
About jobs and job management...........................................................................................188
Manage jobs in Splunk Web..................................................................................................189
Manage jobs in the OS...........................................................................................................190
Use Splunk's command line interface (CLI)..................................................................................192
About the CLI.........................................................................................................................192
Get help with the CLI..............................................................................................................193
Use the CLI to administer a remote Splunk server.................................................................195
CLI admin commands............................................................................................................197
Configuration file reference............................................................................................................199
admon.conf.............................................................................................................................199
alert_actions.conf....................................................................................................................200
app.conf..................................................................................................................................205
audit.conf.................................................................................................................................209
authentication.conf..................................................................................................................212
authorize.conf..........................................................................................................................219
commands.conf.......................................................................................................................223
crawl.conf................................................................................................................................226
default.meta.conf.....................................................................................................................228
iii
Table of Contents
Configuration file reference
deploymentclient.conf.............................................................................................................229
distsearch.conf........................................................................................................................232
eventdiscoverer.conf...............................................................................................................236
event_renderers.conf..............................................................................................................237
eventtypes.conf.......................................................................................................................239
fields.conf................................................................................................................................240
indexes.conf............................................................................................................................242
inputs.conf...............................................................................................................................251
limits.conf................................................................................................................................266
literals.conf..............................................................................................................................281
macros.conf.............................................................................................................................282
multikv.conf.............................................................................................................................284
outputs.conf.............................................................................................................................287
pdf_server.conf.......................................................................................................................300
perfmon.conf...........................................................................................................................304
procmon-filters.conf.................................................................................................................306
props.conf...............................................................................................................................307
pubsub.conf.............................................................................................................................323
regmon-filters.conf..................................................................................................................324
report_server.conf...................................................................................................................326
restmap.conf...........................................................................................................................330
savedsearches.conf................................................................................................................334
searchbnf.conf.........................................................................................................................341
segmenters.conf......................................................................................................................344
server.conf..............................................................................................................................346
serverclass.conf......................................................................................................................354
serverclass.seed.xml.conf.......................................................................................................358
setup.xml.conf.........................................................................................................................360
source-classifier.conf..............................................................................................................363
sourcetypes.conf.....................................................................................................................364
sysmon.conf............................................................................................................................366
tags.conf..................................................................................................................................367
tenants.conf.............................................................................................................................369
times.conf................................................................................................................................370
transactiontypes.conf..............................................................................................................373
transforms.conf.......................................................................................................................375
user-seed.conf........................................................................................................................383
web.conf..................................................................................................................................384
wmi.conf..................................................................................................................................392
workflow_actions.conf.............................................................................................................396
viewstates.conf........................................................................................................................400
iv
Table of Contents
Troubleshooting...............................................................................................................................403
What Splunk logs about itself.................................................................................................403
Work with metrics.log.............................................................................................................406
Contact Support.....................................................................................................................409
Anonymize data samples to send to support.........................................................................413
Not finding the events you're looking for?..............................................................................415
SuSE Linux: unable to get a properly formatted response from the server............................416
Command line tools for use with Support's direction.............................................................417
Troubleshoot configurations with btoo.l..................................................................................419
v
Welcome to Splunk administration
What's in this manual
What's in this manual
This manual contains information and procedures for the Splunk administrator. If you're responsible
for configuring, running, and maintaining Splunk as a service for yourself or other users, this manual
is for you.
For example, learn how to:
• add users and set up roles
• configure data security
• back up index and user data
• manage the search jobs your users run
and much more.
Where is all the information about forwarding, distributed search, and other deployment strategies?
Starting with the 4.2 release, we've moved that information into a separate manual, Distributed
Deployment. There's a lot of new stuff there, as well. Check it out and let us know what you think!
How about the data inputs and event processing stuff?
Yep, we've moved that, too -- to a second new manual, Getting Data In. There you'll find all the great
information previously in the Admin manual, along with a lot of new topics. That manual covers the
entire process of getting data into Splunk -- from setting up your data inputs to configuring event
processing.
And the information about event types and source types etc?
We moved that information into a new manual a while back, actually, with the 4.1 release. For
information about Splunk knowledge, new and old, look in the Knowledge Manager Manual.
Looking for help with searching in Splunk?
Check out the User Manual and the Search Reference Manual for all things search. In particular,
you might want to check out the Search Cheatsheet if you're looking for a quick list of common
examples.
Make a PDF
If you'd like a PDF of any version of this manual, click the pdf version link above the table of
contents bar on the left side of this page. A PDF version of the manual is generated on the fly for you,
and you can save it or print it out to read later.
1
What is Splunk?
What is Splunk?
Splunk is an IT search engine.
• You can use Splunk to search and navigate IT data from applications, servers, and network
devices in real-time.
• Data sources include logs, configurations, messages, alerts, scripts, code, metrics, etc.
• Splunk lets you search, navigate, alert, and report on all your IT data in real-time using Splunk
Web.
Learn more about what Splunk is, what it does, and how it's different.
2
What to do first
Start Splunk
Start Splunk
This topic provides brief instructions for starting Splunk. If you are new to Splunk, we recommend
reviewing the User Manual first.
Start Splunk on Windows
On Windows, Splunk is installed by default into C:\Program Files\Splunk. Many examples in
the Splunk documentation use $SPLUNK_HOME to indicate the Splunk installation, or home, directory.
You can replace the string $SPLUNK_HOME (and the Windows variant %SPLUNK_HOME%) with
C:\Program Files\Splunk if you installed Splunk into the default directory.
You can start and stop Splunk on Windows in one of the following ways:
1. Start and stop Splunk processes via the Windows Services control panel (accessible from Start
-> Control Panel -> Administrative Tools -> Services)
• Server daemon: splunkd
• Web interface: splunkweb
2. Start and stop Splunk services from a command prompt by using the NET START <service> or
NET STOP <service> commands:
• Server daemon: splunkd
• Web interface: splunkweb
3. Start, stop, and restart both processes at once by going to %SPLUNK_HOME%\bin and typing
> splunk [start|stop|restart]
Start Splunk on UNIX
Start Splunk
From a shell prompt on the Splunk sever host, run this command:
# splunk start
This starts both splunkd (indexer and other back-end processes) and splunkweb (the Splunk Web
interface). To start them individually, type:
# splunk start splunkd
or
3
# splunk start splunkweb
Note: If startwebserver is disabled in web.conf, manually starting splunkweb does not
override that setting. If it is disabled in the configuration file, it will not start.
To restart Splunk (splunkd or splunkweb) type:
# splunk restart
# splunk restart splunkd
# splunk restart splunkweb
Stop Splunk
To shut down Splunk, run this command:
# splunk stop
To stop splunkd and Splunk Web individually, type:
# splunk stop splunkd
or
# splunk stop splunkweb
Check if Splunk is running
To check if Splunk is running, type this command at the shell prompt on the server host:
# splunk status
You should see this output:
splunkd is running (PID: 3162).
splunk helpers are running (PIDs: 3164).
splunkweb is running (PID: 3216).
Note: On Unix systems, you must be logged in as the user who runs Splunk to run the splunk
status command. Other users cannot read the necessary files to report status correctly.
You can also use ps to check for running Splunk processes:
# ps aux | grep splunk | grep -v grep
Solaris users, type -ef instead of aux:
# ps -ef | grep splunk | grep -v grep
4