When setting up a new server or application, you may want to get your logs centralised into a Splunk instance for rapid access and analysis. In order to do that you will be required to install the Splunk Universal forwarder on the machine where the logs to be monitored are located.

Guide

The main documentation on how to setup a Splunk can found here: https://docs.splunk.com/Documentation/Splunk/6.5.2/Installation/Whatsinthismanual

There you will see on how to:

Setup a Splunk enterprise server
Setup the certificates
Setup the forwarders

Installation Steps

Login to your machine and gain root access
Download the Splunk Forwarder
Generate Certificates following: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Howtoself-signcertificates

You will have to Edit /opt/splunkforwarder/etc/system/local/outputs.conf to configure the splunk forwarder authentication settings

More details regarding signed certificates: http://docs.splunk.com/Documentation/Splunk/6.5.2/Security/HowtoprepareyoursignedcertificatesforSplunk

Example:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = <splunk master host eg:splunk.maestrano.io>:<splunk master port to receive traffic from this forwarder eg:9897>
compressed = true
sslRootCAPath = <path/to/your/CA/certificate.pem>
sslCertPath = <path/to/your/forwarder/certificate.pem>
sslPassword = <a password that will be encrypted by the forwarder on first startup>
sslVerifyServerCert = true

Finally start you splunk instance and automatically accept the license via:

/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt

If you want to configure Splunk to automatically start at system boot, you can run:
```
/opt/splunkforwarder/bin/splunk enable boot-start
```

You check the Splunk Forwarder logs by running:

 tail -n 150 -f /opt/splunkforwarder/var/log/splunkd.log

You can now monitor a log file or specific folder by running the following command.

splunk add monitor <path-to-my-log> -index <splunk-index-to-use> -hostname <hostname-for-this-log> -auth admin:changeme

# E.g: monitoring logs for the Maestrano rails application in UAT
splunk add monitor /var/log/maestrano -index uat-mno-web -hostname appserver1.apse1 -auth admin:changem

Installation Script Example

In order to simplify your deployments, you may create a script using this template.

You will first need to:

Download the forwarder, and have it available on a CDN: https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/linux
Generate a certificate following: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Howtoself-signcertificates

install-splunk.sh

#!/bin/bash
# This script is provided by Maestrano for convenience and may need to be adapted to match your infrastructure specificities
# This script installs and configures the splunkforwarder
# You will need to edit and replace [YOUR_COMPANY_DEB_PACKAGE_CDN] or [YOUR_COMPANY_RPM_PACKAGE_CDN] and [YOUR_COMPANY_FORWARDER_CERTIFICATE] and [YOUR_COMPANY_FORWARDER_CERTIFICATE]

# Choose environment
clear
echo "Which environment is this?"
echo "1) UAT/Test"
echo "2) Production"
read environment

if [ $environment != "1" ] && [ $environment != "2" ]; then
  echo "Invalid choice. Exiting"
  exit 1
fi

# Choose log type
clear
echo "Which type of logs are you going to track with this forwarder?"
echo "1) Web logs (Maestrano, mCluster, Connec etc.)"
echo "2) Nex! logs (Nex! management scripts, apps logs)"
echo "3) Other"
read logtype

if [ $logtype != "1" ] && [ $logtype != "2" ] && [ $logtype != "3" ]; then
  echo "Invalid choice. Exiting"
  exit 1
fi

# Choose right Splunk port based on choices above
port="9"

if [ $environment == "1" ]; then
  port="${port}6"
else
  port="${port}9"
fi

if [ $logtype == "1" ]; then
  port="${port}98"
elif [ $logtype == "2" ]; then
  port="${port}97"
else
  port="${port}99"
fi
echo "Forwarder will be configured to stream to master on port: ${port}"

# Choose server name
# clear
# echo "Enter the name of this server"
# read servername
#
# if [ -z $servername ]; then
#   echo "Server name is empty. Exiting"
#   exit 1
# fi

# Install based on target OS - APT or Yum based
if which dpkg > /dev/null 2>&1; then
  echo "Installing for Debian"
  rm -f /tmp/splunkforwarder-amd64.deb
  # Have deb package available via your cdn, something like http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/splunkforwarder-amd64.deb
  # Replace [YOUR_COMPANY_DEB_PACKAGE_CDN] by your debian package cdn url
  # example: dpkg -s splunkforwarder || ( wget -P /tmp http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/splunkforwarder-amd64.deb && dpkg -i /tmp/splunkforwarder-amd64.deb )
  dpkg -s splunkforwarder || ( wget -P /tmp [YOUR_COMPANY_DEB_PACKAGE_CDN] && dpkg -i /tmp/splunkforwarder-amd64.deb )
  rm -f /tmp/splunkforwarder-amd64.deb
else
  echo "Installing for RHEL/CentOS/Fedora/AWS Linux"
  [[ $(uname -m) == "i686" ]] && package="splunkforwarder-i386.rpm" || package="splunkforwarder-x86_64.rpm"
  
  rm -f /tmp/$package
  # Replace http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1 by your own cdn
  # Replace [YOUR_COMPANY_RPM_PACKAGE_CDN] by your rpm package cdn url
  # example: rpm -qa | grep -qw splunkforwarder || ( wget -P /tmp http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/$package && rpm -i /tmp/$package )
  rpm -qa | grep -qw splunkforwarder || ( wget -P /tmp [YOUR_COMPANY_RPM_PACKAGE_CDN]$package && rpm -i /tmp/$package )
  rm -f /tmp/$package
fi

# Configure certs
install_path="/opt/splunkforwarder"
mkdir -p $install_path/etc/system/local
mkdir -p $install_path/etc/auth/maestrano-certs


# Replace [YOUR_COMPANY_CACERT_CERTIFICATE] by your cacert.pem certificate, something like:
# -----BEGIN CERTIFICATE-----
# 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG
#(...)
# 4Kf5vAucZZVe7g==
# -----END CERTIFICATE-----

# Configure CA Cert
cat > ${install_path}/etc/auth/maestrano-certs/cacert.pem << 'EOFEOFEOF'
[YOUR_COMPANY_CACERT_CERTIFICATE]
EOFEOFEOF


# Replace [YOUR_COMPANY_FORWARDER_CERTIFICATE] by your forwarder.pem certificate, something like:
# -----BEGIN CERTIFICATE-----
# 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG
# (...)
# 4Kf5vAucZZVe7g==
# -----END CERTIFICATE-----
# -----BEGIN ENCRYPTED PRIVATE KEY-----
# 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG
# (...)
# 4Kf5vAucZZVe7g==
# -----END ENCRYPTED PRIVATE KEY-----
# -----BEGIN CERTIFICATE-----
# 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG
# (...)
# 4Kf5vAucZZVe7g==
#-----END CERTIFICATE-----

# Configure Forwarder PEM
cat > ${install_path}/etc/auth/maestrano-certs/forwarder.pem << 'EOFEOFEOF'
[YOUR_COMPANY_FORWARDER_CERTIFICATE]
EOFEOFEOF

# Configure outputs
cat > ${install_path}/etc/system/local/outputs.conf << EOFEOFEOF
[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = splunk.maestrano.io:${port}
compressed = true
sslRootCAPath = \$SPLUNK_HOME/etc/auth/maestrano-certs/cacert.pem
sslCertPath = \$SPLUNK_HOME/etc/auth/maestrano-certs/forwarder.pem
sslPassword = password
sslVerifyServerCert = true
EOFEOFEOF

# Configure logrotate
cat > /etc/logrotate.d/splunk << 'EOFEOFEOF'
/opt/splunkforwarder/var/log/splunk/*.log
/opt/splunkforwarder/var/log/introspection/*.log
{
        missingok
        size 10M
        rotate 1
        copytruncate
}
EOFEOFEOF

# Link main folders and binaries
ln -s /opt/splunkforwarder/var/log/splunk /var/log/splunk
ln -s /opt/splunkforwarder/etc /etc/splunk
ln -s /opt/splunkforwarder/bin/splunk /usr/local/bin/splunk

# Start splunk - auto accept everything
/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt

# Finally, enable at boot time
/opt/splunkforwarder/bin/splunk enable boot-start

# Notification
echo " "
echo "Splunk forwarder has been installed and configured"
echo "You can now add logs to be monitored using the following command:"
echo "---"
echo "sudo splunk add monitor <path-to-my-log> -index <splunk-index-to-use> -hostname <hostname-for-this-log> -auth admin:changeme"
echo "E.g: sudo splunk add monitor /var/log/maestrano -index uat-mno-web -hostname appserver1.apse1 -auth admin:changeme"
echo "---"

Browser not supported