Install Splunk Universal Forwarder
When setting up a new server or application, you may want to get your logs centralised into a Splunk instance for rapid access and analysis. In order to do that you will be required to install the Splunk Universal forwarder on the machine where the logs to be monitored are located.
Guide
The main documentation on how to setup a Splunk can found here: https://docs.splunk.com/Documentation/Splunk/6.5.2/Installation/Whatsinthismanual
There you will see on how to:
- Setup a Splunk enterprise server
- Setup the certificates
- Setup the forwarders
Installation Steps
- Login to your machine and gain root access
Download the Splunk Forwarder
- Generate Certificates following: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Howtoself-signcertificates
You will have to Edit /opt/splunkforwarder/etc/system/local/outputs.conf to configure the splunk forwarder authentication settings
More details regarding signed certificates: http://docs.splunk.com/Documentation/Splunk/6.5.2/Security/HowtoprepareyoursignedcertificatesforSplunk
Example:
[tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = <splunk master host eg:splunk.maestrano.io>:<splunk master port to receive traffic from this forwarder eg:9897> compressed = true sslRootCAPath = <path/to/your/CA/certificate.pem> sslCertPath = <path/to/your/forwarder/certificate.pem> sslPassword = <a password that will be encrypted by the forwarder on first startup> sslVerifyServerCert = true
Finally start you splunk instance and automatically accept the license via:
/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt
If you want to configure Splunk to automatically start at system boot, you can run:
/opt/splunkforwarder/bin/splunk enable boot-start
You check the Splunk Forwarder logs by running:
tail -n 150 -f /opt/splunkforwarder/var/log/splunkd.log
You can now monitor a log file or specific folder by running the following command.
splunk add monitor <path-to-my-log> -index <splunk-index-to-use> -hostname <hostname-for-this-log> -auth admin:changeme # E.g: monitoring logs for the Maestrano rails application in UAT splunk add monitor /var/log/maestrano -index uat-mno-web -hostname appserver1.apse1 -auth admin:changem
Installation Script Example
In order to simplify your deployments, you may create a script using this template.
You will first need to:
- Download the forwarder, and have it available on a CDN: https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/linux
- Generate a certificate following: https://docs.splunk.com/Documentation/Splunk/6.5.2/Security/Howtoself-signcertificates
#!/bin/bash # This script is provided by Maestrano for convenience and may need to be adapted to match your infrastructure specificities # This script installs and configures the splunkforwarder # You will need to edit and replace [YOUR_COMPANY_DEB_PACKAGE_CDN] or [YOUR_COMPANY_RPM_PACKAGE_CDN] and [YOUR_COMPANY_FORWARDER_CERTIFICATE] and [YOUR_COMPANY_FORWARDER_CERTIFICATE] # Choose environment clear echo "Which environment is this?" echo "1) UAT/Test" echo "2) Production" read environment if [ $environment != "1" ] && [ $environment != "2" ]; then echo "Invalid choice. Exiting" exit 1 fi # Choose log type clear echo "Which type of logs are you going to track with this forwarder?" echo "1) Web logs (Maestrano, mCluster, Connec etc.)" echo "2) Nex! logs (Nex! management scripts, apps logs)" echo "3) Other" read logtype if [ $logtype != "1" ] && [ $logtype != "2" ] && [ $logtype != "3" ]; then echo "Invalid choice. Exiting" exit 1 fi # Choose right Splunk port based on choices above port="9" if [ $environment == "1" ]; then port="${port}6" else port="${port}9" fi if [ $logtype == "1" ]; then port="${port}98" elif [ $logtype == "2" ]; then port="${port}97" else port="${port}99" fi echo "Forwarder will be configured to stream to master on port: ${port}" # Choose server name # clear # echo "Enter the name of this server" # read servername # # if [ -z $servername ]; then # echo "Server name is empty. Exiting" # exit 1 # fi # Install based on target OS - APT or Yum based if which dpkg > /dev/null 2>&1; then echo "Installing for Debian" rm -f /tmp/splunkforwarder-amd64.deb # Have deb package available via your cdn, something like http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/splunkforwarder-amd64.deb # Replace [YOUR_COMPANY_DEB_PACKAGE_CDN] by your debian package cdn url # example: dpkg -s splunkforwarder || ( wget -P /tmp http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/splunkforwarder-amd64.deb && dpkg -i /tmp/splunkforwarder-amd64.deb ) dpkg -s splunkforwarder || ( wget -P /tmp [YOUR_COMPANY_DEB_PACKAGE_CDN] && dpkg -i /tmp/splunkforwarder-amd64.deb ) rm -f /tmp/splunkforwarder-amd64.deb else echo "Installing for RHEL/CentOS/Fedora/AWS Linux" [[ $(uname -m) == "i686" ]] && package="splunkforwarder-i386.rpm" || package="splunkforwarder-x86_64.rpm" rm -f /tmp/$package # Replace http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1 by your own cdn # Replace [YOUR_COMPANY_RPM_PACKAGE_CDN] by your rpm package cdn url # example: rpm -qa | grep -qw splunkforwarder || ( wget -P /tmp http://cdn.yourcompany.com/pkg/splunkforwarder/6.2.1/$package && rpm -i /tmp/$package ) rpm -qa | grep -qw splunkforwarder || ( wget -P /tmp [YOUR_COMPANY_RPM_PACKAGE_CDN]$package && rpm -i /tmp/$package ) rm -f /tmp/$package fi # Configure certs install_path="/opt/splunkforwarder" mkdir -p $install_path/etc/system/local mkdir -p $install_path/etc/auth/maestrano-certs # Replace [YOUR_COMPANY_CACERT_CERTIFICATE] by your cacert.pem certificate, something like: # -----BEGIN CERTIFICATE----- # 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG #(...) # 4Kf5vAucZZVe7g== # -----END CERTIFICATE----- # Configure CA Cert cat > ${install_path}/etc/auth/maestrano-certs/cacert.pem << 'EOFEOFEOF' [YOUR_COMPANY_CACERT_CERTIFICATE] EOFEOFEOF # Replace [YOUR_COMPANY_FORWARDER_CERTIFICATE] by your forwarder.pem certificate, something like: # -----BEGIN CERTIFICATE----- # 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG # (...) # 4Kf5vAucZZVe7g== # -----END CERTIFICATE----- # -----BEGIN ENCRYPTED PRIVATE KEY----- # 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG # (...) # 4Kf5vAucZZVe7g== # -----END ENCRYPTED PRIVATE KEY----- # -----BEGIN CERTIFICATE----- # 0Yr9mDQjfzdLP3GObc7y7rwz8a5ozATwfpqZiWYjM34oKFPSj7kwLdA+otx0glGG # (...) # 4Kf5vAucZZVe7g== #-----END CERTIFICATE----- # Configure Forwarder PEM cat > ${install_path}/etc/auth/maestrano-certs/forwarder.pem << 'EOFEOFEOF' [YOUR_COMPANY_FORWARDER_CERTIFICATE] EOFEOFEOF # Configure outputs cat > ${install_path}/etc/system/local/outputs.conf << EOFEOFEOF [tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = splunk.maestrano.io:${port} compressed = true sslRootCAPath = \$SPLUNK_HOME/etc/auth/maestrano-certs/cacert.pem sslCertPath = \$SPLUNK_HOME/etc/auth/maestrano-certs/forwarder.pem sslPassword = password sslVerifyServerCert = true EOFEOFEOF # Configure logrotate cat > /etc/logrotate.d/splunk << 'EOFEOFEOF' /opt/splunkforwarder/var/log/splunk/*.log /opt/splunkforwarder/var/log/introspection/*.log { missingok size 10M rotate 1 copytruncate } EOFEOFEOF # Link main folders and binaries ln -s /opt/splunkforwarder/var/log/splunk /var/log/splunk ln -s /opt/splunkforwarder/etc /etc/splunk ln -s /opt/splunkforwarder/bin/splunk /usr/local/bin/splunk # Start splunk - auto accept everything /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt # Finally, enable at boot time /opt/splunkforwarder/bin/splunk enable boot-start # Notification echo " " echo "Splunk forwarder has been installed and configured" echo "You can now add logs to be monitored using the following command:" echo "---" echo "sudo splunk add monitor <path-to-my-log> -index <splunk-index-to-use> -hostname <hostname-for-this-log> -auth admin:changeme" echo "E.g: sudo splunk add monitor /var/log/maestrano -index uat-mno-web -hostname appserver1.apse1 -auth admin:changeme" echo "---"