Category Archives: Splunk

HOWTO Install the Splunk Universal Forwarder on FreeBSD

24 Sunday Apr 2022

Posted by in Splunk

≈ Leave a comment

Tags

The following steps were tested on FreeBSD x64 11.4, 12.3, 13.0 and 13.1.

1) There are a few required adjustments before installing the forwarder on FreeBSD. This ensures the forwarder functions properly.

a) As root, add the following to loader.conf.
> vi /boot/loader.conf

kern.maxdsiz=”2147483648″ # 2GB
kern.dfldsiz=”2147483648″ # 2GB
machdep.hlt_cpus=0

b) Add the following to sysctl.conf.
> vi /etc/sysctl.conf

vm.max_proc_mmap=2147483647

c) Restart for the changes to take effect.
> reboot

2) If you have a Forwarder installed, remove it. If you don’t, skip to step 3.

a) Check if running and if so stop it.
> /opt/splunkforwarder/bin/splunk status
> /opt/splunkforwarder/bin/splunk version
> /opt/splunkforwarder/bin/splunk stop

b) Get the package name and delete the older release. And yes, that is a \ before the * as you have to escape it. In this example, splunkforwarder-8.1.6 is being removed.
> pkg info -g splunkforwarder\*
> pkg delete -y splunkforwarder-8.1.6
> rm /etc/rc.d/splunk
> rm -R /opt/splunkforwarder

3) If a new install, check to see if /opt exists. If not, create it.
> ls -l /opt

If not there, create it.
> mkdir /opt

3) Move the Forwarder package to /tmp on your FreeBSD host and install it.
> cd /tmp
> pkg install splunkforwarder-8.2.3-cd0848707637-freebsd-11.3-amd64.txz

Updating FreeBSD repository catalogue…
FreeBSD repository is up to date.
All repositories are up to date.
Checking integrity… done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
splunkforwarder: 8.2.3

Number of packages to be installed: 1

The process will require 64 MiB more space.

Proceed with this action? [y/N]: y
[1/1] Installing splunkforwarder-8.2.3…
Extracting splunkforwarder-8.2.3: 100%
complete

a) Start the Forwarder and accept the license.
> /opt/splunkforwarder/bin/splunk start –accept-license –answer-yes

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: splunkadmin
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:

b) Enable the Forwarder to start when the system is booted.
> /opt/splunkforwarder/bin/splunk enable boot-start

Init script installed at /etc/rc.d/splunk.
Init script is configured to run at boot.

4) If you don’t use a Deployment server, skip this step. If you use a Deployment server, set the deployment server configuration. Typically this comes from your Splunk team so this is an example only.
> vi /opt/splunkforwarder/etc/system/local/deploymentclient.conf

[deployment-client]

[target-broker:deploymentServer]
targetUri=192.168.1.100:8089

5) Restart the Forwarder and verify the configuration.
> /opt/splunkforwarder/bin/splunk restart
> /opt/splunkforwarder/bin/splunk status

a) Verify that its set in rc to start at boot and that the process is running.
> service -e | grep splunk
/etc/rc.d/splunk

> ps -x | grep splunkd
1494 – S 0:02.54 splunkd -p 8089 restart
1495 – Is 0:00.04 splunkd: [splunkd pid=1494] splunkd -p 8089 restart [process-runner] (splunkd)

> cat /etc/rc.conf | grep splunk
splunk_enable=”YES”

6) Installation is complete.

HOWTO install a Splunk Universal Forwarder on Solaris 11 SPARC and x64 Using pkg(p5p) and tar

05 Saturday Mar 2022

Posted by in Solaris, Splunk

≈ 2 Comments

Tags

I recently had to get the Forwarders installed and there are no detailed steps in the Splunk docs for Solaris 11. If you need Solaris 10 steps, see my post here.

1) There are two installation options and platforms supported by Splunk; using pkg and tar on SPARC and x64 CPUs. The platform/CPU type is at the end of the filename shown below. The steps below cover both types of installation scenarios. Choose the steps for the way in which you want to install and the platform you have. There is a Universal Forwarder for SPARC and x64 (Intel/AMD) CPUs. Be sure to su – root before you start, or use sudo in front of the commands.

a) Native pkg (as in .p5p) formatted binary – easy to manage and upgrade, software included in inventory and vuln scans. Installed using pkg:
splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p
splunkforwarder-8.2.3-cd0848707637-solaris-sparc.p5p

b) Extracted via tar – software not seen by inventory and vuln scans, only the running splunkd process identifies it in scans (but not the version running). Installed using tar:
splunkforwarder-8.2.3-cd0848707637-SunOS-x86_64.tar.Z
splunkforwarder-8.2.3-cd0848707637-SunOS-sparc.tar.Z

2) Verify that you have the required libc installed.
> pvs /usr/lib/libc.so.1 | grep SUNW_1.22.7
> pvs /usr/lib/libc.so.1 (to get a list of all libc versions)

3) Backup your files. If you have any locally customized .conf files in the following folders, save a copy now. Note that some older forwarders
may not use upper case letters in the path.

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/
/opt/splunkforwarder/etc/system/local/

4) If an existing Forwarder is running, identify your Splunk index. You will need this value along with your hostname, IP address and platform later.
> cat /opt/splunkforwarder/etc/system/local/inputs.conf | grep index
> hostname
> ifconfig -a | grep inet (or just ifconfig -a if you have multiple NICs plumbed)
> uname -a

5) Check for existing version and state.
If the existing Forwarder was installed with pkg:
> pkg list | splunk
> pkg info -r splunkforwarder | egrep -i “Summary|Version”
> /opt/splunkforwarder/bin/splunk status
> ps aevx | grep splunkd

If the existing Forwarder was installed with tar:
> /opt/splunkforwarder/bin/splunk version
> /opt/splunkforwarder/bin/splunk status
> ps aevx | grep splunkd

6) Stop the currently installed Universal Forwarder and then remove it.
> /opt/splunkforwarder/bin/splunk stop

If installed via pkg:
> pkg uninstall splunkforwarder
> rm /etc/init.d/splunk

If installed via tar:
> cd /opt/
> rm -r splunkforwarder
> rm /etc/init.d/splunk

7) Make sure the Splunk sockets are no longer in use or locked. Copy/paste each line below one at a time and press enter.
> PORT=8089; for PID in /proc/*; do pfiles ${PID} |grep “port: ${PORT}” && echo ${PID}; done
> PORT=9997; for PID in /proc/*; do pfiles ${PID} |grep “port: ${PORT}” && echo ${PID}; done

a) If the commands above return anything (and they shouldn’t), you need to kill the process.
> pfiles process_id (from the output of the commands above to get process details)
> pkill process_id

8) Move the package to your host. Based on your CPU type (SPARC or x64) and the installation method you chose (pkg or tar), copy the Universal Forwarder to your server using scp, Filezilla, WinSCP or your preferred file transfer method.

If installing with tar:
> cp splunkforwarder-8.2.3-cd0848707637-SunOS-x86_64.tar.Z /opt/
> cd /opt
> uncompress splunkforwarder-8.2.3-cd0848707637-SunOS-x86_64.tar.Z
> tar xvf splunkforwarder-8.2.3-cd0848707637-SunOS-x86_64.tar
> chown -R root:root splunkforwarder

If installing with pkg:
> cp splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p /opt/
> cd /opt/
> pkg list -g file:////opt/splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p
> pkgrepo info -s /opt/splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p
> pkg set-publisher -p /opt/splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p splunk
> pkg install -nv splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p
> pkg install -g ./splunkforwarder-8.2.3-cd0848707637-solaris-intel.p5p splunkforwarder

a) For both pkgadd and tar, start and accept the license.
> /opt/splunkforwarder/bin/splunk start –accept-license –answer-yes

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: splunkadm  (<– you can use any account name you want here, its for splunkd, not an OS account)
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:

b) For both pkgadd and tar, enable auto-start when the server is rebooted.
> /opt/splunkforwarder/bin/splunk enable boot-start

c) For both pkgadd and tar, copy the deploymentclient.conf file (provided by the Splunk team) to the path listed below. Note that if a Deployment server is not being used, skip this step.
> cp deploymentclient.conf /opt/splunkforwarder/etc/system/local/
> /opt/splunkforwarder/etc/system/local/
> chown root:root deploymentclient.conf

8) Restart the Forwarder, verify that its running and contact your Splunk team via email. Send them your index, hostname, ip and platform so they can associate your new Forwarder with the proper inputs.
> /opt/splunkforwarder/bin/splunk restart
> /opt/splunkforwarder/bin/splunk status
> ps aevx | grep splunkd

9) Connectivity issues? See if the Solaris firewall is on and has any rules for Splunk ports 8089 and 9997. Adjust rules as needed.
> svcs -x firewall:default
> pfbash pfctl -s rules

HOWTO install a Splunk Universal Forwarder on Solaris 10 SPARC and x64 Using pkgadd and tar

13 Sunday Feb 2022

Posted by in Solaris, Splunk

≈ 1 Comment

OK, OK, I know. Solaris 10 is old and dusty. Extended support has been pushed out to January 2024 so there are still plenty of systems in use out there. I recently had to get the Forwarders installed and there are no detailed steps in the Splunk docs. This covers installing via pkgadd and tar. Also, the last available Forwarder I could find on their site that supports Solaris 10 is v7.3.9.

1) There are two installation options and platforms supported by Splunk; using pkgadd and tar on SPARC and x64 CPUs. The platform/CPU type is at the end of the filename shown below. The steps below cover both types of installation scenarios. Choose the steps for the way in which you want to install and the platform you have. There is a Universal Forwarder for SPARC and x64 (Intel/AMD) CPUs so simply insert the Forwarder filename you need in the steps listed below. This HOWTO was done using Solaris 10 x64 so if you have a SPARC host, use the sparc Forwarder filename.

a) Native pkg formatted binary – easy to manage and upgrade, software included in inventory and vuln scans. Installed using pkgadd:
splunkforwarder-7.3.9-39a78bf1bc5b-solaris-10-intel.pkg.Z
splunkforwarder-7.3.9-39a78bf1bc5b-solaris-10-sparc.pkg.Z

b) Extracted via tar – software not seen by inventory and vuln scans, only the running splunkd process identifies it in scans (but not the version running). Installed using tar:
splunkforwarder-7.3.9-39a78bf1bc5b-SunOS-x86_64.tar.Z
splunkforwarder-7.3.9-39a78bf1bc5b-SunOS-sparc.tar.Z

2) Verify that you have the required libc installed.
> pvs /usr/lib/libc.so.1 | grep SUNW_1.22.7
> pvs /usr/lib/libc.so.1 (to get a list of all libc versions)

a) Solaris 10 hosts must be updated to libc SUNW_1.22.7 or later. See the Oracle Support page that lists patches that include libc changes (Oracle Support required): https://support.oracle.com/rs?type=doc&id=2069855.1

SUNW_1.22.7
SPARC: Patch 144500-19
X64: Patch 144501-19

3) Backup your files. If you have any locally customized .conf files in the following folders, save a copy now. Note that some older forwarders
may not use upper case letters in the path.

/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/local/
/opt/splunkforwarder/etc/system/local/

4) Identify your Splunk index. You will need this value along with your hostname, IP address and platform later.
> cat /opt/splunkforwarder/etc/system/local/inputs.conf | grep index
> hostname
> ifconfig -a | grep inet (or just ifconfig -a if you have multiple NICs plumbed)
> uname -a

5) Check for existing version and state.
If the existing Forwarder was installed with pkgadd:
> pkginfo -l | grep splunk
> pkginfo -l splunkforwarder* | grep VERSION
> /opt/splunkforwarder/bin/splunk status
> ps -ef | grep splunkd

If the existing Forwarder was installed with tar:
> /opt/splunkforwarder/bin/splunk version
> /opt/splunkforwarder/bin/splunk status
> ps -ef | grep splunkd

6) Stop the currently installed Universal Forwarder and then remove it.
> /opt/splunkforwarder/bin/splunk stop

If installed via pkgadd:
> pkgrm splunkforwarder
> rm /etc/init.d/splunk

If installed via tar:
> cd /opt/
> rm -r splunkforwarder
> rm /etc/init.d/splunk

7) Make sure the Splunk sockets are no longer in use or locked. Copy/paste each line below one at a time and press enter.
> PORT=8089; for PID in /proc/*; do pfiles ${PID} |grep “port: ${PORT}” && echo ${PID}; done
> PORT=9997; for PID in /proc/*; do pfiles ${PID} |grep “port: ${PORT}” && echo ${PID}; done

a) If the commands above return anything (and they shouldn’t), you need to kill the process.
> pfiles process_id (from the output of the commands above to get process details)
> pkill process_id

8) Move the package to your host. Based on your CPU type (SPARC or x64) and the installation method you chose (pkgadd or tar), copy the Universal Forwarder to your server using scp, Filezilla, WinSCP or your preferred file transfer method.

If installing with tar:
> cp splunkforwarder-7.3.9-39a78bf1bc5b-SunOS-x86_64.tar.Z /opt/
> cd /opt
> uncompress splunkforwarder-7.3.9-39a78bf1bc5b-SunOS-x86_64.tar.Z
> tar xvf splunkforwarder-7.3.9-39a78bf1bc5b-SunOS-x86_64.tar
> chown -R root:root splunkforwarder

If installing with pkgadd:
> pkgadd -d splunkforwarder-7.3.9-39a78bf1bc5b-solaris-10-intel.pkg all

Do you want to continue with the installation of <splunkforwarder> [y,n,?] y

a) For both pkgadd and tar, start and accept the license.
> /opt/splunkforwarder/bin/splunk start –accept-license –answer-yes

This appears to be your first time running this version of Splunk.

Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.

Please enter an administrator username: splunkadm  (<– you can use any account name you want here, its for splunkd, not an OS account)
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:

b) For both pkgadd and tar, enable auto-start when the server is rebooted.
> /opt/splunkforwarder/bin/splunk enable boot-start

c) For both pkgadd and tar, copy the deploymentclient.conf file to the path listed below. Note that if a Deployment server is not being used, skip this step.
> cp deploymentclient.conf /opt/splunkforwarder/etc/system/local/
> /opt/splunkforwarder/etc/system/local/
> chown root:root deploymentclient.conf

8) Restart the Forwarder, verify that its running and contact your Splunk team via email. Send them your index, hostname, ip and platform so they can associate your new Forwarder with the proper inputs.
> /opt/splunkforwarder/bin/splunk restart
> /opt/splunkforwarder/bin/splunk status
> ps -ef | grep splunkd

9) Connectivity issues? See if the Solaris firewall is on and has any rules for Splunk ports 8089 and 9997. Adjust rules as needed.
> svcs | egrep ‘(pfil|ipfilter)’
> ipfstat -io

Sysmon Event ID 1 Process Creation rules for Splunk Universal Forwarder and McAfee All Access

20 Saturday Nov 2021

Posted by in Security, Splunk

≈ Leave a comment

Tags

, ,

When using Sysmon to monitor processes in Splunk, you can exclude the chattiest processes that eat away at your license and indexing I/O. Splunk recommends excluding its own Universal Forwarder as well as antivirus products.

1) Splunk’s Sysmon Add-on docs list a few options for the sysmon config file. I used the SwiftOnSecurity template and simply added the UF and McAfee executables.

Splunk Add-on for Sysmon Doc:
https://docs.splunk.com/Documentation/AddOns/released/MSSysmon/ConfigureSysmon

Ready to run Sysmon config file:
https://github.com/SwiftOnSecurity/sysmon-config

Sysmon:
https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon

2) Open your Sysmon xml config file with Notepad++ or VSCode. Use Notepad++ or VSCode when editing xml on Windows. They recognize UNIX newline format and XML syntax highlighting so your file will be easier to read. Don’t use Notepad.exe. Under the heading <!–SYSMON EVENT ID 1 : PROCESS CREATION [ProcessCreate]–> enter the text below and save the file.

3) When done editing, reload the active config.

> sysmon -c c:\path\to\your_sysmon_config_file_v2.xml

<!–SECTION: Splunk:Universal Forwarder–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunkd.exe</Image> <!–Splunk:Universal Forwarder: binary that accesses, processes, and indexes streaming data–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\btool.exe</Image> <!–Splunk:Universal Forwarder: cli util used to troubleshoot config file issues–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe</Image> <!–Splunk:Universal Forwarder: the control application for the Windows version of Splunk–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe</Image> <!–Splunk:Universal Forwarder: runs whenever you configure an Active Directory monitoring input–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-perfmon.exe</Image> <!–Splunk:Universal Forwarder: runs when you monitor performance data–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-netmon.exe</Image> <!–Splunk:Universal Forwarder: runs when you configure Splunk to monitor network information–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe</Image> <!–Splunk:Universal Forwarder: runs when you configure a Registry monitoring input–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe</Image> <!–Splunk:Universal Forwarder: event log collections, outputs events as they are collected–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-winhostmon</Image> <!–Splunk:Universal Forwarder: runs when you configure a Windows host monitoring input–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-winprintmon.exe</Image> <!–Splunk:Universal Forwarder: runs when you configure a Windows print monitoring input –>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-wmi.exe</Image> <!–Splunk:Universal Forwarder: runs when you configure a performance monitor–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-powershell.exe</Image> <!–Splunk:Universal Forwarder: runs when you configure a PowerShell command or script monitor–>
<Image condition=”is”>C:\Program Files\SplunkUniversalForwarder\bin\splunk-MonitorNoHandle.exe</Image> <!–Splunk:Universal Forwarder: this input monitors files without using Windows file handles.–>
<!–SECTION: McAfee:AntiVirus–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\Platform\Core\mchost.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\Platform\CommonBuild\McCBEntAndInstru.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\Platform\McPltCmd.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MQS\QcShm.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee.com\Agent\mcupdate.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\MSGSDK\msgrunner.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\TaskScheduler\McAMTaskAgent.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\UPDMGR\8.1.149.1\mcupdatemgr.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\CSP\4.1.106.0\McCSPServiceHost.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MSC\mcinfo.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\Platform\MSM\McSmtFwk.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\Platform\McUICnt.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MSC\OOBE\mcocrollback.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MSC\McInstru.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\UPDMGR\8.1.149.1\mcupdutl.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\DynamicAppDownloader\DADUpdater.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\HackerWatch\hwupdchk.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MSC\mcsync.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\Common Files\McAfee\ChromiumContainer\delegate.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\VUL\McVulCtr.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\VUL\McVulCon.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\MfeAV\MfeAVSvc.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee\CoreUI\Launch.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>
<Image condition=”is”>C:\Program Files\McAfee.com\Agent\mcagent.exe</Image> <!–McAfee:AntiVirus: no description – TBD–>

Enabling TLS 1.2 on the Splunk 6.2x Console and Forwarders using Openssl and self signed certs.

01 Friday Jan 2016

Posted by in Security, Splunk

≈ Leave a comment

Tags

,

Good luck. You will need it. Certificates are a major headache and complicated to implement. Using them with Splunk is no different. Splunk’s penchant for twiddling files all over the place makes this process time consuming an rife with error. This post will hopefully help you get it done. This covers encrypting the management console and forwarder traffic. This HOWTO is not for a clustered deployment although it could be adapted to serve that purpose. It was done on Windows 2012 R2 with a single Splunk Enterprise deployment (search head and indexer on the same server) and several forwarders. Use your own naming conventions and hosts for fqdn. Please don’t ask me questions on this post. I almost didn’t survive the process. I won’t have time to reply for a while anyway.

1) On the Splunk Search Head, set your environment.

> cd C:\Program Files\Splunk\bin
> splunk envvars > setsplunkenv.bat & setsplunkenv.bat
> setsplunkenv.bat

2) Create dir $SPLUNK_HOME\etc\auth\UScerts and cd into it.

> cd C:\Program Files\Splunk\etc\auth\UScerts

3) Create a root key.
> openssl genrsa -aes256 -out USCA_root.key 2048

4) Generate and sign the certificate.
> openssl req -new -key USCA_root.key -out USCA_root.csr

5) Generate the public certificate.
> openssl x509 -req -in USCA_root.csr -sha256 -signkey USCA_root.key -CAcreateserial -out USCA_root.pem -days 3650

6) Generate a key for your Web(search head)server certificate.
> openssl genrsa -aes256 -out me.fqdn.com.key 2048

7) Request and sign a new server certificate.
> openssl req -new -key me.fqdn.com.key -out me.fqdn.com.csr

8) Use the CSR me.fqdn.com.csr and your CA certificate and private key to generate a server certificate.
> openssl x509 -req -in me.fqdn.com.csr -sha256 -CA USCA_root.pem -CAkey USCA_root.key -CAcreateserial -out
me.fqdn.com.pem -days 730

9) Creating a (removing encryption from priv key) priv key without a passphrase. Required for webservers.
> openssl rsa -in me.fqdn.com.key -out me.fqdn.com_nopass.key

10) Create a combined cert file.
> type me.fqdn.com.pem me.fqdn.com_nopass.key USCA_root.pem > me.fqdn.com_nopass_use.pem

11) On the search head, edit the \etc\system\local\web.conf and add the following:
[settings]
enableSplunkWebSSL = 1
httpport = 8843
privKeyPath = etc\auth\UScerts\me.fqdn.com_nopass_use.pem
CaCertPath = etc\auth\UScerts\USCA_root.pem

Add to \etc\system\local\server.conf

enableSplunkdSSL = true
sslVersions = tls1.2
allowSslCompression = false
allowSslRenegotiation = false
cipherSuite = TLSv1+HIGH:@STRENGTH

12) Restart Splunk. Close your browser, relaunch and login to the console to verify (make sure to use the port defined above in web.conf; https://hostname or ip:8843).  If you scan with Nessus, Retina, etc., it should now be free from SSL errors.

Certs for Forwarders:

Create a SAN (subject alternative name) cert. Although not officially supported by Splunk when I originally wrote this, it does work.

1) Create a new folder in etc\auth\UScerts\SANcert.

2) Copy the openssl.cnf to the new folder.  C:\Program Files\Splunk\openssl.cnf to C:\Program Files\Splunk\etc\auth\UScerts\SANcert

> cd C:\Program Files\Splunk\etc\auth\UScerts\SANcert

3) In Windows 2012 R2 – Take ownership of the copied openssl.cnf file. Right-click > properties, and then add your user with Full Control to the file.

4) In Notepad or Wordpad, edit openssl.cnf. Wordpad is preferred.

a) Search (using the Find function in the upper right of Wordpad) and uncomment this line:
# req_extensions = v3_req # The extensions to add to a certificate request

b) Next, search for and modify this section to include the following if it does not already have it:
[ v3_req ] # Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAtlName = @alt_names

c) Create this section next, between [ v3_req ] and [ v3_ca ].
[alt_names]
DNS.1 = your.server.com
DNS.2 = your.next.server.com
DNS.3 = your.other.server.com
IP.1 = an IP address for a server
IP.2 = another IP address for a server
— note, add as many as you like. You will need one for each forwarder if you want to identify them individually.

5) Generate a new CSR.
openssl req -new -key me.fqdn.com.key -out me.fqdn.com_SAN.csr -config “C:\Program Files\Splunk\etc\auth\UScerts\SANcert\openssl.cnf”

Make sure you use: *.your.fqdn for Common name question. This is the wildcard for your domain, such as *.yourdomain.com

6) Check text of cert csr. You should see the items in the alt_names from above.
> openssl req -text -noout -in me.fqdn.com_SAN.csr

7) Create a cert.
> openssl x509 -req -in me.fqdn.com_SAN.csr -sha256 -CA USCA_root.pem -CAkey USCA_root.key -CAcreateserial -out me.fqdn.com_SAN.pem -extensions v3_req -days 730 -extfile “C:\Program Files\Splunk\etc\auth\UScerts\SANcert\openssl.cnf”

8) On the Indexer, edit \etc\system\local\inputs.conf and add the following and restart Splunk.

[SSL]
rootCA = etc\auth\UScerts\USCA_root.pem
servercert = etc\auth\UScerts\me.fqdn.com_SAN.pem
password = your_password
cipherSuite = TLSv1+HIGH:@STRENGTH

[splunktcp-ssl:9997]
compressed = false

9) Now restart splunk:
$SPLUNK_HOME\bin\splunk restart splunkd

10) Configure your Forwarders to use the certificates. Use your Deployment Server to distribute the certs and modified outputs.conf to your forwarders.

a) On the Search head that is acting as your deployment server, edit the outputs.conf file in etc\deployment-apps\<your name for SendToIndexer>\local\ with the following.

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = your.ip.:9997
compressed = false
sslRootCAPath = etc\apps\<your name for SendToIndexer>\USCA_root.pem
sslCertPath = etc\apps\<your name for SendToIndexer>\me.fqdn.com_SAN.pem
sslPassword = <your password>
sslVerifyServerCert = true

b) Copy the etc\auth\UScerts\USCA_root.pem and etc\auth\UScerts\me.fqdn.com_SAN.pem files to the etc\deployment-apps\<your name for SendToIndexer>\local folder on your deployment server and they will be copied to each Forwarder for you.

11) Restart Splunk.
$SPLUNK_HOME\bin\splunk restart splunkd

12) Done. I hope.