Just ran into a problem tonight while upgrading my Splunk instance from 6.1 to 6.3.1. It had been offline for quite a while (it died and it’s just a demo system of mine) so it hadn’t been upgraded in a while.

While following the phenomenal upgrade process here:

http://www.redelijkheid.com/blog/2011/11/10/upgrading-splunk-on-ubuntu-linux.html

I ran into an odd issue where the migration script was attempting to change the permissions on the /splunk/etc/passwd file and erroring out on me without finishing the upgrade. Full error message below (so Google finds it and people don’t have as hard of a time resolving this as I did).

sudo ./splunk start --accept-license

This appears to be an upgrade of Splunk.
--------------------------------------------------------------------------------)

Splunk has detected an older version of Splunk installed on this machine. To
finish upgrading to the new version, Splunk's installer will automatically
update and alter your current configuration files. Deprecated configuration
files will be renamed with a .deprecated extension.

You can choose to preview the changes that will be made to your configuration
files before proceeding with the migration and upgrade:

If you want to migrate and upgrade without previewing the changes that will be
made to your existing configuration files, choose 'y'.
If you want to see what changes will be made before you proceed with the
upgrade, choose 'n'.


Perform migration and upgrade without previewing configuration changes? [y/n] y

-- Migration information is being logged to '/opt/splunk/var/log/splunk/migration.log.2015-11-23.17-59-09' --

Migrating to:
VERSION=6.3.1
BUILD=f3e41e4b37b2
PRODUCT=splunk
PLATFORM=Linux-x86_64

Copying '/opt/splunk/etc/myinstall/splunkd.xml' to '/opt/splunk/etc/myinstall/splunkd.xml-migrate.bak'.

Checking saved search compatibility...

Handling deprecated files...

Checking script configuration...

chmod: changing permissions of `/opt/splunk/etc/passwd': Operation not permitted

Command error: Was unable to change permissions of '%s'. Unwise to continue without figuring out why.

I wasn’t able to find anyone online running into this issue, thus this post. After digging around, I came across Splunk’s code on Github and discovered my issue.

Link to Github: https://github.com/edo17/splunk/blob/master/lib/python2.7/site-packages/splunk/clilib/migration.py

def fixPasswdPermissions(path, isDryRun):
 """
 In some versions prior to 3.2.0, passwd files were created world-readable.
 On Unix systems, just ensure the perms are set properly.
 """

# can't do anything about this on windows, and free doesn't have a passwd file.
 if comm.isWindows or not os.path.exists(path):
 return
 else:
 if 0 != subprocess.call(['chmod', PERMS_OWNER_RW_ONLY, path]):
 raise cex.ArgError, "Was unable to change permissions of '%s'. Unwise to continue without figuring out why."

It looks like with the free version, it doesn’t have a passwd file, even though one exists on my server. I renamed it to passwd.old and then everything ran!