ElasticSearch issues upgrading from Ubuntu 15.04 -> 15.10

For some bizarre reason when we did an upgrade from ubuntu 15.04 to 15.10 and elasticsearch was updated from v1.4.5 to v1.6.2 elasticsearch would not start (although systemctl thought it had each time).

Immediately after the upgrade, we received error messages from our application saying it could not connect to elasticsearch.

Sure enough trying to connect to the elasticsearch service failed when we attempted to use curl -XGET 'localhost:9200'

Another strange thing is that systemctl says that elasticsearch is running perfectly, even though the process is not running when we execute ps -aux | grep elastic …

It turns out the elasticsearch.pid file had been deleted in the upgrade and the elasticsearch user could not create a new one.

touch /var/run/elasticsearch.pid
chown elasticsearch /var/run/elasticsearch.pid

However, that is not all, I started receiving a new and exciting error message (see stack trace below).
This one turned out to be a failed plugin called marvel. In order to fix this error all you have to do is the following:

cd /usr/share/elasticsearch
sudo bin/plugin --remove elasticsearch/marvel/latest
sudo bin/plugin -i elasticsearch/marvel/latest

There are a few items to configure before the newly upgraded elasticsearch will work:
sudo vim /etc/elasticsearch/elasticsearch.yml
Edit the following lines:

network.host: localhost
network.bind_host: 0.0.0.0

Finally, we have to fix a broken init.d script:

sudo vim /etc/init.d/elasticsearch

Comment out this line

test "$START_DAEMON" = true || exit 0

Then you need to run the following:

sudo systemctl daemon-reload
sudo service stop elasticsearch
sudo service start elasticsearch

After this restart elasticsearch came back to life. 🙂



[2015-11-11 11:44:39,433][ERROR][bootstrap ] Exception
org.elasticsearch.common.util.concurrent.ExecutionError: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
at org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
at org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:212)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
at org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:68)
at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:67)
at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 19 more
Caused by: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2509)
at java.lang.Class.getDeclaredFields(Class.java:1819)
at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:378)
at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:376)
at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectorsForMembers(InjectionPoint.java:351)
at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectionPoints(InjectionPoint.java:345)
at org.elasticsearch.common.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:287)
at org.elasticsearch.common.inject.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:80)
at org.elasticsearch.common.inject.MembersInjectorStore.access$000(MembersInjectorStore.java:36)
at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:41)
at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 50 more

  1. #1 by Alexandre Quessy on 3 December 2015 - 17:31

    Hello Steve,
    It works! Thank you for posting this.

  2. #2 by Artyom Nikolaev on 1 February 2016 - 11:07

    Thank you!

  3. #3 by Home Alone on 13 April 2016 - 20:33

    Thanks, this got me started. Also had issues with the java-wrapper script being unable to detect Oracle Java 8 (they changed the naming convention recently), so I just hardcoded JAVA_HOME=`/usr/lib/jvm/jvm-default` and as JAVA=`which java` in there, and symlinked a jvm-default directory to java-8-oracle. I also had to hardcode 0.0.0.0 and localhost as above, other addresses didn’t seem to work (configuration broken?)

    I’m surprised that a popular package like elasticsearch does not have the systemd setup.

Leave a Reply

Your email address will not be published. Required fields are marked *