Steve's Blog

For some bizarre reason when we did an upgrade from ubuntu 15.04 to 15.10 and elasticsearch was updated from v1.4.5 to v1.6.2 elasticsearch would not start (although systemctl thought it had each time).

Immediately after the upgrade, we received error messages from our application saying it could not connect to elasticsearch.

Sure enough trying to connect to the elasticsearch service failed when we attempted to use curl -XGET 'localhost:9200'

Another strange thing is that systemctl says that elasticsearch is running perfectly, even though the process is not running when we execute ps -aux | grep elastic …

It turns out the elasticsearch.pid file had been deleted in the upgrade and the elasticsearch user could not create a new one.

touch /var/run/elasticsearch.pid
chown elasticsearch /var/run/elasticsearch.pid


However, that is not all, I started receiving a new and exciting error message (see stack trace below).
This one turned out to be a failed plugin called marvel. In order to fix this error all you have to do is the following:

cd /usr/share/elasticsearch
sudo bin/plugin --remove elasticsearch/marvel/latest
sudo bin/plugin -i elasticsearch/marvel/latest


There are a few items to configure before the newly upgraded elasticsearch will work:
sudo vim /etc/elasticsearch/elasticsearch.yml
Edit the following lines:

network.host: localhost
network.bind_host: 0.0.0.0


Finally, we have to fix a broken init.d script:

sudo vim /etc/init.d/elasticsearch

Comment out this line

test "$START_DAEMON" = true || exit 0


Then you need to run the following:

sudo systemctl daemon-reload
sudo service stop elasticsearch
sudo service start elasticsearch


After this restart elasticsearch came back to life. 🙂


[2015-11-11 11:44:39,433][ERROR][bootstrap ] Exception
org.elasticsearch.common.util.concurrent.ExecutionError: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
at org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
at org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:212)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
at org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:68)
at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:67)
at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 19 more
Caused by: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2509)
at java.lang.Class.getDeclaredFields(Class.java:1819)
at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:378)
at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:376)
at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectorsForMembers(InjectionPoint.java:351)
at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectionPoints(InjectionPoint.java:345)
at org.elasticsearch.common.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:287)
at org.elasticsearch.common.inject.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:80)
at org.elasticsearch.common.inject.MembersInjectorStore.access$000(MembersInjectorStore.java:36)
at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:41)
at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 33 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 50 more