{"id":84,"date":"2015-11-11T14:04:57","date_gmt":"2015-11-11T14:04:57","guid":{"rendered":"http:\/\/blog.lexxweb.co.uk\/?p=84"},"modified":"2015-12-23T12:27:11","modified_gmt":"2015-12-23T12:27:11","slug":"elasticsearch-issues-upgrading-from-ubuntu-15-04-15-10","status":"publish","type":"post","link":"https:\/\/blog.lexxweb.co.uk\/?p=84","title":{"rendered":"ElasticSearch issues upgrading from Ubuntu 15.04 -> 15.10"},"content":{"rendered":"

For some bizarre reason when we did an upgrade from ubuntu 15.04 to 15.10 and elasticsearch was updated from v1.4.5 to v1.6.2 elasticsearch would not start (although systemctl thought it had each time).<\/p>\n

Immediately after the upgrade, we received error messages from our application saying it could not connect to elasticsearch.<\/p>\n

Sure enough trying to connect to the elasticsearch service failed when we attempted to use curl -XGET 'localhost:9200' <\/code><\/p>\n

Another strange thing is that systemctl says that elasticsearch is running perfectly, even though the process is not running when we execute ps -aux | grep elastic …<\/p>\n

It turns out the elasticsearch.pid file had been deleted in the upgrade and the elasticsearch user could not create a new one.
\n
\ntouch \/var\/run\/elasticsearch.pid
\nchown elasticsearch \/var\/run\/elasticsearch.pid
\n<\/code><\/p>\n

However, that is not all, I started receiving a new and exciting error message (see stack trace below).
\nThis one turned out to be a failed plugin called marvel. In order to fix this error all you have to do is the following:
\n
\ncd \/usr\/share\/elasticsearch
\nsudo bin\/plugin --remove elasticsearch\/marvel\/latest
\nsudo bin\/plugin -i elasticsearch\/marvel\/latest
\n<\/code><\/p>\n

There are a few items to configure before the newly upgraded elasticsearch will work:
\nsudo vim \/etc\/elasticsearch\/elasticsearch.yml <\/code>
\nEdit the following lines:
\n
\nnetwork.host: localhost
\nnetwork.bind_host: 0.0.0.0
\n<\/code><\/p>\n

Finally, we have to fix a broken init.d script:
\n
\nsudo vim \/etc\/init.d\/elasticsearch
\n<\/code>
\nComment out this line
\n
\ntest \"$START_DAEMON\" = true || exit 0
\n<\/code><\/p>\n

Then you need to run the following:
\n
\nsudo systemctl daemon-reload
\nsudo service stop elasticsearch
\nsudo service start elasticsearch
\n<\/code><\/p>\n

After this restart elasticsearch came back to life. \ud83d\ude42<\/p>\n


\n
\n[2015-11-11 11:44:39,433][ERROR][bootstrap ] Exception
\norg.elasticsearch.common.util.concurrent.ExecutionError: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg\/elasticsearch\/indices\/InternalIndicesService;
\n at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
\n at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
\n at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
\n at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
\n at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
\n at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
\n at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
\n at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
\n at org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
\n at org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
\n at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
\n at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
\n at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
\n at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
\n at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
\n at org.elasticsearch.node.internal.InternalNode.(InternalNode.java:212)
\n at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
\n at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:77)
\n at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:245)
\n at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
\nCaused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg\/elasticsearch\/indices\/InternalIndicesService;
\n at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
\n at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
\n at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
\n at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
\n at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
\n at org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:68)
\n at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:67)
\n at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
\n at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
\n at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
\n at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
\n at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
\n at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
\n at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
\n at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
\n... 19 more
\nCaused by: java.lang.NoClassDefFoundError: Lorg\/elasticsearch\/indices\/InternalIndicesService;
\n at java.lang.Class.getDeclaredFields0(Native Method)
\n at java.lang.Class.privateGetDeclaredFields(Class.java:2509)
\n at java.lang.Class.getDeclaredFields(Class.java:1819)
\n at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:378)
\n at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:376)
\n at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectorsForMembers(InjectionPoint.java:351)
\n at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectionPoints(InjectionPoint.java:345)
\n at org.elasticsearch.common.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:287)
\n at org.elasticsearch.common.inject.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:80)
\n at org.elasticsearch.common.inject.MembersInjectorStore.access$000(MembersInjectorStore.java:36)
\n at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
\n at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:41)
\n at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
\n at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
\n at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
\n at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
\n at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
\n ... 33 more
\nCaused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService
\n at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
\n at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
\n at java.security.AccessController.doPrivileged(Native Method)
\n at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
\n at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
\n at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
\n at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
\n ... 50 more
\n<\/init><\/code>
\n<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

For some bizarre reason when we did an upgrade from ubuntu 15.04 to 15.10 and elasticsearch was updated from v1.4.5 to v1.6.2 elasticsearch would not start (although systemctl thought it had each time). Immediately after the upgrade, we received error messages from our application saying it could not connect to elasticsearch. Sure enough trying to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[3,7,21,4],"tags":[],"class_list":["post-84","post","type-post","status-publish","format-standard","hentry","category-general","category-java","category-operating-systems","category-programming"],"_links":{"self":[{"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/84","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=84"}],"version-history":[{"count":7,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/84\/revisions"}],"predecessor-version":[{"id":93,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=\/wp\/v2\/posts\/84\/revisions\/93"}],"wp:attachment":[{"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=84"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=84"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lexxweb.co.uk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=84"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}