Cluster health in state 'RED'

Help
1

Enonic version: 6.4.0
OS: Ubuntu 15.10

Any suggestion on how to solve this error ?

19:09:32.088 INFO  org.elasticsearch.node - [local-node] initialized
19:09:32.089 INFO  org.elasticsearch.node - [local-node] starting ...
19:09:32.137 INFO  org.elasticsearch.transport - [local-node] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
19:09:32.145 INFO  org.elasticsearch.discovery - [local-node] mycluster/Iqwj9DYZRDKFMSqmIdYGtA
19:09:35.160 INFO  org.elasticsearch.cluster.service - [local-node] new_master [local-node][Iqwj9DYZRDKFMSqmIdYGtA][localhost][inet[/127.0.0.1:9300]]{master=true}, reason: zen-disco-join (elected_as_master)
19:09:35.199 INFO  org.elasticsearch.node - [local-node] started
19:09:35.969 WARN  org.elasticsearch.indices.cluster - [local-node] [[storage-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [storage-cms-repo][0] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162) [elasticsearch-1.5.2_1.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [storage-cms-repo][0] failed to upgrade 3x segments
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784) ~[na:na]
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226) ~[na:na]
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) [elasticsearch-1.5.2_1.jar:na]
        ... 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[na:na]
        at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119) ~[na:na]
        ... 9 common frames omitted
19:09:35.972 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[storage-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[storage-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")]; ]]
19:09:35.989 INFO  org.elasticsearch.gateway - [local-node] recovered [4] indices into cluster_state
19:09:35.996 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [master [local-node][Iqwj9DYZRDKFMSqmIdYGtA][localhost][inet[/127.0.0.1:9300]]{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
19:09:41.010 ERROR c.e.x.e.impl.ClientActivator - Cluster health in state 'RED' 
19:09:41.369 INFO  org.eclipse.jetty.util.log - Logging initialized @12785ms
19:09:41.435 INFO  org.eclipse.jetty.server.Server - jetty-9.3.5.v20151012
19:09:41.491 INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@51be0347{/,null,AVAILABLE}
19:09:41.496 INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@dcaefe6{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
19:09:41.499 INFO  org.eclipse.jetty.server.Server - Started @12915ms
19:09:41.514 INFO  c.e.x.w.v.i.c.VirtualHostConfigImpl - Virtual host is enabled and mappings updated.
19:09:41.949 INFO  c.e.x.l.i.framework.FrameworkService - Started Enonic XP in 12943 ms
19:09:45.187 WARN  org.elasticsearch.indices.cluster - [local-node] [[storage-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [storage-cms-repo][0] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162) [elasticsearch-1.5.2_1.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [storage-cms-repo][0] failed to upgrade 3x segments
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784) ~[na:na]
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226) ~[na:na]
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) [elasticsearch-1.5.2_1.jar:na]
        ... 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[na:na]
        at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119) ~[na:na]
        ... 9 common frames omitted
19:09:45.188 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[storage-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[storage-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")]; ]]
19:09:47.013 ERROR c.e.x.e.impl.ClientActivator - Cluster health in state 'RED'
2

Hi.

Have you by any chance started another node on the same machine? If so; try starting that node again and see if it is able to do the recovery. Then if this is successful, do a dump of your data, delete/move the repo-folder and load the data again.

3

Hi rmy,

There is a chance that it might have happened here. I tried to do a recovery of the data as you said but no success. What I did was a clean installation and used the import function to import the data. This solved the problem.

Thank you for your suggestion!

4

Good stuff. We are disabling the localhost auto cluster detection in next version, trouble bound to happen.

1 Like