Cluster health in state 'RED'

Enonic version: 6.4.0
OS: Ubuntu 15.10

Any suggestion on how to solve this error ?

19:09:32.088 INFO  org.elasticsearch.node - [local-node] initialized
19:09:32.089 INFO  org.elasticsearch.node - [local-node] starting ...
19:09:32.137 INFO  org.elasticsearch.transport - [local-node] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
19:09:32.145 INFO  org.elasticsearch.discovery - [local-node] mycluster/Iqwj9DYZRDKFMSqmIdYGtA
19:09:35.160 INFO  org.elasticsearch.cluster.service - [local-node] new_master [local-node][Iqwj9DYZRDKFMSqmIdYGtA][localhost][inet[/127.0.0.1:9300]]{master=true}, reason: zen-disco-join (elected_as_master)
19:09:35.199 INFO  org.elasticsearch.node - [local-node] started
19:09:35.969 WARN  org.elasticsearch.indices.cluster - [local-node] [[storage-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [storage-cms-repo][0] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162) [elasticsearch-1.5.2_1.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [storage-cms-repo][0] failed to upgrade 3x segments
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784) ~[na:na]
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226) ~[na:na]
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) [elasticsearch-1.5.2_1.jar:na]
        ... 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[na:na]
        at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119) ~[na:na]
        ... 9 common frames omitted
19:09:35.972 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[storage-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[storage-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")]; ]]
19:09:35.989 INFO  org.elasticsearch.gateway - [local-node] recovered [4] indices into cluster_state
19:09:35.996 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [master [local-node][Iqwj9DYZRDKFMSqmIdYGtA][localhost][inet[/127.0.0.1:9300]]{master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
19:09:41.010 ERROR c.e.x.e.impl.ClientActivator - Cluster health in state 'RED' 
19:09:41.369 INFO  org.eclipse.jetty.util.log - Logging initialized @12785ms
19:09:41.435 INFO  org.eclipse.jetty.server.Server - jetty-9.3.5.v20151012
19:09:41.491 INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@51be0347{/,null,AVAILABLE}
19:09:41.496 INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@dcaefe6{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
19:09:41.499 INFO  org.eclipse.jetty.server.Server - Started @12915ms
19:09:41.514 INFO  c.e.x.w.v.i.c.VirtualHostConfigImpl - Virtual host is enabled and mappings updated.
19:09:41.949 INFO  c.e.x.l.i.framework.FrameworkService - Started Enonic XP in 12943 ms
19:09:45.187 WARN  org.elasticsearch.indices.cluster - [local-node] [[storage-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [storage-cms-repo][0] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162) [elasticsearch-1.5.2_1.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [storage-cms-repo][0] failed to upgrade 3x segments
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257) ~[na:na]
        at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784) ~[na:na]
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226) ~[na:na]
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112) [elasticsearch-1.5.2_1.jar:na]
        ... 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")
        at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) ~[na:na]
        at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) ~[na:na]
        at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767) ~[na:na]
        at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084) ~[na:na]
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:119) ~[na:na]
        ... 9 common frames omitted
19:09:45.188 WARN  o.elasticsearch.cluster.action.shard - [local-node] [storage-cms-repo][0] received shard failed for [storage-cms-repo][0], node[Iqwj9DYZRDKFMSqmIdYGtA], [P], s[INITIALIZING], indexUUID [40Rd3jAqRXqByU6y-djm8g], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[storage-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[storage-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/var/www/dev1.vetinst.no/enonic-xp-6.4.0/home/repo/index/data/mycluster/nodes/0/indices/storage-cms-repo/0/index/segments_46")]; ]]
19:09:47.013 ERROR c.e.x.e.impl.ClientActivator - Cluster health in state 'RED'

Hi.

Have you by any chance started another node on the same machine? If so; try starting that node again and see if it is able to do the recovery. Then if this is successful, do a dump of your data, delete/move the repo-folder and load the data again.

Hi rmy,

There is a chance that it might have happened here. I tried to do a recovery of the data as you said but no success. What I did was a clean installation and used the import function to import the data. This solved the problem.

Thank you for your suggestion!

Good stuff. We are disabling the localhost auto cluster detection in next version, trouble bound to happen.

1 Like