Hi !
I’m having troubles with local copy. Without any reason after starting server I got “Cluster health in state ‘RED’” and I can’t do anything to make it work. I get this issue every month, I know that I can stop server, cleanup repo folder and import everything again. Problem is that full dump is very big and load process could take a lot of time ( up to 16h ).
Maybe it’s possible to “restore” data somehow ?
Thanks !
What does the log say during startup?
15:29:57.614 INFO c.e.x.l.i.framework.FrameworkService - Starting Enonic XP…
15:29:57.843 INFO c.e.x.l.i.p.ProvisionActivator - Installing 74 bundles…
15:29:58.802 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.dos]
15:29:58.803 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.vhost]
15:29:58.806 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.jetty]
15:29:58.807 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.server.shell]
15:29:58.811 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.repo]
15:29:58.812 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.mail]
15:29:58.836 INFO c.e.x.s.shell.impl.ShellActivator - Remote shell access is disabled
15:29:58.837 INFO c.e.x.s.shell.impl.ShellActivator - Remote shell access is disabled
15:29:58.962 INFO c.e.x.s.internal.trace.TraceService - Call tracing is disabled
15:29:59.024 INFO c.e.x.i.blobstore.BlobStoreActivator - Waiting for blobstore-provider [file]
15:29:59.025 INFO c.e.x.i.blobstore.BlobStoreActivator - Found blobstore-provider [file]
15:29:59.038 INFO c.e.x.i.blobstore.BlobStoreActivator - Registered blobstore [file] successfully
15:29:59.142 INFO org.elasticsearch.node - [local-node] version[1.5.2], pid[1072], build[NA/NA]
15:29:59.142 INFO org.elasticsearch.node - [local-node] initializing …
15:29:59.145 INFO org.elasticsearch.plugins - [local-node] loaded [], sites []
15:30:00.015 INFO org.elasticsearch.node - [local-node] initialized
15:30:00.016 INFO org.elasticsearch.node - [local-node] starting …
15:30:00.017 INFO org.elasticsearch.transport - [local-node] bound_address {local[1]}, publish_address {local[1]}
15:30:00.025 INFO org.elasticsearch.discovery - [local-node] mycluster/WPfx5pkARgWzBHJiXLt28A
15:30:00.026 INFO org.elasticsearch.cluster.service - [local-node] master {new [local-node][WPfx5pkARgWzBHJiXLt28A][navyhod-laptop][local[1]]{local=true;, master=true}}, removed {[local-node][wFsBwHXfQGiqS1mhYl4XtQ][navyhod-laptop][local[1]]{local=true;, master=true},}, reason: local-disco-initial_connect(master)
15:30:00.054 INFO org.elasticsearch.node - [local-node] started
15:30:00.800 WARN org.elasticsearch.indices.cluster - [local-node] [[search-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [search-cms-repo][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [search-cms-repo][0] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
… 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/media/navyhod/sites/enonic/homes/mysite/repo/index/data/mycluster/nodes/0/indices/search-cms-repo/0/index/segments_5")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:119)
… 9 common frames omitted
15:30:00.802 WARN o.elasticsearch.cluster.action.shard - [local-node] [search-cms-repo][0] received shard failed for [search-cms-repo][0], node[WPfx5pkARgWzBHJiXLt28A], [P], s[INITIALIZING], indexUUID [Pj_Ws47VT6eXHhutufWDlA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[search-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[search-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/media/navyhod/sites/enonic/homes/mysite/repo/index/data/mycluster/nodes/0/indices/search-cms-repo/0/index/segments_5")]; ]]
15:30:00.893 INFO org.elasticsearch.gateway - [local-node] recovered [4] indices into cluster_state
15:30:00.897 WARN o.elasticsearch.cluster.action.shard - [local-node] [search-cms-repo][0] received shard failed for [search-cms-repo][0], node[WPfx5pkARgWzBHJiXLt28A], [P], s[INITIALIZING], indexUUID [Pj_Ws47VT6eXHhutufWDlA], reason [master [local-node][WPfx5pkARgWzBHJiXLt28A][navyhod-laptop][local[1]]{local=true;, master=true} marked shard as initializing, but shard is marked as failed, resend shard failure]
15:30:01.464 WARN org.elasticsearch.indices.cluster - [local-node] [[search-cms-repo][0]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [search-cms-repo][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [search-cms-repo][0] failed to upgrade 3x segments
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:121)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:32)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1262)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1257)
at org.elasticsearch.index.shard.IndexShard.prepareForTranslogRecovery(IndexShard.java:784)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:226)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
… 3 common frames omitted
Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/media/navyhod/sites/enonic/homes/mysite/repo/index/data/mycluster/nodes/0/indices/search-cms-repo/0/index/segments_5")
at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:336)
at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:183)
at org.elasticsearch.common.lucene.Lucene.indexNeeds3xUpgrading(Lucene.java:767)
at org.elasticsearch.common.lucene.Lucene.upgradeLucene3xSegmentsMetadata(Lucene.java:778)
at org.elasticsearch.index.engine.InternalEngine.upgrade3xSegments(InternalEngine.java:1084)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:119)
… 9 common frames omitted
15:30:01.465 WARN o.elasticsearch.cluster.action.shard - [local-node] [search-cms-repo][0] received shard failed for [search-cms-repo][0], node[WPfx5pkARgWzBHJiXLt28A], [P], s[INITIALIZING], indexUUID [Pj_Ws47VT6eXHhutufWDlA], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[search-cms-repo][0] failed recovery]; nested: EngineCreationFailureException[[search-cms-repo][0] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/media/navyhod/sites/enonic/homes/mysite/repo/index/data/mycluster/nodes/0/indices/search-cms-repo/0/index/segments_5")]; ]]
15:30:06.052 ERROR c.e.x.e.impl.ClientActivator - Cluster health in state ‘RED’
15:30:06.306 INFO org.eclipse.jetty.util.log - Logging initialized @8968ms
15:30:06.362 INFO org.eclipse.jetty.server.Server - jetty-9.3.5.v20151012
15:30:06.379 INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@46220ead{/,null,AVAILABLE}
15:30:06.395 INFO o.e.jetty.server.ServerConnector - Started ServerConnector@36241938{HTTP/1.1,[http/1.1]}{0.0.0.0:8082}
15:30:06.396 INFO org.eclipse.jetty.server.Server - Started @9057ms
Ok, think this is due to a bug in the used elasticsearch-version.
Could you check this file:
/media/navyhod/sites/enonic/homes/mysite/repo/index/data/mycluster/nodes/0/indices/search-cms-repo/0/index/segments_5
Is it empty?
Yes, it’s empty. It’s always empty in such cases 
Ok, does it work if you delete it?
Thank you ! Now it works. Magic… I tried to do this a lot of times earlier but without any result. Maybe something changed in 6.11.1 version 
Thank you !
Perfect.
Hmm, should be the same since we use the same version of ES.
Anyway, we know about this issue, and will try to do the upgrade into a newer 1.x ES-version as soon as possible.