Enonic version: XP 7.10.3
OS: macOS
** Cluster with 3 master and 2 data nodes**
XP log shows an error message that “it couldn’t find master branch”. Initially, there was only one data node that failed to start but after full cluster restart all cluster nodes are experiencing the same issue and the cluster fails to start.
2022-10-05 09:07:20,155 ERROR c.e.x.c.i.p.ProjectServiceActivator - bundle com.enonic.xp.core.project:7.10.3 (66)[com.enonic.xp.core.impl.project.ProjectServiceActivator(98)] : The activate method has thrown an exception
java.lang.IllegalArgumentException: branches must contain master branch.
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
at com.enonic.xp.repository.Repository$Builder.validate(Repository.java:157)
at com.enonic.xp.repository.Repository$Builder.build(Repository.java:163)
at com.enonic.xp.repo.impl.repository.RepositoryNodeTranslator.toRepository(RepositoryNodeTranslator.java:142)
at com.enonic.xp.repo.impl.repository.RepositoryEntryServiceImpl.getRepositoryEntry(RepositoryEntryServiceImpl.java:99)
at com.enonic.xp.repo.impl.repository.RepositoryServiceImpl.lambda$doGet$3(RepositoryServiceImpl.java:220)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at com.enonic.xp.repo.impl.repository.RepositoryServiceImpl.doGet(RepositoryServiceImpl.java:220)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at com.enonic.xp.repo.impl.repository.RepositoryServiceImpl.list(RepositoryServiceImpl.java:199)
at com.enonic.xp.core.impl.project.ProjectServiceImpl.doList(ProjectServiceImpl.java:351)
at com.enonic.xp.core.impl.project.ProjectServiceImpl.lambda$initialize$2(ProjectServiceImpl.java:101)
at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:84)
at com.enonic.xp.core.impl.project.ProjectServiceImpl.initialize(ProjectServiceImpl.java:99)
at com.enonic.xp.core.impl.project.ProjectServiceActivator.activate(ProjectServiceActivator.java:54)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:245)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:687)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:531)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:317)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:307)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:354)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:1002)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:975)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:785)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:335)
at com.enonic.xp.repo.impl.repository.RepositoryServiceActivator.activate(RepositoryServiceActivator.java:52)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:245)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:687)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:531)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:317)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:307)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:354)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:1002)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:975)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:785)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:335)
at com.enonic.xp.elasticsearch.impl.ElasticsearchActivator.activate(ElasticsearchActivator.java:73)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invokeMethod(BaseMethod.java:245)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.access$500(BaseMethod.java:41)
at org.apache.felix.scr.impl.inject.methods.BaseMethod$Resolved.invoke(BaseMethod.java:687)
at org.apache.felix.scr.impl.inject.methods.BaseMethod.invoke(BaseMethod.java:531)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:317)
at org.apache.felix.scr.impl.inject.methods.ActivateMethod.invoke(ActivateMethod.java:307)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:354)
at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:1002)
at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:975)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:785)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1274)
at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1225)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1232)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1152)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:959)
at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:895)
at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1184)
at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:116)
at org.apache.felix.framework.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:990)
at org.apache.felix.framework.EventDispatcher.fireEventImmediately(EventDispatcher.java:838)
at org.apache.felix.framework.EventDispatcher.fireServiceEvent(EventDispatcher.java:545)
at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4863)
at org.apache.felix.framework.Felix.registerService(Felix.java:3834)
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:328)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915)
at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752)
at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674)
at org.apache.felix.scr.impl.manager.AbstractComponentManager$1.run(AbstractComponentManager.java:461)
at org.apache.felix.scr.impl.ComponentActorThread.run(ComponentActorThread.java:114)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-10-05 09:07:20,249 INFO c.e.x.c.impl.ClusterManagerImpl - Adding cluster-provider: elasticsearch
2022-10-05 09:07:20,326 INFO org.eclipse.jetty.util.log - Logging initialized @438046ms to org.eclipse.jetty.util.log.Slf4jLog
2022-10-05 09:07:21,026 INFO org.eclipse.jetty.server.Server - jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.15+10-jvmci-22.1-b06
2022-10-05 09:07:21,093 INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=
2022-10-05 09:07:21,094 INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2022-10-05 09:07:21,097 INFO org.eclipse.jetty.server.session - Scavenging every 660000ms
2022-10-05 09:07:21,117 INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@4372edbc{/,null,AVAILABLE,@xp}
2022-10-05 09:07:21,121 INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@31f7ae2d{/,null,AVAILABLE,@api}
2022-10-05 09:07:21,122 INFO o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7e1a856b{/,null,AVAILABLE,@status}
2022-10-05 09:07:21,140 INFO o.e.jetty.server.AbstractConnector - Started xp@3eb4c68{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2022-10-05 09:07:21,142 INFO o.e.jetty.server.AbstractConnector - Started api@3bce20e{HTTP/1.1, (http/1.1)}{0.0.0.0:4848}
2022-10-05 09:07:21,144 INFO o.e.jetty.server.AbstractConnector - Started status@64eea44e{HTTP/1.1, (http/1.1)}{0.0.0.0:2609}
2022-10-05 09:07:21,144 INFO org.eclipse.jetty.server.Server - Started @438865ms
2022-10-05 09:07:21,299 INFO c.e.xp.web.jetty.impl.JettyActivator - Started Jetty
2022-10-05 09:07:21,304 INFO c.e.xp.web.jetty.impl.JettyActivator - Listening on ports [8080](xp), [4848](management) and [2609](monitoring)
2022-10-05 09:09:28,664 INFO c.hazelcast.nio.tcp.TcpIpConnection - [...]:5701 [3.12.12#OSS] [3.12.12] Initialized new cluster connection between /172.17.0.3:5701 and /...:55789
2022-10-05 09:09:35,660 INFO c.h.internal.cluster.ClusterService - [...]:5701 [3.12.12#OSS] [3.12.12]
Members {size:5, ver:5} [
.
.
.
.
]