Upgrading cluster from 6.8 to 6.9.2 or 6.10.2 causes: Repository with id [system-repo] not found

ksawery · May 24, 2017, 10:27am

Hi we tried to upgrade our cluster based on two docker containers from 6.8.0 to 6.9.2 or 6.10.2 and unfortunately we get an error: Repository with id [system-repo] not found

What might be wrong? The cluster consists of two nodes in docker containers from enonic repo.
We also tried upgrading from 6.8.0 to 6.8.1 and it worked without any problems.
Here is the log output:

15:22:50.383 INFO  c.e.x.e.impl.ClientActivator - Cluster operational, register elasticsearch-client
15:22:50.699 INFO  c.e.x.c.i.app.ApplicationServiceImpl - Searching for installed applications
15:22:50.723 ERROR com.enonic.xp.core.app - [com.enonic.xp.core.impl.app.ApplicationServiceImpl(23)] The activate method has thrown an exception
com.enonic.xp.repository.RepositoryNotFoundException: Repository with id [system-repo] not found
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.lambda$verifyBranchExists$3(NodeServiceImpl.java:783) ~[na:na]
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:85) ~[na:na]
	at com.enonic.xp.repo.impl.node.NodeHelper.runAsAdmin(NodeHelper.java:15) ~[na:na]
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.verifyBranchExists(NodeServiceImpl.java:777) ~[na:na]
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.verifyContext(NodeServiceImpl.java:772) ~[na:na]
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.findByParent(NodeServiceImpl.java:175) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationRepoServiceImpl.lambda$getApplications$0(ApplicationRepoServiceImpl.java:90) ~[na:na]
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:101) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationHelper.runAsAdmin(ApplicationHelper.java:122) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationRepoServiceImpl.getApplications(ApplicationRepoServiceImpl.java:90) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationServiceImpl.installAllStoredApplications(ApplicationServiceImpl.java:196) ~[na:na]
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:85) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationHelper.runAsAdmin(ApplicationHelper.java:130) ~[na:na]
	at com.enonic.xp.core.impl.app.ApplicationServiceImpl.activate(ApplicationServiceImpl.java:62) ~[na:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
	at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:222) ~[org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:37) ~[org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:615) ~[org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:499) ~[org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:295) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:302) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:113) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:866) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:833) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:774) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:347) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:247) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:343) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.getService(Felix.java:3692) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:470) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:72) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.helper.BindMethod.getServiceObject(BindMethod.java:646) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2137) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1048) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1457) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:983) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:812) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:724) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:954) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1215) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1136) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:945) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:881) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1167) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:120) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:987) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:838) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:545) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4560) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.registerService(Felix.java:3542) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:348) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:869) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:857) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:915) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:715) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:399) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:676) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:339) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:360) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.Activator.access$000(Activator.java:53) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:260) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.utils.extender.AbstractExtender.createExtension(AbstractExtender.java:259) [org.apache.felix.scr-2.0.2.jar:na]
	at org.apache.felix.utils.extender.AbstractExtender.modifiedBundle(AbstractExtender.java:232) [org.apache.felix.scr-2.0.2.jar:na]
	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:482) [org.apache.felix.framework-5.2.0.jar:na]
	at org.osgi.util.tracker.BundleTracker$Tracked.customizerModified(BundleTracker.java:415) [org.apache.felix.framework-5.2.0.jar:na]
	at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:232) [org.apache.felix.framework-5.2.0.jar:na]
	at org.osgi.util.tracker.BundleTracker$Tracked.bundleChanged(BundleTracker.java:444) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:913) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:834) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:516) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4544) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.startBundle(Felix.java:2166) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1365) [org.apache.felix.framework-5.2.0.jar:na]
	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) [org.apache.felix.framework-5.2.0.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
15:22:50.724 ERROR com.enonic.xp.core.app - [com.enonic.xp.core.impl.app.ApplicationServiceImpl(23)] Failed creating the component instance; see log for reason
15:22:50.724 ERROR com.enonic.xp.core.app - [com.enonic.xp.core.impl.app.ApplicationDeployDirectoryWatcher(20)] Cannot create component instance due to failure to bind reference ApplicationService
15:22:50.724 ERROR com.enonic.xp.core.app - [com.enonic.xp.core.impl.app.ApplicationDeployDirectoryWatcher(20)] Failed creating the component instance; see log for reason
15:22:50.726 INFO  c.e.x.c.i.app.ApplicationServiceImpl - Searching for installed applications
15:22:50.742 ERROR com.enonic.xp.core.app - [com.enonic.xp.core.impl.app.ApplicationServiceImpl(23)] The activate method has thrown an exception
com.enonic.xp.repository.RepositoryNotFoundException: Repository with id [system-repo] not found
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.lambda$verifyBranchExists$3(NodeServiceImpl.java:783) ~[na:na]

tsi · May 27, 2017, 11:49am

Essentially it claims the system repo is not found? Has to be something wrong in your setup.

Also, running a 2-node cluster of XP is not the recommended approach, as this could cause split-brain problems. 3 nodes are ideal. Please log a support issue if you have a subscription for this installation.

gri · May 28, 2017, 7:34pm

My guess is that during the update you have two different versions of Enonic XP in the same cluster (a 6.8 and a 6.9.2 or 6.10.2). You should not have 2 different versions in a same cluster. They should be 2 different clusters. (A good practice could be to include the version number in the cluster name).

A first solution (recommended solution) for the update would be to stop all the old installations and then start all the new installations.

A second solution (not recommended since, if your installation is not in read-only and modifications happen, the data will be lost) would be to
1- Stop one node, upgrade it to 6.9.2 and launch it in a second cluster.
2 - Then for the second node: stop it, upgrade it to 6.9.2 and launch it in this second cluster.

Again as tsi said you should not have a 2-nodes cluster of XP. We recommend 3 nodes.

ksawery · May 29, 2017, 9:29am

Thanks, we will try that.
But was there any change in the Docker container config itself between 6.8 and 6.9? We have compared the docker inspect output of 6.8 and 6.9 and all the disk mappings were identical, so both containers should have access to the repo.
We could upgrade without problems from 6.8.0 to 6.8.1 but it fails even if we just upgrade one of the nodes or both to 6.9 / 6.10

gri · May 29, 2017, 8:16pm

To give more information about why it fails with 6.9 and the error message:

With Enonic XP 6.9, there has been the release of the Node and Repository API.
It required to add metadata about the repositories.

In a normal case, the 6.9 node would have simply generated this meta-data at startup the first time.
But in your case it shared the data of the 6.8 node which lacks this metadata. That is why you get the error “Repository with id [system-repo] not found”

rfo · May 30, 2017, 9:24am

Hi @gri. It is in this case important to not be offline during upgrades, so offline upgrade (suggestion 1) is not an option. Yes, using three nodes in a cluster is recommended, but I don’t see how it will affect this scenario with upgrades other then make it even more complicated in the case of a 3-node cluster where one would have minimum_master_nodes = 2?

I understand suggestion 2 is to change the cluster.name in upgrades between versions (index/data/mycluster-6.8, index/data/mycluster-6.10 etc). So in a two node cluster we would:

Redirect all traffic to node 1.
Take down node 2, upgrade it, change cluster.name to reflect the new version.
Spin up node 2 and redirect all traffic to it.
(now we must assume some content have been created / updated on node 1 index and in the shared blobs)
Take down node 1, upgrade it, change cluster.name to reflect the new version.
Spin up node 1 and set the loadbalancer to send traffic to both nodes.

I have some questions in regard to this.

Is this strategy with changing cluster.name only recommended between minor (6.8.0 -> 6.10.3) and major (6.10.3 -> 7.0.0) versions and not bugfix (6.9.0 -> 6.9.3) versions?
What about shared blobs. Can / is it recommended that two versions point to the same shared blob folder during upgrade?
Given 2. is ok. What about content created/updated during downtime of one node in the cluster. Let’s say a new user gets created in system-repo and updates his profile and uploads profile photo - due to first time login with SSO into the system, or someone fills in a contact form during upgrade, and this contact-form get stored in blobstore, but only updates the _versionkey (that point to the new blob key) in the mycluster-6.8 index; so when both nodes are up and pointing to the mycluster-6.10 - this data is lost. What would be the best way to handle this?

In this specific case there is today only non critical updates to content frequently, but in the near future there will be more critical content created from customers - so then it will not be a solution to risk losing content that gets created during upgrades.

tsi · May 30, 2017, 10:06am

WARNING! Upgrading XP an XP cluster between minor versions without going offline is not supported unless explicitly stated in upgrade instructions. Upgrading a live cluster with fix releases should normally work fine.

The reason for this is the distributed nature of XP. If there are changes in the core data engine, running two different versions of XP against the same dataset and cluster might cause problems that could potentially cause a whole lot of downtime.

That said, an XP minor version upgrade can often be done in less than a minute so downtime would be minimal. The only feasible way to do upgrade without downtime is by booting a completely new installation and re-route traffic, but this might naturally cause data-loss.