Problems while creating data dump


#1

Enonic version: 6.13.0
OS: Linux

Hi!

We are trying to create the full data dump via toolbox cli, but it returns us an error. Dump creates successfully via data toolbox.
This is the command we use to create a dump:

./toolbox.sh dump -asu:password -hlocalhost -p8082 --max-version-age=15 -t name

We assume it might happen because of the --max-version-age=15
However, using --max-version-age is highly preferred, because we have lots of objects with lots of versions and it multiplies the size of dump, it’s creation and importing time.

As far as we found, ./toolbox.sh upgrade can fix this issue, but it will also bring us a big downtime on server, which is unwelcome.

Is there any way to fix this?

This is the full stack trace of an error:

10:34:09.846 ERROR c.e.x.j.i.e.JsonExceptionMapper - NullPointerException
java.lang.NullPointerException: null
	at com.enonic.xp.repo.impl.index.IndexFieldNameNormalizer.doNormalize(IndexFieldNameNormalizer.java:18)
	at com.enonic.xp.repo.impl.index.IndexFieldNameNormalizer.normalize(IndexFieldNameNormalizer.java:11)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.StoreQueryFieldNameResolver.appendIndexValueType(StoreQueryFieldNameResolver.java:25)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.AbstractQueryFieldNameResolver.createValueTypeAwareFieldName(AbstractQueryFieldNameResolver.java:75)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.AbstractQueryFieldNameResolver.resolve(AbstractQueryFieldNameResolver.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.StoreQueryFieldNameResolver.resolve(StoreQueryFieldNameResolver.java:10)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.createRangeFilter(FilterBuilderFactory.java:170)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.doCreateFilterBuilder(FilterBuilderFactory.java:91)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.appendFilters(FilterBuilderFactory.java:71)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.doCreate(FilterBuilderFactory.java:53)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.create(FilterBuilderFactory.java:46)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.QueryBuilderFactory.create(QueryBuilderFactory.java:50)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.NodeVersionQueryTranslator.createQueryBuilder(NodeVersionQueryTranslator.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.ESQueryTranslator.doTranslate(ESQueryTranslator.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.ESQueryTranslator.translate(ESQueryTranslator.java:37)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.execute(SearchExecutor.java:35)
	at com.enonic.xp.repo.impl.elasticsearch.search.SearchDaoImpl.search(SearchDaoImpl.java:23)
	at com.enonic.xp.repo.impl.search.NodeSearchServiceImpl.query(NodeSearchServiceImpl.java:73)
	at com.enonic.xp.repo.impl.node.FindNodeVersionsCommand.execute(FindNodeVersionsCommand.java:27)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.findVersions(NodeServiceImpl.java:589)
	at com.enonic.xp.repo.impl.dump.RepoDumper.getVersions(RepoDumper.java:319)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$dumpVersions$2(RepoDumper.java:170)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1548)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
	at com.enonic.xp.repo.impl.dump.RepoDumper.dumpVersions(RepoDumper.java:166)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$execute$1(RepoDumper.java:94)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:85)
	at com.enonic.xp.repo.impl.dump.RepoDumper.execute(RepoDumper.java:94)
	at com.enonic.xp.repo.impl.dump.DumpServiceImpl.dump(DumpServiceImpl.java:107)
	at com.enonic.xp.impl.server.rest.SystemResource.systemDump(SystemResource.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	at com.enonic.xp.jaxrs.impl.JaxRsServlet.service(JaxRsServlet.java:46)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.enonic.xp.web.impl.dispatch.mapping.ServletDefinitionImpl.service(ServletDefinitionImpl.java:39)
	at com.enonic.xp.web.impl.dispatch.pipeline.ServletPipelineImpl.service(ServletPipelineImpl.java:30)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:45)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.portal.impl.auth.AuthFilter.doHandle(AuthFilter.java:45)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.auth.BasicAuthFilter.doHandle(BasicAuthFilter.java:33)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.context.ContextFilter.lambda$doHandle$0(ContextFilter.java:33)
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:101)
	at com.enonic.xp.web.impl.context.ContextFilter.doHandle(ContextFilter.java:32)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:26)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dos.DosFilterWrapper.doFilter(DosFilterWrapper.java:64)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterPipelineImpl.filter(FilterPipelineImpl.java:29)
	at com.enonic.xp.web.impl.dispatch.DispatchServletImpl.service(DispatchServletImpl.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:199)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:74)
	at com.enonic.xp.web.vhost.impl.VirtualHostFilter.doHandle(VirtualHostFilter.java:50)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dos.DosFilterWrapper.doFilter(DosFilterWrapper.java:64)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterPipelineImpl.filter(FilterPipelineImpl.java:29)
	at com.enonic.xp.web.impl.dispatch.DispatchServletImpl.service(DispatchServletImpl.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

#2

Also, found a possible bug when creating dump and using --max-versions. For example, if we use --max-versions 2, and there is an image in the fifth version, the image cannot be found. Here is the error itself:
Response error: {

“status” : 400,
“message” : “Cannot get blob with blobKey: 7e10f67f5e8682ff32c7f188bc402ff0668caaa8: blob is null”,
“context” : {
“authenticated” : true,
“principals” : [ “role:system.admin”, “role:system.admin.login”, “role:system.authenticated”, “role:system.everyone”, “user:system:su” ]
}
}


#3

Are you saying a version that was included in the dump contained a binaryReference but that the binary was not included in the dump?


#4

Hi!

Unfortunately, I am not able to answer this question :slight_smile:
It is a dump, which contains ~25 000 - 30 000 objects and each of them has lots of versions, and I don’t know which object this error is related to.

For the second time we tried to export the same dump to the same instance and it went well without errors. Strange.

Is there any way I can find some more info?

Thanks!


#5

Hi!
I have got one more issue with XP 7.2.0. Versions are ignored. Please, advice, if there is an option to fix this.

Error executing task [f5e2bf00-d291-4513-9abb-e8044e37f885] 'dump': Search request failed after [30s], query: [{
  "from" : 500000,
  "size" : 5000,
  "query" : {
    "match_all" : { }
  },
  "explain" : false,
  "fields" : [ "commitid", "message", "committer", "timestamp" ],
  "highlight" : {
    "require_field_match" : false,
    "type" : "plain"
  }
}]
com.enonic.xp.repository.IndexException: Search request failed after [30s], query: [{
  "from" : 500000,
  "size" : 5000,
  "query" : {
    "match_all" : { }
  },
  "explain" : false,
  "fields" : [ "commitid", "message", "committer", "timestamp" ],
  "highlight" : {
    "require_field_match" : false,
    "type" : "plain"
  }
}]
	at com.enonic.xp.repo.impl.elasticsearch.executor.AbstractExecutor.doSearchRequest(AbstractExecutor.java:66)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.doSearch(SearchExecutor.java:77)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.execute(SearchExecutor.java:63)
	at com.enonic.xp.repo.impl.elasticsearch.search.SearchDaoImpl.search(SearchDaoImpl.java:23)
	at com.enonic.xp.repo.impl.search.NodeSearchServiceImpl.query(NodeSearchServiceImpl.java:93)
	at com.enonic.xp.repo.impl.node.FindNodeCommitsCommand.execute(FindNodeCommitsCommand.java:27)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.findCommits(NodeServiceImpl.java:708)
	at com.enonic.xp.repo.impl.node.executor.BatchedGetCommitsExecutor.execute(BatchedGetCommitsExecutor.java:42)
	at com.enonic.xp.repo.impl.dump.RepoDumper.dumpCommits(RepoDumper.java:208)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$execute$2(RepoDumper.java:100)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:86)
	at com.enonic.xp.repo.impl.dump.RepoDumper.execute(RepoDumper.java:100)
	at com.enonic.xp.repo.impl.dump.DumpServiceImpl.dump(DumpServiceImpl.java:231)
	at com.enonic.xp.impl.server.rest.task.DumpRunnableTask.run(DumpRunnableTask.java:44)
	at com.enonic.xp.impl.task.TaskWrapper.lambda$callTaskWithContext$0(TaskWrapper.java:90)
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:102)
	at com.enonic.xp.impl.task.TaskWrapper.callTaskWithContext(TaskWrapper.java:89)
	at com.enonic.xp.impl.task.TaskWrapper.doRun(TaskWrapper.java:75)
	at com.enonic.xp.impl.task.TaskWrapper.run(TaskWrapper.java:59)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:874)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:852)
	at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:389)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	... 3 common frames omitted
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [500000] but was [505000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.
	at org.elasticsearch.search.internal.DefaultSearchContext.preProcess(DefaultSearchContext.java:212)
	at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:103)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:689)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
	at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:472)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:392)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:389)
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	... 3 common frames omitted

#6

It seems to me like you’re describing a bug that was fixed in XP 7.2.2?

Source:


#7

Thanks! I will try upgrading to it and do a dump again.


#8

NB! Go directly to 7.2.4 as there are also more fixes available


#9

We have updated XP to 7.2.4 and the error is gone. Issue solved. Thanks!


#10