Problems while creating data dump

Help
1

Enonic version: 6.13.0
OS: Linux

Hi!

We are trying to create the full data dump via toolbox cli, but it returns us an error. Dump creates successfully via data toolbox.
This is the command we use to create a dump:

./toolbox.sh dump -asu:password -hlocalhost -p8082 --max-version-age=15 -t name

We assume it might happen because of the --max-version-age=15
However, using --max-version-age is highly preferred, because we have lots of objects with lots of versions and it multiplies the size of dump, it’s creation and importing time.

As far as we found, ./toolbox.sh upgrade can fix this issue, but it will also bring us a big downtime on server, which is unwelcome.

Is there any way to fix this?

This is the full stack trace of an error:

10:34:09.846 ERROR c.e.x.j.i.e.JsonExceptionMapper - NullPointerException
java.lang.NullPointerException: null
	at com.enonic.xp.repo.impl.index.IndexFieldNameNormalizer.doNormalize(IndexFieldNameNormalizer.java:18)
	at com.enonic.xp.repo.impl.index.IndexFieldNameNormalizer.normalize(IndexFieldNameNormalizer.java:11)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.StoreQueryFieldNameResolver.appendIndexValueType(StoreQueryFieldNameResolver.java:25)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.AbstractQueryFieldNameResolver.createValueTypeAwareFieldName(AbstractQueryFieldNameResolver.java:75)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.AbstractQueryFieldNameResolver.resolve(AbstractQueryFieldNameResolver.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.resolver.StoreQueryFieldNameResolver.resolve(StoreQueryFieldNameResolver.java:10)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.createRangeFilter(FilterBuilderFactory.java:170)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.doCreateFilterBuilder(FilterBuilderFactory.java:91)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.appendFilters(FilterBuilderFactory.java:71)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.doCreate(FilterBuilderFactory.java:53)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.FilterBuilderFactory.create(FilterBuilderFactory.java:46)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.factory.QueryBuilderFactory.create(QueryBuilderFactory.java:50)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.NodeVersionQueryTranslator.createQueryBuilder(NodeVersionQueryTranslator.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.ESQueryTranslator.doTranslate(ESQueryTranslator.java:56)
	at com.enonic.xp.repo.impl.elasticsearch.query.translator.ESQueryTranslator.translate(ESQueryTranslator.java:37)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.execute(SearchExecutor.java:35)
	at com.enonic.xp.repo.impl.elasticsearch.search.SearchDaoImpl.search(SearchDaoImpl.java:23)
	at com.enonic.xp.repo.impl.search.NodeSearchServiceImpl.query(NodeSearchServiceImpl.java:73)
	at com.enonic.xp.repo.impl.node.FindNodeVersionsCommand.execute(FindNodeVersionsCommand.java:27)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.findVersions(NodeServiceImpl.java:589)
	at com.enonic.xp.repo.impl.dump.RepoDumper.getVersions(RepoDumper.java:319)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$dumpVersions$2(RepoDumper.java:170)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1548)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
	at com.enonic.xp.repo.impl.dump.RepoDumper.dumpVersions(RepoDumper.java:166)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$execute$1(RepoDumper.java:94)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:85)
	at com.enonic.xp.repo.impl.dump.RepoDumper.execute(RepoDumper.java:94)
	at com.enonic.xp.repo.impl.dump.DumpServiceImpl.dump(DumpServiceImpl.java:107)
	at com.enonic.xp.impl.server.rest.SystemResource.systemDump(SystemResource.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
	at com.enonic.xp.jaxrs.impl.JaxRsServlet.service(JaxRsServlet.java:46)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.enonic.xp.web.impl.dispatch.mapping.ServletDefinitionImpl.service(ServletDefinitionImpl.java:39)
	at com.enonic.xp.web.impl.dispatch.pipeline.ServletPipelineImpl.service(ServletPipelineImpl.java:30)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:45)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.portal.impl.auth.AuthFilter.doHandle(AuthFilter.java:45)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.auth.BasicAuthFilter.doHandle(BasicAuthFilter.java:33)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.context.ContextFilter.lambda$doHandle$0(ContextFilter.java:33)
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:101)
	at com.enonic.xp.web.impl.context.ContextFilter.doHandle(ContextFilter.java:32)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:26)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dos.DosFilterWrapper.doFilter(DosFilterWrapper.java:64)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterPipelineImpl.filter(FilterPipelineImpl.java:29)
	at com.enonic.xp.web.impl.dispatch.DispatchServletImpl.service(DispatchServletImpl.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:199)
	at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:74)
	at com.enonic.xp.web.vhost.impl.VirtualHostFilter.doHandle(VirtualHostFilter.java:50)
	at com.enonic.xp.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:31)
	at com.enonic.xp.web.filter.BaseWebFilter.doFilter(BaseWebFilter.java:33)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dos.DosFilterWrapper.doFilter(DosFilterWrapper.java:64)
	at com.enonic.xp.web.impl.dispatch.mapping.FilterDefinitionImpl.doFilter(FilterDefinitionImpl.java:40)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:50)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterChainImpl.doFilter(FilterChainImpl.java:36)
	at com.enonic.xp.web.impl.dispatch.pipeline.FilterPipelineImpl.filter(FilterPipelineImpl.java:29)
	at com.enonic.xp.web.impl.dispatch.DispatchServletImpl.service(DispatchServletImpl.java:35)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:821)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
	at org.eclipse.jetty.server.Server.handle(Server.java:517)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:306)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
2

Also, found a possible bug when creating dump and using --max-versions. For example, if we use --max-versions 2, and there is an image in the fifth version, the image cannot be found. Here is the error itself:
Response error: {

“status” : 400,
“message” : “Cannot get blob with blobKey: 7e10f67f5e8682ff32c7f188bc402ff0668caaa8: blob is null”,
“context” : {
“authenticated” : true,
“principals” : [ “role:system.admin”, “role:system.admin.login”, “role:system.authenticated”, “role:system.everyone”, “user:system:su” ]
}
}

3

Are you saying a version that was included in the dump contained a binaryReference but that the binary was not included in the dump?

4

Hi!

Unfortunately, I am not able to answer this question :slight_smile:
It is a dump, which contains ~25 000 - 30 000 objects and each of them has lots of versions, and I don’t know which object this error is related to.

For the second time we tried to export the same dump to the same instance and it went well without errors. Strange.

Is there any way I can find some more info?

Thanks!

5

Hi!
I have got one more issue with XP 7.2.0. Versions are ignored. Please, advice, if there is an option to fix this.

Error executing task [f5e2bf00-d291-4513-9abb-e8044e37f885] 'dump': Search request failed after [30s], query: [{
  "from" : 500000,
  "size" : 5000,
  "query" : {
    "match_all" : { }
  },
  "explain" : false,
  "fields" : [ "commitid", "message", "committer", "timestamp" ],
  "highlight" : {
    "require_field_match" : false,
    "type" : "plain"
  }
}]
com.enonic.xp.repository.IndexException: Search request failed after [30s], query: [{
  "from" : 500000,
  "size" : 5000,
  "query" : {
    "match_all" : { }
  },
  "explain" : false,
  "fields" : [ "commitid", "message", "committer", "timestamp" ],
  "highlight" : {
    "require_field_match" : false,
    "type" : "plain"
  }
}]
	at com.enonic.xp.repo.impl.elasticsearch.executor.AbstractExecutor.doSearchRequest(AbstractExecutor.java:66)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.doSearch(SearchExecutor.java:77)
	at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.execute(SearchExecutor.java:63)
	at com.enonic.xp.repo.impl.elasticsearch.search.SearchDaoImpl.search(SearchDaoImpl.java:23)
	at com.enonic.xp.repo.impl.search.NodeSearchServiceImpl.query(NodeSearchServiceImpl.java:93)
	at com.enonic.xp.repo.impl.node.FindNodeCommitsCommand.execute(FindNodeCommitsCommand.java:27)
	at com.enonic.xp.repo.impl.node.NodeServiceImpl.findCommits(NodeServiceImpl.java:708)
	at com.enonic.xp.repo.impl.node.executor.BatchedGetCommitsExecutor.execute(BatchedGetCommitsExecutor.java:42)
	at com.enonic.xp.repo.impl.dump.RepoDumper.dumpCommits(RepoDumper.java:208)
	at com.enonic.xp.repo.impl.dump.RepoDumper.lambda$execute$2(RepoDumper.java:100)
	at com.enonic.xp.context.ContextImpl.runWith(ContextImpl.java:86)
	at com.enonic.xp.repo.impl.dump.RepoDumper.execute(RepoDumper.java:100)
	at com.enonic.xp.repo.impl.dump.DumpServiceImpl.dump(DumpServiceImpl.java:231)
	at com.enonic.xp.impl.server.rest.task.DumpRunnableTask.run(DumpRunnableTask.java:44)
	at com.enonic.xp.impl.task.TaskWrapper.lambda$callTaskWithContext$0(TaskWrapper.java:90)
	at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:102)
	at com.enonic.xp.impl.task.TaskWrapper.callTaskWithContext(TaskWrapper.java:89)
	at com.enonic.xp.impl.task.TaskWrapper.doRun(TaskWrapper.java:75)
	at com.enonic.xp.impl.task.TaskWrapper.run(TaskWrapper.java:59)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:152)
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:874)
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:852)
	at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:389)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
	... 3 common frames omitted
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Result window is too large, from + size must be less than or equal to: [500000] but was [505000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.
	at org.elasticsearch.search.internal.DefaultSearchContext.preProcess(DefaultSearchContext.java:212)
	at org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:103)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:689)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:633)
	at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:472)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:392)
	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:389)
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
	at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	... 3 common frames omitted
6

It seems to me like you’re describing a bug that was fixed in XP 7.2.2?

Source:

7

Thanks! I will try upgrading to it and do a dump again.

8

NB! Go directly to 7.2.4 as there are also more fixes available

9

We have updated XP to 7.2.4 and the error is gone. Issue solved. Thanks!

1 Like