Using import feature with --skipids

Enonic version: 6.6.0
OS: Windows 7 x64

Hi, we are trying to build update system for our “test” enviroment, from a “prod”. When someone adding “hot changes” on “prod”, they should be moved to “test” by a daily cron job.

The thing that this “test” server can be empty(without site), and not empty(site already exist). Currently we are using export/import functionallity for updates because we don’t want to wipe all data from “test” after every update(someone can be working there).

Case #1, if we are using default export\import commands without (–skipids):
http://xp.readthedocs.io/en/stable/reference/toolbox/export.html
http://xp.readthedocs.io/en/stable/reference/toolbox/import.html

  1. If we are importing data on clean server - eveything ok.
  2. If we are importing data on already existing site, we could get error “such id already exist”. And because of id’s clash that node will not be imported. Problem.

Case #2, if we are using export\import commands with --skipids flag:

  1. If we are trying import data on clean server - it create every node, but when it trying to create references - it’s not finding it, because all references are left with “old” id’s, and all nodes have been created with “new” id’s.
  2. Haven’t tested but it seems that situation will be the same.

Is it a bug, or I am doing something wrong?

Hmmm, the part about “id already exists” sounds like a bug, but Im not able to reproduce it in my environment (6.8.0-SNAPSHOT, but no changes to this that I am aware of)

You say “could get”, does this means that this doesnt happen all the time?

Hi again.

Seems like this is related to moved content. We will investigate further and come back to you.

Ok, I see the issue, but how should this behave?

E.g;

  • you have a folder myFolder with subfolder sub1
  • you move the myFolder to myFolderMoved, creates a new folder sub2. sub1 gets new path
  • you then import the existing data, with the original paths

Should the original myFolder replace the newFolderMoved? Or should the existing path structure override the import structure?

This is a bit more complex than the existing import/export with ids can handle. Different switches/modes will be necessary to cover different usecases. We will add this to features to come.

Ok, so I was gathering data for a big reply with attachments:
firstExport: https://drive.google.com/open?id=0BxDCmPPDR-AaQUcwald2ZndmOFU
secondExport: https://drive.google.com/open?id=0BxDCmPPDR-AaMFpaN3lYNWltbHc
thirdExport: https://drive.google.com/open?id=0BxDCmPPDR-AabmJRTlRJOW56Mlk

  1. In first export I have created just two simple contents(let’s call them “first” and “second”). Exported this content into “firstExport”, uploaded it on different server, and moved “first” into folder. Exported this content from second server, and tried to apply it to the first one. If I will do so, I will get:

“importErrors” : [ “Could not import node in folder [d:/work/enonic/enonic-xp-6.6.0/home/data/export/secondExport/content/test/testfolder/first]: Node already exist, id: ca638677-4adb-46fb-b502-33939a704dcb - com.enonic.xp.node.NodeIdExistsException: Node already exist, id: ca638677-4adb-46fb-b502-33939a704dcb” ]

So this is error that you are talking about in previous message.

  1. Same start situation with “firstExport”. We are exporting it, and upploading to different server. Working A LOT on different server, creating parallel content, and after some time we could get auto-generated ID that clashes with id’s on first server. And will get same situation. This is what I have tried to emulate in “thirdExport”

  2. Ok, so id’s are clashing. We have tried to use --skipids key, to solve problem. So we are exporing content to “firstExport”, delete site, and try to export same content with --skipids key. Importing of content goes well, but all the references(in site.xml, for example) are wiped. Also not sure, is it a correct behaviour?

So now we are in strange situation, and really not sure how we should sync content between servers.

As for my thoughts:

Should the original myFolder replace the newFolderMoved? Or should the existing path structure override the import structure?

For me, if they have differnet URL’s they both could exist in same system. So in current situation “newFolderMoved” should be created near “myFolder”. Maybe it will create a bit of dupplication, but currently, as you see, we don’t have possibility to use it between different systems.

All this stuff about export/import is very important for us. Do you have any alternative that could handle this gaps?

Yes. What you want is a bit out of scope for the current implementation of the export/import; this is more of a staging/sync between servers.

Introducing references with paths (to keep references when doing export/import without ids) instead of id’s introduces a whole new world of complications, so we will probably not be doing that in the near future (that is, the 6.x series)

Duplication seems like a dangerous approach, but maybe a strategy for import that does “if existing already under different path, create a new with different id” may be possible. But, if stuff is moved in production, shouldnt the same stuff be moved in test to reflect the prod-system?

I dont see any immediate perfect way of handling this - until we can create a better solution for merging data - except for clearing the test-environment and load the prod data if the goal is to maintain a copy that is up to date with production. A similar export of test could be done, before the unload, then load them after the production data is loaded. Then you will get an environment equal to prod, and stuff added to test should normally (i guess) be applied with few conflicts - if any, it will ignore them in the import.

I have added this as a feature request, but its a bit too soon to say if its going to make it for the 6.9 version (6.8 is under code freeze right now)

It should. But since “production guys” is far away from “test guys” it’s not always possible, and besides no-one wants to do their job twice every time. Also I should say that we have more than just PROD and TEST.