Error when running enonic sandbox start, no such file or directory

thomasheartman · February 11, 2021, 3:37pm

Enonic version: enonic CLI version 1.5.1, linux sdk 7.6.0
OS: NixOS 20.09

Hi,

I’m struggling a bit with getting started and following the guides provided. I have a sneaking suspicion that this is due to NixOS and the way it handles things, but I’m not entirely sure, so I thought I’d post here to see if anyone knows anything about it.

The issue is that when running enonic sandbox start, I get an error message saying:

Could not start process: /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/bin/server.sh fork/exec /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/bin/server.sh: no such file or directory

Turns out it’s trying to use /bin/bash, which should work on most Linux distros. However, on NixOS, the path is different. In fact, it’s /nix/store/vnyfysaya7sblgdyvqjkrjbrb0cy11jf-bash-4.4-p23/bin/bash on my current system, though /run/current-system/sw/bin/bash should work.

So I rewrote the shebang lines in the scripts in the directory to use the latter. Now it complains that:

./enonic-xp-linux-sdk-7.6.0/bin/server.sh: line 91: /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/jdk/bin/java: No such file or directory

That line is the run command, which looks like this:

exec "$JAVACMD" $JAVA_OPTS -Dxp.install="$XP_INSTALL" -Dfile.encoding=UTF8 -Dnashorn.args="--no-deprecation-warning" $XP_OPTS -Dmapper.allow_dots_in_name=true --add-opens java.base/java.net=ALL-UNNAMED --add-opens java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -classpath "$XP_INSTALL/lib/*" com.enonic.xp.launcher.LauncherMain "${ARGS[@]}"

Now, I’ve checked that the file .../bin/java is definitely there, but there is this interesting thing where if I try and run it I get an error message:

In the jdk/bin dir

$ ./java
Failed to execute process './java'. Reason:
The file './java' does not exist or could not be executed.

The file is definitely there and the permissions say it’s executable for all groups, so I don’t really know what’s going on. Again, I suspect this is related to how NixOS handles software and packages, but I’m not 100% certain, so I thought I’d come here for advice first.

Oh, and for the record: I downloaded the binary from https://repo.enonic.com/public/com/enonic/cli/enonic/1.5.1/enonic_1.5.1_Linux_64-bit.tar.gz and not through my regular package manager (nix). Nix does not yet support snaps.

Any thoughts and insights are much appreciated. I’m also happy to provide any other information you may need; just ask.

Cheers!

gbi · February 12, 2021, 9:46am

The issue with the shebang line is a bug. I filed an issue in XP to fix this issue: https://github.com/enonic/xp/issues/8652

The second problem is related to NixOS specifically. It seems to be a linking issue between the java binary and some dynamic libraries. Here are some resources to help you out with that:

If you find a solution to this problem please post it. I am curious how to fix this.

raboof · February 21, 2021, 9:07am

It’s definitely related to how NixOS handles things: the interpreter and libraries are not in the place the binary expects them.

For Java, I think it would be interesting to know why enonic is shipping with a jdk. Is it a somehow-patched JDK, or is it just there for convenience so that everything is in one package and there is no chance of version mismatches? In the latter case, which is pretty common, it might be a fine strategy to just use a nix-packaged jdk.

thomasheartman · February 21, 2021, 2:12pm

I’m not entirely sure whether the JDK is patched in some way or not, though I suspect it isn’t and that it’s included to make the environment as controlled as possible. @gbi, can you speak to that?

I also tried using a nix-packaged JDK, but it still didn’t work. However, it is definitely possible that I didn’t do it correctly.

Side note: I also created a post on the NixOS discourse forums about this issue, and have gotten some good pointers so far. The gradle2nix tool looks particularly promising and might be just what I need in this case.

raboof · February 21, 2021, 2:25pm

I think it’d be interesting in what way that failed :).

Yeah, that’s where I found out about this thread, but I figured the ‘why ship a jdk’ question would be more likely to be answered here than there .

It’s definitely neat and a good learning experience to create a derivation to build from source, but it might not be the easiest route. Hard to predict though: sometimes it actually is the easiest route .

tsi · February 21, 2021, 4:03pm

JVM is bundled inside XP to ensure consistency of version across deployments and upgrades, but also to simplify installation and use of the platform. It is a standard Java distro.

thomasheartman · February 23, 2021, 12:07pm

Thanks for picking up on this. I thought I’d try again, but decided to go with a different approach this time and changed the $JAVACMD variable in the server.sh to just use the java in my path (from the adoptopenjdk-bin package in nixpkgs). Doing it this way, it starts up just fine, which is great!

To be clear, I changed this:

locateJava() {
    if [ -n "$JAVA_HOME" ] ; then
        if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
            JAVACMD="$JAVA_HOME/jre/sh/java"
        else
            JAVACMD="$JAVA_HOME/bin/java"
        fi
        if [ ! -x "$JAVACMD" ] ; then
            die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME."
        fi
    else
        JAVACMD="java"
        which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH."
    fi
}

to this:

locateJava() {
  JAVACMD="java"
  which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH."
}

Obviously, if it means manually modifying the files, then it’s not much of a permanent solution, but at least it works for now. I’ll try and find a better way to handle this. Any ideas?

raboof · February 23, 2021, 12:44pm

thomasheartman:

    if [ -n "$JAVA_HOME" ] ; then
        if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
            JAVACMD="$JAVA_HOME/jre/sh/java"
        else
            JAVACMD="$JAVA_HOME/bin/java"
        fi

Interesting. If I make available adoptopenjdk-bin in nix-shell, JAVA_HOME is set:

$  nix-shell -p adoptopenjdk-bin
$ export | grep JAVA_HOME
declare -x JAVA_HOME="/nix/store/8cgax4522yhc05y4f3cgdagp7py5rac7-adoptopenjdk-hotspot-bin-11.0.9"
$ ls /nix/store/8cgax4522yhc05y4f3cgdagp7py5rac7-adoptopenjdk-hotspot-bin-11.0.9/bin/java
/nix/store/8cgax4522yhc05y4f3cgdagp7py5rac7-adoptopenjdk-hotspot-bin-11.0.9/bin/java

… so that looks like it should actually work with the script? How are you installing adoptopenjdk-bin?

thomasheartman · February 23, 2021, 1:01pm

Interesting. I’ve just created a nix shell with adoptopenjdk: nix-shell -p adoptopenjdk-bin and tried to launch things from there.

When I try to run it with the original function definition, I get this message:

server.sh: ERROR: JAVA_HOME is set to an invalid directory: /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/jdk.

However, if echo $JAVA_HOME from the same shell that launches the ./server.sh script, I get this path:

/nix/store/xy2r5kdxz6j3drqbsh72nyn5pzy6i0h4-adoptopenjdk-hotspot-bin-11.0.7

I don’t know whether it gets overridden somehow or whether the nix shell just doesn’t export the variable, though. There’s no assignment to JAVA_HOME in the script.

thomasheartman · February 23, 2021, 1:14pm

Ah, no, I figured out what’s up: At the start of the script, it calls setenv.sh, which contains this:

# Set JAVA_HOME pointing to the embedded JDK
export JAVA_HOME=$(cd "$(dirname "$0")/../jdk"; pwd)
export PATH="$JAVA_HOME/bin:$PATH"

If I don’t run that script, it works as expected.

thomasheartman · February 23, 2021, 1:35pm

Hmm, now that I’ve made it work:

I wonder if there’s an elegant way to avoid having to fiddle with files on disk and still make it compatible with NixOS.

If the primary means of distribution (and user interface) is the Enonic CLI, it’d be nice to be able to make that available as a nix package, and then have everything else just work. At the same time, special casing something for NixOS feels a bit much. And even if the script detected whether you were on NixOS, it’d still require a JDK to be available.

What’s a good path forward here?

thomasheartman · March 9, 2021, 2:50pm

A little update on this: It seems that there are still some issues with it. While the sandbox starts up just fine (enonic sandbox start), deploying apps does not work using enonic project deploy does not work.

From a little bit of detective work, it looks like it’s this line in the project deploy command that’s responsible. It runs runGradleTask which attempts to find the distribution’s JDK path (function definition here). It seems that this hard codes the expected binary path, so the changes I made to setenv.sh don’t affect this at all.

It fails with the familiar error:

Deploying to sandbox 'Sandbox1'...
./gradlew: line 188: /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/jdk/bin/java: No such file or directory

exit status 127

If I move/rename the bundled java executable, I get this instead:

Deploying to sandbox 'Sandbox1'...

ERROR: JAVA_HOME is set to an invalid directory: /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/jdk

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

Next, if I symlink the java executable in that directory to the one in the nix store, it almost works, but not quite:

Deploying to sandbox 'Sandbox1'...
Downloading https://services.gradle.org/distributions/gradle-6.4-bin.zip
.................................................................................................

Welcome to Gradle 6.4!

Here are the highlights of this release:
 - Support for building, testing and running Java Modules
 - Precompiled script plugins for Groovy DSL
 - Single dependency lock file per project

For more details see https://docs.gradle.org/6.4/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

FAILURE: Build failed with an exception.

* What went wrong:
The newly created daemon process has a different context than expected.
It won't be possible to reconnect to this daemon. Context mismatch:
Java home is different.
Wanted: DefaultDaemonContext[uid=null,javaHome=/home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.0/jdk,daemonRegistryDir=/home/thomas/.gradle/daemon,pid=28929,idleTimeout=null,priority=NORMAL,daemonOpts=--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
Actual: DefaultDaemonContext[uid=dab39198-a209-44c7-857c-ffb8529fe310,javaHome=/nix/store/xy2r5kdxz6j3drqbsh72nyn5pzy6i0h4-adoptopenjdk-hotspot-bin-11.0.7,daemonRegistryDir=/home/thomas/.gradle/daemon,pid=29000,idleTimeout=10800000,priority=NORMAL,daemonOpts=--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

exit status 1

Alternative

Rather than using the CLI, we can deploy manually. It’s a bit tedious, but it works.

You can either:

Set `XP_HOME` and build (easier)

In the project folder, there’s a gradlew executable. This builds and deploys the project. Running ./gradlew build builds the app. This should work out of the box. To deploy it:

Set the XP_HOME env variable to the path to the home directory
in the sandbox you want to use:
```
# fish
set XP_HOME ~/.enonic/sandboxes/Sandbox1/home/

# bash/zsh (untested)
export XP_HOME=~/.enonic/sandboxes/Sandbox1/home/
```
Check and adjust the variable to use your sandbox value.
Run ./gradlew deploy. With the XP_HOME env var set, this should automatically put the generated .jar file in the right directory and you’re done.

Manually copy file to `deploy` folder (more work)

Using the same gradlew executable as above, we can use the gradlew build command to build the .jar file. The file will be located in the build/libs/ directory and have a name that looks like <app>-1.0.0-SNAPSHOT.jar. Copy this file to your sandbox’s deploy folder. In my case, this was ~/.enonic/sandboxes/Sandbox1/home/deploy. Now your app should be available.

thomasheartman · March 18, 2021, 11:50am

I’m going to keep updating this when I make any progress. Today, I managed to manually patch the bundled java executable, making it run just fine. This uses the method that @gbi linked to in his first response.

I’m still looking into how I can make package this and make it easy to work with. Some thoughts are gathered at the end of the post, but they’re far from final.

Manual patching

Following this detailed StackExchange reply on patching binaries for NixOS, I managed to manually patch the Java executable that comes with the Enonic distribution.

Here are the steps I took (Enonic XP 7.6.1):

Find out which interpreter it’s using:

$ patchelf --print-interpreter ./java
/lib64/ld-linux-x86-64.so.2

Find out where I can get this interpreter.

According to the stack exchange response, it’s available with glibc. However, it’s also provided by $NIX_CC (presumably for convenience). In an expression, you should probably use glibc, but when dirty patching like this, $NIX_CC will have to do.
```
$ cat $NIX_CC/nix-support/dynamic-linker
/nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/ld-linux-x86-64.so.2
```

Set the interpreter

patchelf --set-interpreter (cat $NIX_CC/nix-support/dynamic-linker) ./java

Find out that it’s not able to load all required libraries.

When trying to run it with this interpreter, I’m told that:

$ ./java
./java: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

Use ldd to find out what libraries it can’t find

$ ldd ./java
	linux-vdso.so.1 (0x00007ffc2ede1000)
	libz.so.1 => not found
	libpthread.so.0 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libpthread.so.0 (0x00007f261425b000)
	libjli.so => /home/thomas/.enonic/distributions/enonic-xp-linux-sdk-7.6.1/jdk/bin/./../lib/jli/libjli.so (0x00007f261404a000)
	libdl.so.2 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libdl.so.2 (0x00007f2614045000)
	libc.so.6 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/libc.so.6 (0x00007f2613e86000)
	/nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib/ld-linux-x86-64.so.2 => /nix/store/0c7c96gikmzv87i7lv3vq5s1cmfjd6zf-glibc-2.31-74/lib64/ld-linux-x86-64.so.2 (0x00007f2614482000)
	libz.so.1 => not found

As it turns out, it can’t find libz.so.1

Find out if I have this lib available in the store, and if so: where.

$ find /nix/store -name libz.so.1
/nix/store/rldppqna2kya26zpdrl7p1wlbz0jgvj3-zlib-1.2.11/lib/libz.so.1
/nix/store/3yglmszn58qwj3dw94b0z9iy18vxaa1w-zlib-1.2.11/lib/libz.so.1
/nix/store/s06clkz6r628iqzab3plng138dln85h0-zlib-1.2.11/lib/libz.so.1
/nix/store/7bgshg2z70fpcc7adxfag1lgf45yamxh-zlib-1.2.11/lib/libz.so.1
/nix/store/zkswvy1ya0nf5k6108av1zbyp2ns577v-zlib-1.2.11/lib/libz.so.1
/nix/store/1srmyg1a8cxqwd0hd24rj6kw4lqd61yq-zlib-1.2.11/lib/libz.so.1

As it turns out, I’ve got a bunch of copies of it. For a derivation, we’d probably specify zlib as a runtime dependency. For the dirty patch, though, we can use one of the above libs.

Add the path to the found libz.so.1 library to the executable’s rpath

$ patchelf --set-rpath /nix/store/s06clkz6r628iqzab3plng138dln85h0-zlib-1.2.11/lib/:(patchelf --print-rpath ./java) ./java

At this point, the executable should work as expected.

Automatic patching + packaging

Of course, it would be swell if we could package it properly or at least provide an overlay that would take care of it, but that may require more thinking.

Based on the above, I’d probably need glibc and zlip as buildInputs. For nativeBuildInputs: autoPatchelfHook and tar. The distribution is available here https://repo.enonic.com/public/com/enonic/xp/enonic-xp-linux-sdk/7.6.1/ in both tar and zip formats. Either use unzip or tar -xvf.

This extracts the XP distribution. The path to the Java file to patch here would be: <distribution>/jdk/bin/java. If autoPatchelfHook is able to patch the Java executable on its own, that’s great. Otherwise: we might have to do it manually (using patchelf), though I don’t know how that would work.

Now, the packaged app would end up in the Nix store, so we’d probably also want to create a symbolic link to the store directory from the expected ~/.enonic/distributions/enonic-xp-linux-sdk-x.y.z directory.

For now, assuming Linux should be alright. Support macOS (Nix darwin) could be a stretch goal.

Based on the Stack Exchange answer, I think the derivation would look something like this (but this is very much not finished):

{ stdenv, unzip, glibc, zlib, autoPatchelfHook }:

let
  version = "7.6.1";

  url =
    "https://repo.enonic.com/public/com/enonic/xp/enonic-xp-linux-sdk/${version}/enonic-xp-linux-sdk-7.6.1.tgz";

in stdenv.mkDerivation {
  name = "enonic-xp-${version}";
  inherit version;

  src = fetchTarball {
    sha256 = "0vmmqd9d4w0zis839wg62251vvvcd3jmvb3rg1p0bgcl3b2qy5dk";
    inherit url;
  };

  nativeBuildInputs = [ autoPatchelfHook ];

  buildInputs = [ glibc zlib ];

  # add unpackPhase and installPhaase here

  meta = with stdenv.lib; {
    description = "Enonic XP distribution";
    homepage = "https://enonic.com";
    license = licenses.gplv3;
    maintainers = with stdenv.lib.maintainers; [ ];
    platforms = [ "x86_64-linux" ];
  };
}