ElasticSearch thread OutOfMemoryError after snapshot update

Niki · February 21, 2023, 1:49pm

Enonic version: 7.11.0
OS: Linux

Linux
60GB RAM
CPU(s): 16 (Intel(R) Xeon(R) Platinum 8270 CPU @ 2.70GHz).
Core(s) per socket: 2

I use React4xp tempate.

I had 25GB heap size, then I reduced it to 15GB and restarted xp service.

I’ve run Enonic as a linux service.

Then after snapshot update and reload public portal page I sometimes get following error:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “elasticsearch[5e15c08e-06ac-4e38-9fd5-4d26948bf5e5][[ttl_expire]]” Feb 21 18:55:12 nginx-2 server.sh[183678]: java.lang.OutOfMemoryError: Java heap space Feb 21 18:55:12 nginx-2 server.sh[183678]: <> Feb 21 19:14:08 nginx-2 server.sh[183678]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “qtp1037310970-6313”

Ca you please tell me how to avoid this error or may be I need to adjust Elastic config?

bhj · February 21, 2023, 4:15pm

15GB of heap space is a lot. I’ve never come across a production server that is using more than 10GB heap space (on a virtual machine with 32GB RAM), and usually 5GB heap space should be plenty on a production server (virtual machine with 16GB RAM). So if you’re filling 15GB heap space on a test server, then I think there’s something very wrong either in setup, data, or code.

Typical things that can fill up heap space is when code is processing a lot of data at once, for example if a javascript controller is trying to map a huge array of data at once, like if there is a query that either has a very high (or no) limit on the number of hits, or if your application suddenly has to deal with a huge chunk of data.

Niki · February 21, 2023, 6:38pm

I have about 20 simple parts/components on the page, that is not a lot.
And in Content Studio I have about 2 000 pages.

The problem usually happens after following command:
enonic app install --file /home/*.jar —> success
update page —> OutOfMemory

May be Elastic doesn’t see changes of heap size or need reindex?

similar problem - windows - Elastic Search-Java.lang.OutOfMemoryError - Stack Overflow

bhj · February 22, 2023, 11:39am

What version of React4XP are you using? There have been many changes to lib-react4xp the last month, and latest version is 3.4.2

It could be that React4XP is trying to compile a lot of data all at once.

Niki · February 22, 2023, 4:09pm

I use LIB_REACT4XP_VERSION = 3.2.1

And I also use Ant Design UI https://ant.design with React

Thanks for advice, I will try to change to 3.4.2 and watch if it works better for the next week

tsi · February 22, 2023, 4:27pm

Also, can you output the first 50 or so log lines from XP booting. This will provide some more information about your actual system setup.

Niki · February 22, 2023, 4:37pm

Feb 22 15:35:42 nginx-2 server.sh[200843]: # Enonic XP 7.11.0
Feb 22 15:35:42 nginx-2 server.sh[200843]: # Built on 2022-11-07T09:58:01Z (hash = 0eff920ee6f174d2c4f2ce824673d40a0fda5e11, branch = 0eff920ee6f174d2c4f2ce824673d40a0fda5e11)
Feb 22 15:35:42 nginx-2 server.sh[200843]: # OpenJDK 64-Bit Server VM 11.0.15 (GraalVM Community)
Feb 22 15:35:42 nginx-2 server.sh[200843]: # Linux 5.4.17-2102.201.3.el8uek.x86_64 (amd64)
Feb 22 15:35:42 nginx-2 server.sh[200843]: # Install directory is /opt/enonic/distros/enonic-xp-linux-server-7.11.0
Feb 22 15:35:42 nginx-2 server.sh[200843]: # Home directory is /opt/enonic/home
Feb 22 15:35:42 nginx-2 server.sh[200843]: 2023-02-22 15:35:42,939 INFO c.e.x.l.i.framework.FrameworkService - Starting Enonic XP…
Feb 22 15:35:43 nginx-2 server.sh[200843]: 2023-02-22 15:35:43,166 INFO c.e.x.l.i.p.ProvisionActivator - Installing 96 bundles…
Feb 22 15:35:43 nginx-2 server.sh[200843]: 2023-02-22 15:35:43,919 INFO ROOT - bundle org.apache.felix.scr:2.2.2 (10) Starting with globalExtender setting: false
Feb 22 15:35:43 nginx-2 server.sh[200843]: 2023-02-22 15:35:43,928 INFO ROOT - bundle org.apache.felix.scr:2.2.2 (10) Version = 2.2.2
Feb 22 15:35:44 nginx-2 server.sh[200843]: 2023-02-22 15:35:44,694 INFO c.h.osgi.HazelcastOSGiService - HazelcastOSGiServiceImpl{ownerBundle=com.hazelcast [23], hazelcastInstance=null, active=false, id=3.12.13#OSS} has been registered as OSGI service and activated now
Feb 22 15:35:45 nginx-2 server.sh[200843]: ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console…
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,794 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.export]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,804 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.repo]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,810 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.server.shell]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,811 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.server.trace]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,815 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.dos]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,820 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.app.standardidprovider]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,822 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.jetty]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,824 INFO c.e.x.s.i.config.ConfigInstallerImpl - Loaded config for [com.enonic.xp.web.vhost]
Feb 22 15:35:45 nginx-2 server.sh[200843]: 2023-02-22 15:35:45,841 INFO c.e.x.s.shell.impl.ShellActivator - Remote shell access is disabled
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,052 INFO c.e.x.s.internal.trace.TraceService - Call tracing is disabled in config
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,126 INFO c.e.x.c.i.a.c.AuditLogConfigImpl - Audit log is enabled and mappings updated.
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,138 INFO c.e.x.i.blobstore.BlobStoreActivator - Waiting for blobstore-provider [file]
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,139 INFO c.e.x.i.blobstore.BlobStoreActivator - Found blobstore-provider [file]
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,155 INFO c.e.x.i.blobstore.BlobStoreActivator - Registered blobstore [file] successfully
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,442 INFO c.e.x.s.i.ScriptRuntimeFactoryActivator - Using Nashorn as script engine
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,600 INFO c.e.x.w.v.i.c.VirtualHostServiceImpl - Virtual host is enabled and mappings updated.
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,740 INFO c.e.x.l.i.framework.FrameworkService - Started Enonic XP in 3801 ms
Feb 22 15:35:46 nginx-2 server.sh[200843]: 2023-02-22 15:35:46,740 INFO E.F.org.apache.felix.framework - FrameworkEvent STARTLEVEL CHANGED
Feb 22 15:35:56 nginx-2 server.sh[200843]: 2023-02-22 15:35:56,586 INFO org.elasticsearch.node - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] version[2.4.6], pid[200843], build[NA/NA]
Feb 22 15:35:56 nginx-2 server.sh[200843]: 2023-02-22 15:35:56,586 INFO org.elasticsearch.node - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] initializing …
Feb 22 15:35:56 nginx-2 server.sh[200843]: 2023-02-22 15:35:56,590 INFO org.elasticsearch.plugins - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] modules [], plugins [], sites []
Feb 22 15:35:56 nginx-2 server.sh[200843]: 2023-02-22 15:35:56,601 INFO org.elasticsearch.env - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] using [1] data paths, mounts [[/ (/dev/mapper/ol-root)]], net usable_space [27.2gb], net total_space [59.3gb], spins? [possibly], types [xfs]
Feb 22 15:35:56 nginx-2 server.sh[200843]: 2023-02-22 15:35:56,601 INFO org.elasticsearch.env - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] heap size [15gb], compressed ordinary object pointers [true]
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,759 INFO org.elasticsearch.node - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] initialized
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,760 INFO org.elasticsearch.node - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] starting …
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,763 INFO org.elasticsearch.transport - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] publish_address {local[1]}, bound_addresses {local[1]}
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,766 INFO org.elasticsearch.discovery - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] mycluster/p4BJa2XAS7iZrbZZM9hLUg
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,771 INFO org.elasticsearch.cluster.service - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] new_master {ef3ac602-f58f-4cd9-a7bf-8cc6268557c7}{p4BJa2XAS7iZrbZZM9hLUg}{local}{local[1]}{local=true, master=true}, reason: local-disco-initial_connect(master)
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,777 INFO org.elasticsearch.node - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] started
Feb 22 15:35:57 nginx-2 server.sh[200843]: 2023-02-22 15:35:57,909 INFO org.elasticsearch.gateway - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] recovered [16] indices into cluster_state
Feb 22 15:35:58 nginx-2 server.sh[200843]: 2023-02-22 15:35:58,813 INFO o.e.cluster.routing.allocation - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[storage-system-repo][0]] …]).
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,208 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering configured application com.enonic.xp.app.system bundle 91
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,209 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering configured application com.enonic.xp.app.applications bundle 93
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,211 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering configured application com.enonic.xp.app.main bundle 94
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,212 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering configured application com.enonic.xp.app.standardidprovider bundle 95
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,213 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering configured application com.enonic.xp.app.users bundle 96
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,860 INFO c.e.x.c.impl.ClusterManagerImpl - Adding cluster-provider: elasticsearch
Feb 22 15:35:59 nginx-2 server.sh[200843]: 2023-02-22 15:35:59,887 INFO org.eclipse.jetty.util.log - Logging initialized @18298ms to org.eclipse.jetty.util.log.Slf4jLog
Feb 22 15:36:00 nginx-2 server.sh[200843]: 2023-02-22 15:36:00,003 INFO org.eclipse.jetty.server.Server - jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.15+10-jvmci-22.1-b06
Feb 22 15:36:00 nginx-2 server.sh[200843]: 2023-02-22 15:36:00,026 INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=
Feb 22 15:36:00 nginx-2 server.sh[200843]: 2023-02-22 15:36:00,027 INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
Feb 22 15:36:00 nginx-2 server.sh[200843]: 2023-02-22 15:36:00,028 INFO org.eclipse.jetty.server.session - Scavenging every 600000ms

tsi · February 22, 2023, 5:04pm

Also do you have the boot parameters used for the linux service? I.e. where you tune heap size. Recommended setup is 1/3 of vm/container mem should be heap space, ES will use the rest as off-heap mem.

As mentioned by Bjørnar, an oom with this amount of ram is not normal.

FYI. By installing live-trace app, you can confirm actual memory avail/usage directly from Admin.

Niki · February 23, 2023, 2:48am

[Unit]

Description=Enonic XP

Documentation=https://developer.enonic.com/docs

Wants=network-online.target

After=network-online.target

[Service]

Type=simple

PrivateTmp=true

Environment=XP_INSTALL=/opt/enonic/distros/enonic-xp-linux-server-7.11.0

Environment=XP_JAVA_HOME=/opt/enonic/distros/enonic-xp-linux-server-7.11.0/jdk

Environment=XP_HOME=/opt/enonic/home

Environment=XP_OPTS="-Xms15G -Xmx15G"

User=xp

Group=xp

ExecStart=/opt/enonic/distros/enonic-xp-linux-server-7.11.0/bin/server.sh

StandardOutput=journal

StandardError=inherit

LimitNOFILE=65536

LimitNPROC=4096

LimitAS=infinity

LimitFSIZE=infinity

KillSignal=SIGTERM

KillMode=process

SendSIGKILL=no

SuccessExitStatus=143

[Install]

WantedBy=multi-user.target

May be I need to clear buff/cache? Is it safe for ES and XP?

Niki · February 23, 2023, 3:23am

Niki · February 23, 2023, 3:24pm

I’ve updated React4XP version to 3.4.2

but still continue getting Java heap space error after snapshot update

app and xp service freezes and doesn’t respond

and I should kill it manually (

Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:06 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:19:10 nginx-2 server.sh[200843]: Feb 23 19:39:11 nginx-2 server.sh[200843]: Feb 23 19:39:16 nginx-2 server.sh[200843]: Feb 23 19:39:21 nginx-2 server.sh[200843]: Feb 23 19:39:26 nginx-2 server.sh[200843]: Feb 23 20:56:29 nginx-2 server.sh[200843]: Feb 23 20:56:29 nginx-2 server.sh[200843]: Feb 23 20:56:29 nginx-2 server.sh[200843]: Feb 23 20:56:33 nginx-2 server.sh[200843]: Feb 23 20:56:33 nginx-2 server.sh[200843]: Feb 23 20:56:41 nginx-2 server.sh[200843]: Feb 23 20:56:41 nginx-2 server.sh[200843]: Feb 23 20:56:41 nginx-2 server.sh[200843]: Feb 23 20:56:41 nginx-2 server.sh[200843]: 2023-02-23 19:19:02,104 WARN org.elasticsearch.indices.ttl - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] failed to execute ttl purge
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
2023-02-23 19:19:06,182 ERROR c.e.x.i.s.distributed.RescheduleTask - Problem during tasks scheduling
com.enonic.xp.repository.IndexException: Search request failed after [30s], query: [{
“from” : 0,
“size” : 1,
“query” : {
“bool” : {
“must” : {
“match_all” : { }
},
“filter” : {
“bool” : {
“must” : [ {
“terms” : {
“path” : [ “/” ]
}
}, {
“terms” : {
“branch” : [ “master” ]
}
} ]
}
}
}
},
“explain” : false,
“fields” : [ “nodeid”, “versionid”, “nodeblobkey”, “indexconfigblobkey”, “accesscontrolblobkey”, “state”, “path”, “timestamp”, “references” ],
“highlight” : {
“require_field_match” : false,
“type” : “plain”
}
}]
at com.enonic.xp.repo.impl.elasticsearch.executor.AbstractExecutor.doSearchRequest(AbstractExecutor.java:58)
at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.doSearch(SearchExecutor.java:78)
at com.enonic.xp.repo.impl.elasticsearch.executor.SearchExecutor.execute(SearchExecutor.java:63)
at com.enonic.xp.repo.impl.elasticsearch.search.SearchDaoImpl.search(SearchDaoImpl.java:23)
at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.doGetByPath(BranchServiceImpl.java:256)
at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.get(BranchServiceImpl.java:196)
at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.get(NodeStorageServiceImpl.java:300)
at com.enonic.xp.repo.impl.node.GetNodeByPathCommand.execute(GetNodeByPathCommand.java:26)
at com.enonic.xp.repo.impl.node.NodeChildOrderResolver.doResolve(NodeChildOrderResolver.java:36)
at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:100)
at com.enonic.xp.repo.impl.node.NodeHelper.runAsAdmin(NodeHelper.java:23)
at com.enonic.xp.repo.impl.node.NodeChildOrderResolver.resolve(NodeChildOrderResolver.java:23)
at com.enonic.xp.repo.impl.node.FindNodesByParentCommand.execute(FindNodesByParentCommand.java:61)
at com.enonic.xp.repo.impl.node.NodeServiceImpl.executeFindByParent(NodeServiceImpl.java:386)
at com.enonic.xp.repo.impl.node.NodeServiceImpl.findByParent(NodeServiceImpl.java:346)
at com.enonic.xp.impl.scheduler.ListScheduledJobsCommand.doExecute(ListScheduledJobsCommand.java:33)
at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:100)
at com.enonic.xp.impl.scheduler.ListScheduledJobsCommand.execute(ListScheduledJobsCommand.java:28)
at com.enonic.xp.impl.scheduler.SchedulerServiceImpl.list(SchedulerServiceImpl.java:116)
at com.enonic.xp.context.ContextImpl.callWith(ContextImpl.java:100)
at com.enonic.xp.impl.scheduler.distributed.RescheduleTask.lambda$doRun$0(RescheduleTask.java:69)
at com.enonic.xp.core.internal.osgi.OsgiSupport.withServiceOrElseGet(OsgiSupport.java:68)
at com.enonic.xp.core.internal.osgi.OsgiSupport.withService(OsgiSupport.java:96)
at com.enonic.xp.impl.scheduler.distributed.RescheduleTask.doRun(RescheduleTask.java:69)
at com.enonic.xp.impl.scheduler.distributed.RescheduleTask.run(RescheduleTask.java:47)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:70)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:62)
at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:52)
at com.enonic.xp.repo.impl.elasticsearch.executor.AbstractExecutor.doSearchRequest(AbstractExecutor.java:51)
… 30 common frames omitted
2023-02-23 19:19:10,232 ERROR c.e.l.r.ssr.resources.AssetLoader - com.enonic.lib.react4xp.ssr.errors.RenderException: org.graalvm.polyglot.PolyglotException: Java heap space
com.enonic.lib.react4xp.ssr.errors.RenderException: org.graalvm.polyglot.PolyglotException: Java heap space
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.eval(AssetLoader.java:147)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.loadAssetIntoEngine(AssetLoader.java:82)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.loadAssetsIntoEngine(AssetLoader.java:55)
at com.enonic.lib.react4xp.ssr.pool.Renderer.(Renderer.java:58)
at com.enonic.lib.react4xp.ssr.pool.RendererFactory.create(RendererFactory.java:33)
at com.enonic.lib.react4xp.ssr.pool.RendererFactory.create(RendererFactory.java:15)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:70)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:637)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:659)
at org.apache.commons.pool2.impl.GenericObjectPool.preparePool(GenericObjectPool.java:974)
at com.enonic.lib.react4xp.ssr.ServerSideRenderer.lambda$asyncInitRenderers$1(ServerSideRenderer.java:119)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.script.ScriptException: org.graalvm.polyglot.PolyglotException: Java heap space
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.toScriptException(GraalJSScriptEngine.java:492)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:469)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:435)
at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.eval(AssetLoader.java:129)
… 16 more
Caused by: org.graalvm.polyglot.PolyglotException: Java heap space
at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:467)
… 19 more
RenderException: org.graalvm.polyglot.PolyglotException: Java heap space
Engine #16
in com.enonic.lib.react4xp.ssr.ServerSideRenderer.loadAsset
assetName = ‘/assets/react4xp/vendors.dc90ba8fdf2d47de10c9.js’
SOLUTION TIPS: The previous error message may refer to lines in compiled/mangled code. To increase readability, you can try react4xp clientside-rendering or building react4xp with buildEnv = development or gradle CLI argument -Pdev. Remember to clear all cached behavior first (stop continuous builds, clear/rebuild your project, restart the XP server, clear browser cache).
2023-02-23 19:19:10,246 ERROR c.e.l.r.ssr.ServerSideRenderer - Error during async init Renderers
com.enonic.lib.react4xp.ssr.errors.RenderException: Couldn’t init Renderer#16
at com.enonic.lib.react4xp.ssr.pool.Renderer.(Renderer.java:63)
at com.enonic.lib.react4xp.ssr.pool.RendererFactory.create(RendererFactory.java:33)
at com.enonic.lib.react4xp.ssr.pool.RendererFactory.create(RendererFactory.java:15)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:70)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:571)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureIdle(GenericObjectPool.java:637)
at org.apache.commons.pool2.impl.GenericObjectPool.ensureMinIdle(GenericObjectPool.java:659)
at org.apache.commons.pool2.impl.GenericObjectPool.preparePool(GenericObjectPool.java:974)
at com.enonic.lib.react4xp.ssr.ServerSideRenderer.lambda$asyncInitRenderers$1(ServerSideRenderer.java:119)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.enonic.lib.react4xp.ssr.errors.RenderException: org.graalvm.polyglot.PolyglotException: Java heap space
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.eval(AssetLoader.java:147)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.loadAssetIntoEngine(AssetLoader.java:82)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.loadAssetsIntoEngine(AssetLoader.java:55)
at com.enonic.lib.react4xp.ssr.pool.Renderer.(Renderer.java:58)
… 13 common frames omitted
Caused by: javax.script.ScriptException: org.graalvm.polyglot.PolyglotException: Java heap space
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.toScriptException(GraalJSScriptEngine.java:492)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:469)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:435)
at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.enonic.lib.react4xp.ssr.resources.AssetLoader.eval(AssetLoader.java:129)
… 16 common frames omitted
Caused by: org.graalvm.polyglot.PolyglotException: Java heap space
at org.graalvm.sdk/org.graalvm.polyglot.Context.eval(Context.java:399)
at org.graalvm.js.scriptengine/com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:467)
… 19 common frames omitted
2023-02-23 19:39:11,032 INFO org.elasticsearch.monitor.jvm - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] [gc][old][97761][852] duration [5.1s], collections [1]/[5.1s], total [5.1s]/[1h], memory [14.9gb]->[14.9gb]/[15gb], all_pools {[young] [8mb]->[0b]/[0b]}{[old] [14.9gb]->[14.9gb]/[15gb]}{[survivor] [0b]->[0b]/[0b]}
2023-02-23 19:39:16,117 INFO org.elasticsearch.monitor.jvm - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] [gc][old][97762][853] duration [5s], collections [1]/[5s], total [5s]/[1h], memory [14.9gb]->[14.9gb]/[15gb], all_pools {[young] [0b]->[0b]/[0b]}{[old] [14.9gb]->[14.9gb]/[15gb]}{[survivor] [0b]->[0b]/[0b]}
2023-02-23 19:39:21,184 INFO org.elasticsearch.monitor.jvm - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] [gc][old][97763][854] duration [5s], collections [1]/[5s], total [5s]/[1h], memory [14.9gb]->[14.9gb]/[15gb], all_pools {[young] [0b]->[4mb]/[0b]}{[old] [14.9gb]->[14.9gb]/[15gb]}{[survivor] [0b]->[0b]/[0b]}
2023-02-23 19:39:26,350 INFO org.elasticsearch.monitor.jvm - [ef3ac602-f58f-4cd9-a7bf-8cc6268557c7] [gc][old][97764][855] duration [5.1s], collections [1]/[5.1s], total [5.1s]/[1h], memory [14.9gb]->[14.9gb]/[15gb], all_pools {[young] [4mb]->[8mb]/[0b]}{[old] [14.9gb]->[14.9gb]/[15gb]}{[survivor] [0b]->[0b]/[0b]}
2023-02-23 20:56:29,015 INFO c.e.x.s.i.e.ScriptEventManagerImpl - Removed all Script Event Listeners for isnareactcms
2023-02-23 20:56:29,015 INFO E.F.org.apache.felix.framework - FrameworkEvent PACKAGES REFRESHED
2023-02-23 20:56:29,015 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Uninstalled application isnareactcms bundle 108
2023-02-23 20:56:33,291 INFO c.e.x.c.i.app.ApplicationServiceImpl - Installed application isnareactcms bundle 109
2023-02-23 20:56:33,291 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Registering application isnareactcms bundle 109
2023-02-23 20:56:41,656 INFO c.e.x.c.i.app.ApplicationServiceImpl - Global Application [isnareactcms] installed successfully
2023-02-23 20:56:41,660 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Started application isnareactcms bundle 109
2023-02-23 20:56:41,660 INFO c.e.x.c.i.a.ApplicationRegistryImpl - Configuring application isnareactcms bundle 109
2023-02-23 20:56:41,660 INFO c.e.x.c.i.app.ApplicationServiceImpl - Application [isnareactcms] started successfully

Niki · February 23, 2023, 3:35pm

Live trace memory screenshot

rymsha · February 28, 2023, 2:21pm

You have quite a few CPU cores…
Try to lower number of react4xp engines via react4xp.ssr.maxThreads = 1
(This value is for an experiment only.) It may allow to fit react4xp assets into memory. They might be abnormally big since you are using UI react components.

Niki · February 28, 2023, 3:03pm

I have cleared buff/cache

sync; echo 1 > /proc/sys/vm/drop_caches

and have changed back heap to 25G.

Now I have 31Gi free RAM and I don’t get OutOfMemoryError anymore.
The main idea as a temp solution to clear buff/cache time to time to get at least 2G or more as a free memory.

If there will be less than 2G, you may have a risk of getting OutOfMemoryError after snapshot update.

tsi · February 28, 2023, 8:37pm

Niki.
There must be something wrong with the code, or the queries you are doing to cause this amount of ram usage. Normally, XP will be able to spin a small site with just a few Gig or ram.

Do you have a license to run LiveTrace performance analysis?

Niki · March 1, 2023, 2:52am

No, I don’t have a license.

ElasticSearch thread OutOfMemoryError after snapshot update

Enonic version: 7.11.0 OS: Linux

Enonic version: 7.11.0
OS: Linux