How to set Elastic Search heap size?

Enonic version: 7.11.0
OS: Linux

How to set memory limit (heap size) for Elastic Search?
Elastic eat all free memory and application get down!

Caused by: org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.

at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:70)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:62)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:52)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.getById(StorageDaoImpl.java:181)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.doGetById(BranchServiceImpl.java:172)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.get(BranchServiceImpl.java:166)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.get(NodeStorageServiceImpl.java:292)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.node.GetNodeByIdCommand.execute(GetNodeByIdCommand.java:26)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.node.NodeServiceImpl.doGetById(NodeServiceImpl.java:187)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.node.NodeServiceImpl.executeGetById(NodeServiceImpl.java:154)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.repo.impl.node.NodeServiceImpl.getById(NodeServiceImpl.java:122)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         at com.enonic.xp.core.impl.content.GetContentByIdCommand.execute(GetContentByIdCommand.java:30)
Apr 04 14:45:23 nginx-2 server.sh[373703]:         ... 167 common frames omitted
Apr 04 14:45:35 nginx-2 server.sh[373703]: 2023-04-04 14:45:35,297 INFO  org.elasticsearch.monitor.jvm - [393a83f9-08db-43bc-ba48-c980619267a0] [gc][old][336107][989] duration [7.6s], collections [1]/[8s], total [7.6s]/[2h], memory [24.6gb]->[24.5gb]/[25gb], all_pools {[young] [128mb]->[0b]/[0b]}{[old] [24.5gb]->[24.5gb]/[25gb]}{[survivor] [0b]->[0b]/[0b]}

How are you running XP? Are you running it from the docker image? This documentation page says that a good rule of thumb is to set heap size to 30% of available memory and never above 26GB:

1 Like

I ran XP as service. I set heap size to 25GB.
Total 62GB.

Ok, thanks, I will try to reduce to 15GB (25% of total size).

OK. When you run XP as a service, you set the environment variable
XP_OPTS=-Xms15G -Xmx15G

Here is the documentation for XP as a service using systemd on Linux:

1 Like

I’ve set environment variable to 15G (XP_OPTS=-Xms15G -Xmx15G)
Then reloaded xp service.

But still I’m getting Elastic (Timeout waiting for task) errors such flllowing:

Caused by: org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:70)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:62)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:52)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.elasticsearch.storage.StorageDaoImpl.getById(StorageDaoImpl.java:181)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.doGetById(BranchServiceImpl.java:172)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.getFromCache(BranchServiceImpl.java:277)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.doGetByPath(BranchServiceImpl.java:241)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.branch.storage.BranchServiceImpl.get(BranchServiceImpl.java:196)
Apr 05 14:38:35 nginx-2 server.sh[388389]:         at com.enonic.xp.repo.impl.storage.NodeStorageServiceImpl.get(NodeStorageServiceImpl.java:300)

Although now I have enough free RAM on server (about free 13G), but application become unavailable time-to time and I’m getting Timeout waiting for task exceptions.
What can be the reason of ElasticsearchTimeoutException and how to fix that ?
Can it be because of large data?

My application got down again.
It was stable about 2 - 3 days and then became unavailable ;(
In logs I see elastic timeout exceptions

org.elasticsearch.ElasticsearchTimeoutException: Timeout waiting for task.

Can you please tell me what can be the reason?
PS: I will try to reindex data, may be that will help with the problem.

This is most likely related to your application, possibly you have created unbounded queries or similar that fill up memory. Ensure you have strict count limits for hits etc.

1 Like