Resources and Usage¶
As with every application, you will need to plan and maintain the usage of resources like CPU time, memory and storage for Dataverse and its components.
Details about storage can be found at Storage and Backup.
Dataverse and Solr both being Java technology based components need to be tweaked for memory usage. Obviously the values below count per instance.
Recommendend min. RAM for production use
Application server w/ Dataverse
Solr Search Index
The JVM will by default use 70% of RAM for it’s heap. No need to add an operating system reserve when running in containers on K8s.
Configuring pods memory¶
Since Java 8u192 the Hotspot VM is natively supporting container resource limits.
To configure these limits, simply configure it in the
spec: template: spec: containers: - name: dataverse resources: requests: memory: "2Gi" limits: memory: "4Gi"
You can easily apply your settings by using a patch and kustomize. Please find
an example in the demo persona directory
For development or demo use, you’ll be good in most cases with much less. You need to ensure the JVM uses at least ~800 MiB for heap space. Using less heap space will not even deploy successfully.
How much RAM is used at max for Java Heap can be easily adjusted by using the
-XX:MaxRAMPercentage=xx.x. For your convenience this has been
simplified by supporting an environment variable
see hidden example below. Please keep in mind: value must be a floating point
Development values are hidden on purpose to avoid confusion. Expand/hide
1.5 GiB RAM means ~1 GiB of heap space using the 70% default setting, which is safe. You can tweak the setting to match your necessities like below:
spec: template: spec: containers: - name: dataverse resources: requests: memory: "1.0Gi" limits: memory: "1.5Gi" env: - name: MEM_MAX_RAM_PERCENTAGE value: "50.0"
Fighting “Out Of Memory” situations¶
Java Memory usage is a very complex topic and you should take great care of monitoring it’s usage within your deployment.
Most operators are aware of the Heap, where values of POJOs are stored during the applications runtime. You should also be aware of the Stack, Metaspace and other memory terms, like Native Memory.
The following sections is not so much about prevention of these situations (that’s why you’re doing monitoring, right?), but what you can do to troubleshoot when bad things happened.
Shortage of Heap Space¶
Many times, you’re application is killed by running out of memory. Often this is related to running out of Heap Space, the most important type of memory for a Java application.
If you don’t know what Garbage Collection is and how memory allocation works in Java, you can find lots of resources online. Some examples:
When running out of Heap Space, your JVM will throw an
exception (see also Oracle docs on OutOfMemoryError).
In these cases, a heap dump will be written to
$DUMPS_DIR, which is
/dumps by default. Those can be analyzed using tools like Eclipse MAT.
dev persona, a sidecar container is deployed with whom you
kubectl cp to download the
.hprof dump file for analysis.
You might want to deploy your own solution, maybe uploading to an object store, sending notifications or other great ideas. Some inspirations:
Shortage of other memory spaces¶
For many users of Java applications, other types of memory than the Heap are less commonly known. Compared to the huge amounts of Heap spaces, those are often rather small, yet they might get you into trouble.
To understand how this might happen, you need to be aware that the Linux kernel
will non-gracefully kill processes exceeding their memory limits. A container
running on a Kubernetes cluster usually should have resource limits restrictions
applied. (Java will align it’s memory usage to these as outlined above.) Once
the containers starts using more RAM than the limits allow, the out of memory
killer will stop the process (usually the only one running in a single container)
and Kubernetes will log an event
Depending on how much RAM budget you have left on your nodes, you might either simply raise the limits. Or you might want to do deeper analysis of the problem, because there might be a memory leak, coming back no matter how much you raise the limits.
There are some excellent resources to read when you go for a hunt:
So regarding to monitoring, you should always keep an eye not only on heap and GC stats, but also on the very basic containers metrics offered by K8s. Try to match the JVM memory stats with those from the container. If things fall apart, there is a good chance you’ll see it before it dies from memory leaks.