GC analysis and troubleshooting with SapMachine
source link: https://blogs.sap.com/2023/05/05/gc-analysis-and-troubleshooting-with-sapmachine/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
GC analysis and troubleshooting with SapMachine
There are several possibilities to analyze and troubleshoot GC issues. You can activate the gc log via Unified JVM logging(-Xlog:gc) to get a textual log output. This can be also set for detailed GC phases and to specific levels. The analysis can be done on the textual log file or with available open source tools like JIFA or commercial tools like GCeasy to parse and analyze the log and visualize the findings.
Another possibility is to use the Java Flight Recorder (JFR) to profile your Java application and analyze this with Java Mission Control (JMC).
We, from SapMachine development team, build and provide also a JMC version on SapMachine.
SapMachine will provide two additional Flight Recording configurations
The SapMachine starting with 11.0.19 and 17.0.7 provides two additional Flight Recording configurations(located in the directory lib/jfr of the JDK/JRE), especially for GC profiling.
The gc.jfc, which is a lightweight GC profiling. This can be used also for longer profiling runs and will provide general GC profiling data with limited details and small recording size.
The gc_details.jfc will record all GC events and details. This will have a higher impact caused by heap inspection initiated GCs to get e.g. heap statistics and have a large recording size.
How to do the JFR profiling:
In general there are three ways to do the recording:
- The Flight Recording can be enabled by JVM parameter to start this with the startup already(-XX:StartFlightRecording). Per option you can specify the filename to be used to save the recording and you can specify the settings/configuration file.
E.g. -XX:StartFlightRecording,filename=./my_recording.jfr,settings=gc_details.jfc - Start the Flight Recorder via jcmd. Connect to the running Java application(jcmd <main class|PID>) and start/dump/stop the recording by specific command(JFR.start, JFR.dump, JFR.stop). You can specify the recording name, settings(configuration), etc. Check the help of the commands to get all available options.
In SAP BTP you have to enable ssh in CF first to call the jcmd via cf ssh.
E.g.:cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) JFR.start name=gc_recording settings=gc_details.jfc filename=/home/vcap/tmp/gc.jfr"
- Connect with JMC to a running local or remote Java application to start/dump/stop the recording without restart. You have to open JMX protocol listener. This can be done with e.g. jcmd <main class|PID> ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555.
If you want to profile a Java application running behind a firewall you may need port forwarding.
In SAP BTP you have to start your app with -Djava.rmi.server.hostname=127.0.0.1 and start the Management Agent via cf ssh and use a ssh tunnel. E.g.:cf ssh <your app name> -c "app/META-INF/.sap_java_buildpack/sap_machine_jre/bin/jcmd $(pgrep java) ManagementAgent.start jmxremote.authenticate=false jmxremote.ssl=false jmxremote.port=5555 jmxremote.rmi.port=5555"
cf ssh <your app name> -N -T -L 5555:127.0.0.1:5555
Connect with JMC to 127.0.0.1:5555 to get the JMX Console or starting the Flight Recording.
Note:
Depending on what your SapMachine is using (JRE or JDK), specify the path accordingly (sap_machine_jre or sap_machine_jdk). To learn more, see: SapMachine.
How to analyze the JFR recording:
The collected jfr recording can be analyzed with JMC. Open the JFR recording file to get the Automated Analysis Results as a first overview.
Figure 1: Analysis overview
The provided findings can be expanded to get more details with hints for optimization.
Figure 2: Expanded analysis details with hints for optimization
In addition JMC is providing specific Outlines e.g. Java Application, JVM Internals, Environment and the Event Browser. The Outlines will show the specific data in tables or graphs.
In JVM Internals you will find the GC Summary with average, maximum, total GC times of the collections:
Figure 3: GC summary
The GC configuration details:
Figure 4: The GC configuration details
The Garbage Collections with the GC details:
Figure 5: Garbage collection details
Depending of the enabled events in the used JFR configuration, the table will show some or all the GC Phase Pause Levels with detailed pause time.
All collected events can be found in the Event Browser:
Figure 6: Event Browser
There you can also create your own specific outlines/pages for events your are interested in and no outline exists already. In addition you can create outlines/pages for your own custom events.
Conclusion
With the additional JFR configurations the SapMachine provides, general GC analysis with low overhead and also details GC analysis is possible out of the box as the configurations are delivered with the SapMachine.
In case of questions about the usage of JFR and JMC for GC analysis or other profiling don’t hesitate to contact me and any feedback about this post is very welcome.
References:
Ask questions about SAP Java Virtual Machine and follow (https://answers.sap.com/tags/01200615320800003576)
Read other SAP Java Virtual Machine blog posts and follow (https://blogs.sap.com/tags/01200615320800003576/)
Recommend
-
29
An OpenJDK release maintained and supported by SAP View on GitHub
-
4
``` docker pull sapmachine ``` # Quick reference - **Maintained by**: [The SapMachine Team](https://github.com/SAP/SapMachine) - **Where to get help**: send an email to [email protected] # Supported tags and respective `Dockerfile`...
-
9
Troubleshooting Rails and Sidekiq Feb 17, 2017 I recently had a problem with Rails and Sidekiq that confused the heck out of me for an embarrassingly long time so I thought I'...
-
10
Virtualization openSUSE 13.2: Day-1 troubles and troubleshooting - VLC, wireless etcetera ...
-
4
Virtualization Virtualbox on OpenSUSE 13.1: vboxdrv troubles and troubleshooting ...
-
5
A Lesson in DR, Azure Site Recovery, and Troubleshooting – The SQL HeraldSkip to content
-
7
The logging in Jersey, the reference JAX-RS implementation, is little sub-optimal. For example if it cannot find a method producing the expected MIME type then it will return "Unsupported mime type" to the client but won't log anything (which mi...
-
5
Troubleshooting And Improving HTTPS/TLS Connection Performance November 27, 2015 Our team has struggled with slow calls to th...
-
11
Sahil Grover April 22, 2022 2 minute read
-
2
Kubernetes troubleshooting: 6 ways to find and fix issues Try these troubleshooting methods the next time something is behaving u...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK