Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save khatwaniNikhil/2398f456d593a98eb5abf5ce08e42f39 to your computer and use it in GitHub Desktop.
Save khatwaniNikhil/2398f456d593a98eb5abf5ce08e42f39 to your computer and use it in GitHub Desktop.
Java Native memory leak analysis
Native Leak example 1
******************************************************************************
https://technology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/
leak
1GB in java.util.zip.Inflater.inflateBytes -> inflate -> updatewindow.
solution
stopped the decompression of frontend assets by
1) Using an uncompressed jar (jars are effectively zip files and are compressed by default).
2) Serve assets from the nginx instance sitting in front of the frontend server.
******************************************************************************
Native leak example 2
******************************************************************************
https://www.evanjones.ca/java-native-leak-bug.html
an exception logger that was compressing messages without closing the GZIPOutputStream
faces issue only in case of multiple exceptions between gc cycles and OOM occurs
otherwise, eventually the finalizers would run to clean up the native allocations
solution
Adding a try/finally block to close the stream fixed the problem, and stabilized our service's memory usage under load.
steps to generate memory profile graphs via jemalloc and jeprof
******************************************************************************
sudo su
wget https://github.com/jemalloc/jemalloc/archive/5.2.0.zip
cd jemalloc-5.2.0
./autogen.sh
./configure --enable-prof
make
make install
// force our jvm to use jemalloc (instead of glibc’s malloc)
export LD_PRELOAD=/usr/local/lib/libjemalloc.so
// write a profile to the disk every few 1Gb allocations and record a stack trace
export MALLOC_CONF=prof:true,lg_prof_interval:30,lg_prof_sample:17
service tomcat start
jemalloc-5.2.0/bin/jeprof --show_bytes --svg `which java` jeprof*.heap > ~/test.svg
study the image file ...check for major memory consumtption areas like java.util.zip.Inflater.inflateBytes
Notes:
what is lg_prof_sample
Average interval (log base 2) between allocation samples, as measured in bytes of allocation activity. The default sample interval is 512 KiB
what is lg_prof_interval
Average interval (log base 2) between memory profile dumps, as measured in bytes runs o of allocation activity
******************************************************************************
TODO
******************************************************************************
1. Dry Runs on Beta - DONE
Found 74 MB memory consumption in java.util.zip.Inflater.inflateBytes. Refer Beta_env_jeprofile_output.svg on desktop
2.
https://stackoverflow.com/questions/53451103/java-using-much-more-memory-than-heap-size-or-size-correctly-docker-memory-limi/53624438#53624438
https://vimeo.com/364039638
https://blogs.oracle.com/poonam/troubleshooting-native-memory-leaks-in-java-applications
https://medium.com/swlh/native-memory-the-silent-jvm-killer-595913cba8e7
https://github.com/jeffgriffith/native-jvm-leaks
https://sleeplessinslc.blogspot.com/2014/08/jvm-native-memory-leak.html
***************
using async profiler with jemalloc
//install debug symbols
debuginfo-install java-1.8.0-openjdk
// use case, Java process RSS constant but os level free command free is continously reduced and buffer/cache continously reduced
// fix - log rotation was not setup in the sample code with infinite loop with logger line.
./profiler.sh -d 5 -e filemap:mm_filemap_add_to_page_cache -f cache.svg jps
// use jvm native leak, profiling mprotect(used to increase committed size, point to increase in memory allocation) instead of malloc (as malloc based allocations can be immediately freed but still shown in flame graph)
// fix Outputstream not closed
./profiler.sh -d 5 -e mprotect -f cache.svg jps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment