【第22期】观点:IT 行业加班,到底有没有价值?

Java程序在linux系统下运行,为什么会占用很多虚拟内存?

转载 2017年01月02日 23:41:59

转自:Virtual Memory Usage from Java under Linux, too much memory used

Question:

I have a problem with a Java application running under Linux.

When I launch the application, using the default maximum heap size (64mb), I see using the tops application that 240 MB of virtual Memory are allocated to the application. This creates some issues with some other software on the computer, which is relatively resource-limited.

The reserved virtual memory will not be used anyway, as far as I understand, because once we reach the heap limit an OutOfMemoryError is thrown. I ran the same application under windows and I see that the Virtual Memory size and the Heap size are similar.

Is there anyway that I can configure the Virtual Memory in use for a Java process under Linux?

Edit 1: The problem is not the Heap. The problem is that if I set a Heap of 128M, for example, still linux allocates 210 MB of Virtual Memory, which is not needed, ever.**

Edit 2: Using ulimit -v allows limiting the amount of virtual memory. If the size set is below 204 MB, then the application won't run even though it doesn't need 204MB, only 64MB. So I want to understand why java requires so much virtual memory. Can this be changed?

Edit 3: There are several other applications running in the system, which is embedded. And the system does have a virtual memory limit. (from comments, important detail)


Answer:

This has been a long-standing complaint with Java, but it's largely meaningless, and usually based on looking at the wrong information. The usual phrasing is something like "Hello World on Java takes 10 megabytes! Why does it need that?" Well, here's a way to make Hello World on a 64-bit JVM claim to take over 4 gigabytes ... at least by one form of measurement.

java -Xms1024m -Xmx4096m com.example.Hello

Different Ways to Measure Memory

On Linux, the top command gives you several different numbers for memory. Here's what it says about the Hello World example:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2120 kgregory  20   0 4373m  15m 7152 S    0  0.2   0:00.10 java
  • VIRT is the virtual memory space: the sum of everything in the virtual memory map (see below). It is largely meaningless, except when it isn't (see below).
  • RES is the resident set size: the number of pages that are currently resident in RAM. In almost all cases, this is the only number that you should use when saying "too big." But it's still not a very good number, especially when talking about Java.
  • SHR is the amount of resident memory that is shared with other processes. For a Java process, this is typically limited to shared libraries and memory-mapped JARfiles. In this example, I only had one Java process running, so I suspect that the 7k is a result of libraries used by the OS.
  • SWAP isn't turned on by default, and isn't shown here. It indicates the amount of virtual memory that is currently resident on disk, whether or not it's actually in the swap space. The OS is very good about keeping active pages in RAM, and the only cures for swapping are (1) buy more memory, or (2) reduce the number of processes, so it's best to ignore this number.

The situation for Windows Task Manager is a bit more complicated. Under Windows XP, there are "Memory Usage" and "Virtual Memory Size" columns, but the official documentation is silent on what they mean. Windows Vista and Windows 7 add more columns, and they're actually documented. Of these, the "Working Set" measurement is the most useful; it roughly corresponds to the sum of RES and SHR on Linux.

Understanding the Virtual Memory Map

The virtual memory consumed by a process is the total of everything that's in the process memory map. This includes data (eg, the Java heap), but also all of the shared libraries and memory-mapped files used by the program. On Linux, you can use the pmap command to see all of the things mapped into the process space (from here on out I'm only going to refer to Linux, becuase it's what I use; I'm sure there are equivalent tools for Windows). Here's an excerpt from the memory map of the "Hello World" program; the entire memory map is over 100 lines long, and it's not unusual to have a thousand-line list.

0000000040000000     36K r-x--  /usr/local/java/jdk-1.6-x64/bin/java
0000000040108000      8K rwx--  /usr/local/java/jdk-1.6-x64/bin/java
0000000040eba000    676K rwx--    [ anon ]
00000006fae00000  21248K rwx--    [ anon ]
00000006fc2c0000  62720K rwx--    [ anon ]
0000000700000000 699072K rwx--    [ anon ]
000000072aab0000 2097152K rwx--    [ anon ]
00000007aaab0000 349504K rwx--    [ anon ]
00000007c0000000 1048576K rwx--    [ anon ]
...
00007fa1ed00d000   1652K r-xs-  /usr/local/java/jdk-1.6-x64/jre/lib/rt.jar
...
00007fa1ed1d3000   1024K rwx--    [ anon ]
00007fa1ed2d3000      4K -----    [ anon ]
00007fa1ed2d4000   1024K rwx--    [ anon ]
00007fa1ed3d4000      4K -----    [ anon ]
...
00007fa1f20d3000    164K r-x--  /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
00007fa1f20fc000   1020K -----  /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
00007fa1f21fb000     28K rwx--  /usr/local/java/jdk-1.6-x64/jre/lib/amd64/libjava.so
...
00007fa1f34aa000   1576K r-x--  /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3634000   2044K -----  /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3833000     16K r-x--  /lib/x86_64-linux-gnu/libc-2.13.so
00007fa1f3837000      4K rwx--  /lib/x86_64-linux-gnu/libc-2.13.so
...

A quick explanation of the format: each row starts with the virtual memory address of the segment. This is followed by the segment size, permissions, and the source of the segment. This last item is either a file or "anon", which indicates a block of memory allocated via mmap.

Starting from the top, we have

  • The JVM loader (ie, the program that gets run when you type java). This is very small; all it does is load in the shared libraries where the real JVM code is stored.
  • A bunch of anon blocks holding the Java heap and internal data. This is a Sun JVM, so the heap is broken into multiple generations, each of which is its own memory block. Note that the JVM allocates virtual memory space based on the -Xmx value; this allows it to have a contiguous heap. The -Xms value is used internally to say how much of the heap is "in use" when the program starts, and to trigger garbage collection as that limit is approached.
  • A memory-mapped JARfile, in this case the file that holds the "JDK classes." When you memory-map a JAR, you can access the files within it very efficiently (versus reading it from the start each time). The Sun JVM will memory-map all JARs on the classpath; if your application code needs to access a JAR, you can also memory-map it.
  • Per-thread data for two threads. The 1M block is a thread stack; I don't know what goes into the 4K block. For a real app, you will see dozens if not hundreds of these entries repeated through the memory map.
  • One of the shared libraries that holds the actual JVM code. There are several of these.
  • The shared library for the C standard library. This is just one of many things that the JVM loads that are not strictly part of Java.

The shared libraries are particularly interesting: each shared library has at least two segments: a read-only segment containing the library code, and a read-write segment that contains global per-process data for the library (I don't know what the segment with no permissions is; I've only seen it on x64 Linux). The read-only portion of the library can be shared between all processes that use the library; for example, libc has 1.5M of virtual memory space that can be shared.

When is Virtual Memory Size Important?

The virtual memory map contains a lot of stuff. Some of it is read-only, some of it is shared, and some of it is allocated but never touched (eg, almost all of the 4Gb of heap in this example). But the operating system is smart enough to only load what it needs, so the virtual memory size is largely irrelevant.

Where virtual memory size is important is if you're running on a 32-bit operating system, where you can only allocate 2Gb (or, in some cases, 3Gb) of process address space. In that case you're dealing with a scarce resource, and might have to make tradeoffs, such as reducing your heap size in order to memory-map a large file or create lots of threads.

But, given that 64-bit machines are ubiquitous, I don't think it will be long before Virtual Memory Size is a completely irrelevant statistic.

When is Resident Set Size Important?

Resident Set size is that portion of the virtual memory space that is actually in RAM. If your RSS grows to be a significant portion of your total physical memory, it might be time to start worrying. If your RSS grows to take up all your physical memory, and your system starts swapping, it's well past time to start worrying.

But RSS is also misleading, especially on a lightly loaded machine. The operating system doesn't expend a lot of effort to reclaiming the pages used be a process. There's little benefit to be gained by doing so, and the potential for an expensive page fault if the process touches the page in the future. As a result, the RSS statistic may include lots of pages that aren't in active use.

Bottom Line

Unless you're swapping, don't get overly concerned about what the various memory statistics are telling you. With the caveat that an ever-growing RSS may indicate some sort of memory leak.

With a Java program, it's far more important to pay attention to what's happening in the heap. The total amount of space consumed is important, and there are some steps that you can take to reduce that. More important is the amount of time that you spend in garbage collection, and which parts of the heap are getting collected.

Accessing the disk (ie, a database) is expensive, and memory is cheap. If you can trade one for the other, do so.


举报

相关文章推荐

Java虚拟内存

可以给Java虚拟机设置使用的内存,但是如果你的选择不对的话,虚拟机不会补偿。可通过命令行的方式改变虚拟机使用内存的大小。如下表所示有两个参数用来设置虚拟机使用内存的大小。  参数描述  -Xms...

Linux笔记

Linux操作系统学习 linux的特点 优点: 1、免费的/开源的系统 2、支持多线程/多用户的系统 3、安全性好 4、对内存和文件管理优越 linux体积...

欢迎关注CSDN程序人生公众号

关注程序员生活,汇聚开发轶事。

java 虚拟机内存设置和性能调优

默认的java虚拟机的大小比较小,在对大数据进行处理时java就会报错:java.lang.OutOfMemoryError。设置jvm内存的方法,对于单独的.class,可以用下面的方法对Test运行时的jvm内存进行设置。java -Xms64m -Xmx256m Test-Xms是设置内存初始化的大小-Xmx是设置最大能够使用内存的大小(最好不要超过物理内存大小)在weblogic中,可以在startweblogic.cmd中对每个domain虚拟内存的大小进

Linux 多线程VS多进程

鱼还是熊掌:浅谈多进程多线程的选择 关于多进程和多线程,教科书上最经典的一句话是“进程是资源分配的最小单位,线程是CPU调度的最小单位”,这句话应付考试基本上够了,但如果在工作中遇到类似的选择问题,...

JVM内存的调优

一.JVM内存的设置的原理 默认的java虚拟机的大小比较小,在对大数据进行处理时java就会报错:java.lang.OutOfMemoryError。设置jvm内存的方法,对于单独的.class,可以用下面的方法对Test运行时的jvm内存进行设置。 java -Xms64m -Xmx256m Test -Xms是设置内存初始化的大小 -Xmx是设置最大能够使用内存的大小(最好不要超过物理内存大小) 在weblogic中,可以在startweblogic.cmd中对每个domain虚拟内存的大小进行设置,默认的设置是在commEnv.cmd里面。 -vmargs -X

linux服务器多线程还是多进程的选择及区别

鱼还是熊掌:浅谈多进程多线程的选择 关于多进程和多线程,教科书上最经典的一句话是“进程是资源分配的最小单位,线程是CPU调度的最小单位”,这句话应付考试基本上够了,但如果在工作中遇到类似的选择问题,...
  • zmbbb
  • zmbbb
  • 2016-03-17 13:49
  • 1152

java 虚拟机内存设置和性能调优

本文来源:http://zhli986-yahoo-cn.iteye.com/blog/1149233   默认的java虚拟机的大小比较小,在对大数据进行处理时java就会报错:java.lang.OutOfMemoryError。设置jvm内存的方法,对于单独的.class,可以用下面的方法对Test运行时的jvm内存进行设置。java -Xms64m -Xmx256

linux状态及原理全剖析

linux Table of Contents 1 linux 1.1 proc filesystem 1.1.1 /proc 1.1.1.1 /proc/meminfo...

Windows和Linux系统下的JAVA程序运行

最近做了一个项目,客户要求写个bat能在window下直接运行,还得能在Linux下运行,当然Linux下不可能是bat文件了。客户也是搞开发的,他是只是外包了一部分让我们做,也许是觉得HttpUnit的东西他们做不如外包的化算。Window下写bat运行到很简单,只要在DOS操作能成功运行就OK了,偶是新手,没有太深入学习过JAVA,运行时需要添加很多包,靠靠的偶都不会,后来测试了N多偏被偶搞定,哄哄。JAVA命令中 -classpath即是引入包,如果一个工程运行时需要包含其它包的话可以这样写:"java -classpath .;..\lib\XXX.jar; javafilena

【经典】《Java170道面试笔试题全面含答案》涉及java/数据库/框架/系统/设计模式相关

《Java170道面试题全集》 1、面向对象的特征有哪些方面?  答:面向对象的特征主要有以下几(四)个方面:  - 抽象:抽象是将一类对象的共同特征总结出来构造类的过程,包括数据抽象和行为抽...
收藏助手
不良信息举报
您举报文章:深度学习:神经网络中的前向传播和反向传播算法推导
举报原因:
原因补充:

(最多只允许输入30个字)