Memory management questions

https://access.redhat.com/solutions/58689

Memory management questions

 SOLUTION VERIFIED - Updated January 8 2019 at 7:47 PM - 

English 

Environment

• Red Hat Enterprise Linux (ALL)

Issue

• How do check how much memory is available for sizing purposes?
• How check how much memory is cached and/or by what processes? What process "owns" or reports the cache?
• Which figures has to be used when trying to determine how many more database instances can be added to a server or can be used for other processed or instance?
• The sum of memory reported by:

Raw

#sar -r (kbmemfree + kbmemused) = 49322992
#vmstat 3 3 (free + buff + cache) = 46767488  
#free (total)= 49322992  
#ps -aux RSS = 31090168 (used)

• Why does vmstat report different total memory than free and sar?
• How can there be ~31GB of memory in use by processes in the form of RSS AND ~38GB of FS(?) cache on a system with 48GB of RAM?
• Lets consider a scenario, Suppose there is a server that is currently running 3 oracle instances and want to add 2 more:
• In order to do this, how to be insured that there are enough resources available, particularly RAM ?
• As per the server's current state, free shows 44GB "freeable". Is that the number should be considered ?

Raw

 # free -g  
  total used free shared buffers cached  
 Mem: 47 41 6 0 0 37  
 -/+ buffers/cache: 2 44  
 Swap: 31 0 31  
# free  
  total used free shared buffers cached  
 Mem: 49322992 43075896 6247096 0 712936 39799284  
 -/+ buffers/cache: 2563676 46759316  
 Swap: 33554424 673416 32881008

 # vmstat 3 3  
 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------  
 r b swpd free buff cache si so bi bo in cs us sy id wa st  
 1 0 674544 6263992 701620 39795264 0 0 58 249 2 2 3 1 94 2 0  
 0 0 674540 6263076 701660 39795248 0 0 0 1605 1140 3846 7 3 88 2 0  
 0 0 674540 6263688 701660 39795268 0 0 0 89 1038 3824 0 0 99 1 0

 # sar -r  
 07:20:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad  
 07:30:01 AM 6416104 42906888 86.99 664508 39730540 32876180 678244 2.02 98260  
 07:40:01 AM 6412164 42910828 87.00 668628 39731476 32876316 678108 2.02 98244  
 07:50:01 AM 6408016 42914976 87.01 672636 39733016 32876508 677916 2.02 98204  
 08:00:01 AM 6397940 42925052 87.03 676484 39734048 32876524 677900 2.02 98188  
 08:10:01 AM 6392680 42930312 87.04 680520 39736572 32876900 677524 2.02 98396  
 08:20:01 AM 6347280 42975712 87.13 684280 39738236 32877084 677340 2.02 98448  
 08:30:01 AM 6340504 42982488 87.14 688400 39738932 32877084 677340 2.02 98448  
 08:40:01 AM 6334748 42988244 87.16 692524 39739840 32877128 677296 2.02 98412  
 08:50:01 AM 6324936 42998056 87.18 696772 39744052 32879644 674780 2.01 99164  
 09:00:01 AM 6313072 43009920 87.20 700884 39744648 32879644 674780 2.01 99164  
 Average: 6553141 42769851 86.71 605553 39652062 32850845 703579 2.10 108613

Resolution

How to check how much memory is available for sizing purposes ?
It can be figured out through output of command "free", In this case

Raw

 # free  
 total used free shared buffers cached  
 Mem: 49322992 43075896 6247096 0 712936 39799284  
 -/+ buffers/cache: 2563676 46759316  
 Swap: 33554424 673416 32881008  

The total physical memory(RAM) is 49322992KB.
The free physical memory (RAM) is swap+buffer+cached = 46759316 KB.
The total available free memory is 'swap+memory' , for more clarity it would be like 'free swap+ free memory(free memory+buffer+cached)
How to check how much memory is cached and/or by what processes? What process "owns" or reports the cache ?
How much memory is being used as "cache" is cached can be viewed by : # grep -iE "cache" /proc/meminfo
Caches are kernel structures used to improve overall system performance through utilizing memory smartly to avoid workload degradation.
While it is managed by the kernel, in general there's no single process 'owning' the cache, as it is a result of the whole system workload but in your case as your are using oracle that might be owning the cache. Caches are considered as 'system memory' and they are reported as "freeable" because they can be easily shrunk or reclaimed as a system's workload demands.
Which figures has to be used when trying to determine how many more database instances can be added to a server or can be used for other processed or instance ?
In a general scenario one can use the values from 'free', the amount of memory available would be 'free + buffer + cache'. If memory was the only constraint, one should consider avoid swapping out database buffer/cache as much as one can. As a rule of thumb, It is suggested not to use SGA size more that 70% of system memory. As you are using Oracle, it would be better to take an opinion from them.
**The sum of memory reported by: **

Raw

#sar -r (kbmemfree + kbmemused) = 49322992
#vmstat 3 3 (free + buff + cache) = 46767488  
#free (total)= 49322992  
#ps -aux RSS = 31090168 (used)

Why does vmstat report different total memory than free and sar ?
In 'vmstat' the addition of free + buff + cache shows the free memory available on the system which is approximately equal to 'free'. The values of free, buffer and cache respectively,

Raw

 from free = 6247096 +712936+39799284 =46759316  
 from vmstat = 6263992 +701620+39795264 =43760876  

which is approximately same. Also, its worth to mention that while "free" and "sar -r" reports the physical memory snapshot and "vmstat" reports information about processes, memory, paging, block IO, traps, and cpu activity instantaneously producing information on a sampling period of length delay. So, it is usual to observe minor discrepancies among their reported values, when one start to compare them in a timely manner.
How can there be ~31GB of memory in use by processes in the form of RSS AND ~38GB of FS(?) cache on a system with 48GB of RAM?
RSS does not render an accurate process' memory footprint, as it includes the shared memory segments (memory mappings, libs).
Lets a consider a scenario, Suppose there is a server that is currently running 3 oracle instances and want to add 2 more.
In order to do this, How to be insured that there are enough resources available, particularly RAM.
As per the server's current state, free shows that there should be 44GB "freeable". Is that the number we should use?
No, that figure shows the amount of memory potentially available. The free available memory would be 'free + buff + cache'.

But there is something tricky in this. One cannot just consider free + buffer + cache when it comes to Oracle database. Oracle uses SGA. Oracle SGA would keep all the shared memory in RAM, which will be unused by other applications. The SGA will be shown under "cached" in free output. The memory allocated for SGA under cache cannot be considered as available.
** There is a system and requirement for new DB instance with an expected total SGA of 8GB. Assume that Memory is the only constraint. How do we determine if there is enough RAM available?**
In general as a thumb of rule the SGA should not more than 70% of total physical memory available as concerning to operating system.
In normal case amount of available free memory would be : 'free + cached + buffer' but as this case represent 3 instances of Oracle on this system and it can be seen most of the memory is utilized in cache, It can not be calculated how much amount of memory is available/free as it will be depend upon size of SGA. Its better to take the suggestion from Oracle on this.
Does SGA resides in cache ? Does ALL of the SGA reside in cache or only a portion? And if only a portion, which portion?
Yes, All of the SGA resides in cache only.
Here's an example SGA, Which Components would be represented in cache?

Raw

SGA Component Current Allocation (MB)  
Shared Pool 444  
Buffer Cache 280  
Large Pool 4  
Java Pool 4  
Other 16

All these components are a part of cache memory.