How to Profile Memory in Linux.

Disclaimer:

I found this great article on a Linux mailing list. It is written by Jake Dawley-Carr and I honestly do not know what the copyright is on it but since it was posted on a mailing list, I assume it is in public domain. If the author contacts me, I will remove it but meanwhile, I would like to republish it because there is no knowing how long the article is going to be indexed and it would be too bad it to get lost.


HOWTO: Profile Memory in a Linux System

1. Introduction

It's important to determine how your system utilizes it's
resources. If your systems performance is unacceptable, it is
necessary to determine which resource is slowing the system
down. This document attempts to identify the following:

a. What is the system memory usage per unit time?
b. How much swap is being used per unit time?
c. What does each process' memory use look like over time?
d. What processes are using the most memory?

I used a RedHat-7.3 machine (kernel-2.4.18) for my experiments,
but any modern Linux distribution with the commands "ps" and
"free" would work.

2. Definitions

RAM (Random Access Memory) - Location where programs reside when
they are running. Other names for this are system memory or
physical memory. The purpose of this document is to determine if
you have enough of this.

Memory Buffers - A page cache for the virtual memory system. The
kernel keeps track of frequently accessed memory and stores the
pages here.

Memory Cached - Any modern operating system will cache files
frequently accessed. You can see the effects of this with the
following commands:

for i in 1 2 ; do
free -o
time grep -r foo /usr/bin >/dev/null 2>/dev/null
done

Memory Used - Amount of RAM in use by the computer. The kernel
will attempt to use as much of this as possible through buffers
and caching.

Swap - It is possible to extend the memory space of the computer
by using the hard drive as memory. This is called swap. Hard
drives are typically several orders of magnitude slower than RAM
so swap is only used when no RAM is available.

Swap Used - Amount of swap space used by the computer.

PID (Process IDentifier) - Each process (or instance of a running
program) has a unique number. This number is called a PID.

PPID (Parent Process IDentifier) - A process (or running program)
can create new processes. The new process created is called a
child process. The original process is called the parent
process. The child process has a PPID equal to the PID of the
parent process. There are two exceptions to this rule. The first
is a program called "init". This process always has a PID of 1 and
a PPID of 0. The second exception is when a parent process exit
all of the child processes are adopted by the "init" process and
have a PPID of 1.

VSIZE (Virtual memory SIZE) - The amount of memory the process is
currently using. This includes the amount in RAM and the amount in
swap.

RSS (Resident Set Size) - The portion of a process that exists in
physical memory (RAM). The rest of the program exists in swap. If
the computer has not used swap, this number will be equal to
VSIZE.

3. What consumes System Memory?

The kernel - The kernel will consume a couple of MB of memory. The
memory that the kernel consumes can not be swapped out to
disk. This memory is not reported by commands such as "free" or
"ps".

Running programs - Programs that have been executed will consume
memory while they run.

Memory Buffers - The amount of memory used is managed by the
kernel. You can get the amount with "free".

Memory Cached - The amount of memory used is managed by the
kernel. You can get the amount with "free".

4. Determining System Memory Usage

The inputs to this section were obtained with the command:

free -o

The command "free" is a c program that reads the "/proc"
filesystem.

There are three elements that are useful when determining the
system memory usage. They are:

a. Memory Used
b. Memory Used - Memory Buffers - Memory Cached
c. Swap Used

A graph of "Memory Used" per unit time will show the "Memory Used"
asymptotically approach the total amount of memory in the system
under heavy use. This is normal, as RAM unused is RAM wasted.

A graph of "Memory Used - Memory Buffered - Memory Cached" per
unit time will give a good sense of the memory use of your
applications minus the effects of your operating system. As you
start new applications, this value should go up. As you quit
applications, this value should go down. If an application has a
severe memory leak, this line will have a positive slope.

A graph of "Swap Used" per unit time will display the swap
usage. When the system is low on RAM, a program called kswapd will
swap parts of process if they haven't been used for some time. If
the amount of swap continues to climb at a steady rate, you may
have a memory leak or you might need more RAM.

5. Per Process Memory Usage

The inputs to this section were obtained with the command:

ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid

The command "ps" is a c program that reads the "/proc"
filesystem.

There are two elements that are useful when determining the per
process memory usage. They are:

a. RSS
b. VSIZE

A graph of RSS per unit time will show how much RAM the process is
using over time.

A graph of VSIZE per unit time will show how large the process is
over time.

6. Collecting Data

a. Reboot the system. This will reset your systems memory use

b. Run the following commands every ten seconds and redirect the
results to a file.

free -o
ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid

c. Do whatever you normally do on your system

d. Stop logging your data

7. Generate a Graph

a. System Memory Use

For the output of "free", place the following on one graph

1. X-axis is "MB Used"

2. Y-axis is unit time

3. Memory Used per unit time

4. Memory Used - Memory Buffered - Memory Cached per unit time

5. Swap Used per unit time

b. Per Process Memory Use

For the output of "ps", place the following on one graph

1. X-axis is "MB Used"

2. Y-axis is unit time

3. For each process with %MEM > 10.0

a. RSS per unit time

b. VSIZE per unit time

8. Understand the Graphs

a. System Memory Use

"Memory Used" will approach "Memory Total"

If "Memory Used - Memory Buffered - Memory Cached" is 75% of
"Memory Used", you either have a memory leak or you need to
purchase more memory.

b. Per Process Memory Use

This graph will tell you what processes are hogging the
memory.

If the VSIZE of any of these programs has a constant, positive
slope, it may have a memory leak.

Reference : http://www.freshblurbs.com/how-profile-memory-linux

HIstory of Google in 2 minutes !

Just found this video on youtube. Its nicely developed to showcase the history of google. Have a look at it.

Apayee Linux !

Hello All,

I have been in the process making an linux distribution for the server environment. I have been making some progress in what ever time I get. As part of this project, I would like to understand the ways that we can customize the Linux and then I would be working on the customization of the linux kernel. As of now, I have compiled the linux from sources and booted with virtual box. Now, the distribution is setup with command line and networking support. My next task would be creating an live cd out of this. You can find below the screencast of Apayee Linux running on virtual box.




For more information, please visit http://www/apayee.com


Thanks and Regards,
Bala

GUIDE to USING OPEN-SOURCE SOFTWARE to DEVELOP WEB APPLICATIONS - Sun Micro Systems

You can successfully deliver more web services — despite tight budgets and reduced resources — by using open source software and applying proven best practices. Open source software can help your enterprise build and deploy rich Internet applications safely, affordably, and quickly — if done right.

Our complimentary guide will arm you with the knowledge you need to determine whether you should consider open source for your application platform and to understand how it can reduce your platform costs and complexities.

In 11 pages, this free guide describes how using open source will:

» Lower your risk as Web application implementations grow
» Reduce operational costs
» Speed exploration/trial and deployment of new software

Reference : https://www.sun.com/offers/docs/open_web_platform.pdf

IE within Firefox ? Try IE Tab Today !

IE Tab allows users to view pages using the Internet Explorer rendering engine from within Firefox. This may be useful for viewing pages that only work in Internet Explorer (e.g. Windows Update) without exiting Firefox. Pages viewed through the IE Tab extension will be recorded in Internet Explorer's history, cache, and so on, as if they had been viewed directly in Internet Explorer. The extension has become popular among web developers, since they can display and compare their websites in the two browsers simultaneously.

Download and try it from here. (https://addons.mozilla.org/en-US/firefox/addon/1419)


Screenshots :







Reference : http://ietab.mozdev.org/

Creating Tiny ELF Executables For Linux !

Recently got a nice article link from my friend.

If you're a programmer who's become fed up with software bloat, then may you find herein the perfect antidote.

This document explores methods for squeezing excess bytes out of simple programs. (Of course, the more practical purpose of this document is to describe a few of the inner workings of the ELF file format and the Linux operating system. But hopefully you can also learn something about how to make really teensy ELF executables in the process.)

Please note that the information and examples given here are, for the most part, specific to ELF executables on a Linux platform running under an Intel-386 architecture. I imagine that a good bit of the information is applicable to other ELF-based Unices, but my experiences with such are too limited for me to say with certainty.

Please also note that if you aren't a little bit familiar with assembly code, you may find parts of this document sort of hard to follow. (The assembly code that appears in this document is written using Nasm; see http://www.nasm.us/.)

---

In order to start, we need a program. Almost any program will do, but the simpler the program the better, since we're more interested in how small we can make the executable than what the program does.

Let's take an incredibly simple program, one that does nothing but return a number back to the operating system. Why not? After all, Unix already comes with no less than two such programs: true and false. Since 0 and 1 are already taken, we'll use the number 42.

So, here is our first version:

  /* tiny.c */
 int main(void) { return 42; }

which we can compile and test like so:

  $ gcc -Wall tiny.c
 $ ./a.out ; echo $?
 42

So. How big is it? Well, on my machine, I get:

  $ wc -c a.out
    3998 a.out

(Yours will probably differ some.) Admittedly, that's pretty small by today's standards, but it's almost certainly bigger than it needs to be.

The obvious first step is to strip the executable:

  $ gcc -Wall -s tiny.c
 $ ./a.out ; echo $?
 42
 $ wc -c a.out
    2632 a.out

That's certainly an improvement. For the next step, how about optimizing?

  $ gcc -Wall -s -O3 tiny.c
 $ wc -c a.out
    2616 a.out

That also helped, but only just. Which makes sense: there's hardly anything there to optimize.

It seems unlikely that there's much else we can do to shrink a one-statement C program. We're going to have to leave C behind, and use assembler instead. Hopefully, this will cut out all the extra overhead that C programs automatically incur.

So, on to our second version. All we need to do is return 42 from main(). In assembly language, this means that the function should set the accumulator, eax, to 42, and then return:

  ; tiny.asm
 BITS 32
 GLOBAL main
 SECTION .text
 main:
               mov     eax, 42
               ret

We can then build and test like so:

  $ nasm -f elf tiny.asm
 $ gcc -Wall -s tiny.o
 $ ./a.out ; echo $?
 42

(Hey, who says assembly code is difficult?) And now how big is it?

  $ wc -c a.out
    2604 a.out

Looks like we shaved off a measly twelve bytes. So much for all the extra overhead that C automatically incurs, eh?

..................

Please continue reading from below :

Reference : http://www.muppetlabs.com/~breadbox/software/tiny/teensyps.html

Apayee User Group

I have started a new user group and it can be accessed over here. (http://www.apayee.com)