Ah Lam's Note

MTU is really a matter

2007-08-25T08:52:00.000+08:00

Long time ago, I lower the MTU of my router to solve a networking issue (see my previous blog entry). However, it is not the end of story.
Recently, I have changed my job. My new company provides Citrix Presentation Server for employees to work at home via remote desktop.
Then, at home, I tried to connect to Citrix server and run remote desktop. I can login but the link is very unstable. It disconnected every 10 to 20 seconds.
After some investigation, I found it is the MTU problem again. The problem is fully explained at the following site:
http://www.netheaven.com/pmtu.html
I should change the MTU of my desktop at home to match the router. Finally, it works!

Boost C++ Libraries

2007-07-01T09:59:00.000+08:00

Recently, I checked some job ad. on the web and found some jobs require knowledge about "Boost". I have not heard "Boost" before. After some googling, I visit the web site of Boost. Boost is really a great thing. It is a set of C++ libraries contains many useful programming constructs with good documentation, e.g. smart pointers, regular expression, object pools, state machines.... Lastly, it is open source and free.

Dig out dead looping thread

2007-06-17T11:27:00.000+08:00

It is not easy to write multi-threading applications. One of the common bugs is dead looping of a thread. To kill this kind of bugs, the first step is to find out which thread causes dead looping. However, an application may have dozens of threads. How to dig out the dead looping thread? My trick is to issue the ps -eLf command to list the information of threads in the whole system. The "time" column of the output of ps shows the CPU time have spent by the threads. Most likely, the dead looping thread would be the thread spending most CPU time. Then, drop down the LWP number of the thread. Next, you use gdb to attach the process and to debug the thread.

Job Interview Questions

2007-06-10T11:16:00.000+08:00

Recently, I got some job interviews. The interviewers asked me tones of questions. Most of them were very technical. The most difficult question was an IQ puzzle. Under stress, it is difficult to overclock my brain :<
At home, I found a web site which included the IQ puzzle:
http://www.techinterview.org/index.html

Undefined symbols in C++

2007-05-12T13:38:00.000+08:00

In C++ programming, we sometimes encounter "undefined symbols" problem during compilation or dlopen. The name of undefined symbols looks obfuscated. E.g.:

Unable to dlopen(test.so): test.so: Undefined symbol "_ZN6moduleD2Ev"

You may wonder why the symbol looks so ugly. Actually, this conversion of symbol is called name mangling. C++ supports polymorphism, this means functions can have same name but different types and numbers of parameters. Therefore, compiler cannot just use the function name as the symbol. Instead, both function name and parameter types should be included in symbol naming. Name mangling is the technique to encode a function name and parameter types into one symbol.

To translate the mangled symbols to more meaningful text, we can use the c++flit utility. E.g.

ahlam@oxygen:~$ c++filt _ZN6moduleD2Ev
module::~module()

Now, you know "_ZN6moduleD2Ev" is the destructor of class module.

More than "command not found"

2007-05-11T21:44:00.000+08:00

In old days, if I try to run a command which doesn't exist in my Linux box, it just prompts "command not found". Today, Ubuntu gives me a nice response:

ahlam@oxygen:/usr/bin$ cdecl
The program 'cdecl' can be found in the following packages:
* cutils
* cdecl
Try: sudo apt-get install <selected package>
Make sure you have the 'universe' component enabled
bash: cdecl: command not found

It's surprising!

Using new Chinese fonts from M$

2007-05-09T17:28:00.000+08:00

One of my friends told me that the Chinese fonts in M$ Vista is much more beautiful. So, I try to install the fonts to my Ubuntu box today. The new Chinese fonts called YaHei and JhengHei. If you don't have Vista, you can download from somewhere (:P). The screenshot below shows the result.

For details about how to install the new fonts, you can refer to http://imtx.cn/feed.php?go=entry_332

Upgrade to Ubuntu 7.04

2007-04-22T06:42:00.000+08:00

Yesterday, I follow the instruction here to upgrade my Ubuntu to 7.04. Unfortunately, the Update Manager crashes during package installation. To solve the problem, I use the following command:

dpkg --configure -a

Then, I reboot the Linux box. Some errors occurs. It fails to mount all partitions, excepts "/" partition. After some investigation, under the new kernel (2.6.20), IDE harddisks/CD-ROM are considered as SCSI. The following is the dmesg output:

ata1: PATA max UDMA/100 cmd 0x000101f0 ctl 0x000103f6 bmdma 0x0001f000 irq 14
ata2: PATA max UDMA/100 cmd 0x00010170 ctl 0x00010376 bmdma 0x0001f008 irq 15
scsi0 : ata_piix
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: ATA-6: WDC WD800BB-55HEA0, 13.03G13, max UDMA/100
ata1.00: 156301488 sectors, multi 16: LBA 
ata1.00: ata_hpa_resize 1: sectors = 156301488, hpa_sectors = 156301488
ata1.00: configured for UDMA/100
scsi1 : ata_piix
ATA: abnormal status 0x7F on port 0x00010177
ata2.01: ATAPI, max UDMA/33
ata2.01: configured for UDMA/33
scsi 0:0:0:0: Direct-Access     ATA      WDC WD800BB-55HE 13.0 PQ: 0 ANSI: 5
scsi 1:0:1:0: CD-ROM                     RW-481248        1.00 PQ: 0 ANSI: 5

Therefore, the entries in /etc/fstab should be changed from "/dev/hdan" to "/dev/sdan".

Know more about a process

2007-04-19T19:53:00.000+08:00

Today, I would like to introduce some ways to get more information of a process, and hope those can help in trouble-shooting and debugging.

List environment variables

To list the environment variables of a process you can issue the command:
ps ewww pid
E.g.:


ahlam@oxygen:~/test/malloc$ ps ewww 4752
PID TTY      STAT   TIME COMMAND
4752 pts/2    Ss+    0:00 bash USER=ahlam HOME=/home/ahlam DESKTOP_SESSION=default GDM_XSERVER_LOCATION=loca
l GTK_IM_MODULE=gcin LOGNAME=ahlam USERNAME=ahlam GDM_LANG=en_HK.UTF-8 PATH=/usr/local/sbin:/usr/local/bin:/u
sr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games DISPLAY=:0.0 LANG=en_HK.UTF-8 XMODIFIERS=@im=gcin XAUTHOR
ITY=/home/ahlam/.Xauthority SHELL=/bin/bash GDMSESSION=default QT_IM_MODULE=gcin PWD=/home/ahlam SSH_AUTH_SOC
K=/tmp/ssh-nRdglP4521/agent.4521 SSH_AGENT_PID=4563 DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-ix9My5o6
DJ,guid=8abd2646dc897f03d1cfb07e3a3d4b00 GTK_RC_FILES=/etc/gtk/gtkrc:/home/ahlam/.gtkrc-1.2-gnome2 SESSION_MA
NAGER=local/oxygen:/tmp/.ICE-unix/4521 GNOME_KEYRING_SOCKET=/tmp/keyring-fXmYul/socket GNOME_DESKTOP_SESSION_
ID=Default TERM=xterm COLORTERM=gnome-terminal WINDOWID=31457489

List out the threads of a process

ps ms pid
E.g:


ahlam@oxygen:~/test/malloc$ ps ms 4752
UID   PID          PENDING          BLOCKED          IGNORED           CAUGHT STAT TTY        TIME COMMAND
1000  4752 0000000000000000                -                -                - -    pts/2      0:00 bash
1000     - 0000000000000000 0000000000000000 0000000000384004 000000004b813efb Ss+  -          0:00 -

Print the memory map

To list the memory map of a process, you can use the pmap command. It lists out the mapped memory regions and the purpose of the memory regions. It also lists out the .so files loaded by the process.
pmap pid
E.g:


ahlam@oxygen:~/test/malloc$ pmap 4752
4752:   bash
08048000    644K r-x--  /bin/bash
080e9000     20K rw---  /bin/bash
080ee000   1804K rw---    [ anon ]
b7c94000     36K r-x--  /lib/tls/i686/cmov/libnss_files-2.4.so
b7c9d000      8K rw---  /lib/tls/i686/cmov/libnss_files-2.4.so
b7c9f000     32K r-x--  /lib/tls/i686/cmov/libnss_nis-2.4.so
b7ca7000      8K rw---  /lib/tls/i686/cmov/libnss_nis-2.4.so
b7ca9000     72K r-x--  /lib/tls/i686/cmov/libnsl-2.4.so
b7cbb000      8K rw---  /lib/tls/i686/cmov/libnsl-2.4.so
b7cbd000      8K rw---    [ anon ]
b7cbf000     28K r-x--  /lib/tls/i686/cmov/libnss_compat-2.4.so
b7cc6000      8K rw---  /lib/tls/i686/cmov/libnss_compat-2.4.so
b7cd7000    204K r----  /usr/lib/locale/en_HK.utf8/LC_CTYPE
b7d0a000      4K r----  /usr/lib/locale/en_HK.utf8/LC_NUMERIC
b7d0b000      4K r----  /usr/lib/locale/en_HK.utf8/LC_TIME
b7d0c000    860K r----  /usr/lib/locale/en_HK.utf8/LC_COLLATE
b7de3000      8K rw---    [ anon ]
b7de5000   1204K r-x--  /lib/tls/i686/cmov/libc-2.4.so
b7f12000      8K r----  /lib/tls/i686/cmov/libc-2.4.so
b7f14000      8K rw---  /lib/tls/i686/cmov/libc-2.4.so
b7f16000     12K rw---    [ anon ]
b7f19000      8K r-x--  /lib/tls/i686/cmov/libdl-2.4.so
b7f1b000      8K rw---  /lib/tls/i686/cmov/libdl-2.4.so
b7f1d000    220K r-x--  /lib/libncurses.so.5.5
b7f54000     32K rw---  /lib/libncurses.so.5.5
b7f5c000      4K rw---    [ anon ]
b7f5d000      4K r----  /usr/lib/locale/en_HK.utf8/LC_MONETARY
b7f5e000      4K r----  /usr/lib/locale/en_HK.utf8/LC_MESSAGES/SYS_LC_MESSAGES
b7f5f000      4K r----  /usr/lib/locale/en_HK.utf8/LC_PAPER
b7f60000      4K r----  /usr/lib/locale/en_HK.utf8/LC_NAME
b7f61000      4K r----  /usr/lib/locale/en_HK.utf8/LC_ADDRESS
b7f62000      4K r----  /usr/lib/locale/en_HK.utf8/LC_TELEPHONE
b7f63000      4K r----  /usr/lib/locale/en_HK.utf8/LC_MEASUREMENT
b7f64000     28K r--s-  /usr/lib/gconv/gconv-modules.cache
b7f6b000      4K r----  /usr/lib/locale/en_HK.utf8/LC_IDENTIFICATION
b7f6c000      8K rw---    [ anon ]
b7f6e000    100K r-x--  /lib/ld-2.4.so
b7f87000      8K rw---  /lib/ld-2.4.so
bfa1c000     88K rw---    [ stack ]
ffffe000      4K -----    [ anon ]
total     5528K

/proc/pid

In proc/pid directory, it contains much information of a process.


ahlam@oxygen:~/test/malloc$ ls -l /proc/4752
total 0
dr-xr-xr-x 2 ahlam ahlam 0 2007-04-19 20:17 attr
-r-------- 1 ahlam ahlam 0 2007-04-19 20:17 auxv
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 19:57 cmdline
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 cpuset
lrwxrwxrwx 1 ahlam ahlam 0 2007-04-19 18:20 cwd -> /home/ahlam/test/malloc
-r-------- 1 ahlam ahlam 0 2007-04-19 19:58 environ
lrwxrwxrwx 1 ahlam ahlam 0 2007-04-19 20:17 exe -> /bin/bash
dr-x------ 2 ahlam ahlam 0 2007-04-19 19:57 fd
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:06 maps
-rw------- 1 ahlam ahlam 0 2007-04-19 20:17 mem
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 mounts
-r-------- 1 ahlam ahlam 0 2007-04-19 20:17 mountstats
-rw-r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 oom_adj
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 oom_score
lrwxrwxrwx 1 ahlam ahlam 0 2007-04-19 20:17 root -> /
-rw------- 1 ahlam ahlam 0 2007-04-19 20:17 seccomp
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 smaps
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 19:57 stat
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 statm
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 19:57 status
dr-xr-xr-x 3 ahlam ahlam 0 2007-04-19 20:17 task
-r--r--r-- 1 ahlam ahlam 0 2007-04-19 20:17 wchan

For details, please man proc.

Optimistic memory allocation strategy in Linux

2007-04-15T14:27:00.000+08:00

If you know C programming language, you must know what is malloc. malloc is for dynamic memory allocation. In colleges, we learned that malloc should return NULL in case of out-of-memory. However, it is not the case in Linux. By default, Linux uses optimistic memory allocation strategy. Under this strategy, Linux assumes there always exists free memory. The memory region returns by malloc is not actually allocated until the process touches the memory region. This means the memory region returns by malloc may not be available. In case of out-of-memory, the OOM Killer in Linux will pick up one or more process to kill. This sounds strange!

Reference:
man malloc
http://linux-mm.org/OOM_Killer

Change the title of xterm

2007-04-13T09:53:00.000+08:00

In most modern Linux distribution, the title of terminal follows the working directory. The mechanism behind this feature is not simple. You can even customize the title. Please note that, in this article, it is assumed to use xterm and bash.

In xterm, the following escape sequence can change the title of terminal windows:

ESC]0;title_stringBEL

where ESC and BEL is 033 and 007 (in octal ASCII) respectively. For example, we can use the following echo command to set the terminal title as "Hello World.":

echo -ne "\033]0;Hello World.\007"

However, if you issue the above command in your terminal, the title may not be changed. It is because the PROMPT_COMMAND environment variable have already been defined for changing the terminal title. The value in PROMPT_COMMAND environment variable will be executed as command prior to issuing each prompt. In my terminal, the value of PROMPT_COMMAND is:

echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD/$HOME/~}\007"

In the above command, ${USER} and ${HOSTNAME} is the username and hostname respectively, while the ${PWD/$HOME/~} is the current working directory with $HOME abbreviated as ~. Therefore, in my terminal, the title will be changed to show username, hostname and current working directory, e.g.:

ahlam@oxygen: ~/download

To customize your terminal title, you should update the value of PROMPT_COMMAND, e.g.:

export PROMPT_COMMAND='echo -ne "\033]0;Hello World.\007"'

Reference:
How to change the title of an xterm
man bash

Adobe Reader crashes with SCIM

2007-03-31T21:45:00.000+08:00

I am using Ubuntu 6.10. Recently, I have installed Adobe Reader 7.0.9. However, Adobe Reader crashed at startup. In ubuntuguide.org , it mentions Adobe Reader won't work with SCIM. Finally, I found the solution -- scim bridge. To solve the problem, install the scim-bridge package and edit the "acroread" script to insert the line "GTK_IM_MODULE=scim-bridge" at the beginning of the script, like:


#!/bin/sh
#

GTK_IM_MODULE=scim-bridge
...

Then, Adobe Reader launches normally!

Be careful with STL strings

2007-03-10T08:33:00.000+08:00

Please refer to the following C++ code fragment:

    char charArray[4]={'a','b','c',0,};
    string str1(charArray);
    string str2;
    str2.append(charArray, 4);

Use the following lines to print out the contents of str1 and str2:

    cout << "str1 [" << str1 << ']' << endl;
    cout << "str2 [" << str2 << ']' << endl;

The output would be:

str1 [abc]
str2 [abc]

Two strings looks same. However, does str1 equal to str2?

    cout << (str1 == str2? "equal" : "not equal") << endl;

The output:

not equal

Why not!? Let's print out the size of the strings:

    cout << "str1 size = " << str1.size() << endl;
    cout << "str2 size = " << str2.size() << endl;

The output:

str1 size = 3
str2 size = 4

In short, we should be careful about the == operator of strings. It does not only compare the contents of strings, but it also compares the size of strings.

Does it really need to Lock?

2007-03-03T15:54:00.000+08:00

In multi-threading environment, we always face the problem of race condition -- concurrent accessing sharing data among threads. To solve the problem, we can use locking primitives, e.g. mutex, to avoid concurrent accessing of sharing data. However, those locking primitives are expensive, since they involve system calls. In some cases, we can avoid using locks.

Counting
Suppose threads updating a count variable concurrently. To avoid race condition, I saw some implementation like:

mutex.lock();
count++;
mutex.unlock();

Increment an integer just takes one CPU instruction, but locking and unlocking of mutex takes hundreds or thousands of CPU instructions. To avoid race condition, we can use atomic operations provided by CPU. Referring to /usr/include/asm-i386, there is implementation of atomic add:

static __inline__ void atomic_add(int i, atomic_t *v)
{
__asm__ __volatile__(
LOCK_PREFIX "addl %1,%0"
:"=m" (v->counter)
:"ir" (i), "m" (v->counter));
}

In Apache Portable Runtime project, it provides a set of atomic operations for different platforms.

Circular Buffers
In general, without locking, a circular buffer cannot be thread-safe. However, under a restricted condition and implementation, there would be no race condition problem. In short, for a fixed size circular buffer, if there is exactly one reader and one writer, it does not need locking. It is because the reader only updates the read-pointer and the writer only updates the write-pointer. This issue have been discussed in http://ddj.com/dept/cpp/184401814.

Is Port 25 blocked?

2007-02-04T10:35:00.000+08:00

Port 25, default port of SMTP, is for E-mail transferring. To reduce SPAM, ISPs normally block this port, especially for those household DSL accounts. Therefore, people may think that it is impossible to setup E-mail server at home.
I have experiences of using HKBN and Netvigator, they are not really block all port 25 traffics. They only block outgoing traffics, but not incoming traffic. To setup the E-mail server, we can use the "smarthost" trick.

Stack Size of Threads

2007-01-27T21:50:00.000+08:00

Several months ago, my boss assigned me to do performance tuning on several applications. One of them was a server application, which was for distributing data to clients. This application used a lot of threads, two threads per client -.-" It can support up to about 80 clients. If the number was over 80, the application generated a core dump. The core dump was due to out of memory. From top, I found the virtual memory size increases 4x MB for each newly connected client. It confused me. How could one client eat 4xMB?
Finally, I found the answer -- stack size of a thread. In the Linux system, the default stack size of a thread (in case of pthread library) is set to 20MB!
To change the stack size of a thread,

use pthread_attr_setstacksize (&attr, stacksize) to set the attribute during thread creation. For more details, you can refer to the following link: http://www.llnl.gov/computing/tutorials/pthreads/#Stack
use ulimit -s nnnn command to change the default stack size of pthread, where nnnn is the size in KBytes. (http://kbase.redhat.com/faq/FAQ_43_8710.shtm)

Find files by last modification time

2007-01-20T11:01:00.000+08:00

"find" is a powerful Unix/Linux utility for searching files. With the -mtime (or -mmin) options, "find" can search files which modified within a specific days (or minutes). However, there is no option for "find" to search files modified after a point of time. To do this, you need to write a shell script:

#!/bin/sh
touch -t "$1" /tmp/$$ \
&& find . -newer /tmp/$$ \
&& rm /tmp/$$

This script creates a temp file with a specific last modification time in /tmp directory and uses the temp file's last modification time as a reference time point for "find" to search files newer than the specific time. The -newer option of "find" is for searching files which are newer than a specific file. touch -t "$1" /tmp/$$ creates a temp file with a specific last modified time using the -t option.

Speed up DNS Lookup in Linux

2007-01-14T16:37:00.000+08:00

When I was first time to use Linux for web surfing, I found it was slower than MS Windows. Recently, I found the reason form the Internet. The difference is due to the DNS lookup speed. In Windows, it caches DNS lookup results but Linux doesn't. To improve the lookup speed in Linux, we can install dnsmasq, which is a light-weight DNS server for Linux.

Bufferring Behaviour of stdout(Stardard Out)

2007-01-01T13:24:00.000+08:00

Recently, I encounter a problem of capturing stdout of a program. The program is expected to run for a long time. When the output of the program is on the console, everything works fine. However, when the output of the program is redirected to a pipe or a file, the output is buffered for a long time. This means the output of the program is not appeared immediately. This behaviour is not desired. To illustrate the problem, I write the following simple program:

int main(int argc, char** argv)
{
    while (1)
    {
            sleep(1);
            printf("hello!\n");
    }
}

The program prints "hello!" for every second. However, the output will be buffered if it is
redirected to a pipe, like:
a.out | tee tmp.log
After some investigation, I found the behaviour is documented in setbuf(3) man page:

The three types of buffering available are unbuffered, block buffered, and line buffered. When an output stream is unbuffered, information appears on the destination file or terminal as soon as written; when it is block buffered many characters are saved up and written as a block; when it is line buffered characters are saved up until a newline is output or input is read from any stream attached to a terminal device (typically stdin).... Normally all files are block buffered. When the first I/O operation occurs on a file, malloc(3) is called, and a buffer is obtained. If a stream refers to a terminal (as stdout normally does) it is line buffered. The standard error stream stderr is always unbuffered by default....

There are two ways to solve the problem. First, call fflush(stdout) after printf.
Second, call setlinebuf(stdout) at the startup of the program.

A Strange ICQ Message

2006-12-26T10:05:00.000+08:00

Today, I received a very strange ICQ message from one of my friend:

Hi, you've just received a postcard.

To view the postcard click this link or copy it to your browser's address bar.
http://cards.vertionpilinhertunfendsa.com/1/show.xml?id=201e624becfd36e0b7663089da805de0

The postcard will be kept for 10 weeks.

The link in the message points to an executable file named "postcard_flash.exe". Why does a postcard need to be executable? It most likely is a virus or a worm. From the Internet, I found some information about this kind of virus:
http://www.sophos.com/security/analyses/w32mydoomal.html
A very "clever" worm which spreads via email and ICQ messages. The worm tries to disable the firewall, as well as anti-virus program. Moreover, the worm changes the HOST file to prevent the compromised computer connecting to anti-virus and security web sites.... So what! I am using Linux.

C Macro Tricks

2006-12-21T21:34:00.000+08:00

If you know C programming, you should know C macro as well. Macro is very powerful. However, it also makes the code difficult to read, especially macro with multi-level expansion. In complication of source code, the macro expansion is handled by "C Preprocessor". To do trouble-shooting on macro expansion, we can use the "-E" option in gcc. The "-E" option tells the compiler to stop after "C Preporcessor". With "-E" option the output file of gcc will be a text file containing the source code with all macro expanded and #include files merged.

gdb stops at SIGPIPE

2006-12-14T22:03:00.000+08:00

By default, gdb captures SIGPIPE of a process and pauses it. However, some program ignores SIGPIPE. So, the default behavour of gdb is not desired when debugging those program. To avoid gdb stopping in SIGPIPE, use the folloing command in gdb:

handle SIGPIPE nostop noprint pass

Installing Solaris 10 x86 in VMWare

2006-12-09T11:50:00.000+08:00

Yesterday, I tried to install Solaris 10 x86 in VMWare virtual machine. Overall, the installation was quite smooth. I faced a little problem in the harddisk partitioning stage. I selected "auto-layout" but failed at first time. Then, I went back one step and try "auto-layout" again and it worked for second time. Maybe there is a bug in the installation.
After installation, the resolution of X window was low. I had to run "/usr/X11/bin/xorgconfig" to config the resolution.
Next, I installed vmware-tool for Solaris. After installed the vmware-tool the NIC is changed from pcn0 to vmxnet0. So, I need to copy the hostname.pcn0 to hostname.vmxnet0 and dhcp.pcn0 to dhcp.vmxnet0.
Then, I tried to run mozilla to check the Internet connection. Unluckly, the browser always report unknown host. I have verified the "/etc/resolve.conf" for DNS setting. It is correct. I also tried to use "dig www.google.com" to resolve the IP address successfully. Finally, from forum of Internet, we had to modify the "/etc/nsswitch.conf" as well.... -.-"

Linux VPN Client for Cisco VPN Router

2006-12-08T14:09:00.000+08:00

The desktop of my office is in private network behind a firewall. If I need to access the desktop of my office, I need to use VPN by connecting the Cisco VPN Router in my office. It is not a trivial task to connecting VPN router in Linux platform, and there are seems several ways:

Use the official Cisco VPN Client for Linux. During the installation, it needs the kernel headers for module complication. It should work in most cases. However, this software package is not freely available and open source.
VPNC is an open source VPNC client target for cisco3000 VPN Concentrator. It works well for me. However, some features are not supported, e.g. re-keying, digital certificate authentication....
Openswan is an implementation of IPsec for Linux. It is not only a VPN client, it can act as VPN router. I found some document about how to use Openswan for connecting Cisco VPN router. However, it fails in my Ubuntu box :<.

There is some web pages mentions that the XAuth of Cisco VPN router is non-standard. No wonder open source software packages don't support it well.

Free Advertising for Your Blog

2006-12-08T13:39:00.000+08:00

Here is some suggestion for advertising your blog. First you can register your blog to some blog directories:

Second, you can reply posts in some popular newsgroups and forums. Include your blog's info in the signature of your replies.
Third, ask your friends to add the link of your blog in their homepage or blog.
The more web pages pointing to your blog, the higher ranking you can get in Google's search engine. To know the ranking of your homepage for some specific keywords in search engines, you can visit http://www.googlerankings.com/index.php. Now, it supports Google, MSN Search, Yahoo and Ask Search.

Ubuntu with Compiz

2006-12-01T15:57:00.000+08:00

Compiz is a window manager with amusing visual effects. You can find some screen captures in YouTube. Yesterday, I installed Compiz in my Ubuntu (Edgy) Linux and it works great. The following the screen capture of my desktop:

The following is the How-to about installing Compiz in Ubuntu:
http://gandalfn.wordpress.com/howto/howto-compiz-aiglx-on-edgy/

Rename a JPG File to Photo Taking Time

2006-11-26T10:12:00.000+08:00

If you use DC to take photos, the file names are in form of a prefix following by a sequence number. This form of file name is meaningless and not convenient for looking up. Recently, I found a tools called jhead which can rename JPG files to their photo taking time in a batch. To do the batch renaming, just issue a single command:
jhead -n%Y%m%d-%H%M%S *.jpg
jhead is more than a file rename util. It is a exif Jpeg header and thumbnail manipulator utitliy. Please visit jhead's web site for details: http://www.sentex.net/~mwandel/jhead/

STL list size() method is slow

2006-11-24T14:34:00.000+08:00

In STL list, the size() method is o(n), where n is number of elements in the list. The implementation of size() is by traversing the linked list and counting the nodes one by one. So, it sounds stupid.
I have written a simple program to test the performance of size() method in STL list and STL vector. For a STL list with 10M integers, it takes 0.17 sec. to get the size. However, for a STL vector with 10M integers, it takes 0.4 micro sec to get the size() in same machine.
There are some suggestions:

Use vector instead of list.
If the application need to check whether the list is empty or not, uses "list.emtpy()" instead of "list.size() != 0".
use an extra counter variable to counting the size of a list.

VMWare Server with ALSA DMix

2006-11-19T15:23:00.000+08:00

In Linux, the sound-card drivers normally do not support concurrent access. This means when you playing MP3, your instant-chat program (e.g. gaim) will be mute. To solve this problem, there are some projects -- ALSA DMix, ESD, ARTSD, JACK.... However, most of them are not compatible each other and the applications needs to use the API provided by the specific project.

In my Ubuntu desktop, I use ALSA DMix with ESD approach. Most applications should support ALSA or ESD. However, VMWare support neither of them. VMWare only use native /dev/dsp device for sounding. After some searching on the web, we can use the LD_PRELOAD libossa.so trick.

After half day of try-and-error, the following is the summary of how to make VMWare work with ALSA:

make sure you installed the package "olsa-oss".
add suid bit to /usr/lib/libaoss.so by command chmod +s /usr/lib/libaoss.so
rename the /usr/lib/vmware/bin/vmware-vmx to vmware-vmx-real
create a shell script /usr/lib/vmware/bin/vmware-vmx with the following contents

#!/bin/sh
LD_PRELOAD=libaoss.so exec /usr/lib/vmware/bin/vmware-vmx-real $@

make sure the shell script is executable (i.e. chmod a+x /usr/lib/vmware/bin/vmware-vmx-real)
If your guest OS's disk image file is "single growable virtual disk" type with size large than 2G, you should use the vmware-vdiskmanager to split it into several files. Otherwise, you will encounter the problem of "File too large".

Switch To Ubuntu

2006-11-10T13:46:00.000+08:00

I have been used Linux as my primary desktop OS since 2001. At that time, I used Mandrake because it was easy to install. Then I switched to Debain because of its good package management and package update. Today, I decided to switch to Ubuntu.

Ubuntu is based on Debian's package management and adds user-friendly installation. No wonder Ubuntu is the most popular Linux distribution (http://distrowatch.com).

Today, I tried to install Ubuntu and don't encounter any difficulty. It can auto detect and configure all my hardware (sound card, video card, network card....). So, everything works properly!

One year ago, I have tried to install Ubuntu and I found it was not mature. So, I remained to use Debian. Now, Ubuntu 6.10 is much better. I love it!

Does MTU matter?

2006-01-07T13:47:00.000+08:00

Recently, I have changed my ISP to HKBN. I chose the plan BB10 which provide 10Mb upload and download speed. This is much better than ADSL. For such high bandwidth, I want to use my NSLU2 Linux box to setup a web server at my home.

However, there is some problems. I found that not every machine can access my web server. Ellen using Pacific Supernet cannot access the web server. Besides, a friend of mine using Netvigator faces the same problem. However, the W3C's HTML validate can access my web server to validate my HTML files. Moreover, my company can also access the server.

This is really strange. Today, I try to tackle the problem. I tried to set the server in DMZ at my router but fail. Then, I look for other setting in the router's configuration. I find the item "MTU". MTU stands for "maximum transmission unit". Currently setting is 1500 which is the maximum allowed value. I try to change the value smaller to 1400. It works! However, I want to ask, "Does MTU matter?".

Install Ubuntu Linux Cont'

2005-12-29T13:46:00.000+08:00

Today, I go to fine tune the Ubuntu Linux in my desktop. One disadvantage of Ubuntu comparing with Debian is Ubuntu supports less packages officially. For example, the SCIM, a Input Method modules support many languages, is not included in Ubuntu.

To install the SCIM, I includes the "Universe" packages pool in my Ubuntu and install the SCIM. The "Universe" pool contains some extra packages which is not officially support by Ubuntu team. After install the SCIM, I try to run the "Setup" of SCIM. However it can't run properly -- seg. fault. As a Debian user, this problem becomes the weak points of Ubuntu. In Debian, it officially support much more packages than Ubuntu. I think Ubuntu need to put more effort to support more packages.

Finally, I decide to install a Debian on my desktop and give up the fine tuning the Ubuntu. The installation of Debian is more complicated than Ubuntu. Fortunately, I am a experienced Debian User.

After installation complete, the XWindow can start correctly. However, my sound card doesn't work. To solve this, I need to install ALSA package via the "aptitude" (Note: aptitude is a Debian package manager). Moreover, the middle button of my mouse doesn't work and this is similar to the case of Ubuntu. The solution was mentioned yesterday's BLOG entry. Then, I install the SCIM and other Chinese related packages. The SCIM works properly!

Install Ubuntu Linux

2005-12-28T13:44:00.000+08:00

According to DistroWatch, Ubuntu Linux is the most popular Linux distribution. Today, I want to experience how good it is.

The installation is quite simple and straight forward. It can detect all my hardware and can run up XWindow without any manual settings. My sound card and CD RW also works properly.

There is one problem on my mouse. The middle key doesn't works. With hours of investigation, I find the solution. The problem is due to the "psmouse" kernel module. It detects my mouse incorrectly. To solve the problem, the parameter "proto=imps" should be used while loading the kernel module. So, I create a file "psmouse" in "/etc/modprode.d" with the following contents:

options psmouse proto=imps

CSV File Format

2005-12-14T13:43:00.000+08:00

CSV stands for Comma Separated Values. It is a common file format for storing tabular data, e.g. spread sheet data. The file format of CSV is simple -- a text file, values are separated by comma (,) and rows are separated by newline. However, there is still some tricky in the format.

The tuck point is how to escape the comma in the values. If a value contains commas, the value should be quoted with double-quote ("). e.g.:

value one,"value two with ',' inside",value three

If a value contains a double-quote, the value should be quoted with double-quote and the double-quote in the value should be escaped with another double-quote. e.g.:

value one,"value two with '""' inside",value three

The specification of the CSV is described in RFC4180.

Even there is an RFC standard for CSV format, there are some deviation standard exists. In some applications, the comma and double-quote is escaped by a back-slash (\). Moreover, the character encoding is not stored in the file format and causes ambiguity. So, it is not trivial while using this format.

Missing Samba after Upgrade Packages

2005-11-24T13:39:00.000+08:00

My Old PC is a Debian Linux box. I have not upgraded the packages for a long time. Therefore, yesterday, I run Aptitude to do a mass updates. Today, I find the Samba is removed after the mass updates.

I run Aptitude to figure out the reason. I found the Samba has some broken dependency -- libdb4.1. Then, I search the libdb4.1 on the Web. In the unstable branch of Debian, it only supports AMD64 Architecture. Oh! Some packages in Debian is moving from unstable to stable branch.

So, I edit the "/etc/apt/sources.list" to add the stable repository:

deb http://ftp.hk.debian.org/debian/ stable main non-free contrib

Then run Aptitude and update the package list. The broken dependency problem solve and Samba can be installed.

Free Virtual Machine -- QEMU

2005-11-23T13:38:00.000+08:00

Long time ago, I have tried QEMU. I run QEMU in Debian and the guest OS is Win2000. It worked but it was very slow. Recently, I visit the site of QEMU and there is a QEMU Accelerator Module. According to the site, with the accelerator, QEMU's performance is comparable to commercial products.

I want to try QEMU in Windows XP. However, the accelerator for windows is alpha version. Anyway, I still want to try to download and run it on my Win XP. The guest OS is the sample Linux which can be download the QEMU's website. It crashes in LILO. Too bad!!!! After that, I try to run QEMU without the accelerator. It works fine. I think the accelerator for windows need to be improved.

Then, I try another accelerator module -- qvm86. This is a open source accelerator module. This module is even worse. It crashes my WinXP.

Trojan Found

2005-11-20T13:35:00.000+08:00

Two days ago, my sister reported to me that her computer had torjan. On one hand, IE has many many security bugs. On the other hand, she have not done Windows Update for a long long time. I guess the trojan was planted by visiting some bad web-site. Actually, I have told her don't use IE, use Firefox!

The trojan was found by AVG virus scanner. Actually, there are two trojans found: PSW.Generic.BWM and PSW.Generic.DAP. I search them in google, and nothing related found. The infected file are "WINDOWS\SYSTEM32\msd.dll" and "WINDOWS\SYSTEM32\roddll.dll".

For the filename "roddll.dll", I find some information on the Web. There is a trojan named "Troj/Ragrok-A". This trojan creates "WINDOWS\rundll132.exe" and the "roddll.dll" file. Moreover, it also creates a registry entry to execute the "rundll132.exe" in HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run. So, it is not difficult to remove.

For the "msd.dll", there is not much information find. Anyway, I inspect the HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run in the registry, I find a suspect entry: "WINDOWS\system\rundll32.exe". I search on Google by key words "rundll32.exe" and "trojan". I find that rundll32.exe file is located in the c:\windows\System32 folder. In other cases, rundll32.exe is a virus, spyware, trojan or worm! Then, I remove the entry in the registry, remove the "rundll32.exe" in "WINDOWS\SYSTEM", as well as the "msd.dll". Then, reboot. Everything works fine! The AGV scanner doesn't find any trojan.

Obfuscate the Spammer's Robot

2005-11-18T13:33:00.000+08:00

Most email users should have experience of receiving junk email. For me, my HongKong.com account, it receives around 100 junk email each day. This wastes our time to filter the email and makes us feel annoying.

Do you know how a spammer get your email addresses? Actually, your email addresses may already be put on the Web which can be accessed by the public. For example, in web-sites of many companies, they put contact lists containing email addresses of its staff. spammers use a program, called robot, to automatically retrieve email addresses on the Web. The spammers' robots move on the Web and try to get the email addresses. So, don't put your email addresses publicly.

However, sometimes you can't avoid it. So, it is suggested to post your email addresses on the Web in a obfuscated form. For example:

john [at] google [dot] com

The robot is a very simple program which can only recognize the email addresses in the form of "xxx@xxxx.xxx". Once the email address is transformed, the robot can't recognize it, but human can. However, it is not user-friendly. I suggest to use JavaScript to transform the the obfuscated email addresses back when mouse is over the email addresses. Here is a real example in my homepage: Contact. Such that, someone can copy and past the email address directly.

Get the Date of Tomorrow/Yesterday in Shell Script

2005-11-16T13:31:00.000+08:00

In shell script, we may need to get the date of tomorrow/yesterday. If you use GNU version of "date", it is a piece of cake:

date -d tomorrow +%Y-%m-%d

The above command will display the date of tomorrow in "yyyy-mm-dd" format. Similarly, to find the date of yesterday:

date -d yesterday +%Y-%m-%d

The power of the GNU date command is out of my imagination, see the following examples:

date -d "2 days ago"
date -d "3 weeks ago"
date -d "1 month ago"
date -d "3 days"
date -d "4 weeks"
date -d "2 month"
date -d "2 month 1 week"

However, if your "date" is non-GNU version (e.g. in Solaris), it doesn't support "-d" option. Then, there seems no simple command can do this. You may need to write Perl script or use other more powerful programming/scripting language to achieve same results.

Power of "find"

2005-11-15T13:30:00.000+08:00

"find" is a command line utility. Its basic usage is to find some files. e.g.

find . -name '*.html'

The above example is to find all HTML files under current directory.

While building a homepage, it is common to change HTML and graphic files to 0644 permission. We can use the "-exec" option of find to execute the "chmod"; command to change the permission of the files, e.g:

find . \( -name '*.html' -o -name '*.png' -o -name '*.jpg' \) -exec chmod 0644 {} \;

The parameters after "-exec" is the command to be executed. In the command, "{}" is replaced by the found file. The "\;" is the terminator of the command of "-exec".

Comments Submission

2005-11-14T13:28:00.000+08:00

I continued to my Blog development. In the comments text area of comments submission, my original implementation did NOT allow any HTML tag. It was mainly due to security reasons. However, this made the text of comments looks dull. I think it is better to allow some simple formatting tag e.g. , , ....

After did some survey on the web, I found the most popular technology is BBCode. BBCode defines a set of tags in form of "[xxx]" which will be converted to HTML tags. For example, "[b]bold text[/b]" is converted to "bold text". There are several free BBCode parsers available. It is convenient to deploy this technology.

One disadvantage of BBCode is not user-friendly. Users need to learn a new set of code. I thinks it is better to have a WYSIWYG editor. Unfortunately, I cannot find any web-based WYSIWYG editor for BBCode.

On the other hand, I found some open source web-based WYSIWYG HTML editors. It is great! I have tried the Cross-Browser Rich Text Editor and it works well in both Firefox and IE.

However, it is not secure to accept all HTML tags. On server side, it should filter out some dangerous tags, e.g. , .... There already exist some HTML filters. One of them is safehtml. Finally, I decided to use the WYSIWYG HTML editor plus HTML filter.

Java Generics

2005-10-31T13:26:00.000+08:00

J2SE 5.0 have been released for a long time. However, I didn't have time to study it. Today, I spend some time to have a look of it's new features. One new feature introduced in J2SE 5.0 is generics.

In Wikipedia, the term generics is defined as:

generics is a technique that allows one value to take different datatypes (so-called polymorphism) as long as certain contracts such as subtypes and signature are kept. The programming style emphasizing use of this technique is called generic.

For example, a List object can contains different elements String, Integer, etc. In generics, we can declare a List object only contains certain types in elements. E.g, List as a List object only contains String objects.

Generics is easily mixing up with the term template. In Java, the syntax of generics is similar to Template in C++. However, generics and Template are two different concept. Template is about code generation, and generics is about type checking. You can refer to a Wikipedia's article: Comparison of generics to templates.

With generics, it can avoid type-safe problem of collection framework of Java. However, the generics in Java is very complicated. It is not elegant. After you read the tutorial of generics, you may feel very confusing, and you may find that there are many exceptional cases need to take care when using generics in Java.

Fix the RAID

2005-10-30T13:20:00.000+08:00

Yesterday, I found that one of my harddisk in my Old PC has bad sectors. I decided not to replace the bad harddisk but to relocate the root partition to other free spaces in the harddisk. Originally, the RAID of root filesystem contains the hda8 and hdc8 partitions. The bad sectors were found in hda8. I try to search whether there exists unused partition on hda. I find hda7 is not used and the size matches the hdc8. To add hda7 to the RAID:

raidhotadd /dev/md0 /dev/hda7

From "/proc/mdstat", it shows hda7 is adding to md0. Wait for a while, mirroring of data from hdc8 to hda7 finished. I restart the PC.

After startup, it fails. The OS still try to use hda8 as the RAID. After some inspection, it is due to the configuration problem. First, it needs to correct the "/etc/mdadm/mdadm.conf" file. Second, it needs to regenerate the initrd image using the "mkinitrd" command. Lastly, it needs to correct the GURB config. Then, I restart te PC and this round, it works!

GIF vs PNG

2005-10-29T12:16:00.000+08:00

Today, I test my homepage with IE. IE cannot display transparent PNG graphics properly, and the background becomes grey in color.

Fortunately, I find a workaround on the Web -- http://homepage.ntlworld.com/bobosola/. It has details about the problem and how to solve it. In short, it uses javascript to patch all <img> tag of PNG graphic's CSS using the "filter:progid:DXImageTransform.Microsoft.AlphaImageLoader" of IE.

Actually, PNG has some advantages over GIF.

Max color depth of PNG is 24-bit but that of GIF is 8-bit.
PNG has higher compression rate than GIF.
PNG support alpha channel transparency but GIF only support binary transparency.

However, GIF supports animation but PNG not. This is a weak point of PNG.

Makefile to Build Homepage

2005-10-25T11:43:00.000+08:00

After study GNU Make yesterday, I write a generic Makefile for building my homepage today.

The the following is the Generic Makefile:

PHP=php

all: premake all_subdirs all_curdirpages postmake

premake:
 if test -x ./premake.sh; then ./premake.sh; fi

postmake:
 if test -x ./postmake.sh; then ./postmake.sh; fi

include Makefile.dep

all_subdirs: $(subdirs)
 for dir in $(subdirs); do (cd $$dir; make); done

all_curdirpages: $(pages)

$(filter %.html,$(pages)): %.html: %.tpl
 $(PHP)  $< > $@
 touch timestamp

clean:
 for dir in $(subdirs); do (cd $$dir; make clean); done
 rm -f *.html timestamp *.gen

.PHONY: all clean all_subdirs all_curdirpages premake postmake $(subdirs) $(dep_phony)

This Makefile is put (symbolic-link) in all sub-directories of the homepage. In other words, all sub-directories use the same Makefile. So, it is called generic Makefile. If it needs to add some extra logics before or after make, we can create "premake.sh" and "postmake.sh" shell scripts respectively. The scripts will be executed accordingly. In the Makefile, the marco "pages", "subdirs" and "dep_phony" are provided by the "Makefile.dep" The customization depends on the "Makefile.dep". In each directory, it should contain a "Makefile.dep". The following is a sample "Makefile.dep":

subdirs=blog secImage

pages=index.html tba.html contact.html

index.html: index.tpl header.tpl footer.tpl rightbar.tpl \
 lib/search.tpl lib/acknowledgement.tpl blog/recent/index.html

contact.html: contact.tpl header.tpl footer.tpl rightbar.tpl \
 lib/search.tpl lib/acknowledgement.tpl

tba.html: tba.tpl header.tpl footer.tpl

The "subdirs" macro defines the the sub-directories requiring to process, and the "pages" macro defines the HTML pages requiring to generated. The "Makefile.dep" file also defines the dependency of HTML files to the template files (*.tpl).

To build the homepage, I only need to issue the "make" command in the root directory of the homepage source.

Designing My Homepage

2005-10-24T13:22:00.000+08:00

I want to host my homepage in my NLSU2-Linux box. Today, it's the time to think about it. Since the NLSU2-Linux box is not a powerful machine, it is not appropriate to use Java technology. To put less load on the CPU, it should avoid using too much dynamic pages.

In my past experience, using static HTML, there are many duplication parts among different pages, for example, the page header, page footer, menus.... This make the web-site difficult to maintain. If I need to changes the page header, I need to edit nearly all HTML pages repeatedly. In programming, we avoid code duplication by using functions. How about make web pages? I think template is a solution.

In my homepage, I decided to use PHP as the template language. (To avoid confusion with PHP scripts for dynamic pages, the extension of template files is ".tpl") The following shows a typical page contains header and footer in PHP:

<? include 'header.tpl' ?>

.... the contents here .....

<? include 'footer.tpl' ?>

The "header.tpl" and "footer.tpl" are PHP scripts which generate the header and footer HTML fragments respectively. In PHP, the "include" function embedded the contents of the specific file in current pages, similar to the "include" directive in C/C++. In the homepage, every pages contains header and footer will be like the above example. If we want to change the header, we only need to change the "header.tpl" file once.

However, we need to convert the PHP file to HTML. For example, we have an "index.tpl" file which is a PHP script for generating "index.html". We can issue the following commands:

php index.tpl > index.html

For a complete homepage, there should be many ".tpl" files. I think nobody want to type the above command line again and again. GNU Make is a good utility to do this task.

Since I only have some experience to create a simple "Makefile", it is not enough for handling a complete homepage. I spent some hours to study the GNU Make and hope I can create a generic Makefile tomorrow.

Purchased Linksys NSLU2

2005-10-21T12:42:00.000+08:00

I want to setup a 7x24 server at home. The main purpose of this server is for hosting my homepage and files sharing with my friends. After some googling, I found the Linksys NSLU2 is quite suitable for this purposes.

NSLU2 is just a SOHO-grade network attached device with simple functions. However, it can be turned into a more powerful Linux box with the help of NSLU2-Linux project. Then, I can have a lower power consumption, ultra-slim and fanless server :P

Finally, I bought it at Golden Shopping Center. The following is a photo of my running NSLU2 : )

Start Hacking the NSLU2

2005-10-18T13:16:00.000+08:00

Today, I had annual leave. I had full day to hack my NSLU2 box. After read the wiki of NSLU2-Linux project, I decided to install the Unslung firmware. I followed the user guide and reflashed the firmware of the NSLU2 box. Everything worked fine. :)

Next, I setup the SSH, so I can remotely access the NSLU2 box. Then, PHP, Perl, and other useful utilities have been installed. I also setup a native compilation environment in the NSLU2 box, such that it can compile additional software packages.

Moreover, I tried to setup a cross compilation environment in my old PC which running Debian. This can help me to compile any additional packages faster. However, I fail to compile the cross compilation environment. Then, I read the guide in Wiki carefully, I found that I missed to install some necessary packages (flex, bison, rsync...). After installed the necessary packages, the cross compilation environment was compiled successfully.