Sunday, February 05, 2017

Recovery Ubuntu boot entry in UEFI after upgraded motherboard firmware

Nowadays, UEFI is commonly used to replace the BIOS. UEFI doesn't use boot sector in disks. Instead, UEFI comes with a boot manager. During installation of an OS, e.g. Ubuntu Linux, the OS installer usually setup the boot loader (e.g. GRUB) in EFI partition and update the boot config. The boot config provides information for boot manager to locate the boot loader. The tricky part is the boot config is stored in NVRAM on the motherboard. During firmware upgrade, NVRAM can be reset. This means the boot config will also be reset and lost the boot loader information.
I encountered this issue when I upgrade the firmware of my Xubuntu box. Now, I would like to show you how to recovery the boot config. This demo is using Ubuntu 16.04 on Virtual Box.

  1.  Get an installation CD or a flash drive with installer of Ubuntu 16.04 (You should have it when you install the Ubuntu first time). If you don't have it, get it from here.
  2.  Boot up the system using the CD or flash drive. In GRUB, select "Try Ubuntu without installation"
  3. Then, you should be able to boot into a fresh Ubuntu, like
  4. Open a terminal and run "sudo su" to switch to root user.
  5. Now, we need to know where the EFI partition located. We can use "fdisk -l /dev/xxx" command to list out the paration, like:
    Refer to the above screen, we know that the EFI partition is on the first partition of device "sda" (i.e. /dev/sda1). (Note: the actual location of the EFI partition on your box can be different from this demo.)
  6. Then, we mount the EFI partition to verify if the Ubuntu GRUB loader is installed. To do this, mount the drive using command "udiskctrl mount -b /dev/xxx" and check if the EFI partition contains "EFI/ubuntu/shimx64.efi" file. E.g.:
  7. After the above verification, we have enough information to create the add the boot entry using the command:
    efibootmgr -c -d /dev/xxx -p n -l '\EFI\ubuntu\shimx64.efi'
    where "xxx" is the disk device contains the EFI partition, and "n" is the partition number. E.g.:

  • shutdown the box, remove the CD or flash drive and reboot. Then, we should be able to see the GRUB again.
  • Monday, January 02, 2017

    jq - a powerful unix command line tool for handling JSON data

    Nowadays, JSON is a popular data format, especially in web services domain. However, traditionally, Unix command line tools, e.g. sed, awk, grep, etc., are not capable to handle JSON data format. So, it is hard to write shell scripts to reading JSON data.
    Fortunately, I found a handy and powerful tool - jq. In this page, I want to show you an example of using jq to extract weather forecast in JSON format to CSV. First of all, I get the data from weather forecast serveice of and save it to "forcecast.json" file:
    Here is the raw data, hard to read by human,
    jq can print the JSON in pretty format, with color:
    Now, I show you how to extract the dt_txt(forecast time), temp (forecast temperature) and humidity (forecast humidity) from the JSON data to CSV:
    Just a  single command can covert JSON to CSV. How powerful is it! For the usage details of jq, you can refer to the manual

    Friday, December 23, 2016

    Log TCP Connections to WAN in OpenWrt

    I have a router running OpenWrt 15.05. I would like to log all the TCP connections. To achieve this, I just need to add a line in the firewall custom rule like:
    iptables -A forwarding_lan_rule -p tcp -m state --state NEW -m limit --limit 30/sec -j LOG --log-prefix "NEW Conn "
    Then, login the router as root, and restart the firewall with the following command:
    /etc/init.d/firewall restart

    The following shows how the System Log looks like after enable the custom rule:

    Tuesday, December 20, 2016

    Audio Volume Normalizer

    When I switched from MS Windows to Linux as my primary desktop, I found that I needed to adjust the audio volume frequently. For example, different YouTube videos have different levels of audio volume, but I didn't notice this in MS Windows. With some searching on the web, I found that MS Windows did some tricks behind the scene, called "Loudness Equalization".

    Do we have similar ticks in Linux? After some digging on the web, I found the solution. In the following, I will show you how I do that in my XUbuntu 16.04 LTS box:
    1. First of all, need to install "swh-plugins". Actually, we only need the Compressor Plugin.
      sudo apt-get install swh-plugins
      I am not an expert on Sound Engineering, but I found a page which explained the parameters well.
    2. Next, we need to config PulseAudio to load the plugin and set the "default sink". To do this, create a file at $HOME/.config/pulse/ with below contents:
      #include system-wise config
      .include /etc/pulse/
      #compressor -- for normalize audio volume
      load-module module-ladspa-sink sink_name=ladspa_out label=sc1 control=3,401,-30,4,5,12
      set-default-sink ladspa_out
    3. Then, you just need to kill PulseAudio daemon to restart with the new config:
      pulseaudio -k
    4. If everything goes well, you can run any application with audio playback (e.g. Mpalyer). To confirm the setup, you can run "pavucontrol" command to bring up the "Volume Control" GUI. In the "Volume Control" GUI you should see "SC1" plugin is being used, like:

    Monday, December 19, 2016

    Install Remix OS on VirtualBox

    Remix OS is an Android OS which can run on PC. Because I want to run some Android Apps in my Ubuntu desktop, I tried to install Remix OS in VirtualBox VM. Here are the steps:
    1. Download the Remix OS (64-bit version) at
    2. Unzip We need the file Remix_OS_for_PC_Android_M_64bit_B2016112101.iso, which is a CD image of Remix OS.
    3. Create a new Virtual Machine in VirtualBox like below. Please ensure you select the "Remix_OS_for_PC_Android_M_64bit_B2016112101.iso" disk image for the CD driver of the VM. For Audio, please select "ASLA Audio Driver" and "ICH AC97".
    4. Start the VM, and you will see the GRUB menu. Press "Tab" immediately to edit the boot command.

    5. Append "INSTALL=1" in the boot command and then press "Enter"

    6. Follow the steps to create a new partition:

    7. Choose the newly created partition to install Remix OS
    8. Select "ext4" as the format of the new partition

    9. Choose "Yes" to install GRUB.
    10. Choose "Yes" to install "/system" dir as read-write.
    11. After a few minutes, you will see the installation completed screen. Now, you have to eject the CD of the VM (Device --> Optical Drives --> remove disk from virtual drive). Then, select "Reboot".
    12. If everything goes smooth, you should see the GRUB screen.
    13. After a few minutes you will see the welcome screen. Please note that you need to disable the mouse integration (Input --> Mouse Integration) of the VM, otherwise, you can't use mouse in Remix OS.
    14. Just follow the steps on the screen to initialize the Remix OS. Then, you should see "home" screen. To enable Google Play, you need to double click the "Play activator" and follow the steps on the screen.

    Software versions:
    XUbuntu 16.04.1 LTS
    Linux Kernel 4.4.0-53-generic (64-bit)
    Virtual Box 5.0.24_Ubuntu r108355


    Saturday, November 26, 2016

    Theory vs Practice -- Linked List and Array

    When I was a student, in algorithm course about data structures, we were comparing arrays and linked lists. Arrays are faster than linked lists on random access (O(1) vs O(n)). However, listed lists are faster than arrays for random insertion/deletion operations (O(1) vs O(n)).

    In reality, when we consider the L1 and L2 cache in a computer system, the above is not always true. Recently, I found a good page comparing the performance of C++ std::vector (dynamic array), std::list (linked list) and std::deque. When the size of an element is small, vectors and deques perform better than lists. It's because the spatial locality. In arrays, elements are packed next to each others in memory, but in lists, elements are located randomly and linked by pointers. Therefore, when iterating the elements in lists, it will have much higher chance of cache missed. In other words, lists can't get the benefit from cache.

    Monday, October 10, 2016

    Setup Another Instance of sshd Support Google's Two-Factor Authentication in Raspberry Pi

    For security reason, I want to enhance the sshd running in my Raspberry Pi to make use of Two-Factor Authentication (i.e. SSH Key plus Google Authenticator), if an user logins from WAN (from public network). However, if an user logins from LAN (local network), it just needs password or SSH Key for authentication.

    To archive this, I need to setup a new instance of sshd with Two-Factor Authentication. The original sshd remains unchange for LAN users. Moreover, I also need to install google-authenticator related packages.

    (Note: need to use root account to do below steps)

    1. install the google-authenticator
      apt-get install libpam-google-authenticator libqrencode3
    2. link sshd-second to sshd. The name of executable sshd-second will be used to load the corresponding PAM config.
      ln /usr/sbin/sshd /usr/sbin/sshd-second
    3. create sshd_config-second by copying from sshd_config
      cp /etc/ssh/sshd_config /etc/ssh/sshd_config-second
    4. update the sshd_config-second: Change the "Port" to 9022 (you can use any available port you want); change "ChallengeResponseAuthentication" to "yes"; Set "PasswordAuthentication" to "no"; add "AuthenticationMethods publickey,keyboard-interactive:pam" and add "PidFile /var/run/". Here shows the diff between /etc/ssh/sshd_config and /etc/ssh/sshd_config-second after edited:
      # diff sshd_config sshd_config-second 
      < Port 22
      > Port 9022
      < ChallengeResponseAuthentication no
      > ChallengeResponseAuthentication yes
      < #PasswordAuthentication yes
      > PasswordAuthentication no
      > AuthenticationMethods publickey,keyboard-interactive:pam
      > PidFile /var/run/
    5. create /etc/pam.d/sshd-second by copying from /etc/pam.d/sshd
    6. cp /etc/pam.d/sshd /etc/pam.d/sshd-second
    7. updte the /etc/pam.d/sshd-second: comment out the line "@include common-auth" and add a line "auth required" after "@include common-auth". Here shows the diff between /etc/pam.d/sshd-second and /etc/pam.d/sshd after edited:
      # diff sshd sshd-second 
      < @include common-auth
      > #@include common-auth
      > auth required
      (I comment out the "common-auth" here because I don't want it prompts for password during login.)
    8. create /lib/systemd/system/sshd-second.service by copying from /lib/systemd/system/sshd.service
      cp /lib/systemd/system/sshd.service /lib/systemd/system/sshd-second.service
    9. edit /lib/systemd/system/sshd-second.service: Update the "Description"; update the "ExecStart" to "/usr/sbin/sshd-second -f /etc/ssh/sshd_config-second -D $SSHD_OPTS"; update the "Alias" to "sshd-second.service". Here shows the diff between sshd-second.service and sshd.service after edited:
      # diff sshd.service sshd-second.service 
      < Description=OpenBSD Secure Shell server
      > Description=OpenBSD Secure Shell server (2nd)
      < ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
      > ExecStart=/usr/sbin/sshd-second -f /etc/ssh/sshd_config-second -D $SSHD_OPTS
      < Alias=sshd.service
      > Alias=sshd-second.service
    10. enable and start the new sshd instance:
      systemctl enable sshd-second
      systemctl start sshd-second
      (To check the status of the new sshd, you can run "systemctl status sshd-second")
    Up to now, the new sshd setup is done. Next, need to setup Google Authenticator for each users.
    1. Login as user and run "google-authenticator". Like below:
    2. On user's smart phone, open the Google Authenticator app. Then, add the account using the QRCode or input the secret key. (If the user's phone doesn't have the app, please follow this link to install).
    3. To test the login, use ssh command, e.g:
      ssh localhost -o Port=9022
      Then, it prompts for "Verification code". The user can get the code from the Google Authenticator App. As long as the user input the code correctly and with correct ssh-key, he/she can login.
    Last but not least, need to update router/firewall setting, such that it blocks the original sshd (port 22) from public network and opens for the port (9022 in my case) of the new sshd instance to be access from public network.

    [the demo environment is Raspbian GNU/Linux 8 (jessie)]