| |
[How to deploy kerrighed nodes massively using DRBL]
http://trac.nchc.org.tw/grid/wiki/krg_DRBL
--------------------------------------------------------------------------------
Search: Login(l)Settings(t)Help/Guide(6)About Trac
Wiki(1)Timeline(2)Roadmap(3)Search(4)Browse Source(5)New Ticket(7)View Tickets(8)
Wiki Navigation
Start Page Index by Title Index by Date Last Change(h)
--------------------------------------------------------------------------------
How to deploy kerrighed nodes massively using DRBL
1. Introduction
2. Software
3. Install Kerrighed
4. Install DRBL
5. Test Kerrighed
6. Reference
How to deploy kerrighed nodes massively using DRBL ?
1. Introduction ?
People need powerful computing resources to resolve complex problems. A important issue is how to easy build cheap and effective computing resource. We use DRBL and Kerighed to build SSI cluster. It quick to deploy your cluster environment. You just install them in one machine (Server), clients don't install any OS. Client only setup PXE (network boot) in BIOS, when client finish network booting that your cluster is OK!
2. Software ?
We use below Software:
Ubuntu or Debian
Linux Distribution.
DRBL
Diskless Remote Boot in Linux (DRBL) provides a diskless or systemless environment for client machines. It works on Debian, Ubuntu, Mandriva, Red Hat, Fedora, CentOS and SuSE. DRBL uses distributed hardware resources and makes it possible for clients to fully access local hardware. It also includes Clonezilla, a partitioning and disk cloning utility similar to Symantec Ghost.
http://drbl.sourceforge.net/
Kerrighed
Kerrighed is a Single System Image operating system for clusters. Kerrighed offers the view of a unique SMP machine on top of a cluster of standard PCs. The new version is Kerrighed 2.2.0, and it used for 2.6.20 kernel.
http://www.kerrighed.org/wiki/index.php/Main_Page
3. Install Kerrighed ?
Install Basic Package
$ sudo aptitude install gcc-3.3 automake autoconf libtool initramfs-tools make
$ sudo aptitude install kernel-package libncurses5-dev build-essential fakeroot wget bzip2
$ sudo aptitude install xmlto lsb-release
$ sudo aptitude install nfsbooted
Download Kerighed tar ball
$ cd /usr/src
$ sudo wget https://gforge.inria.fr/frs/download.php/3791/kerrighed-2.2.1.tar.gz
$ sudo wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.20.tar.bz2
$ sudo tar -zxvf kerrighed-2.2.1.tar.gz
$ sudo tar -jxvf linux-2.6.20.tar.bz2
Config and make Kerrighed-enable kernel, modules
$ cd kerrighed-2.2.0
$ sudo ./configure --with-kernel=/usr/src/linux-2.6.20 --disable-kernel-config --with-kernel-config=/usr/src/linux-2.6.20/.config
$ sudo make patch
$ sudo mv /bin/sh /bin/sh.old
$ sudo ln -s /bin/bash /bin/sh
(Ubuntu default sh is link to dash)
(1)Choose kernel options
$ cd ../linux-2.6.20
$ sudo make defconfig
$ sudo make menuconfig
(0.If your CPU arch is intel, choose i386
1.Cancel Preemption, KEYS and SMP options
2.Enable netwok block device, plug and play, NIC Driver and boot from nfs must to build-in
)
(2)Make
$ cd /usr/src/kerrighed-2.2.0
$ sudo make kernel
$ sudo make
$ sudo make kernel-install
$ sudo make install
Make initrd image
$ sudo mkinitramfs -o /boot/initrd.img-2.6.20-krg 2.6.20-krg
$ sudo vim /boot/grub/menu.lst
(ex:
title Kerrighed 2.6.20-krg
root (hd0,0)
kernel /boot/vmlinuz-2.6.20-krg root=UUID=8bc93e80-35f9-41db-a349-02e839f2ed7c ro quiet splash
initrd /boot/initrd.img-2.6.20-krg
quiet
savedefault
)
Config nodes information
$ vim /etc/kerrighed_nodes
(ex:
session=7
nbmin=8
krg001:0:eth0
krg002:1:eth0
krg003:2:eth0
krg004:3:eth0
krg005:4:eth0
krg006:5:eth0
krg007:6:eth0
krg008:7:eth0
)
4. Install DRBL ?
Add deb source
$ sudo vim /etc/apt/sources.list
(add : deb http://free.nchc.org.tw/drbl-core drbl stable)
$ wget http://drbl.nchc.org.tw/GPG-KEY-DRBL sudo apt-key add GPG-KEY-DRBL
$ sudo apt-get update
Install DRBL
Before we install DRBL, we must clear plan our DRBL environment. The below layout is our environment, eth0 used to connect WAN, eth1 used for DRBL internal clients.
NIC NIC IP Clients
+-----------------------------+
| DRBL SERVER |
| |
| +-- [eth0] 140.110.X.X +- to WAN
| |
| +-- [eth1] 192.168.0.1 +- to clients group 1 [ 7 clients, their IP from 192.168.0.2 - 192.168.0.8]
| |
+-----------------------------+
$ sudo aptitude install drbl
(DRBL will be installed in directory /opt/drbl )
$ sudo /opt/drbl/sbin/drblsrv -i
$ sudo /opt/drbl/sbin/drblpush-offline -s `uname -r`
(The command used interactive mothod help user to install. It install related packages (nfs, dhcp, tftp......) and create /tftpboot directory. The /tftpboot include:
nbi_img: kenrel , initrd image and grub menu
node_root: server directories copy
nodes: each nodes' individual directories
)
$ sudo /opt/drbl/sbin/drblpush -i
(the command will deploy client environment, like client name, DRBL mode, swap ...)
Setup each node's grub menu
$ cd /tftpboot/nbi_im/pxelinux.cfg
(named rule is IP's hexadecimal
ex. 192.168.0.2 -> C0A80002)
$ cp default C0A80002
$ vim C0A80002
( add node_id in append line:
ex.
label drbl
MENU DEFAULT
# MENU HIDE
MENU LABEL Ubuntu 7.04 Linux (DRBL mode, mostly local resources)
# MENU PASSWD
kernel vmlinuz-pxe
append initrd=initrd-pxe.img devfs=nomount drblthincli=off selinux=0 node_id=0 session_id=9
)
5. Test Kerrighed ?
Running
If kerrighed module don't auto load when booting:
$ sudo /etc/init.d/kerrighed start
(all node must load, and we can use command dmesg see node message
ex.
TIPC: Established link <1.1.1:eth0-1.1.3:eth0> on network plane A
krg_node_arrival: 2
)
$ sudo krgadm cluster start
(Kerrighed is running on 7 nodes)
$ top
(we cane see all clients' CPU and Memory are combined
ex.
top - 18:53:16 up 10 min, 2 users, load average: 0.10, 0.07, 0.04
Tasks: 221 total, 1 running, 220 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.7%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu1 : 0.7%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu2 : 0.7%us, 0.3%sy, 0.0%ni, 98.7%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu3 : 0.8%us, 0.4%sy, 0.0%ni, 98.4%id, 0.3%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu4 : 0.8%us, 0.4%sy, 0.0%ni, 98.5%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu5 : 0.7%us, 0.4%sy, 0.0%ni, 98.5%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu6 : 0.8%us, 0.3%sy, 0.0%ni, 98.6%id, 0.2%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 14530264k total, 1508584k used, 13021680k free, 560k buffers
Swap: 2650684k total, 0k used, 2650684k free, 1282652k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 1580 548 480 S 0 0.0 0:00.30 init.orig
2 root 34 19 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
3 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
4 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/0
)
Test Kerrighed command
$ sudo krgadm nodes status
(ex.
[rock@krg002 ~]$ krgadm nodes
0:1 1:1 2:1 3:1 4:1 5:1 6:1
)
$ sudo krgcapset -s
(ex.
Permitted Capabilities: 037777777777
CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
SEE_LOCAL_PROC_STAT
Effective Capabilities: 01
CHANGE_KERRIGHED_CAP
Inheritable Permitted Capabilities: 037777777777
CHANGE_KERRIGHED_CAP, CAN_MIGRATE, DISTANT_FORK, FORK_DELAY
CHECKPOINTABLE, USE_REMOTE_MEMORY, USE_INTRA_CLUSTER_KERSTREAMS
USE_INTER_CLUSTER_KERSTREAMS, USE_WORLD_VISIBLE_KERSTREAMS
SEE_LOCAL_PROC_STAT
Inheritable Effective Capabilities: 01
CHANGE_KERRIGHED_CAP
)
Test process migration
DRBL Server:
$ mkdir /home/ker ; chmod 777 /home/ker
$ cd /home/ker
$ wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.22.tar.gz
$ tar zxvf linux-2.6.22.tar.gz
Client:
$ krgcapset -d +CAN_MIGRATE
$ cd /home/linux-2.6.22
$ sudo make -j 24 bzImage
$ sudo dmesg
(you can command dmesg to see message of process migration
ex.
send_kerrighed_signal: 36647 (Migration Mgr) -> 36885 (bzip2)
send_kerrighed_signal: 36647 (Migration Mgr) -> 37449 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39064 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39234 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39269 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39325 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39402 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39465 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39543 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39599 (cc1)
send_kerrighed_signal: 36647 (Migration Mgr) -> 39660 (cc1)
)
6. Reference ?
DRBL http://drbl.sourceforge.net/
Kerrighed http:///www.kerrighed.org/wiki/index.php/Main_Page
Attachments
config-2.6.20-krg (75.5 kB) -kerrighed .config for 2.6.20 kernel, added by rock on 02/22/08 14:40:29.
Download in other formats:
Plain Text
--------------------------------------------------------------------------------
Powered by Trac 0.10.3
By Edgewall Software.
Power by Open Source Taskforce, NCHC, Taiwan
|
|
|
返回討論區
本篇主題：How to deploy kerrighed nodes massively using DRBL
發表文章
回覆文章
列印本文
線上管理
我也要加入討論