OS X Built-in tftp Server:  

You can start it with:

sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist

sudo launchctl start com.apple.tftpd

You can stop it with:

sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist

Default path:

/private/tftpboot

Performance Testing with SIPp

Performance Testing with SIPp:  

Installation:

Download SIPp from http://sourceforge.net/projects/sipp/files/sipp/

./configure --with-pcap
make
make install

Testing:

Server:

• ./sipp -sn uas

Client:

• ./sipp -sn uac 192.168.1.10

Advance client options:

• ./sipp -sd uac > uac.xml
• ./sipp -sf uac.xml
• ./sipp -sn uac -i 192.168.1.5 -d 60000 -l 1 -m 1 -r 1 -s 1001 sip.example.com

-i   bind local IP

-d  call duration ( 60s )

-l  limit concurrent calls

-r  call rate per second

-s username or extension

sip.example.com is remote SIP server.

Send RTP stream:

• ./sipp -sn uac_pcap -i 192.168.1.5 -d 60000 -l 1 -m 1 -r 1 -s 1001 sip.example.com

MySQL Replication setup in 10 minutes

  ________ ________

| | | |
| | | |
| master|>----->----->----->| slave |
| | | |
|_______| |_______|
     


1. configure master's /etc/mysql/my.cnf and add following line in [mysqld] section:

server-id=1 binlog-format = mixed log-bin=mysql-bin datadir=/var/lib/mysql innodb_flush_log_at_trx_commit=1 sync_binlog=1

2. Restart mysqld server and create replication user that your slave will use to authenticate and connect to master:

CREATE USER replicant@<slave-server-ip>;
GRANT REPLICATION SLAVE ON *.* TO replicant@<<slave-server-ip>> IDENTIFIED BY 'choose-secret-password';

3. Create backup file with the binlog position. It will affect performance of your existing database server, but won't lock your tables:

mysqldump -u root -p --skip-lock-tables --single-transaction --flush-logs --hex-blob --events --master-data=2 -A > ~/mysql_full_dump.sql

4. Now, find out MASTER_LOG_FILE and MASTER_LOG_POS values from "mysql_full_dump.sql" using following command, we will use them later:

head dump.sql -n80 | grep "MASTER_LOG_POS"

Above command output look like following:

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=107;

5. scp mysql_full_dump.sql user@slave-server-ip:~/

6. Edit slave server's /etc/my.cnf file to add following lines:

server-id = 101 binlog-format = mixed log_bin = mysql-bin relay-log = mysql-relay-bin log-slave-updates = 1 read-only = 1

7. Restart the mysql slave daemon and then import SQL dump: 

mysql -u root -p < ~/mysql_full_dump.sql

8. Log into mysql console on slave and run following command to setup replication:

mysql> CHANGE MASTER TO MASTER_HOST='master-server-ip',MASTER_USER='replicant',MASTER_PASSWORD=''slave-server-password', MASTER_LOG_FILE='Above head command output', MASTER_LOG_POS='Above head command output';

9. Run command command to start slave:

mysql> START SLAVE;

10. Check replication progress:

mysql> SHOW SLAVE STATUS \G

If all is well then you will see "Waiting for master to send event"

Notes: My mistake if you accidentally change any row or data on slave server, then revert your change and run following command:

mysql> STOP SLAVE;SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;START SLAVE;

BIGIP F5 Command Line (bigpipe Vs tmsh)

b arp show	show /net arp all
b arp all delete	tmsh delete /net arp all
b class DATA-GROUP mode read	modify ltm data-group DATA-GROUP access-mode read-only
b class show	show running-config /ltm data-group
b cluster show	show /sys cluster all-properties
b config load file.ucs	load /sys ucs file.ucs
b config save file.ucs	save /sys ucs file.ucs
b config sync	run /cm config-sync from-group/to-group DEVICEGROUPNAME
b conn show	show /sys connection
b conn show all	show /sys connection all-properties	Show all connection table properties
b conn ss server node-ip:node-port delete	delete /sys connection ss-server-addr node-ip ss-server-port node-port	Delete connection table entries for node-ip node-port
b daemon list	list /sys daemon-ha all-properties
b db < key name > < value >	modify /sys db < key name > value < value >	Modify database values
b db Platform.PowerSupplyMonitor disable	tmsh modify sys db platform.powersupplymonitor value disable	Disables PSU alert if only one PSU in use on Dual PSU system
b db show	show running-config /sys db -hidden all-properties
b export my.config.scf	save /sys scf my.config.scf	v10.x only
b export my.config.scf	save /sys config file my.config.scf tar-file my.config.tar	v11.0+
b failover standby	run /sys failover standby	v11+
b fo show	show /sys failover
b fo standby	run /util bigpipe fo standby	v10+
b ha table	show /sys ha-status all-properties
b hardware baud rate	modify /sys console baud-rate	v10: sol10621 | v11: sol13325
b ha table show	show /sys ha-status all-properties
b httpd list	list /sys httpd	To list httpd configuration.
b import my.config.scf	load /sys scf my.config.scf	v10.x only
b import my.config.scf	load /sys config file my.config.scf tar-file my.config.tar	v11.0+
b interface show -j	show /net interface -hidden all-properties	-hidden is not tab completable, but should be shown in the command output on iHealth.
b load	load sys config partitions all
b merge	load /sys config merge	Added in v11. In v10 use bigpipe
b merge /path/to/file.txt	tmsh load /sys config file /path/to/file.txt merge	Merge a file into the BIG-IP configuration. Added in v11. In v10, use bigpipe
b mgmt show	show running-config /sys management-ip
b monitor show	show running-config /ltm monitor (?)
b nat show	show /ltm nat all or list /ltm nat all-properties	The two tmsh commands are required here since b nat show will list the unit preference and ARP status. Statistical information is shown via “show” while configuration information is shown via “list”.
b node all monitor show	list ltm node monitor
b node show	show /ltm node
b ntp servers 10.10.10.10	modify sys ntp servers add { 10.10.10.10 }
b packet filter all show	show /net packet-fliter
b partition	list auth partition	no “show” command yet, list will only show written partitions
b persist	tmsh show ltm persistence persist-records
b platform	show /sys hardware
b pool list	list /ltm pool
b pool show	show /ltm pool members
b profile access all stats
b profile auth all show all	show /ltm auth profile all	The tmsh auth command does not display associated OCSP information shown by bigpipe.
b profile http ramcache show	show /ltm profile http
b profile http stats	show /ltm profile http
b profile ssl stats	show /ltm profile ssl
b profile persist profile_name list all	tmsh list ltm persistence profile_name all-properties
b profile tcp show	show /ltm profile tcp
b profile tcp stats	show /ltm profile tcp
b profile udp show	show /ltm profile udp
b profile udp stats	show /ltm profile udp
b profile xml show	show /ltm profile xml
b reset	load / sys default-config	v10.x
b reset	load / sys config default	v11.x
b route show	show /net route all
b rule < rule > show all	show /ltm rule < rule >
b rule show	show /ltm rule all
b rule stats reset	reset-stats /ltm rule < rule >
b save	save sys config partitions all
b self show	show running-config /net self
b snat	show /ltm snat
b snatpool show	show /ltm snatpool
b software	show sys software
b software desired	install sys software image NAME volume HDX.Y reboot
b software desired	install sys software image NAME create-volume volume HDX.Y	v11.0+ : Creates volume and installs software. (Cannot create empty volumes in v11)
b software desired	install sys software hotfix NAME volume HDX.Y	Installs desired Hotfix to the specified Volume.
b stp show	show running-config /net stp all-properties
b syslog list all	list sys syslog all-properties
b syslog remote server none	modify sys syslog remote-servers none
b syslog remote server test-srv host 192.168.206.47	modify sys syslog remote-servers add {test-srv{host 192.168.206.47}}	You can append “remote-port 517″ for example to the end of the command to specify the port
b syslog remote server test-srv local ip 172.28.72.90	modify sys syslog remote-servers modify {test-srv{local-ip 172.28.72.90}}	The self ip must be non-floating
b system hostname	modify sys global-settings hostname NEWHOST.EXAMPLE.COM
b trunk show -j	show /net trunk -hidden all
b trunk all lacp show	show /net trunk detail
b unit show
b verify load	load sys config verify
b version	show /sys version	Takes grep (but not “head” as in “b version |head”) – for example, grep on build: tmsh show sys version |grep -i build
b virtual address show	show /ltm virtual-address all-properties	“show” does not show the objects used by the virtual, and list does not show statistics.
b virtual all show all	show /ltm virtual all-properties or list /ltm virtual all-properties	“show” does not show the objects used by the virtual, and list does not show statistics.
b vlan all show all -j	show /net vlan -hidden
b vlangroup all show all	show /net vlan-group all
bigstart status|start|stop|restart SERVICE_NAME	show|start|stop|restart sys service SERVICE_NAME
bpsh (?)	load sys config from-terminal merge

UML (User Mode Linux)

1. Build the Kernel

# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.tar.gz

# bunzip2 linux-2.6.39.tar.gz Build the Kernel 

# make ARCH=um defconfig

# make ARCH=um

2. Download Root-fs filesystem for Debian 

# wget http://fs.devloop.org.uk/filesystems/Debian-Squeeze/Debian-Squeeze-AMD64-root_fs.bz2

# bunzip2 Debian-Squeeze-AMD64-root_fs.bz2

3. Run it!!! Voila!! 

# ./vmlinux ubda=/root/Debian-Squeeze-AMD64-root_fs mem=128m umid=debian
Core dump limits :
        soft - 0
        hard - NONE
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking for tmpfs mount on /dev/shm...OK
Checking PROT_EXEC mmap in /dev/shm/...OK
Checking for the skas3 patch in the host:
  - /proc/mm...not found: No such file or directory
  - PTRACE_FAULTINFO...not found
  - PTRACE_LDT...not found
UML running in SKAS0 mode
Adding 5627904 bytes to physical memory to account for exec-shield gap
Linux version 2.6.39 (root@instructor.example.com) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 Wed Jan 30 11:49:25 PST 2013
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 33675
Kernel command line: ubda=/root/Debian-Squeeze-AMD64-root_fs mem=128m root=98:0
PID hash table entries: 1024 (order: 1, 8192 bytes)
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Memory: 123904k available
NR_IRQS:15
Calibrating delay loop... 525.92 BogoMIPS (lpj=2629632)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 256
Checking that host ptys support output SIGIO...Yes
Checking that host ptys support SIGIO on close...No, enabling workaround
Using 2.6 host AIO
NET: Registered protocol family 16
bio: create slab  at 0
Switching to clocksource itimer
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 2, 16384 bytes)
TCP established hash table entries: 8192 (order: 5, 131072 bytes)
TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
UDP hash table entries: 128 (order: 0, 4096 bytes)
UDP-Lite hash table entries: 128 (order: 0, 4096 bytes)
NET: Registered protocol family 1
mconsole (version 2) initialized on /root/.uml/debian/mconsole
Checking host MADV_REMOVE support...OK
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 242
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
TCP cubic registered
NET: Registered protocol family 17
Initialized stdio console driver
Console initialized on /dev/tty0
console [tty0] enabled
Initializing software serial port version 1
console [mc-1] enabled
ubda: detected capacity change from 0 to 1073741824
 ubda: unknown partition table
Switched to NOHz mode on CPU #0
EXT3-fs: barriers not enabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (ubda): mounted filesystem with ordered data mode
VFS: Mounted root (ext3 filesystem) readonly on device 98:0.
modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
.udev/ already exists on the static /dev! ... (warning).
Starting the hotplug events dispatcher: udevd.
Synthesizing the initial hotplug events...udevd-work[272]: kernel-provided name 'hw_random' and NAME= 'hwrng' disagree, please use SYMLINK+= or change the kernel to provide the proper name

done.
Waiting for /dev to be fully populated...done.
hostname: the specified hostname is invalid
Activating swap...done.
EXT3-fs (ubda): using internal journal
Cleaning up ifupdown....
Setting up networking....
Loading kernel modules...done.
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...failed.
Activating swapfile swap...done.
Cleaning up temporary files....
Configuring network interfaces...Internet Systems Consortium DHCP Client 4.1.1-P1
Copyright 2004-2010 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

SIOCSIFADDR: No such device
modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

eth0: ERROR while getting interface flags: No such device
modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

modprobe: FATAL: Could not load /lib/modules/2.6.39/modules.dep: No such file or directory

eth0: ERROR while getting interface flags: No such device
Bind socket to interface: No such device
Failed to bring up eth0.
done.
Cleaning up temporary files....
Setting kernel variables ...done.
startpar: service(s) returned failure: hostname.sh ... failed!
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.
Serial line 0 assigned device '/dev/pts/0'

Debian GNU/Linux 6.0 (none) tty0

(none) login: root
Last login: Wed Jan 30 19:57:33 UTC 2013 on tty0
Linux (none) 2.6.39 #1 Wed Jan 30 11:49:25 PST 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@(none):~# whoami
root
  

Build RPM - Deploy files using RPM command

1. Deploy files inside tar package  /usr/src/redhat/SOURCES/appdynamics-3.5.8.tar.gz

2. Create  appdynamics.spec file

#===============================================================================

# No debuginfo:
%define debug_package %{nil}


%define name      appdynamics
%define summary   AppDynamics is client agent of AppDynamic Server.
%define version   3.5.8
%define release   Base
%define license   GPL
%define group     Applications/System
%define source    %{name}-%{version}.tar.gz
%define url       http://www.outcome.com
%define vendor    AppDynamic
%define packager  Satish Patel
%define buildroot %{_tmppath}/%{name}-root
%define _prefix   /opt

Name:      %{name}
Summary:   %{summary}
Version:   %{version}
Release:   %{release}
License:   %{license}
Group:     %{group}
Source0:   %{source}
BuildArch: x86_64
Provides:  %{name}
URL:       %{url}
Buildroot: %{buildroot}

%description
AppDynamics is client agent of AppDynamic Server.

%prep
%setup -q

%build

%install
install -d %{buildroot}%{_prefix}/appdynamics

tar cf - . | (cd %{buildroot}%{_prefix}/appdynamics; tar xfp -)

%post
echo " "
echo "-------------------------------------------------"
echo "- %{name} installed in %{_prefix}"
echo "- Add /opt/%{name}/jsvc in tomcat startup script."
echo "-------------------------------------------------"

%clean
rm -rf %{buildroot}

%files
%defattr(-,tomcat5,tomcat5)
%{_prefix}/appdynamics/*
# Add cron entries

%changelog
* Mon Jan 08 2013 Satish Patel
- First rpmbuild version 3.5.8

Python CSV to XLS conversion script using openpyxl

Q. How to convert CSV file with ( : ) colon delimiter to Excel sheet?

A. Required - Python2.6 version with Openpyxl module

#!/usr/bin/python2.6
import csv
from openpyxl import Workbook
from openpyxl.cell import get_column_letter

f = open('/tmp/csvfile.txt', "rU")

csv.register_dialect('colons', delimiter=':')

reader = csv.reader(f, dialect='colons')

wb = Workbook()
dest_filename = r"/tmp/xls_out.xlsx"

ws = wb.worksheets[0]
ws.title = "A Snazzy Title"

for row_index, row in enumerate(reader):
    for column_index, cell in enumerate(row):
        column_letter = get_column_letter((column_index + 1))
        ws.cell('%s%s'%(column_letter, (row_index + 1))).value = cell

wb.save(filename = dest_filename)

How to mount LVM partitions from rescue mode (Fedora/CentOS/RedHat)

Boot from your rescue media and "skip" filesystem mount.

1. scan for volume groups:

#lvm vgscan -v

2. Activate all volume groups:

#lvm vgchange -a y

3. List logical volume

#lvm lvscan

4. Mount your filesystem and do your work :) 

Perl Expect script (auto login)

This script basically takes hostname from "serverlist" and login one by one on servers and run specified commands.

Required: 

Perl

Expect perl module

Install Expect module using CPAN shell:

#perl -MCPAN -e shell

cpan>install Expect

Script:

#!/usr/bin/perl
use Expect;
my $timeout = 1;

$i = 0;

@hostnm = `cat ./serverlist`;

foreach(@hostnm)
{
my $sys_user_pass = 'myPassword';
my $exp = Expect->spawn("ssh -l spatel $hostnm[$i]");
$exp->expect($timeout,

[ 'password: $' => sub {
                         $exp->send("$sys_user_pass\r");
                         exp_continue; }
       ],
         '-re', qr'[#>:] $'
);

$exp->send("sudo -k\n");
$exp->send("sudo touch /root/foo.txt\r");

$exp->expect($timeout,
       [ ': $' => sub {
                          $exp->send("$sys_user_pass\n");
                          exp_continue; }
        ],
          '-re', qr'[#>:] $'
);

$exp->soft_close();
$i++;
}

My first assembly shell code

My first "Hello World" program in assembly language (shell code) .

  #------------------start--------------------------------------

.data

HelloWorldString:
        .ascii "Hello World\n"

.text

.globl _start

_start:

        movl $4, %eax
        movl $1, %ebx
        movl $HelloWorldString, %ecx
        movl $12, %edx
        int $0x80

        movl $1, %eax
        movl $0, %ebx
        int $0x80

#------------------end--------------------------------------

Let compile it

$ as -o Helloword.o Helloword.s

Let link it.

$ ld -o Helloword Helloword.o

Let run it.

 $ ./Helloword

Hello World

Voila!!!!

Python script send email using smtplib

#!/usr/bin/python

import smtplib

SERVER = "mail.example.com"

FROM = "noreply@example.com"
TO = ["User@example.com"]
SUBJECT = "Hello! I am Python"

TEXT = "This message was sent with Python's smtplib."

# Prepare actual message

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# Send the mail

server = smtplib.SMTP(SERVER)
server.sendmail(FROM, TO, message)
server.quit()






# You can buy me a beer.

AWK Commands Tips

Using awk to extract lines in a text file

$ cat > somefile.txt
Line 1
Line 2
Line 3
Line 4


To print a single line number, say line 2:

$ awk 'NR==2' somefile.txt
Line 2



If the text file is huge, you can cheat by exiting the program on the first match. Note that this hack will not work if multiple lines are being extracted.

$ awk 'NR==2 {print;exit}' somefile.txt
Line 2



To extract a section of the text file, say lines 2 to 3:

awk 'NR==2,NR==3' somefile.txt
Line 2
Line 3



A more interesting task is to extract every nth line from a text file. I showed previously how to do it using sed and perl.

Using awk, to print every second line counting from line 0 (first printed line is line 2):

$ awk '0 == NR % 2' somefile.txt
Line 2
Line 4



To print every second line counting from line 1 (first printed line is 1):

$ awk '0 == (NR + 1) % 2' somefile.txt
Line 1
Line 3

VMware - Hot adding memory in Linux VM

POC CentOS 5.5

1. Add memory in VM

2. Run this command to check the state of the memory, looking for memory that appears offline.

#grep offline /sys/devices/system/memory/*/state

3. If memory appears as offline, set it to online with the command:

#echo online > /sys/devices/system/memory/memory[number]/state

Note: I'm using for loop to run above command in one shot on all memory slots.

[root@linuxbug]# for qw in `grep offline /sys/devices/system/memory/*/state | awk -F':' '{print $1}'`; do echo online > $qw; done

4. Verify memory

#free -m

Enjoy!!!

How do I send email attachments from a command prompt?

$ mutt -s "Test mail" -a /tmp/file.tar.gz spatel@linuxbug.org < /tmp/mailmessage.txt

EXT4 Filesystem

1. Compatibility

Any existing Ext3 filesystem can be migrated to Ext4 with an easy procedure which consists in running a couple of commands in read-only mode (described in the next section). This means that you can improve the performance, storage limits and features of your current filesystems without reformatting and/or reinstalling your OS and software environment. If you need the advantages of Ext4 on a production system, you can upgrade the filesystem. The procedure is safe and doesn't risk your data (obviously, backup of critical data is recommended, even if you aren't updating your filesystem :). Ext4 will use the new data structures only on new data, the old structures will remain untouched and it will be possible to read/modify them when needed. This means, that, of course, that once you convert your filesystem to Ext4 you won't be able to go back to Ext3 again (although there's a possibility, described in the next section, of mounting a Ext3 filesystem with Ext4 without using the new disk format and you'll be able to mount it with Ext3 again, but you lose many of the advantages of Ext4).

2. Bigger filesystem/file sizes

Currently, Ext3 support 16 TB of maximum filesystem size, and 2 TB of maximum file size. Ext4 adds 48-bit block addressing, so it will have 1 EB of maximum filesystem size and 16 TB of maximum file size. 1 EB = 1,048,576 TB (1 EB = 1024 PB, 1 PB = 1024 TB, 1 TB = 1024 GB). Why 48-bit and not 64-bit? There are some limitations that would need to be fixed before making Ext4 fully 64-bit capable, which have not been addressed in Ext4. The Ext4 data structures have been designed keeping this in mind, so a future update to Ext4 will implement full 64-bit support at some point. 1 EB will be enough (really :)) until that happens. (Note: The code to create filesystems bigger than 16 TB is -at the time of writing this article- not in any stable release of e2fsprogs. It will be in future releases.)

3. Sub directory scalability

Right now the maximum possible number of sub directories contained in a single directory in Ext3 is 32000. Ext4 breaks that limit and allows a unlimited number of sub directories.

4. Extents

The traditionally Unix-derived filesystems like Ext3 use a indirect block mapping scheme to keep track of each block used for the blocks corresponding to the data of a file. This is inefficient for large files, specially on large file delete and truncate operations, because the mapping keeps a entry for every single block, and big files have many blocks -> huge mappings, slow to handle. Modern filesystems use a different approach called "extents". An extent is basically a bunch of contiguous physical blocks. It basically says "The data is in the next n blocks". For example, a 100 MB file can be allocated into a single extent of that size, instead of needing to create the indirect mapping for 25600 blocks (4 KB per block). Huge files are split in several extents. Extents improve the performance and also help to reduce the fragmentation, since an extent encourages continuous layouts on the disk.

5. Multiblock allocation

When Ext3 needs to write new data to the disk, there's a block allocator that decides which free blocks will be used to write the data. But the Ext3 block allocator only allocates one block (4KB) at a time. That means that if the system needs to write the 100 MB data mentioned in the previous point, it will need to call the block allocator 25600 times (and it was just 100 MB!). Not only this is inefficient, it doesn't allow the block allocator to optimize the allocation policy because it doesn't knows how many total data is being allocated, it only knows about a single block. Ext4 uses a "multiblock allocator" (mballoc) which allocates many blocks in a single call, instead of a single block per call, avoiding a lot of overhead. This improves the performance, and it's particularly useful with delayed allocation and extents. This feature doesn't affect the disk format. Also, note that the Ext4 block/inode allocator has other improvements, described in detail in this paper.

6. Delayed allocation

Delayed allocation is a performance feature (it doesn't change the disk format) found in a few modern filesystems such as XFS, ZFS, btrfs or Reiser 4, and it consists in delaying the allocation of blocks as much as possible, contrary to what traditionally filesystems (such as Ext3, reiser3, etc) do: allocate the blocks as soon as possible. For example, if a process write()s, the filesystem code will allocate immediately the blocks where the data will be placed - even if the data is not being written right now to the disk and it's going to be kept in the cache for some time. This approach has disadvantages. For example when a process is writing continually to a file that grows, successive write()s allocate blocks for the data, but they don't know if the file will keep growing. Delayed allocation, on the other hand, does not allocate the blocks immediately when the process write()s, rather, it delays the allocation of the blocks while the file is kept in cache, until it is really going to be written to the disk. This gives the block allocator the opportunity to optimize the allocation in situations where the old system couldn't. Delayed allocation plays very nicely with the two previous features mentioned, extents and multiblock allocation, because in many workloads when the file is written finally to the disk it will be allocated in extents whose block allocation is done with the mballoc allocator. The performance is much better, and the fragmentation is much improved in some workloads.

7. Fast fsck

Fsck is a very slow operation, especially the first step: checking all the inodes in the file system. In Ext4, at the end of each group's inode table will be stored a list of unused inodes (with a checksum, for safety), so fsck will not check those inodes. The result is that total fsck time improves from 2 to 20 times, depending on the number of used inodes (http://kerneltrap.org/Linux/Improving_fsck_Speeds_in_Ext4). It must be noticed that it's fsck, and not Ext4, who will build the list of unused inodes. This means that you must run fsck to get the list of unused inodes built, and only the next fsck run will be faster (you need to pass a fsck in order to convert a Ext3 filesystem to Ext4 anyway). There's also a feature that takes part in this fsck speed up - "flexible block groups" - that also speeds up filesystem operations.

8. Journal checksumming

The journal is the most used part of the disk, making the blocks that form part of it more prone to hardware failure. And recovering from a corrupted journal can lead to massive corruption. Ext4 checksums the journal data to know if the journal blocks are failing or corrupted. But journal checksumming has a bonus: it allows one to convert the two-phase commit system of Ext3's journaling to a single phase, speeding the filesystem operation up to 20% in some cases - so reliability and performance are improved at the same time. (Note: the part of the feature that improves the performance, the asynchronous logging, is turned off by default for now, and will be enabled in future releases, when its reliability improves)

9. "No Journaling" mode

Journaling ensures the integrity of the filesystem by keeping a log of the ongoing disk changes. However, it is know to have a small overhead. Some people with special requirements and workloads can run without a journal and its integrity advantages. In Ext4 the journaling feature can be disabled, which provides a small performance improvement.

10. Online defragmentation

(This feature is being developed and will be included in future releases). While delayed allocation, extents and multiblock allocation help to reduce the fragmentation, with usage filesystems can still fragment. For example: You write three files in a directory and continually on the disk. Some day you need to update the file of the middle, but the updated file has grown a bit, so there's not enough room for it. You have no option but fragment the excess of data to another place of the disk, which will cause a seek, or allocate the updated file continually in another place, far from the other two files, resulting in seeks if an application needs to read all the files on a directory (say, a file manager doing thumbnails on a directory full of images). Besides, the filesystem can only care about certain types of fragmentation, it can't know, for example, that it must keep all the boot-related files contiguous, because it doesn't know which files are boot-related. To solve this issue, Ext4 will support online fragmentation, and there's a e4defrag tool which can defragment individual files or the whole filesystem.

11. Inode-related features

Larger inodes, nanosecond timestamps, fast extended attributes, inodes reservation...

* Larger inodes: Ext3 supports configurable inode sizes (via the -I mkfs parameter), but the default inode size is 128 bytes. Ext4 will default to 256 bytes. This is needed to accommodate some extra fields (like nanosecond timestamps or inode versioning), and the remaining space of the inode will be used to store extend attributes that are small enough to fit it that space. This will make the access to those attributes much faster, and improves the performance of applications that use extend attributes by a factor of 3-7 times.
* Inode reservation consists in reserving several inodes when a directory is created, expecting that they will be used in the future. This improves the performance, because when new files are created in that directory they'll be able to use the reserved inodes. File creation and deletion is hence more efficient.
* Nanoseconds timestamps means that inode fields like "modified time" will be able to use nanosecond resolution instead of the second resolution of Ext3.

12. Persistent preallocation

This feature, available in Ext3 in the latest kernel versions, and emulated by glibc in the filesystems that don't support it, allows applications to preallocate disk space: Applications tell the filesystem to preallocate the space, and the filesystem preallocates the necessary blocks and data structures, but there's no data on it until the application really needs to write the data in the future. This is what P2P applications do in their own when they "preallocate" the necessary space for a download that will last hours or days, but implemented much more efficiently by the filesystem and with a generic API. This have several uses: first, to avoid applications (like P2P apps) doing it themselves inefficiently by filling a file with zeros. Second, to improve fragmentation, since the blocks will be allocated at one time, as contiguously as possible. Third, to ensure that applications has always the space they know they will need, which is important for RT-ish applications, since without preallocation the filesystem could get full in the middle of an important operation. The feature is available via the libc posix_fallocate() interface.

13. Barriers on by default

This is an option that improves the integrity of the filesystem at the cost of some performance (you can disable it with "mount -o barrier=0", recommended trying it if you're benchmarking). From this LWN article: "The filesystem code must, before writing the [journaling] commit record, be absolutely sure that all of the transaction's information has made it to the journal. Just doing the writes in the proper order is insufficient; contemporary drives maintain large internal caches and will reorder operations for better performance. So the filesystem must explicitly instruct the disk to get all of the journal data onto the media before writing the commit record; if the commit record gets written first, the journal may be corrupted. The kernel's block I/O subsystem makes this capability available through the use of barriers; in essence, a barrier forbids the writing of any blocks after the barrier until all blocks written before the barrier are committed to the media. By using barriers, filesystems can make sure that their on-disk structures remain consistent at all times."

How To Compile Linux Kernel

Compiling custom kernel has its own advantages and disadvantages.Compiling kernel needs to understand few things and then just type couple of commands. This step by step howto covers compiling Linux kernel version 2.6.xx under Debian GNU Linux.

Few step to compile kernel.

# cd /usr/src
# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-x.y.z.tar.bz2

Notes : Make sure GCC and GNU make utility installed on your system before compile kernel.

# tar cvjf linux-x.y.z.tar.bz2
# cd linux-x.y.z

Compile kernel

# make menuconfig

Start compiling to create a compressed kernel image, enter:
# make

compiling to kernel modules
# make modules

Install kernel modules
# make modules_install

To install kernel
# make install

It will install three files into /boot directory as well as modification to your kernel grub configuration file

System.map-2.6.25
config-2.6.25
vmlinuz-2.6.25

Create an initrd image for new installed kernel
# cd /boot
# mkinitrd -o initrd.img-2.6.25 2.6.25

Modify Grub configuration file - /boot/grub/menu.lst
# vi /boot/grub/menu.lst

title Debian GNU/Linux, kernel 2.6.25 Default
root (hd0,0)
kernel /boot/vmlinuz root=/dev/hdb1 ro
initrd /boot/initrd.img-2.6.25
savedefault
boot

Note: its hard to edit this file without knowledge of options so better way is use update-grub command it will do it automaticaly for you.

# update-grub

Now time comes up for testing reboot your system and boot with new kernel and enjoy your system. Good Bye

Netcat - Swiss army knife

One of the Linux command line tools I had initially under-estimated is netcat or just nc. By default, netcat creates a TCP socket either in listening mode (server socket) or a socket that is used in order to connect to a server (client mode). Actually, netcat does not care whether the socket is meant to be a server or a client. All it does is to take the data from stdin and transfer it to the other end across the network.

The simplest example of its usage is to create a server-client chat system. Although this is a very primitive way to chat, it shows how netcat works. In the following examples it is assumed that the machine that creates the listening socket (server) has the 192.168.0.1 IP address. So, create the chat server on this machine and set it to listen to 3333 TCP port:

$ nc -l 3333


On the other end, connect to the server with the following:

$ nc 192.168.0.1 3333


In this case, the keyboard acts as the stdin. Anything you type in the server machine’s terminal is transfered to the client machine and vice-versa.

Transfering Files

In the very same way it can be used to transfer files between two computers. You can create a server that serves the file with the following:

$ cat backup.iso | nc -l 3333


Receive backup.iso on the client machine with the following:

$ nc 192.168.0.1 3333 > backup.iso


As you may have noticed, netcat does not show any info about the progress of the data transfer. This is inconvenient when dealing with large files. In such cases, a pipe-monitoring utility like pv can be used to show a progress indicator. For example, the following shows the total amount of data that has been transfered in real-time on the server side:

$ cat backup.iso | pv -b | nc -l 3333


Of course, the same can be implemented on the client side by piping netcat’s output through pv:

$ nc 192.168.0.1 3333 | pv -b > backup.iso

Other Examples

Netcat is extremely useful for creating a partition image and sending it to a remote machine on-the-fly:

$ dd if=/dev/hdb5 | gzip -9 | nc -l 3333


On the remote machine, connect to the server and receive the partition image with the following command:

$ nc 192.168.0.1 3333 | pv -b > myhdb5partition.img.gz

This might not be as classy as the partition backups using partimage, but it is efficient.

Another useful thing is to compress the critical files on the server machine with tar and have them pulled by a remote machine:

$ tar -czf - /etc/ | nc -l 3333


As you can see, there is a dash in the tar options instead of a filename. This is because tar’s output needs to be passed to netcat.

On the remote machine, the backup is pulled in the same way as before:

$ nc 192.168.0.1 3333 | pv -b > mybackup.tar.gz


Security

It is obvious that using netcat in the way described above, the data travels in the clear across the network. This is acceptable in case of a local network, but, in case of transfers across the internet, then it would be a wise choice to do it through an SSH tunnel.

Using an SSH tunnel has two advantages:

1. The data is transfered inside an encrypted tunnel, so it is well-protected.
2. You do not need to keep any open ports in the firewall configuration of the machine that will act as the server, as the connections will take place through SSH.

You pipe the file to a listening socket on the server machine in the same way as before. It is assumed that an SSH server runs on this machine too.

$ cat backup.iso | nc -l 3333


On the client machine connect to the listening socket through an SSH tunnel:

$ ssh -f -L 23333:127.0.0.1:3333 me@192.168.0.1 sleep 10; \
nc 127.0.0.1 23333 | pv -b > backup.iso


This way of creating and using the SSH tunnel has the advantage that the tunnel is automagically closed after file transfer finishes. For more information and explanation about it please read my article about auto-closing SSH tunnels.

Telnet-like Usage

Netcat can be used in order to talk to servers like telnet does. For example, in order to get the definition of the word “server” from the “WordNet” database at the dict.org dictionary server, I’d do:

$ nc dict.org 2628
220 ..............some WELCOME.....
DEFINE wn server
150 1 definitions retrieved
151 "server" wn "WordNet (r) 2.0"
server
n 1: a person whose occupation is to serve at table (as in a
restaurant) [syn: {waiter}]
2: (court games) the player who serves to start a point
3: (computer science) a computer that provides client stations
with access to files and printers as shared resources to a
computer network [syn: {host}]
4: utensil used in serving food or drink
.
250 ok [d/m/c = 1/0/18; 0.000r 0.000u 0.000s]
QUIT
221 bye [d/m/c = 0/0/0; 16.000r 0.000u 0.000s]


Works as a Port Scanner too

A useful command line flag is -z. When it is used, netcat does not initiate a connection to the server, but just informs about the open port it has found. Also, instead of a single port, it can accept a port-range to scan. For example:

$ nc -z 192.168.0.1 80-90
Connection to 192.168.0.1 80 port [tcp/http] succeeded!


In this example, netcat scanned the 80-90 range of ports and reported that port 80 is open on the remote machine.

The man page contains some more interesting examples, so take the time to read it.
Notes

All the above examples have been performed on Fedora 5/6. Netcat syntax may vary slightly among Linux distributions, so read the man page carefully.

Netcat provides a primitive way to transfer data between two networked computers. I wouldn’t say it’s an absolutely necessary tool in the everyday use, but there are times that this primitive functionality is very useful.

SFTP/SCP autologin Perl script

I have wrote script to autologin scp and transfer specified file. I am using perl Net::SCP::Expect module for it.

Requirement:   Net::SCP::Expect  perl module is required for script.
 
Install required module:

perl -MCPAN -e 'install Net::SCP::Expect' 

Script name is autossh.pl


-----------------------START--------------------------

#!/usr/bin/perl -w

use strict;
use Net::SCP::Expect;

if ($#ARGV != 0 ) {
print "usage: autossh.pl \n";
exit;
}

my $server=$ARGV[0];

my $user = "orion";
my $password = 'mypassword';
my $remotedir = "/home/local/mydata/";
my $filelocation = "/home/remote/backup/";

print "Login...Starting scp...";
my $scpe = Net::SCP::Expect->new(host=>$server, user=>$user, password=>$password, recursive=>'1', auto_yes => '1', auto_quote => '0');

print "\nFILELOCATION:" . $filelocation . "*\n";
print "REMOTEDIR: " . $remotedir . "\n";

$scpe->scp($filelocation, $remotedir);

print "SCP complete\n";

-----------------------END--------------------------

To run script type.

#./autossh.pl servername.com

This script will copy /home/mydata local directory to remove server "remoteserver.com" at /home/remote/backup directory.

Migrate / Move MySQL Database And Users To New Server

Get Current MySQL, Usernames, Hostname, And Database Names

Type the following command at shell prompt to list username and hostname list, enter:

mysql -u root -B -N -p -e "SELECT user, host FROM user" mysql

Sample outputs:

satish 192.168.1.5
tom 192.168.1.5
blog 192.168.1.7
root localhost
db1.vm.linuxbug.net.in
root db1.vm.linuxbug.net.in

The first column is mysql username and second one is network host names. Now, type the following command to get exact details about grants and password for each user from above list:

mysql -u root -p -B -N -e"SHOW GRANTS FOR 'userName'@hostName"
mysql -u root -p -B -N -e"SHOW GRANTS FOR 'satish'@192.168.1.5"

Sample outputs:

GRANT USAGE ON *.* TO 'satish'@'192.168.1.5' IDENTIFIED BY PASSWORD 'somePasswordMd5'
GRANT ALL PRIVILEGES ON `blogdb`.* TO 'satish'@'192.168.1.5'

Where,

satiah - MySQL login username
192.168.1.5 - Another server or workstation to access this mysql server
somePasswordMd5 - Password stored in mysql database which is not in a clear text format
blogdb - Your database name

GDB Example Debugging Session: Segmentation Fault Example

We are going to use gdb to figure out why the following program causes a segmentation fault. The program is meant to read in a line of text from the user and print it. However, we will see that in it's current state it doesn't work as expected...

1 : #include 

2 : #include 



3 : int main(int argc, char **argv)

4 : {

5 :   char *buf;

6 :

7 :   buf = malloc(1<<31);

8 :

9 :   fgets(buf, 1024, stdin);

10:   printf("%s\n", buf);

11:

12:   return 1;

13: }

---

The first step is to compile the program with debugging flags:

prompt> gcc -g segfault.c

Now we run the program:

prompt > a.out

Hello World!

Segmentation fault

prompt >

This is not what we want. Time to fire up gdb:

prompt > gdb a.out

GNU gdb 5.0

Copyright 2000 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type "show copying" to see the conditions.

There is absolutely no warranty for GDB.  Type "show warranty" for details.

This GDB was configured as "i686-pc-linux-gnu"...

(gdb)

We'll just run it and see what happens:

(gdb) run

Starting program: /home/dgawd/cpsc/363/a.out

test string



Program received signal SIGSEGV, Segmentation fault.

0x4007fc13 in _IO_getline_info () from /lib/libc.so.6

So we received the SIGSEGV signal from the operating system. This means that we tried to access an invalid memory address. Let's take a backtrace:

(gdb) backtrace

#0  0x4007fc13 in _IO_getline_info () from /lib/libc.so.6

#1  0x4007fb6c in _IO_getline () from /lib/libc.so.6

#2  0x4007ef51 in fgets () from /lib/libc.so.6

#3  0x80484b2 in main (argc=1, argv=0xbffffaf4) at segfault.c:10

#4  0x40037f5c in __libc_start_main () from /lib/libc.so.6

We are only interested in our own code here, so we want to switch to stack frame 3 and see where the program crashed:

(gdb) frame 3

#3  0x80484b2 in main (argc=1, argv=0xbffffaf4) at segfault.c:10

10        fgets(buf, 1024, stdin)

We crashed inside the call to fgets. In general, we can assume that library functions such as fgets work properly (if this isn't the case, we are in a lot of trouble). So the problem must be one of our arguments. You may not know that 'stdin' is a global variable that is created by the stdio libraries. So we can assume this one is ok. That leaves us with 'buf':

(gdb) print buf

$1 = 0x0

The value of buf is 0x0, which is the NULL pointer. This is not what we want - buf should point to the memory we allocated on line 8. So we're going to have to find out what happened there. First we want to kill the currently-running invocation of our program:

(gdb) kill

Kill the program being debugged? (y or n) y

Now set a breakpoint on line 8:

(gdb) break segfault.c:8

Breakpoint 1 at 0x8048486: file segfault.c, line 8.

Now run the program again:

(gdb) run

Starting program: /home/dgawd/cpsc/363/a.out



Breakpoint 1, main (argc=1, argv=0xbffffaf4) at segfault.c:8

8         buf = malloc(1<<31);

We're going to check the value of buf before the malloc call. Since buf wasn't initialized, the value should be garbage, and it is:

(gdb) print buf

$2 = 0xbffffaa8 "Èúÿ¿#\177\003@t`\001@\001"

Now step over the malloc call and examine buf again:

(gdb) next

10        fgets(buf, 1024, stdin);

(gdb) print buf

$3 = 0x0

After the call to malloc, buf is NULL. If you were to go check the man page for malloc, you would discover that malloc returns NULL when it cannot allocate the amount of memory requested. So our malloc must have failed. Let's go back and look at it again:

7 :   buf = malloc(1<<31);

Well, the value of the expression 1 <<>

prompt >

Hello World!

Hello World!



prompt >

So now you know how to debug segmentation faults with gdb. This is extremely useful (I use it more often then I care to admit). The example also illustrated another very important point: ALWAYS CHECK THE RETURN VALUE OF MALLOC! Have a nice day.