2012. március 27., kedd

dangerous bash shell expansions

Recently I saw a forum thread about removing files with filename expansions can be tricky and dangerous when you use the '[A-Z]*' pattern. See the following example.

[qmi@debian: ~/scr/tmp]$ touch ALMA.txt GGG.txt bela.txt lowercase.txt zzz.txt aaa.txt
[qmi@debian: ~/scr/tmp]$ ls
ALMA.txt  GGG.txt  aaa.txt  bela.txt  lowercase.txt  zzz.txt
[qmi@debian: ~/scr/tmp]$ rm -v [A-Z]*.txt
removed `ALMA.txt'
removed `bela.txt'
removed `GGG.txt'
removed `lowercase.txt'
removed `zzz.txt'

Now, the above should have removed all the uppercase filenames, but in fact, it removed all files except for the 'aaa.txt'! Assuming your default settings on Debian Linux (and on other Linuxes as the forum members confirmed) in Bash and the LC_COLLATE variable. After little research into the Bash man page, I found that the LC_COLLATE variable should be set up to 'POSIX' to avoid happening this again. Let's try again.

[qmi@debian: ~/scr/tmp]$ touch ALMA.txt GGG.txt bela.txt lowercase.txt zzz.txt aaa.txt
[qmi@debian: ~/scr/tmp]$ export LC_COLLATE=POSIX
[qmi@debian: ~/scr/tmp]$ rm -v [A-Z]*.txt
removed `ALMA.txt'
removed `GGG.txt'
[qmi@debian: ~/scr/tmp]$

Instead of using the filename expansion metacharacters, we can use the "[[:upper:]]" regex pattern match to the shell to remove the unwanted filenames with uppercase letters. See below.

[qmi@debian: ~/scr/tmp]$ shopt -u nocaseglob
[qmi@debian: ~/scr/tmp]$ touch ALMA.txt GGG.txt bela.txt lowercase.txt zzz.txt aaa.txt
[qmi@debian: ~/scr/tmp]$ ls
ALMA.txt  GGG.txt  aaa.txt  bela.txt  lowercase.txt  zzz.txt
[qmi@debian: ~/scr/tmp]$ rm -v [[:upper:]]*.txt
removed `ALMA.txt'
removed `GGG.txt'
[qmi@debian: ~/scr/tmp]$ ls
aaa.txt  bela.txt  lowercase.txt  zzz.txt
[qmi@debian: ~/scr/tmp]$

Perfect. It removed just what we wanted! All the lowercase filenames remained.

2012. március 16., péntek

Debian Linux can boot an LVM root partition

I installed Debian Squeeze on my Dell laptop and partitioned the disk to be fully managed by LVM. Including the root (/) partition and I got GRUB set up the MBR as the primary boot loader. After the successful installation, GRUB can load the root filesystem with an LVM partition! In other words, it can boot it without any extra tricks (i.e.: /boot filesystem separate partition, etc.) contrary to Ege Turgay, a Linux sysadmin from Turkey, who claimed that Linux cannot boot an LVM partition unless /boot is on a separate partition not being part of LVM. All the files / partitions are on LVM and Debian Linux Squeeze is able to boot the OS seamlessly. Here are the setup below.

root@debian:/boot/grub# cat /etc/mtab
/dev/mapper/debian-root / ext4 rw,errors=remount-ro 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid,mode=0755 0 0
proc /proc proc rw,noexec,nosuid,nodev 0 0
sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
udev /dev tmpfs rw,mode=0755 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
/dev/mapper/debian-var /var ext4 rw 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0

Above you can see that the root partition is on /dev/mapper/debian-root on LVM and there is no separate /boot partition. Another partition, /var is on /dev/mapper/debian-var on LVM as well (but that does not make any difference to the point) and no other tricks or glitches. Below I am going to show you my grub.cfg which contains an important line that makes it possible. 

root@debian:/boot/grub# cat grub.cfg |grep -C3 lvm

menuentry 'Debian GNU/Linux, with Linux 2.6.XX-X' --class debian --class gnu-linux --class gnu --class os {
insmod lvm
insmod part_msdos
insmod ext2
set root='(debian-root)'
initrd /boot/initrd.img-2.6.XX-X

Pay attention to the red line above. When GRUB is loaded from the MBR, it loads an lvm module so that it can handle LVM partitions without issues. This is so simple! :-) And it's not even the latest, most up-to-date Linux version or distro, but it is a rather old version, the stable distro from Debian, with a relatively obsolete kernel. See the root partition below that is indeed an LVM.

root@debian:/boot/grub# uname -a
Linux debian 2.6.32-5-686-bigmem #1 SMP Mon Jan 16 16:42:05 UTC 2012 i686 GNU/Linux
root@debian:/boot/grub# grub-install -v
grub-install (GRUB) 1.98+20100804-14+squeeze1
root@debian:/boot/grub# lvdisplay /dev/debian/root

--- Logical volume ---
  LV Name                /dev/debian/root
  VG Name                debian
  LV UUID                yYjrfl-keey-KNrH-tbC8-R7B3-Luwi-OKMAMH
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                186.26 GiB
  Current LE             47683
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0


2012. március 10., szombat

ext4 filesystem online resizing

After looking into how to resize an ext3/ext4 filesystem, I found that it works! :) The tool called resize2fs can be used to resize a filesystem online. Actually, when the fs is mounted - online - only extending possible, shrink is not. It did not work for me. Here is a practical application about a successful shrink or resize demonstrated on my 8Gb pendrive. I have got a 2Gb filesystem mounted on /dev/sdb2 as shown below.

root@debian:~# df -m -T /media/usb
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/sdb2      ext4      2030    48      1880   3% /media/usb

List what data what we have, if any before we'll try to shrink it to 1Gb.

root@debian:~# ls -l /media/usb
total 20
-rw-r--r-- 1 root root    26 Mar 10 00:36 bela
drwx------ 2 root root 16384 Mar 10 00:32 lost+found 

We see that a file named 'bela' is contained with the size of 26 bytes. Remember, we want to retain data and be able to see it after we will have resized the filesystem. Now, let's expand the filesystem. The underlying partition has more space , actually the no.of blocks is 3145728 which means it is a 3Gb partition holding a 2Gb filesystem on /dev/sdb2.

root@debian:~# fdisk -l /dev/sdb

Disk /dev/sdb: 7751 MB, 7751073792 bytes
239 heads, 62 sectors/track, 1021 cylinders, total 15138816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x8ef631df

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048     8390655     4194304    b  W95 FAT32
/dev/sdb2         8390656    14682111     3145728   83  Linux


Let's expand now with the following command 'online', without unmounting the filesystem. Your kernel needs to support online resizing. If we don't give the size option to the command, the resize2fs command will expand the filesystem to the end of the partition.

root@debian:~# resize2fs /dev/sdb2
resize2fs 1.42.1 (17-Feb-2012)
Filesystem at /dev/sdb2 is mounted on /media/iomega; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/sdb2 to 786432 (4k) blocks.
The filesystem on /dev/sdb2 is now 786432 blocks long.

It's done! The filesystem's size now is 3Gb, the increased size. Let's check if the OS can see the expanded size with 'df' utility.

root@debian:~# df -m -T /dev/sdb2
Filesystem     Type 1M-blocks  Used Available Use% Mounted on
/dev/sdb2      ext4      3053    96      2804   4% /media/iomega
Voila! In megabytes, the size is 3053 in Mb which is roughly a 3Gb filesystem. Finally, the data remained intact just as it was before, as we did not have to unmount the filesystem and the filesystem resize operation did not alter the data. Note that if you use LVM, you can expand the logical volume's size without unmounting the filesystem and then apply the resize2fs command. In an another article, I will show you how to use LVM to do that.

2012. március 1., csütörtök

how to use tripwire simply

Just recently I decided to try out the famous file integrity checking and intrusion detection tool: Tripwire. It is a brilliant software product and it's FOSS. It can be used to track down the changes on your Unix system by running periodic checks. Report is generated each time the checks run to see what has changed since the database was generated. Let's have a short look how to use Tripwire.

First, you need to initialise the database on your system. This is called Database Initialization Mode. This involves generating a site and local key file with the passwords and the tw.pol file which is part of the Debian package install process. The file mentioned is a binary file generated from the text file version twpol.txt on Debian systems. If you have accidentally removed the binary version and want to regenerate it (like it happened to me), you can easily do it as root without reinstalling the entire package.

root@debian:/etc/tripwire# twadmin -m P --polfile tw.pol twpol.txt
Please enter your site passphrase:
Wrote policy file: /etc/tripwire/tw.pol

That's it. Now, let's go ahead and initialise the Tripwire database.

root@debian:/etc/tripwire# tripwire --init
Please enter your local passphrase:
Parsing policy file: /etc/tripwire/tw.pol
Generating the database...
*** Processing Unix File System ***
### Warning: File system error.
### Filename: /var/lib/tripwire/debian.twd
### No such file or directory
### Continuing...
Wrote database file: /var/lib/tripwire/debian.twd
The database was successfully generated.

As seen above, the database has been generated into the filename /var/lib/tripwire/debian.twd. This will be used as a reference file later on during the integrity checks. Let's run an integrity check and see the report. This is called Integrity Checking Mode .

root@debian:/etc/tripwire# tripwire --check
Parsing policy file: /etc/tripwire/tw.pol
*** Processing Unix File System ***
Performing integrity check...
Wrote report file: /var/lib/tripwire/report/debian-20120301-122000.twr

Open Source Tripwire(R) Integrity Check Report

Report generated by:          root
Report created on:            Thu Mar  1 12:20:00 2012
Database last updated on:     Never

Report Summary:

Host name:                    debian
Host IP address:    
Host ID:                      None
Policy file used:             /etc/tripwire/tw.pol
Configuration file used:      /etc/tripwire/tw.cfg
Database file used:           /var/lib/tripwire/debian.twd
Command line used:            tripwire --check

Rule Summary:

  Section: Unix File System

  Rule Name                       Severity Level    Added    Removed  Modified
  ---------                       --------------    -----    -------  --------
  Other binaries                  66                0        0        0       
  Tripwire Binaries               100               0        0        0       
  Other libraries                 66                0        0        0       
  Root file-system executables    100               0        0        0       
* Tripwire Data Files             100               1        0        0       
  System boot changes             100               0        0        0       
  Root file-system libraries      100               0        0        0       
  Critical system boot files      100               0        0        0       
  Other configuration files       66                0        0        0       
  Boot Scripts                    100               0        0        0       
  Security Control                66                0        0        0       
  Root config files               100               0        0        0       
  Invariant Directories           66                0        0        0       
* Low security filesystems        33                0        0        1       

Total objects scanned:  29964
Total violations found:  2

Object Summary:

# Section: Unix File System

Rule Name: Tripwire Data Files (/var/lib/tripwire/debian.twd)
Severity Level: 100


Rule Name: Low security filesystems (/proc)
Severity Level: 33


Error Report:

No Errors

*** End of report ***

Open Source Tripwire 2.4 Portions copyright 2000 Tripwire, Inc. Tripwire is a registered
trademark of Tripwire, Inc. This software comes with ABSOLUTELY NO WARRANTY;
for details use --version. This is free software which may be redistributed
or modified only under certain conditions; see COPYING for details.
All rights reserved.
Integrity check complete.

It looks brilliant! The only thing looks a bit strange, is that it says 'Total violations found: 2'. One is that it finds the very file, the database as a new file added to the system: '/var/lib/tripwire/debian.twd'. That is true. Now, each time we will integrity checks in the future, the reports will complain about any changes that happen on our system. But we don't want to see the same thing over and over again, we want to avoid the false positives. Tripwire can give an option for that. It is called Database Update Mode. This allows your changes to be reconciled with the Tripwire database, meaning you allow the changes into your system. Let's do it.

root@debian:/etc/tripwire# tripwire --update --twrfile /var/lib/tripwire/report/debian-20120301-122052.twr

The report file is given above in the command line as a reference what needs to be updated. Right after, it will jump you into an editor to review and allow changes. You only need to do is to exit with save from your favourite editor.

Please enter your local passphrase:
Wrote database file: /var/lib/tripwire/debian.twd

Brilliant! A new database has been written. Now, let's run our check again. You can see it from the report that the violations have been disappeared and number of total violations have been either changed or disappeared depending on what changes you have allowed. To summarise and to better understand, the following three commands have been used.

tripwire --init
tripwire --check
tripwire --update

From the above list, the first command is typically run once during setting up Tripwire on a clean system. The next 2 commands are to be run periodically.

Hope this article helped to understand how Tripwire works. Any comments, let me know!

2012. február 29., szerda

sudo -e is to edit

Another interesting thing I just realised lately while reading the sudo manual page on Linux. The sudo command has an inbuilt option, '-e' to edit files. You do not have to invoke your favourite editor, provided that your EDITOR or VISUAL environment is set correctly. The '-e' option will do it for you. See the example below.

For many years before, when I wanted to edit a file with root permissions, I just did

[qmi@debian: ~]$ sudo vim /etc/shadow

Now, I use the '-e' option to sudo and leaving the editor out.

[qmi@debian: ~]$ sudo -e /etc/shadow
sudo: /etc/shadow unchanged
[qmi@debian: ~]$

A good sysadmin reads manuals, it pays off ;-)

2012. február 20., hétfő

rename - a tool to rename files in batch

A long-forgotten Linux tool I found lately, called rename. It is a Perl script on Debian in the /usr/bin/ folder available as 'prename'. It is a very useful tool to rename files in a batch! Let's take an example. Suppose you've got a bunch of .sh extension files you want to rename to .bash in one step, of course.

$ ls -1 *.sh
$ rename 's/\.sh/\.bash/' *.sh
$ ls -1 *.bash

Brilliant! You can of course do it with writing a code shell script. I already had a Python solution as well, which I liked, but now it's good to know that rename is available as part of the main distribution. Look at the Python code below.

# File: rename.py
# Desc: renames files in a directory according to a given
#       global pattern
# Date: Tue, 23 Mar 2004 17:01:03 +0200
import glob, os


for file in glob.glob('*.jpg'):
        (base, extension) = os.path.splitext(file)
        os.rename(file, os.path.join(DIR2, base + '.png'))

The above code renames files with '.jpg' extension to '.png' in the current directory.  

2012. február 13., hétfő

My work terms

After failing more than one time working with dodgy companies and finding unacceptable working conditions, I have decided to write a few words about my acceptable work terms. These outline the ideal necessary conditions under which I am willing to conduct all business at any employer's environment, including daily communication with fellow employees, code of ethics and code of conduct. This applies to working at any company regardless of the country or culture. To make it simple, this blog describes my work terms.  By following these terms I show example how to behave at a working place. These terms are a must and a mandatory requirement everywhere I will be working in the future. No IT Manager, no CTO, no other boss or no work contracts are allowed override these terms on the basis of his/her position or authority. NO EXCEPTIONS.

Individual accountibility
This means that every person, including myself must be able to stand by and take responsibility for the acts or actions during work. Each employee's acts must be able to be tracked down by the company information systems to a certain individual. This strengthens IT security and ensures the transparency of the employee's actions. Blurred lines that could jeopardise this requirement or providing unjustifiable, equivocal traces (i.e. unclear separation of duties between teams or within a group of employees) are not acceptable. Using others' credentials impersonating someone else is by no means acceptable. Each worker must use his/her credentials to conduct or complete the given task. Sharing credentials (i.e. user, password information) is only acceptable with the owner's presence and clear consent.

Focus on task
During business hours, everyone's responsitibility is to focus on the work that is assigned. No deliberate distractions are allowed. The only reason to interrupt the work is to eat or taking ergonomic breaks for general health and well being and for the reason of being ill / unwell (feeling pain).  The duration of such breaks are laid out by the work law of the country.

Respecting others
This means that the employee must respect others and only interrupt fellow employees if the topic is work related. Disturbing others with non-work related matters  (i.e. playing loud music, watching youtube/facebook etc. videos, making dirty jokes or ironical references or any other unlawful behaviour) as well as act of dishonesty, indecent or immoral acts, verbal or written harrassment, bullying (i.e. swearing, using bad or inappropriate language in email, letter) is strictly forbidden. Employees must maintain professional demeanour at all times.

Proper use of the company resources
During business hours it is every employee's responsibility to use the company work assets, network, infrastructure and equipment for only work related matters. Using the company assets, network, equipment for personal reasons is only limited to incidental situations when it is absolutely necessary and cannot be carried out otherwise. Any other personal use is not cultivated (i.e. using company email address for personal reasons, visiting social networking sites, sports, betting or online games, adult sites or any other entertainment). In the company environment (i.e. office) no picture, poster or sign is allowed that could negatively affect good morals or by any means disturbing or discriminating against any religion, human race, age, skin colour, gender, political affiliation.