Linux Bash Scripting: How to combine two lines by removing newline character using sed

In this post we will see how to remove the newline character and combine two lines in a text file using "sed".

In sed, we have single line version and multi-line version of few commands. One such command is the next command. Click here to see an example for single-line version of next command in sed.

You might want to look at :

How to remove a blank line after a particular line using sed

The processing space of the sed editor is known as the pattern space. The multi-line next command (N) adds the next line to the existing pattern space which already contains the previous line. So, the sed editor can treat two lines as a single line. Below is an example for the N command.

$ cat sample.txt This is the header line. This is the first line. This is the second line. This is the last line. $ $ sed ‘/first/{ N ; s/\n/ / }’ sample.txt This is the header line. This is the first line. This is the second line. This is the last line. $  

The above sed editor script searches for the line of text that contains the word “first” in it. When it finds the line, it uses the N command to combine the next line with that line. It then uses the substitution command (s) to replace the newline character with a space. The result is that the two lines in the text file appear as one line in the sed editor output.

Linux Bash Scripting: How to create directories for each day of the current month

 In this post I have written a script that creates an empty directory for each day of the current month. I used cal command to get the date and month instead of using date command. The reason is that the date command does not work in Solaris. To make it portable, I have used cal command.

The below script does the following

• It takes the total number of days of the current month 
• It gets the current month name from the cal command
• It gets the current year from the cal command
• It creates an empty directory for each day of the current month
• Before creating, this checks whether the directory already exists, if yes, the script exits by displaying a message.

#!/bin/bash days=`echo $(cal) | tail -c 3` month=`echo $(cal) | head -c 3` year=`echo $(cal) | cut -d' ' -f2` for day in `seq $days` do folname=$day$month$year echo $folname if [ -d dir/$folname ] then echo "directory $folname already exists" exit else mkdir -p "dir/$folname" fi done

You can change the "dir" to the directory in which you want the folders to be created.

Linux Bash Scripting: How to remove a blank line after a particular line using sed

Removing a blank line after a particular line can be achieved using the single-line next command (n) of the sed editor.

The lowercase n command tells the sed editor to move to the next line of text in the data stream, without going back to the beginning of the sed commands. Normally the sed editor processes all of the defined commands on a line before moving to the next line of text in the data stream. The single‐line next command alters this flow.

In the below example, the data file that contains five lines, two of them empty. The goal is to remove the blank line after the header line but leave the blank line before the last line intact. If you write a sed script to just remove blank lines, you will remove both blank lines.

$ cat data This is the header line. This is a data line. This is the last line. $ $ sed ‘/ˆ$/d’ data This is the header line. This is a data line. This is the last line. $  

This happened because there was no specific way to find the unique blank line that you wanted to delete. The same can be achieved using the n command as shown in the below script.

$ sed ‘/header/{n ; d}’ data This is the header line. This is a data line. This is the last line. $

The above script searches for the keyword "header" and then moves to the next line when it executes the n command and then applies the d command to delete the blank line below it.

Linux Bash Scripting: How to modify the ls command output using pr command

I have received the below question in our "Ask a question" section of this blog.

Q: Using set -A write a script to print the output of the ls command in 5 columns with two spaces between each column. Pretend that ls does not have multi column output.

Normally we will get a multi column output if we use the command ls -A as shown below.

/home/sittik/programs $ ls -A arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp

To print the output of the ls command in five columns with two spaces between each column. You can use the following command.

ls -A | pr -a5 -S" " -T

The pr utility is a printing and pagination filter for text files.

From the man page,

-COLUMN, --columns=COLUMN

output COLUMN columns and print columns down, unless -a is used. Balance number of lines in the columns on each page.

-a, --across

print columns across rather than down, used together with -COLUMN 

-SSTRING, --sep-string[=STRING]

separate columns by STRING, without -S: Default separator <TAB> with -J and <space> otherwise (same as -S" "), no effect on column options

-T, --omit-pagination

omit page headers and trailers, eliminate any pagination by form feeds set in input files

The output of the command would be,

/home/sittik/programs $ ls -A | pr -a5 -S" " -T arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp setting .setting.swp

suppose if you want to have delimiter as :::: instead of spaces, then you can change the delimiter in the following command.

/home/sittik/programs $ ls -A | pr -5 -S"::::" -T arr.sh ::::com ::::gcom ::::loops ::::setting bigchess.sh::::file ::::lbigchess.s::::samp ::::.setting.sw chess.sh ::::fileread.sh

you can see the difference in the order in which the files are displayed from the above two commands. That is because I purposely did not use -a option to show the difference.

To pretend that ls does not have a multi column output, you need to add the above command as the alias to the ls command as below.

alias ls='ls -A | pr -a5 -S" " -T'

After adding alias, if you use the ls command, you will get the desired output.

/home/sittik/programs $ ls arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp setting .setting.swp

Linux Bash Scripting: Standard file descriptors STDIN, STDOUT and STDERR and input/ouput redirection

 

Linux system handles every object as a file. This includes input and output processes also. The linux identifies each file using a file descriptor.

What is a file descriptor?
A file descriptor is a non negative integer which uniquely identifies the open files. Each process can have upto 9 open files at a time. The bash shell reserved three file descriptors(0, 1 and 2) for special purposes.

Linux Standard File Descriptors
File descriptor	Abbreviations	Description
0	STDIN	Standard Input
1	STDOUT	Standard output
2	STDERR	Standard Error

The above three file descriptor controls the input and output of your scripts. The shell uses these descriptors to assign the input and output to appropriate locations (default is the monitor or the terminal).  Now let us see each one in detail.

STDIN

The STDIN file descriptor refers the keyboard in the terminal environment. So, the Linux will read the input from keyboard always unless it is changed.

$ cat This is test This is second test  

The above “cat” command accepts input from the keyboard.  If you want to force the “cat” command to accept the input other than STDIN, you can use STDIN redirect symbol (<).

$ cat < filetest This is test This is second test $

STDOUT

The STDOUT descriptor refers to the standard output of the shell (i.e. the monitor). So, the Linux directs all the output to the monitor unless it is changed.

$ ls –l > test $ cat test -rwxrwxrwx 1 sittik owner 260 Oct 16 07:03 arr.sh -rwx--x--x 1 sittik owner 963 Oct 11 01:25 bigchess.sh -rwx--x--x 1 sittik owner 389 Oct 4 00:47 chess.sh

You can also append the data to a file by using the >> symbol.

$ who >> test $ cat test -rwxrwxrwx 1 sittik owner 260 Oct 16 07:03 arr.sh -rwx--x--x 1 sittik owner 963 Oct 11 01:25 bigchess.sh -rwx--x--x 1 sittik owner 389 Oct 4 00:47 chess.sh sittik pts/0 2012-10-20 05:50 (132.127.131.35)

But whenever the command produces an error, it will be displayed in the monitor even if the output is redirected to a file. That is because the error details have a separate file descriptor (STDERR).

STDERR

By default the STDERR descriptor refers to monitor.

$ ls –al badfile > test ls: badfile: No such file or directory $ cat test $

The above example shows that the error message is displayed even after the output is redirected to the file called test. If you want to redirect the error message to some other file, you can do it as below.

$ ls –al badfile 2> test $ cat test ls: badfile: No such file or directory $

If you want to redirect the output and the error message to different files, you can do as below.

$ ls –al goodfile badfile 2> badtest 1> goodtest $ cat badtest ls: badfile: No such file or directory $ cat goodtest -rwx--x--x 1 sittik owner 389 Oct 14 10:47 goodfile $

If you want to redirect both the error message and the output to the same file, you can use the & operator as shown below.

$ ls –al badfile goodfile &> commonfile $ cat commonfile ls: badfile: No such file or directory -rwx--x--x 1 sittik owner 389 Oct 14 10:47 goodfile $

Linux Bash Scripting: how to rename or delete a file which has special characters in its filename

Suppose if you have created a file with a special character in its name because of a typo, you may not be able to access it. You may not even be able to delete the file. For example, see the below files.

$ touch "test-file1 " [Note: There is a space at the end] $ touch "test'file2" [Note: There is a single quote in the middle] $ touch "test?file3" [Note: There is a question mark in the middle] $ ls -1 test* test-file1 test'file2 test?file3

If you see the output of ls command, you will not be able to identify the space at the end of the file. Hence you may not be able to access the file unless you specify a space at the end of the file using quotes.

The ? is a wild character for which any character can be substituted. Hence you may end up deleting or reading some other file.

To handle this situation, we can make use the inode of the file. Every file has a unique inode number with which we can identify the file. 

$ ls -i1 test* 13287439 test-file1 13287623 test'file2 13287981 test?file3

To rename the file:

You can specify the inode number in the find command as below to rename the files. The below command renames the file using  the inode number.

$ find -inum 13287623 -exec mv {} testfile3new \; $ ls -i1 test* 13287439 test-file1 13287623 test'file2 13287981 testfile3new

To delete the file:

You can delete the file using the inode number of the file with the help of "find" command as below.

$ find -inum 13287623 -exec rm {} \; $ ls -i1 test* 13287439 test-file1 13287981 testfile3new [Note: The file with name "test'file" is now removed]  

Linux Bash Scripting: Perform floating point calculation/arithmetic/operation using BC

 

As we have discussed on performing Arithmetic operations on our previous post, now we will look at the floating point operations in the shell script in this post.

You might want to look at : 
Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

A huge limitation in the Bash shell is that it supports only integer arithmetic. There are several solutions to overcome this limitation. The famous solution is Bash Calculator.

Basics of BC(Bash Calculator):

The bash calculator is actually a programming language that allows you to enter floating point expressions at the command line and then interprets the expressions, calculates and then returns the result. The Bash calculator recognizes the following:

• Numbers (both integer and floating point)
• Variables (both simple variables and arrays)
• Comments (lines starting with a pound sign or the C language /* */ pair
• Expressions
• Programming statements (such as if‐then statements)
• Functions

 The Bash calculator can be invoked from the command line using "bc" command.

$ bc bc 1.06.95 Copyright 1991‐1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type ‘warranty’. 9.5 * 3.5 33.25 3.156 * (3 + 5) 25.248 quit $

As you can see from the above example, each time you enter the expression, you get the result immediately. To exit the BC you need to type quit.

Floating point decimal is controlled by a variable called "scale". Using this variable we can specify no. of decimal places that we would like to see in our result.

$ bc ‐q 3.44 / 5 0 scale=4 3.44 / 5 .6880 quit $

The default value of the scale is zero. As you can see from the above example, the Bash calculator returns the answer with zero decimal places. After setting the scale to 4, it displays the result with 4 decimal places.

The -q parameter suppresses the lengthy welcome banner of the Bash Calculator. In addition to the normal numbers, the bash calculator also understands the variables.

$ bc ‐q var1=10 var1 * 4 40 var2 = var1 / 5 print var2 2 quit $

Once a variable is defined you can use the variable throughout the Bash calculator session. The "print" statement is used to print the variables and numbers.

Using BC in the scripts:

We can use the BC in our script in two ways.

1) Using the back tick character to run the "bc" command and assign the result to the variable as shown below.

        variable=‘echo “options; expression” | bc’

Below is a sample script which illustrates the above usage.


$ cat test11 #!/bin/bash var1=20 var2=3.14159 var3=‘echo “scale=4; $var1 * $var1” | bc’ var4=‘echo “scale=4; $var3 * $var2” | bc’ echo The final result is $var4 $

This method will be useful for small calculations. For the calculations which involves more complex expressions, the next method will be useful.

2) Using the inline input redirection, you can perform calculations which involves more numbers. The syntax of this method would be,

        variable=‘bc << EOF
        options
        statements
        expressions
        EOF
        ’
The EOF text string indicates the beginning and end of the inline redirection data. Remember that the backtick characters are still needed to assign the output of the bc command to the variable.

Below is an example script to show the usage of input redirection in BC.

$ cat test12 #!/bin/bash var1=10.46 var2=43.67 var3=33.2 var4=71 var5=‘bc << EOF scale = 4 a1 = ( $var1 * $var2) b1 = ($var3 * $var4) a1 + b1 EOF ’ echo The final answer for this mess is $var5 $

Note: The variables declared inside the Bash calculator is valid only inside the BC and can't be used in the shell script.

Linux Bash Scripting: How to ignore specific commands from being stored in the history in Linux

If you see the history of your commands using the "history" command, you can see all the commands that you executed in the command line. If you don't want certain basic commands to be stored in your history, you can ignore them using HISTIGNORE. You can specify all the commands that you want to ignore in the HISTIGNORE. 

In the below code, I have ignored the commands pwd, ls, ls -ltr and l using HISTIGNORE. Then I am executing some commands in the command line. Finally I'm checking the output of the history command.

$ export HISTIGNORE="pwd:ls:ls -ltr:l:" $ ls $ l $ ls -ltr $ pwd $ cd $HOME $ history | tail -3 379 export HISTIGNORE="pwd:ls:ls -ltr:l:" 380 cd $HOME 381 history
[Note that history did not record ls, l, ls -ltr and pwd]

Please note that adding ls does not exclude the command ls -al from being included in the history. So, you need to specify the complete command to make it as an exception.

Linux bash scripting: How to repeat the previous command quickly in the Linux command line

There are four different ways by which we can repeat the previous commands that we executed in the command line.

1) use the "UP arrow" to view the previous command and press Enter to execute it.
2) Type !! and press enter from the command line
3) Type !-1 and press enter from the command line.
4) Press Control+P to display the previous commands. You can keep pressing the Ctrl+P to older commands. When you see the required command you can press enter to execute it

Linux bash scripting: How to search the command history (previously executed commands)

We often need to type long commands in the command line which we typed earlier. Instead of typing it again we can search the command history for the command we executed and execute it again. This technique will come handy when you are required to type a long command that you executed previously.

To search the command history and execute the command without editing it, Press Ctrl+R and type the keyword present in the previously executed command.

In the below example, I searched for the word "work", which displayed the previous command "cd /home/sittik/test/DDI/work" in the history that contained the word work.

$ [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`work': cat /home/sittik/test/DDI/work [Note: Press enter when you see your command, which will execute the command from the history] $ cat /home/sittik/test/DDI/work This is my work data

Suppose if you need to edit the command before executing it, you can use the left or right arrow key without pressing enter. For example, you can search for "httpd" and which will display the command "service httpd stop" from the command history. You can change this command to "service httpd start" and execute it as shown below.

$ [Press Ctrl+R from the command prompt, which will display the reverse-i-search prompt] (reverse-i-search)`httpd': service httpd stop [Note: Press either left arrow or right arrow key when you see your command, which will display the command for you to edit, before executing it] $ service httpd start

linux Bash Scripting: how to display the history of commands along with the timestamp

Usually the history command will display the command number and the command when you execute the command from the command line. Suppose if you want to add the timestamp along with the command history for auditing purpose, you can do it using the HISTTIMEFORMAT as shown below.

$ export HISTTIMEFORMAT='%F %T ' $ history | less 1 2012-10-11 04:12:36 vi .profile 2 2012-10-11 04:12:36 ls -al 3 2012-10-11 04:12:36 vi .bash_profile 4 2012-10-11 04:12:36 source ./.bash_profile

Linux Bash Scripting: How to rename multiple files in a single go

You might come across a situation where you need to rename lot of files with similar file names either by editing the file name or by truncating the last part of the file name. There are two ways by which you can accomplish the task.

Method 1:

You can use the "rename" command to rename multiple files in a single go.

Syntax:

rename from to file...

- from is the word that you want to change in the file name
- to is the word that you want to replace with
- file is the file that want to rename ( you can use regular expression to select multiple files here

Example:

rename .html .htm *.html

This will change the extension of all the html files present in the current directory. The from and to keywords need not be the last characters of the file names. It can be anywhere in the file names

Method 2:

This method utilizes the "for" loop and "mv" command to rename all the files in a single go.

# This is program to rename multiple files for i in *.html do echo "`basename $i l`"; mv "$i" "`basename $i l`"; done


The above script will change all the html file name from *.html to *.htm

Windows tips and Tricks

I have personally tried and tested most of the tweaks discussed in this guide. However, I request fellow readers to be cautious while trying it out with your system. Always take a backup copy of all important data/registry before attempting to change the system/registry settings. Remember, prevention is always better than cure.

Linux Bash Scripting: how to print a big chess board anywhere on the screen

In this post, we will see how to create a big chess board that can be displayed anywhere on the screen rather than being displayed left aligned. Below is the script which displays the Chess board on the screen.

#!/bin/bash # SCRIPT : bigchess.sh # PURPOSE : Prints big chess board on the screen using Linux style for loop. clear # clears the screen row=6 # This indicates after how many lines the chess board should be displayed col=15 # This indicates after how many cols the chess board should be displayed for i in `seq 9` do for j in 1 2 # prints same line twice to increase height of the block do tput cup $row $col # moves cursor to partcular ROW(here row) & COLUMN(here col) for k in `seq 9` do c=`expr $((i+k)) % 2` # to determine whether it is odd or even block if [ $c -eq 0 ] then echo -e -n "\033[47m " # White background, no. of spaces indicates the width of the block else echo -e -n "\033[40m " # Black background, no. of spaces indicates the width of the block fi done row=`expr $row + 1` done done echo -e "\033[0m" # Restores color settings read key # Waits for enter

If you want to make changes to the above script for changing the size and the position of the chess board, you can change the following variables in the script.

1) row - to change the number of lines to leave from top before printing the chess board
2) col - to change the number of columns to leave from left before printing the chess board
3) j in second for loop - to increase or decrease the height of the block
4) No. of spaces in echo statement to change the width of the block


The same script using C style for loop:

#!/bin/bash # SCRIPT : bigchess.sh # PURPOSE : Prints big chess board on the screen using C style for loop. clear # clears the screen row=6 # This indicates after how many lines the chess board should be displayed col=15 # This indicates after how many cols the chess board should be displayed for (( i=1 ; i<=9; i++ )) do for (( j=1 ;j<=2; j++ )) # prints same line twice to increase height of the block do tput cup $row $col # moves cursor to partcular ROW(here row) & COLUMN(here col) for (( k=1 ; k<=9; k++ )) do c=`expr $((i+k)) % 2` # to determine whether it is odd or even block if [ $c -eq 0 ] then echo -e -n "\033[47m " # White background, no. of spaces indicates the width of the block else echo -e -n "\033[40m " # Black background, no. of spaces indicates the width of the block fi done let row=row+1 done done echo -e "\033[0m" # Restores color settings read key # Waits for enter

Have questions?? Need Clarifications???

Hi All,

I know that lot of us would get stuck while doing programming in Shell scripting or doing something in Linux. If you are badly in help and need to find an answer, you can make use of this page. Please post your questions on shell scripting in the comments box along with your email. Your questions will be answered and will be sent to your email. Please make use of this free service :)

Thanks,
Beneficial Knowledge

how to change the default size of the terminal window in Solaris

To change the default size of the terminal window in Solaris, you need to do the following.

1) Go to the directory /usr/share/vte/termcap/ and edit the file xterm

     i.e., $ vi /usr/share/vte/termcap/xterm

2) Look for :co#115:it#8:li#36:\

3) In the above line you can change the numbers like for example, put 77 after "co#" and put 48 after "li#"

4) Then every terminal you open will open with the size that you have mentioned in the above line.

OR

You can also add an alias as below in the .profile or .bashrc or .cshrc, wherever it is applicable for you.

alias xterm='xterm -geometry 77x48'

Any one of the above mentioned methods should do the work of changing the default terminal size. 

Note : You need to reboot your PC after you have made the above changes for your changes to be effective.

Linux Shell Scripting: how to list the files alone in a directory

 Often we tend to look into the directories to check what files are present in the directory. When we try to list the files using ls command, all files along with the directories will be displayed. What if we want to list only the files and not the directories???

We will see how to do that in this post.

1) Using "find" command:

$ find . -type f -maxdepth 1

The above command will list also the hidden files. But if you want to avoid the hidden files to be displayed in your results, you need to use the below command.

$ find . -type f -maxdepth 1 \( ! -iname ".*" \)


2) To derive it from the "ls" command, we can use the below commands.

$ ls -l | grep -v ^d

$ ls -l | awk 'NR!=1 && !/^d/ {print $NF}'

Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

 

 The command which is used to perform mathematical or Arithmetic calculations is "expr" command.

Syntax:
expr <operand 1> operator <operand 2>



Examples:
$ expr 2 + 3
$ expr 5 - 3
$ expr 7 % 3
$ expr 12 \* 3
$ expr 12 / 6

However, even though the operations looks easy for performing, the "expr" command has a drawback. It uses special character (*) for multiplication which is recognized as a wild character by Linux/Unix. So, if you try to do multiplication as

$ expr 12 * 3

Then you will receive an error. You need to use an escape character (\) along with the (*) character to indicate that this should not be considered as a wild character. Hence the expression should be

$ expr 12 \* 3

If you are going to display the result or assign the result to some variable, then you need to include the expr command in a back tick(`). This symbol is not commonly used. You can find this symbol along with the (~) key in the US keyboard. This is not a single quote('). This is different from single quote and the double quote. Let's see the examples below illustrating the usage.

$ echo "expr 3 + 2" # Using double quotes, considered as a string expr 3 + 2 $ echo 'expr 3 + 2' # Using single quotes also same as above expr 3 + 2 $ echo `expr 3 + 2` # By using back tick, we got the desired result. 5

The back tick is used to inform the Linux system that whatever included within the back tick should be considered as a command and not a string.

To overcome this complexity, Bash shell introduced a new method of doing arithmetic calculations by using brackets. But still the Bash shell includes the expr command to stay compatible with Bourne shell.

Using Brackets:
Another easy way of doing Arithmetic calculation is to enclose the calculation using a dollar sign($) and square brackets ([]).

Examples:

$ var1=$[1+6] $ echo $var1 7 $ var2=$[$var1 * 3] $ echo $var2 21

When you are using brackets for doing arithmetic calculations, you don't need to worry about the wild character (*). Linux will consider whatever inside the bracket as  an expression and not as a wild character.

Finally, there is a major limitation in performing math in Bash shell. That is the Bash supports only  integer numbers and not floating numbers. For example if you do the below calculation, you will get only the integer as a result.

$ echo $[100 / 45]
2

But, the z shell (zsh) provides full floating point arithmetic operations. There is also an alternative way to do the floating point calculations in Bash shell. That is by using BC (Bash Calculator).

How to print the last field in a file in Linux/Unix Shell scripting

Let us see how to print the last field alone from a file which contains several fields.

Let's say that the input is the below file.



$ cat priority.txt Bash:P:Sun:7 NW:F:Mon:4 DB:P:Tue:8 SE:P:Fri:8


Now we need to Print the last field. i.e.

The required Output would be:
----------------------------------
7
4
8
8

We can do this by three ways.

1) The normal awk way of doing it:

$ awk 'BEGIN {FS=":"} {print $NF}' priority.txt

2) Using sed:

$ sed -n 's/.*://;p' priority.txt

3) Using grep:

$ grep -o '[^:]*$' priority.txt

How to find the latest file in a directory

Suppose if you hava a directory with a huge set of files(more than 500) in a log directory, you would want to find the latest file in the directory. Now we can see few ways to that.

for example, Let us take the below scenario

$ ls -lrt total 476744 -rw-r--r-- 1 sittik sdefault 13764 Aug 20 06:34 cache_log.23219.txt -rw-r--r-- 1 sittik sdefault 95451 Aug 20 06:38 cache_log.23236.txt ... ... -rw-r--r-- 1 sittik sdefault 96269 Sep 30 11:59 cache_log.23233.txt -rw-r--r-- 1 sittik sdefault 91236 Sep 30 12:03 cache_log.23239.txt drwxr-xr-x 2 sittik sdefault 4096 Sep 30 12:04 old


1) Now our goal is to print the last field of the last line (except the dir old) from the "ls -lrt" output in that directory.

$ ls -lrt | awk '/cache_log/ { f=$NF };END{ print f }' cache_log.23239.txt


So basically to print the last field of last line(in this case the last field of last line of "ls -lrt" command), we can use the below command.

$ ls -lrt | awk '{ f=$NF }; END{ print f }'


2) This can also be acheived by using Head command.

$ ls -t1 | head -n1


3) This can also be acheived by using Find command.

$ find . -not -type d -printf "%T+ %p\n" | sort -n | tail -1

Linux/Unix Shell Scripting topics explained on requests

• Are you new to Shell scripting? 
• Are you learning Shell Scripting either in Linux or Unix?
• Is it tough for you to understand? 
• Do you feel confused while trying to learn a topic?
• Is there anything in shell scripting that you are not able to understand?

Please let us know (leave your comments below). We are here to help you understand(Don't worry!!! We won't charge u for it :P). We can explain things neatly for you with good examples and pictures.

Thanks,
Beneficial Knowledge
 

Linux terminal freezing problem [resolved] :-)

I often wonder why my Linux terminal freezes when I press certain key combinations unknowingly. Finally when I found the key combination, I found that I pressed Ctrl+s key. Which is used for enabling the scroll lock. Hence the screen was shocked. To release the lock, we have to press the Ctrl+q. This key combination will unlock the scroll lock.

Please share any such instances with us that you encountered in your Linux career.

How to display the text in Bold and Underline in Linux

 

 To display the test in "BOLD" and "UNDERLINE", the following ANSI Escape Sequences can be used.

To display the text in BOLD
$ echo -e "\033[1mThis is a BOLD line\033[0m"
This is a BOLD line

#Using tput
$ tput bold
$ echo "This"     #BOLD
$ tput sgr0     #Reset text attributes to normal without clear.
$ echo "This"     #NORMAL

To display the text in UNDERLINE
$ echo -e "\033[4mThis is an underlined line.\033[0m"
This is an underlined line.

How to sort the files and directories based on their size

Let us see how to sort the files and directories. In my current directory, I have the following files and sub-directories:

$ file *
config: directory
val.dat: ASCII text
sample.txt: ASCII text
log: directory

i.e., 2 files and 2 directories

Now to list the files and directories (in the current directory) in the descending order of file size.

$ du -sk * | sort -nr
4730 log
2779 config
1100 val.dat
968 sample.txt

To list only the directories (in the current directory) in descending order of size:

$ du --max-depth=1 . | sort -nr
9308 .
4730 ./log
2779 ./config


Since the sub-directories also contain files, let's list all the files (not directories) in the descending order of file sizes

$ find . -type f -exec du -sk {} \; | sort -nr
2196 ./log/main1.dat
2064 ./config/pi.dat
1100 ./log/huo.txt
1100 ./val.dat
968 ./sample.txt
964 ./log/as.txt
916 ./config/lis.txt


If you would like to restrict find to the current directory only and not the sub-directories, use need to use maxdepth=1. i.e.

$ find . -maxdepth 1 -type f -exec du -sk {} \; | sort -nr
1100 ./val.dat
968 ./sample.txt

How to list the empty directories in Linux

 

The find command in Linux has an option called 'empty'. This option can be used to list empty regular files or empty directories.

e.g.

To list all the empty directories

$ find . -type d -empty

Output:

./dir1/praset
./salo/new/data5
./premoc/test1

How to find the past and future dates using date command

 

 

The "date" command in Linux is used to display the current date and time. But, we can also make the date command to display past and future dates based on the inputs/parameters that we pass to it. Below are some of the ways to find the future and past dates.

 

 

 

1) Normal dates:

Today
$ date +%Y-%m-%d
2012-09-29

 

After 2 days
$ date +%Y-%m-%d -d "2 day"
2012-10-01

 

Before 2 days
$ date +%Y-%m-%d -d "-2 day"
2012-09-27

 

Yesterday
$ date +%Y-%m-%d -d "yesterday"
2012-09-28

 

Tomorrow
$ date +%Y-%m-%d -d "next day"
2012-09-30

 

After 3 weeks
$ date +%Y-%m-%d -d "3 weeks"
2012-10-20

 

Before 3 weeks
$ date +%Y-%m-%d -d "-3 weeks"
2012-09-08

 

One Year 3 days after
$ date -d "1 year 3 days"
Wed Oct  2 02:24:53 CDT 2013

 

Last Sunday what was the date ?
$ date -d 'last sun'
Sun Sep 23 00:00:00 CDT 2012

 

How about coming Sunday?
$ date -d 'sun'
Sun Sep 30 00:00:00 CDT 2012

 

This year, what day does "Independence Day" fall on?
$ date --date='15 Aug' +%A
Wednesday

 

2) GMT Time?
$ date -u
Sat Sep 29 07:27:33 UTC 2012

 

3) Epoch time?

Number of seconds from 1970-01-01 UTC (epoch time,it is the number of seconds elapsed since midnight UTC of January 1, 1970, not counting leap seconds)
$ date +%s
1348903671

 

The opposite way conversion, from epoch time to date is shown below.
$ date --date '1970-01-01 UTC 1199968580 seconds'
Sat Sep 29 02:27:51 CDT 2012

 

To make the above command as a function which can be used always, you need to do the following steps.

Make the below function in ~/.bash_profile

epochtime () {
date --date '1970-01-01 UTC '$1' seconds'
}

 

So that you can use it as below in the command line or in a script.

$ epochtime 1199968580
Sat Sep 29 02:27:51 CDT 2012

How to calculate the size of a directory in Linux

 

Let us see how to calculate the size of a directory including its sub directory.

To calculate the total folder size of folder "bin"

$ du -sh bin/
280K bin/

-s, --summarize (display only a total for each argument)
-h, --human-readable ( print sizes in human readable format (e.g., 1K 234M 2G))

How to find largest files in a directory and its sub directories

A situation may occur in your office that you might have to clear the space in the server by deleting the largest files in the log directory. If you want to find and print the top 10 largest files' names (not directories) in a particular directory and its sub directories you can use the following commands.

$ find . -printf '%s %p\n'|sort -nr|head

To restrict the search to the present directory, you can use the parameter "-maxdepth 1" with find command.

$ find . -maxdepth 1 -printf '%s %p\n'|sort -nr|head

And to print the top 10 largest "files and directories":

$ du -a . | sort -nr | head

you can vary the number of files to be printed on the screen by using "head" command as "head -n X" instead of using only "head" in all the above commands to print the top X largest files

How to remove duplicate entries from a file without sorting

 

Usually whenever we want to remove duplicate entries or lines from a file, we need to sort the entries and then eliminate the duplicates using "uniq" command.

But if we want to remove the duplicates and preserve the entries in the same order or sequence, here is the way:

 Sample file:

$ cat sample.txt AAAA FFFF BBBB BBBB CCCC AAAA FFFF DDDD

Using the "uniq" command without sorting the file will not remove all duplicates.

$ uniq sample.txt AAAA FFFF BBBB CCCC AAAA FFFF DDDD

The "sort" command has an option (-u) to sort and uniq, but we will lose the original sequence of the entries

$ sort -u sample.txt AAAA BBBB CCCC DDDD FFFF

Using sort and then uniq commands will remove the duplicates, but sequence?? (Same as above)

$ sort sample.txt | uniq AAAA BBBB CCCC DDDD FFFF

Finally, here is the solution using AWK:

$ awk ' !x[$0]++' sample.txt AAAA FFFF BBBB CCCC DDDD

How to exclude or skip a directory while searching for a file using FIND command

Find command is used to search the files in the Linux environment. This find command will search all the directories and the sub directories in the path which we give in the find command. Often we would like to exclude some directories from our FIND command. Now we will see how to do it.

The below command is used to search all the *.sh files starting from current directory.

$ find . -name "*.sh" -print ./dir11/dir311/helps.sh ./dir11/sold.sh ./dir22/dir322/ace.sh ./dir22/gon.sh ./dir33/dertime.sh ./dir33/tty.sh

Suppose if you want to skip or exclude the directory './dir22' and all files and directories under it, you need to use -prune option with bash find command.

$ find . -path './dir22' -prune -o -name "*.sh" -print ./dir11/dir311/helps.sh ./dir11/sold.sh ./dir33/dertime.sh ./dir33/tty.sh

Sorting the file leaving the first line and writing in a file

Now we can see how to sort a file by leaving the first line as it is. For example, we take a file sample.txt which contains the below data.

chrom:index:forward:reverse

chr01:13:1:2

chr13:23432:4:7

chr01:3445:1:6

chr02:2311:3:1

chr01:212:5:2

chr03:12:1:4

chr02:345:12:6

chr01:45:45:0

If we sort the above data by leaving the first line, the output would be

chrom:index:forward:reverse

chr01:13:1:2

chr01:45:45:0

chr01:212:5:2

chr01:3445:1:6

chr02:345:12:6

chr02:2311:3:1

chr03:12:1:4

chr13:23432:4:7

Now we are going to see the command which sorts the content of the file by ignoring the header line (i.e. the first line). This command sorts the first field in ascending order(String) and the second field in the ascending order(numeric)

head -1 sample.txt  

tail -n +2 sample.txt | sort -t : -k1,1 -k2,2n

1) The first command prints the first line of the file on to the screen.

2) The second command will print all the lines except the first line and then sorts the data using the first field (string sort) and then with the second field(numeric sort).

If you want the output to be written in a file, redirect the output using the re-directional operators.

head -1 sample.txt > output.txt

tail -n +2 sample.txt | sort -t : -k1,1 -k2,2n >> output.txt

The symbol > is for writing into a file. This will clear the output.txt if exists and then writes the data. If the file does not exist, it will create the file and then write the output to it.

The symbol >> will append the data to the existing data in the output.txt file.

How to use Arrays in Shell scripts

We know what an array is. An array is a variable which can store multiple values in the same name. These values can be accessed by using the subscripts. The whole of the array can be accessed by just mentioning the array name without any subscripts. Let us see some of the important operations using array in BASH with the help of the below script.

#!/bin/sh # the name of this script is array.sh array=(ram rick sittik rahim) # assigning values to the array len=${#array[*]} # Number of elements of the array echo "The array contains $len members. They are:" i=0 # Displaying the members of the array while [ $i -lt $len ]; do echo "$i: ${array[$i]}" let i++ done #Some of the operations echo "Adding "bill" to the end of the array" array=( "${array[@]}" "bill" ) echo "Listing all the elements in the array" echo "${array[@]}" echo "Listng all the elements in the array" # This is another way echo "${array[*]}" echo "Deleting \"rick\"" unset array[1] # Same as array[1]="" echo "Now the array is" echo "${array[@]}" echo "length of 3rd element in the array" len1=${#array[2]} echo $len echo "Deleting the whole array" unset array echo "${array[@]}" # The array is empty


The output of the above script would be

$ ./array.sh
The array contains 4 members. They are:
0: ram
1: rick
2: sittik
3: rahim
Adding bill to the end of the array
Listing all the elements in the array
ram rick sittik rahim bill
Listing all the elements in the array
ram rick sittik rahim bill
Deleting "rick"
Now the array is
ram sittik rahim bill
length of 3rd element in the array
6
Deleting the whole array