Linux Bash Scripting: How to combine two lines by removing newline character using sed

In this post we will see how to remove the newline character and combine two lines in a text file using "sed".

In sed, we have single line version and multi-line version of few commands. One such command is the next command. Click here to see an example for single-line version of next command in sed.

You might want to look at :

How to remove a blank line after a particular line using sed

The processing space of the sed editor is known as the pattern space. The multi-line next command (N) adds the next line to the existing pattern space which already contains the previous line. So, the sed editor can treat two lines as a single line. Below is an example for the N command.

$ cat sample.txt This is the header line. This is the first line. This is the second line. This is the last line. $ $ sed ‘/first/{ N ; s/\n/ / }’ sample.txt This is the header line. This is the first line. This is the second line. This is the last line. $  

The above sed editor script searches for the line of text that contains the word “first” in it. When it finds the line, it uses the N command to combine the next line with that line. It then uses the substitution command (s) to replace the newline character with a space. The result is that the two lines in the text file appear as one line in the sed editor output.

Linux Bash Scripting: How to create directories for each day of the current month

 In this post I have written a script that creates an empty directory for each day of the current month. I used cal command to get the date and month instead of using date command. The reason is that the date command does not work in Solaris. To make it portable, I have used cal command.

The below script does the following

• It takes the total number of days of the current month 
• It gets the current month name from the cal command
• It gets the current year from the cal command
• It creates an empty directory for each day of the current month
• Before creating, this checks whether the directory already exists, if yes, the script exits by displaying a message.

#!/bin/bash days=`echo $(cal) | tail -c 3` month=`echo $(cal) | head -c 3` year=`echo $(cal) | cut -d' ' -f2` for day in `seq $days` do folname=$day$month$year echo $folname if [ -d dir/$folname ] then echo "directory $folname already exists" exit else mkdir -p "dir/$folname" fi done

You can change the "dir" to the directory in which you want the folders to be created.

Linux Bash Scripting: How to remove a blank line after a particular line using sed

Removing a blank line after a particular line can be achieved using the single-line next command (n) of the sed editor.

The lowercase n command tells the sed editor to move to the next line of text in the data stream, without going back to the beginning of the sed commands. Normally the sed editor processes all of the defined commands on a line before moving to the next line of text in the data stream. The single‐line next command alters this flow.

In the below example, the data file that contains five lines, two of them empty. The goal is to remove the blank line after the header line but leave the blank line before the last line intact. If you write a sed script to just remove blank lines, you will remove both blank lines.

$ cat data This is the header line. This is a data line. This is the last line. $ $ sed ‘/ˆ$/d’ data This is the header line. This is a data line. This is the last line. $  

This happened because there was no specific way to find the unique blank line that you wanted to delete. The same can be achieved using the n command as shown in the below script.

$ sed ‘/header/{n ; d}’ data This is the header line. This is a data line. This is the last line. $

The above script searches for the keyword "header" and then moves to the next line when it executes the n command and then applies the d command to delete the blank line below it.

Linux Bash Scripting: How to modify the ls command output using pr command

I have received the below question in our "Ask a question" section of this blog.

Q: Using set -A write a script to print the output of the ls command in 5 columns with two spaces between each column. Pretend that ls does not have multi column output.

Normally we will get a multi column output if we use the command ls -A as shown below.

/home/sittik/programs $ ls -A arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp

To print the output of the ls command in five columns with two spaces between each column. You can use the following command.

ls -A | pr -a5 -S" " -T

The pr utility is a printing and pagination filter for text files.

From the man page,

-COLUMN, --columns=COLUMN

output COLUMN columns and print columns down, unless -a is used. Balance number of lines in the columns on each page.

-a, --across

print columns across rather than down, used together with -COLUMN 

-SSTRING, --sep-string[=STRING]

separate columns by STRING, without -S: Default separator <TAB> with -J and <space> otherwise (same as -S" "), no effect on column options

-T, --omit-pagination

omit page headers and trailers, eliminate any pagination by form feeds set in input files

The output of the command would be,

/home/sittik/programs $ ls -A | pr -a5 -S" " -T arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp setting .setting.swp

suppose if you want to have delimiter as :::: instead of spaces, then you can change the delimiter in the following command.

/home/sittik/programs $ ls -A | pr -5 -S"::::" -T arr.sh ::::com ::::gcom ::::loops ::::setting bigchess.sh::::file ::::lbigchess.s::::samp ::::.setting.sw chess.sh ::::fileread.sh

you can see the difference in the order in which the files are displayed from the above two commands. That is because I purposely did not use -a option to show the difference.

To pretend that ls does not have a multi column output, you need to add the above command as the alias to the ls command as below.

alias ls='ls -A | pr -a5 -S" " -T'

After adding alias, if you use the ls command, you will get the desired output.

/home/sittik/programs $ ls arr.sh bigchess.sh chess.sh com file fileread.sh gcom lbigchess.sh loops samp setting .setting.swp

Linux Bash Scripting: Standard file descriptors STDIN, STDOUT and STDERR and input/ouput redirection

 

Linux system handles every object as a file. This includes input and output processes also. The linux identifies each file using a file descriptor.

What is a file descriptor?
A file descriptor is a non negative integer which uniquely identifies the open files. Each process can have upto 9 open files at a time. The bash shell reserved three file descriptors(0, 1 and 2) for special purposes.

Linux Standard File Descriptors
File descriptor	Abbreviations	Description
0	STDIN	Standard Input
1	STDOUT	Standard output
2	STDERR	Standard Error

The above three file descriptor controls the input and output of your scripts. The shell uses these descriptors to assign the input and output to appropriate locations (default is the monitor or the terminal).  Now let us see each one in detail.

STDIN

The STDIN file descriptor refers the keyboard in the terminal environment. So, the Linux will read the input from keyboard always unless it is changed.

$ cat This is test This is second test  

The above “cat” command accepts input from the keyboard.  If you want to force the “cat” command to accept the input other than STDIN, you can use STDIN redirect symbol (<).

$ cat < filetest This is test This is second test $

STDOUT

The STDOUT descriptor refers to the standard output of the shell (i.e. the monitor). So, the Linux directs all the output to the monitor unless it is changed.

$ ls –l > test $ cat test -rwxrwxrwx 1 sittik owner 260 Oct 16 07:03 arr.sh -rwx--x--x 1 sittik owner 963 Oct 11 01:25 bigchess.sh -rwx--x--x 1 sittik owner 389 Oct 4 00:47 chess.sh

You can also append the data to a file by using the >> symbol.

$ who >> test $ cat test -rwxrwxrwx 1 sittik owner 260 Oct 16 07:03 arr.sh -rwx--x--x 1 sittik owner 963 Oct 11 01:25 bigchess.sh -rwx--x--x 1 sittik owner 389 Oct 4 00:47 chess.sh sittik pts/0 2012-10-20 05:50 (132.127.131.35)

But whenever the command produces an error, it will be displayed in the monitor even if the output is redirected to a file. That is because the error details have a separate file descriptor (STDERR).

STDERR

By default the STDERR descriptor refers to monitor.

$ ls –al badfile > test ls: badfile: No such file or directory $ cat test $

The above example shows that the error message is displayed even after the output is redirected to the file called test. If you want to redirect the error message to some other file, you can do it as below.

$ ls –al badfile 2> test $ cat test ls: badfile: No such file or directory $

If you want to redirect the output and the error message to different files, you can do as below.

$ ls –al goodfile badfile 2> badtest 1> goodtest $ cat badtest ls: badfile: No such file or directory $ cat goodtest -rwx--x--x 1 sittik owner 389 Oct 14 10:47 goodfile $

If you want to redirect both the error message and the output to the same file, you can use the & operator as shown below.

$ ls –al badfile goodfile &> commonfile $ cat commonfile ls: badfile: No such file or directory -rwx--x--x 1 sittik owner 389 Oct 14 10:47 goodfile $

Linux Bash Scripting: how to rename or delete a file which has special characters in its filename

Suppose if you have created a file with a special character in its name because of a typo, you may not be able to access it. You may not even be able to delete the file. For example, see the below files.

$ touch "test-file1 " [Note: There is a space at the end] $ touch "test'file2" [Note: There is a single quote in the middle] $ touch "test?file3" [Note: There is a question mark in the middle] $ ls -1 test* test-file1 test'file2 test?file3

If you see the output of ls command, you will not be able to identify the space at the end of the file. Hence you may not be able to access the file unless you specify a space at the end of the file using quotes.

The ? is a wild character for which any character can be substituted. Hence you may end up deleting or reading some other file.

To handle this situation, we can make use the inode of the file. Every file has a unique inode number with which we can identify the file. 

$ ls -i1 test* 13287439 test-file1 13287623 test'file2 13287981 test?file3

To rename the file:

You can specify the inode number in the find command as below to rename the files. The below command renames the file using  the inode number.

$ find -inum 13287623 -exec mv {} testfile3new \; $ ls -i1 test* 13287439 test-file1 13287623 test'file2 13287981 testfile3new

To delete the file:

You can delete the file using the inode number of the file with the help of "find" command as below.

$ find -inum 13287623 -exec rm {} \; $ ls -i1 test* 13287439 test-file1 13287981 testfile3new [Note: The file with name "test'file" is now removed]  

Linux Bash Scripting: Perform floating point calculation/arithmetic/operation using BC

 

As we have discussed on performing Arithmetic operations on our previous post, now we will look at the floating point operations in the shell script in this post.

You might want to look at : 
Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

A huge limitation in the Bash shell is that it supports only integer arithmetic. There are several solutions to overcome this limitation. The famous solution is Bash Calculator.

Basics of BC(Bash Calculator):

The bash calculator is actually a programming language that allows you to enter floating point expressions at the command line and then interprets the expressions, calculates and then returns the result. The Bash calculator recognizes the following:

• Numbers (both integer and floating point)
• Variables (both simple variables and arrays)
• Comments (lines starting with a pound sign or the C language /* */ pair
• Expressions
• Programming statements (such as if‐then statements)
• Functions

 The Bash calculator can be invoked from the command line using "bc" command.

$ bc bc 1.06.95 Copyright 1991‐1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type ‘warranty’. 9.5 * 3.5 33.25 3.156 * (3 + 5) 25.248 quit $

As you can see from the above example, each time you enter the expression, you get the result immediately. To exit the BC you need to type quit.

Floating point decimal is controlled by a variable called "scale". Using this variable we can specify no. of decimal places that we would like to see in our result.

$ bc ‐q 3.44 / 5 0 scale=4 3.44 / 5 .6880 quit $

The default value of the scale is zero. As you can see from the above example, the Bash calculator returns the answer with zero decimal places. After setting the scale to 4, it displays the result with 4 decimal places.

The -q parameter suppresses the lengthy welcome banner of the Bash Calculator. In addition to the normal numbers, the bash calculator also understands the variables.

$ bc ‐q var1=10 var1 * 4 40 var2 = var1 / 5 print var2 2 quit $

Once a variable is defined you can use the variable throughout the Bash calculator session. The "print" statement is used to print the variables and numbers.

Using BC in the scripts:

We can use the BC in our script in two ways.

1) Using the back tick character to run the "bc" command and assign the result to the variable as shown below.

        variable=‘echo “options; expression” | bc’

Below is a sample script which illustrates the above usage.


$ cat test11 #!/bin/bash var1=20 var2=3.14159 var3=‘echo “scale=4; $var1 * $var1” | bc’ var4=‘echo “scale=4; $var3 * $var2” | bc’ echo The final result is $var4 $

This method will be useful for small calculations. For the calculations which involves more complex expressions, the next method will be useful.

2) Using the inline input redirection, you can perform calculations which involves more numbers. The syntax of this method would be,

        variable=‘bc << EOF
        options
        statements
        expressions
        EOF
        ’
The EOF text string indicates the beginning and end of the inline redirection data. Remember that the backtick characters are still needed to assign the output of the bc command to the variable.

Below is an example script to show the usage of input redirection in BC.

$ cat test12 #!/bin/bash var1=10.46 var2=43.67 var3=33.2 var4=71 var5=‘bc << EOF scale = 4 a1 = ( $var1 * $var2) b1 = ($var3 * $var4) a1 + b1 EOF ’ echo The final answer for this mess is $var5 $

Note: The variables declared inside the Bash calculator is valid only inside the BC and can't be used in the shell script.

Linux Bash Scripting: How to ignore specific commands from being stored in the history in Linux

If you see the history of your commands using the "history" command, you can see all the commands that you executed in the command line. If you don't want certain basic commands to be stored in your history, you can ignore them using HISTIGNORE. You can specify all the commands that you want to ignore in the HISTIGNORE. 

In the below code, I have ignored the commands pwd, ls, ls -ltr and l using HISTIGNORE. Then I am executing some commands in the command line. Finally I'm checking the output of the history command.

$ export HISTIGNORE="pwd:ls:ls -ltr:l:" $ ls $ l $ ls -ltr $ pwd $ cd $HOME $ history | tail -3 379 export HISTIGNORE="pwd:ls:ls -ltr:l:" 380 cd $HOME 381 history
[Note that history did not record ls, l, ls -ltr and pwd]

Please note that adding ls does not exclude the command ls -al from being included in the history. So, you need to specify the complete command to make it as an exception.