Shell Programming Blog Shell Programming Blog

Wednesday, 31 October 2012

Linux Bash Scripting: How to combine two lines by removing newline character using sed

Unknown | 08:57 | | Be the first to comment!

Bash Linux Unix shell scripting

In this post we will see how to remove the newline character and combine two lines in a text file using "sed".

In sed, we have single line version and multi-line version of few commands. One such command is the next command. Click here to see an example for single-line version of next command in sed.

You might want to look at :

The processing space of the sed editor is known as the pattern space. The multi-line next command (N) adds the next line to the existing pattern space which already contains the previous line. So, the sed editor can treat two lines as a single line. Below is an example for the N command.

 

The above sed editor script searches for the line of text that contains the word “first” in it. When it finds the line, it uses the N command to combine the next line with that line. It then uses the substitution command (s) to replace the newline character with a space. The result is that the two lines in the text file appear as one line in the sed editor output.

Tuesday, 30 October 2012

Linux Bash Scripting: How to create directories for each day of the current month

Unknown | 09:59 | | 3 Comments so far


Bash Linux Unix shell scripting

 In this post I have written a script that creates an empty directory for each day of the current month. I used cal command to get the date and month instead of using date command. The reason is that the date command does not work in Solaris. To make it portable, I have used cal command.

The below script does the following
  • It takes the total number of days of the current month 
  • It gets the current month name from the cal command
  • It gets the current year from the cal command
  • It creates an empty directory for each day of the current month
  • Before creating, this checks whether the directory already exists, if yes, the script exits by displaying a message.



You can change the "dir" to the directory in which you want the folders to be created.

Sunday, 28 October 2012

Linux Bash Scripting: How to remove a blank line after a particular line using sed

Unknown | 07:35 | | Be the first to comment!

Bash Linux Unix shell scripting
Removing a blank line after a particular line can be achieved using the single-line next command (n) of the sed editor.

The lowercase n command tells the sed editor to move to the next line of text in the data stream, without going back to the beginning of the sed commands. Normally the sed editor processes all of the defined commands on a line before moving to the next line of text in the data stream. The single‐line next command alters this flow.

In the below example, the data file that contains five lines, two of them empty. The goal is to remove the blank line after the header line but leave the blank line before the last line intact. If you write a sed script to just remove blank lines, you will remove both blank lines.

 

This happened because there was no specific way to find the unique blank line that you wanted to delete. The same can be achieved using the n command as shown in the below script.


The above script searches for the keyword "header" and then moves to the next line when it executes the n command and then applies the d command to delete the blank line below it.

Tuesday, 23 October 2012

Linux Bash Scripting: How to modify the ls command output using pr command

Unknown | 11:29 | | Be the first to comment!

I have received the below question in our "Ask a question" section of this blog.
Q: Using set -A write a script to print the output of the ls command in 5 columns with two spaces between each column. Pretend that ls does not have multi column output.

Normally we will get a multi column output if we use the command ls -A as shown below.


To print the output of the ls command in five columns with two spaces between each column. You can use the following command.


The pr utility is a printing and pagination filter for text files.

From the man page,

-COLUMN, --columns=COLUMN
output COLUMN columns and print columns down, unless -a is used. Balance number of lines in the columns on each page.

-a, --across
print columns across rather than down, used together with -COLUMN 

-SSTRING, --sep-string[=STRING]
separate columns by STRING, without -S: Default separator <TAB> with -J and <space> otherwise (same as -S" "), no effect on column options

-T, --omit-pagination
omit page headers and trailers, eliminate any pagination by form feeds set in input files

The output of the command would be,


suppose if you want to have delimiter as :::: instead of spaces, then you can change the delimiter in the following command.


you can see the difference in the order in which the files are displayed from the above two commands. That is because I purposely did not use -a option to show the difference.

To pretend that ls does not have a multi column output, you need to add the above command as the alias to the ls command as below.


After adding alias, if you use the ls command, you will get the desired output.

Sunday, 21 October 2012

Linux Bash Scripting: Standard file descriptors STDIN, STDOUT and STDERR and input/ouput redirection

Unknown | 08:52 | | Be the first to comment!


 Bash Linux Unix shell scripting

Linux system handles every object as a file. This includes input and output processes also. The linux identifies each file using a file descriptor.

What is a file descriptor?
A file descriptor is a non negative integer which uniquely identifies the open files. Each process can have upto 9 open files at a time. The bash shell reserved three file descriptors(0, 1 and 2) for special purposes.



Linux Standard File Descriptors
File descriptor
Abbreviations
Description
0
STDIN
Standard Input
1
STDOUT
Standard output
2
STDERR
Standard Error






The above three file descriptor controls the input and output of your scripts. The shell uses these descriptors to assign the input and output to appropriate locations (default is the monitor or the terminal).  Now let us see each one in detail.

STDIN

The STDIN file descriptor refers the keyboard in the terminal environment. So, the Linux will read the input from keyboard always unless it is changed.

 

The above “cat” command accepts input from the keyboard.  If you want to force the “cat” command to accept the input other than STDIN, you can use STDIN redirect symbol (<).



STDOUT

The STDOUT descriptor refers to the standard output of the shell (i.e. the monitor). So, the Linux directs all the output to the monitor unless it is changed.



You can also append the data to a file by using the >> symbol.



But whenever the command produces an error, it will be displayed in the monitor even if the output is redirected to a file. That is because the error details have a separate file descriptor (STDERR).

STDERR

By default the STDERR descriptor refers to monitor.



The above example shows that the error message is displayed even after the output is redirected to the file called test. If you want to redirect the error message to some other file, you can do it as below.



If you want to redirect the output and the error message to different files, you can do as below.



If you want to redirect both the error message and the output to the same file, you can use the & operator as shown below.

Wednesday, 17 October 2012

Linux Bash Scripting: how to rename or delete a file which has special characters in its filename

Unknown | 19:46 | | Be the first to comment!


Bash Linux Unix shell scripting

Suppose if you have created a file with a special character in its name because of a typo, you may not be able to access it. You may not even be able to delete the file. For example, see the below files.



If you see the output of ls command, you will not be able to identify the space at the end of the file. Hence you may not be able to access the file unless you specify a space at the end of the file using quotes.

The ? is a wild character for which any character can be substituted. Hence you may end up deleting or reading some other file.

To handle this situation, we can make use the inode of the file. Every file has a unique inode number with which we can identify the file. 



To rename the file:

You can specify the inode number in the find command as below to rename the files. The below command renames the file using  the inode number.



To delete the file:

You can delete the file using the inode number of the file with the help of "find" command as below.

 

Monday, 15 October 2012

Linux Bash Scripting: Perform floating point calculation/arithmetic/operation using BC

Unknown | 03:07 | | Be the first to comment!


 Bash Linux Unix shell scripting

As we have discussed on performing Arithmetic operations on our previous post, now we will look at the floating point operations in the shell script in this post.

You might want to look at : 
Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

A huge limitation in the Bash shell is that it supports only integer arithmetic. There are several solutions to overcome this limitation. The famous solution is Bash Calculator.

Basics of BC(Bash Calculator):

The bash calculator is actually a programming language that allows you to enter floating point expressions at the command line and then interprets the expressions, calculates and then returns the result. The Bash calculator recognizes the following:
  • Numbers (both integer and floating point)
  • Variables (both simple variables and arrays)
  • Comments (lines starting with a pound sign or the C language /* */ pair
  • Expressions
  • Programming statements (such as ifthen statements)
  • Functions
 The Bash calculator can be invoked from the command line using "bc" command.



As you can see from the above example, each time you enter the expression, you get the result immediately. To exit the BC you need to type quit.

Floating point decimal is controlled by a variable called "scale". Using this variable we can specify no. of decimal places that we would like to see in our result.



The default value of the scale is zero. As you can see from the above example, the Bash calculator returns the answer with zero decimal places. After setting the scale to 4, it displays the result with 4 decimal places.

The -q parameter suppresses the lengthy welcome banner of the Bash Calculator. In addition to the normal numbers, the bash calculator also understands the variables.



Once a variable is defined you can use the variable throughout the Bash calculator session. The "print" statement is used to print the variables and numbers.

Using BC in the scripts:

We can use the BC in our script in two ways.

1) Using the back tick character to run the "bc" command and assign the result to the variable as shown below.

        variable=‘echo “options; expression” | bc’

Below is a sample script which illustrates the above usage.




This method will be useful for small calculations. For the calculations which involves more complex expressions, the next method will be useful.

2) Using the inline input redirection, you can perform calculations which involves more numbers. The syntax of this method would be,

        variable=‘bc << EOF
        options
        statements
        expressions
        EOF
        ’

The EOF text string indicates the beginning and end of the inline redirection data. Remember that the backtick characters are still needed to assign the output of the bc command to the variable.

Below is an example script to show the usage of input redirection in BC.



Note: The variables declared inside the Bash calculator is valid only inside the BC and can't be used in the shell script.

Sunday, 14 October 2012

Linux Bash Scripting: How to ignore specific commands from being stored in the history in Linux

Unknown | 19:39 | | 3 Comments so far


Bash Linux Unix shell scripting

If you see the history of your commands using the "history" command, you can see all the commands that you executed in the command line. If you don't want certain basic commands to be stored in your history, you can ignore them using HISTIGNORE. You can specify all the commands that you want to ignore in the HISTIGNORE. 

In the below code, I have ignored the commands pwd, ls, ls -ltr and l using HISTIGNORE. Then I am executing some commands in the command line. Finally I'm checking the output of the history command.


[Note that history did not record ls, l, ls -ltr and pwd]

Please note that adding ls does not exclude the command ls -al from being included in the history. So, you need to specify the complete command to make it as an exception.

Linux bash scripting: How to repeat the previous command quickly in the Linux command line

Unknown | 11:32 | | Be the first to comment!


There are four different ways by which we can repeat the previous commands that we executed in the command line.

1) use the "UP arrow" to view the previous command and press Enter to execute it.
2) Type !! and press enter from the command line
3) Type !-1 and press enter from the command line.
4) Press Control+P to display the previous commands. You can keep pressing the Ctrl+P to older commands. When you see the required command you can press enter to execute it

Saturday, 13 October 2012

Linux bash scripting: How to search the command history (previously executed commands)

Unknown | 10:47 | | Be the first to comment!


Bash Linux Unix shell scripting

We often need to type long commands in the command line which we typed earlier. Instead of typing it again we can search the command history for the command we executed and execute it again. This technique will come handy when you are required to type a long command that you executed previously.

To search the command history and execute the command without editing it, Press Ctrl+R and type the keyword present in the previously executed command.

In the below example, I searched for the word "work", which displayed the previous command "cd /home/sittik/test/DDI/work" in the history that contained the word work.



Suppose if you need to edit the command before executing it, you can use the left or right arrow key without pressing enter. For example, you can search for "httpd" and which will display the command "service httpd stop" from the command history. You can change this command to "service httpd start" and execute it as shown below.




linux Bash Scripting: how to display the history of commands along with the timestamp

Unknown | 10:35 | | Be the first to comment!


Bash Linux Unix shell scripting

Usually the history command will display the command number and the command when you execute the command from the command line. Suppose if you want to add the timestamp along with the command history for auditing purpose, you can do it using the HISTTIMEFORMAT as shown below.



Friday, 12 October 2012

Linux Bash Scripting: How to rename multiple files in a single go

Unknown | 10:39 | 1 Comment so far


Bash Linux Unix shell scripting

You might come across a situation where you need to rename lot of files with similar file names either by editing the file name or by truncating the last part of the file name. There are two ways by which you can accomplish the task.

Method 1:

You can use the "rename" command to rename multiple files in a single go.

Syntax:

rename from to file...

- from is the word that you want to change in the file name
- to is the word that you want to replace with
- file is the file that want to rename ( you can use regular expression to select multiple files here

Example:

rename .html .htm *.html

This will change the extension of all the html files present in the current directory. The from and to keywords need not be the last characters of the file names. It can be anywhere in the file names

Method 2:

This method utilizes the "for" loop and "mv" command to rename all the files in a single go.




The above script will change all the html file name from *.html to *.htm

Windows tips and Tricks

Unknown | 10:16 | Be the first to comment!


I have personally tried and tested most of the tweaks discussed in this guide. However, I request fellow readers to be cautious while trying it out with your system. Always take a backup copy of all important data/registry before attempting to change the system/registry settings. Remember, prevention is always better than cure.



Thursday, 11 October 2012

Linux Bash Scripting: how to print a big chess board anywhere on the screen

Unknown | 11:12 | Be the first to comment!


Bash Linux Unix shell scripting


In this post, we will see how to create a big chess board that can be displayed anywhere on the screen rather than being displayed left aligned. Below is the script which displays the Chess board on the screen.



If you want to make changes to the above script for changing the size and the position of the chess board, you can change the following variables in the script.

1) row - to change the number of lines to leave from top before printing the chess board
2) col - to change the number of columns to leave from left before printing the chess board
3) j in second for loop - to increase or decrease the height of the block
4) No. of spaces in echo statement to change the width of the block


The same script using C style for loop:



Friday, 5 October 2012

Have questions?? Need Clarifications???

Unknown | 08:34 | 6 Comments so far


Hi All,

I know that lot of us would get stuck while doing programming in Shell scripting or doing something in Linux. If you are badly in help and need to find an answer, you can make use of this page. Please post your questions on shell scripting in the comments box along with your email. Your questions will be answered and will be sent to your email. Please make use of this free service :)

Thanks,
Beneficial Knowledge

Thursday, 4 October 2012

how to change the default size of the terminal window in Solaris

Unknown | 09:13 | | Be the first to comment!


To change the default size of the terminal window in Solaris, you need to do the following.

1) Go to the directory /usr/share/vte/termcap/ and edit the file xterm

     i.e., $ vi /usr/share/vte/termcap/xterm

2) Look for :co#115:it#8:li#36:\

3) In the above line you can change the numbers like for example, put 77 after "co#" and put 48 after "li#"

4) Then every terminal you open will open with the size that you have mentioned in the above line.

OR

You can also add an alias as below in the .profile or .bashrc or .cshrc, wherever it is applicable for you.

alias xterm='xterm -geometry 77x48'

Any one of the above mentioned methods should do the work of changing the default terminal size. 

Note : You need to reboot your PC after you have made the above changes for your changes to be effective.

Tuesday, 2 October 2012

Linux Shell Scripting: how to list the files alone in a directory

Unknown | 09:21 | Be the first to comment!


Bash Linux Unix shell scripting

 Often we tend to look into the directories to check what files are present in the directory. When we try to list the files using ls command, all files along with the directories will be displayed. What if we want to list only the files and not the directories???

We will see how to do that in this post.

1) Using "find" command:

$ find . -type f -maxdepth 1

The above command will list also the hidden files. But if you want to avoid the hidden files to be displayed in your results, you need to use the below command.

$ find . -type f -maxdepth 1 \( ! -iname ".*" \)


2) To derive it from the "ls" command, we can use the below commands.

$ ls -l | grep -v ^d

$ ls -l | awk 'NR!=1 && !/^d/ {print $NF}'


Monday, 1 October 2012

Perform Arithmetic operations or mathematical calculations in Shell scripting - expr command

Unknown | 09:54 | Be the first to comment!


 Bash Linux Unix shell scripting

 The command which is used to perform mathematical or Arithmetic calculations is "expr" command.

Syntax:
expr <operand 1> operator <operand 2>



Examples:
$ expr 2 + 3
$ expr 5 - 3
$ expr 7 % 3
$ expr 12 \* 3
$ expr 12 / 6

However, even though the operations looks easy for performing, the "expr" command has a drawback. It uses special character (*) for multiplication which is recognized as a wild character by Linux/Unix. So, if you try to do multiplication as

$ expr 12 * 3

Then you will receive an error. You need to use an escape character (\) along with the (*) character to indicate that this should not be considered as a wild character. Hence the expression should be

$ expr 12 \* 3

If you are going to display the result or assign the result to some variable, then you need to include the expr command in a back tick(`). This symbol is not commonly used. You can find this symbol along with the (~) key in the US keyboard. This is not a single quote('). This is different from single quote and the double quote. Let's see the examples below illustrating the usage.



The back tick is used to inform the Linux system that whatever included within the back tick should be considered as a command and not a string.

To overcome this complexity, Bash shell introduced a new method of doing arithmetic calculations by using brackets. But still the Bash shell includes the expr command to stay compatible with Bourne shell.

Using Brackets:
Another easy way of doing Arithmetic calculation is to enclose the calculation using a dollar sign($) and square brackets ([]).

Examples:



When you are using brackets for doing arithmetic calculations, you don't need to worry about the wild character (*). Linux will consider whatever inside the bracket as  an expression and not as a wild character.

Finally, there is a major limitation in performing math in Bash shell. That is the Bash supports only  integer numbers and not floating numbers. For example if you do the below calculation, you will get only the integer as a result.

$ echo $[100 / 45]
2

But, the z shell (zsh) provides full floating point arithmetic operations. There is also an alternative way to do the floating point calculations in Bash shell. That is by using BC (Bash Calculator).

How to print the last field in a file in Linux/Unix Shell scripting

Unknown | 08:32 | Be the first to comment!


Bash Linux Unix shell scripting


Let us see how to print the last field alone from a file which contains several fields.

Let's say that the input is the below file.






Now we need to Print the last field. i.e.

The required Output would be:
----------------------------------
7
4
8
8

We can do this by three ways.

1) The normal awk way of doing it:

$ awk 'BEGIN {FS=":"} {print $NF}' priority.txt

2) Using sed:

$ sed -n 's/.*://;p' priority.txt

3) Using grep:

$ grep -o '[^:]*$' priority.txt

How to find the latest file in a directory

Unknown | 07:21 | Be the first to comment!


Bash Linux Unix shell scripting

Suppose if you hava a directory with a huge set of files(more than 500) in a log directory, you would want to find the latest file in the directory. Now we can see few ways to that.

for example, Let us take the below scenario




1) Now our goal is to print the last field of the last line (except the dir old) from the "ls -lrt" output in that directory.




So basically to print the last field of last line(in this case the last field of last line of "ls -lrt" command), we can use the below command.

$ ls -lrt | awk '{ f=$NF }; END{ print f }'


2) This can also be acheived by using Head command.

$ ls -t1 | head -n1


3) This can also be acheived by using Find command.

$ find . -not -type d -printf "%T+ %p\n" | sort -n | tail -1

Linux/Unix Shell Scripting topics explained on requests

Unknown | 02:11 | Be the first to comment!

  • Are you new to Shell scripting? 
  • Are you learning Shell Scripting either in Linux or Unix?
  • Is it tough for you to understand? 
  • Do you feel confused while trying to learn a topic?
  • Is there anything in shell scripting that you are not able to understand?

Please let us know (leave your comments below). We are here to help you understand(Don't worry!!! We won't charge u for it :P). We can explain things neatly for you with good examples and pictures.

Thanks,
Beneficial Knowledge
 

 

Shell Programming Copyright © 2012 Shell Programming theme is Designed by Abusittik, Shell Programming