Small fry Unix commands can get the job done

A look at the tail and date utilities

Summary
In this first of a series on "small fry" commands, Mo shows you some nifty tricks you can do with two simple Unix utilities.

 

The tail display command is particularly useful for examining log files, while date can be used to custom format date fields. (1,700 words)

Unix is filled with hundreds of fascinating little commands that do one or more small jobs. They are designed to work in conjunction with other programs, usually as part of a pipe, in order to get another job done.

Most of them are too simple to warrant a whole article, but these commands are more than just simple short-cuts.

tail
The tail command displays the last 10 lines of a file on standard output (usually the terminal). The number can be modified by using -nn, where nn is the number of lines to display. For example tail -20 log.txt will display the last 20 lines of the file log.txt.

If a + is used, the number of lines specified is from the beginning of the file. For example tail +20 log.txt will skip lines 1 through 19 of log.txt and display lines 20 through to the end of the file.

The tail utility is particularly useful for looking at the last lines logged in a log file. It also has a very useful option called the -f flag. If tail is used with a -f flag, the file being displayed is not closed, but is kept open. The tail program sleeps for one second, wakes, and checks to see if more lines have been added to the file. If they have, the new lines are displayed. This option is particularly useful for monitoring a log file that is currently active and being written to.

Assuming that log.txt is a logging file being written to by one or more programs, tail -f log.txt will display the last 10 lines of log.txt, and then update the screen each second with any new records added to log.txt by other programs.

tail -f is an excellent debugging tool. The first time I used it was to monitor a file that was being filled with transactions by cash registers as cashiers rang up purchases. For testing purposes, we set up the register, rang up different types of transactions, and then examined the result on the Unix screen, immediately. In this way we were quickly able to isolate transactions that were being written to the file incorrectly.

Because tail routes its output to standard output, it is possible to pipe the results of tail into another process. At one point in the debugging, we were concerned that the cash registers were writing garbage or nulls to the transaction file. tail processed the files as text, and it was not possible to see nulls in the data. We used tail to pump the data into od (octal dump) and displayed the files in hex, so that we could examine them for nulls. Using tail -f at the beginning of the stream meant that the transaction file was kept open and constantly pumped into od where the bytes were translated into hex and displayed.

Here is the command:

tail -f trx.txt|od -xc

Testing tail
You might want to try the following to test tail. First we will create a process that writes to a log file. Start the Korn shell by typing ksh and pressing Enter.

Type each of the following lines. As you press Enter after each line, the > prompt will appear indicating that more input is expected. The following lines have been numbered for explanation. The initial $ on line 1 and the > on lines 2 through 6 are provided as prompts for your input. You will be typing everything to the right of the first character.

The command listed below creates a process that sleeps for two seconds and then awakens and appends the date and time to log.txt. The process is submitted to the background by enclosing all of the commands in parentheses at lines 1 and 6, and a final ampersand (&) at line 6. When you press Enter at the end of line 6, the process is submitted to the background and runs as a detached job with no terminal to write to. It doesn't need a terminal since it is writing to log.txt.

On line 7, the operating system responds by giving you the job number of the job that is now running in the background. Make a note of the process id number that appears on your terminal, as you will need it later to kill the process. In this example it is 657.

1    $ ( while true
2    > do
3    > sleep 2
4    > echo `date` >>log.txt
5    > done
6    > )&
7    [1]   657
8    $

Now that log.txt is being filled with date and time stamps, enter the following command:

tail -f log.txt

As you watch the screen, log.txt is filled with more information, and tail -f continues to display the information on the screen as in the following example.

$ tail -f log.txt
Sun Apr 12 14:03:01 PDT 1998
Sun Apr 12 14:03:03 PDT 1998
Sun Apr 12 14:03:05 PDT 1998
Sun Apr 12 14:03:07 PDT 1998

Press Control-C or the Delete key to stop your tail -f process depending on how your terminal is set up.

Finally you need to stop the background process that is logging to the log.txt file. Using the process id number you noted after the process started type:

kill 657

The system will usually respond with a message that the specified job has been terminated as in the following example.

$ kill 657
[1] + Terminated     (while true;do;sleep 2;echo `date` >>log.txt;done)&
$

Formatting with date
The date command is frequently used to set the date and time, but it can also be used to extract the date and time in numerous formats.

The simplest format is the default:

$ date
Sun Apr 12 14:03:01 PDT 1998

A simple logging shell script can be created by using this format. After you have edited this with vi, save it as logit.sh. Then change the permissions on it by typing chmod a+x logit.sh to make it executable.

The logit.sh script

# logit.sh
# log all the arguments passed on the command line, with a date and time stamp
echo `date` $@ >> log.txt

You can run logit.sh by typing:

$ logit.sh This is a line to log

If you then type out the last line of log.txt it will contain:

$ cat log.txt
Sun Apr 12 14:03:01 PDT 1998 This is a line to log

The date may also be formatted by using a + followed by formatting characters as in:

$ date +%D
04/12/98

In this example %D stands for the date in mm/dd/yy format. Other formatting characters are listed below. Your version of date may not support all of these, and may support others. Check your man pages for a list of formatting characters.

 

Formatting characters
%A	the full weekday name (Sunday, Monday etc.)	%n	a newline character
%a	the abbreviation for the weekday (Sun, Mon etc.)	%p	the equivalent of AM or PM
%B	the full month name (January, February etc.)	%R	the 24-hour time (for example, 13:22)
%b	the abbreviated month name (Jan, Feb etc.)	%r	the 12-hour time with AM/PM notation (11:53:29 AM)
%C	the first two digits of the year (00 to 99)	%S	the seconds (00 to 59)
%D	the date in the form mm/dd/yy	%T	the 24-hour time (14:53:29)
%d	the two-digit day of the month (01 to 31)	%U	the week number in the year (00-53) Sunday is the first day of the week All days before the first Sunday of a year are in week 0
%H	the hour in a 24-hour clock (00 to 23)	%u	the weekday number (Monday = 1, Sunday = 7)
%I	the hour in a 12-hour clock (01 to 12)	%w	the weekday number (Sunday = 0, Saturday = 6)
%j	the numeric day of the year (001 to 366)	%Y	the four-digit number of the year (1999)
%M	the minute (00 to 59)	%y	the two-digit year (99)
%m	the month number (01 to 12)	%Z	the time zone name (PDT, EDT)

It is frequently necessary to include spaces in a date format so it is usual to include the date format string in quotes. In the following example, the quotes are necessary because of the space in the format string.

$ date "+%D %R"
04/12/98 13:22

This ability to format the date fields can be used to create log file names that contain a date stamp. In the following version of logit.sh, the log file name is created from the year, month, day, and hour. This ensures that log files don't become too large to be easily handled.

# logit.sh
# log all the arguments passed on the command line, in an
# hourly log file.
logtime=`date "+%y%m%d%H"`
logfile=${logtime}.log
echo `date` $@ >> $logfile

This could be taken down to the minute if a lot of logging is going on by using:

# logit.sh
# log all the arguments passed on the command line, in an
# hourly log file.
logtime=`date "+%y%m%d%H%M"`
logfile=${logtime}.log
echo `date` $@ >> $logfile

One of the advantages of using date to create log file names is the natural sort order provided by the ls command. If the file name is created by concatenating year, month, day, and hour (or hour and minute) the log files will sort together, in time order when a ls command is used.

Contact us for a free consultation.