Log tailing tips

The following examples assume you can tail live logs as follows:

tail -f service.log

That command will show lines being added to the file service.log. Replace it with whatever command will stream logs in your environment.

Although many unix tools (head, wc, sort, xargs, etc.) are available with the same name and similar functionality on both BSD-based platforms (such as Mac OS X) and GNU-based platforms (such as Linux), they often have small differences. All of the examples here assume GNU tools. If you are not sure which version of a tool you have on your system, or what options are available for it, check its man page.

Show only the first 5k new error lines

tail -f service.log | grep ERROR | head -n5000

Lines not including ‘SUCCESS’ or ‘200 OK’ or backtraces

tail -f service.log | grep -v SUCCESS | grep -v '200 OK' | grep -Pv '\tat '

Grep’s -v will print lines that don’t match the pattern. -P is a Perl-style regular expression (used here to indicate a tab character).

Show the 3 lines before and 10 lines after each exception

tail -f service.log | grep -B3 -A10 Exception

Grep’s -B says how many lines before the matching line to print, -A is for lines after. Blocks of matching input will be separated by lines with --.

Watch the rate of a certain exception

tail -f service.log | grep IndividualRequestTimeoutException | \
  pv -lra > /dev/null

pv is Pipe Viewer. It is commonly used to show a progress bar or counter when writing or transferring data using pipes in the shell. In this case we specify -l (line mode - counting newline characters instead of bytes), -r (display the current rate of data transfer) and -a (display the average rate of data transferred so far). pv echos its STDIN to STDOUT (often to another program), but in this case we just want to see the rates and not keep the data, so we redirect the STDOUT to /dev/null to discard it.

Show 10k log lines and save them in a file

tail -f service.log | head -n10000 | tee captured.log

tee lets you write input into a file for later use, but also to see the data as it passes through. In this case the file will be overwritten if it already exists.

Show 10k log lines and append them to a file

tail -f service.log | head -n10000 | tee -a captured.log

You can append rather than overwrite by adding -a.

List instances in captured logs

Assuming each log line starts with an instance ID:

cat captured.log | cut -f1 | uniq

cat simply prints the contents of a file. cut lets you pick out a certain field from each line of input data. By default, fields are delimited by tabs, and we use that to select the instance ID (the first field) from each line. You can specify a different field delimiter with -d. For example, a space: -d' '. uniq prints only unique lines of input (removing repeat lines).

Count instances in captured logs

cat captured.log | cut -f1 | uniq | wc -l

wc is the word counter. -l tells it to just count lines.

Group captured logs by instance, in one file

cat captured.log | sort -sk 1,1 | tee grouped.log

sort will sort lines. -k tells it to sort based on certain (whitespace separated) fields. In this case we want to sort using field 1 to 1 (so, only 1 - the instance ID) (note that if the ending field is unspecified it defaults to the end of the line). Usually when two lines have equal sort fields, the whole lines will also be compared to determine the final order, but -s specifies ‘stable sort’, which preserves the incoming order for lines with equal sort keys.

Group captured logs by instance, in separate files

cat captured.log | cut -f1 | uniq | \
  xargs -n1 -I{} bash -c 'grep ^{} captured.log > instance-{}.log'

xargs transforms lines in its STDIN to arguments to another command. By default, multiple input lines will be converted to multiple arguments to a single command. In this case we specify -n1 to use just one input line at a time. -I{} says that in the command, {} should be replaced with the input string. We use grep ^{} captured.log > instance-{}.log to find the lines in captured.log that begin with the given instance ID and write them to a file with the instance ID in the name. Since this this command involves a shell redirect (>), it needs to be run in a shell rather than as a direct command, so we wrap it in bash -c '...'.

Count log lines in each instance file, sort ascending

wc -l instance-* | LC_ALL=C sort

In most locales, by default, sort will consider whitespace insignificant. This means that a 2 will be ordered after 123, because the first significant character in the former (2) comes after the first significant character in the latter (1). We can specify the C locale in an environment variable to force sort to consider whitespace significant and sort 2 before 123. Note that here we are asking wc to count multiple files rather than the STDIN data (as we did above).