In this post, I describe simple methods for scanning and searching log files – a common activity when researching error messages or FIX rejects.

As an example in this posting, I’m going to trace the source of this reject message:

8=FIX.4.2 9=166 35=j 49=broker 56=buySide 34=11 52=20100622-04:56:07 45=12 372=D 379=Order1 380=4 58=MissingDataException: Missing field. Type 50 10=070

Now we’ll take a look at the mechanics of finding rejects within a log and then use that data to find the source of the error.

FIX Format

Full details of the FIX Format can be found at fixprotocol.org, but for this posting, I’m just going to highlight one element of FIX mechanics.

FIX messages contain binary data (ASCII 1’s). Viewing FIX logs with a GUI text editor can be rather confusing. It can look like this:

8=FIX.4.29=0045535=849=TTD34=446952=20190521-05:00:00.02555=GC48=00...

Or this:

8=FIX.4.2^A9=00455^A35=8^A34=4469^A52=20190521-05:00:00.025^A55=GC^A48=00...

Neither is helpful to understand the underlying message content. To make this more readable, substitute the binary separators with a space or pipe (“|”) character. We do this using command line tools.

Command Line Tools

The command line is available in Linux and Mac natively. Simply open a terminal window and you will get a “bash” command line prompt ($).

On Windows, you’ll need to install command line add-ons such as “babun” (pronounced “baboon”) or “cygwin” or other bash shell add-on for windows.

Typing an ASCII 1 on the Keyboard

In the commands below, we sometimes need to type an ASCII 1. This is also referred to as ctrl-A. Unfortunately, one cannot type ctrl-A directly since that signals to the command line that you want the cursor to move to the start of the line. If that happens, just type the arrow keys to move the cursor back where you want it.

Instead, to enter ctrl-A, you first type ctrl-V (stands for verbatim) and then ctrl-A. If you do it successfully, it should look like this:

$ ^A

This technique will be used in the commands that follow.

Searching for a String

A common reason to search a FIX log is to search for a FIX reject. Here’s the command line I use:

grep -a 35=[3j] fixlogfile.log | sed "s/^A/ /g"

WARNING: DO NOT cut & paste the above command line. Cut and paste will not properly enter the ctrl-A (^A) character. It must by typed (ctrl-V, ctrl-A) as described in the previous section.

Considering the above command line, let’s breakdown the two commands: grep & sed.

Grep is a search tool and will search the file named fixlogfile.log (change that to whatever name you applies to you). The string 35=[3j] tells grep to search for lines containing “35=” followed by either a 3 or j (which in FIX defines a session level reject or a business level reject, respectively).

Sed executes the command (s/^A/ /g) to substitute ASCII 1’s (denoted by ^A, also called ctrl-A) with a space (the “g” tells sed to repeat the command “globally” on the line, not just the first instance).

If you have many rejects, another helpful set of commands is the “head” and “tail” command. With these, we can limit the amount of lines that we write to screen (to stdout, in bash terms). For example, we could limit our output to 5 lines like this:

grep -a 35=[3j] fixlogfile.log | sed "s/^A/ /g" | head -5

You could list the last 5 rejects by using “tail” above instead of “head.”

In the example I used at the beginning of this posting, I showed a reject message with these important fields:

35=j ... 45=12 372=D 379=Order1

Searching for the Related FIX Message

Armed with these details, we can re-scan the log to search for the particular order that caused the reject.

Tag 379 is the RefID, Order1. So we can scan the log for 11=Order1, like this:

grep -a 35=D.*11=Order1 fixlogfile.log | sed "s/^A/ /g" 

The search phrase (or more properly, the “regex”) is “35=D.*11=Order1”. This is simply two search strings in one with the middle wildcard of “.*” meaning any string of chars can be between 35=D and 11=Order1. (In regex, a period means any char and an asterisk means any number of occurrences, zero or more. So, “.*” mean any number of any char.)

In my test log, this yields:

8=FIX.4.2 9=141 35=D 49=buySide 56=broker 34=12 52=20100622-04:33:03 54=1 55=I 38=1000 44=1 40=2 11=Order1 21=2 60=20100622-03:10:39 48=IBM 22=8 99=0 200=BM 10=231 

Counting FIX Messages

To count the number of New Order messages in a log, you can use the “wc” command:

grep -a 35=D fixlogfile.log | wc -l

Conclusion

This posting showed the use of command lines strung together to help analyze log files. There are many other uses of the command line one can use to help isolate problems and understand a large log file without having to read over it manually with a text editor. I’ll go into some additional tools in a followup posting.