1. Introduction
FAST (FIX Adapted for Streaming) was developed by the FIX Protocol organization to bring the greater benefits of standardization to market data and deliver optimized performance for the exchange of electronic financial information. Built around a data compression algorithm, it significantly reduces bandwidth requirements and latency between sender and receiver. FAST works particularly well at improving performance during periods of peak message rates. While FAST has grown out of market data, it is applicable and can be used with all FIX messages to offer flexibility in the way that FIX data is formatted for transmission.
FAST uses several techniques to reduce bandwidth. This FAST tutorial covers these techniques briefly on this page and in more detail in later sections.
1.1. FAST Template
FAST Templates define the field layout of messages so the message itself does not describe individual field names or tags. Instead, fields’ identities are inferred from their position within the message as described by the template. This template is shared by both the sender and the receiver (that is, both the encoder and the decoder) and is usually held in an XML file.
Picture 1.1 – “Hello World message transfer”
1.2. FAST Presence Map (PMap)
The Presence Map (or PMap) is a variable length bit field used to indicate whether or not a particular field is present in a message.
This allows the encoder leave out fields from the message in certain cases, such as:
- field value is the same as in the previous message (common in fields like trade date, settlement date, etc) or
- is one greater that last message (common case for sequence number).
The template defines the rules to use when a field is not present in the message.
1.3. Variable Sized Fields
Fields in FAST do not have a fixed size and do not use a field separator. Instead, there is a notion of a stop bit (the high order bit on each byte of the message acts as a stop bit) signaling the end of the field.
All of the above concepts used together allow the sender to compress a message (sometimes as much as 90%) and the receiver to restore the original message content from the encoded bytes.
The next section looks at a small message example to show encoding and decoding in practice.
2. Hello World Example
To introduce the encoding and decoding process, we describe how a trivial message with only 1 field would be FAST encoded and decoded.
2.1. Input Message
Here’s the message we’ll start with:
58=HelloWorld<SOH>.
The SOH (ASCII 1) is the FIX separator byte.
2.2. The Template
This message requires a very simple template (expressed in XML and described in greater detail in later sections):
2.3. Stop Bits
The high order bit of each byte of the message is reserved to indicate the end of the field.
So each of the fields described below will only use 7 bit bytes and the 8th bit will be set on the last byte of the field.
2.4. Encoding the Message:
To encode the message, we need to produce a Presence Map (PMap) that goes onto the front of the compressed FAST message followed by the encoded fields.
PMap is described in detail in section 3.1, but simply has 1 bit for each field to indicate whether the field has been included in subsequent bytes, or has been omitted.
Note: In our example, if the Text field is omitted, it will default to an empty string. (Because of the xml element: <default value=””/> in the above template.)
When encoding the message the template ID is usually put into the message right after the PMap. (We’ll discuss this more later.) This requires a PMap bit, too.
So for our trivial one field message, we need a PMap with 2 bits:
- indicate template ID is in the message
- indicate Text field is in the message
Encoding steps:
-
- These are the high order bits and the rest are zeros: 110 0000.
This is a 1 byte PMap, so its 8th bit is set to 1 to indicate it is the last byte of the PMap: 1110 0000 = 0xE0.
- These are the high order bits and the rest are zeros: 110 0000.
-
- The next field is the template field (for template 1): 000 0001.
Again, this is a 1 byte field so we set the 8th bit to indicate the end: 1000 0001 = 0x81.
- The next field is the template field (for template 1): 000 0001.
- The next field is the text field contents: “HelloWorld”.
In the hex format it is: H=0x48, e=0x65, l=0x6C, l=0x6C, o=0x6F, W=0x57, o=0x6F, r=0x72, l=0x6C, d=0x64.
The last byte will have its high order bit set to “1”. Using this, we have the following binary representation of our text field:
01001000 01100101 01101100 01101100 01101111 01010111 01101111 01110010 01101100 11100100
2.5. Encoded FAST Message “HelloWorld”
The fields strung together are shown below:
Picture 2.1 – “HelloWorld message encoding”
2.6. Decoding the Message
Looking at the just the above binary data and the template file, we can reconstruct the original message content by reversing the encoding process.
Our input (encoded) FAST message is:
HEX format: 0xE0 0x81 0x48 0x65 0x6C 0x6C 0x6F 0x57 0x6F 0x72 0x6C 0xE4
Binary format: 11100000 10000001 01001000 01100101 01101100 01101100 01101111 01010111 01101111 01110010 01101100 11100100
Lets decode it step by step:
-
- Extracting Message PMap. Message PMap must be present before each encoded message so the first step of decoding process must be defining PMap.
We get the first byte from the stream: “11100000“, check its stop bit (high order bit or first bit).
It is set to “1”, so our encoded PMap consists of 1 byte: 11100000. We remove the stop bit which yields the decoded message PMap: 1100000
- Extracting Message PMap. Message PMap must be present before each encoded message so the first step of decoding process must be defining PMap.
-
- Decoding template ID. We get the first bit from our PMap which is “1” and it indicates presence of template ID value in the input message.
Using the same way as for PMap decoding we decode template ID:
- We get the next byte after PMap from the stream. It is the second byte: “10000001” and the stop bit of this byte is “1”. So our template ID is the 1 byte value.
- After removing stop bit from template ID encoded value we receive: "0000001" = "1" which is the actual value of template ID.
Template ID is used to define FAST message template which contains references for field decoding.
- Decoding template ID. We get the first bit from our PMap which is “1” and it indicates presence of template ID value in the input message.
- “HelloWorld” field decoding. The next bit (second bit in this case) from our PMap is set to “1” and it indicates that the “HelloWorld” field is present in the input message too.
As our text field consists of more than one byte we must get next input bytes (after template ID) one by one. This is required to find the byte which has stop bit set to “1” and indicates the end of “HelloWorld” encoded field.
Byte which has high order bit (stop bit) set to “1” is the last byte in input stream so our encoded text field is:01001000 01100101 01101100 01101100 01101111 01010111 01101111 01110010 01101100 11100100
Stop bit decoding of this text field is in following. We must replace the stop bit of last byte of text field from “1” to “0”.
After this we receive actual value of our text field:Binary format: 01001000 01100101 01101100 01101100 01101111 01010111 01101111 01110010 01101100 01100100 HEX format: 0x48 0x65 0x6C 0x6C 0x6F 0x57 0x6F 0x72 0x6C 0x64 Character format: HelloWorld
We combine decoded output fields and receive message: 58=HelloWorld<SOH>.
In the picture below are shown all of these steps:
Picture 2.2 – “HelloWorld message decoding”
The next section will describe the different FAST elements (such as the FAST Template and the PMap) in much greater detail and walk through more complex decoding examples.
3. FAST Message Decoding Process
When decoding a FAST message, we need to determine the template ID, retrieve the template, and combine the message content with the template definitions and operators, reproduce the original message.
FAST decoder decodes each message using the following algorithms:
Picture 3.2 – “Message decoding algorithm”
Picture 3.3 – “Field decoding algorithm”
To summarize, we follow these steps to decode FAST messages:
- 3.1 Extract Presence Map
- 3.2 Determine Message Template ID
- 3.3 Decode each message field
3.1. Extract Presence Map
A Presence Map (or “PMap” as it is commonly called) occurs at the beginning of each FAST message and sometimes within a FAST message. It has bits representing each field in the message (that is not ‘required’, defined later) to indicate whether or not that field is present in the message body. If our FAST message includes a repeating sequence where any field requires a PMap bit, then a PMap will be present before each group of the sequence.
General structure of full FAST message is shown on the Picture 3.4
Picture 3.4 – “FAST message structure”
if (presenceMap.hasNextPresenceBit()) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
...
}
Logically a PresenceMap has an infinite suffix of zeroes. This makes it possible to truncate a PresenceMap that ends in a sequence where the bits are all zero. This concept is shown below.
Picture 3.5 – “PMap infinite suffix of zeroes”
public boolean hasNextPresenceBit() {
if (isEmptyPMap) {
return false;
}
if (offset > lastPMapBytePosition) {
//Implies that presence map is empty and has the infinitive suffix of zeros
isEmptyPMap = true;
return false;
}
boolean bitOnPosition = hasBitOnPosition(currentByte, currentBitPosition);
currentBitPosition++;
if (currentBitPosition == NUMBER_BITS_IN_BYTE) {
currentBitPosition = 1;
offset++;
currentByte = data[offset];
}
return bitOnPosition;
}
Our Presence Map class example does not perform stop bit decoding to store PMap data. Encoded bytes are stored in the actual encoded format. PMap does not use stop bit values so it passes them.
Checking if a bit for a specific position is set can be easily defined as follows:
public static boolean isPositionSet(byte byteToCheck, int bitPosition) {
if (bitPosition > 0) {
byteToCheck = (byte) (byteToCheck << bitPosition);
}
if (byteToCheck < 0) {
return true;
}
return false;
}
In the sample above we are shifting byte to have required bit on the first position and if first bit in the shifted byte is 1 we return true.
For example we have the FAST message which contains only templateID field: 0xC0 0xA9
Here its bits representation: 11000000 10101001.
Using stop bit definition we define amount of bytes which encode the FAST message PMap by the following steps:
- Gets bytes one by one.
The first is 0xC0 and has 1100 0000 bits representation. - Check the stop bit value of current byte. If it is equals to “1”, our PMap completed by this byte.
In our case the 0xC0 has first bit equals to “1”, so our PMap is 1100 0000.
In case of current byte does not have first bit equals to “1” we increment length of our PMap by 1 and repeat our process from the step 1 using next byte.
3.2. Determine Message Template ID
The first bit (after stop bit) of Presence Map in any message shows presence of templateID which is required to define template for our message. Template ID uses an implicit copy operator. So if this first bit is 1, then the Template ID is present in the stream, and if the bit is 0, then the Template ID is the same as the Template ID of the previous message. In any case, we must always determine templateID as it is required for decoding the FAST message.
In our example, template ID is present in the stream, as illustrated below:
Picture 3.6 – “Presence of TemplateID PMap bit”
The same FAST message template is shared between the FAST encoder and decoder (typically the message producer and consumer). This relationship is depicted in the following diagram.
Picture 3.7 – “Message template relationship”
3.2.1. Template
In general, FAST message templates are stored in a single XML file and define how to encode and decode message content. These templates must be identical for both the encoder and decoder. The syntax of standard XML” should be “Standard XML”
DocumentBuilder docBuilder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
File xmlFile = new File(templateFileName);
Document doc = null;
try {
doc = docBuilder.parse(xmlFile);
} catch (Exception ex) {
//handle exception ...
}
The template xml file contains a set of FAST message templates that describe the format of encoded messages.
3.2.2. Message Template
To illustrate message template processing, we’ll use a template for a FIX Market Data Incremental Update message, FIX message type X.
In the sample java snippet below, we are importing the FAST message templates from an xml into a java HashMap for easy access in the rest of our application.
Map<integer, messagetemplate=""> messageTemplates = new HashMap<integer, messagetemplate="">();
public void readTemplates() {
int templateId;
for (int i =0; i < msgTemplatesCount; i++) {
templateId = parseTemplateId();
messageTemplates.put(templateId, new MessageTemplate(parseTemplateFields()));
}
}
</integer,></integer,>
The second decoding step is to identify the template ID. The following code snipping receives one FAST message, extract this message PMap, decodes the template ID, obtains the message template, and decode the message content using the template:
int templateId;
Message outputMessage;
int offset;
public Object decodeMessage(byte[] fastMessage) {
offset = 0;
// Get a presence map of the whole message
presenceMap = new PresenceMap();
offset = presenceMap.init(data, offset);
if (presenceMap.hasNextPrecenceBit()) {
templateId = dataTypeDecoder.processUnInt32Decoding(fastMessage, offset);
}
messageTemplate = templateParser.getTemplate(templateId);
outputMessage = new Message();
messageTemplate.decode(fastMessage, offset, presenceMap, outputMessage);
...
}
In FAST, template ID is encoded as an unsigned integer, so we decode it with the rules for this data type and get the matching template. We then use that template and the rest of the encoded data and, following the definitions within the template, generate the original message.
Let’s decode templateID for FAST message 0xC0 0xA9 from section 3.1.
- As we defined above our PMap looks like: 1100 0000. First bit is StopBit and the second bit says us that we have next FAST entry in the message.
As per message template definition it is templateID. So we take the second byte and decode it as templateID. - Second byte is 0xA9 and its bit representation: 1010 1001.
As this field type is “Unsigned Integer” and according to the this Data type rules, we can get its value as just removing the first bit: 0101001. - We convert these bits into decimal format and receive 41. So our templateID is 41.
3.3. Decode each message field
FAST message templates consist of field objects. Each of these entities should be coded separately.
3.3.1. Terminology
To introduce information in the next sections we will consider the following concepts which represent message template structure:
- Template. Contains one or more template units.
- Template unit. Is either a template field or a sequence.
- Template field. Defines a data type, presence attribute and an operator.
- Sequence. Is a count followed by repetitions of a field group..
- Field group. Is a number of template units that may have its own PMap.
3.3.2. Template Unit
According to the FAST specification, the template defines the following internal entities:
- Template Field
- Template Sequence
Each is represented by TemplateUnit in the following java interface:
public interface TemplateUnit {
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap,
OutputMessage message);
public void reset();
}
The purpose of the reset() method is to return Template Unit object to the initial (pre-decoding) state.
The Message Template itself consists of set of fields is also represented by TemplateUnit.
public class MessageTemplate implements TemplateUnit {
private TemplateUnit[] templateFields;
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap,
OutputMessage message) {
for (TemplateUnit templateField : templateFields) {
offset = templateField.decode(buffer, offset, presenceMap, message);
}
return 0;
}
public void reset() {
for (TemplateUnit templateField : templateFields) {
templateField.reset();
}
}
}
Before we continue, there are several FAST message template concepts to introduce:
3.3.2.1. Mandatory and Optional Fields
In a FAST message template, template units (that is, fields or sequences) can have an optional attribute. If set, the field or sequence is optional, otherwise mandatory.
This attribute has the following implications:
- If a template unit is mandatory, it must be present in the resulting decoded message (output stream).
- If a field is optional, it is possible for the field to not appear in the final decoded message (a special encoded NULL representation translates to no-value in the decoded message).
Mandatory fields are required in the decoded message. The following code example shows how you can test this constraint as you iterate over each template field.
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
Object decodedValue = getFieldValue();
if (decodedValue == null) {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
} else {
// do nothing
}
...
}
...
3.3.2.2. Previous, Initial, Base Values
Some template instructions (or “operators”) rely on the previous value. This is considered an internal property of the operator.
For fields using operators that rely on a previous value, this value is used to derive the decoded value of the current message.
The previous value can be in one of three states: undefined, empty and assigned.
- All values are in the state undefined when processing starts.
- The assigned state indicates that the previous value is present and has some value.
- The empty state is the assigned state but it indicates that previous value is absent. In such case previous values has “null” representation of value.
At some defined moments the state of previous value of template fields need to be reset to the undefined state. These moments are defined by the protocol which is used to transfer data:
- Protocols with packet boundaries (UPD, TCP stream with separator) considers to automatically reset state
- For protocols without packet boundaries, the reset normally occurs at start of connection only.
However, It is important that reset must be preformed in the same order in the encoder and decoder with respect to the order of the content of the stream.
The following example shows reset action performing after each FAST message receiving
public Object decodeMessage(byte[] fastMessage) {
...
messageTemplate.decode(fastMessage, offset, presenceMap, outputMessage);
messageTemplate.reset();
return outputMessage;
}
Resetting state means that all fields are set to their initial pre-processing state (undefined state).
The undefined state of fields implies non-existent of previous values. In such case on the first iteration of decoding process is used initial value instead of previous value. (An initial value is specified in the Template by the value attribute on the operator element.)
If the initial value is not defined by the template (if it has the “null” value representation), the base value can be used instead of initial value. Each field defined its own appropriate base value according to data type property.
3.3.3. Template Field
Template Field defines the following main decoding actions:
- Defining if the field is presents in the stream (input FAST mesage)
- Creating field values by extracted bytes which were decoded using stop bit definition
- Defining calculations against decoded field values to achieve the final output field value.
When we implement this concept in code, Template Field has a set of TemplateUnit methods that decode according to its FieldOperator property. And Field Operators are described next.
3.3.4. Field Operators
A field operator can be assigned to a template field. One simple example is the “copy” operator, where you want to use the previous value if no value is in this message to override. This is common for dates, which are often repeated.
But there are a number of operators available that define how fields are encoded and decoded. These are all listed below.
3.3.4.1. Field Presence Map Bit Requirement
In except of Presence Map there is another parameter which can define the presence field value in the stream. Here is it description:
In some cases Presence Map may not contains appropriate bits for some Template fields. In relation of field properties (Operator type and Mandatory) we define presence of PMap bit which is associated to this field. The field does not require the presence bit in the PMap if this field value (not depending whether the value is null or not null) must always be presenent in the input encoded message. In such cases PMap bit for this field is always absent in the PMap and we always decode field value from the input message. In the other cases we decode field value from the input message on relation of PMap bit value (‘1’ or ‘0’).
The following Table 3.1 shows the presence of PMap bit depending on Operator type and Mandatory property of the field.
Table 3.1
| Operation | Mandatory | Optional |
| NoOperator | No | No |
| Constant | No | Yes |
| Copy | Yes | Yes |
| Default | Yes | Yes |
| Delta | No | No |
| Increment | Yes | Yes |
| Tail | Yes | Yes |
Each template field entity object should implement the following method for defining requirement of presence bit value in the PMap:
public boolean isPMapBitExist();
Here is an example of using this method:
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
if (isPMapBitExist()) {
boolean presenceBit = presenceMap.hasNextPresenceBit();
...
As we can see from the Table 2.1 we should use this method only in the case if field operator is Constant. In other cases we don’t need to check the requirement of presence bit value. This functionality should be implemented directly.
3.3.4.2. Nullable Fields
As described earlier, template fields have an “optional” attribute, meaning the field is optional in the output message. In order to indicate that a field is not to exist in the output message, a special NULL value is put into the message.
A field is nullable in certain cases and can affect the normal decoding of encoded fields. For example, optional integer fields are offset by 1 if positive and to make room for the reserved value 0 to indicate NULL. This is not the case for mandatory integer fields.
The following Table 3.2 shows the rules of determining field nullability.
Table 3.2
| Operation | Mandatory | Optional |
| NoOperator | No | Yes |
| Constant | No | No |
| Copy | No | Yes |
| Default | No | Yes |
| Delta | No | Yes |
| Increment | No | Yes |
| Tail | No | Yes |
In your code, you will want each template field object to implement a method for determining field nullability.
public boolean isFieldNullable();
In our FAST library, we have two decoders which perform field value extracts from the stream:
- NullableDataTypeDecoder for nullable fields (can return null as field value).
- NonNullableDataTypeDecoder for non-nullable fields (cannot return null as field value).
Before assigning a decoder for the template field object, we check field nullability to choose the right one:
dataTypeDecoder = isFieldNullable() ? NullableDataTypeDecoder.getDefaultInstance()
: NonNullableDataTypeDecoder.getDefaultInstance();
3.3.4.3. NoOperator
The fields with no defined operator perform only the encoded value extract and perform no other operation on the extracted value.
The following diagram illustrates this decoding process:
Picture 3.8 – “NoOperator fields decoding algorithm”
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
Object decodedValue = getFieldValue();
if (decodedValue != null) {
message.addValue(tagId, decodedValue);
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
}
}
return offset;
}
public void reset() {
// Since NoOp fields do not use a dictionary entry there is nothing to reset.
}
public boolean isPMapBitExist() {
// Without operator fields not require presence map bit. In any case it should be in encoded message
return false;
}
public boolean isFieldNullable() {
// If isMandatory false the NoOp field could has the 'null' representation of field value.
// In such case it could not apear in the output FIX message
return !isMandatory;
}
3.3.4.4. Constant Operator
The constant operator specifies that the field value will always be given value.
This value is never transferred in the message itself, but is suppressed by the encoder and replaced by the decoder.
An example of this might be 8=FIX.4.4, where you would never sent FIX.4.4 over the wire.
If optional, the field will use a PMap bit. If not set, the field will not be present in the output message.
The decoding algorithm of template fields using the Constant operator:
Picture 3.9 – “Decoding algorithm of fields with Constant Operator”
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
if (isMandatory || (isPMapBitExist() && presenceMap.hasNextPresenceBit())) {
message.addValue(tagId, initialValue);
}
return offset;
}
public void reset() {
// Constant field never use the previous value
}
public boolean isPMapBitExist() {
// If isMandatory true the Constant field do not require a presence map bit.
// Otherwise it is require the presence map bit.
return !isMandatory;
}
public boolean isFieldNullable() {
// Constant field is not nullable
return false;
}
3.3.4.5. Copy Operator
This operator instructs the decoder to use the previous value if the value is not present in the encoded message (PMap bit is not set). Here's the decoding algorithm of these fields:
Picture 3.10 – “Decoding algorithm of fields with Copy Operator”
Code example of this algorithm:
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
if (presenceMap.hasNextPresenceBit()) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
Object decodedValue = getFieldValue();
if (decodedValue != null) {
message.addValue(tagId, decodedValue);
setPreviousValue(decodedValue);
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
} else {
setPreviousValue(null);
}
}
} else {
// there is no undefinite status for the previous value
Object prevValue = getPreviousValue();
if (prevValue != null) {
// add field into the message
message.addValue(tagId, prevValue);
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
}
}
}
return offset;
}
public void reset() {
// the field delivers previous value from the undefinite status
setPreviousValue(getInitialValue());
}
public boolean isPMapBitExist() {
// Copy fields require presence map bit
return true;
}
public boolean isFieldNullable() {
// If isMandatory false the Copy field could has the null representation of field value and does not apear in the output FIX message
return !isMandatory;
}
3.3.4.6. Default Operator
The default value associated with this operator is added to the output message unless the value is included in the encoded message.
This diagram shows the decoding algorithm for fields defined with the Default Operator:
Decoding algorithm of fields with Default Operator | FIX Fast Tutorial | JetTek Fix
Picture 3.11 – “Decoding algorithm of fields with Default Operator”
An implementation of this algorithm:
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
Object decodedValue = null;
if (presenceMap.hasNextPresenceBit()) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
decodedValue = getFieldValue();
} else {
decodedValue = getInitialValue();
}
if (decodedValue != null) {
message.addValue(tagId, decodedValue);
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
}
return offset;
}
public void reset() {
// Default fields never change their initial value so there is no need to reset them
}
public boolean isPMapBitExist() {
// Default fields require presence map bit
return true;
}
public boolean isFieldNullable() {
// If isMandatory false the Default field could has the null representation of field value and does not apear in the output fix message
return !isMandatory;
}
3.3.4.7. Delta Operator
The delta operator combines the incoming value in the encoded message with the previous value. So if the last decoded value came to 150 (for an integer field) and the incoming value within the encoded message is -5, then the output value is 145.
The diagram below illustrates this operator’s algorithm.
Picture 3.12 – “Decoding algorithm of fields with Delta Operator”
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
if (getFieldValue() != null) {
if (getPreviousValue() != null) {
calculateDeltaFieldValue();
Object fieldValue = getFieldValue();
message.addValue(tagId, fieldValue);
setPreviousValue(fieldValue);
} else {
throw new RuntimeException("Mandatory decoded field can not be NULL");
}
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
}
}
return offset;
}
public void reset() {
// This functionality delivers previous value from the undefinite status
Object prevValue = getInitialValue();
if (prevValue == null) {
prevValue = getFieldBaseValue();
}
setPreviousValue(prevValue);
}
public boolean isPMapBitExist() {
// Delta fields not require presence map bit. In any casy it should be in encoded message
return false;
}
public boolean isFieldNullable() {
// If isMandatory false the Delta field could has the null representation of field value and does not apear in the output fix message
return !isMandatory;
}
2.3.4.8. Increment Operator
If the field value is present in the encoded message (its PMap bit is set) the decoded value is used as the output value. Otherwise, the increment operator uses the previous value and adds one to it to yield the output value. That output value updates the previous value for next time.
This operator works with integers only.
The below diagram illustrates the Increment operator decoding algorithm:
Picture 3.13 – “Decoding algorithm of fields with Increment Operator”
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
if (presenceMap.hasNextPresenceBit()) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
Object decodedValue = getFieldValue();
if (decodedValue != null) {
message.addValue(tagId, decodedValue);
setPreviousValue(decodedValue);
isUndefinedPrevValue = false;
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
} else {
setPreviousValue(null);
}
}
} else {
if (getPreviousValue() != null) {
if (!isUndefinedPrevValue) {
incrementPrevFieldValue();
}
// add field into the message
message.addValue(tagId, getPreviousValue());
} else {
if (isMandatory) {
throw new RuntimeException("Mandatory field decoded value can not be NULL");
}
}
}
return offset;
}
public void reset() {
// The field delivers previous value from the undefinite status
setPreviousValue(getInitialValue());
isUndefinedPrevValue = true;
}
public boolean isPMapBitExist() {
// Increment fields require presence map bit
return true;
}
public boolean isFieldNullable() {
// If isMandatory false the Increment field could has the null representation of field value and does not apear in the output fix message
return !isMandatory;
}
3.3.4.9. Tail Operator
This operator specifies the number of chars to remove and chars to append to the previous value. That is, the output value is obtained by combining the previous value – ‘n’ chars (where ‘n’ is an int from the input message) + decoded field value from input encoded message (the tail value).
For this operator, the tail value is optional.
This operator can be applied to data type string and byte vector, both defined later.
This diagram shows the tail operator’s decoding algorithm:
Picture 3.14 – “Decoding algorithm of fields with Tail Operator”
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
if (presenceMap.hasNextPresenceBit()) {
offset = dataTypeDecoder.decode(encodedData, offset, templateField);
Object decodedValue = getFieldValue();
if (decodedValue != null) {
if (getPreviousValue() == null) {
Object prevValue = getInitialValue();
if (prevValue == null) {
prevValue = getFieldBaseValue();
}
setPreviousValue(prevValue);
}
calculateCombineValue();
decodedValue = getFieldValue();
message.addValue(tagId, decodedValue);
setPreviousValue(decodedValue);
} else {
if (isMandatory) {
logger.logError("Mandatory field decoded value can not be NULL");
} else {
setPreviousValue(null);
}
}
} else {
Object prevValue = getPreviousValue();
if (prevValue != null) {
message.addValue(tagId, prevValue);
} else {
if (isMandatory) {
logger.logError("Mandatory field decoded value can not be NULL");
}
}
}
return offset;
}
public void reset() {
// The field delivers previous value from the undefinite status
setPreviousValue(getInitialValue());
}
public boolean isPMapBitExist() {
// ~~ Tail fields require presence map bit
return true;
}
public boolean isFieldNullable() {
// If isMandatory false the Delta field could has the null representation of field value and does not apear in the output fix message
return !isMandatory;
}
3.3.5. Sequence
As described earlier, a FAST message templates are composed of template units. Template units are either template fields or sequences.
A sequence is a set of fields and is comprised of:
- A sequence length that represents the number if instances of this sequence follow.
- Field group repeating “sequence length” times, each beginning with its own PMap.
The decoding algorithm for a sequence follows:
Picture 3.15 – “Sequence decoding algorithm”
Picture 3.16 – “Field decoding algorithm”
The following example shows an example implementation of the TemplateUnit interface for “sequence”:
private TemplateField sequenceLength;
private TemplateUnit[] templateFields;
public int decode(byte[] encodedData, int offset, PresenceMap presenceMap, OutputMessage message) {
offset = sequenceLength.decode(buffer, offset, presenceMap, message);
if (sequenceLength.getFieldValue() != null) {
int groupsCount = sequenceLength.getFieldValue();
if (groupsCount > 0) {
message.createFieldsSequence(sequenceLength.getTagId(), groupsCount);
for (int iter = 0; iter < groupsCount; iter++) {
presenceMap = new PresenceMap();
if (isPMapRequired) {
presenceMap.init(buffer, offset);
offset = presenceMap.getLastPMapBytePosition() + 1;
}
for (int i = 1; i < templateFields.length; i++) {
offset = templateFields[i].decode(buffer, offset, presenceMap, message);
}
}
}
} else {
if (isMandatory) {
throw new RuntimeException("A mandatory sequence is missing from the stream.");
}
}
return offset;
}
public void reset() {
for (int i = 1; i < templateFields.length; i++) {
templateFields[i].reset();
}
}
Note:
In the above code, boolean variable isPMapRequired is true if any template field in the sequence requires a PMap bit. According to FAST, if no PMap bits are required (i.e., all fields are will always be present in the encoded message or are mandatory constants), then no PMap will precede the field groups so an empty PMap will be used.
The method isPMapRequired() can be implemented with the following code snippet:
isPMapRequired = false;
for (TemplateField field : templateFields) {
if (field.isPMapBitExist()) {
isPMapRequired = true;
break;
}
}
Sequence decoding example:
Let’s consider a small message that has the following template
Our input FAST message: 0xC0 0xA3 0x81 0xA0 0x80 = 11000000 10100011 10000001 10100000 10000000
The following table shows complete decoding of this message:
Picture 3.17 – “Decoding of message which contains sequence”
3.3.6. Data Types
FAST defines several data types for encoding message fields. The template declares a data type for each template field. The choice of data type controls the binary format of the encoded field.
For the decoder, data type controls the following:
- Extracting field value from the encoded message.
- Stop bit rules.
- Null value representation and value adjustments to non-null values of optional fields.
In our code, we combine these steps and consider it one process. The available data types are:
- Unsigned Integer
- Signed Integer
- Decimal
- ASCII String
- Byte Vector and Unicode String
The next sections detail the above data types and describe how the decoder will extract each type, the stop bit rules, and null value representation.
3.3.6.1. Integer
Integers are represented as stop bit encoded entities.
The stop bit decoding process of integer fields is:
- Determine the length by the stop bit algorithm.
- Remove stop bit from each byte.
- Combine 7-bit words (without stop bits) to determine actual integer.
In addition, a nullable field continues with this processing:
- If the value is zero (0), the output value is NULL.
- If the value is positive, subtract 1 from that value to use as the output value.
2.3.6.1.1. Unsigned Integer
Represents unsigned integers using the FAST 7-bit binary encoding.
The example of this decoding process is shown below in Picture 3.18
Picture 3.18 – “Unsigned Integer stop bit decoding”
public int processUInt32Decoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
// The primari value is only 0 as this is unsigned
long value = 0l;
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value = (value << 7) | (buffer[offset++]);
}
value = (value << 7) | (buffer[offset++] & Byte.MAX_VALUE);
fieldValue.setFieldValue(value);
return offset;
}
Decoding code sample of nullable fields:
public int processUInt32Decoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
// The primari value is only 0 as this is unsigned
long value = 0l;
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value = (value << 7) | (buffer[offset++]);
}
value = (value << 7) | (buffer[offset++] & Byte.MAX_VALUE);
if (value == 0) {
fieldValue.setFieldValue(null);
} else {
fieldValue.setFieldValue(value - 1);
}
return offset;
}
2.3.6.1.2. Signed Integer
Used to represent a signed (+/-) integer using the FAST 7-bit binary encoding.
The following diagram illustrates signed integer decoding of a negative number.
Picture 3.19 – “Signed integer stop bit decoding”
public int processInt32Decoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
int value = 0;
if (PresenceMap.hasBitOnPosition(buffer[offset], 1)) {
value = 0xFFFFFFFF;
}
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value = (value << 7) | (buffer[offset++]);
}
value = (value << 7) | (buffer[offset++] & Byte.MAX_VALUE);
fieldValue.setFieldValue(value);
return offset;
}
Decoding code sample of nullable fields:
public int processInt32Decoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
int value = 0;
if (PresenceMap.hasBitOnPosition(buffer[offset], 1)) {
value = 0xFFFFFFFF;
}
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value = (value << 7) | (buffer[offset++]);
}
value = (value << 7) | (buffer[offset++] & Byte.MAX_VALUE);
if (value == 0) {
fieldValue.setFieldValue(null);
} else {
if (value > 0) {
value -= 1;
}
fieldValue.setFieldValue(value);
}
return offset;
}
3.3.6.2. Decimal
In FAST, Decimal numbers are represented by two signed integers, an exponent and mantissa. This is similar to, but not the same as the IEEE standard. FAST variable length encoding differs from IEEE, which has a fixed size encoding of a set number of bytes and other special values, such as infinity.
The numerical value of a decimal data type is obtained by extracting the encoded signed integer exponent followed by the signed integer mantissa values from the encoded message and multiplying the mantissa by base 10 power of the exponent.
output value=mantissa * 10 exponent
EXAMPLE:
Take, for example, the encoded decimal value: 0xFE 0x9 0xD2 = 11111110 00001001 11010010.
Picture 3.20 shows the decoding process of this decimal field.
Picture 3.20 – “Decimal value decoding”
decimalValue=mantissa * 10 exponent = 1234 * 10 -2 = 12.34
Notes:
- The presence attribute of exponent equals to presence attribute of decimal field. Mantissa must always have mandatory presence. Thus the exponent can be either nullable or non-nullable and the mantissa is always non-nullable.
- The mantissa is present in the stream if the exponent is not NULL.
- If the template field is optional the field of this data type can have empty value (NULL) and will not be added into the output message. This is represented by a NULL in the exponent (in which case, no mantissa field is encoded or decoded).
Here is the code example:
public int decode(byte[] buffer, int offset, PresenceMap presenceMap, OutputMessage message) {
offset = exponentField.decode(buffer, offset, presenceMap, message);
if (exponentField.getValue() != null) {
offset = mantissaField.decode(buffer, offset, presenceMap, message);
Object decimal = calculateDecimalValue(exponentField.getValue(),
mantissaField.getValue());
message.addValue(tagId, decimal);
}
return offset;
}
Lets consider more detailed another example of decimal field decoding:
Our FAST message template:
Note: The presence attribute of our decimal field is specified by our template and it is mandatory. According to the decimal field definition the presence attribute of exponent is the same as presence attribute of full decimal field and is mandatory too. Presence attribute of mantissa is always mandatory.
Input encoded message:
HEX format: 0xE0 0x81 0x03 0x3B 0xD5
Binary format: 11100000 10000001 00000011 00111011 11010101
Decoding process of this message is shown below:
Picture 3.21 – “Decoding of message which contains decimal field”
3.3.6.3. ASCII String
An ASCII String is represented as a stop bit encoded entity. The entity value is interpreted as a sequence of 7-bit ASCII characters.
The stop bit decoding of these fields is executed as follows:
- Determine the byte count of the String value using the stop bit algorithm.
- In the final byte, change the stop bit value from “1” to “0”.
- The resultant byte array is the output string value.
An example of this decoding process is shown below.
Picture 3.22 – “String value stop bit decoding”
public int processStringDecoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
// Handle empty string
if(buffer[offset] == (byte)0x80) {
fieldValue.setFieldValue("");
return (offset+1);
}
// Extract the bytes from the buffer until a stop bit is encountered
StringBuffer value = new StringBuffer();
while(!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value.append((char)(buffer[offset++]));
}
// Extract the last byte and get rid of the stop bit
value.append((char)((buffer[offset++] & Byte.MAX_VALUE)));
fieldValue.setFieldValue(value);
return offset;
}
Note:
- Field value represented by byte equals to 0x80 implies the empty String – “” of decoded String value.
Decoding code sample for nullable string fields:
public int processStringDecoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
// Handle null value
if(buffer[offset] == (byte)0x80) {
fieldValue.setFieldValue(null);
return (offset+1);
}
// Handle empty string value
if(buffer[offset] == 0x00 && buffer[offset+1] == (byte)0x80) {
fieldValue.setFieldValue("");
return (offset+2);
}
// Extract the bytes from the buffer until a stop bit is encountered
StringBuffer value = new StringBuffer();
while(!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
value.append((char)(buffer[offset++]));
}
// Extract the last byte and get rid of the stop bit
value.append((char)((buffer[offset++] & Byte.MAX_VALUE)));
fieldValue.setFieldValue(value);
return (offset);
}
Note:
- Field value represented by 0x80 byte implies the NULL decoded String value (not “” as above).
- Field value represented by 0x00 0x80 bytes implies the empty String – “” of decoded String value.
3.3.6.4. Byte Vector and Unicode String
Byte Vector and Unicode String fields are represented as an Unsigned Integer size preamble followed by the specified number of raw bytes.
Each byte in the data part has eight significant data bits. As a consequence, the data part is not stop bit encoded.
A nullable field has a nullable size preamble. A NULL size preample represents a NULL value of this field. s
The following diagram illustrates a unicode-8 string decoding example.
Picture 3.23 – “Byte Vector and Unicode String fields stop bit decoding”
The following sample code shows the decoding process for byte vectors and unicode string for non-nullable fields:
public int processByteVectorDecoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
int valueByteCount = 0;
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
valueByteCount = (valueByteCount << 7) | (buffer[offset++]);
}
valueByteCount = (valueByteCount << 7) | (buffer[offset++] & Byte.MAX_VALUE);
String strValue = null;
if (valueByteCount == 0) {
strValue = "";
}
if (valueByteCount > 0) {
strValue = new String(buffer, offset, valueByteCount);
}
fieldValue.setFieldValue(strValue);
return offset;
}
The next code sample shows the decoding process for the same type (byte vectors and unicode string) but for nullable fields:
public int processByteVectorDecoding(byte[] buffer, int offset, TemplateFieldValue fieldValue) {
int valueByteCount = 0;
while (!PresenceMap.hasBitOnPosition(buffer[offset], STOP_BIT_POSTION)) {
valueByteCount = (valueByteCount << 7) | (buffer[offset++]);
}
valueByteCount = (valueByteCount << 7) | (buffer[offset++] & Byte.MAX_VALUE);
String strValue = null;
valueByteCount--;
if (valueByteCount == 0) {
strValue = "";
}
if (valueByteCount > 0) {
strValue = new String(buffer, offset, valueByteCount);
}
fieldValue.setFieldValue(strValue);
return offset;
}