As we browse over some of the concepts behind this meta-model we will bring an overall description of the fundamentals behind the QuantLET framework.

Models

Models represent a very specific problem in terms of dependent and independent parameters.

An applied example of an (unrealistic) model would be something like:  given LIBOR, benchmark interest and rates in US, US/EUR spread and crude oil spot price, what is the limit price of a long position and how many pips for a trailing stop in a conditional order with a highest chance of maximized profit?

A typical problem of stochastic optimization.

On this example, independent variables are given by:

• LIBOR
• Benchmark US rates
• US/EUR spread in the spot rate
• Crude oil spot price

And dependent variables:

• Limit price of the long position
• Rate maximum move in the stop loss leg
• Exit price of the long position

Models might have one or more Execution Modes associated to it.

Execution Mode

An execution mode defines source and destination of financial data, represented by a high-frequency stream of events. For the example above, we could think about a few execution modes: simulation, back-testing, paper-trading and production.

Simulation

In simulation mode, our model reads a stream of events from an endpoint simulating a synthetic Geometric Brownian Motion with pre-defined parameters for drift and volatility.

Back-Testing

In back testing, our model reads a stream of events from historical parameters for each of the independent variables.

Paper Trading

In paper trading, our model posts orders to a simulated exchange environment, based on inbound data from real-time live data.

Production

Reads inbound data from real-time live inbound channels, posts orders to real exchange channels

Shocks

Shocks are re-configuration of execution modes with different parameters. For obvious reasons, not all execution modes allow the association of shocks. On the example before, we could think about a few shocks:

• Simulation mode, with drifts in the range of (-0.30 to 0.30) in increments of 0.05
• Back-testing mode, with LIBOR rates shocked up (or down) in the range  of (-2.0 to +2.00) in increments of 0.1

Benchmark

A benchmark allows the comparison of results from various shocks, looking for the lowest cost of the optimization function, or any other dimension you could think of. Some examples.

• Highest single profit in a day
• Highest accumulated profit in a period (week, month)
• Highest number of profitable periods (days, weeks)
• Highest value at risk exposure in a period

You can have multiple benchmarks active at any given time, and historical benchmarks are available for comparison.

The Remaining Framework

QuantLET is a framework – and as such it provides many battle tested, time saving components with a number of rich features. A few other pieces of the framework:

• Tapestry: A powerful framework for data and CPU intensive based on map-reduce abstractions and data partitioning
• Dashboard: A UNO-based front-end for OpenOffice family of spreadsheets.
• Mediation: Configuration of models, modes, shocks and benchmarks at run-time

Among many other features – check QuantLET home often for more and updates.

Let’s illustrate with an example: cumulative moving averages of real-time price points .

In PyQuantLET we define a simple model to read ticks from an inbound endpoint, generate a cumulative moving average, and forward the result to an outbound endpoint in two lines:

f = quantlet.handler.stat.cma()
 [outbound((x, f(x))) for x in inbound]

Concrete inbound and outbound endpoints are defined as part of the execution mode of this model. A mode of execution could be a simulation in which the inbound endpoint is a real-time feed for trades of a specific underlying, and an outbound plot of price points and cumulative moving average.

execute_mode(id='simulation',
     model=cma(inbound=feed(''tcp://localhost:61616?wireFormat=openwire'),
               outbound=graph(['plot']
              )
     )

Of course, nothing realistic, but that’s all it takes.  Ok, let’s now define the function for cumulative moving average (in reality we would use classes for the same purpose, but let’s stick to this for now):

def cma(): // this is wrong
     total, counter = (0,0)
     def result(value):
            total = total + value
            counter = counter + 1
            return total / counter
     return result

If you try this version, you will get something like this:

UnboundLocalError: local variable 'total' referenced before assignment

The message is self explanatory – seems like the reason is that rebinding names in a closed scope is not allowed in Python. Nothing like a dictatorship for life…

So, now the good part – the solution – rewrite your function so variables in the outer scope are associated to a container object, see…

def cma():
    total, counter =([0],[0])
    def result(value):
        total[0] = total[0] + value
        counter[0] = counter[0] + 1
        return total[0] / counter[0]
return result

Looks weird, but works – you can try yourself

As a reminder, RMS is a collection of patterns that are ideal for event-driven, low-latency applications, especially useful in high-frequency finance. Events in RMS are described in terms of a header and a body. The header is usually a dictionary, and the body must support the representation of elaborate graphs of objects, representing real world analytical use cases. QuantLET is an open source RMS framework.

Protocol Buffers

“Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data”. Comparisons with other alternatives like XML or JSON come to mind immediately, but just let’s say for now that Protocol Buffer translates to a smaller binary wire format, and therefore transfers are usually done in a fraction of a time of alternatives like XML or JSON.

Benefits?

A few features of Protocol Buffers are a great match to represent RMS events and high frequency applications

Fast

The transport format is based in a binary format, compact, and therefore really fast for transfers and serialization. It is ideal for high frequency solutions where the expected latency is less than a millisecond.

Backward Compatible

We can see that backward compatibility is a notion that is present from the grounds up. The notion of optional attributes and unique identification of fields through a number allow messages to be consumed by older clients as you roll out new versions of your information model.

Drawbacks?

A few features of Protocol Buffers that speak against its use as a transport protocol in RMS architectures.

Not Plain Objects

In RMS lingo we define ‘plain object’ as a structure capable of describing cyclical graphs of objects, in which objects are represented through an interface allowing access to its members either directly or through getters and setters.

With that definition in mind, Protocol Buffer objects cannot be seen as RMS plain objects. Protocol Buffers objects carry programmatic features – like fluent builders, and binding to language specific structures (like streams in Java or C++).

While bringing benefits, these same features make objects dependent on the framework when Protocol Buffers are not really necessary (in RMS a transport format is only necessary on endpoints, usually during marshalling/unmarshalling and transformation).

Lacks Inheritance Support

Ok, I know – Protocol Buffers allows the representation of a quasi inheritance in a few different ways, depending on how you intend on using it.

When a consumer knows in advance the expected subtype of a message, you can simply embed an instance of your base type on the derived type, you can embed subtypes as optional aggregated messages:

message Animal { // General animal properties.

optional double weight = 1;
extensions 1000 to max;
}

message Dog { // Dog-specific properties.
optional float average_bark_frequency = 1; }

message Cat { // Cat-specific properties.
enum Breed { TABBY = 1; CALICO = 2; ... }
optional Breed breed = 1;
}

extend Animal {
optional Dog dog = 1000;
optional Cat cat = 1001;
}

Ok, it might work. The problem with that is that you will need to know all the possible derived types in advance, for anything that is a bit more than really trivial use cases this is an impediment. Here is the alternative, for when a  consumer does not know in advance what types to expect, you have to play with a mix of features.

message Animal reserved 10 {
optional double weight = 1;
optional Color color = 2;
}

message Dog extends Animal reserved 10 {
optional float average_bark_frequence = 1;
}

message Cat extends Animal reserved 10 {
optional Breed breed = 1;
}

message Lion extends Cat reserved 10 {
optional boolean isAlphaMale = 1;
}

Now the dirty details: the description above is made in a ‘protoext’ file, and in order to give you something useful it has to be compiled to produce a proto file like this:

//@Hierarchi Lion:Cat:Animal
message Lion {
//-- Animal's indices start at 1 because it is the base class (Lifeform does not count since it is an interface)
//@Class Animal
optional double weight = 1;
//@Class Animal
optional Color color = 2;

//-- Cat's indices starts at 11 because reserved is set to 10: 10+1
//@Class Cat
optional Breed breed = 11;

//-- Lion's indices starts at 21 because reserved is set to 10: 10+10+1
//@Class Lion
optional boolean isAlphaMale = 21;
}

//@Hierarchi Dog:Animal
message Dog {
//-- See comments for Lion above
//@Class Animal
optional double weight = 1;
//@Class Animal
optional Color color = 2;

//-- See comments for Cat above
//@Class Dog
optional float aveage_bark_frequence = 11;
}

So you have already compiled it once, and this thing here will have to be compiled again, to give you something you can finally use. You got the picture: yes, it is complicated.

Alternatives

So are there any alternatives to Protocol Buffers out there? Yes there are a few, each bringing their own benefits and drawbacks:

Specialized Types

You can use a set of collection type of your language of choice to represent graphs through keys and values. Some examples are maps of maps in Java, or dictionary sets in Python. Expensive and loosely typed, I just can’t think of any good reason to use it, you should avoid it if you can.

Native Serialization

This is language dependent, and brings its obvious and critical drawbacks. Java’s serialization, Python pickles, C++ streams, and so forth. Of course those formats are hardly interchangeable. In some cases (Java) the format is so large and cumbersome that makes it a hard fit for any serious low-latency application.

XML

Large, slow, over engineered, over complicated, human readable (why if it is intended to be used by computers?). I still did not figure out why people seem to like it and its use is so overspread. I can only see justification for XML use away from low-latency solutions, representing maybe complex and text-driven structures like documents.

JSON

JSON (JavaScript Object Notation) is a lightweight data-interchange format, based on a subset of JavaScript. Again, human readable (can someone explain me why this is a good thing?) and supported by virtually any major platform out there. Faster and a bit smarter than XML, JSON is still too slow for low-latency applications.

SLICE

SLICE (Specification Language for ICE) is the abstraction mechanism for separating ICE object interfaces from their implementations, on various languages. Really fast and lightweight, the problem is that it forces in the ICE paraphernalia. It is an option if you plan on using the whole stack.

So?

Overall, as long as you clearly understand its limitations and pitfalls, and plan a work around them, given the alternatives and pros and cons, Protocol Buffers is a reasonable option to define and transport events and the information model in RMS systems.

1. Make sure you install Parallels Tools
   You do that from Parallels menu: Virtual Machine -> Install Parallels Tools

2. Modify X configuration to account for MacBook Pro proper resolution modes

   jfaleiro@ubuntu:~$ sudo vi /etc/X11/xorg.conf

   enter your admin password, and then change the file, commenting old line and adding a new one with your desired configuration:

   #                Modes   "800x600"
                    Modes   "1280x800" "1152x820" "1024x768"

3. Re-apply configuration to all additional xserver configuration files

   jfaleiro@ubuntu:~$ sudo dpkg-reconfigure -phigh xserver-xorg

4. Modify the amount of video memory on the virtual machineAgain, the Parallels menu: Virtual Machine -> Configure -> Video On my case, 16Mb did the job
5. Finally, a fresh Restart

   jfaleiro@ubuntu:~$ sudo reboot

The highest resolution is fine for a full screen view. You can switch Ubuntu ( System -> Preferences -> Display ) to a lower resolution (1152×820 or 1024×768) if you intend on using a window view.

First of all, they both have to be upgraded to Mac OS X 10.6.1 – so, to the uninstall:

Fink:

%% sudo rm -rf /sw

and MacPort:

%% sudo rm -rf \
    /opt/local \
    /Applications/DarwinPorts \
    /Applications/MacPorts \
    /Library/LaunchDaemons/org.macports.* \
    /Library/Receipts/DarwinPorts*.pkg \
    /Library/Receipts/MacPorts*.pkg \
    /Library/StartupItems/DarwinPortsStartup \
    /Library/Tcl/darwinports1.0 \
    /Library/Tcl/macports1.0 \
    ~/.macports

Finally the installation: I decided to stay with MacPort, seems to provide more packages and most of the tools I usually look for are included – downloaded the 10.6.1 DMG for it and

%% sudo port -v selfupdate

Simple and straightforward, all works like a breeze.

Specialized Declarative

Mostly through a declarative language (CEP guys/gals love that) like EQL or dialects. In other words, a language specialized for event handling (DSL). But be advised – despite of its ‘coolness’ and being (most times at least) a time saver and a shortcut it requires specialized skills and tools not available on every corner (translation: lots of $$$).

Functional Declarative

Similar to above, one example would be something like a lisp-like dialect. Another example are constructs based on paradigms like map-reduction for large data fabric clusters. All as above applies

Second-Order Declarative

Based on inferences derived from second-order logic descriptions. You: “What? Why are you bringing this up here?” Me: Well, inference engines, specially RETE engines, are basically a correlated forwarded-chain of rules and facts, optimized in a tree-like fashion for speed in which “triggering” can be seen as an event’s action.

A time saver, flexible, and ideal when proper tools and resources are not available.

Finite State Machines

Actions are associated to changes in state represented by a directed graph. Conditional transitions are also supported by ‘guards’. Incoming events carry either a next state of the graph, or the indication of a transition. Different types of actions are then performed based on the transition and the association, i.e: actions performed when entering a state (entry), leaving a state (exit), or when changed from one specific state to another (transition).

Observer-Observable Imperative

Simplest, but can lead to a lot of code to write that *will* turn into spaghetti if done by the uninitiated. Base yourself on proven paradigms that enforce separation of control and state to avoid adding to your own pain.

Procedural Imperative

Read ‘anything-else-imperative-except-the-observer-pattern-mentioned-above’ – Don’t even think about trying that unless your model is *really* simple…

Ok — Now, pick one, any one…

% sudo update-alternatives --config java

Will show you all java alternatives, so you can select which one to run

Adding Alternatives of Binaries to run

Some binaries do not have alternative versions, or do not come configured as such – an example is Python

% sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.4 10
% sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.5 20

This case adds versions 2.4 and 2.5 of Python – then later you can select the version you want with

% sudo update-alternatives --config python

Searching for a file using a string

You will need apt-file:

% aptitude install apt-file

apt-file works on a local image of the repository image:

% sudo apt-file update

you can then search for a matching string

% apt-file search 'tomcat-user'

Searching for a package using wildcards

% apt-cache search '*tomcat*'

or

% dpkg -l '*tomcat*''

Starting/Stopping Services

Individual services can be started/stopped through scripts on /etc/init.d, e.g.
% sudo /etc/init.d/tomcat6 restart

garbage collection (GC) is a form of automatic memory management. It is a special case of resource management, in which the limited resource being managed is memory. The garbage collector, or just collector, attempts to reclaim garbage, or memory occupied by objects that are no longer in use by the program.

Yes, sounds magical – let’s look at two examples of when the magic seems pretty much gone: promotion failures and concurrent mode failures

Promotion Failure

No, this is not related to your last performance review. Nothing we can do about that. Sorry. This is related to a failure while promoting objects from the younger to older generation space in the heap. You can detect a promotion failure by entries like this on your GC trace:

nnnnnn.nnn: [GC nnnnnn.nnn: [ParNew (promotion failed): (...)

See, the way promotion of objects work, all younger objects to be promoted have to be allocated in a contiguous address space in the older generation memory.

Almost all cases of that (expensive) failure is due to fragmentation on the older generation area in the heap. This forces compaction to kick in right after that failure.

The most common way to deal with that is to reduce eden size and increase heap size (to a limit of course – bare in mind that compaction time does not increase linearly with the heap space to be compacted).

Concurrent Mode Failure

This is a result of an interruption of the usual (c0ncurrent) collection before completion. It is indicated in the GC trace by something similar to this:

nnnnnn.nnn: [GC ...: [ParNew: (...) ...: [CMS (concurrent mode failure):

Why is that exactly? The GC algorithm predicts that the concurrent collection will not end before the heap becomes full, so it decides to stop everything and run a full (stop the world) GC.

The cause is usually related to promotion of objects that will become eligible for collection right after the promotion takes place, and not enough time for the concurrent collection to go over the entire heap in time before new objects are promoted. How to deal with that?

• Make the CMS kick in sooner by reducing the initiating occupancy fraction (-XX:CMSInitiatingOccupancyFraction) to less than the default value
• Make CMS life’s simpler by reducing the size of the heap
• Enable incremental mode (-XX:+CMSIncrementalMode)

What Else?

Moving to a 1.6 environment and using the default JVM configuration in your environment might solve your issue right away – why? escape analysis. This means that not all short lived objects are allocated on the execution stack, saving precious space on the young generation and avoiding all the promotion issues and the consequences we listed here.

Why did it take Java so long to get there?

1. One or more solutions that cut through all the layers of an architecture
2. Time constrained
3. It is not intended to be thrown away, however there is no commitment on using spikes ‘as is’ – should be driven to allow re-use of ideas and concepts, not format.

Basically, Spikes are a tool to reduce uncertainty. Use them often and wisely. And you should be prepared to refer back to them from time to time in search of alternatives, benchmarks and so on.

You should expect some controversy on the definition of a spike. Some authors seem to think that it should not be time constrained, some others question retaining anything from it at all – use your best judgement.

Price, Volume, Gain & Loss in 3D

The example above from MIT’s financial visualization lab shows price, volume over a time series and instantaneous gain/loss. As you can see this and other similar applications out there resemble a neat 3D gaming interface and not grandpa’s trading blotter.

Anyway, needless to say, the choice of a framework to support your user experience can go as far as determining the success or failure of your application – in some cases it can even balance for limitations on other functions performed in the back-end.

As of now thankfully there are many good options out there. Hands down RIA (rich internet [or interactive in MS parlance] applications) frameworks are best suited for the task. In essence RIA frameworks allow thick, full featured clients to be deployed and launched over the internet. The main current players are JavaFX, Silverlight, Flex, GWT and Openlazslo.

JavaFX

The JavaFX platform was announced by Sun about a year ago. A long waited response to something to counter balance deficiencies in the standard Swing/AWT. It is available by default on JRE 6 and can optionally rely on battle tested JNLP for remote deployment, execution and security.

Some of its major limitations come from the fact that even after one year, good development tools are still not available. Most common development tools are provided as part of the NetBeans IDE, some limited plugins are available for Eclipse IDE. We could not find many commercial applications relying on the framework as well.

Strong points are embedded JDK support, reliance on stable Java protocols and Java language binding and a neat scripting language for view definition, for example:

Button {
  text: "Click Me"
  action:
    function():Void {
      MessageDialog {
        title: "JavaFX Script Rocks!"
        // This string has a newline in the source code
        message: "JavaFX Script is Simple, Elegant,
         and Leverages the Power of Java"
        visible: true
       }
     }
}

Defines a button, and a function to be triggered after an action to that button – this sample code is an exact copy of code found in the scripting tutorial.

Silverlight

Silverlight is MIcrosoft’s RIA framework for .NET, very popular in financial software shops. Runtime is supported for Mac and Windows, development environment in Visual Studio requires Windows. Views are declared following XML descriptors as well.

<UserControl x:Class="DiggSample.Page"
    xmlns="http://schemas.microsoft.com/client/2007"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:Digg="clr-namespace:DiggSample;assembly=DiggSample">
    <Grid Style="{StaticResource TopGrid}">
            <Button x:Name="btnSearch"
                    Content="Search"
                    Click="SearchBtn_Click"
                    Style="{StaticResource SearchButton}" />
    </Grid>
</UserControl>

Again, the same usage sample: a button and an action. This code snippet – reduced for simplicity – is from Silverlight’s tutorial and samples. Silverlight is target mainly at Windows platforms, Mac is supported as a run time as well. For an open source implementation covering other OSes as of this moment you will have to rely on Moonlight, Mono‘s implementation.

Flex

Adobe’s Flex has a commercial and an open source SDK distribution. Since it was one of the early products on this arena, is the most widely used, specially for public web applications.

It is not an exception when it comes to XML-like language for UI definition called MXML and a “powerful object-oriented language” called ActionScript (yes, we need yet another). Below is a sample MXML and script of a Flickr-like photo application.

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml"
    backgroundGradientColors="[0xFFFFFF, 0xAAAAAA]"
    horizontalAlign="left"
    verticalGap="15"
    horizontalGap="15">

    <mx:Script>
        <![CDATA[
            import mx.collections.ArrayCollection;
            import mx.rpc.events.ResultEvent;

            [Bindable]
            private var photoFeed:ArrayCollection;

            private function requestPhotos():void {
                photoService.cancel();
                var params:Object = new Object();
                params.format = 'rss_200_enc';
                params.tags = searchTerms.text;
                photoService.send(params);
            }

            private function photoHandler(event:ResultEvent):void {
                 photoFeed = event.result.rss.channel.item as ArrayCollection;
            }
         ]]>
    </mx:Script>

    <mx:HTTPService id="photoService"
        url="http://api.flickr.com/services/feeds/photos_public.gne"
        result="photoHandler(event)" />

	<mx:HBox>
		<mx:Label text="Flickr tags or search terms:" />
		<mx:TextInput id="searchTerms" />
		<mx:Button label="Search"
			click="requestPhotos()" />
	</mx:HBox>

	<mx:TileList width="100%" height="100%"
		dataProvider="{photoFeed}"
		itemRenderer="FlickrThumbnail">
	</mx:TileList>

</mx:Application>

There is extensive documentation and tutorials available. This code snippet came from there as well.

It is interesting to note the recent swap of designers and developers working on JavaFX and Flex SDK – some common strategies on these two arenas started to take shape. It would not come as a surprise if these platforms converge in the near future.

GWT

The GWT platform from all powerful Google is basically a set of development tools and IDE plugins that allow you to use Java to define RIA solutions. No surprises here as well, although you should not need to manipulate them directly, XML is used again to define GWT “modules”. You can find plenty of examples and documentation on GWT’s tutorial resources.

OpenLaszlo

My favorite, an DHTML (optionally FLASH) based framework, many strong points: stable, open source, simple to deploy, test. A variety of resources to support data biding and scripting. My only comment would be once more, the over use of XML as a language called LZX:

<canvas height="200" width="500">
  <window x="20" y="20" width="150" title="Simple Window" resizable="true">
    <button text="My button" onclick="this.parent.setAttribute('title', 'You clicked it');"/>
  </window>
</canvas>

Well, self explanatory. A button again and an action. You can embed scripts in LZX as well:

<canvas height="120">
   <script>
     <![CDATA[
     for (var i = 0; i < 11; i++) {
       Debug.write(i);
     }
     ]]>
   </script>
</canvas>

Again, these examples and many others can be found in Openlaszlo’s scripting tutorial.

Finally

Great options, you might want to try these tutorials and find out which ones fit your use cases better. Sorry if you were expecting this article to provide a definitive answer here, far from that.

This is an active area of development currently, so you should expect many improvements for each of these contenders over the upcoming months.