poetiq / poetiq Goto Github PK

View Code? Open in Web Editor NEW

37.0 6.0 10.0 9.15 MB

poetiq - Platform O' Electronic Trading In Q

q 100.00%

q kdb algorithmic-trading trading-simulator trading-platform backtester trading

poetiq's Introduction

POETIQ - Platform O' Electronic Trading In Q

Poetiq is a backtesting and algorithmic trading engine built in kdb+/Q.

Requirements

qutil - command line parsing utility
For testing and simulation use the buildhdb.q script to create a HDB equitysim with simulated equity data.

cd hdb
q buildhdb.q -S 104831        # seed random numbers generator for reproducible results

Installation

$ alias q='QPATH=~/q/lib:~/q/repo/poetiq'
$ alias bt='q ~/q/repo/poetiq/init.q -p 5000 --backtest'
$ cd poetiq

Run

$ bt src strategy/strategy1.q

Interactively:

q init.q
q) env.cfg.paths: `:src/`:strategy/strategy1.q / or write your own strategy
q) backtest[env]

poetiq's People

Contributors

Stargazers

Watchers

Forkers

minhpascal fisherx logtest bayao jtzharry sahanduiuc secwang ashwinrajesh8 ghlian 864811699

poetiq's Issues

Replace integer type with date in `dt` column (PnL, fills, prices)

Integrate with TorQ

See https://github.com/AquaQAnalytics/TorQ

Scope database tables within namespaces named by source

src.tablename - prepend src to file/directory names on disk?
Rename this to e.g. equitysim.trade

Otherwise var names may clash with other modules.

Source: Namespaces vs Directories chapter of Psaris: Q Tips.

Fill simulator: fill 100% of the order at last trade price - daily or intraday option

Replace this with asof price logic
https://github.com/danielkrizian/poetiq/blob/backtest-unit/code/fillsim/f.q#L3

backtestfeed: drip feed batches of events manually

Useful for testing and debugging purposes

OMS: Propose order size as delta between current and target portfolio weights/shares/value

Retrieve from disk only the data necessary

When strategy queries data from disk, eliminate from retrieval any records that don’t fall within one of the sym-date-time pairs while only performing one access to the on disk database. Also minimises memory footprint.

Inspiration from:
https://github.com/nugend/qspec/wiki/Tutorial#curve-v4q

Fill simulator: Fill on the last traded price at the time of the order plus delay

Show equity curve in the browser

Modularity (replace strategy components)

This is part of the effort to separate TorQ and business logic.

For example, switching easily between target size and target weight portfolio construction:
https://github.com/poetiq/poetiq/blob/master/src/optimiser/opt.q#L3-L6

Rewrite into state transition model from stinky loops

The timings for https://a.kx.com/q/state.q are interesting. State transition matrix is the fastest

\t 0b{$[x;y>6;y>11]}\p
    151
\t 0 m\p
    5
\t 0(0 0 1;0 1 1)\1+(p>11)-p<7
    24

Rewrite backtester from stinky loops into state transition model
Currently I am eaching through queue of events and assigning to global state, which is only step better than looping. Proxy timings of current code:

\t {s::$[x>11;1;x<6;0]} each p
     311
\t {s::x} each p
    217

`.proc.proctype` to replace command line args

From q torq.q -usage:

Currently each process has to know two things about itself - it's name and type. These can be

read from a file (the default behaviour)

supplied on the command line

set previously if sourced from other files (set .proc.proctype and .proc.procname)

Consolidate repetitive code

    \d .servers
STARTUP:1b
CONNECTIONS:`discovery`bttickerplant

\d .proc
loadprocesscode: 1b

.servers.startup[]
hbtt: .servers.gethandlebytype[`bttickerplant;`any]
btt: .sub.getsubscriptionhandles[`bttickerplant;();()!()]
subinfo:.sub.subscribe[`order;`;1b;0b;first btt]

.DELETE/REUSE AT SIGHT

Usage of Projects

I just noticed the Projects page and found it quite useful. I was thinking if we should start logging everything we do in Issues or Projects and refer to that in our commit messages.

I could have a project called Launch Control (id #123). I would prefix each commit message related to that project with e.g. "#123 update something".

Not sure if you're using JIRA at your company? But it's the same concept. It's quite useful to see what the other guys are working on. And if you're in doubt why I just pushed a particular commit, you can refer to the Issue page for more details.

.DELETE/REUSE AT SIGHT

source bin/control.sh : Q binary not found on win

Using aedd514 throws an error:

MINGW64 /f/gdrive/q/poetiq (master)
$ source bin/control.sh
2016.10.15T18:23:07.361 |ERROR| usr/bin/bash[11356] | - Q binary not found

My Q path is:

MINGW64 /f/gdrive/q/poetiq (master)
$ ${QHOME}
bash: F:\gdrive\q: Is a directory

Patchy hdb loading from other processes

Find unified point of access to historical database and adhoc datasets outside the process.

Remove the need for ad-hoc loading of hdb dependencies such as \l hdb/equitysim here:
https://github.com/poetiq/poetiq/blob/modular-folders/src/fillsim/f.q#L14

The above depends on correct directory so we are currently hacking with cd here:
https://github.com/poetiq/poetiq/blob/modular-folders/config/settings/default.q#L1

The issue is that the process (fillsim in this case, but all subscribers as well) ends up in this state:

q)system "cd"
"F:\\gdrive\\q\\poetiq\\hdb\\equitysim"

.DELETE/REUSE AT SIGHT

Pull (vs push) backtest market prices

The biggest backtest performance hog is the construction of event queue. Market prices used for e.g. mtm take up at least half of the queue size (and for low freqency strats significantly more than half).
Therefore, the idea is to preload market price table into memory and presort by tstamp once.

Then market component of the backtest will pull price on demand from the table:
select first price from pricetbl where tstamp > .clock.then

The tradeoff here is between the costs of the following two strategies:

cost of a) presort + b) pull some prices on demand
cost of c) build event queue all + d) push all

a:select sym, close by date from daily where int<20 / , date since 1973, rows:11630, size: 634MB, time: 2000ms
\t:10000 select close from a where date=2012.01.03     / 8ms

counter::0
\t:1 {counter+::1} each a                                                 / 6ms

Where can we store event schemas better?

At the moment, the path is tickerplant dependent legacy:
/code/tick/tick/equitysim.q

Event schemas currently look like this:

quote:flip`time`sym`bid`ask`bsize`asize`mode`ex!"tsffjjcc"$\:()
trade:flip`time`sym`price`size`stop`cond`ex!"tsfjbcc"$\:()
mtm:flip`time`date`sym`close!"ndsf"$\:()
signal:flip`sym`date`signal`time!"sdin"$\:()
targetsz:flip`sym`date`sz`time!"sdin"$\:()
targetw:flip`sym`date`w`time!"sdfn"$\:()
order:flip`sym`date`size`time!"sdin"$\:()
fill:flip`date`sym`time`price`size!"dsnfj"$\:()

Data in global namespaces, functions and config vars in directories/contexts

Follow advice by Nick Psaris (Q Tips book and here)

Optimiser: On +1(-1) signal, be long(short) one share/unit

Command line arguments with qutil - upgrade

The current command line argument parsing is hand-made:
2432793

Improve it with https://github.com/nugend/qutil/wiki/Option-Parsing-Facility, which is more flexible:

it parses required arguments and positional options, handles binary flags, provides automatic casting of variables, automatic variable assignment, calling functions, and allows for required arguments and positional options to be mixed along with keyword options.

`.module.upd[eventtype;data]` - dictionaries type dispatch

Expected benefits in addition to performance: no if statements, shorter code

Move HTML to Poetic-GUI

Move HTML interface to separate repo Poetiq-GUI.

Demo Strategies Specs

SAMPLING-FREQUENCY: 1 month (1 day, 1 year, 1 quarter, 1 week, 1 minute, 5 minutes, 1 hour, open:close)
ENTRY: P >= MA(P, 12)
EXIT: P < MA(P, 12)
STOP-LOSS: None
RANKING (ascending): P/Earnings - 1
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight (Risk-parity)
TRANSACTION COST: 0.001

SAMPLING-FREQUENCY: 1 day
ENTRY: P >= MA(P, 100) and RSI(2, P) < 20
EXIT: P < MA(P, 100)
STOP-LOSS: Drawdown(Equity_Curve) < -0.2
RANKING (ascending): None
NUMBER_OF_ASSETS: ALL
UNIVERSE: (AAPL, MSFT)
WEIGHTING: (50, 50)
TRANSACTION COST: 0

SAMPLING-FREQUENCY: 5 minutes
ENTRY: P >= MA(P, 100) and RSI(2, P) < 20 and TIME=1400EST
EXIT: P < MA(P, 100) OR TIME>=1459EST
STOP-LOSS: Drawdown(P) < -0.2
RANKING (ascending): None
NUMBER_OF_ASSETS: ALL
UNIVERSE: (AAPL, MSFT)
WEIGHTING: (50, 50)
TRANSACTION COST: 0

SAMPLING-FREQUENCY: open:close
ENTRY: TIME=1555
EXIT: TIME=1550
STOP-LOSS: None
RANKING (ascending): -Ret(P, P-1)
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight
TRANSACTION COST: 0

SAMPLING-FREQUENCY: 5 minutes
ENTRY: DAY=Tue and TIME = 1555EST
EXIT:  DAY=Tue and TIME = 1550EST
STOP-LOSS: None
RANKING (ascending): -Kurtosis(P, 300)
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight
TRANSACTION COST: 0

Equity calculation prototype - mark-to-market method

https://drive.google.com/open?id=0ByqnOlY7vMfyOThENnhTcGo4VG8

Target portfolio weights

Strategy converts signals into target portfolio weights by instrument.
Orders are created comparing current portfolio weights to target portfolio weights.

Setup CI server

As the code base grows and we are each working on different parts of the project, I think it is becoming increasingly important to focus on testing and TDD. This obviously makes sure our components work individually and together, but also helps document their usage.

.DELETE/REUSE AT SIGHT

`startp` does not start (returns Usage only)

Using latest commit 039e84d (on MINGW64), trying to launch my first process doesn't seem to do anything and just returns usage help:

User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ echo $QBIN                                                                                              
F:\gdrive\q/w32/q.exe                                                                                     

User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ source bin/control.sh                                                                                   

User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ startp                                                                                                  
Usage:                                                                                                    
        startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]       


User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ startp discovery discovery1 -debug -new_console:nc:t:'discovery'                                        
Usage:                                                                                                    
        startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]       


User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ startp discovery discovery1 -debug                                                                      
Usage:                                                                                                    
        startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]       


User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)                                                        
$ startp discovery discovery1 -debug --args -new_console:nc:t:'discovery'                                 
Usage:                                                                                                    
        startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]

Parallelise backtestfeed.q: Feed chunk 1 and prepare chunk 2 in parallel

/code/tick/backtestfeed.q

The performance can be improved by separating/quasi-parallelising "events"+"orderdata" into one process and "feed" into own process.

While "feed" works its way down the chunk 1 queue, "events"+"orderdata" pre-processes chunk 2 and when done appends to the queue.

Feed chunk 1 and prepare (eventise and order) chunk 2 in parallel. The event queue is the conduit between the two processes.

Refresh equity curve display when view updates

Interfaces in directory namespace (contexts)

interfaces are prepended with directory (context) notation (.)

.market; .bt; oms; port;
dt ?

External interfaces (.market) can then be abstracted and applied uniformly in both backtesting and live trading mode. See https://github.com/poetiq/poetiq/wiki/Code-Organisation#backtestinglive-trading-modes-duality

Example of manipulating parse tree at load time

.market.upd.order: {[x] handletomarket (`upd;`order;x) } / x - order data

value .market.upd.order
/
0x78a0a1a20a0381520004
,`x
`symbol$()
``handletomarket
`order
`upd
enlist
"{[x] handletomarket (`upd;`order;x) }"

Fill generator

For prototyping and testing purposes. Virtual exchange generates fills and publishes them to the portfolio tracker.

poetiq / poetiq Goto Github PK

poetiq's Introduction

POETIQ - Platform O' Electronic Trading In Q

Requirements

Installation

Run

poetiq's People

Contributors

Stargazers

Watchers

Forkers

poetiq's Issues

Recommend Projects

Recommend Topics

Recommend Org