poetiq / poetiq Goto Github PK
View Code? Open in Web Editor NEWpoetiq - Platform O' Electronic Trading In Q
poetiq - Platform O' Electronic Trading In Q
For prototyping and testing purposes. Virtual exchange generates fills and publishes them to the portfolio tracker.
The timings for https://a.kx.com/q/state.q are interesting. State transition matrix is the fastest
\t 0b{$[x;y>6;y>11]}\p
151
\t 0 m\p
5
\t 0(0 0 1;0 1 1)\1+(p>11)-p<7
24
Rewrite backtester from stinky loops into state transition model
Currently I am eaching through queue of events and assigning to global state, which is only step better than looping. Proxy timings of current code:
\t {s::$[x>11;1;x<6;0]} each p
311
\t {s::x} each p
217
The current command line argument parsing is hand-made:
2432793
Improve it with https://github.com/nugend/qutil/wiki/Option-Parsing-Facility, which is more flexible:
it parses required arguments and positional options, handles binary flags, provides automatic casting of variables, automatic variable assignment, calling functions, and allows for required arguments and positional options to be mixed along with keyword options.
Using aedd514 throws an error:
MINGW64 /f/gdrive/q/poetiq (master)
$ source bin/control.sh
2016.10.15T18:23:07.361 |ERROR| usr/bin/bash[11356] | - Q binary not found
My Q path is:
MINGW64 /f/gdrive/q/poetiq (master)
$ ${QHOME}
bash: F:\gdrive\q: Is a directory
\d .servers
STARTUP:1b
CONNECTIONS:`discovery`bttickerplant
\d .proc
loadprocesscode: 1b
.servers.startup[]
hbtt: .servers.gethandlebytype[`bttickerplant;`any]
btt: .sub.getsubscriptionhandles[`bttickerplant;();()!()]
subinfo:.sub.subscribe[`order;`;1b;0b;first btt]
src.tablename
- prepend src
to file/directory names on disk?
Rename this to e.g. equitysim.trade
Otherwise var names may clash with other modules.
Source: Namespaces vs Directories chapter of Psaris: Q Tips.
SAMPLING-FREQUENCY: 1 month (1 day, 1 year, 1 quarter, 1 week, 1 minute, 5 minutes, 1 hour, open:close)
ENTRY: P >= MA(P, 12)
EXIT: P < MA(P, 12)
STOP-LOSS: None
RANKING (ascending): P/Earnings - 1
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight (Risk-parity)
TRANSACTION COST: 0.001
SAMPLING-FREQUENCY: 1 day
ENTRY: P >= MA(P, 100) and RSI(2, P) < 20
EXIT: P < MA(P, 100)
STOP-LOSS: Drawdown(Equity_Curve) < -0.2
RANKING (ascending): None
NUMBER_OF_ASSETS: ALL
UNIVERSE: (AAPL, MSFT)
WEIGHTING: (50, 50)
TRANSACTION COST: 0
SAMPLING-FREQUENCY: 5 minutes
ENTRY: P >= MA(P, 100) and RSI(2, P) < 20 and TIME=1400EST
EXIT: P < MA(P, 100) OR TIME>=1459EST
STOP-LOSS: Drawdown(P) < -0.2
RANKING (ascending): None
NUMBER_OF_ASSETS: ALL
UNIVERSE: (AAPL, MSFT)
WEIGHTING: (50, 50)
TRANSACTION COST: 0
SAMPLING-FREQUENCY: open:close
ENTRY: TIME=1555
EXIT: TIME=1550
STOP-LOSS: None
RANKING (ascending): -Ret(P, P-1)
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight
TRANSACTION COST: 0
SAMPLING-FREQUENCY: 5 minutes
ENTRY: DAY=Tue and TIME = 1555EST
EXIT: DAY=Tue and TIME = 1550EST
STOP-LOSS: None
RANKING (ascending): -Kurtosis(P, 300)
NUMBER_OF_ASSETS: 20
UNIVERSE: SPX Index
WEIGHTING: Equal-weight
TRANSACTION COST: 0
At the moment, the path is tickerplant dependent legacy:
/code/tick/tick/equitysim.q
Event schemas currently look like this:
quote:flip`time`sym`bid`ask`bsize`asize`mode`ex!"tsffjjcc"$\:()
trade:flip`time`sym`price`size`stop`cond`ex!"tsfjbcc"$\:()
mtm:flip`time`date`sym`close!"ndsf"$\:()
signal:flip`sym`date`signal`time!"sdin"$\:()
targetsz:flip`sym`date`sz`time!"sdin"$\:()
targetw:flip`sym`date`w`time!"sdfn"$\:()
order:flip`sym`date`size`time!"sdin"$\:()
fill:flip`date`sym`time`price`size!"dsnfj"$\:()
Move HTML interface to separate repo Poetiq-GUI.
From q torq.q -usage
:
Currently each process has to know two things about itself - it's name and type. These can be
- read from a file (the default behaviour)
- supplied on the command line
- set previously if sourced from other files (set .proc.proctype and .proc.procname)
Strategy converts signals into target portfolio weights by instrument.
Orders are created comparing current portfolio weights to target portfolio weights.
This is part of the effort to separate TorQ and business logic.
For example, switching easily between target size and target weight portfolio construction:
https://github.com/poetiq/poetiq/blob/master/src/optimiser/opt.q#L3-L6
Replace this with asof price logic
https://github.com/danielkrizian/poetiq/blob/backtest-unit/code/fillsim/f.q#L3
I just noticed the Projects page and found it quite useful. I was thinking if we should start logging everything we do in Issues or Projects and refer to that in our commit messages.
I could have a project called Launch Control (id #123). I would prefix each commit message related to that project with e.g. "#123 update something".
Not sure if you're using JIRA at your company? But it's the same concept. It's quite useful to see what the other guys are working on. And if you're in doubt why I just pushed a particular commit, you can refer to the Issue page for more details.
Follow advice by Nick Psaris (Q Tips book and here)
Useful for testing and debugging purposes
Using latest commit 039e84d (on MINGW64), trying to launch my first process doesn't seem to do anything and just returns usage help:
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ echo $QBIN
F:\gdrive\q/w32/q.exe
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ source bin/control.sh
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ startp
Usage:
startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ startp discovery discovery1 -debug -new_console:nc:t:'discovery'
Usage:
startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ startp discovery discovery1 -debug
Usage:
startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]
User@DeathStar MINGW64 /f/gdrive/q/poetiq (master)
$ startp discovery discovery1 -debug --args -new_console:nc:t:'discovery'
Usage:
startp <proctype> <procname> [-w <size>] [--log <loglevel>] [--debug] [--args <customargs>]
/code/tick/backtestfeed.q
The performance can be improved by separating/quasi-parallelising "events"+"orderdata" into one process and "feed" into own process.
While "feed" works its way down the chunk 1 queue, "events"+"orderdata" pre-processes chunk 2 and when done appends to the queue.
Feed chunk 1 and prepare (eventise and order) chunk 2 in parallel. The event queue is the conduit between the two processes.
Expected benefits in addition to performance: no if statements, shorter code
The biggest backtest performance hog is the construction of event queue. Market prices used for e.g. mtm take up at least half of the queue size (and for low freqency strats significantly more than half).
Therefore, the idea is to preload market price table into memory and presort by tstamp once.
Then market component of the backtest will pull price on demand from the table:
select first price from pricetbl where tstamp > .clock.then
The tradeoff here is between the costs of the following two strategies:
a:select sym, close by date from daily where int<20 / , date since 1973, rows:11630, size: 634MB, time: 2000ms
\t:10000 select close from a where date=2012.01.03 / 8ms
counter::0
\t:1 {counter+::1} each a / 6ms
Find unified point of access to historical database and adhoc datasets outside the process.
Remove the need for ad-hoc loading of hdb dependencies such as \l hdb/equitysim
here:
https://github.com/poetiq/poetiq/blob/modular-folders/src/fillsim/f.q#L14
The above depends on correct directory so we are currently hacking with cd
here:
https://github.com/poetiq/poetiq/blob/modular-folders/config/settings/default.q#L1
The issue is that the process (fillsim in this case, but all subscribers as well) ends up in this state:
q)system "cd"
"F:\\gdrive\\q\\poetiq\\hdb\\equitysim"
interfaces are prepended with directory (context) notation (.)
.market
; .bt
; oms
; port
;
dt
?
External interfaces (.market
) can then be abstracted and applied uniformly in both backtesting and live trading mode. See https://github.com/poetiq/poetiq/wiki/Code-Organisation#backtestinglive-trading-modes-duality
Example of manipulating parse tree at load time
.market.upd.order: {[x] handletomarket (`upd;`order;x) } / x - order data
value .market.upd.order
/
0x78a0a1a20a0381520004
,`x
`symbol$()
``handletomarket
`order
`upd
enlist
"{[x] handletomarket (`upd;`order;x) }"
As the code base grows and we are each working on different parts of the project, I think it is becoming increasingly important to focus on testing and TDD. This obviously makes sure our components work individually and together, but also helps document their usage.
When strategy queries data from disk, eliminate from retrieval any records that don’t fall within one of the sym-date-time pairs while only performing one access to the on disk database. Also minimises memory footprint.
Inspiration from:
https://github.com/nugend/qspec/wiki/Tutorial#curve-v4q
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.