robcarver17 / pysystemtrade Goto Github PK
View Code? Open in Web Editor NEWSystematic Trading in python
License: GNU General Public License v3.0
Systematic Trading in python
License: GNU General Public License v3.0
In [50]: pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
...: for code in instrument_codes]
TypeError Traceback (most recent call last)
in ()
1 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
----> 2 for code in instrument_codes]
in (.0)
1 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
----> 2 for code in instrument_codes]
TypeError: pandl_for_subsystem() got an unexpected keyword argument 'percentage'
Hi Rob,
I am testing new version of the code (0.12.0) and noticed one thing. If there are no cheap enough rules for an instrument available, the calc crashes in syscore\accounting.py. To reproduce the bug, it is enough to increase instrument costs in cost_analysis.csv column Slippage.
Best regards,
Pavel
PS. I am attaching the stack (long read):
Error: ValueError: No objects to concatenate
Traceback (most recent call last):
File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3066, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 2, in
acc_curve_group=system.accounts.portfolio()
File "C:\Project\Project\systems\account.py", line 1769, in portfolio
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1757, in _portfolio
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 1757, in
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 1101, in pandl_for_instrument
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1056, in _pandl_for_instrument
positions = this_stage.get_buffered_position(instrument_code, roundpositions = roundpositions)
File "C:\Project\Project\systems\account.py", line 991, in get_buffered_position
flags="roundpositions%s" % TorF(roundpositions))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 977, in _get_buffered_position
optimal_position=this_stage.get_notional_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 364, in get_notional_position
return self.parent.portfolio.get_notional_position(instrument_code)
File "C:\Project\Project\systems\portfolio.py", line 347, in get_notional_position
"get_notional_position", instrument_code, _get_notional_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 330, in _get_notional_position
idm = this_stage.get_instrument_diversification_multiplier()
File "C:\Project\Project\systems\portfolio.py", line 719, in get_instrument_diversification_multiplier
self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 710, in _get_instrument_div_multiplier
correlation_list_object=this_stage.get_instrument_correlation_matrix()
File "C:\Project\Project\systems\portfolio.py", line 673, in get_instrument_correlation_matrix
self, corr_func, *_corr_params)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, *args, *_kwargs)
File "C:\Project\Project\systems\portfolio.py", line 646, in _get_instrument_correlation_matrix
pandl=this_stage.pandl_across_subsystems().to_frame()
File "C:\Project\Project\systems\portfolio.py", line 776, in pandl_across_subsystems
return self.parent.accounts.pandl_across_subsystems()
File "C:\Project\Project\systems\account.py", line 945, in pandl_across_subsystems
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 933, in _pandl_across_subsystems
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 933, in
roundpositions=roundpositions) for instrument_code in instruments]
File "C:\Project\Project\systems\account.py", line 891, in pandl_for_subsystem
TorF(delayfill), TorF(roundpositions)))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 860, in _pandl_for_subsystem
positions = this_stage.get_aligned_subsystem_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 257, in get_aligned_subsystem_position
_get_aligned_subsystem_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 249, in _get_aligned_subsystem_position
sspos=this_stage.get_subsystem_position(instrument_code)
File "C:\Project\Project\systems\account.py", line 233, in get_subsystem_position
return self.parent.positionSize.get_subsystem_position(instrument_code)
File "C:\Project\Project\systems\positionsizing.py", line 493, in get_subsystem_position
'get_subsystem_position', instrument_code, _get_subsystem_position, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\positionsizing.py", line 484, in _get_subsystem_position
forecast = this_stage.get_combined_forecast(instrument_code)
File "C:\Project\Project\systems\positionsizing.py", line 76, in get_combined_forecast
return self.parent.combForecast.get_combined_forecast(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 538, in get_combined_forecast
'get_combined_forecast', instrument_code, _get_combined_forecast, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 512, in _get_combined_forecast
forecast_weights = this_stage.get_forecast_weights(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 392, in get_forecast_weights
'get_forecast_weights', instrument_code, _get_forecast_weights, self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 376, in _get_forecast_weights
instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 1092, in get_raw_forecast_weights
self)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 1086, in _get_raw_forecast_weights
return this_stage.calculation_of_raw_forecast_weights(instrument_code).weights
File "C:\Project\Project\systems\forecast_combine.py", line 875, in calculation_of_raw_forecast_weights
return self.calculation_of_raw_forecast_weights_for_instrument(instrument_code)
File "C:\Project\Project\systems\forecast_combine.py", line 959, in calculation_of_raw_forecast_weights_for_instrument
self, codes_to_use, weighting_func, pool_costs, *_weighting_params)
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, *args, *_kwargs)
File "C:\Project\Project\systems\forecast_combine.py", line 907, in _calculation_of_raw_forecast_weights
for code in codes_to_use]
File "C:\Project\Project\systems\forecast_combine.py", line 907, in
for code in codes_to_use]
File "C:\Project\Project\systems\forecast_combine.py", line 773, in get_returns_for_optimisation
return self.parent.accounts.pandl_for_instrument_rules_unweighted(instrument_code, rule_list)
File "C:\Project\Project\systems\account.py", line 1311, in pandl_for_instrument_rules_unweighted
delayfill))
File "C:\Project\Project\systems\basesystem.py", line 630, in calc_or_cache
value = func(self, instrument_code, this_stage, _args, *_kwargs)
File "C:\Project\Project\systems\account.py", line 1300, in _pandl_for_instrument_rules_unweighted
weighted_flag=False)
File "C:\Project\Project\syscore\accounting.py", line 1323, in init
curve_type="net")
File "C:\Project\Project\syscore\accounting.py", line 1058, in init
(acc_total, capital)=total_from_list(acc_curve_for_type_list, asset_columns, capital)
File "C:\Project\Project\syscore\accounting.py", line 979, in total_from_list
pdframe=acc_list_to_pd_frame(list_of_ac_curves, asset_columns)
File "C:\Project\Project\syscore\accounting.py", line 955, in acc_list_to_pd_frame
ans=pd.concat(list_of_df, axis=1, join="outer")
File "D:\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 834, in concat
copy=copy)
File "D:\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 867, in init
raise ValueError('No objects to concatenate')
ValueError: No objects to concatenate
Hi,
On https://github.com/robcarver17/pysystemtrade/blob/master/docs/introduction.md there is a line stating from systems.forecast_combine import ForecastCombine
; however, when I do this, I receive the error ImportError: cannot import name 'ForecastCombine'
.
I checked the forecast_combine.py file and there are only two options in there: i) ForecastCombineFixed
, and ii) ForecastCombineEstimated
.
What should the relevant code be at the site mentioned above? Should it read: from systems.forecast_combine import ForecastCombine
?
Hi,
I updated data in pysystemtrade today, so the last data entry was 2016-09-26.
Each time I run pysytemtrade I get an error during the correlation estimate. The last data point in all of the legacycsv files I have is 2016-09-26 (I have triple checked this). When pysytemtrade goes to estimate the correlations, for some reason it attempts to estimate out to 2016-10-02 even though there is no data after 2016-09-26. My log states:
"Estimating from 2016-01-31 00:00:00 to 2016-10-02 00:00:00"
Why is it trying to estimate out to 2016-10-02?
I believe it is an error in fit_period. I've done this every day for a month and this is the first time I've encountered this. I am using expanding estimates.
I’m not sure if this is an issue or not.
When daily cash volatility trader is calculated the trading capital is constant throughout time. It doesn’t take in account history profit and loses.
Another thing regarding capital is that the system ignore buying power when generate positions. And in some scenarios you end up buying more than your capital allow.
Hello, Rob. Thanks for the book and this repo.
I am running the following code:
from systems.provided.futures_chapter15.estimatedsystem import futures_system
system=futures_system(log_level="on")
system.combForecast.get_combined_forecast("EDOLLAR")
And getting the following error:
Traceback (most recent call last):
File "<ipython-input-1-15484267b926>", line 1, in <module>
runfile('C:/Users/paulj/Documents/Upwork/Ravi_Balsa/futures_system.py', wdir='C:/Users/paulj/Documents/Upwork/Ravi_Balsa')
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 880, in runfile
execfile(filename, namespace)
File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/paulj/Documents/Upwork/Ravi_Balsa/futures_system.py", line 6, in <module>
system.combForecast.get_combined_forecast("EDOLLAR")
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 685, in wrapper
**kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 582, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 899, in get_combined_forecast
raw_multiplied_combined_forecast = self._get_raw_combined_forecast(instrument_code)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 914, in _get_raw_combined_forecast
instrument_code)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 656, in wrapper
return func(*args, **kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 849, in get_forecast_diversification_multiplier
return self.get_forecast_diversification_multiplier_estimated(instrument_code)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 685, in wrapper
**kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\system_cache.py", line 582, in calc_or_cache
value = func(this_stage, *args, **kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\systems\forecast_combine.py", line 838, in get_forecast_diversification_multiplier_estimated
**div_mult_params)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\syscore\divmultipliers.py", line 92, in diversification_multiplier_from_list
corrmatrix, weights, **kwargs)
File "c:\users\paulj\documents\upwork\richard spencer\pysystemtrade\syscore\divmultipliers.py", line 38, in diversification_mult_single_period
(float(np.dot(np.dot(weights, corrmatrix), weights.transpose()))**.5),
ValueError: shapes (1,) and (4,1) not aligned: 1 (dim 0) != 4 (dim 0)
That happens in an advanced stage. After the system outputs this:
Calculating forecast correlations over CORN, EDOLLAR, EUROSTX, MXP, US10, V2X
[================================================================================] 100.0%
Estimating correlations
Hi Rob,
system.portfolio.get_actual_position("CORN") seems to not work. I have tried using both fixed and estimated portfolios. The errors show up as follows:
AttributeError: 'PortfoliosFixed' object has no attribute 'accounts'
AttributeError: 'PortfoliosEstimated' object has no attribute 'accounts'
Same error shows up with get_actual_buffers_for_position as well. For what it's worth, I am only running one trading rule on one instrument (corn). Thank you very much.
Regards,
Tony
I am trying to use pysystemtrade to backtest your estimated system across all instruments you include in the csv files (on Python 3 in Windows 10).
I create new config in pysystemtrade/private/my_system/futuresestimateconfigme.yaml
which exactly copies your pysystemtrade\systems\provided\futures_chapter15\futuresestimateconfig.yaml
file, but adds in all of the instruments in the last item in the yaml file as a list. instruments: ['CRUDE_W', 'GOLD', 'PLAT', 'CORN', 'KR3', 'SOYBEAN', 'BOBL', 'SHATZ'...etc]
I then run the following code:
from sysdata.configdata import Config
from systems.provided.futures_chapter15.estimatedsystem import futures_system
my_config = Config("private.my_system.futuresestimateconfigme.yaml")
Each time I try to run this, I obtain the following 'KeyError: 'private'
.
I have tried this with and without __init__.py
files in the 'private' and 'my_system' folders.
What am I missing?
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-11-24c926ebb4e0> in <module>()
----> 1 my_config = Config("private.my_system.futuresestimateconfigme.yaml")
C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\sysdata\configdata.py in __init__(self, config_object)
52 self._create_config_from_item(config_item)
53 else:
---> 54 self._create_config_from_item(config_object)
55
56 ## this will normally be overriden by the base system
C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\sysdata\configdata.py in _create_config_from_item(self, config_item)
65 elif isinstance(config_item, str):
66 # must be a file YAML'able, from which we load the
---> 67 filename = get_filename_for_package(config_item)
68 with open(filename) as file_to_parse:
69 dict_to_parse = yaml.load(file_to_parse)
C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\syscore\fileutils.py in get_filename_for_package(name_with_dots)
24 path_as_list[-1] = path_as_list[-2] + "." + path_as_list.pop()
25
---> 26 return get_pathname_for_package_from_list(path_as_list)
27
28
C:\Anaconda3\Python_Library\Python_Folder\pysystemtrade\syscore\fileutils.py in get_pathname_for_package_from_list(path_as_list)
54 package_name = path_as_list[0]
55 paths_or_files = path_as_list[1:]
---> 56 d = os.path.dirname(sys.modules[package_name].__file__)
57
58 if len(paths_or_files) == 0:
KeyError: 'private'
Hi,
I have cloned pysystemtrade and am examining your (ipynb) example in examples/introduction/asimpletradingrule.ipynb. When running price = data.get_instrument_price(instrument_code)
I receive the error AttributeError: 'csvFuturesData' object has no attribute 'get_instrument_price'
When I examined csvFuturesData I see a method called get_daily_price
, which I believe should be get_instrument_price
. Is this correct?
When setting use_SR_costs to false, an error was popping up when using pandl_for_subsystem, I think because SR_cost would not be defined in that situation. I seemed to have fixed it by doing the following:
if SR_cost_per_turnover is not None:
SR_cost = self.subsystem_SR_costs(instrument_code, roundpositions)
else:
SR_cost = None
Regards,
Tony
Hi Rob,
Your book is terrific -- it's the most thorough, logical, and practical one I've ever read on systematic trading and futures. Your blog and EliteTrader posts are also very helpful, and the pysystemtrade code is a great resource. Thank you for sharing your knowledge.
After the latest update to algos.py (in commit 2a68e99), this error started to appear: "'Series' object has no attribute 'columns'." Python and YAML for reproducing the bug are attached, along with a traceback.
The error goes away if algos.py is reverted to the version in the previous commit (f899ab1).
It's possible that the error is related to my using later versions of Python and Pandas (3.4, 0.18) than the ones on which pysystemtrade was developed, but I thought I should mention it for future reference.
-- Michael
Need to handle conversion of daily / sparse intraday to daily.
Need to properly handle intraday data.
Hi Rob
I really to reallywant to get a grip with Algo trading and your book is such a valuable ressource. I am just starting with your package, however I was having difficulties with the 1st line of code (great strart I know..)
from sysdata.csvdata import csvFuturesData
File "/home/olivier/pysystemtrade/sysdata/csvdata.py", line 15, in
from sysdata.futuresdata import FuturesData
File "/home/olivier/pysystemtrade/sysdata/futuresdata.py", line 512, in
class futuresData(pd.dataFrame):
I am ample to change it once I adopt CamelStyle type of writing on DataFrame (as opposed to dataFrame). But as I am relatively new to the whole Python language I wanted to check with you if this was the right edit?
Thanks again
If I run Ch. 15 trading system I get two datasets of my interest: gross and net. You documented it well, and I find the idea very good. Then I delete "costs_analysis.csv" file and I assume that in this case there will be no costs included into the calculations. Indeed, I get the warning when I run calculations - "Cost file not found sysdata\legacycsv\costs_analysis.csv". That is great. But when I look into .net.daily.stats() and gross.daily.stats()stats, I see different numbers. Assuming that the costs are zeroed, I expected those stats to be identical. Am I wrong? I am attaching the code and output in the end of this email.
from systems.provided.futures_chapter15.basesystem import futures_system
from matplotlib.pyplot import show
**system = futures_system(log_level="on")
acc_curve_group=system.accounts.portfolio(
percentage
True
)
(
acc_curve_group
.gross.daily.stats())
(
acc_curve_group
.net.daily.stats())
system = futures_system(log_level="on")
acc_curve_group=system.accounts.portfolio(
percentage
True
)
(
acc_curve_group
.gross.daily.stats())
(
acc_curve_group
.net.daily.stats())
# coding: utf-8
import syscore
from syscore.algos import robust_vol_calc
import pandas as pd
import numpy as np
import matplotlib as plt
#import pyyaml
import scipy
from sysdata.csvdata import csvFuturesData
data=csvFuturesData()
from systems.provided.example.rules import ewmac_forecast_with_defaults as ewmac
from systems.forecasting import Rules
my_rules=Rules(dict(ewmac=ewmac))
my_rules.trading_rules()
from systems.basesystem import System
my_system=System([my_rules], data)
my_system
# We can now get forecasts:
my_system.rules.get_raw_forecast("EDOLLAR", "ewmac").tail(5)
from systems.forecasting import TradingRule
ewmac_rule=TradingRule(ewmac)
my_rules=Rules(dict(ewmac=ewmac_rule))
ewmac_rule
ewmac_8=TradingRule((ewmac, [], dict(Lfast=8, Lslow=32))) ## as a tuple (function, data, other_args) notice the empty element in the middle
ewmac_32=TradingRule(dict(function=ewmac, other_args=dict(Lfast=32, Lslow=128))) ## as a dict
my_rules=Rules(dict(ewmac8=ewmac_8, ewmac32=ewmac_32))
my_rules.trading_rules()['ewmac32']
from sysdata.configdata import Config
my_config=Config()
my_config
empty_rules=Rules()
my_config.trading_rules=dict(ewmac8=ewmac_8, ewmac32=ewmac_32)
my_system=System([empty_rules], data, my_config)
from systems.forecast_scale_cap import ForecastScaleCap
## By default we pool esimates across instruments. It's worth telling the system what instruments we want to use:
#
my_config.instruments=["EDOLLAR", "US10", "EDOLLAR", "CORN", "SP500"]
## this parameter ensures we estimate:
my_config.use_forecast_scale_estimates=True
fcs=ForecastScaleCap()
my_system = System([fcs, my_rules], data, my_config)
print(my_system.forecastScaleCap.get_forecast_scalar("EDOLLAR", "ewmac32").tail(5))
my_config.forecast_scalars=dict(ewmac8=5.3, ewmac32=2.65)
## this parameter ensures we don't estimate:
#my_config.use_forecast_scale_estimates=False
## we need a new object
fcs=ForecastScaleCap()
my_system=System([fcs, empty_rules], data, my_config)
my_system.forecastScaleCap.get_capped_forecast("EDOLLAR", "ewmac32")
from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5)
my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5)
from systems.account import Account
my_account = Account()
## let's use naive markowitz to get more interesting results...
my_config.forecast_weight_estimate=dict(method="one_period")
my_config.use_forecast_weight_estimates=True
combiner = ForecastCombine()
my_system = System([my_account, fcs, my_rules, combiner], data, my_config)
## this is a bit slow, better to know what's going on
my_system.set_logging_level("on")
print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))
my_config.forecast_weights=dict(ewmac8=0.5, ewmac32=0.5)
my_config.forecast_div_multiplier=1.1
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)
from systems.positionsizing import PositionSizing
possizer=PositionSizing()
my_config.percentage_vol_target=25
my_config.notional_trading_capital=500000
my_config.base_currency="GBP"
my_system=System([ fcs, empty_rules, combiner, possizer], data, my_config)
my_system.positionSize.get_subsystem_position("EDOLLAR").tail(5)
Hi Rob,
I recently bought your book and have read it once and will soon start reading it again as I think there are a lot of concepts which are going to be really useful as I start looking at trading my own money systematically so thank you for that.
My issue is that when I try and install the package (using python3 setup.py install
from the terminal on my mac os x machine - when in the relevant directory) I get the following error:
error: can't copy 'examples/longonly.py': doesn't exist or not a regular file
I know you're not "tech support" but I'm at a loss as to how to fix this as I tried just copying the files into the directory where all the other packages are but it seems to not work either - when I try and import a function in iPython it doesn't like it and says it doesn't exist.
If there is something I'm missing or another resource you can point to to help me out that would be great as I'm at a loss as to how to fix this now.
Thanks
I have encountered the following error message when I try running the tutorial codes:
from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)
File "c:\pysystemtrade\systems\forecast_combine.py", line 352, in get_SR_cost_for_instrument_forecast return self.parent.accounts.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
AttributeError: 'System' object has no attribute 'accounts'
There is no 'accounts' attribute defined within \systems\basesystem.py
I am finding that the correlation module is not working as expected. It's likely to be me but I have checked just using 2 instruments and get a one dimensional array with
'ans = system.portfolio.get_instrument_correlation_matrix()' whereas before it worked beautifully.
Hi, I am trying to utilise the breakout method you outlined (code below). I notice that when I use this on my own csv data, the code seems to work; however, when I try to use this on pysystemtrade data using e.g., system.rawdata.get_daily_prices("CRUDE_W")
it yields error messages. This means that I cannot incorporate the breakout into pysystemtrade. I believe it may be related to the index associated with system.rawdata.get_daily_prices("CRUDE_W")
. I have checked and both my own csv data and system.rawdata.get_daily_prices("CRUDE_W")
yield pandas dataframes with Datetime indexes. Do you have any advice on how to adapt the breakout code so that it works with pysystemtrade and pysystemtrade data?
Here is the code:
def breakout(x, ws):
max_x = pd.rolling_max(x, ws, min_periods=min(len(x), int(ws/2)))
min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
sig = pd.Series(sig, index=x.index)
sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))
return sig
def breakout_one_row(idx, x, max_x, min_x):
r_px = x[idx]
r_min = min_x[idx]
r_max = max_x[idx]
return 40*(r_px - np.mean([r_min, r_max]))/(r_max - r_min)
Here is the error using pysystemtrade:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
C:\Anaconda3\lib\site-packages\pandas\indexes\base.py in get_loc(self, key, method, tolerance)
1875 try:
-> 1876 return self._engine.get_loc(key)
1877 except KeyError:
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4027)()
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:3891)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12408)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12359)()
KeyError: 0
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-69-079ba8216ec5> in <module>()
----> 1 breakout(pd.DataFrame(system.rawdata.get_daily_prices("CRUDE_W")), 320)
<ipython-input-27-61d55c59b2f4> in breakout(x, ws)
3 min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
4
----> 5 sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
6 sig = pd.Series(sig, index=x.index)
7 sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))
<ipython-input-27-61d55c59b2f4> in <listcomp>(.0)
3 min_x = pd.rolling_min(x, ws, min_periods=min(len(x), int(ws/2)))
4
----> 5 sig = [breakout_one_row(idx, x, max_x, min_x) for idx in range(len(x.index))]
6 sig = pd.Series(sig, index=x.index)
7 sig = pd.ewma(sig, span = int(ws/4.0), min_periods=int(ws/8.0))
<ipython-input-27-61d55c59b2f4> in breakout_one_row(idx, x, max_x, min_x)
10
11 def breakout_one_row(idx, x, max_x, min_x):
---> 12 r_px = x[idx]
13 r_min = min_x[idx]
14 r_max = max_x[idx]
C:\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
1990 return self._getitem_multilevel(key)
1991 else:
-> 1992 return self._getitem_column(key)
1993
1994 def _getitem_column(self, key):
C:\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_column(self, key)
1997 # get column
1998 if self.columns.is_unique:
-> 1999 return self._get_item_cache(key)
2000
2001 # duplicate columns & possible reduce dimensionality
C:\Anaconda3\lib\site-packages\pandas\core\generic.py in _get_item_cache(self, item)
1343 res = cache.get(item)
1344 if res is None:
-> 1345 values = self._data.get(item)
1346 res = self._box_item_values(item, values)
1347 cache[item] = res
C:\Anaconda3\lib\site-packages\pandas\core\internals.py in get(self, item, fastpath)
3223
3224 if not isnull(item):
-> 3225 loc = self.items.get_loc(item)
3226 else:
3227 indexer = np.arange(len(self.items))[isnull(self.items)]
C:\Anaconda3\lib\site-packages\pandas\indexes\base.py in get_loc(self, key, method, tolerance)
1876 return self._engine.get_loc(key)
1877 except KeyError:
-> 1878 return self._engine.get_loc(self._maybe_cast_indexer(key))
1879
1880 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:4027)()
pandas\index.pyx in pandas.index.IndexEngine.get_loc (pandas\index.c:3891)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12408)()
pandas\hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas\hashtable.c:12359)()
KeyError: 0
Sorry for another issue, but just wanted to bring up that every time I would try system.accounts.portfolio().stats(), python would crash. After reviewing the output to just up to the crash point, I isolated the issue to the drawdown function in pdutils.py. For some reason, changing the "window" to six digits did the trick and it doesn't crash anymore.
maxx = pd.rolling_max(x, 999999, min_periods=1) #six digits instead of 8 for python crashing error
If it matters, I'm running Python 3.4.4 on Windows 7.
Thank you,
Tony
Robert, running the script below gives me an error. Full trace is attached. Thanks. -Zhen
from systems.account import Account
my_account = Account()
## let's use naive markowitz to get more interesting results...
my_config.forecast_weight_estimate=dict(method="one_period")
my_config.use_forecast_weight_estimates=True
combiner = ForecastCombine()
my_system = System([my_account, fcs, my_rules, combiner], data, my_config)
## this is a bit slow, better to know what's going on
my_system.set_logging_level("on")
print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))
AttributeError Traceback (most recent call last)
in ()
13
14 print(my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5))
---> 15 print(my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5))
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_diversification_multiplier(self, instrument_code)
764 forecast_div_multiplier = self.parent.calc_or_cache(
765 'get_forecast_diversification_multiplier', instrument_code, _get_forecast_div_multiplier,
--> 766 self)
767 return forecast_div_multiplier
768
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_div_multiplier(system, instrument_code, this_stage)
752
753 correlation_list_object = this_stage.get_forecast_correlation_matrices(
--> 754 instrument_code)
755 weight_df = this_stage.get_forecast_weights(instrument_code)
756
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_correlation_matrices(self, instrument_code)
705 'get_forecast_correlation_matrices', instrument_code_ref, label,
706 _get_forecast_correlation_matrices,
--> 707 self, codes_to_use, corr_func, **corr_params)
708
709 return forecast_corr_list
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache_nested(self, itemname, instrument_code, keyname, func, this_stage, *args, **kwargs)
684 this_stage,
685 *args,
--> 686 **kwargs)
687 self.set_item_in_cache(value, cache_ref, instrument_code, keyname)
688
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_correlation_matrices(system, NotUsed1, NotUsed2, this_stage, codes_to_use, corr_func, **corr_params)
668
669 return corr_func(forecast_data, log=self.log.setup(
--> 670 call="correlation"), **corr_params)
671
672 # Get some useful stuff from the config
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\syscore\correlations.py in init(self, data, log, frequency, date_method, rollyears, dict_group, boring_offdiag, cleaning, **kwargs)
278 column_names = list(data.columns)
279
--> 280 data = frequency.resample.last()
281
282 # Generate time periods
AttributeError: 'str' object has no attribute 'resample'
The function get_instrument_value_vol
generate a time series that contains the weekend dates.
That happen because the get_instrument_currency_vol
is multiply with the get_fx_rate
, this create time series with weekend date.
this is propagate up to:
get_instrument_value_vol
get_volatility_scalar
get_subsystem_position
get_notional_position
get_buffers_for_position
get_buffers_for_position
is called within get_buffered_position this generate pos_buffers
time series that contains weekend with NaN values.
This time series then pass into apply_buffer
that call ffill()
on pos_buffers
this generate buffered positions the contains positions on the weekend.
I don’t know it that’s a problem or not in the end stats and curve, but it’s just something I notice.
Some more info:
I'm running the simplesystemconfig.yaml with base currency USD, I don’t see the same problem when base currency is GBP
Hello,
In my pursuit to try to figure out why the carry data affects PandL/Stats(), I think I may have uncovered a bug (please forgive me if I have not, still learning Python). The code for daily_denominator_price in rawdata.py is essentially:
def _daily_denominator_returns(system, instrument_code, this_stage):
dem_returns = this_stage.get_daily_prices(instrument_code)
return dem_returns
To me, this just means getting the regular daily prices (although I note you state "This won't always be the same as the normal 'price'"). However, when I run the code it seems to be pulling the price from the carry data file:
>>> mysystem.rawdata.daily_denominator_price("CORN").head(5)
1981-09-25 326.75
1981-09-28 327.75
1981-09-29 332.75
1981-09-30 334.75
1981-10-01 334.00
Freq: B, Name: PRICE, dtype: float64
>>> mysystem.rawdata.get_daily_prices("CORN").head(5)
1981-09-25 693.50
1981-09-28 694.50
1981-09-29 699.50
1981-09-30 701.50
1981-10-01 700.75
Freq: B, Name: price, dtype: float64
For good measure, here is the output from the data stage:
>>> mysystem.data.daily_prices("CORN").head(5)
1981-09-25 693.50
1981-09-28 694.50
1981-09-29 699.50
1981-09-30 701.50
1981-10-01 700.75
Freq: B, Name: price, dtype: float64
Am I missing something (probably), or is this a bug? And if you could point me in the right direction as to my original pursuit (why carry would affect PandL/Stats() if I'm not using any carry trading rules), I would greatly appreciate it.
Thank you,
Tony
Hi Rob, while running the simplesystem.py, it got interrupted halfway and threw out this error above. I google to search for it and it seems that the data set is not in the correct form? Am I heading in the right direction to resolve this? Thanks.
Ax
Dear Rob, Dear colleagues,
Thank you very much such a good library. I have some questions on its usage.
I have some 10 trading rules, that work well on 15-minute data bars time frame. I see that the library works with daily data. I am wondering if the library has some feature that would allow me to build my portfolio of 20 cheap instruments with 10 trading rules using 15 minute pricing data bars. If not, what would be your recommendation on how I could add this feature myself?
In my case above, I have noticed that the best performance I am getting not when I trade to update my positions every 15 minutes (even including position inertia), but at some fixed time frame of larger scale: e.g. every 4 hours. That means that I trade every 4 hours, and when it is time to trade, I take the latest position info available (16 of them is produced per every 4 hour interval). Wanted to ask your thoughts if this is possible to have with current implementation, and if not, what would be your recommendation on how I could add this feature myself while working with your library?
And the last question is about commissions. Many brokers, including IB, have non-flat commissions for some of the markets that depend on monthly volumes of the traded instruments and other factors. As far as I understand, currently I can only specify some flat numbers (estimates) in the library configuration file. I have checked that with one of my systematic portfolios if I specify the flat numbers, then the resulting P/L is effectively eaten out by the commissions. If I model the commissions more precisely and separately (after the gross curves are produced), I see that commissions eat out only 12% of the performance. The difference is remarkable. But I believe it is not be the right approach because this information about commissions was not used by the library itself during its calculations. Wanted to ask you what would be your recommendation on how I could model more precise non-linear commissions (to stay in style of your library design)?
I would really appreciate your thoughts on those three points.
Best regards,
Max
Hi,
I just synced my version of pysystemtrade with the latest version in order to test on the latest data, and experiment with the added trading rules, but now unfortunately I receive an error. It seems that systems.basesystem has changed.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-10-48c5ec1b28eb> in <module>()
----> 1 from systems.provided.futures_chapter15.basesystem import *
2 from syscore.pdutils import align_to_joint, uniquets, divide_df_single_column
3 from syscore.dateutils import generate_fitting_dates
4 from syscore.algos import robust_vol_calc
5
H:\pysystemtrade-master\systems\provided\futures_chapter15\basesystem.py in <module>()
9
10 from systems.forecasting import Rules
---> 11 from systems.basesystem import System
12 from systems.forecast_combine import ForecastCombine
13 from systems.forecast_scale_cap import ForecastScaleCap
H:\pysystemtrade-master\systems\basesystem.py in <module>()
9
10 from systems.forecasting import Rules
---> 11 from systems.basesystem import System
12 from systems.forecast_combine import ForecastCombine
13 from systems.forecast_scale_cap import ForecastScaleCap
ImportError: cannot import name 'System'
Hi Rob,
tested your new code with the breakout example and the standard estimated system example using your defaults. In both cases the resulting instrument weights seem to be incorrect.
Code:
from systems.provided.futures_chapter15.estimatedsystem import futures_system
from sysdata.configdata import Config
my_config=Config("examples.breakout.breakoutfuturesestimateconfig.yaml")
system=futures_system(config=my_config)
system.set_logging_level("on")
In your previous version a similar issue occurred when increasing the notional trading capital (above $1M). Now it seems to occur independent of the trading capital chosen. I tested with your default and 100000.
Regards
robooeko
Thank you for doing this project. I get the error when I test this example in the introduction,
from systems.forecast_combine import ForecastCombine
combiner=ForecastCombine()
my_system=System([fcs, empty_rules, combiner], data, my_config)
my_system.combForecast.get_forecast_weights("EDOLLAR").tail(5)
my_system.combForecast.get_forecast_diversification_multiplier("EDOLLAR").tail(5)
Please help.
Hi Rob,
Would you be so kind as to update the CSV data? I would like to simulate performance over Brexit.
Additionally, are you able to provide and script for back-adjusting the futures contracts? I have attempted to create one so I can update the data myself. It is based on the traded price for the new future and subtracting the difference between the traded price of the new future from the price of the old future at the time of trade across the historical data series, but this is yielding some strange results for me. I assume you have something similar you must used when rolling contracts and back-adjusting the price series?
Thanks
under "small account size" and for the above referenced file:
from systems.provided.futures_chapter15.estimatedsystem import PortfoliosEstimated
ImportError: cannot import name 'PortfoliosEstimated'
Hi there, when I run examples/introduction/asimpletradingrule.py, the graphs are fine, but I also get,
File "../..\syscore\accounting.py", line 932, in resolve_capital
daily_risk_capital = base_capital * ann_risk_target / ROOT_BDAYS_INYEAR
UnboundLocalError: local variable 'base_capital' referenced before assignment
When I pass capital=10000 to accountCurve, I received a segmentation fault (at least on Windows).
After some tries, I was able to narrow it down to accountCurveSingleElementOneFreq class, to stats(self) function; in the loop while the stats are being calculated, the avg_drawdown stat, time_in_drawdown, calmar, avg_return_to_drawdown are the stats that are causing problems. If these are skipped, then I get the output fine, it says 0.5084 for Sharpe which matches the output in the doc.
I have the latest code.
Hi Rob,
with your new code the estimation or cleaning of instrument weights behaves different when using non pooled scalars. The code calculates weights only when scalar values exist for all instruments e.g. US10 and SP500. Before your code changes all weight before 1998 went to US10 as no values existed for SP500. Now no weight is given at all (neither to US10 nor SP500) and weights are calculated when both instruments have scalar values.
Regards
Robooeko
I thought I'd let you know that using concurrent.futures ProcessPoolExecutor sped up bootstrap performance by about 10x. I prep all the samples in advance and then bootstrap them in parallel.
If you're interested in code, it's here https://gist.github.com/chrism2671/256ec51515b6849dd695f15fb28caecf (start reading at def bootstrap() )
When I try to load the asimpletradingrule.py , it gives me the following error.
FileNotFoundError Traceback (most recent call last)
C:\Users\jngri\Documents\Python Scripts\pysystemtrade\examples\introduction\asimpletradingrule.py in ()
62
63 import pandas as pd
---> 64 from syscore.algos import robust_vol_calc
65
66
C:\Users\jngri\Anaconda3\lib\site-packages\syscore\algos.py in ()
11
12 from syscore.genutils import str2Bool
---> 13 from systems.defaults import system_defaults
14
15 LARGE_NUMBER_OF_DAYS = 250 * 100 * 100
C:\Users\jngri\Anaconda3\lib\site-packages\systems\defaults.py in ()
24 return default_dict
25
---> 26 system_defaults = get_system_defaults()
27
28 if name == 'main':
C:\Users\jngri\Anaconda3\lib\site-packages\systems\defaults.py in get_system_defaults()
19 """
20 default_file = get_filename_for_package(DEFAULT_FILENAME)
---> 21 with open(default_file) as file_to_parse:
22 default_dict = yaml.load(file_to_parse)
23
FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\jngri\Anaconda3\lib\site-packages\systems\provided\defaults.yaml'
I try to run other notebooks like prebakedsystem and I get this..
ModuleNotFoundError Traceback (most recent call last)
C:\Users\jngri\Documents\Python Scripts\pysystemtrade\examples\introduction\prebakedsystems.py in ()
----> 1 from systems.provided.example.simplesystem import simplesystem
2
3 my_system = simplesystem(log_level="on")
4 print(my_system)
5 print(my_system.portfolio.get_notional_position("EDOLLAR").tail(5))
ModuleNotFoundError: No module named 'systems.provided'
I am on a windows 10 machine using anaconda. I added the environment path for for pysystemtrade and it seems to be having trouble findings things. ALL Required packages are installed Just an FYI- I am teaching myself python, so I could quite possibly be down something wrong.
Hi,
I just downloaded the latest version of Anaconda/python and have received the following error in pysystemtrade. Any thoughts on debugging this?
ValueError Traceback (most recent call last)
in ()
16 = System([Account(), Portfolios(), PositionSizing(), FuturesRawData(), ForecastCombine(), ForecastScaleCap(), Rules()], csvFuturesData(), config_)
17 .set_logging_level("on")
---> 18 _portfolio = File.accounts.portfolio(delayfill=False)
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in portfolio(self, percentage, delayfill, roundpositions)
1219 port_pandl = self.parent.calc_or_cache(
1220 itemname, ALL_KEYNAME, _portfolio, self, percentage, delayfill,
-> 1221 roundpositions)
1222
1223 return port_pandl
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _portfolio(system, not_used, this_stage, percentage, delayfill, roundpositions)
1208 percentage=percentage,
1209 delayfill=delayfill,
-> 1210 roundpositions=roundpositions) for instrument_code in instruments]
1211
1212 port_pandl = accountCurveGroup(port_pandl, instruments)
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in (.0)
1208 percentage=percentage,
1209 delayfill=delayfill,
-> 1210 roundpositions=roundpositions) for instrument_code in instruments]
1211
1212 port_pandl = accountCurveGroup(port_pandl, instruments)
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in pandl_for_instrument(self, instrument_code, percentage, delayfill, roundpositions)
884 TorF(percentage), TorF(delayfill), TorF(roundpositions))
885 instr_pandl = self.parent.calc_or_cache(
--> 886 itemname, instrument_code, _pandl_for_instrument, self, percentage, delayfill, roundpositions)
887
888 return instr_pandl
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _pandl_for_instrument(system, instrument_code, this_stage, percentage, delayfill, roundpositions)
844
845 price = this_stage.get_daily_price(instrument_code)
--> 846 positions = this_stage.get_buffered_position(instrument_code, roundpositions = roundpositions)
847 fx = this_stage.get_fx_rate(instrument_code)
848 value_of_price_point = this_stage.get_value_of_price_move(
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in get_buffered_position(self, instrument_code, roundpositions)
775
776 buffered_position = self.parent.calc_or_cache(
--> 777 itemname, instrument_code, _get_buffered_position, self, roundpositions)
778
779 return buffered_position
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in _get_buffered_position(system, instrument_code, this_stage, roundpositions)
761
762 this_stage.log.msg("Calculating buffered positions")
--> 763 optimal_position=this_stage.get_notional_position(instrument_code)
764 pos_buffers=this_stage.get_buffers_for_position(instrument_code)
765 trade_to_edge=system.config.buffer_trade_to_edge
C:\Users\me\Google Drive\File\pysystemtrade\systems\account.py in get_notional_position(self, instrument_code)
276 >>>
277 """
--> 278 return self.parent.portfolio.get_notional_position(instrument_code)
279
280
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_notional_position(self, instrument_code)
341
342 notional_position = self.parent.calc_or_cache(
--> 343 "get_notional_position", instrument_code, _get_notional_position, self)
344 return notional_position
345
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_notional_position(system, instrument_code, this_stage)
321 instrument_code=instrument_code)
322
--> 323 idm = this_stage.get_instrument_diversification_multiplier()
324 instr_weights = this_stage.get_instrument_weights()
325 subsys_position = this_stage.get_subsystem_position(
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_instrument_diversification_multiplier(self)
641 instrument_div_multiplier = self.parent.calc_or_cache(
642 'get_instrument_diversification_multiplier', ALL_KEYNAME, _get_instrument_div_multiplier,
--> 643 self)
644 return instrument_div_multiplier
645
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_instrument_div_multiplier(system, NotUsed, this_stage)
630 idm_func=resolve_function(div_mult_params.pop("func"))
631
--> 632 correlation_list_object=this_stage.get_instrument_correlation_matrix()
633 weight_df=this_stage.get_instrument_weights()
634
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in get_instrument_correlation_matrix(self)
593 'get_instrument_correlation_matrix', ALL_KEYNAME,
594 _get_instrument_correlation_matrix,
--> 595 self, corr_func, **corr_params)
596
597 return forecast_corr_list
C:\Users\me\Google Drive\File\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
628
629 if value is None:
--> 630 value = func(self, instrument_code, this_stage, *args, **kwargs)
631 self.set_item_in_cache(value, cache_ref, instrument_code)
632
C:\Users\me\Google Drive\File\pysystemtrade\systems\portfolio.py in _get_instrument_correlation_matrix(system, NotUsed, this_stage, corr_func, **corr_params)
576 pandl=pandl.cumsum().resample(frequency).diff()
577
--> 578 return corr_func(pandl, log=this_stage.log.setup(call="correlation"), **corr_params)
579
580 ## Get some useful stuff from the config
C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in init(self, data, log, frequency, date_method, rollyears, dict_group, boring_offdiag, cleaning, **kwargs)
306
307 # means we can use earlier correlations with sensible values
--> 308 corrmat=clean_correlation(corrmat, corr_with_no_data, must_haves)
309
310 corr_list.append(corrmat)
C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in clean_correlation(corrmat, corr_with_no_data, must_haves)
99 size_range=range(corrmat.shape[0])
100
--> 101 avgcorr=get_avg_corr(corrmat)
102
103 def _good_correlation(i,j,corrmat, avgcorr, must_haves, corr_with_no_data):
C:\Users\me\Google Drive\File\pysystemtrade\syscore\correlations.py in get_avg_corr(sigma)
24 """
25 new_sigma=copy(sigma)
---> 26 np.fill_diagonal(new_sigma,np.nan)
27 if np.all(np.isnan(new_sigma)):
28 return np.nan
C:\ProgramData\Anaconda3\lib\site-packages\numpy\lib\index_tricks.py in fill_diagonal(a, val, wrap)
757 """
758 if a.ndim < 2:
--> 759 raise ValueError("array must be at least 2-d")
760 end = None
761 if a.ndim == 2:
ValueError: array must be at least 2-d`
Hello Rob,
New version of the method get_raw_cost_data in csvdata returns objects of different type (either dict or tuple) depending on cost data availability. This leads to a runtime crash later in account.py _get_SR_cost.
File: sysdata\csvdata.py
Method: get_raw_cost_data
Line: 117, 123
Still returns tuples, not dicts.
Best regards,
Pavel
is ForecastWithThreshold (examples/smallaccountsize) still a valid way to achieve thresholding? is the API still valid?
Hi, I am trying to examine the equity curves of subsystems as per your post "Instrument Weights"
referring to the following code:
import pandas as pd
instrument_codes=system.get_instrument_list()
pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
for code in instrument_codes]
pandl=pd.concat(pandl_subsystems, axis=1)
pandl.columns=instrument_codes
pandl=pandl.cumsum().plot()
show()
This yields the following error: AttributeError: 'BlockManager' object has no attribute 'index'
Here is the error message:
AttributeError Traceback (most recent call last)
<ipython-input-8-f5d38cdd5c65> in <module>()
2 pandl_subsystems=[system.accounts.pandl_for_subsystem(code, percentage=True)
3 for code in instrument_codes]
----> 4 pandl=pd.concat(pandl_subsystems, axis=1)
C:\Anaconda3\lib\site-packages\pandas\tools\merge.py in concat(objs, axis, join, join_axes, ignore_index, keys, levels, names, verify_integrity, copy)
833 verify_integrity=verify_integrity,
834 copy=copy)
--> 835 return op.get_result()
836
837
C:\Anaconda3\lib\site-packages\pandas\tools\merge.py in get_result(self)
1027 new_data._consolidate_inplace()
1028
-> 1029 return (self.objs[0]._from_axes(new_data, self.new_axes)
1030 .__finalize__(self, method='concat'))
1031
C:\Anaconda3\lib\site-packages\pandas\core\generic.py in _from_axes(cls, data, axes, **kwargs)
294 # for construction from BlockManager
295 if isinstance(data, BlockManager):
--> 296 return cls(data, **kwargs)
297 else:
298 if cls._AXIS_REVERSED:
C:\Anaconda3\pysystemtrade\syscore\accounting.py in __init__(self, price, percentage, cash_costs, SR_cost, capital, ann_risk_target, weighting, apply_weight_to_costs_only, **kwargs)
709 """
710
--> 711 returns_data=pandl_with_data(price, capital=capital, ann_risk_target=ann_risk_target, **kwargs)
712
713 (cum_trades, trades, instr_ccy_returns,
C:\Anaconda3\pysystemtrade\syscore\accounting.py in pandl_with_data(price, trades, marktomarket, positions, delayfill, roundpositions, get_daily_returns_volatility, forecast, fx, capital, ann_risk_target, value_of_price_point)
139 if fx is None:
140 # assume it's 1.0
--> 141 fx = pd.Series([1.0] * len(price.index),
142 index=price.index).to_frame("fx")
143
AttributeError: 'BlockManager' object has no attribute 'index'
Is there an error within pysystemtrade or am I missing something?
Hello Rob,
I have noticed that this project has ups and downs in terms of test pass rate and test coverage. Pass rate is often not 100%. Currently 16 tests are red. Do you have any plans to improve software quality of the project? e.g. higher coverage, CI (most importantly), regression tests, etc. I am not a python expert, but I am sure those things will be valued by end-users.
My environment: Win10, Python 3.5, Pandas 0.18.0, Matplotlib 1.5.1, pyyaml 3.11, numpy 1.10.4, scipy 0.17.0.
Red tests:
Best regards,
Pavel
Why do the returns and metrics in pysystemtrade change so markedly if you change the base currency? Shouldn't the returns be similar for various currencies given they are mostly USD instruments and most of the account stays in cash? The vol is often much lower than target vol when using non-USD currency balances. How do you grapple with this?
Traceback (most recent call last):
File "experiment2.py", line 6, in
system.accounts.portfolio().pandl_for_instrument("US10").percent().stats() #
File "C:\Users\xxxxx\AppData\Local\Continuum\Anaconda2\envs\python3\lib\site-
packages\pandas\core\generic.py", line 2672, in getattr
return object.getattribute(self, name)
AttributeError: 'accountCurveGroup' object has no attribute 'pandl_for_instrument'
One possibility is it might be my conda environment (my Python3 is conda installed, my native installation is Python 2.7) causing an error - any one facing a similar issue?
Hi Rob - I am walking through your introduction.md document. Intermittently (when I run from scratch) when I run line 134-136:-
from syscore.accounting import accountCurve
account = accountCurve(price, forecast=ewmac, percentage=True)
account.percent().stats()
I get an error:-
pysystemtrade/syscore/accounting.py in resolve_capital(ts_to_scale_to, capital, ann_risk_target)
931
932 ## might be a float or a Series, depending on capital
--> 933 daily_risk_capital = base_capital * ann_risk_target / ROOT_BDAYS_INYEAR
934
935 if type(base_capital) is float or type(base_capital) is int:
UnboundLocalError: local variable 'base_capital' referenced before assignment
If I make a change to the resolve_capital function within accounting.py as follows, that seems to fix it.
if capital is None:
capital=DEFAULT_CAPITAL
base_capital = copy(capital) #add this line
else:
base_capital = copy(capital)
Full disclosure - not that confident that this is the cause of my error - rusty in programming and quite new to python.
Hi Rob, I am trying to compute instrument weights for two instruments, US20 and SP500, with two rules on them, ewmac_8 and ewmac_32 (not estimated, equal weights).
from systems.provided.example.rules import ewmac_forecast_with_defaults as ewmac
from sysdata.configdata import Config
from systems.account import Account
from systems.forecast_combine import ForecastCombine
from systems.forecast_scale_cap import ForecastScaleCap
from systems.basesystem import System
from sysdata.csvdata import csvFuturesData
from systems.forecasting import Rules
from systems.forecasting import TradingRule
from systems.portfolio import PortfoliosEstimated
from systems.positionsizing import PositionSizing
data = csvFuturesData()
my_config = Config()
my_config.instruments=["US20", "SP500"]
ewmac_8 = TradingRule((ewmac, [], dict(Lfast=8, Lslow=32)))
ewmac_32 = TradingRule(dict(function=ewmac, other_args=dict(Lfast=8, Lslow=32)))
my_rules = Rules(dict(ewmac8=ewmac_8, ewmac32=ewmac_32))
my_system = System([Account(), PortfoliosEstimated(), PositionSizing(), ForecastScaleCap(), my_rules, ForecastCombine()], data, my_config)
my_system.config.forecast_weight_estimate['method']="equal_weights"
my_system.config.instrument_weight_estimate['method']="bootstrap"
my_system.config.instrument_weight_estimate["monte_runs"]=10
my_system.config.instrument_weight_estimate["bootstrap_length"]=250
print (my_system.portfolio.get_instrument_weights())
print(my_system.portfolio.get_instrument_diversification_multiplier())
When I run this system, I get
Cost multiplier of 0.0 will be ignored as equalising SR in optimisation (equalise_SR=True)
Zero cost multiplier and not applying cost weightings - so costs won't be used at all
Cost multiplier of %2.f is less than one and not applying cost weightings - effect of costs may be underestimated
WARNING: No forecast weights - using equal weights of 0.5000 over all 2 trading rules in system
WARNING: No forecast weights - using equal weights of 0.5000 over all 2 trading rules in system
....
...
Anaconda2\envs\py3k\lib\site-packages\numpy\lib\nanfunctions.py:675: RuntimeWarning: Mean of empty slice
warnings.warn("Mean of empty slice", RuntimeWarning)
Anaconda2\envs\py3k\lib\site-packages\numpy\lib\nanfunctions.py:1136: RuntimeWarning: Degrees of freedom <= 0 for slice.
warnings.warn("Degrees of freedom <= 0 for slice.", RuntimeWarning)
...
...
This output goes on for a while, and I ran it for a long time, it did not finish. Am I doing something wrong during the configuration, or did I stumble upon a bug?
Thanks,
When trying to run the 'optimisation.py':
In [86]: print(system.accounts.portfolio().stats())
Calculating pandl for portfolio
Calculating instrument div. multiplier
Calculating instrument correlations
C:\Anaconda3\lib\site-packages\systems\portfolio.py:665: FutureWarning:
.resample() is now a deferred operation
You called diff(...) on this deferred object which materialized it into a dataframe
by implicitly taking the mean. Use .resample(...).mean() instead
pandl = pandl.cumsum().resample(frequency).diff()
Correlation estimate
C:\Anaconda3\lib\site-packages\syscore\correlations.py:181: RuntimeWarning: invalid value encountered in less
corrmat[corrmat < 0] = 0.0
I cannot figure out how to solve this when running the script below. The full trace is attached. Thanks.
my_config.forecast_weights=dict(ewmac8=0.5, ewmac32=0.5) my_config.forecast_div_multiplier=1.1 my_system=System([fcs, empty_rules, combiner], data, my_config) my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)
AttributeError Traceback (most recent call last)
in ()
2 my_config.forecast_div_multiplier=1.1
3 my_system=System([fcs, empty_rules, combiner], data, my_config)
----> 4 my_system.combForecast.get_combined_forecast("EDOLLAR").tail(5)
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_combined_forecast(self, instrument_code)
541
542 combined_forecast = self.parent.calc_or_cache(
--> 543 'get_combined_forecast', instrument_code, _get_combined_forecast, self)
544 return combined_forecast
545
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_combined_forecast(system, instrument_code, this_stage)
513 instrument_code=instrument_code)
514
--> 515 forecast_weights = this_stage.get_forecast_weights(instrument_code)
516 rule_variation_list = list(forecast_weights.columns)
517
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_forecast_weights(self, instrument_code)
392
393 forecast_weights = self.parent.calc_or_cache(
--> 394 'get_forecast_weights', instrument_code, _get_forecast_weights, self)
395 return forecast_weights
396
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_forecast_weights(system, instrument_code, this_stage)
375
376 forecast_weights = this_stage.get_raw_forecast_weights(
--> 377 instrument_code)
378 rule_variation_list = list(forecast_weights.columns)
379 forecasts = this_stage.get_all_forecasts(
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_raw_forecast_weights(self, instrument_code)
1132 'get_raw_forecast_weights', instrument_code,
1133 _get_raw_forecast_weights,
-> 1134 self)
1135
1136 return raw_forecast_weights
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _get_raw_forecast_weights(system, instrument_code, this_stage)
1126
1127 return this_stage.calculation_of_raw_forecast_weights(
-> 1128 instrument_code).weights
1129
1130 ##
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in calculation_of_raw_forecast_weights(self, instrument_code)
894 # could still be using pooled returns
895 return self.calculation_of_raw_forecast_weights_for_instrument(
--> 896 instrument_code)
897
898 def calculation_of_raw_forecast_weights_for_instrument(
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in calculation_of_raw_forecast_weights_for_instrument(self, instrument_code)
976 if pooling_returns:
977 # find set of instruments with same trading rules as I have
--> 978 codes_to_use = self.has_same_cheap_rules_as_code(instrument_code)
979 else:
980 codes_to_use = [instrument_code]
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in has_same_cheap_rules_as_code(self, instrument_code)
861 """
862
--> 863 my_rules = self.apply_cost_weighting(instrument_code)
864 instrument_list = self.parent.get_instrument_list()
865
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in apply_cost_weighting(self, instrument_code)
846 'apply_cost_weighting', instrument_code,
847 _apply_cost_weighting,
--> 848 self, ceiling_cost_SR)
849
850 return cheap_rules
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\basesystem.py in calc_or_cache(self, itemname, instrument_code, func, this_stage, flags, *args, **kwargs)
633
634 if value is None:
--> 635 value = func(self, instrument_code, this_stage, *args, **kwargs)
636 self.set_item_in_cache(value, cache_ref, instrument_code)
637
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in _apply_cost_weighting(system, instrument_code, this_stage, ceiling_cost_SR)
824 rule_list = this_stage.get_trading_rule_list(instrument_code)
825 SR_cost_list = [this_stage.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
--> 826 for rule_variation_name in rule_list]
827
828 cheap_rule_list = [rule_name for (rule_name, rule_cost) in zip(rule_list, SR_cost_list)
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in (.0)
824 rule_list = this_stage.get_trading_rule_list(instrument_code)
825 SR_cost_list = [this_stage.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
--> 826 for rule_variation_name in rule_list]
827
828 cheap_rule_list = [rule_name for (rule_name, rule_cost) in zip(rule_list, SR_cost_list)
C:\Users\mih\WinPython-64bit-3.5.2.3Qt5\notebooks\pysystemtrade\systems\forecast_combine.py in get_SR_cost_for_instrument_forecast(self, instrument_code, rule_variation_name)
804 """
805
--> 806 return self.parent.accounts.get_SR_cost_for_instrument_forecast(instrument_code, rule_variation_name)
807
808 def apply_cost_weighting(self, instrument_code):
AttributeError: 'System' object has no attribute 'accounts'
Hi Rob,
I've bought your book and started looking at the code, working through the examples. Since I normally use Python 2.7 I set up a fresh environment with the latest... 3.5 and everything else. In the introduction there is an implementation of the ewma forecast which accesses a pandas function .ewma(...). It turns out that in pandas 0.18 they have changed the functionality so that you have to create an .ewm object then calculate its .mean(). See --> http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.ewm.html.
When I downgraded to 0.17.1 this problem went away, so you might want to flag that as a requirement. Currently pysystemtrade/requirements.txt states pandas >= 0.17.0. (One other note - I reset to 3.4 to dup your environment as best I could).
Getting another error also but I'll comment on that in the file where the error is occurring.
Best
Davis
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.