Black-Litterman model of expected returns about pyportfolioopt HOT 15 CLOSED

Hi, great work on this package.
I was just curious about the progress made on implementing the BL model? If either of you are working on a branch/PR?

Also about the views, do we leave the creation of the view matrix up to the user, or is there any plans to create some helper functions to do this?

from pyportfolioopt.

Hi @Busteren, essentially the code is just as given above.

Your question about the views is quite a tough one and is the main reason why I haven't implemented BL yet (apart from being generally busy).

It is not trivial for users to enter views - they have to know the precise specification and interpretation of the each row. But likewise, views are so flexible that it's hard to define useful helper methods.

I suppose one of the possible helper functions is to map absolute views to a views matrix (I've implemented this before). But even then, there'd have to be pretty comprehensive docs to make it usable

from pyportfolioopt.

@schneiderfelipe

I've just pushed the finished Black-Litterman code. If you have a moment, could you please take a look? There are a few examples in tests/test_black_litterman.py which demonstrate its usage:

test_bl_returns_no_prior()
test_bl_relative_views()
test_bl_returns_all_views()

from pyportfolioopt.

@schneiderfelipe Cheers for the review! Always very useful to have a second pair of eyes.

I believe the formula you're using for implied prior returns in market_implied_prior_returns is actually excess returns

I'll look through some of the papers to get to the bottom of this.

I would suggest naming absolute_views something like absolute_Q or abs_Q, or calling Q views.

This is a fair comment, but for now, I am inclined to leave it because I want to make the point that Q is the views vector (a mathematical object that BL requires) while absolute_views is a dictionary of views supplied by the user and is thus conceptually very different to Q. My feeling is that if I were to call it absolute_Q, it could cause some confusion as the user might expect it to have the same type as Q.

I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

Good catch! Will do this.

I believe inversion can be avoided in bl_weights

As always, nice spot with the linear systems. I'll run some tests to make sure it works as expected.

I believe portfolio_performance should use the result from bl_cov

Agreed, that makes more sense!

from pyportfolioopt.

You were right about the excess returns thing. It's a simple fix, just adding back the risk_free_rate. I'm leaving it to the user to ensure that time periods match. It fits the rest of the API because covariance matrices are invariant to linear shifts and the computation of the risk aversion coefficient uses the excess return.

I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

I have added a comment to the docs, but I also added a mechanism to support a custom omega_inv matrix. During my summer internship, I was building BL portfolios and non-diagonal
uncertainty matrices were possible. So I'm exposing it as an attribute such that advanced users can write:

bl.omega_inv = np.linalg.inv(my_custom_uncertainty_matrix)
w = bl.bl_returns()

I want to add a few more bits and pieces to PyPortfolioOpt before the v0.5.0 release (unrelated to BL), so I will hopefully be in a position to fully merge by early next week.

from pyportfolioopt.

@schneiderfelipe

Hi, hope everything is going well. I actually implemented BL separately for myself, and it's not too difficult to do. I suppose I am more concerned about user-entered parameters. For example, what is a good way to allow users to specify their views? Should the function accept the BL inputs as matrices? or should we help users to construct them?

In any case, this is the implementation of the formula itself:

        tau_sigma_inv = np.linalg.inv(tau * sigma)
        omega_inv = np.diag(1 / np.diag(omega))  # efficient inverse of diagonal matrix
        p_omega_inv = P.T.dot(omega_inv)
        term1 = np.linalg.inv(tau_sigma_inv + p_omega_inv.dot(P))
        rets = term1.dot(p_omega_inv.dot(Q))

But the code for producing the inputs (particularly the omega matrix) is a little bit more tricky.

What do you think?

from pyportfolioopt.

Hi @robertmartin8,

That's great! I was thinking in something similar to your solution:

def black_litterman_return(gamma, sigma, Q, omega, P=None, tau=0.01):
    # should we accept the implied returns gamma or a reference portfolio?
    # we might keep things flexible, checking for both and using the one provided

    omega_inv = np.diag(1 / np.diag(omega))
    if P is None:
        P = np.eye(omega_inv.shape[0])

    p_omega_inv = P.T @ omega_inv
    tau_sigma_inv = tau * np.linalg.inv(sigma)

    A = tau_sigma_inv + p_omega_inv @ P
    b = (tau_sigma_inv @ gamma + p_omega_inv @ Q)
    return np.linalg.solve(A, b)

A linear system is solved to avoid an extra matrix inversion, which I believe is more efficient (but I haven't tested yet!). Overall, only the sigma needs to be inverted.

I suppose I am more concerned about user-entered parameters. For example, what is a good way to allow users to specify their views? Should the function accept the BL inputs as matrices? or should we help users to construct them?

I think we should start with sensible defaults (like above for P and tau, what do you think about those default values?) and build up from that. The most basic here would be to accept no views at all by default (i.e. return the equilibrium return vector?) but receive the matrices provided by the user for now.

But the code for producing the inputs (particularly the omega matrix) is a little bit more tricky.

Since there's more than one way to do it, we could implement one or two after we know the code with the BL matrices works.

I'm particularly interested in Idzorek's method (section 3.3.2 of this) and methods involving valuation (I can't remember the source right now).

If there's a concern about poluting the interface or making the function too complex, a separate function might be used to build up the views using various methods.

(As a side question, when you use this, do you adjust the covariance matrix to reflect the BL model too? (equation (3) here).)

from pyportfolioopt.

In your code, what is gamma? Haven't seen it in the literature.

A linear system is solved

Good spot!

I think we should start with sensible defaults (like above for P and tau, what do you think about those default values?)

Agreed for tau, but not sure it makes sense to set P to be None as default. I would expect users to supply views.

return the equilibrium return vector?

Specification of the prior should happen in a separate function, possibly with the default being no prior. One other concern is that constructing the BL prior requires market cap data.

As a side question, when you use this, do you adjust the covariance matrix to reflect the BL model too?

I've been experimenting with that but haven't come to any conclusions one way or another as to which is better. It's also an open question as to whether the output should be shrunk, or whether you should compute the shrinkage estimate of the sample cov matrix before using the BL "weighted average".

from pyportfolioopt.

In your code, what is gamma? Haven't seen it in the literature.

Sorry for not mentioning it. It's the vector of implied returns (capital gamma).

We might go with asking the user for all the data (only giving a default value for tau?) and explicitly asking for the implied returns, so that it's their job to calculate the market portfolio and so on. Alternatively, another function could be used to calculate the implied returns from the market portfolio, as you said (which I think is the best idea actually).

I've been experimenting with that but haven't come to any conclusions one way or another as to which is better. It's also an open question as to whether the output should be shrunk, or whether you should compute the shrinkage estimate of the sample cov matrix before using the BL "weighted average".

I asked because I believe there would be room for a risk_models.black_litterman_cov:

# "cov_matrix" might be more meaningful than "sigma"
def black_litterman_cov(cov_matrix, omega, P, tau=0.01):
    omega_inv = np.diag(1 / np.diag(omega))
    p_omega_inv = P.T @ omega_inv
    tau_cov_matrix_inv = tau * np.linalg.inv(cov_matrix)

    A = tau_cov_matrix_inv + p_omega_inv @ P
    return cov_matrix + np.linalg.inv(A)

The idea and its proof can be seen here. The author says it's due to He and Litterman (2002), although I haven't read the original paper yet.

Using the function above, the user could provide any covariance matrix she likes, probably calculated using one of the functions in risk_models.

The drawback I see here is that there's a lot of repeated code among my proposed functions (black_litterman_return and black_litterman_cov). What do you think about making a class (such as CovarianceShrinkage)? A basic sketch follows:

class BlackLitterman:
    def __init__(self, implied_returns, cov_matrix, Q, omega, P, tau=0.01):
        ...
    def posterior_return(self):
        ...
    def posterior_cov(self):
        ...

What do you think?

from pyportfolioopt.

It's the vector of implied returns (capital gamma).

In the papers I've read that's normally denoted by capital Pi?

The drawback I see here is that there's a lot of repeated code

This is actually quite a tricky problem from a design perspective. In the current code I'm using at work, I've written BL as a class and it works very well. But that structure doesn't really fit with the PyPortfolioOpt API. It seems there are two options:

1. We repeat a bit of code

2. We define a BlackLitterman class in expected_returns.py, as you've suggested, then just import it into risk_models.

Option 2 feels a bit unexpected, so I think it might just be best to repeat a little: "Simple is better than complex. Complex is better than complicated"

Here's another question that I'd like your input on. When it comes to implementing BL, should we refer to the variables by their symbols (P, Q, Pi, Sigma, Omega) – which seem to be almost universally consistent in the literature – or should we use what they represent, i.e picking_matrix, views_vector, implied_returns, cov_matrix, views_cov_matrix?

from pyportfolioopt.

It's the vector of implied returns (capital gamma).

In the papers I've read that's normally denoted by capital Pi?

You're absolutely right, I mixed up the letters 😅

Option 2 feels a bit unexpected, so I think it might just be best to repeat a little: "Simple is better than complex. Complex is better than complicated"

Fine, let's go simple then. Only two functions (expected_returns.black_litterman_return and risk_models.black_litterman_cov) with the BL model in general terms. Since there's more than one way of calculating the implied returns, views, etc., we can leave all that to the user.

Here's another question that I'd like your input on. When it comes to implementing BL, should we refer to the variables by their symbols (P, Q, Pi, Sigma, Omega) – which seem to be almost universally consistent in the literature – or should we use what they represent, i.e picking_matrix, views_vector, implied_returns, cov_matrix, views_cov_matrix?

If the user is required to provide the inputs, having terms as close as possible to the standard literature might make it easier for new users. Equations can be included, referenced and comented in the documentation, so that people know what is each term in the code.

from pyportfolioopt.

@Busteren @robertmartin8 I plan on doing a PR real soon (probably today or tomorrow). Sorry for the delay, time was really short lately.

from pyportfolioopt.

@schneiderfelipe

I've been working to integrate the code from your PR #55. I've decided that BL won't be much use for the majority of users unless it's got many helper methods to format views properly, so I'm working on some functionality where a user can input (absolute) views and the function will figure out the picking matrix and views vector. Right now I'm hoping to accept either a pandas series or dict, e.g:

{ "AAPL": 0.20,
  "GOOG": 0.15
  "XOM":-0.3}

will result in a 3xN picking matrix and 3x1 views vector. I will also make sure the API allows for advanced users who want to input relative views and who know what they are doing.

However, with this in mind, it doesn't make sense to split BL between expected_returns and risk_models anymore. TL;DR, you were right and I was wrong :)

I'll be adapting your code into the new class (which I am also putting in a new module since it doesn't fit anywhere else).

On a side note, when I get round to writing the docs, the Black-Litterman page will attribute your GitHub profile – let me know if you would prefer this to be linkedin/email/webpage instead!

from pyportfolioopt.

An example of this (with some working tests) has now been pushed to the dev branch. Tomorrow I'll work on adapting your BL covariance code, as well as implementing a few more methods. For example, BL can be used to allocate a portfolio directly (without passing through another optimiser) by combining the expected returns with a market-implied risk-aversion parameter.

from pyportfolioopt.

That's awesome, thanks for the hard work @robertmartin8! I just proposed minor changes to the tests in PR #57 (minor code duplication).

Besides, I think the code and the concept are very good. Only a couple of things:

• I believe the formula you're using for implied prior returns in market_implied_prior_returns is actually excess returns, which raises the question whether this should be added the risk-free rate. We must check if it would fit the rest of the API. If that's the case, a minor change should be set to bl_weights so that the risk-free rate is subtracted from returns there.

• market_implied_risk_aversion is awesome and very very useful!

• I would suggest naming absolute_views something like absolute_Q or abs_Q, or calling Q views. This would ensure that the user understands the relationship between the two.

• I think both bl_returns and bl_cov should explicitly say in the docs that it's assumed that omega is diagonal.

• I believe inversion can be avoided in bl_weights, which is probably more robust/efficient. Probably something like

    raw_weights = np.linalg.solve(risk_aversion * self.cov_matrix, self.posterior_rets)
  

• I believe portfolio_performance should use the result from bl_cov instead of the prior covariance matrix (self.cov_matrix is currently used).

from pyportfolioopt.