Phase Change Index about technical HOT 18 CLOSED

Ok no problems, I'll keep that in mind for the future. I think we can call this done now, thanks again for your help and patience

from technical.

honestly, your description of the indicator leaves more questions than answers/explanations ... (ignoring the code for now - as that could lead in a wrong way right away).

Assuming a lookback of 35 for simplicity...

for point T - would you then have exactly one gradiend (df[T] vs. df[T - 35]) - or would you have 35 gradients to compare to?
If it's one - it should be pretty simple (you get exactly one value per timepoint - so using .shift() should work just fine.

If you're looking for 35 gradients (assuming mid-dataframe) - then you'll need to keep in mind that as you approach "now" (the end of the dataframe) - the number of possible gradients will decrease - ending with exactly one gradient (Now - 35 candles ago).
You also need to be very careful how each of the lines is used to really avoid a lookback, but without actually seeing / understanding what you're trying to do (the above is many words, but little (or confusing) information.

from technical.

Thanks for your reply. The situation is the second one - there are 35 gradient points, and each gradient point is compared to its equivalent past close price, ie the first gradient value is subtracted from close[t-35], then gradient 2 - close[t-34] etc.

I realise my explanation is not overly clear - I've found it very hard to explain how it works and the original article took me a few attempts to understand. Not to be condescending but I found the best way for me to visualise the indicator's machinations was to go to a chart and draw a straight line 35 bars long between the current close and the close 35 bars ago. This is the gradient line, and has actual values on the chart. There will be close values above and below this line, and by subtracting the gradient value from the close and summing the values above and below, you can find out the proportion of close values that were greater or less than the gradient.

The purpose of the indicator is to identify when the market is moving between uptrend, downtrend, and consolidation phases (hence phase change index).

I hope that helps, my paper calculations for this indicator look very promising so it would be great to find a way to implement it in Freqtrade. Thanks

from technical.

if one requirement is to look at all 35 gradient points (not a partial amount, like 34 or fewer) - then your indicator will stop 35 candles from the current candle.
you'll therefore only see "35 candles ago, we were moving up" - but have 0 information on what happened in between (this same thing is true for EVERY point in the dataframe ... even though it may not look like that).

your approach will probably something around this to populate the gradient columns (i'm currently too lazy to calculate the actual gradient - and your "+" application doesn't actually do that).

for i in range(35):
    dataframe[f'gradient_{i}'] = dataframe['close'].shift(0-i)  ... dataframe['close'].shift(i)

you then have a list of 35 columns - each of which you'll compare against close (i assume).
(careful, the above intentionally looks into the future).

You can't really do this without looking into the future - as at any point T - you'll need 35 gradients, comparing "close vs close-35", "close+1 vs. close-34", ..."close+34 vs. close-1" ...).

In intermediate calculations, that's also not necessarily a problem.
However, when using the indicator, you must shift it forward by 35 candles, as the comparison result looks into the future by 35 candles.

To me, this sounds like a very visual indicator - which works well when looking at a chart - but not so much when using it in an automated fashion - unless you consider it a long-term indicator, and only use each value once the full values have populated (basically by shifting it forward by 35 candles).

from technical.

Thanks again for your response, that's really helpful. I modified your loop slightly with this:

def phase_change_index(self, dataframe: DataFrame, period):
        df = dataframe.copy()

        # Calculate momentum
        df['momentum']=ta.MOM(df, period)

        data=pd.DataFrame()

        for i in range(35):
            data[f'gradient_{i}'] = df['close'].shift(0-i) + (df['momentum'] * (i/35))

        print(data)

When I print to see what's happening, it looks like the most recent gradient value is in column 0 position 0, then value 2 is in column 1 position -1, column 2 position -2 etc. It actually almost traces out a gradient line across the columns, which is kind of cool. And when I check the values against my hand calculations they're correct. We're winning!

With that data layout I'm not unsure how to subtract the close and the gradient, as I'll need to iterate through both columns and rows. Here's my attempt below, in pseudocode since I don't know how to do it properly, but should give an idea of what I'm trying to achieve:

df['up'] = 0.00
        df['down'] = 0.00
        
        for i in range(35):
            if df['close'].shift(0-i) > df['gradient_i'].shift(0-i):
                df['up'] = df['close'][i] - df['gradient_i'].shift(i)
            else:
                df['down'] = df['gradient_i'].shift(i) - df['close'][i]

Shifting the close price incrementally looks easy, but I don't know how to shift both row and column for the gradient values. Any help is much appreciated!

[edited for formatting]

from technical.

well you have 35 gradients per candle - so shifting close again will be wrong.
you'll simply need to compare the single close 35 times (to the 35 gradients you created before - without any shift) - which should give you 35 x 2 new columns (up + down) per candle.

from technical.

I don't think that's quite right though, I'm not comparing each gradient to a single close value, I need to compare each gradient to the close value at that point along the line - gradient.shift(1) compared with close.shift(1), gradient.shift(2) with close.shift(2) etc for the 35 points. That's what I meant by shifting the close as well. Unless I'm misunderstanding what you mean?

from technical.

for each point T - you've got 35 gradients.

If you shift the close as well and compare it 35 times (so 35 * 35) - then you're doing the comparison twice.

from technical.

Ok I think I see what you mean, I found a mistake in my code. Instead of:

for i in range(35):
            data[f'gradient_{i}'] = df['close'].shift(0-i) + (df['momentum'] * (i/35))

It should be:

for i in range(35):
            data[f'gradient_{i}'] = df['close'].shift(35) + (df['momentum'] * (i/35))

Which gives me this output:

      gradient_0    gradient_1    gradient_2    gradient_3  ...   gradient_31   gradient_32   gradient_33   gradient_34
0            NaN           NaN           NaN           NaN  ...           NaN           NaN           NaN           NaN
1            NaN           NaN           NaN           NaN  ...           NaN           NaN           NaN           NaN
2            NaN           NaN           NaN           NaN  ...           NaN           NaN           NaN           NaN
3            NaN           NaN           NaN           NaN  ...           NaN           NaN           NaN           NaN
4            NaN           NaN           NaN           NaN  ...           NaN           NaN           NaN           NaN
...          ...           ...           ...           ...  ...           ...           ...           ...           ...
7311    26968.94  27003.831143  27038.722286  27073.613429  ...  28050.565429  28085.456571  28120.347714  28155.238857
7312    27124.91  27152.764571  27180.619143  27208.473714  ...  27988.401714  28016.256286  28044.110857  28071.965429
7313    26971.71  27001.623143  27031.536286  27061.449429  ...  27899.017429  27928.930571  27958.843714  27988.756857
7314    26962.88  26985.226857  27007.573714  27029.920571  ...  27655.632571  27677.979429  27700.326286  27722.673143
7315    27057.36  27085.813143  27114.266286  27142.719429  ...  27939.407429  27967.860571  27996.313714  28024.766857

Now each row is the whole gradient line at that close value (I think).

Which means I should be able to compare close.shift(0) with gradient_0, close.shift(1) with gradient_1, etc without doubling up.
I think it should look something like this, but it doesn't work and I don't know what's wrong:

for columns in data:
            if df['close'].shift(0-i) > data[columns].values:
                df['up'] = df['close'].shift(i) - df[columns].value
            else:
                df['down'] = df[columns].value - df['close'].shift(i)

Once I have the up and down columns populated, I think the rest of the indicator should simply be:

       df['up_sum'] = df['up'].rolling(36).sum()
       df['down_sum'] = df['down'].rolling(36).sum()

        df['deviation_sum'] = df['up_sum'] + df['down_sum']

        df['PCI'] = (df['up_sum']/df['deviation_sum']) * 100

How does this look to you? I think it's nearly there. Thanks for your patience with my questions, I appreciate you taking the time to help.

from technical.

well your up/down assignment must be vectorized (look at np.where if you need conditions per row)

the way you have it, it'll be a "last column wins" - which is not what you want

from technical.

Ok, thanks. I've looked into np.where and tried this:

        deviation['up'] = np.where(data['gradient_1'] > df['close'].shift(34), data['gradient_1'] - df['close'].shift(34), 0)
        deviation['down'] = np.where(data['gradient_1'] < df['close'].shift(34), df['close'].shift(34)- data['gradient_1'], 0)

Which gives me the output:

              up        down
0       0.000000    0.000000
1       0.000000    0.000000
2       0.000000    0.000000
3       0.000000    0.000000
4       0.000000    0.000000
...          ...         ...
7311    0.000000  121.078857
7312  181.054571    0.000000
7313   38.743143    0.000000
7314    0.000000   72.133143
7315   87.183143    0.000000

I think this output means that each value in each row of the up column corresponds with gradient_1 - close.shift(34). And it will be updated with every new bar, right? So how do I use np.where to create a function where gradient_1 - close(34), gradient_2 - close(33), gradient_3 - close(32), etc? Each gradient calculation would need its own column, eg up_1, up_2, up_3, down_1, down_2, down_3... etc. Is that right, or is there a better way? Thanks again

from technical.

well probably yes - you need 35 up/down values for evern candle - which you then aggregate over- but i think you're messing up the calculations again - by again double-shifting close.

This is all VERY confusing - as the actual intent is totally unclear.
Quite honestly, you're best off first learning pandas, before trying to write an algorithm which you clearly can't explain clearly (which suggests you yourself are not entirely sure how it's actually supposed to work).

from technical.

I do 100% understand how it works, what I lack is the knowledge of python to implement it. As I said right at the start, I know it's hard to explain and it took me a few goes to get my head around it. I'll try one last time to explain it. I don't want to waste your time, I realise you volunteer to help out for an open-source project but I know this will be really simple to calculate with the knowledge of Python and Freqtrade. I know I keep saying it but I do appreciate you taking the time to read all of this.

Perhaps a worked example would be the best way to explain, and to keep it simple I'll use a lookback period of 5 bars. Let's say you have 5 close prices - $1, $2, $2, $1, $5. The first step is to calculate the difference between the latest close and the close 5 bars ago - so 5 -1 = $4 difference. That also happens to be what the momentum indicator does, so I though I could use ta-lib or pandas-ta to calculate momentum.

The next step is to calculate an imaginary straight line between the close 5 periods ago and the current close, which is called the gradient. Take the momentum and divide by 5 (the number of samples), and add that incrementally to the first close value to calculate the points along the line. In this example, a momentum of 4, divided by 5 samples = 0.8. Note that the we end up using n-1 because the first and last values of the gradient line will equal the close values. You'll see what I mean below.

To get the first point along the gradient line, you take close.shift(5) + (momentum*(0/4)), so 1+(4*(0/4) = 1. This is gradient value 1, which will be the same as close.shift(5).
For the second point you take close.shift(4) + (momentum*(1/4)) = 2+(41/4)=3.
For the third point, close.shift(3) + (momentum(2/4)) =2+(4*(2/4)= 4
Fourth point, close.shift(2) + (momentum*(4/5)) = 1+(4*(3/4)) = 4
The last value will be the same as the close at the end of the gradient line, as that's close + (momentum*(4/4)) = 5
( I know there's aliasing in the gradient line in this example, it's because of the low sample rate and round numbers, ignore it)

The next step is to compare the gradient line values with the actual close value at that point. We can call this the deviation. If the gradient > close, then the value is gradient - close, and the result is saved in the up-deviation column. If gradient < close then the value is close - gradient, and that is saved as a down deviation.

So for gradient 1, the close at that point on the gradient line is 1 and the gradient is 1, so the deviation is 0.
For gradient 2, the close at that point on the gradient line is 2 and the gradient is 3, so 3 - 2=1, which is stored in the down column because the gradient > close
For gradient 3, the close is 2 and the gradient is 4, so 4-2 = 2 in the down column because gradient > close
Fourth point is 4 so 4-1 = 3 in the down column
Fifth point is 0 as it's the close itself so there's no deviation.

Now that the up and down columns are populated, we need to work out the sum of those columns. So the sum of down is 1+2+3 = 6. The sum of up is obviously 0 as there were no values stored.

Next add the sum of up with the sum of down. In this case that's 6.

Then the final step is calculate the PCI by dividing the sum of up by (sum of up + sum of down) then * 100 to create a percentage value. So (0/6) * 100 = 0.

The final value tells us is that 0% of the close values were > the gradient line. Without muddying the waters further by trying to explain the interpretation of the results, that's how the indicator is calculated. I can see I've sent a wall of text and I have a headache now, but I'm not sure I can explain it any other way. For the sake of completeness I've attached a pdf of the original publication of the indicator in case you want to read it.

Thanks again

phase_change_index(1).pdf

from technical.

Looking at that paper raises more questions than answers ...

Looking at the below image - how is 35.41 + 1.8/0/4 = 35.41 ... and then the next line, it's 35.86, ...
it's wrong in both instances ... from a pure mathematical standpoint.

from technical.

i really lack the patience/time to fully understand this - but this is a topic where chatbots might actually do a decent job.

Disclaimer: using chatGPT (GPT4), providing it with your explanation and the following query (without the PDF):

can you code the above in python, using pandas, assuming a dataframe with the columns "date, open, high, low, close, volume", thanks.

It was even nice, thanking for the great explanation 😆

import pandas as pd


def pci(data, lookback=5):
    # Calculate momentum
    data['momentum'] = data['close'] - data['close'].shift(lookback)

    # Calculate gradient values
    gradient_values = []
    for i in range(lookback):
        gradient_value = data['close'].shift(
            lookback) + (data['momentum'] * (i / (lookback - 1)))
        gradient_values.append(gradient_value)

    # Calculate deviations
    up_deviation = pd.DataFrame(index=data.index, columns=[
                                f'up_dev_{i}' for i in range(lookback)])
    down_deviation = pd.DataFrame(index=data.index, columns=[
                                  f'down_dev_{i}' for i in range(lookback)])
    for i, gradient_value in enumerate(gradient_values):
        up_deviation[f'up_dev_{i}'] = (data['close'].shift(
            i) - gradient_value).where(data['close'].shift(i) > gradient_value, 0)
        down_deviation[f'down_dev_{i}'] = (
            gradient_value - data['close'].shift(i)).where(data['close'].shift(i) < gradient_value, 0)

    # Sum up and down deviations
    data['up_dev_sum'] = up_deviation.sum(axis=1)
    data['down_dev_sum'] = down_deviation.sum(axis=1)

    # Calculate PCI
    data['pci'] = (data['up_dev_sum'] /
                   (data['up_dev_sum'] + data['down_dev_sum'])) * 100
    # data.drop(columns=['momentum', 'up_dev_sum', 'down_dev_sum'], inplace=True)

    return data


# Example data
data = pd.DataFrame({
    'date': pd.date_range('2023-01-01', periods=20, freq='D'),
    'open': [0.5, 1.5, 1.5, 0.5, 4.5, 4.0, 5.0, 5.5, 6.0, 7.0, 8.0, 7.5, 7.0, 6.5, 9.0, 9.5, 8.5, 10.0, 11.0, 12.0],
    'high': [1.5, 2.5, 2.5, 1.5, 5.5, 5.0, 6.0, 6.5, 7.0, 8.0, 9.0, 8.5, 8.0, 7.5, 10.0, 10.5, 9.5, 11.0, 12.0, 13.0],
    'low': [0, 1, 1, 0, 4, 3.5, 4.5, 5.0, 5.5, 6.5, 7.5, 7.0, 6.5, 6.0, 8.5, 9.0, 8.0, 9.5, 10.5, 11.5],
    'close': [1, 2, 2, 1, 5, 4, 5, 6, 6, 7, 8, 8, 7, 7, 9, 10, 9, 10, 11, 12],
    'volume': [1000, 2000, 2000, 1000, 5000, 4000, 5000, 6000, 6000, 7000, 8000, 8000, 7000, 7000, 9000, 10000, 9000, 10000, 11000, 12000]
})

result = pci(data)
print(result)

from technical.

Hahaha that's great, AI really will make us all redundant. I'll try it out. My only other concerns is, from your perspective, does the indicator look forward at all? I'd like to be able to use it in backtesting. It looks ok to me but I've been wrong before...

from technical.

Ok I've had a chance to test out the indicator and after a few tweaks to suit my strategy it's already providing some solid results. So thanks for your time and energy helping me with this - between you, me and ChatGPT we got there in the end 😆

Briefly re ChatGPT for Freqtrade if I may - did you actually just put my explanation in along with your instructions and it produced that completed output? No other steps? If so I know how I'll be building my indicators from now on!

Cheers

from technical.

i used this post #322 (comment) - without the first paragraph - and with the query i posted above.

You can easily test out if an indicator is forward looking by calculating the indicator - then adding (or removing) a new row - and calculating it again (for the whole dataframe).

You'd expect that

• no old values are changed, and the calculation yields the same resut
• the new row get's a value (it didn't have one before).
• this repeats for an unlimited amount of candles
• this also repeats if old candles (beyond the lookback horizon) are removed, so the dataframe length is kept at an equal length (obviously with the caveat that the "startup candles" won't have values calculated anymore)

from technical.