[Question] One-sided p-value for WEAT? about wefe HOT 8 CLOSED

Hello @santoshbs

A few moments ago I made a deploy of a new version of the library.
It includes the calculation of the p-value in WEAT, based on the code you proposed.
Thank you very much!

It is implemented in the WEAT.py file in case you need to see its implementation.
Here is the API documentation.

By default, it takes the original statistical test of weat: right one sided. However, it also includes the option for left sided and two sided, which gives more control over the test.

It remains as a future job to implement it with joblib to parallelize this operation.

Best regards,
Pablo.

from wefe.

Hello @santoshbs

At this time the option to perform the One-sided p-value of WEAT is not currently available.

However, this feature is currently under development and will be implemented in the next software release.

Best regards,
Pablo.

from wefe.

Hello,

Thank you very much!
I will integrate this code in the next version of the library (hopefully this week).

Best regards,
Pablo.

from wefe.

I almost forgot,

With respect to :

By the way, I noticed that the use of the terms 'target' and 'attribute' is different in WEFE as compared to that in the above paper.

We decided to restrict what the target and attribute set can include in the design of the framework.
While we kept the names defined in the WEAT paper, we tried to validate the definitions of other metrics such as social group words and neutral words, etc..

Regards,
Pablo.

from wefe.

Thanks, @pabloBad

from wefe.

Hello @pabloBad , I attempted to create the p-value calculation function based on some code here.

def calc_p_value(model, target, attribute, iterations= 100000):
    '''
    Reference: https://wefe.readthedocs.io/en/latest/about.html#motivation-and-objectives
    target: he, she, etc.
    attribute: doctor, nurse, etc.
    '''
    u= attribute[0] + attribute[1]
    n= int(len(u)/2)
    runs= np.min((iterations, math.factorial(len(u)))) 
    seen= set()
    
    q= Query(target, attribute)
    w= weat.run_query(q, model)
    r_original= w['result']
    count_greater= 0
    
    for _ in range(runs):
        permutation= tuple(random.sample(u, len(u)))
        if permutation not in seen:
            a1= list(permutation[0:n])
            a2= list(permutation[n:])
            attribute_hat= [a1, a2]
            q_hat= Query(target, attribute_hat)
            w_hat= weat.run_query(q_hat, model)
            r_hat= w_hat['result']
            if r_hat > r_original:
                count_greater += 1
            seen.add(permutation)
    
    p_value= count_greater/runs
    return p_value

Hope this is the correct implementation of the methodology outlined in the paper:
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186.

By the way, I noticed that the use of the terms 'target' and 'attribute' is different in WEFE as compared to that in the above paper.

from wefe.

No (bare, at most) experience in developing software packages, but social scientists really need this function. Thank you all! Looking forward to the improvements!

from wefe.

Hello @santoshbs

A few moments ago I made a deploy of a new version of the library.
It includes the calculation of the p-value in WEAT, based on the code you proposed.
Thank you very much!

It is implemented in the WEAT.py file in case you need to see its implementation.
Here is the API documentation.

By default, it takes the original statistical test of weat: right one sided. However, it also includes the option for left sided and two sided, which gives more control over the test.

It remains as a future job to implement it with joblib to parallelize this operation.

Best regards,
Pablo.

Thank you!

from wefe.