Giter Club home page Giter Club logo

Comments (7)

thenav56 avatar thenav56 commented on June 12, 2024 1

Hey @szabozoltan69 Looks like the API endpoint from go was changed. We will need to update the codebase here according to it.

We need to update the merged part to use the object field-value.

-    merged = aadf.merge(apdo, left_on='id', right_on='appeal')
+    merged = aadf.merge(apdo, left_on=aadf['id'].astype(int), right_on=apdo.appeal.str['id'])

Here is a working example.

import requests
import pandas as pd

def download_api_results(call='appeal'):
    href = "https://goadmin.ifrc.org/api/v2/"+call+"/?format=json&limit=300000"
    aa = requests.get(href).json()
    return pd.DataFrame(aa['results'])

DREF_Final_Report_names = [
    'DREF Operation Final Report',
    'DREF Final Report',
    'DREF Operation Final Report 1'
]

# Filter only relevant DREF reports. Ignore case. 
def filter_DREF_Final_Reports(df, names=DREF_Final_Report_names):
    names_lower = [name.lower() for name in names]
    return df[df.name.apply(lambda x: x.lower() in names_lower)].copy().reset_index(drop=True)


appeal_df = download_api_results(call='appeal')
raw_apdo = download_api_results(call='appeal_document')
apdo = filter_DREF_Final_Reports(raw_apdo)

merged = appeal_df.merge(apdo, left_on=appeal_df['id'].astype(int), right_on=apdo.appeal.str['id'])

def find(lead):
    url_list = merged[merged.code==lead].document_url
    if not url_list.empty:
        return url_list.values[0]

print(merged.code)
print(find('MDRGH018'))

from dref-nlp.

szabozoltan69 avatar szabozoltan69 commented on June 12, 2024

I discovered that a point the DREF-parser uses such a request:
.../api/v2/appeal_document/?format=json&limit=300000
Which is quite resource intensive, not mentioning the 7-8 seconds while it runs.
(There is also a .../api/v2/appeal/?format=json&limit=300000 request which is a bit faster, but also quite resource intensive.)

from dref-nlp.

szabozoltan69 avatar szabozoltan69 commented on June 12, 2024

Many thanks, @thenav56 ! Great catch and elegant solution (to fix the merge of an id and a dictionary).

from dref-nlp.

szabozoltan69 avatar szabozoltan69 commented on June 12, 2024

Recently I get this runtime error:

(Pdb) p aadf.merge(apdo, left_on=aadf['id'].astype(int), right_on=apdo.appeal.str['id'])
*** pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer

These are the merged entities.

from dref-nlp.

szabozoltan69 avatar szabozoltan69 commented on June 12, 2024

Hi @thenav56, I tried to solve this ^ issue via some "fillna(value=0)", but dit not help. :-(

from dref-nlp.

thenav56 avatar thenav56 commented on June 12, 2024

Hey @szabozoltan69 , We need to remove this line as well.
https://github.com/IFRCGo/DREF-NLP/blob/master/dref_parsing/dref_parsing/parser_utils.py#L267

-         apdo.appeal = apdo.appeal.astype(str)

from dref-nlp.

szabozoltan69 avatar szabozoltan69 commented on June 12, 2024

Thanks so much @thenav56!
Now it works fine.

kép
@ypyelab

from dref-nlp.

Related Issues (2)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.