Comments (7)
Hey @szabozoltan69 Looks like the API endpoint from go was changed. We will need to update the codebase here according to it.
We need to update the merged part to use the object field-value.
- merged = aadf.merge(apdo, left_on='id', right_on='appeal')
+ merged = aadf.merge(apdo, left_on=aadf['id'].astype(int), right_on=apdo.appeal.str['id'])
Here is a working example.
import requests
import pandas as pd
def download_api_results(call='appeal'):
href = "https://goadmin.ifrc.org/api/v2/"+call+"/?format=json&limit=300000"
aa = requests.get(href).json()
return pd.DataFrame(aa['results'])
DREF_Final_Report_names = [
'DREF Operation Final Report',
'DREF Final Report',
'DREF Operation Final Report 1'
]
# Filter only relevant DREF reports. Ignore case.
def filter_DREF_Final_Reports(df, names=DREF_Final_Report_names):
names_lower = [name.lower() for name in names]
return df[df.name.apply(lambda x: x.lower() in names_lower)].copy().reset_index(drop=True)
appeal_df = download_api_results(call='appeal')
raw_apdo = download_api_results(call='appeal_document')
apdo = filter_DREF_Final_Reports(raw_apdo)
merged = appeal_df.merge(apdo, left_on=appeal_df['id'].astype(int), right_on=apdo.appeal.str['id'])
def find(lead):
url_list = merged[merged.code==lead].document_url
if not url_list.empty:
return url_list.values[0]
print(merged.code)
print(find('MDRGH018'))
from dref-nlp.
I discovered that a point the DREF-parser uses such a request:
.../api/v2/appeal_document/?format=json&limit=300000
Which is quite resource intensive, not mentioning the 7-8 seconds while it runs.
(There is also a .../api/v2/appeal/?format=json&limit=300000 request which is a bit faster, but also quite resource intensive.)
from dref-nlp.
Many thanks, @thenav56 ! Great catch and elegant solution (to fix the merge of an id and a dictionary).
from dref-nlp.
Recently I get this runtime error:
(Pdb) p aadf.merge(apdo, left_on=aadf['id'].astype(int), right_on=apdo.appeal.str['id'])
*** pandas.errors.IntCastingNaNError: Cannot convert non-finite values (NA or inf) to integer
These are the merged entities.
from dref-nlp.
Hi @thenav56, I tried to solve this ^ issue via some "fillna(value=0)", but dit not help. :-(
from dref-nlp.
Hey @szabozoltan69 , We need to remove this line as well.
https://github.com/IFRCGo/DREF-NLP/blob/master/dref_parsing/dref_parsing/parser_utils.py#L267
- apdo.appeal = apdo.appeal.astype(str)
from dref-nlp.
Thanks so much @thenav56!
Now it works fine.
from dref-nlp.
Related Issues (2)
- Site can’t be reached HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dref-nlp.