Comments (19)
sckott
commented on May 31, 2024
2
ok, thanks @stijnvanhoey - pandas docs sounds fun and impt.
I'll probably take a crack at it, but will make sure you two have a look at it
from pygbif.
damianooldoni
commented on May 31, 2024
1
Thanks @sckott . Just back from two weeks holidays and I don't see time to do it even. Still, available for review. So, ping me if needed.
from pygbif.
sckott
commented on May 31, 2024
thanks for your question @damianooldoni
@stijnvanhoey @peterdesmet you two I think did most of the download methods. Any thoughts on the above?
Here's the JSON body that's sent in that example you gave:
{
"creator": "<hidden>",
"notification_address": [
"<hidden>"
],
"send_notification": "true",
"created": 2019,
"predicate": {
"type": "and",
"predicates": [
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "OBSERVATION"
},
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "LITERATURE"
},
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "PRESERVED_SPECIMEN"
},
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "MATERIAL_SAMPLE"
},
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "UNKNOWN"
},
{
"type": "equals",
"key": "BASIS_OF_RECORD",
"value": "HUMAN_OBSERVATION"
},
{
"type": "equals",
"key": "COUNTRY",
"value": "BE"
},
{
"type": "greaterThanOrEquals",
"key": "YEAR",
"value": "1000"
},
{
"type": "lessThanOrEquals",
"key": "YEAR",
"value": "2019"
},
{
"type": "equals",
"key": "HAS_COORDINATE",
"value": "TRUE"
}
]
}
}does that look as expected?
from pygbif.
sckott
commented on May 31, 2024
any thoughts @stijnvanhoey @peterdesmet ?
from pygbif.
stijnvanhoey
commented on May 31, 2024
The current GbifDownload class provides the different building blocks required to handle this case by using the object oriented apporach instead of the occurrences.download() shortcut function:
from pygbif.occurrences.download import GbifDownload
# initiate the download class
gbif_query = GbifDownload('xxxxxxxx', 'xxxxxxxxx') # user name and email
# setup the query
gbif_query.add_predicate('COUNTRY', 'BE', predicate_type='equals')
gbif_query.add_predicate('YEAR', 1000, predicate_type='>=')
gbif_query.add_predicate('YEAR', 2019, predicate_type='<=')
gbif_query.add_predicate('hasCoordinate', TRUE, predicate_type='equals')
# add the multiple values predicate:
gbif_invasive.add_iterative_predicate('basisOfRecord',
['LITERATURE', 'OBSERVATION', 'PRESERVED_SPECIMEN',
'MATERIAL_SAMPLE', 'UNKNOWN', 'HUMAN_OBSERVATION'])
# post request download
gbif_query.post_download('xxxxxxxxx', 'xxxxxx')
So, it is more a matter of missing documentation...
from pygbif.
stijnvanhoey
commented on May 31, 2024
the gbif_invasive.predicates attribute shows the predicates as setup:
> gbif_invasive.predicates
[{'key': 'COUNTRY', 'type': 'equals', 'value': 'BE'},
{'key': 'YEAR', 'type': 'greaterThanOrEquals', 'value': 1000},
{'key': 'YEAR', 'type': 'lessThanOrEquals', 'value': 2019},
{'key': 'hasCoordinate', 'type': 'equals', 'value': True},
{'predicates': [
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'HUMAN_OBSERVATION'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'UNKNOWN'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'MATERIAL_SAMPLE'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'PRESERVED_SPECIMEN'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'OBSERVATION'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'LITERATURE'}],
'type': 'or'}]
and these are combined with the gbif_invasive.main_pred_type (default and) which is combined into the gbif_invasive.payload:
{'created': 2019,
'creator': <hidden>,
'notification_address': <hidden>,
'send_notification': 'true'
'predicate': {'predicates': [
{'key': 'COUNTRY', 'type': 'equals', 'value': 'BE'},
{'key': 'YEAR', 'type': 'greaterThanOrEquals', 'value': 1000},
{'key': 'YEAR', 'type': 'lessThanOrEquals', 'value': 2019},
{'key': 'hasCoordinate', 'type': 'equals', 'value': 'TRUE'},
{'predicates': [
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'HUMAN_OBSERVATION'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'UNKNOWN'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'MATERIAL_SAMPLE'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'PRESERVED_SPECIMEN'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'OBSERVATION'},
{'key': 'basisOfRecord', 'type': 'equals', 'value': 'LITERATURE'}],
'type': 'or'}],
'type': 'and'}
}
@damianooldoni can you have a check if this is correct and similar to the rgbif request?
from pygbif.
sckott
commented on May 31, 2024
@damianooldoni any time to take a look at question from @stijnvanhoey ? if not, i'll take a look
from pygbif.
damianooldoni
commented on May 31, 2024
yes, @sckott . Actually It totally slipped my mind.
Indeed, the query posted by @stijnvanhoey is similar to the query sent via rgbif. An example in R here below:
countries <- c("BE", "NL")
basis_of_record <- c("HUMAN_OBSERVATION", "LITERATURE")
year_begin <- 1990
year_end <- 1991
rgbif::occ_download(
paste0("basisOfRecord = ", paste(basis_of_record, collapse = ",")),
paste0("country = ", paste(countries, collapse = ",")),
paste0("hasCoordinate = TRUE"),
paste0("year >= ", year_begin),
paste0("year <= ", year_end)
)which results in following API query:
{
"type": "and",
"predicates": [
{"type": "or", "predicates": [
{"type": "equals", "key": "BASIS_OF_RECORD", "value": "HUMAN_OBSERVATION"},
{"type": "equals", "key": "BASIS_OF_RECORD", "value": "LITERATURE"}
]},
{"type": "or", "predicates": [
{"type": "equals", "key": "COUNTRY", "value": "BE"},
{"type": "equals", "key": "COUNTRY", "value": "NL"}
]},
{"type": "equals", "key": "HAS_COORDINATE", "value": "TRUE"},
{"type": "greaterThanOrEquals", "key": "YEAR", "value": "1990"},
{"type": "lessThanOrEquals", "key": "YEAR", "value": "1991"}]
}
This has same structure of the query posted by @stijnvanhoey: only the order changes (type-key-value vs key-type-value), which doesn't change anything of course.
I will double check the solution provided by @stijnvanhoey and if it works this issue can be closed.
from pygbif.
stijnvanhoey
commented on May 31, 2024
As the result is the same, we should improve the documentation of pygbif to make sure this use case is explained to other users as well. Or we could improve the documentation by providing an explanation of the object oriented way of using pygbif more in general?
from pygbif.
sckott
commented on May 31, 2024
+1 to improving docs/adding examples
from pygbif.
damianooldoni
commented on May 31, 2024
I test is again to be completely sure. Yes, documentation should be improved as well. I can give a try.
from pygbif.
damianooldoni
commented on May 31, 2024
I found that this doesn't work:
gbif_query = GbifDownload(xxxxxxx, xxxxxxxxx) # user name and pwd
gbif_query.add_iterative_predicate('basisOfRecord', ['LITERATURE', 'HUMAN_OBSERVATION'])
gbif_query.add_iterative_predicate('taxonKey', [1898286, 1894840])
gbif_query.add_predicate('hasCoordinate', 'TRUE', predicate_type='equals')
gbif_query.post_download(xxxxxxx, xxxxxxxxx) # user name and pwdwhile this works:
gbif_query.add_iterative_predicate('BASIS_OF_RECORD', ['LITERATURE', 'HUMAN_OBSERVATION'])
gbif_query.add_iterative_predicate('TAXON_KEY', [1898286, 1894840])
gbif_query.add_predicate('HAS_COORDINATE', 'TRUE', predicate_type='equals')
gbif_query.post_download(xxxxxxx, xxxxxxxxx) # user name and email
This means that the parameters of the shortcut function occurrences.download() are the typical ones (same as the rgbif's ones) while we have to use the "raw" versions of them if we want to build queries with .add_predicate() and .add_iterative_predicate().
This has to be documented as well or, even better I think, should be changed. Converting keys automatically (as in occurrence.download()) allows the user to not change key style (e.g. hasCoordinate vs HAS_COORDINATE) while writing complex queries.
@stijnvanhoey , @sckott : what do you think about?
from pygbif.
sckott
commented on May 31, 2024
converting for the user makes sense, what do you think @stijnvanhoey ?
from pygbif.
sckott
commented on May 31, 2024
from pygbif.
stijnvanhoey
commented on May 31, 2024
I'm sorry, I agree that using the Darwin-core terms make much more sense for the user. I would refactor the input of it before updating the documentation.
from pygbif.
sckott
commented on May 31, 2024
thanks @stijnvanhoey - agree we should refactor. Does one of you have time for this? or should I put it on my to do list?
from pygbif.
stijnvanhoey
commented on May 31, 2024
I won't be able to do it the coming weeks, so it would rather be November that I can contribute on this. Currently too busy on remake of pandas documentation ;-)
from pygbif.
glaroc
commented on May 31, 2024
+1 for adding this to the docs. I had to search through the issues to find this info.
from pygbif.
CecSve
commented on May 31, 2024
duplicate of #104
from pygbif.
Related Issues (20)
- Include predicates IS NULL and IS NOT NULL in the download script HOT 1
- Run integration tests against live GBIF APIs HOT 1
- Add support of in predicate HOT 1
- support the not predicate
- support dictionaries (nested queries)
- Support multiple download format
- Add examples to the documentation of newly implemented features
- Check operators and look-up tables
- add_predicate update
- Option "verbatim" in method species.name_usage throws error for specific taxon HOT 2
- add compatibility to requests-cache = 0.8 HOT 2
- "SyntaxError: not a PNG file" in maps.map() call with 204 response HOT 1
- add requests-cache to dependencies HOT 1
- Add support for isNull, isNotNull predicates in download
- ModuleNotFoundError: No module named 'requests_cache.core' HOT 10
- [DOCUMENTATION] GBIF_USER and GBIF_PWD missing but needed HOT 2
- Docs: invalid field and invalid page
- Circular Occurences?
- Lack of HTTP connection reuse makes repeated queries slow
- Update limit of characters statement for download queries
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pygbif.