Way to get a count of query results without downloading the dataset? about direct-access-py HOT 6 CLOSED

@eliasvitali You can get counts by substituting HEAD for GET for any Direct Access v2 operation. It respects any filter provided. The count is returned in the HTTP headers as X-Query-Record-Count.

However, performance isn't as good as GET because the entire data structure has to be scanned to get an accurate count, so please adjust any TTL you have on the request accordingly.

from direct-access-py.

Ah I see, that is stable enough for my liking, thanks for the suggestion!

from direct-access-py.

Hi @rbastian . Thanks for the super quick reply! I'm not quite sure how to implement your solution, for I'm not using the keywords GET or HEAD anywhere, but rather the DirectAccessV2.query() method. I did however, after quite the rigorous Google keyword search, come across this snippet of code, which appears to be an implemented count method of DirectAccessV2 pbjects. After testing it, it does appear to work; thanks anyways for the help and prompt reply :)

def count(self, dataset, **options):
        """
        Get the count of records given a dataset and query options

        :param dataset: a valid dataset name. See the Direct Access documentation for valid values
        :param options: query parameters as keyword arguments
        :return: record count as int
        """
        url = self.url + '/' + dataset
        response = self.session.head(url, params=options)
        count = response.headers.get('X-Query-Record-Count')
        return int(count)

from direct-access-py.

Hi @eliasvitali!

There's an existing method in the DirectAccessV2 class to get a count for a query.

@rbastian's suggestion of adjusting any timing for your process still stands! In some cases, it can be a rather long time for the count to return.

Here's an example:

client = DirectAccessV2(client_id, client_secret, api_key)
count = client.count('rigs', deleteddate='null', drilltype='H')

from direct-access-py.

Thank you! Perhaps you can help me solve a mystery, because this is the reason I wanted to return a count:

• DirectAccessV2.count('producing-entity-details', upddateddate = 'eq(2020-07-14)') returns 0
• DirectAccessV2.count('producing-entity-details', upddateddate = 'eq(2020-07-15)') returns 0
• DirectAccessV2.count('producing-entity-details', upddateddate = 'btw(2020-07-14, 2020-07-15)') returns 7436932

Any idea why this would be? No results on either the 14th or 15th but 7M+ in between?

from direct-access-py.

@eliasvitali that's because there are no records with an updated date that exactly equals (eq()) only a date without a time, so a count of 0 is correct.

This might not be the most graceful way to handle but here's how I've gotten counts for a single day's update previously:

count = client.count('producing-entity-details', updateddate='btw(2020-07-14T00:00:00, 2020-07-14T23:59:59)')

from direct-access-py.