Hello, first of all compliments for your library, it's a great evolution over RSel

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Hello, I may have found a solution: <div class="snippet-clipboard-content notransl

Logic for returning results from a function that get XHR calls response body about crrri HOT 3 OPEN

bakaburg1 commented on June 3, 2024

Logic for returning results from a function that get XHR calls response body

from crrri.

Comments (3)

RLesur commented on June 3, 2024 1

Hi @bakaburg1,

We're glad you like crrri! I haven't had time to dive in your example but here is what I would do to create an async counter with R promises:

# we can use a closure to define an async counter:
async_counter <- function(resolve_at = 20) {
  # initialize objects
  count <- 0
  resolve_function <- NULL
  
  # build a promise and store the resolve function
  pr <- promises::promise(function(resolve, reject) {
    resolve_function <<- resolve
  })
  
  # build a function to increment the counter
  increment_counter <- function() {
    count <<- count + 1
    if (count >= resolve_at) {
      resolve_function(count)
    }
  }
  
  # return both the counter and the promise:
  list(increment_counter = increment_counter, promise = pr)
}

# demo #1 -----------------------------------------------------------------
my_counter <- async_counter(resolve_at = 1)
# check the promise:
my_counter$promise
# increment the counter
my_counter$increment_counter()
# check the promise
my_counter$promise

# demo #2 -----------------------------------------------------------------
my_counter <- async_counter(resolve_at = 1)
my_counter$promise$then(function(value) {
  cat("promise resolved, value:", value)
})
my_counter$increment_counter()

I hope it will help.

from crrri.

bakaburg1 commented on June 3, 2024

Hello, I may have found a solution:

get_website_resources <- function(url, url_filter = '*', type_filter = '*', wait_for = 20) {

  out <- new.env()

  out$l <- list()

  perform_with_chrome( function(client) {
    Fetch <- client$Fetch
    Page <- client$Page

    client$inspect()

    Fetch$enable(patterns = list(list(urlPattern="*", requestStage="Response"))) %...>% {
      Fetch$requestPaused(callback = function(params) {

        if (str_detect(params$request$url, url_filter) & str_detect(params$resourceType, type_filter)) {

          Fetch$getResponseBody(requestId = params$requestId) %...>% {
            resp <- .

            if (resp$body != '') {
              if (resp$base64Encoded) resp$body = base64_dec(resp$body) %>% rawToChar()

              body <- list(list(
                url = params$request$url,
                response = resp
              )) %>% set_names(params$requestId)

              out$l <- append(out$l, body)
            }

          }
        }

        Fetch$continueRequest(requestId = params$requestId)
      })
    } %...>% {
      Page$navigate(url)
    } %>% wait(wait_for)
  })

  out$l
}

This does the trick but is not very efficient (20s is a lot but you need a good margin). Is there a way to resolve the promise instead based on a condition?
In the javascript world, I would trigger an event if the correct number of resources is catcher, but I have no idea how to do it with R promises.

Thank you

from crrri.

bakaburg1 commented on June 3, 2024

Thank you! it took me a while but I was able to more or less understand the logic of what you did and integrate it in my function. My problem was to stop a promise started by the crrri client.
I'm not sure if what I did was the most efficient solution: I enclosed all the async calls in a promise which also exports a resolve function in the parent environment. Once the conditions are met, the resolve function is called, closing the open async calls.

get_website_resources <- function(url, url_filter = '*', type_filter = '*', wait_for = 15, n_of_resources = NULL) {

  crrri::perform_with_chrome(function(client) {
    Fetch <- client$Fetch
    Page <- client$Page

    client$inspect() # not needed, just to check what's going on

    out <- new.env()

    out$results <- list()
    out$resolve_function <- NULL

    # wrap everything into a promise which produces a resolution function
    out$pr <- promises::promise(function(resolve, reject) {
      out$resolve_function <- resolve

      Fetch$enable(patterns = list(list(urlPattern="*", requestStage="Response"))) %...>% {
        Fetch$requestPaused(callback = function(params) {

          if (str_detect(params$request$url, url_filter) & str_detect(params$resourceType, type_filter)) {

            Fetch$getResponseBody(requestId = params$requestId) %...>% {
              resp <- .

              if (resp$body != '') {
                if (resp$base64Encoded) resp$body = base64_dec(resp$body) %>% rawToChar()

                body <- list(list(
                  url = params$request$url,
                  response = resp
                )) %>% set_names(params$requestId)

                #str(body)

                out$results <- append(out$results, body)

                if (!is.null(n_of_resources) & length(out$results) >= n_of_resources) out$resolve_function(out$results)
              }

            }
          }

          Fetch$continueRequest(requestId = params$requestId)
        })
      } %...>% {
        Page$navigate(url)
      } %>% crrri::wait(wait_for) %>%
        then(~ out$resolve_function(out$results))

    })

    out$pr$then(function(x) x)
  }, timeouts = max(wait_for + 3, 30), cleaning_timeout = max(wait_for + 3, 30))
}

from crrri.

Logic for returning results from a function that get XHR calls response body about crrri HOT 3 OPEN

Comments (3)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent