Capture multiple series simultaneously? about fredr HOT 5 CLOSED

Looks like this may have been what I was looking for re capturing multiple series at once:
#16

from fredr.

Yea for the multiple series question I would just purrr::map_dfr(series_ids, fredr)

There is no way to bulk get series from FRED in a single API hit

from fredr.

similarly i think you can purrr::map_dfr(my_tags, ~fredr_category_series(my_category, tag_names = .x))

from fredr.

Thank you, I was most curious about the bulk series download from a single API hit - so that answers my question.

I'd like to note that when running a bulk capture of data I would often hit the rate limit of 120 hits/min, which would prevent me from pinging the API again for 20 seconds. This would often occur even at runs to capture 10 series, where I am not sending a total of >=120 requests, so I assumed that the rate of requests itself was being detected. I could only assume that the loop I was using was iterating through at a rate faster than 120/min and spending a significant amount of time in delay.

To avoid this, I simply added a 'pause' function that will ensure the loop will not ping the API at a rate faster than 120 hits/min. It seems to have increase the speed of iteration. This does improve to appear my runtime when pulling many series, but I didn't do extensive profiling, so correct me if this doesn't make sense.

Define a Function to pause the for loop:

pause <- function(x){
p1 <- proc.time()
Sys.sleep(x)
proc.time() - p1 # The cpu usage should be negligible
} # use ~ value slightly greater than 0.5 seconds

Here is the function I have been using to iterate through the series:

Define function to iterate through the series ID variables and return one long data.frame .
IDs.toDF = function(ids){
bigdata = data.table()
for (i in ids){
singleObs = fredr(i)
bigdata = rbind(bigdata, singleObs)
pause(0.53) # avoid hitting rate limit and incurring 20 sec. pause
}
bigdata = dcast(bigdata, date~series_id, fill = NA)
return(bigdata)
}

Run list of series and get results
df = IDs.toDF(seriesIDs)

Above is the function I was using to iterate through the list of series and capture observations. I hadn't seen the purrr method before building this, but I like your purrr solution a lot. From a purely computational perspective it's obviously superior. I'm curious how you see the purrr solution playing out with the API rate limit. Does the vectorizion increase the frequency the API is being pinged and thus lead to delays?

from fredr.

Thanks for taking the time to document this issue, @jsolson4.

You can pass a custom function to most of the purrr methods to induce timing of the requests to comply with the API limitations. Just wrap the logic inside your for loop a function and pass that function to the purrr method.

from fredr.