Possibly related to the other issue:

> x = WDIbulk()
trying URL 'https://databank.worldbank.org/data/download/WDI_csv.zip'
Content type 'application/x-zip-compressed' length 65341191 bytes (62.3 MB)
===========
downloaded 14.6 MB

Error in utils::download.file(url, temp) : 
  download from 'https://databank.worldbank.org/data/download/WDI_csv.zip' failed
In addition: Warning messages:
1: In utils::download.file(url, temp) :
  downloaded length 15286272 != reported length 65341191
2: In utils::download.file(url, temp) :
  URL 'https://databank.worldbank.org/data/download/WDI_csv.zip': status was 'Failure when receiving data from the peer'

Hello,

I'm trying to download Social protection data using WDI R package. The WDIsearch("Social protection") gives more than 140 social protectiojn indicator but only one of them works. ("IQ.CPA.PROT.XQ"). Other indicators such as ""per_allsp.adq_ep_preT_tot" , "per_allsp.adq_ep_tot" "per_allsp.adq_pop_preT_tot", along the list don't work. here is the code as well as the error message i got:

HIC<- WDI(indicator= c("per_allsp.adq_ep_preT_tot"),country= c("BH", "KW", "OM", "QA", "SA", "AE", "BN"), start=2000, end=2014)

Warning message:
In WDI(indicator = c("per_allsp.adq_ep_preT_tot"), country = c("BH", :
Unable to download indicators perallsp.adqeppreTtot

Many thanks for your help

All the best
hicham

I was looking for the poverty headcount ratios that WB recently changed. Even though API returns up-to-date data, the data/WDI_data.RData file has old names, for example showing, $1,25 poverty line instead of the current $1.90. Any plans to update that file?

Thanks,
Markus

Links initially posted by @etiennebacher

For reference, here are a few links that could give ideas for new features:

• New features in V2
• Advanced Data API Queries (BETA), to get the dataset as it existed in 1997 for example.
• Other World Bank APIs (this is maybe out of the scope of WDI, but it gives ideas).

Notes on API / Call structure:

Example:

http://api.worldbank.org/en/countries/ind;chn/indicators/DPANUSSPF?date=2002M01:2008M08

Questions:

• WDIsearch(): How to find a list of series that include quarterly/monthly data? How to distinguish those from yearly data?

If you search for more elements than those extant in the database
WDIsearch('industrial')[1:13,]

Not a bug, just a question, why if you download any indicator in csv format you get the country code with three letters, but if you download it using your package the country code has just 2 letters?

According to the documentation, WDIcache() returns a data frame

I actually get a list back but created a data.frame, latestInfo

However, when I use the data in WDI I get mixed results

choice <- latestInfo$indicator[1] # "1.0.HCount.1.25usd"

data <- WDI(country = "all", 
       indicator = choice, 
       start = 1980, 
       end = 2015)

works but with this as input

choice <- latestInfo$indicator[10000] # "per_si_allsi.avt_q4_rur"
Warning message:
In WDI(country = "all", indicator = choice, start = 1980, end = 2015) :
Unable to download indicators  persiallsi.avtq4rur

Many WDI indicators are not calculated every year for every country, and they are updated at different times (see, for example, SP.POP.TECH.RD.P6). The World Bank wbopendata Stata module has a "latest" functionality, and I would love to see the same option for the R package. I've created a quick workaround in my code, but this requires pulling the last 5 years of data, ordering it by country and year, and removing duplicate country records.

library(WDI)
WDI(country='SZ', indicator='NY.GDP.MKTP.CD', extra = TRUE)

Returned data set has iso3c code for Swaziland missing.
Same codes are do exist in excel download from WDI site

Matthieu Stigler

Submitted code via email:

WDIsearch <- function (string = "gdp", field = "name", short = TRUE, cache = NULL)
{
if (field %in% colnames(WDI_data$series)){
smatrix = "series"
out_col = c("indicator", "name")
} else if (field %in% colnames(WDI_data$country)){
smatrix = "country"
out_col = c("iso2c", "country")
} else {
stop("Invalid field name.")
}

if (!is.null(cache)) {
series = cache[[smatrix]]
} else {
series = WDI_data[[smatrix]]
}
matches = grep(string, series[, field], ignore.case = TRUE)
if (short) {
out = series[matches, out_col]
}
else {
out = series[matches, ]
}
return(out)
}

In PR #48 @etiennebacher writes:

Question about defaults

The observation status is particularly useful when we import a particular indicator without specifying the dates. On the World Bank API, when no dates are specified, the whole serie is returned. E.g for this indicator, the data goes until 2023, which is why knowing which observation is a forecast is helpful. However, the behavior of WDI() regarding dates is a bit strange:

• if I don't specify any start and end dates, the data will be downloaded from the year of creation of the indicator (1960 in most cases) to 2020

WDI("CHN", "NYGDPMKTPKDZ")
   iso2c country NYGDPMKTPKDZ year
1     CN   China          2.3 2020
2     CN   China          6.0 2019
...
21    CN   China           NA 2000
22    CN   China           NA 1999

• if I specify end = NULL then it gets until the most recent year

> WDI("CHN", "NYGDPMKTPKDZ", end = NULL)
   iso2c country NYGDPMKTPKDZ year
1     CN   China          5.3 2023
2     CN   China          5.4 2022
3     CN   China          8.5 2021
4     CN   China          2.3 2020
...
24    CN   China           NA 2000
25    CN   China           NA 1999

• if I specify start = NULL and end = NULL, then it errors

> WDI("CHN", "NYGDPMKTPKDZ", start = NULL, end = NULL)
Error in WDI("CHN", "NYGDPMKTPKDZ", start = NULL, end = NULL) : 
  Need to specify dates or number of latest values.

So I'm wondering if it would be better to change the default values for start and end, and to change the default behavior. To me, both start and end should be NULL if I don't specify anything, and if they are NULL then it should get all years, even after 2020. I don't know if this would be a breaking change in the code of other people. What do you think?

Via email:

Hi Vincent Arel-Bundock,
Congratulation for the WDI package.
While reproducing the Readme routine, I found a small inconsistency relative to the NY.GDP.PCAP.KD variable.
In the WDIsearch('gdp.*capita.*constant') line, the result is "GDP per capita (constant 2010 US$)",
but when I download the series, the result shows (constant 2015 US$).
I checked the https://data.worldbank.org/indicator/NY.GDP.PCAP.KD and it is in fact 2015.
Thanks
Adriano

From an email:

Vincent,
I got the following with a WDIsearch:

WDIsearch("Stock Markets")
indicator name
[1,] "DSTKMKTXD" "Stock Markets, US$"
[2,] "DSTKMKTXN" "Stock Markets, LCU"

And replaced the formulas (which did not produce values in the earlier emails) with the above numbers, and produced monthly values.

http://api.worldbank.org/countries/chn;bra/indicators/DSTKMKTXD?date=2009M01:2010M08

Abbreviated example:

<wb:data xmlns:wb="http://www.worldbank.org" page="1" pages="1" per_page="50" total="40">
wb:data
<wb:indicator id="DSTKMKTXD">Stock Markets, US$,,,/wb:indicator
<wb:country id="BR">Brazil/wb:country
wb:date2010M08/wb:date
wb:value98.14421329725/wb:value
wb:decimal0/wb:decimal
/wb:data
wb:data
<wb:indicator id="DSTKMKTXD">Stock Markets, US$,,,/wb:indicator
<wb:country id="BR">Brazil/wb:country
wb:date2010M07/wb:date
wb:value93.99118676983/wb:value
wb:decimal0/wb:decimal
/wb:data
wb:data
<wb:indicator id="DSTKMKTXD">Stock Markets, US$,,,/wb:indicator
<wb:country id="BR">Brazil/wb:country
wb:date2010M06/wb:date
wb:value90.86708797275/wb:value
wb:decimal0/wb:decimal
/wb:data
wb:data
<wb:indicator id="DSTKMKTXD">Stock Markets, US$,,,/wb:indicator
<wb:country id="BR">Brazil/wb:country
wb:date2010M05/wb:date
wb:value89.5377288438/wb:value
wb:decimal0/wb:decimal
/wb:data

Christine Zhang
@ChristineZhang

@Vincentab FYI, i think the longitude/latitudes from 'extra=TRUE' in your WDI package are of the capital, rather than the country

Bonjour / Hi,

First, let me say that WDI is a GREAT package.

I had to reinstall my ubuntu system (now 13.04), including R. I have now R 3.0.1 installed. After installing the WDI package, I now run into an error if I attempt to run WDIcache(): wdicache k$id : $ operator is invalid for atomic vectors.

This is a new behaviour.

Now, because I already have a cache, the rest of the functions seem to work.

Thank you for giving attention to this issue.

J'ai jeté un coup d'oeil et y'aun ou deux trucs qui clochent. Il semble que XML soit toujours chargé au lieu de RJSONIO. Sinon, j'ai pas réussi à faire marcher les examples. Il semble que JSON arrive avec des valeurs nulles pour VALUE, ce qui crée un problème:
daturl<-"http://api.worldbank.org/countries/all/indicators/AG.AGR.TRAC.NO?date=1990:2000&per_page=25000&format=json"
dat     = fromJSON(daturl)[[2]]
dat[[2720]]$value

un petit workaround ( au cas ou on veut pasgarder des valeurs nulles):
dat_noNULL<-unlist(lapply(dat, function(i) is.null(i$value)))
dat     = data.frame(do.call('rbind', lapply(dat[dat_noNULL], function(j) cbind(j$country[[1]], j$country[[2]], j$value, j$date))))

Hi Vincent,
Only just learnt of this very useful library. It would be great if country = (...) could be extended to allow a vector of names in lieu of ISO2 code. Eg country = c("New Zealand", "Australia", ...).
Kind regards,
Pete McMillen

This is true even after updating the WDIcache

Can we tell ahead of time and strip those series from search results?

Hi
First of all thank you for this package.
This is not really an issue but a proposed (small) improvement in the code.
In WDI.R at wdi.dl function at line 67 you wrote

dat = data.frame(do.call('rbind', dat))

And just after you run a for loop to turn factors to characters.
But you can just do

dat = data.frame(do.call('rbind', dat), stringsAsFactors = FALSE)

regards,

I was using your R package to export subsets of data by income levels of countries to Excel as I was not finding Stata helpful for this.

However, I noticed differences in the data itself. Using the following in Stata

wbopendata, indicator(SP.POP.TOTL) long clear latest
list if countrycode == "NPL"

results in

With R::WDI

df <- WDI::WDI(country = "all", indicator = "SP.POP.TOTL", latest = 1, extra = TRUE)
df[df$iso3c == "NPL",]

results in

So year and population numbers match. Nepal had 29,136,808 inhabitants in 2020. However, using Stata I am told that Nepal is a "lower middle income" country while using the R tool I am told Nepal is "lower income".

What explains this behaviour? Am I doing something wrong?

P.S.: Otherwise and of course, thanks for the package! :)

Can WDI library use the iso3c code in the WDI function?

Have you considered updating the default end year for WDI? It is currently set to 2005 , but many of the series are updated through at least 2011.

I am trying to download per capital GDP (NY.GDP.PCAP.KD) for all countries. But the data are mostly blanks (except for region aggregates), while the WDI website has the data.

Hi Vincent,

I have written the following small function to get a few indicators from the WDI dataset:

WB_input <- function (year, countries) {
  dataSeries = c("DT.DOD.DECT.GN.ZS",
                 "DT.DOD.DECT.EX.ZS",
                 "DT.TDS.DECT.EX.ZS")
  data = WDI(
    indicator = dataSeries,
    country = countries,
    start = year,
    end = year
  )
  return(data)
}

and I executed it for a list of countries for 2018 yesterday, all worked fine. Today, I now get the following error message:

Error in 1:nrow(lab) : argument of length 0
In addition: Warning messages:
1: In WDI(indicator = dataSeries, country = countries, start = year,  :
  Unable to download data on countries:  ATA, ESH, GUF, RKS
2: In WDI(indicator = dataSeries, country = countries, start = year,  :
  Unable to download indicators  DT.DOD.DECT.GN.ZS ; DT.DOD.DECT.EX.ZS ; DT.TDS.DECT.EX.ZS

Literally yesterday, all worked fine (except for the missing countries ATA, ESH, GUF, RKS, of course).

My question is if anything has changed in the API settings that you're aware of or that this is a problem that occurs when no connection to the API could be established. I see from your repo that there was no change in your code so I suspect that it might be something on the WDI side.

When attempting to extract data, the data comes duplicated for most developing countries.

For instance in the code below, Angola, Mozambique, and Congo Republic come duplicated. with instances containing ISO2 and others ISO3

dat <- WDI(indicator=c("DT.ODA.ODAT.GN.ZS","NY.GDP.MKTP.CD","DT.ODA.ODAT.GD.ZS",
                       "DT.ODA.ALLD.GD.ZS","DT.ODA.ODAT.XP.ZS","DT.ODA.ALLD.XP.ZS","NY.GNP.MKTP.CD"),
           country = c("AO","CG","SA","MZ"), start=1980, end=2020)

but it seems like some indicators are assigned to different ISO codes as the code below does not return duplicates for any of those countries.

dat <- WDI(indicator=c("DT.ODA.ODAT.GN.ZS"),
           country = c("AGO","CG","SA","MZ"), start=1980, end=2020)

The following calls:

WDIsearch('NET.user')
a=WDI(indicator='IT.NET.USER', country='all', start=2010, end=2017)
b=WDI(indicator='IT.NET.USER.P2', country='all', start=2010, end=2017)
c=WDI(indicator='IT.NET.USER.P3', country='all', start=2010, end=2017)

yield an error message for the third call (IT.NET.USER.P2):

Unable to download indicators IT.NET.USER.P2NULL

in R version 3.3.2 (2016-10-31), RStudio version 1.0.136 on OS X.
a and c will contain values up to 2012 only, where all values in 2012 are NA.

Checking https://data.worldbank.org/indicator/IT.NET.USER.ZS?view=chart reveals that there is data available for many countries up until 2016.

Generate a nicely formatted rst file with codebook info about a list of indicators

Hello,

Using the WDI command, is it possible to get a certain older data release, instead of the latest ones. For example, I need the April 2022 release for a project, and not the July 2022 release. I was not clear on how to do this

Thanks

Hi Vincent. I've been using your WDI package in my classes. I've found that sometimes the regions for countries (extra = T) have trailing spaces. This was the source of a bit of confusion in two cases for my students. One example is in subsetting with a Boolean (region == "Latin America ...")). If you don't include the trailing space, then no rows are returned. Do you think that you could do a cleaning step (str_trim()) for regions? Great package!

Following #35, I ran:

library(remotes)
remove.packages(c('RJSONIO', 'WDI'))
install_github('vincentarelbundock/WDI')
library(WDI)

I would like to download region names as well and added extra = TRUE:

WDI(country = "FRA", indicator = "NV.AGR.TOTL.ZS", extra = TRUE)

This worked before (I used it three months ago but not until today), but now the column region provides some code like Z7 instead of Europe & Central Asia. Is it possible to fix it?

Thanks for the nice package.

Hi Vincent,

I saw a similar issue has been open on Git before, however I couldn't find the solution to my problem, as I think it is slightly different: in my case I could download most of the indicators I wanted, except for these ones specifically. The common issue with the previous post is that it worked also fine yesterday I could retrieve all indicators.

I also tried to run the code you posted on this post, but I get a similar error where only one of the 3 series could not be downloaded... (These indicators could not be downloaded: DT.DOD.DECT.EX.ZS)

Do you know what the issue might be?

country <- c("AU", "AT", "BE")
indicator <- c("FR.INR.RINR",
"FR.INR.DPST",
"FR.INR.LNDP",
"FR.INR.RISK",
"FR.INR.LEND",
"FM.AST.NFRG.CN",
"NY.GNS.ICTR.ZS",
"NY.ADJ.NNAT.GN.ZS",
"NY.ADJ.NNAT.CD",
"SP.DYN.LE00.IN",
"SP.POP.1564.TO.ZS",
"SP.POP.65UP.TO",
"SP.POP.DPND.OL",
"SP.POP.DPND",
"FP.CPI.TOTL.ZG",
"NY.GEN.SVNG.GD.ZS",
"NY.GEN.NDOM.GD.ZS",
"NY.GDP.MKTP.KN.87",
"BN.CUR.GDPM.ZS",
"BN.CAB.XOKA.GDP.ZS",
"6.0.GDP_growth")

wi_data <- WDI(
country = country,
indicator = indicator,
start = 1960,
end = 2020,
extra = FALSE,
cache = NULL,
latest = NULL,
language = "en"
)

Many thanks for your help!

Hi, I was wondering if you plan on submitting WDI to rOpenSci. This would allow to have a proper peer-review (along with other advantages), though the package is quite simple and works very well. This would take some additional time if there are some modifications needed after the review, but I think that having a package supported by rOpenSci is a great indicator of reliability for users.

Of course, WDI is already very good as-is (and thanks for having built it!). It was just something crossing my mind while checking the list of rOpenSci packages.

However, I'm having problems with pulling the data for East Asia & Pacific (4E) and Europe & Central Asia (7E). I've pasted the code below, in case that helps. Do you know what I'm doing that's wrong, or is it a bug? The other developing area codes (e.g. for SSA, LAC and MNA) work just fine. I'm wondering whether it's something to do with the WB code starting with a number?

EAP<-WDI(country="4E", indicator="NY.GDP.MKTP.KD.ZG", start=2000, end=2008)
Error in WDI(country = "4E", indicator = "NY.GDP.MKTP.KD.ZG", start = 2000, :
No valid country was requested
In addition: Warning message:
In WDI(country = "4E", indicator = "NY.GDP.MKTP.KD.ZG", start = 2000, :

Maybe it's unique to me, but every time I run WDIbulk() there's an error reading Country.Name. It always reads as "ï..Country.Name". It's pretty easy to fix for myself, but was wondering if others have seen this issue and if there's a way to fix it short of just rewriting the function (and/or just writing your own code to download and unpack the indicators.

Hi Vincent,
I just stumbled upon one of your comments on the wbstats package repo, which talks about deprecating WDI and orienting users towards wbstats. Since the comment was almost two years ago, I was wondering if this is still what you think you're gonna do, or if you will push new features to CRAN.
Personally, I discovered WDI before wbstats and I'd like to stick to it (unless there's a game changer feature in wbstats).
I am also asking this because I'm thinking of adding new features to WDI (such as the World Bank Projects API), unless you choose to deprecate the package.

Test with documentation example:

# Rename indicators on the fly
WDI(country = 'CA', indicator = c('women_private_sector' = 'BI.PWK.PRVS.FE.ZS',
                                  'women_public_sector' = 'BI.PWK.PUBS.FE.ZS'))

iso2c country year BI.PWK.PRVS.FE.ZS BI.PWK.PUBS.FE.ZS
1    CA  Canada 2000                NA                NA
2    CA  Canada 2001                NA                NA
3    CA  Canada 2002                NA                NA
4    CA  Canada 2003                NA                NA
5    CA  Canada 2004                NA                NA
6    CA  Canada 2005                NA                NA
...

Tested with last CRAN and GitHub versions (2.7.3)

data <-WDIsearch(string='AG.AGR.TRAC.NO', field='indicator', cache=NULL, short=FALSE)
class(data)
[1] "matrix"

Hi...
This code grabs data correctly
WDI::WDI(country = "all", indicator = c("GE.EST", "RQ.EST"),
start = 2000, end = 2020)

However when i change the start year, i receive an error.
WDI::WDI(country = "all", indicator = c("GE.EST", "RQ.EST"),
start = 1990, end = 2020)

None of the indicators your requested could be downloaded. Please verify the arguments of the WDI() function. You can also type a URL of this form in your browser to check if the World Bank web API is currently serving the indicator(s) of interest: https://api.worldbank.org/v2/country/all/indicator/GE.EST?format=json&date=1960:2020&per_page=32500&page=1

Please note that the data before 1996 is not available (https://databank.worldbank.org/reports.aspx?source=1181&series=GE.EST)

Please Advise

Salut Vincent

Une petite ligne utile (127) au cas où une erreur pour argument field:

if(!field%in%colnames(series)) stop("Wrong value for argument 'field', should be in: ", paste(colnames(series), collapse=", "))

Permet d'avoir:

WDIsearch2("SI.POV.GINI", field="indicators")
Erreur dans WDIsearch2("SI.POV.GINI", field = "indicators") :
Wrong value for argument 'field', should be in: indicator, name, description, sourceDatabase, sourceOrganization

au lieu de:

WDIsearch("SI.POV.GINI", field="indicators")
Erreur dans grep(string, series[, field], ignore.case = TRUE) :
indice hors limites
Calls: WDIsearch -> grep

Salutations

Just one tiny remark: The examples you have in your current documentation have a "years" argument (which apparently is no longer used), so, it may be helpful for beginners to have working examples.

Could WDI package has some function or parameter that support to choose language other than English?

Here is a list of the aggregate categories:

aggregates <- c("Arab World", "Central Europe and the Baltics",
                "Caribbean small states", "East Asia & Pacific (developing only)",
                "East Asia & Pacific (all income levels)", "Europe & Central Asia (developing only)",
                "Europe & Central Asia (all income levels)", "Euro area",
                "European Union", "Fragile and conflict affected situations",
                "High income", "Heavily indebted poor countries (HIPC)",
                "Not classified", "Latin America & Caribbean (developing only)",
                "Latin America & Caribbean (all income levels)",
                "Least developed countries: UN classification",
                "Low income", "Lower middle income", "Low & middle income",
                "Middle East & North Africa (all income levels)",
                "Middle income", "Middle East & North Africa (developing only)",
                "North America", "High income: nonOECD", "High income: OECD",
                "OECD members", "Other small states", "Pacific island small states",
                "South Asia", "Sub-Saharan Africa (developing only)",
                "Sub-Saharan Africa (all income levels)", "Small states",
                "Upper middle income", "World")

Specifying extra = TRUE includes the column region, which indicates which row is an aggregate. Based on this, one can exclude aggregates by himself.

However, it would be convenient to have an option in the main command to exclude aggregates.