nutterb / redcapapi Goto Github PK

Most of the core code won't change, but I'm going through each of the functions and trying to tidy up the code, improve readability, add all of the currently available features (as of 6.5.0), optimize things as much as possible, and transition to using checkmate for as many error checks as possible.

API Functions

• exportRecords
• exportReports
• importRecords
• exportMetaData (still needs version logic)
• exportFieldNames
• exportFile
• importFile
• deleteFile
• exportInstruments (still needs version logic)
• exportPDF
• exportSurveyLink
• exportSurveyQueueLink (combine with exportSurveyLink with queue = TRUE?)
• exportSurveyReturnCode
• exportEvents (still needs version logic)
• exportArms
• exportMappings
• exportUsers (still needs version logic)
• exportProjectInformation (still needs version logic)
• exportVersion

And since I tend to be sloppy about documenting as I go, I want to make sure I do a separate checklist of writing the documentation with each function. Things that should be included in the documentation for each function:

1. @section REDCap Version: in which the API function was introduced
2. @section Known limitations: of the API (such as fields and forms not active in exportMetaData; or arms forcing empty data frames in exportArms), if any
3. @section REDCap Documentation: that gives, verbatim, the description of the action from the REDCap documentation page.

Documentation

• exportRecords
• exportRecords_offline
• exportReports
• importRecords
• exportMetaData
• exportFieldNames
• exportFile
• importFile
• deleteFile
• exportInstruments
• exportPDF
• exportSurveyLink
• exportSurveyQueueLink (combine with exportSurveyLink with queue = TRUE?)
• exportSurveyReturnCode
• exportEvents
• exportArms
• exportMappings
• exportUsers
• exportProjectInformation
• exportVersion

Miscellaneous

• exportRecords_offline
• allocationTable
• allocationTable_offline
• parseBranchingLogic
• recodeCheck
• redcapFactorFlip
• Deprecate redcapDbConnection methods

Some more notes for myself:

1. fieldToVar should be a big switch statement. It would make sense for there to be unexported methods for each variable type.
2. The lapply that calls fieldToVar probably ought to be an mapply. Nope, changed my mind. This is begging for a for loop.
3. Factors that are created by redcapAPI should have a new class that inherits factor. That would make it easier to find them and toggle between labels and levels with redcapFactorFlip.
4. An additional option for how to handle errors would be good. For example, how to handle errors from exportEvents on classic projects, or any method that isn't supported by the version of REDCap. The options should be to return the error, or to return NULL
5. For convenience, I'm going to assign anything that happens before REDCap 6.0.0 to REDCap 5.12.2. The only only inefficiency I can think of that this will create is in deciding when to run syncUnderscoreCodings, since those were fixed in 5.5.21. That's pretty low computational overhead, though, so I'm willing to live with it in order to have the benefit of exportVersion always returning something that can be compared.

Hi @nutterb,
In a recent experiment I realized that disabling the factors flag in exportRecords results in some unexpected warnings when attributing labels. This leads to NA's appearing during data processing, with a bunch of:

In print.factor(x) : factor levels must be "character"

or

In class(x) <- class(x)[!class(x) %in% c("labelled", "redcapFactor")] :
  NAs introduced by coercion

This seems to happen rather late during the export, around here in makeRedcapFactor(). before giving that class, just printing the content of x returns its untouched value. After allocation of this class, it starts failing, with R somehow mistaking it for an actual factor ?

This puzzles me a bit because there's no reason for R to interprete x as a "factor" here.

edit: This may actually be caused by the call to print.factor() in the print.redcapFactor() function. When x is not a factor, obviously print.factor() fails with a warning.

We are finding that text field where no data has been entered in a form for that field, are returned as logical instead of the correct field type from the metadata.

Once data are entered in the field on any record, the export correctly returns the field as chr.

Factor fields, even when blank on the entire form, are returned correctly, as are date fields.

Attached are screenshots for redcapR and REDCapAPI. Help!!
scrREDCapExtractions-20220718.pdf

From an e-mail I received in January 2020

Also, I'm not sure how much you investigated the exportUsers() error, but I ran into it with one of my studies. The problem was that the column containing form access permissions only contains form names for forms which that user has some level of access to... I think. I didn't 100% verify, but rbind was definitely kicking an error because the list of DFs had different numbers of columns where each column was a form name.

• importArms
• deleteArms
• importEvents
• deleteEvents
• importMetaData
• createProject
• importProjectInfo
• generateNextRecordName
• deleteRecords
• exportSurveyParticipants
• importUsers

Please,
Can someone help me? I'm trying to export the Survey Participants Lists with the survey link but this error appears:
Error in checkmate::assert_data_frame(x = arms_data, ncols = 2, add = coll) :
argument "arms_data" is missing, with no default

Thanks

How do we use the importRecords so that we can have REDCap import the record id's with what's available? Basically I need to use the forceAutoNumber parameter of the REDCAP API.

I keep getting errors that there is no file for the record when I know there is one.

Hello,

I keep getting the error message below when using importRecords(). The data get successfully imported into R but the error persists. This only occurs when importing a larger # of records (~5,000). Is there a maximum number of records the action can handle?

Code used

importRecords(rcon_tests,data=updateFac,overwriteBehavior = c("normal"),
              returnContent = c("count"), returnData = FALSE,
              logfile = "")

Error

Error in redcap_error(x, error_handling = "error") : 504: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

Hi there -- so appreciate this library! Hoping this is the right place to ask this:

On first glance, the missingSummary ancillary function for redcapAPI works perfectly. However, when cross-checking true missingness with REDcap open, there are a number of branching logic values where conditions are met, missingness still occurs, but the code doesn't report the missing value. Is it possible this is related to one of the options in the exportRecords function being the wrong value or perhaps the function isn't pulling in all available data?

Here's a link to the missingSummary gist: https://gist.github.com/nutterb/501c370418abb58bee78

A week ago, my code (which had been working perfectly) stopped working. When I run the exportRecords function, I get the error

Error in redcap_error(x, error_handling = error_handling) : 
  400: ERROR: An unknown error occurred. Please check your API parameters.

I suspect a RedCap update caused this, but I'm not sure. What's going wrong here?

Thanks for this REDCap API interface library for R, it's proven pretty useful already. At the moment we're converting research data from an older system to REDCap projects. We have created a REDCap dictionary based on the old system and exported the data as csv files.

Instead of importing the files one-by-one through the REDCap web user interface (too many files, too slow, freezes etc) we're trying to import the data using R scripts. This works for most files, except it doesn't work for certain decimal values. Specifically, for columns that contain numeric 1 decimal values with trailing zeros, we get this error:

The value you provided could not be validated because it does not follow the expected format. Please try again.

This only happens for variable when Validation = Number 1 decimal, and the values contain trailing zeros like 5.0 10.0 etc, but there is no error message for other decimal values like 2.5 9.3 etc. See the code below, it reads a CSV file with data, builds a dataframe and send the dataframe to the Redcap API.

library(redcapAPI)
library(dplyr)

# Set workdirectory
setwd('C:/Users/XxxxxX/source/ConvertStudydata')

# config API-token + REDCap url
api_url = "https://redcap.xxxxxxxx.xxx/redcap/api/"
api_token = "1234567890ABCDEF1234567890ABCDEF"

# Setup API Connection
rconnection <- redcapConnection(url = api_url, token = api_token)

recordId_colClasses <- c("studynr"="character", "mutation_date"="Date")
csvALGEMEEN <- read.csv('./Data/Import_ALGEMEEN.csv', sep = ',', dec='.', fileEncoding = 'UTF-8-BOM', colClasses = recordId_colClasses)
dfALGEMEEN <- importRecords(rconnection, csvALGEMEEN, returnData = FALSE, logfile = "import_logfile_ALGEMEEN.txt")

The csv file contains values like 20.0, 5.0, 10.0 etc. and importing it into a dataframe in RStudio using read.csv(.. the dataframe still contains the correct values, see screenshot.

However, when I try to import the dataframe using the importRecords function I get the error message:

dfALGEMEEN <- importRecords(rconnection, csvALGEMEEN, returnData = FALSE, logfile = "import_logfile_ALGEMEEN.txt")
Error in redcap_error(x, error_handling = "error") :
400: ERROR: "20240012","rookaant","10","The value you provided could not be validated because it does not follow the expected format. Please try again."
"20240014","rookaant","20","The value you provided could not be validated because it does not follow the expected format. Please try again."
"20240017","rookaant","5","The value you provided could not be validated because it does not follow the expected format. Please try again."

The error message only appears for the values ending on .0 decimals, it doesn't give any error on the values 8.1 or 5.2 etc. I also tried to force the CSV reader to interpret the column as character but then I still get the same error when importing. The variable in REDCap is defined as "Number (1 decimal)", see screenshot below:

When I import the same file in the REDCap web gui then I don't get the error message. Any idea how to import these decimal values using R and the API library? Is this a bug in the library maybe, or are there extra parameter I need to set? fyi I'm using REDCap v12.4.6 and Rstudio 2022.07.0 Build 548

Wondering if you had an anticipated date for when you will push the package 2.3.1 version to CRAN? 2.3.1 fixed a bug that users are running into and they don't have build tools. Thanks for the work you do...

I'm compiling notes here because I have no better place to do it where I can keep track of it. It wouldn't hurt to have someone confirm the behavior. However we decide to handle these issues needs to be documented as well.

• In exportFieldNames, if you give a vector of field names, it returns "ERROR: Invalid field." But the REDCap documentation states that it will accept multiple fields. Until this gets fixed, I suspect the appropriate behavior would be to honor the fields argument only when it has length 1, and print a message that the entire field list is being returned until the REDCap API is corrected.
• exportFieldNames also doesn't return field names for calculated values, sliders, or signature draws. I think it probably should--note that REDCap disagrees. They document that the calculated fields, signatures, files, and descriptive fields are not included because they cannot be used in data imports. It would be nice to have an option to get these anyway, as it could help streamline the process of formatting variable names upon export. Then again, the time spent sending the request to the API is probably longer than the time spent parsing things through regex, so I probably won't ask them to change this.
• In exportArms, if you use the arms argument, the API returns an empty data frame
• In exportEvents, if you use the arms argument, the API returns an empty data frame

When a field is specified within a particular arm in branching logic (e.g. [screening_arm_4][co_enrol_report]), parseBranchingLogic concatenates these as 'screening_arm_4co_enrol_report' with the result that attempting to evaluate the rule (e.g. with(allrecords, eval(l)) ) returns an object not found error.

I'm looking for a way to lock/unlock records in bulk using the API. I found this similar project - https://github.com/lsgs/redcap-locking-api - but cannot find the functionality in the R API. Has anyone explored this?

Hi! I'm trying to create an allocation table with the following code, taken from this Wiki page:

Randomize <- allocationTable(rcon, random="Randomization",
strata=NULL,
dag.id=c("[X]", "[Y]"),
replicates=200,
block.size=c(8, 4, 2),
block.size.shift = c(0, .48, .9),
seed.dev=205, seed.prod=1510,
bundle=proj)

No matter what I do with "random" (leave it blank, put random=NULL, or what I did above, which is to put the field name of the variable we want randomized), I get the following error:

Error in allocationTable(rcon, random = "Randomization", strata = NULL, dag.id = c("[X]", :
1 assertions failed:

• Variable 'random': Must be a subset of the empty set, i.e. also empty.

Could someone explain to me (in simple terms, please, I'm not a mathematician or coder) what this means and how to fix it? Thanks!

Hi, I am trying to export data from REDCap to R using the exportRecords() function. While most of the data are fully exported, two variables appear to be only partially exported, as in the data populated for these variables is incomplete. Specifically, these variables are populated for nearly all patients in REDcap, but missing for most (but not all) patients in the exported data. Have you ever seen this before? Or do you have any suggestions on how to approach this? Thank you!

We can alleviate this by building in a check for duplicate identifiers and casting an error when it occurs. "Duplicate identifiers" will need to account for record_id and redcap_event_name. Are there other identifiers (that will appear conditionally) that we need to account for?

Nothing comes to mind, but I don't want to code something that can't account for additional identifiers.

I notice that longer API calls have been disconnecting. Our project has ~2,500 variables. A record JSON is ~6 MB. When I try to upload the record to REDCap using API, REDCap disconnects yielding the following error:

Traceback (most recent call last):
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/miniconda3/envs/dpimport/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/miniconda3/envs/dpimport/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/miniconda3/envs/dpimport/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/miniconda3/envs/dpimport/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

I have found that some smaller forms upload successfully. But breaking the above record down to smaller forms does not work for us as we have multiple large forms e.g. PSYCHS, SCID-5, and FIGS.

What do you think I can do to remedy the above?

I think this has been renamed to "api_export" in the actual API.

@nutterb, if you ever have bureaucratic obstacles getting access to a REDCap server, please tell us. Some processes changed recently that would make it easy for us to give you an account and any projects you wanted for testing.

I've been working on getting access to REDCap again. I got access a couple of weeks ago, but am just now getting to using it.

I haven't tried this myself yet, but I'm about to try now. I wonder if doing the following might help:

library(redcapAPI)
rcon <- redcapConnection(url = '[your_url]',
                         token = '[your_token]',
                         config = list(encoding = 'UTF-8'))

_Originally posted by @nutterb in #27

I am exporting a report (using exportReports) with dag = TRUE. The redcap_data_access_group variable appears in the dataframe, but when I use importRecords to import the dataframe into another REDCap project (with the same DAGs), none of the records are being assigned a DAG. When I export the dataframe as a .csv and import it manually, the DAG is assigned.

Is there a parameter I need to add to importRecords to get the DAGs to assign correctly? Has anyone else had this issue?

Thanks!

Hi @nutterb

This is repeat of the error from issue 158:

Error in set_label.default(x[[nm]], lab) : 
  labels may not be added to `NULL` objects.

I tried to install the issue-158 branch, but it doesn't seem to print out the variables. I'm using the latest version:

> xfun::session_info('redcapAPI')
R version 4.0.0 (2020-04-24)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Locale:
  LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
  LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
  LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
  LC_PAPER=en_US.UTF-8       LC_NAME=C                 
  LC_ADDRESS=C               LC_TELEPHONE=C            
  LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

Package version:
  askpass_1.1       assertthat_0.2.1  backports_1.2.1   BH_1.75.0.0      
  checkmate_2.0.0   chron_2.3.56      cli_2.2.0         clipr_0.7.1      
  cpp11_0.2.6       crayon_1.3.4      curl_4.3          DBI_1.1.1        
  digest_0.6.27     dplyr_1.0.3       ellipsis_0.3.1    fansi_0.4.2      
  generics_0.1.0    glue_1.4.2        graphics_4.0.0    grDevices_4.0.0  
  hms_1.0.0         httr_1.4.2        jsonlite_1.7.2    labelVector_0.1.1
  lifecycle_0.2.0   lubridate_1.7.9.2 magrittr_2.0.1    methods_4.0.0    
  mime_0.9          openssl_1.4.3     pillar_1.4.7      pkgconfig_2.0.3  
  purrr_0.3.4       R6_2.5.0          Rcpp_1.0.6        readr_1.4.0      
  redcapAPI_2.3.1   rlang_0.4.10      stats_4.0.0       stringi_1.5.3    
  stringr_1.4.0     sys_3.4           tibble_3.0.6      tidyr_1.1.2      
  tidyselect_1.1.0  tools_4.0.0       utf8_1.1.4        utils_4.0.0      
  vctrs_0.3.6      

Hi Team,

I am getting below error while running redcap api to get pdf.

Error in function (type, msg, asError = TRUE) :
SSL: certificate subject name 'redcap-admproj' does not match target host name 'redcap-rsdm'

Hi,

I'm having a newfound error (Error in redcap_error(x, error_handling = "error") : 400:) when trying to import an R data frame to a REDCap project using importRecords(). I have successfully imported other data in this way today, but I'm having problems here with the largest data frame (~17,000 records with ~400 variables, some of which are long character fields). Thanks in advance.

Code is below:

api_url <- "https://global.redcap.unc.edu/api/"
api_key <- "[my token]"
rcon <- redcapConnection(url=api_url, token=api_key)

df <-read.csv("[my file]")

importRecords(rcon, data=df, overwriteBehavior = c("normal"), returnContent = c("count"), returnData = F)

I posted back in October regarding the exportRecords function no longer working, but after a day or two, it suddenly started working again. It worked most of the time, but I would occasionally still get the following error:

Error in redcap_error(x, error_handling = error_handling) : 
  400: ERROR: An unknown error occurred. Please check your API parameters.

After restarting my computer, this error would usually stop occurring. Sometimes I would have to wait, but it would usually resolve itself within a day or two. However, I just changed to a new PC and the error persists.

Why might this be happening? I change nothing in the code -- sometimes I get the error, other times I'm able to pull all the data with no issues.

Hi I'm getting an invalid sender error when I'm uploading the alerts & Notifications from one project to another. My email address is valid, and user permissions are the same across both projects. Any suggestions?

Hello,
When I tired to use function redcapAPI:::deleteRecords.redcapApiConnection, got error: 400: <?xml version="1.0" encoding="UTF-8" ?><hash><error>Incorrect format for 'records' parameter. It must be passed as an array, even if it only contains a single record name.</error></hash>.
I changed records name to records = array(c(9984, 9983), dim = c(2, 1, 1)), but still got the same error massage. Is there any way can fix this issue?

Thank you!

@nutterb, just a heads up. I've had two recent issues created that ended up being caused by curly quotes in the data dictionary. The solution involved being more explicit with the text & encoding passed to httr::content().

I know you use httr::POST() instead (and I like that there are two approaches available to the public) and I don't know if there's an analog for your approach. Maybe httr::POST() avoids it entirely.

If you start seeing things like these two REDCapR issues, it may be worth investigating. It may address redcapAPI's nutterb/redcapAPI#126 & nutterb/redcapAPI#129?

• OuhscBbmc/REDCapR#269
• OuhscBbmc/REDCapR#272