It says path not found. We need to fix this in other to upload correctly.

When I try to use an R file from the path (which incidentally isn't listed as an option in the documentation :D) I get the error:

Error in source(input_path) :
invalid multibyte character in parser at line 41

Following the installation guide I run:
devtools::install_github("lifecycle-project/ds-upload")

and get the following error:

Error: object ‘armadillo.assume_role_with_web_identity’ is not exported by 'namespace:MolgenisArmadillo'
Execution halted

devtools::session_info()
─ Session info ───────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.2 Patched (2022-11-10 r83330)
 os       Ubuntu 20.04.5 LTS
 system   x86_64, linux-gnu
 ui       RStudio
 language en_GB:en_US:nl
 collate  nl_NL.UTF-8
 ctype    nl_NL.UTF-8
 tz       Europe/Amsterdam
 date     2023-03-01
 rstudio  2022.12.0+353 Elsbeth Geranium (desktop)
 pandoc   NA

─ Packages ───────────────────────────────────────────────────────────────
 package     * version date (UTC) lib source
 cachem        1.0.6   2021-08-19 [1] CRAN (R 4.2.1)
 callr         3.7.3   2022-11-02 [1] CRAN (R 4.2.2)
 cli           3.6.0   2023-01-09 [1] CRAN (R 4.2.2)
 crayon        1.5.2   2022-09-29 [1] CRAN (R 4.2.1)
 curl          5.0.0   2023-01-12 [1] CRAN (R 4.2.2)
 desc          1.4.2   2022-09-08 [1] CRAN (R 4.2.1)
 devtools    * 2.4.5   2022-10-11 [1] CRAN (R 4.2.2)
 digest        0.6.31  2022-12-11 [1] CRAN (R 4.2.2)
 ellipsis      0.3.2   2021-04-29 [1] CRAN (R 4.2.0)
 fastmap       1.1.0   2021-01-25 [1] CRAN (R 4.2.0)
 fs            1.6.1   2023-02-06 [1] CRAN (R 4.2.2)
 glue          1.6.2   2022-02-24 [1] CRAN (R 4.2.0)
 htmltools     0.5.4   2022-12-07 [1] CRAN (R 4.2.2)
 htmlwidgets   1.6.1   2023-01-07 [1] CRAN (R 4.2.2)
 httpuv        1.6.9   2023-02-14 [1] CRAN (R 4.2.2)
 later         1.3.0   2021-08-18 [1] CRAN (R 4.2.1)
 lifecycle     1.0.3   2022-10-07 [1] CRAN (R 4.2.1)
 magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.2.0)
 memoise       2.0.1   2021-11-26 [1] CRAN (R 4.2.1)
 mime          0.12    2021-09-28 [1] CRAN (R 4.2.0)
 miniUI        0.1.1.1 2018-05-18 [1] CRAN (R 4.2.1)
 pkgbuild      1.4.0   2022-11-27 [1] CRAN (R 4.2.2)
 pkgload       1.3.2   2022-11-16 [1] CRAN (R 4.2.2)
 prettyunits   1.1.1   2020-01-24 [1] CRAN (R 4.2.0)
 processx      3.8.0   2022-10-26 [1] CRAN (R 4.2.2)
 profvis       0.3.7   2020-11-02 [1] CRAN (R 4.2.1)
 promises      1.2.0.1 2021-02-11 [1] CRAN (R 4.2.1)
 ps            1.7.2   2022-10-26 [1] CRAN (R 4.2.2)
 purrr         1.0.1   2023-01-10 [1] CRAN (R 4.2.2)
 R6            2.5.1   2021-08-19 [1] CRAN (R 4.2.0)
 Rcpp          1.0.10  2023-01-22 [1] CRAN (R 4.2.2)
 remotes       2.4.2   2021-11-30 [1] CRAN (R 4.2.0)
 rlang         1.0.6   2022-09-24 [1] CRAN (R 4.2.1)
 rprojroot     2.0.3   2022-04-02 [1] CRAN (R 4.2.0)
 rstudioapi    0.14    2022-08-22 [1] CRAN (R 4.2.1)
 sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.2.1)
 shiny         1.7.4   2022-12-15 [1] CRAN (R 4.2.2)
 stringi       1.7.12  2023-01-11 [1] CRAN (R 4.2.2)
 stringr       1.5.0   2022-12-02 [1] CRAN (R 4.2.2)
 urlchecker    1.0.1   2021-11-30 [1] CRAN (R 4.2.1)
 usethis     * 2.1.6   2022-05-25 [1] CRAN (R 4.2.1)
 vctrs         0.5.2   2023-01-23 [1] CRAN (R 4.2.2)
 withr         2.5.0   2022-03-03 [1] CRAN (R 4.2.0)
 xtable        1.8-4   2019-04-21 [1] CRAN (R 4.2.1)

 [1] /home/gcc/R/x86_64-pc-linux-gnu-library/4.2
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

It looks like [MolgenisArmadillo::assume_role_with_web_identity](https://github.com/lifecycle-project/ds-upload/blob/a4a1df00f9e719debb9708fec36bde47f11e2e90/R/backend_armadillo.R) produces this error.

[molgenis-r-armadillo](https://github.com/molgenis/molgenis-r-armadillo) update 2.0.0 seems to break things.

The function du.check.package.version is not used anymore (dsUpload will not be availlable through registry.molgenis.org) and should be removed. When users login with du.login() and the registry is down then dsUpload will fail to run.
Note. dsUpload installation relies on devtools::install_github("lifecycle-project/ds-upload")

• Add a project parameter
• Make the dict_kind project agnostic
• Do not hardcode the dictionary url

How to reproduce:
Reshape the randomized set of WP5 and you will see that the variable: FEV1FVC_z_ will not contain any data.

What was expected:
All data should be reshaped.

What was observed:
The variables is left out of the generated data.

The following variables are getting QCd out of the core non_rep: lden_preg, ln_preg, pm10_preg, nox_preg

The following variables are getting QCd out of the yearly rep: lden_, ln_, pm10_, nox_

Run the tutorial for opal (https://lifecycle-project.github.io/ds-upload/articles/dsUpload.html)
The first du.upload fails because the call is async. It will work if you try again.

The upload doesn't work; same error for either CSV and SAS file. The table only have trimester variables (no yearly) :

#EDEN
du.upload(

• cohort_id = 'eden',
• dict_version = '1_0',
• dict_kind = 'chemicals_ath',
• data_version = '1_0',
• data_input_format = 'SAS',
• data_input_path = 'G:/ATHLETE/athlete_expo_chemicals.sas7bdat'
• #,run_mode = "non_interactive"
• )
  ######################################################
  Start upload data into DataSHIELD backend

• Create temporary workdir
  ######################################################
  Start download dictionaries

• Download: [ 1_0_non_rep.xlsx ]
• Download: [ 1_0_trimester_rep.xlsx ]
• Download: [ 1_0_yearly_rep.xlsx ]
  Successfully downloaded dictionaries
  ######################################################
  Start importing data dictionaries
  ######################################################
• Check released dictionaries
• Project : eden already exists
  ######################################################
  Start converting and uploading data
  ######################################################
• Setup: load data and set output directory

• Generating: non-repeated measures
• Generating: yearly-repeated measures
  [WARNING] No yearly-repeated measures found in this set
  Error in write_delim(x, file, delim = ",", na = na, append = append, col_names = col_names, :
  is.data.frame(x) n'est pas TRUE
• Reinstate default working directory
• Cleanup temporary directory

• Core trimester
• Outcome monthly
• Outcome yearly

Follow the first part of the manual for armadillo without logging in (if something went wrong with the login the error will also occur and it's unclear what went wrong:

login_data <- [data.frame](https://rdrr.io/r/base/data.frame.html)(
  server = "https://armadillo.test.molgenis.org", 
  storage = "https://armadillo-minio.test.molgenis.org", 
  driver = "ArmadilloDriver")
du.upload(
  cohort_id = 'gecko', 
  dict_version = '2_1', 
  dict_kind = 'core', 
  data_version = '1_0', 
  data_input_format = 'CSV',
  data_input_path = 'https://github.com/lifecycle-project/ds-upload/blob/master/inst/examples/data/WP1/data/all_measurements_v1_2.csv?raw=true',
  run_mode = "non_interactive"
)

Error:

 * Check released dictionaries
Error in if (ds_upload.globals$login_data$driver == du.enum.backends()$ARMADILLO) { : 
  argument is of length zero

Expected something like:
Armadillo driver not specified in login_data. Please take a look at your login_data and login using du.login(login_data = login_data).

Consider the following code:

library(dsUpload)
armadillo_url <- "armadillo-demo.molgenis.net"

login_data <- data.frame(
  server = armadillo_url,
  driver = "ArmadilloDriver"
)

du.login(login_data = login_data)

This works as expected however next I want to upload some data:

du.upload(
  cohort_id = "inma",
  dict_version = "1_3",
  dict_kind = "outcome_ath",
  data_version = "1_1",
  data_input_format = "CSV",
  data_input_path = "<PATH>/ds-upload/inst/examples/data/demo/demo-athlete-outcome.csv",
  action = "all",
  run_mode = "NORMAL",
  upload = TRUE
)

DsUpload fails:

URL using bad/illegal format or missing URL

How to fix

Make sure the server url contains https://

Add a check in du.login() that will make sure the user is prompted to provide a valid url:

armadillo_url <- "https://armadillo-demo.molgenis.net"

Tested with dsUpload version: dsUpload_5.0.3

Less hacky code, with backward compatibility:

• to apply a data dictionary to a tibble (i.e. before saving it): https://www.obiba.org/opalr/reference/dictionary.apply.html
• to save a tibble as a opal table: https://www.obiba.org/opalr/reference/opal.table_save.html
• to update a table data dictionary (i.e. after a tibbla has been saved): https://www.obiba.org/opalr/reference/opal.table_dictionary_update.html

See also: https://opaldoc.obiba.org/en/latest/cookbook/import-data/r.html

When I try to specify a file path relative to my working directory:

Error: './data/alspac_all.csv' does not exist in current working directory ('/Users/timcadman/OneDrive - University of Bristol/repos/lc-alspac-upload/2021-03-11_16-56-50').

• Reinstate default working directory
• Be advised: you need to cleanup the temporary directories yourself now.

Maybe I'm doing something stupid? Ah, or maybe the function changes the working directory and then it can't find the file.

Session check fails and it states it cannot find the backend$ARMADILLO or OPAL properly. It should say login again.