trias-project / alien-species-checklist Goto Github PK

This can only be done once all the other issues have been resolved.

If yes, we can ignore the Annex B file in our combined checklist. @stijnvanhoey will do a join on scientificNames to see how many match.

We will add the following columns for species that occur in GRIIS and the concatenated file (based on identical acceptedKey):

• PresenceBE: one of our 7 columns
• Status: standardized
• Source: dataset name from which this info was derived
• GBIF acceptedKey
• Our taxonID ?

Order of sources to use info from:

• Plants
• Wrims
• Fishes
• Macroinvertebrates (all considered established)
• Harmonia (beware, not all present in BE)
• Rinse + annex: no status info

Matches

• Records in both lists considered validated
• Records in GRIIS only: are probably errors in one of both datasets
• Records in concatenated only: we will submit this as an additional file: potentially missing from GRIIS.

Create a scientificName for all records in concatenated where there currently isn't one. Since none of those have a subspecies, this can be done quite easily:

1. Filter on blank scientificName
2. Edit cells > Transform
3. value: cells["genus"].value + " " + cells["specificEpithet"].value

Make sure to escape quotes before importing the file, so we don't get this in taxonID:

plants "Populus x jackii Sargent ""Gileadensis"" (P. balsamifera x deltoides)" Salicaceae Nat.? Hort. D NAM X 2010 N?

In WRIMS, one can search on distributions: http://www.marinespecies.org/introduced/

There are 4 (sub)regions that might be applicable for a Belgian checklist:

• Belgian coast (Coast): 13 records
• Belgian Exclusive Economic Zone (EEZ): 55 records
• Belgian part of the North Sea (Marine Region): 55 records
• Belgium (Nation): 114 records

I verified and Belgium (Nation) includes the 3 other regions completely, so a search on Belgium (nation) is the one we should use.

Once we have #26:

1. Do a facet on presenceBE
2. Copy all the values and the number of times they occur
3. Paste this information in the relevant sheet in this spreadsheet
4. In the spreadsheet, indicate the field name in the column Field
5. Repeat this for the 6 other presence columns.

• Create one flat file (source format unknown)
• Transform to CSV
• Put CSV in https://github.com/LifeWatchINBO/alien-species-checklist/tree/master/source-datasets/wrims

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with macroinvertebrates
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

First scan & basic cleaning of the initial files

• Harmonia
• Rinse
• WRIMS
• Plants
• Macroinvertebrates
• Neobiota

Dear Tim
I would like you to help to define the main columns tittles (Darwin Core Terms) we would need for the checklist.

I try to do till the Assignment # 30. Peter, you can check the spread sheet for comments
For the following, I think, I'm not technically qualified for them; Right?
If I have to do them, I need to be at INBO, with one of you, Stijn, Peter or Dimitri
This is possible by next Monday. Or Stijn has already completed them?
Anyway, I would like to know how these have been done

Thanks

• Create one flat file (source format unknown)
• Transform to CSV
• Put CSV in https://github.com/LifeWatchINBO/alien-species-checklist/tree/master/source-datasets/harmonia

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with plants
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

Once we have #26:

1. Do a facet on introductionPathways
2. Copy all the values and the number of times they occur
3. Paste this information in the relevant sheet in this spreadsheet

This table gives an overview of the current GBIF name matching status:

Some observations:

• The first 3 categories can be considered OK, which is 93,3% of the dataset! The only caveat is that we have to trust the accepted names GBIF gives for synonyms (745 records + 15 doubtful), which we don't always do: e.g. Tripolium pannonicum is not a synonym of A. salignus
• The 96 EXACT < 100% matches will have to be examined case by case.
• The 95 FUZZY matches are probably typos, and are addressed in #41
• The 440 HIGHERRANK matches are mostly plants and a lot of hybrids. Chances are we can only correct half of those to match in GBIF.
• And then there are 9 records with no match: those are some viruses and names in Harmonia that appear twice in GBIF (a bug that will be fixed in April).

@timadriaens, how do you want to prioritize going forward?

In order to do the matching of the GRISS dataset with GBIF, some transformation is needed. However, @timadriaens, is there any taxonID in the current set to use as an identifier?

We want to add presence data to the WRIMS dataset:

1. Open this file in Open Refine: https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/source-datasets/wrims/data.tsv
2. Do a facet on PlaceName
3. Filter on Belgium and add a column based on this column, named presenceBE
4. Filter on Belgian part of the North Sea and add a column based on this column, named presenceBPNS
5. Filter on Belgian Exclusive Economic Zone and add a column based on this column, named presenceBEEZ
6. Filter on Belgian Coast and add a column based on this column, named presenceBECoast

Once this is done:

1. Copy and apply https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/source-datasets/wrims/data-with-common-terms-refine.json
2. Move the columns presenceBE, presenceBEEZ, presenceBPNSandpresenceBECoast` to the correct position (see https://github.com/LifeWatchINBO/alien-species-checklist#process)
3. Export the data as data-with-common-terms.tsv
4. Export the Refine as data-with-common-terms-refine.json

When I open the concatenated file in Refine, I still have the same problem: 1256 records rather than about 9000 and more.

Because of a Refine issue in automatically interpreting data from Excel, years are written as 2015.0. We need to remove those decimals.

• Plants
• WRIMS

• Combine Excel sheets into one flat file
• Transform Excel file to CSV
• Put CSV in https://github.com/LifeWatchINBO/alien-species-checklist/tree/master/source-datasets/rinse

Question is asked by @DimEvil to VLIZ. Ideally there is a public bulk dataset.

In what format should the GRISS information be provided when returning? is CSV appropriate, or should it be excel?

We will add the following columns for species that occur in wn.be and the concatenated file (based on identical acceptedKey:

• Status: standardized
• GBIF acceptedKey
• Our taxonID ?

Order of sources to use info from:

• Plants
• Wrims
• Fishes
• Macroinvertebrates (all considered established)
• Harmonia (beware, not all present in BE)
• Rinse + annex: no status info

Matches

• Records in both lists ok: maybe check discrepancy in status wn.be and ours
• Records in wn.be only: mostly natives, check those flagged by wn.be as exotic
• Records in concatenated only: ok

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with fishes
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

Once we have #26:

1. Do a facet on origin
2. Copy all the values and the number of times they occur
3. Paste this information in the relevant sheet in this spreadsheet

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the columns Belgian exclusive economic zone; Belgian Part of the North sea; Belgian coast
• Add the other common terms as columns
• Populate datasetName with wrims
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

What are the meaning of the columns "D/N" and "V/I"?

• Combine Excel sheets into one flat file
• Transform Excel file to CSV
• Put CSV in https://github.com/LifeWatchINBO/alien-species-checklist/tree/master/source-datasets/plants

Hello Dear Tim.
Do we need to keep other countries than Belgium? (ie: GB, France, Netherlands)

@timadriaens, I also asked you this by email, but I'm recording it here so we won't forget. What file do we use for the RINSE dataset?

1. File emailed by you: https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/source-datasets/rinse/AnnexB%20RINSE%20Registry%20of%20NNS.xlsx
2. File in supplementary material of Zieritz et al.: https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/source-datasets/rinse/neobiota-023-065-s001.xlsx

My preference would be 2, as that is publicly available and easier to reference. I just need to know that all the core info is there.

Once we have #26:

1. Do a facet on habitat
2. Copy all the values and the number of times they occur
3. Paste this information in the relevant sheet in this spreadsheet

Is the Harmonia dataset publicly available in bulk? If not, how can we obtain this?

Once we have #26:

1. Do a facet on status
2. Copy all the values and the number of times they occur
3. Paste this information in the relevant sheet in this spreadsheet

23 records:

OK

• 20 have an exact match with gbifapi_canonicalName which is ACCEPTED.

Fuzzy

• 19 Proterorhinus semilunairs → Proterorhinus semilunaris Correct typo

Synonyms

• 0 Acipenser baeri → Acipenser baerii: Correct typo
• 1 Acipenser guldenstaedti → Acipenser gueldenstaedtii: Correct typo

Similar to the concatenated and waarnemingen lists, a matching with GBIF is needed to make comparison possible.

Use http://api.gbif.org/v1/species/5406539

For the macroinvertebrate data, we are currently relying on a final proof paper emailed by Tim: https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/source-datasets/macroinvertebrates/AI15-039_Boets_etal_almost%20ready%207%20Dec%202015_Tim.doc

When Boets et al. is published, we should use the actual pdf paper as the source, as that is easier to reference.

• Extract data from table in article
• Transform to CSV
• Put CSV in https://github.com/LifeWatchINBO/alien-species-checklist/tree/master/source-datasets/macroinvertebrates

Input file is at https://github.com/LifeWatchINBO/alien-species-checklist/blob/master/reference-datasets/natuurpunt/waarnemingen-be-species.csv

140 records:

Considered OK

• 119 records are exact match with gbifapi_canonicalName + ACCEPTED: 114 SPECIES and 5 GENUS. The genera got a lower confidence level, but are all correct.
• 2 records have EXACT 100% + ACCEPTED. Those are nothotaxa and are also correctly matched.

Synonyms

• 30 Alopochen aegyptiacus → Alopochen aegyptiaca
• 31 Ambrosia artemisiifolia → Ambrosia polystachya Incorrect, only applies to invalid Ambrosia artemisiifolia Besser, not Ambrosia artemisiifolia L.
• 37 Aster divaricatus → Croptilon divaricatum. Update to Eurybia divaricata
• 38 Aster laevis → Symphyotrichum laeve
• 39 Aster lanceolatus → Symphyotrichum lanceolatum Is now just Aster lanceolatus in August update of GBIF backbone
• 40,41 Aster novi-belgii → Symphyotrichum novi-belgii
• 42 Aster x salignus → Tripolium pannonicum -> Update to: Symphyotrichum salignum → Symphyotrichum ×salignum
• 59 Conyza canadensis → Erigeron canadensis
• 105 Mahonia aquifolium → Berberis aquifolium
• 109 Mustela vison → Neovison vison
• 137 Rana catesbeiana → Lithobates catesbeianus
• 157 Spiraea x billardii → Spiraea pseudosalicifolia

Other issues

• 74 Elodea canadensis → Multiple matches, hardcode to http://www.gbif.org/species/2865448 = http://mdoering.github.io/nub-browser/app/#/taxon/2865448
• 88 Hyacinthoides hispanica → Multiple matches, hardcode to http://www.gbif.org/species/5304257 = http://mdoering.github.io/nub-browser/app/#/taxon/5304257
• 143 Rudbeckia laciniata → Higher taxon: Rudbeckia. Multiple matches in fact, hardcode to http://www.gbif.org/species/3114229 = http://mdoering.github.io/nub-browser/app/#/taxon/3114229
• 107 Mimulus guttatus → Multiple matches, http://www.gbif.org/species/7333256 will disappear: http://mdoering.github.io/nub-browser/app/#/taxon/7333256. Update to Erythranthe guttata
• 36 Aster americ. → Higher taxon: Aster No variations can be found. Removed from list (i.e. name emptied)
• 127 Pinus nigra nigra → Higher taxon: Pinus nigra Can match to subsp, form or var: http://mdoering.github.io/nub-browser/app/#/search/Pinus%20nigra%20nigra Higher match considered ok

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with harmonia
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with rinse-annex-b
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

Using this file:

• Rename the columns we want to keep to the common term names
• Remove the columns we are not planning to use
• Add the other common terms as columns
• Populate datasetName with rinse
• Order the columns alphabetically
• Export the final file to the appropriate directory as data-with-common-terms.tsv
• Store the refine steps as data-with-common-terms-refine.json

• Select in concatenated.tsv file for the fuzzy matches
• Compare the original scientificName with the scientificName_gbif
• double check the names (eol.org , google...)
• Correct typos!

(17 fuzzy matches)