FAQs and Useful Tools

We have put together a list of frequently asked questions and useful tools related to transcribing records. These are  based on discussion threads in the NfN discussion forums (linked at the bottom). Please take a look, and let us know if there are any questions we missed or useful tools that you can suggest. These will eventually go on a page of the Notes from Nature website.

We really appreciate your input and all that you do for Notes from Nature. Thank You!

Common issues or questions that you may encounter while transcribing Notes from Nature records:

1.) Interpretation: In general, you should minimize interpretation of open-ended fields and enter information verbatim. This way, we can better achieve consensus when checking multiple records against one another (see below, on that process). However, some discretion would be nice. Here are examples:

Interpretation that you should make: Simple spacing errors (e.g. “3miN. of Oakland” should be “3 mi N. of Oakland”)

Interpretation you should leave to us: Don’t interpret abbreviations, we’ll sort that out. (e.g. “Convict Lk.” )

2.) Not in English: Transcribe exactly as written. Match label content to transcription fields as best as you can.

3.) Abbreviations: Transcribe exactly as written.

4.) Spelling mistakes: Transcribe exactly as written, unless you have looked it up and are absolutely certain of a simple spelling mistake. In this case, you can enter the correct spelling.

5.) Problem records: If you come across a problem record that may need to be addressed by a scientist, like a faulty image or a record with illegible handwriting, you can flag the record by commenting on it (e.g. with the hashtag #error) and indicate what is in error.

6.) Provinces: Provinces go in the Location field (e.g. Coastal Plain Province, Piedmont Province).

7.) Capitalization: Sometimes information may be in all capital letters on the labels. Unless this is an abbreviation, you should capitalize only the first letter of every word in your transcription (e.g. “COASTAL PLAIN PROVINCE” should be “Coastal Plain Province”).

8.) Many collectors: In many cases, collectors may be listed on different lines of the label with no punctuation separating them. In your transcription, please separate the collectors with commas.

9.) Missing information: What should you do when there is no information available for a field? When information is not given on the label, you should leave the field blank (in the case of open-ended fields) or select “Unknown” or “Not Shown” in the drop-down lists.

10.) Inconsistent collector names: You will often find several variations of the same collector name (e.g. “R. Kral” or “R.Kral”, “RWG” or “R.W.Garrison”). Use similar discretion when transcribing these variations as in the localities.

Interpretation that you should make: Simple spacing errors (e.g. “R.Kral” should be “R. Kral”)

Interpretation you should leave to us: Don’t interpret abbreviations, we’ll sort that out. (e.g. “RWG” should remain “RWG”)

11.) Many scientific names: For Calbug, you do not need to enter the species name (we have this info already), but if there is a scientific name that is different from what is listed in the record, put it into the “Other Notes” field. These could be old names, or plant host names.

For SERNEC Herbarium specimens, copy only the most recent name. This can be determined based on the date that appears on the ‘annotation label.’ If you do not see a date then enter the name that appears on the primary label.

12.) Variations and subspecies (SERNEC): Record the subspecies, but omit the scientific author’s name. So “Cyperus odoratus var. squarrosus (Britton) Jones, Wipff & Carter” becomes “Cyperus odoratus var. squarrosus”. “Echinodorus cordifolius (Linnaeus) Grisebach ssp. cordifolius” becomes “Echinodorus cordifolius ssp. cordifolius”.

13.) Scientific name (SERNEC): Provide the most recent name, whether it is a species name (a two-word combination of the genus and what is called the “specific epithet” in botanical nomenclature) or a one-word name that is at a higher taxonomic rank (e.g., just the genus or family name). Names at higher taxonomic ranks than species are used when a more precise identification has not been made.  The species name should typically take the form of a genus name that begins with a capital letter and a specific epithet that begins with a lowercase letter.  If any of the names are given in all capitals, such as “CYPERUS ODORATUS”, the name should be entered using the typical convention, “Cyperus odoratus” in this case.

14.) Latitude and Longitude: How do you enter latitude and longitude values, and where do these values go? Enter exactly as written, you can find symbols in Word or by searching online (e.g. 33° 62’ 22” N  116° 41’ 42” W). You can also produce the degree symbol ° using key combinations (alt + 0 on a mac; alt + 0176 on a PC, with the key pad on the right side of your keyboard). This information should go into the “Location” or “Locality” field, depending on the project you are working on.

15.) Special Characters: What should you type when there is a special character in a text string, such as a degree symbol or language-specific characters? You can do a google search for the symbol or copy and paste it from Microsoft Word symbols. There are also key combinations for common symbols. As mentioned above, you can produce the degree symbol ° using key combinations (alt + 0 on a mac; alt + 0176 on a PC, with the key pad on the right side of your keyboard).

16.) Elevation: Enter elevation verbatim into the “Other Notes” field for Calbug and the “Habitat and Description” field for SERNEC Herbarium records.

17.) County: If the county is not given on the label, please find the appropriate county using google search. However, if there are multiple potential counties for a locality, please leave the county field blank.

18.) Checking your transcription: You can use the link to the left of the “Finish Record” button (e.g. “1/9” or “9/9”) to check the information that you entered. Just click on any of the fields to make any necessary edits to your transcription.

19.) When is a record finished?: These blog posts describe the data checking process that uses 4 transcriptions of the same record to derive a consensus.

http://blog.notesfromnature.org/2014/01/14/checking-notes-from-nature-data/

http://soyouthinkyoucandigitize.wordpress.com/2014/01/14/412/

 

Some Useful Tools (discovered by NfN users)

Counties and Cities: Good tools for finding counties etc. are lists on wikipedia, there are lists of municipalities in each state of the USA (there are also similar lists for others). For example, https://en.wikipedia.org/wiki/List_of_municipalities_in_Florida (via the linkbox you can also change the state).

Mountains: https://en.wikipedia.org/wiki/Category:Lists_of_mountains_of_the_United_States

Uncertain Localities: Geographic Names Information System, U.S. Geological Survey.

https://geonames.usgs.gov/pls/gnispublic

Mapping tool with topo quads: To find uncertain counties or localitieshttp://mapper.acme.com

Collector Names: The Essig Museum of Entomology database has lists of collector names and periods of activity for many collectors you will find in Calbug records.  http://essigdb.berkeley.edu/query_people.html

Hard-to-read text: Use “Sheen”, the visual webpage filter, for some hard-to-read handwriting written in pencil. (Tip was from the War Diary Zooniverse project) https://chrome.google.com/webstore/detail/sheen/mopkplcglehjfbedbngcglkmajhflnjk?hl=en-GB

Special symbols: You should be able to find symbols in word or by doing a google search and copy and paste. Here are a few:

-           degree symbol for coordinates:  °

-          plus minus: ±

-          fractions: ⅛ ¼ ⅓ ⅜ ½ ⅝ ⅔ ¾ ⅞

-          non-English symbols: Ä ä å Å ð ë ğ Ñ ñ õ Ö ö Ü ü Ž ž

The Plant List: Search for scientific names of plants -http://www.theplantlist.org/

List of Trees:   https://en.wikipedia.org/wiki/Category:Trees_of_the_United_States

Integrated Taxonomic Information System (ITIS): http://www.itis.gov/

Essig database (for Calbug): Unsure if you spelled a collector name or species name right? It might also be worth checking similar entries already in the database, http://essigdb.berkeley.edu/.

 

NfN Discussion threads that this is based on:

FAQs

http://talk.notesfromnature.org/#/boards/BNN0000001/discussions/DNN000024q

Useful Tools

http://talk.notesfromnature.org/#/boards/BNN0000001/discussions/DNN00001vl

A Big Thank You to Ornithology Ledger Transcribers

Phase 1 of the Ornithological Collections transcription project has been successfully completed.   Thank you to everyone who participated in the first stage of the project.   Over a thousand users worked on the project producing nearly 100,000 transcriptions from 1037 register pages.

Image

The resulting dataset is going to be tremendously valuable to Natural History Museum curators and researchers all over the world.   We have included some pictures of museum specimens that now have useful electronic data thanks to your work as citizen scientists.  Initial analysis suggests citizen scientists were especially busy over the mid-winter period. 

Image

Special mention must go to Snowysky, Estron21, OneUniverse, Dmbrgn, mbhook and AstroboyOW for their sterling work.  We’d love to hear from you.   But all contributions are valued and we’d love to hear from anyone who has participated in the first phase.  Especially, as we enter an exciting second stage with six more registers that need transcribing.  We hope you will all support us in this new phase ultimately aimed at getting all the Natural History Museum’s bird specimens on line.

Image

With thanks and best wishes,

NHM Bird Group

 

Making Progress Clear on Notes from Nature

Notes from Nature is something of a departure for a Zooniverse project.  Rather than a single organization asking for help with the exact same tasks, Notes from Nature is, like its subject matter, diverse.   So we have labels of bugs, sheets of plants, fungal specimen labels, and ledgers of birds.  And we have a lot – and I mean A LOT— of images that need transcription.   Not only that, but each of those images are transcribed more than once—as mentioned in previous posts, right now each image gets 4 separate transcriptions.

All of this is preface to the main topic of this post – how do we measure “progress” with the tasks of transcribing all of this data.  The science team on Notes from Nature has talked a lot about this, and a number of complexities related to making sure that the numbers are transparent to you, our volunteers.  This post covers a fair amount about how to measure overall progress.  We also know that there have been issues with transcription counts for individual volunteers. We believe that we have solved those issues, but we’ll cover those separately in another blog post.

So, here are two of the main issues we have been dealing with and some recent solutions that have been implemented across Notes from Nature:

Issue 1: Do we measure total number of transcriptions or total number of images that are “finished” (e.g. transcribed four times)?

Solution:  We have decided to measure total transcriptions completed across all projects and within projects.  This is a change from our previous strategy which had mixed and matched these different counts on different pages.  We think the most obvious measure is overall effort put in, even if this means it is harder to know how many images have been done.

Issue 2:  Should we even measure “completeness” within a project (e.g., Calbugs)?  The reason this is an issue is that most projects on Notes From Nature have only posted a small subset of available images and there are many more “waiting in the wings”.  We don’t want to say “hey, only a 1000 more images to transcribe” and then just a little later go “Oh!  Just kidding, there are now 50000 more!” Our ultimate goal is to stage the many remaining images as smaller batches with compelling themes derived from their research or other societal values (e.g., all specimens from a particular national park or collected by an important historical figure).   This will give us a chance to celebrate the success of completion more regularly.  At the moment, we are seeking funding to do this.

Solution:  We do want to show that progress is being made on the current batch of images on Notes from Nature, but we want to avoid any confusion if more images are made available once the current sets are close to be done.  So we are showing a percentage that represents total number of transcriptions completed over the total number needed for a batch, but we link to this very blog post to explain why those may change.  We are also providing some information on progress with the images themselves, and here we provide counts of “total images”, “active images”, “complete images”.  Below is a definition of each of those terms:

total images - The number of source images currently available.
active images - The number of images that are either in progress with being transcribed or waiting for transcription.
complete images -  The number of images that have been independently transcribed four times

We hope the new changes lead to greater transparency!  We are also so awestruck and appreciative of all the participation as we pass the 5000 registered participants and 400,000 transcriptions mark- we can’t thank you enough or tell you how important the public participation has been for this endeavor.  Finally, we love comments and suggestions. Please feel free to provide some, and we’ll get right back to you.

The Valdosta State University Herbarium

Image

Cabinets in the Valdosta State University Herbarium

The Valdosta State University Herbarium is a museum-quality collection preserving more than 65,000 dried plant specimens useful in research and teaching. The VSU Herbarium, a unit of the Biology Department of Valdosta State University and the second largest herbarium in Georgia, is a rich repository of data emphasizing the diverse flora of the coastal plain region of Georgia and, more generally, the flora of the southeastern United States. In addition to this geographic focus, the VSU Herbarium has taxonomic specialization beyond the southeastern region, with extensive holdings of sedges (Cyperaceae) and other graminoid families, and bryophytes (mosses).

Plant specimens in herbaria are the basis for the knowledge about where and when plants grow and their physical characteristics.  Herbarium specimens and associated data are standards for the application of plant names and are widely used by scientists as a basis for the descriptions and distributional maps in specialized literature related to plants. Consequently, they are an essential resource for anyone who needs plant names consistently and accurately derived. The herbarium is also employed extensively to document the locations of rare species and how their populations change over time. Data from herbaria are now being used to study shifts in the timing of reproductive patterns (flowering and fruiting) of plants relating to climate change.  Thus, herbarium specimens and data are useful to a variety of scientific researchers, not only botanists, but also ecologists, agricultural scientists and natural resource managers.  The VSU Herbarium is used intensively in research and teaching at Valdosta State University, and it provides materials used by researchers at other institutions through lending and exchange of specimens.

Although it originated in the 1930s as a teaching resource of several hundred specimens collected by Professor Beatrice Nevins, the VSU Herbarium was founded as a research collection in 1967 by Professor Wayne R. Faircloth.  In addition to Faircloth’s specimens, the VSU Herbarium includes significant collections of Charles Bryson, Richard Carter, Delzie Demaree, Robert Godfrey, Robert Kral, and Sidney McDaniel.  Since 1984, the VSU herbarium has more than doubled in size, growing at the rate of 1000-2000 specimens per year.  In 2001, the VSU Herbarium occupied new quarters with about 1500 sq. ft., more than twice the space of the old facility, and a modern dedicated climate control system with the capacity to maintain relative humidity below 60%.  Additional information about the VSU Herbarium can be found here. Through support from the National Science Foundation, all of accessions in the VSU Herbarium have been imaged, and we are currently building a database of label data from these specimens. Through a local collaborative effort with the VSU Odum Library, many of these images are currently available on-line at http://herb.valdosta.edu.

The VSU Herbarium needs your help in building this database!

–Richard Carter, Director of the Valdosta State University Herbarium

The University of Michigan Herbarium — Mecca for Macrofungi

This article was written by Matthew Foltz, who is the manager of the Macrofungi digitization project at the University of Michigan.  We commissioned this article because most of the labels that are currently available  for transcription in Macrofungi are from this institution.  The University of Michigan Herbarium has an unparalleled history of contributions to the scientific study of fungi, and also for contributions to bringing an understanding of fungi to a general audience.

The University of Michigan Herbarium is internationally recognized as one of the leading repositories for natural history collections including Vascular Plants, Bryophytes (that is, mosses and related plants), Algae, Lichens, and Fungi. It has a rich history dating back to the late 1830s when state geologist Douglass Houghton conducted a geological survey of Michigan and deposited about 800 collections at the University of Michigan.  The herbarium is now home to over 1.7 million specimens of plants and fungi collected from around the world, including about 280,000 collections of fungi.

Figure 1. The University of Michigan Herbarium houses over 1.7 million collections in 1,200 cabinets in a 16,000 square-foot climate-controlled range. Photo by M. Foltz.

Figure 1. The University of Michigan Herbarium houses over 1.7 million collections in 1,200 cabinets in a 16,000 square-foot climate-controlled range. Photo by M. Foltz.llections of fungi.

The University of Michigan Herbarium has strong roots in mycology. In 1921 the herbarium became its own department under the directorship of mycologist Calvin H. Kauffman. Directorship of the herbarium went on to several other mycologists in the 1900s including Edwin B. Mains, Alexander H. Smith, and Robert L. Shaffer. Calvin Kauffman was a mycological and scientific pioneer. His publication on the Agaricaceae of Michigan (as well as the series of fungal monographs that preceded it) was not only the most comprehensive for the state, but at the time it was also one of the best fungal surveys for North America. The family Agaricaceae contains the common grocery store mushroom.

Figure 2. Calvin H. Kauffman. Photo courtesy of University of Michigan.

Figure 2. Calvin H. Kauffman. Photo courtesy of University of Michigan.

To truly appreciate the impact that Kauffman and his predecessors have had on mycology, one needs to look no further than the Kauffman Lineage which is a part of Meredith Blackwell and Robert Gilbertson’s Genealogy of North American Mycologists. This lineage of education includes many of the most significant mycologists of all time, including some of today’s most prominent scientists.

The fungal collection at the herbarium is strong in both Macrofungi (mushrooms and shelf fungi) and microfungi (molds and mildews, etc.). Kauffman’s work was strong in the agarics (mushrooms) of the Great Lakes region, but he also collected in many localities across North American and internationally. Alex Smith followed in his footsteps and went on to become a leading expert of the agarics as well as other groups such as the gasteroid fungi (puffballs, earthballs, stinkhorns, etc.) and the boletes. Like Kauffman, Smith contributed to both the Great Lakes region flora, and the North American flora. Smith was a prominent collector and deposited over 92,000 specimens at the herbarium during his lifetime.

Figure 3. Dow V. Baxter. Photo courtesy of University of Michigan.

Figure 3. Dow V. Baxter. Photo courtesy of University of Michigan.

The value and depth of the Michigan fungal collection is strengthened by Dow Baxter’s extensive collection of wood-decay fungi, along with collections of agarics from R.L. Shaffer and hypogeous (underground) fungi from R. Fogel. The Michigan herbarium also owns several classical exsiccati (historical sets of reference collections), as well as historically important personal collections from H.A. Kelly, H.C. Beardslee Jr., and many others. In addition to macrofungi, Michigan also has extensive collections of microfungi from prominent mycologists including Bessie B. Kanouse (discomycetes), E.B. Mains (Uredinales, Geoglossaceae, insecticolous fungi), L.E. Wehmeyer (pyrenomycetes), F.K. Sparrow (aquatic fungi), and others.

Michigan mycologists have a history of supporting citizen science and collaborating with amateur mycologists and the general public. Alex Smith was an advisor and supporter of the premier amateur mycology group the North American Mycological Association (NAMA) in its early days.

Figure 4. Alexander H. Smith. Photo courtesy of University of Michigan.

Figure 4. Alexander H. Smith. Photo courtesy of University of Michigan.

Smith was a close friend to many of the top “amateur” mycologists such as Ellen Trueblood, Virginia Wells, Phyllis Kempton, and others.

These mycologists kept detailed records and notes with their specimens, and these important collections and their ancillary materials are deposited at the Michigan herbarium. More recently, Robert Fogel, a past curator of fungi at Michigan, was one of the first mycologists to create a website for learning about fungi in the 1990s (Fun Facts About Fungi).

The tradition of outreach continues today through the efforts of mycologist Tim James, the assistant curator of fungi at the herbarium. Recently, James has held lectures and forays for several amateur groups including the Michigan Botanical Club and the Michigan Mushroom Hunters Club. He is also leading the efforts at Michigan as part of the Macrofungi Collection Consortium, a nationwide project to digitize the fungal collections and make them available online to researchers and the general public. The digital records produced by that project are the ones posted here on Notes from Nature.

Figure 5. James lab (Umich) on a John Cage inspired mushroom foray with the Michigan Mushroom Hunters Club. Photo courtesy of T. James.

Figure 5. James lab (Umich) on a John Cage inspired mushroom foray with the Michigan Mushroom Hunters Club. Photo courtesy of T. James.

On behalf of the University of Michigan Herbarium, we thank you for your participation in this effort to transcribe information from these historical records.

An Ode to Collecting: following the path of an early 20th century dragonfly collector

As you enter information from old labels, have you ever wondered about the collector’s experience—20, 60, or even 100 years ago? Have you thought about what the river, lake or meadow was like?  Or how the place may have changed since the collector was there?  I have, and so I spent a couple of years revisiting sites originally sampled by C.H. Kennedy for dragonflies in 1914-1915. Check out my recent blog post on this experience, entitled An ode to collecting: following the path of an early 20th century dragonfly collector.

CASENT8127985

Checking Notes from Nature Data

You may have wondered how we will use the data you are transcribing… Especially after you spend a few minutes trying to figure out the meaning of some unclear handwriting, and are uncertain whether the year on the label actually says “1976” or “1879,” or if the locality is “Vino” or “Vina” in California.

The first step in using Notes from Nature transcription data in research is checking data quality. Each specimen is transcribed independently by four different users, so mistakes in a single transcription do not impact the accuracy of our data very much. But how do we determine the right one out of the multiple transcriptions? For controlled fields such as country and state, this is simple: we perform a “vote-counting” procedure, where the value with the most “votes” is considered correct. However, this is much more difficult for open-ended fields such as locality or collector names.

One of the early approaches was to quantify differences between transcriptions. We did this by using the Levenshtein edit distance, named for the Russian information scientist, Vladimir Levenshtein, who developed a way to quantify the minimum number of single-character edits required to change one string of text to another. For example, “12 mi W Oakland” and “12 mi. W Oakland” would have a score of 1 because of the period after “mi”. We then looked for highly similar pairs of transcriptions for each specimen (pairs that were separated by the smallest edit distance), and then chose one of the pair. Our assumption was that if two separate transcriptions are almost identical to one another then either one must be very close to being correct. However, this assumption becomes violated if users are consistently wrong (either because the label is partially obscured, or the handwriting difficult to read, etc.). Furthermore, this was not a true consensus in the sense that our algorithm only decides which transcription out of the four is the best.

During the Hackathon last month (discussed in an earlier blog), it was quickly obvious that this approach was not ideal. In discussions with other researchers and experts, we developed an approach to combine the best elements of each individual transcription into a locality string that was greater than the sum of its parts. We did this using a concept from bioinformatics known as “sequence alignment.”

Genetics researchers and molecular ecologists use sequence alignment to match up multiple DNA or amino acid sequences in studies of evolutionary history or DNA mutation. To evaluate alignment, algorithms look for particular strings of amino acids or nucleotides within multiple sequences, and match similar or identical regions. Even closely related sequences, however, are often not identical and have small differences due to substitutions, deletions or insertions. So, alignment algorithms use complicated scoring mechanisms to generate alignments requiring as little “arrangement” as possible. The goal is to balance potential insertions, deletions and substitutions in characters from one sequence to the next.

In Notes from Nature data assessment, we use this method and assume that differences between transcriptions in a particular location may be due to substitutions. Or, we introduce gaps so that two separate chunks of sequence might be better aligned. This is especially useful if there are certain chunks of text within an individual transcription that are not consistently shared among all other transcriptions.

We implemented sequence alignment in two ways, using token and character alignment. For token alignment, we converted each chunk of text (usually individual words) and assigned each unique chunk a “token.” We then aligned the tokens. For character alignment, we compared each character in the text string directly. This proved to be the better method, as we inherently had a lot more information for the alignment software to work on.

An Illustration of Character and Token Alignment

Character consensus.001

To compare how well our algorithm was doing, we manually transcribed around 200 of Calbug’s specimens, and ran our algorithm on user transcriptions. 70% of the consensus transcriptions were either EXACTLY identical or within 5 edits to our manual transcriptions. While this result is preliminary and we’re still working on improvements, it is worth mentioning that the testing data set included quite a lot of the very worst examples of handwriting on a teeny tiny label.

Proportion of records that were exactly identical or within 5 edits to manual transcriptions 

Method

Field

Proportion exact       (n =158)

Proportion      < 5 edits

Token

Collector

37.3

74.1

Token

Locality

38.6

58.8

Character

Collector

38.0

88.0

Character

Locality

37.3

69.6

Proportion of specimens that were a certain edit distance away from our manually-transcribed set of specimens

Alignment

In closing, we’d like to emphasize the importance of typing exactly what you see in the open-ended fields for collector and locality. A certain degree of interpretation may be necessary. For example, if there are no apparent spaces between words (e.g. DeathCanyonValley) it would be useful to add them. But, it is often very difficult (probably impossible) to derive a consensus if the inconsistency of label data is compounded by inconsistencies in data entry. We have ways of dealing with the huge diversity of abbreviations, so it would be helpful if you enter them as they are when you see them with minimal interpretation. The overall lesson from our data quality comparisons is that so far we are doing great, thanks to your careful work!

Want to learn more? Check out a related blog post, here.

By Junying Lim

Follow

Get every new post delivered to your Inbox.

Join 680 other followers

%d bloggers like this: