We wanted to explain more about what happens behind the scenes after our awesome Notes from Nature volunteers do transcriptions or classifications. What do we do with it and how do we get it back to curators or other scientists at Museums? One thing you may not know is that every label is transcribed by three different people. The idea is that more folks examining labels will lead to better results. For example, if two people enter Wisconsin for the state, and one person accidentally enters Wyoming then we can assume Wisconsin is correct and that Wyoming was a mistake. We also know that some labels are tough to interpret, and sometimes a couple different guesses can get closer to the right answer than just one.
This seems pretty easy right? Well… it gets more complicated when we start working with free text labels. Those text boxes where you enter sentences and phrases from the label. Things like locality information “Route 46 next to a tree by the stop sign on 4th street”, or habitat data “in a field”. How do we compare answers for these kinds of labels. What do we do with extra punctuations? Extra spaces? Extra words? Different words?
We have spent the last few months writing code that helps handle these kinds of situations. Essentially we want to first find labels that match and if not then we want to select the best label we can from the set of answers. We have set up a series of decisions rules to go through your answers. First, we ask if two of the three answers are identical including spaces and punctuation. If they match we are done. If not, then we remove extra spaces and punctuation and ignore capitals and ask if two of the three answers are identical. If so then we select the one with the most characters- with the idea of getting more information.
These two labels would be found to match after removing punctuation, spaces and ignoring capitals. Here we generally take the one with more characters to include as much information as possible.
Rd. 10 KM 24 *RD. 10. KM 24 *this one gets selected more characters
At this next stage things get a little more complicated and we want to use our decision rules to select the best answer we can among the three. First we look for labels where all of the words from one are found in another – partial ratio match. If we find this then we take the label with the most words.
North Fork of Salmon River at Deep Creek, by US-93 *North Fork of the Salmon River at Deep Creek, by US-93 *partial match selection– more words
Finally, we compare the answers using both a ‘fuzzy matching’ scheme. The fuzzy matching looks partial matches on words for example someone may have written ‘rd’ whereas someone else wrote ‘road’, our fuzzy matching will allow those to be considered the same. This strategy also allows for slight misspellings between words. If we get a fuzzy match between the two labels then we take the label with the most words. That ensures that we get the most data we can from these answers.
*County Line Road 2 mi E of airport County Line Rd. 2 mi. E. of airport *fuzzy match select this one
The end result of all this is a reconciliation “toolkit”. We pass all transcripts from finished expeditions through this toolkit, and it delivers three products. The first is just the raw data. The second is a best guess transcription based on the field by field reconciliations described above. The third is perhaps the most important – a summary of what we did and how we did it as a .html file. The summary output is something we are extending, as we think of new things that providers might want to see. Here is an example from the New World Swallowtail Expedition, one of the more difficult expeditions we’ve launched.
More recently, we have added some new features, including information about how many transcriptions were done by transcribers (based on their login names at Zooniverse) and a plot of transcription “effort” and how that looks over all transcribers. The effort plot is very new, but we wanted to provide information on whether most of the effort is done by a very few people, or there is more even spread across transcribers. Here is an example for a different expedition, “WeDigFLPlants’ Laurels of Florida”:
Finally, we give them the information about how labels were reconciled (if there was an exact match, partial or fuzzy match). We do this so the providers can go through them and decide if there are some they want to check. We also highlight any problem record, those for which we could not get a match, or those for which there was only one answer – so we could not compare the answers. Here is an example from one label. The areas in green are the three different answers, the top row is the ‘best guess’ reconciled record and the gray row is information about how the reconciliation was done. For example on the first column Country all three answers were Myanmar – and in gray it says we had an exact match with three answers. The ones in red are potential issues (in this case only one answer given).
The goal of all of this is to make it easy for providers to use these data right away. And we’ll note that this tool allows us to also get an overall look at transcription “success” rates, something we may come back to future posts, because these numbers are striking and illustrate the high value of this effort.
– Julie Allen, Notes from Nature data scientist
Thanks to an awesome group of volunteers who came to visit us at the Natural History Museum London today (Visiteers), the second batch of Chalcid slides in the Magnified section have now been completely classified! Hurray!
Visiteering is one of the ways in which we invite the public to take part in our science at the museum, and today’s stellar group has set an all-time record for number of classifications in one day. Absolutely Awesome.
We’ll try to get the third and final batch of Chalcid slides launched in a new Expedition soon – but in the meantime, a VERY big thank-you to the Notes from Nature community and our Visiteers in the museum for all of your help in setting this data free.
Help get a genome for the amazing Bombardier beetle #seqthebeetle
Vote daily from now until 5 April here https://tinyurl.com/gn84mu8
A huge shout out to our volunteers for quick work on our first NFN Ideas project, which focused on oak phenology. We completed the expedition a week after launching it, with 1944 transcripts of 644 subjects. 53 awesome transcribers took part. A lot of discussion on talk focused on some of the challenges with denoting flowers and fruits — it is harder than it first looks! So folks were interested in whether there was consistency among transcribers, and if the results would be consistent with an expert assessment. We have some initial answers to those questions and more! And a note that ALL of these data – the label data and phenological scoring – were ALL done by Notes from Nature volunteers.
So to get right to it! Transcriber consistency on this expedition was absolutely remarkable. Well above 99%. Yeah. We were surprised, too. There were three cases where we didn’t get consistent results. Just 3! Out of 664 subjects. So apparently there was very strong agreement.
We took a closer look at the three that seemed to prove difficult.
- subject_id: 4308678 –http://www.sernecportal.org/portal/collections/individual/index.php?occid=11108535
- subject_id: 4308659 –http://www.sernecportal.org/portal/collections/individual/index.php?occid=11108069
- subject_id: 4308844 —http://www.sernecportal.org/portal/collections/individual/index.php?occid=11130030
The consensus scoring for those from transcribers were:
- subject_id: 4308678: Flowers: No, Fruits: No
- subject_id: 4308659: Flowers: No, Fruits: Yes
- subject_id: Flowers: Yes, Fruits: No
I then asked NFN’s own Michael Denslow, who is also a darn fine botanist, for his assessment (without reporting anything about transcriber’s scoring), and he was 100% consistent with the three above. He noted for 4308678, “Funky one for sure” and for 4308659, “The terminal buds might be confusing people on these. Based on the collection date (and presence of terminal buds) fruits could be from pervious fall.”
And finally, we wanted to see if we could use these data to look at phenology patterns, so our data scientist Julie Allen did some quick visualizations of the data using the statistical package, R, which has some great plotting functions. You can see our plot above, for two species, Quercus falcata (top) and Quercus marilandica (bottom), two common oaks where we had enough data to examine patterns. The plot shows time on x-axis measured from March through November, and the y-axis is just a yes-no response. For yesses, we show a little emoji, and for no’s you can see those no reports over time for fruits and flowers in different colors. Yup, we decided to go with a tropical flower and fruit motif here, despite oaks definitively not producing pineapples!
The really neat thing is that we do pick up the short, and early flowering period for oaks during Spring, and in Q. falcata, a seemingly quick transition to acorns, and a slower cadence for Q. marilandica (note the longer period between flowering and appearance of acorns). There are still some great questions to examine here — these records were not all from the same year, and maybe some variation we are seeing is due to climate variation year to year. There were a couple “no flower” records during a typical flowering period and these might be either limited information from the sample, or perhaps something about that particular year. We are more than happy to share the raw data from this expedition with anyone who wants a closer look!
Many people have heard of leafcutter ants carrying bits of leaves around the forest like little parasols. Ants are all in the family Formicidae, in the order Hymenoptera, which includes bees, wasps, and ants. Leafcutter ants do not eat the leaves, but rather compost them to feed their fungus gardens. They only eat a particular species of fungus, removing all other species, and constantly tend their gardens to keep them clean.
Thanks to everyone who contributed to the Trechine Ground Beetles 2 expedition. With its completion, we have also finished off our remaining specimens from this ground beetle subfamily, and brought our total number of ground beetle specimens digitized to 4391. That’s a little less than 25% of the total number of specimens we had when we started this project back in July, 2016.
For our next expedition we’ll be focusing on the Bombardier beetles (subfamily Brachininae). The University of Alberta’s E.H. Strickland Entomological Museum has a large number of Bombardier beetle specimens, with 860 already digitized and publicly available from GBIF, and another ~4300 with images remaining to be transcribed as part of this effort. Most of these belong to just one genus, Brachinus, though there are about a dozen genera known in the subfamily. Bombardier beetles are found in temperate and subtropical regions around the world. They get their name from their ability to spray boiling hot chemicals from their abdomen, a characteristic whose mechanical/biological basis eluded explanation until a couple of years ago (a must see article and video here).
You’ll notice that our expedition image has changed from what use to be a photo of a specimen drawer to something a little more lively. We now feature an in situ photograph of a Brachinus specimen courtesy of PhD student and carabidologist, Wesley Hunting. Thanks Wes!
Make sure to check out our Bombardier beetle expedition!
–Bryan Brunet, PhD
Collections Management Advisor (Natural Sciences), University of Alberta Museums, Edmonton, Alberta, Canada