Research Tools: Automatic Interview Transcription

Hi all, I'm writing this post to share some resources for automatic transcription - very handy for interview data (but probably not conversation analysis). I think this is a topic that has come up in the past, but some of the tools I found recently were new (to me), so I thought to share.

Context: I've done about 30 hours of interviews; interviews were done in either English (My L1, interviewee L2+) or Korean (my L2, interviewee L2). Transcribing the English interviews by hand wasn't terribly slow, but it was tedious. Transcribing the Korean interviews was rough (okay, brutal) - my proficiency and typing speed slowed me down a lot. So I got to searching for additional help.

Where things stand with speech-to-text, you can't expect perfect transcriptions, but you can get fairly decent accuracy when speaker proficiency is high and not strongly accented. These can form a basis for manual clean-up, which is proving to be a lot quicker for me than starting from scratch. Here are a few of the tools I tried:

temi.com - This site only does English, but the accuracy seems quite good (even for non-native speakers). Lots of bells and whistles- there is a very nice interactive transcription editor. The rates are quite cheap, too: 10 cents per minute.

happyscribe.co - This site handles many different languages and has a lot of bells and whistles - automatic punctuation, automatically splitting speaker turns, highlighting parts of transcriptions that the system was less sure of, and really nice integrated playback (you can click a part of the transcript and the audio plays from there). I found the speaker segmentation to be a little off, and the transcription accuracy for this one seems a bit low for Korean (could just be the speakers I fed it). Hourly rates are between 9 and 12 Euro per hour (depending on whether you have a monthly membership).

vocalmatic.com - Handles a fairly wide range of languages, and I found it was decently accurate for Korean. This is pretty no-frills compared to the other options- no automated speaker separation, no punctuation, limited editing tools - but it turns out transcripts with helpful timestamps. I was able to get a lot of mileage out of their free trial (they say you get 30 minutes free, but....). Otherwise, rates are similar to happyscribe.

Other tools to look into are Trint, Transcribe (wreally.transcribe.com). I was focused on tools that offered Korean transcription, but European languages are more common across other platforms.

Difficulty of Korean Phoneme Production and Perception for L2 Learners

It's a new year, and a (renewed) resolution not to let this blog fall completely by the wayside... so, time to start sharing little slices of my dissertation research!

Very briefly, my dissertation is a language assessment project related to the pronunciation of L2 Korean phonemes. I developed a diagnostic test that provides information on how well learners can produce and perceive Korean phonemes. I collected test data from 198 adult learners from a range of first language backgrounds and overall proficiency levels.

After converting each phoneme score to percentages and averaging across learners, here's what I found for the accuracy of production (y axis) and perception (x axis) (graphs made in R with ggplot2):

For those of you who don't read Korean, but do read the International Phonetic Alphabet, here's another version of the chart:

In many ways, these results aren't so surprising. For example, the tensed consonants of Korean were the most difficult to produce. And in general, there's a fairly strong (moderate) correlation between production and perception accuracy (of course, this is averaged across learners, so should be taken with a grain of salt).

But there are some interesting things going on. For one, learners actually did reasonably well with perceiving some of the tensed sounds, like /d*/ (about 80% accurate). There were also some cases in which production accuracy exceeded perception accuracy; this is something that goes against some stronger versions of L2 speech learning theory, but I'd chalk much of it up to differences in task difficulty. Because the production items were scored according to criteria in line with John Levis' (2005) Intelligibility Principle, productions that were confidently identifiable were scored as correct, even if they weren't exactly within native-like ranges in terms of temporal/acoustic qualities. Reception items, however, were simply right or wrong based on a choice, and these items utilized native-speaker recording of standard Korean phonological contrasts. 

Reference:

Levis, J. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39, 39-377. https://doi.org/10.2307/3588485

A few reflections on being back in the language classroom again- as a student

This summer, I had the great privilege of spending 200 hours over the course of 10 weeks in an intensive Korean program in Seoul (thanks to Michigan State University Asian Studies Center's Foreign Language and Area Studies summer fellowship program!). Ever since I started learning Korean at the age of ~23, I have craved opportunities to devote (most of) my attention to learning Korean, rather than squeeze in bits of self-study and chats with my wife or friends here and there.

Aside from noticeably improving my knowledge of and ability in Korean, I also find that being a full-time(-ish, more on this later) student is great for reflecting on language teaching and learning. Here are a few reflections that have stuck with me:

• Obligation is a Good Thing: Maybe one of the biggest things I get out of language classes is the obligation to show up, participate, do homework, etc. Outside of language classes, it is incredibly easy to just not use the language, and this can be true whether you are in an immersion setting or in foreign language setting. Hell, I'm married to a speaker of my target language, but because she speaks English incredibly well, it affords me the opportunity to be lazy. And I take that opportunity much more than I should. This summer I had to schlep 1.5 hours each way across Seoul for my 5-days-a-week classes, during morning rush hour. It was not always a fun commute, but I would do it again in a heartbeat. Having an obligation to learn and use the language is key. In my research on online language learning platforms, this is often the biggest missing element.
• I am 99.99999% in favor of Target Language Only classrooms: With all due respect to translanguaging/plurilingualism/plurilingual repertoires/etc., in a language-focused classroom there's nothing quite like near-exclusive use of the target language. Having to struggle through your misunderstandings and botched expression pushes you to recall words and structures as you re-work things. Asking for help or clarification in the TL in the classroom prepares you to do the same outside of the classroom. I relished being the only L1 English speaker in my class this summer. There were multiple Vietnamese and Chinese speakers, who all made great efforts to stick to Korean during class, which I really appreciated. Occasionally, though, there would be a side-huddle among some shared L1 speakers... which was a little jarring, honestly. Everyone else just had to sit and wait! Luckily my classmates were very kind about reporting back to the group in Korean. 
• Bilingual Dictionaries: Oh, right, that .00001%? I fully support individual use of bilingual (electronic/web) dictionaries. Especially as vocabulary gets more advanced and abstract. It's just such a huge time saver and I found my understanding of Korean words was better after a few seconds looking at definitions and examples in the KOR-ENG Naver dictionary. In some cases, teacher explanations led to confusions for me, likely due to my own deficiencies in lexical knowledge. 
• Meaningful Communication - 말처럼 쉽지 않다 ("Not as easy as saying it"): The program I attended prides itself on it's communicative focus, and in many ways, they get it right. For example, we had an assignment where were gave a little presentation on a news article and then presented three topically related questions for whole-class discussion. This was probably my favorite week in class- we spent so much time talking about current events in Korea and elsewhere that we were genuinely interested in, and using grammar and vocabulary we had learned came naturally when we needed it. On the other hand, I think it is still common pedagogical practice to also spend a fair amount of time doing the whole "go around the room and have everyone make a sentence using the target grammar structure." This does involve speaking, but it's not really meaningful or contextualized communication. It also means no one else is talking!
• Language students can have a lot on their plates: I hinted at this previously, but even though you might be enrolled full-time in an intensive language program, you might end up having a lot of other responsibilities to deal with. In my case, I had some work scoring essays and teaching an online class. I also became a father! Some of my classmates were putting in lots of hours at part-time jobs to support themselves. This kind of stuff makes it hard to do homework/projects as well as you'd like, and can negatively impact your sleep schedule, too. Some students do kind of get to "live the dream" where they have lots of time to study class material and really take advantage of immersion/social opportunities, but not everyone gets to do that. This really made me think about the expectations we have for international students in intensive English programs back home. Many are young and on generous scholarships, but not all of them. On the teaching side of things, I think that means we should really do our best to take advantage of class time, such as creating opportunities to write or work on presentations in class (with teacher and peer support!), and perhaps not assign too much weight to "busywork" like workbook pages that are so easy to just assign as homework.

Well, I could probably go on and on, but I'll stop here. These were some of the bigger thoughts floating around in my head since finishing the class last week. Even though a little negativity slipped in to those reflections, it was overall a really positive experience and I'm already looking forward to my next chance to be a language student.

Speech Intelligibility and Grain Size - a SLRF 2017 preview post

Next month, I'll be giving a talk called "Explaining Intelligibility: What matters most in L2 Speech?" at Second Language Research Forum 2017 in Columbus, Ohio. That talk will examine features of L2 Korean speech that caused intelligibility issues, based on data from 30 Korean native speaking listeners. This post is a preview, where I'll show some of my initial summary data.

In second language speech, a handful of constructs are widely studied and considered important: intelligibility, comprehensibility, accentedness, and fluency. Arguably, intelligibility is the most important, as you can't really have successful communication without it. When we think about intelligibility, we might think of it holistically to describe a person's general ability or a person's performance in some speaking context. Language tests are a good example of this- the word "intelligibility" pops up in rubrics that are used to assess someone's speaking performance on a test task. Visually, we might think of people having different degrees of intelligibility looking like this:

Fig 1. Average proportion of eojeols (words) in a picture description task correctly transcribed by 30 Korean listeners. 

In Fig. 1, we do see some variation- around 80% or so of Speaker B's words (actually 어절, eojeol, a word + bound morphemes, the preferred unit of analysis in Korean linguistics) were intelligible to Korean listeners, on average, while speaker F clocked in at around 50%.  But it isn't necessarily the case that Speaker B is always 30% more intelligible than Speaker F- everybody stumbles sometimes, right? What if we look at each utterance (sentence) that the speakers produced?

Fig 2. Average proportion of eojeols correctly transcribed in each utterance.

We can see here that Speaker F, while generally having troubles with intelligibility, really dropped the ball on his/her first sentence, which was almost completely unintelligible to the listeners in the study. Speaker B is relatively intelligible throughout, but his/her first sentence was a little harder to grasp compared to the following two. Speaker A shows one of the starkest contrasts, with his/her first sentence around 50% and the rest being 80% or so. It's worth noting that the person and sentence level is about as fine-grained as a lot of L2 intelligibility research using naturalistic or contextualized speech has gone. Some studies focused on single-word intelligibility (i.e., a learner reads single words, or names single objects) do get to the word level, but I am curious about what leads to intelligibility issues in more realistic contexts. After all, these utterances aren't uniformly 50% intelligible- each word is either intelligible, or not. So we can dial in here and look at things this way:

Fig 3. Proportions of correct transcriptions for each eojeol.

To me, this is where things get really interesting. For one, we can see much more variation- there's more red and orange in this plot compared to the utterance-level depiction in Fig 2. Some words were almost completely unintelligible to listeners. Those who read Korean might notice that many of these words are names! This is interesting, and was intentional in the task design for the speakers- a name that involves a nasal assimilation at the meeting of its two syllables was chosen. But other words, often involving times and days of the week, were also quite difficult for listeners to understand. What I'm more interested in, though, is the speech features that might cause these words to be unintelligible- is it phoneme substitutions? Deletions? Pauses or repetitions in the utterance? Lexical errors? Grammatical errors? And that's my next task- building models to examine the relative impacts of these (and other) features on intelligibility.

Stay tuned!

P.S. - It's also worth pointing out that the 30 listeners were not monolithic in their overall ability to understand and correctly transcribe words. I'll be looking at listener factors in another analysis at a later time, but here's a little preview of that:

Fig 4. Proportion of eojeols correctly transcribed by each listener.

Communication Breakdown in Conversation: Pronunciation, Phonological Reception, Vocabulary... or a little bit of everything?

Alternate title: Dan overthinks a misunderstanding

As part of my ongoing efforts to at least retain, if not develop, my Korean abilities, I have been attending the Korean Conversation Table at my university (also, my wife helps put these on, so I kind of have to go- which is a good thing!).

I usually get grouped with a native speaker or two and some undergraduates who have finished all the Korean coursework and spent some time abroad, so we get some good conversation going on a range of topics. Generally, communication is doable, if a bit laborious at times, but occasionally someone experiences a breakdown, small or large.

At the most recent meeting, I experienced a major breakdown, but it was due to a really small bit that, once I realized my error, made me want to smack myself on the forehead. Another learner was talking about an internship she applied for, and she was worried that the working hours would be long- a topic I could generally follow along. In trying to explain her concerns about long working hours, she was talking about keeping a 9-5 schedule. In Korean, she was using the word 퇴근 (/twɛ.gɯn/), meaning "leaving work time", repeatedly (which makes sense). But for some reason, I kept hearing what I thought was something like 제근 (/dʑɛ.gɯn/)- a word that doesn't exist. It's worth mentioning that I think her pronunciation is quite clear (moreso than my own!), so I don't really know what was going on with my perception. I keyed on to the 근 morpheme, which I recognized as "work" and it was something that made sense given the context, so I started to think that "제근" was something like "regular work" (제 can mean something like "system" or "correct").

After sitting and listening, and spinning the gears in my head over what this word I thought I was hearing could mean, eventually I had to ask some clarification questions.  This helped me understand that she was actually saying 퇴근, a word I knew which makes total sense in this context (hence the smacking myself on the forehead). To indicate that I knew what she meant in relation to the topic, I offered the word 야근, which means "overtime"- a word that maybe I was expecting, given the topic, but she was unfamiliar with.

Meanwhile, the native speaker in the group naturally had no trouble following and helping both of us.

More L2 Korean Phonological Error Rates

Following up on my previous post on L2 Korean pronunciation over time, I have finished coding and counting for another task (this time, a more spontaneous picture description task) and have some fresh plots to share:

Lots of individual variation in direction and magnitude of changes. The treatment group, at least for the read-aloud task, is a bit less "noisy" and seems to show a stronger trend towards reduction in error rates.

Some higher-level descriptive statistics below. Lots to talk about considering the plot above and table below! L1 Chinese seem to have noticeably higher syllable error rates, but groups showed improvement overall. Back to writing this up...

		Segmental Error Rate			Syllable Error Rate			Combined Error Rate
		Pretest	Posttest		Pretest	Posttest		Pretest	Posttest
	n	M (SD)	M (SD)		M (SD)	M (SD)		M (SD)	M (SD)
Chinese	24	0.12 (0.07)	0.10 (0.04)		0.09 (0.05)	0.07 (0.04)		0.21 (0.11)	0.17 (0.07)
English	47	0.10 (0.06)	0.08 (0.05)		0.05 (0.03)	0.04 (0.03)		0.15 (0.08)	0.12 (0.06)
1st Year	47	0.13 (0.07)	0.09 (0.05)		0.08 (0.04)	0.06 (0.04)		0.21 (0.10)	0.15 (0.07)
2nd Year	24	0.06 (0.03)	0.07 (0.04)		0.04 (0.03)	0.04 (0.02)		0.10 (0.04)	0.10 (0.06)
Treatment	38	0.11 (0.06)	0.08 (0.04)		0.06 (0.04)	0.05 (0.03)		0.17 (0.09)	0.13 (0.07)
Control	33	0.11 (0.08)	0.08 (0.05)		0.06 (0.04)	0.06 (0.04)		0.17 (0.11)	0.14 (0.08)
Picture	36	0.09 (0.06)	0.08 (0.04)		0.07 (0.04)	0.06 (0.04)		0.16 (0.07)	0.14 (0.06)
Read-Aloud	35	0.12 (0.07)	0.08 (0.05)		0.06 (0.04)	0.04 (0.03)		0.18 (0.11)	0.13 (0.08)
All	71	0.11 (0.07)	0.08 (0.05)		0.06 (0.04)	0.05 (0.04)		0.17 (0.10)	0.14 (0.07)