Nuance’s 2012 Understanding Healthcare Challenge: Natural Language Processing Meets Clinical Language Understanding

I’ve written a lot recently about natural language processing in healthcare.

Language technology and workflow technology have lots of interesting connections. As I previously discussed:

  • NLP and workflow often use similar representations of sequential behavior.
  • Speech recognition promises to improve EHR workflow usability but needs to fit into EHR workflow.
  • Workflow models can provide context useful for interpreting speech and recognizing user goals.
  • NLP “pipelines” are managed by workflow management systems.
  • Workflows among EHRs and other HIT systems need to become more conversational (“What patient do you mean?”, “I promise to get back to you”, “Why did you ask me for that information?”)

So, writing about NLP reflects the name of this blog: EHR Workflow Management Systems.

Therefore I was delighted when Nuance Communications, provider of medical speech recognition and clinical language understanding technology, approached me to highlight their 2012 Understanding Healthcare Challenge. An interview with Jonathon Dreyer, Director, Mobile Solutions Marketing follows. In a postscript I add my impression of going through the process of gaining access to Nuance’s speech recognition and clinical language understanding SDKs (Software Development Kits).

By the way, I think the idea and model of a vendor sponsoring a challenge and giving away developer support packages is one of the best ideas I’ve come across in quite a while. If anyone else decides to follow suite, I’d love to interview and highlight your SDK and related resources too!

Interview with Jonathon Dreyer, Director, Mobile Solutions Marketing, Healthcare Division at Nuance Communications, about the 2012 Understanding Healthcare Challenge

Jonathan, thanks for taking time out of your schedule for speaking with me! I enjoyed interviewing Nuance’s Chief Medical Informatics Officer, Dr. Nick van Terheyden and turning that into a blog post.

Video Interview and 10 Questions for Nuance’s Dr. Nick on Clinical Language Understanding

So I look forward to doing the same with you!

Dr. Nick did a great job of putting speech recognition and natural language understanding into clinical context. That interview was directed toward user-clinicians. Let’s focus this interview on developers. (As well as curious clinicians; lots of physicians are learning about IT these days!)

1. What’s your name and role at Nuance?

Jonathon Dreyer, Director, Mobile Solutions Marketing, Healthcare Division at Nuance Communications. I manage our 360 | Development Platform. That’s speech recognition and clinical language understanding in the cloud. It includes a variety of technologies for desktop, web and mobile, including speech-to-text, text-to-speech, voice control and navigation, and clinical fact extraction.

2. I understand Nuance is sponsoring a contest or challenge. What’s it called? What does it entail?

It’s called the 2012 Understanding Healthcare Challenge. The deadline is Friday, October 5th. Just go here…


…and fill out answers to some questions and submit them to Nuance.

About a year and a half ago we launched our speech recognition platform in the cloud. Earlier this year we had sponsored a successful challenge in which several dozen developers participated. Recently we released our clinical understanding (CLU) services platform and software development kit (SDK). The CLU engine can take unstructured free text from dictation (generated by our speech recognition engine), or existing text documents, and extract a variety of data sets.

The key difference between the current 2012 Understanding Healthcare Challenge and the previous speech recognition challenge is that in the previous challenge developers integrated speech recognition into applications, but in this challenge we’re looking for great ideas. The 2012 Healthcare Understanding Challenge has list of questions: What clinical data types would you use? What value is provided to end-users? And so on.

At the end of the challenge application deadline, October 5th, we’ll select three winners. Each will get a free developer evaluation and a year of developer support. These packages are worth $5,000. Essentially Nuance will help developers bring their idea to life and then to help market it.

[CWW: The 2012 Understanding Healthcare Challenge application form lists the following areas application: EMR/Point-of-Care Documentation, Access to Resources, Professional Communications, Pharm, Clinical Trials, Disease Management, Patient Communication, Education Programs, Administrative, Financial, Public Health, Ambulance/EMS, Body Area Network.]

3. Pretend I’m a programmer (which I occasionally am): how does Nuance work and how does Nuance work with mobile apps?

Our platform supports both speech and understanding. That’s both the speech-to-text service and then the text-to-meaning structured data service. A developer can sign up for one or both of these services. Depending on country and language they’ll access different relevant content and resources.

For example, a US developer can sign up for a ninety-day speech recognition evaluation (eval) account (360 |SpeechAnywhere Services), including SDKs and documentation, or the 30-day CLU eval (360 | Understanding Services). The developer portal has lots of educational and technical documentation, plus online forums and contacts for support. The SDKs are relatively simple to use. All you need are just a couple of lines of code. Within an hour most developers are generating their first speech to text transactions.

[CWW: I signed up for access to both the speech recognition and clinical language understanding evaluation documentation, software, and services. I’ll tell you what I found in a postscript at the end of this post.]

4. I’m seeing more and more speech-enabled mobile apps in healthcare. It’s not always obvious which speech-engine or language technology powers them. Have any numbers you’d care to share?

We’ve had about 300 developers come through our program. Several dozen have reached commercial status and their products are commercially available today. To date, we’ve worked with a lot of startup vendors. But in the next few weeks we’ll also be announcing partnerships that focus on providing speech recognition mobility to a number of well-known EHR vendors. It’s safe to say we are powering a “fair number” of these mobile healthcare applications.

5. Is there a “Nuance Inside” option? (after “Intel Inside”)

The phrase we use is “Powered by Nuance Healthcare” Plus there are a couple of visual indicators. In an iPhone or Android app, or in a web browser, there’ll be a little Dragon flame. This automatically appears in text fields when the speech recognition SDK is integrated into a product. And there’s a little button with the Dragon flame. Help menus also have a “Powered by Nuance Healthcare” badge.


6. How cross-platform is the technology? Does it rely on specific libraries compiled into iOS or Android apps?

SDKs include iPhone/iPad, Android, Web, and a .NET version for desktop windows.

7. I’m @EHRworkflow on Twitter and my blog is called EHR Workflow Management System, where I talk about workflow, usability, and natural language processing. It seems to me that a bunch of interesting technologies are coming together, all of which potentially contribute to more usable EHR workflow. Here’s just a few of these ideas: workflow, process, flexibility, customization, context of use, user intent, intelligent assistants, etc. I know that was a long preamble for this question, but could you react to some of these topics with respect to speech and language technology?

Well, for example, the latest version of the speech engine has some conversational capability. It can also do text-to-speech, so EHR can, potentially, speak up. Command-and-control functionality allows users to ask questions (such as “What are the vitals for my patient” ) and to navigate through an application.

The clinical language-understanding engine is a different use case, from the developer’s point of view, because it’s not directly dependent on an audio control. You send narrative text to the server and you get back useful data set extracted from the text. So CLU depends on the use case and what our development partner is trying to accomplish.

[CWW: The 2012 Understanding Healthcare Challenge application form lists the following datasets: Problems/Diagnoses, Medications, Allergies, Procedures, Social History, Vital Signs.]

However, if you’re doing pure speech recognition, then you add a couple lines of code and enable every text field. With our new command-and-control functionality users can also directly address such controls as checkboxes “Check this, check that, etc.” You can also integrate a medical assistant as in “Who are my patients for the day” or “Show me patient Mary Smith”, “When was her last visit?”

So we’re going beyond speech to text to more intelligent voice interactions. Speech recognition and clinical language understanding allow the intent of users to directly drive workflow or process.

We’ve leveraged our speech recognition and clinical language understanding technology to build our own workflow solutions, such as Dragon Medical 360 | M.D. Assist. It not only improves workflow from a technical perspective, streamlining and so forth, but also asks intelligent questions. So if a user mentions heart failure the system can check for specificity and ask the user for more information if needed.

Relative to workflow-related use cases, we’re seeing a lot of specialty-specific integrations, from general medicine and emergency medicine to dermatology and chiropractic. We’re also seeing a lot of EHR-agnostic front ends. These mobile workflow tools sit on top of legacy EHRs or even connect to multiple EHRs applications. One example even uses location services to reason from clinic or facility location to which back-end EHR system to which to connect. These systems intelligently recognize and reason from context to user-intended workflow. Adding speech recognition and clinical language understanding into this mix provides even more value. Every week I see something new and exciting from our development partners.

We have a base set of functionality we provide. If you want to do simple things, you can just add our code to your application. But you can also customize voice commands to work with your preferred keystrokes or macros.

We know that users were going to be doing some form of touching, speaking, typing, swiping and so on. For example, natively in iOS and Android, if you swipe to the right that starts your dictation. If you swipe to the right again, it stops. If you swipe to the left it “scratches” the last phrase.

Here’s an interesting workflow. If you tap a sequence of text fields, and speak into each, you don’t need to wait until the text appears in one text field before moving onto the next. All we need is a low-bandwidth connection, but if the network is congested or slow for any reason, users can move on while text catches up. We call this “Speak Ahead”. In other words, the user can forge ahead at own pace and we’ll accommodate them.

8. Tell me about how you support third-party developers.

[CWW: Jonathan and I spoke briefly about this. Since I personally registered as a developer, I’ll spill those beans in a postscript below.]

9. What are the ideal technical skill prerequisites for third-party developers?

The technical skills are not so much about our end. If developers are already building mobile apps for the iPhone and Android then they already have more than enough technical skill to integrate our technology. You’ll get more of an appreciation for this when actually get a chance to see the SDKs. We tried very hard to make integration as easy and effortless as possible. Relative to the current developer portal education and support resources, you’ll see that too. But we will also be updating the portal before the end of the year. We’ve had a lot of feedback from hundreds of developers and we’re continuing to leverage this to feedback to aim for rich educational content and a robust developer experience.

10. OK! Let’s close with the deadline for you 2012 Understanding Healthcare developer challenge and where folks go to apply.

Friday, October 5th. Go to


Thank you again, Jonathon!

Thank you Chuck!

Well, that’s my interview with Jonathan Dreyer, Director, Mobile Solutions Marketing, Healthcare Division at Nuance Communications. I certainly learned a lot. I hope you did too! And, please, get in there and apply for one of those $5,000 development support packages. Create something great. Then come back here and tell me about it!

Many thanks to Gordon Segersten, of Nuance Healthcare Business Development, for walking me through the application for access to the speech recognition and clinical language understanding evaluation materials and services.

PS In order to get my own impression of at least the first couple steps of becoming an Nuance third-party developer, I went to Nuance’s 360 | Development Platform developer support site at and signed up for 90-day and 30-day free evaluation access for speech recognition and clinical language understanding SDK material and support.

When I investigate integrating a third-party product or service, I start of with a short list of questions.

  1. Is there an SDK (Software Development Kit)?
  2. Does the SDK appear well documented? Lots of content? Well organized? Current? Etc.
  3. Is there sample code? In the right programming language? (i.e., the language the application you’re integrating the speech/language tech into, though there are often workarounds if not)
  4. Are data formats based on standards familiar to the developers in question? XML (eXtensible Markup Language), CDA (Clinical Document Architecture), etc.
  5. Are there support forums? Are they well populated with recent discussions: questions, answers from support, contributions from other users, etc?

After being accepted into the developer evaluation programs, I observed that the answers to these questions was “Yes.” In fact, I quite liked what I saw!

PSS One additional postscript: Whenever I get an opportunity to look under the hood of an EHR or HIT systems, I always look for workflow technology such as I tout in in my blog or via my Twitter account. It’s not always obvious! But it is often the secret in the sauce that makes some systems more customizable than others. Intriguingly I found what I was looking for:

“The data extraction platform contains a number of components that can be assembled in a pipeline to perform a specific extraction task. The actual execution of that task is performed by a workflow engine that takes this pipeline as a configuration parameter.”

Ha! Workflow engine sighting! I’ve written about NLP pipelines and workflow engines elsewhere in this blog, just in case you are interested!

Cited! My “Process-aware EHR BPM Systems: Two Prototypes and a Conceptual Framework”

I’m always looking for new material on the Web about process-aware information systems in healthcare. They go by many names: EMR workflow systems, EHR workflow management systems, mentions of “workflow engine” in healthcare contexts, clinical groupware (see my definition), EHR BPM systems (for Business Process Management), and adaptive case management, to list a few. Since I also write about these subjects, I often stumble upon my own blog posts, white papers, and tweets. Occasionally I even find my work cited, which really sends my serotonin through the roof (figuratively speaking, of course).

Here’s a mention of my Process-aware EHR BPM Systems: Two Prototypes and a Conceptual Framework that I presented at MedInfo’s 2010 conference in South Africa. It’s in Issues in Information Science: Informatics: 2011 Edition. Fascinating how one can so easily embed the actual page into a blog, such as I have done below.

If you’re interested in more information about “Process-aware EHR BPM Systems” and its associated presentation, the slides and speaker notes are in my blog post Clinical Intelligence, Complex Event Processing and Process Mining in Process-Aware EMR / EHR BPM Systems.

There’s also a follow-on paper about process mining EHRs and process-aware ideas in healthcare called EHR BPM: From Process Mining to Process Improvement to Process Usability that I presented at the 2012 Healthcare Systems Process Improvement Conference in Las Vegas.

I continue to see more-and-more references to workflow management systems, business process management systems, and process-aware information systems in healthcare. In fact, if you’ll humor me, I’d like to make an analogy, between the ragtag band of BPM enthusiasts (and not so ragtag, there are some really big BPM companies with crosshairs on healthcare) and the Star Wars Rebel Alliance. Hmm. Who or what is the Death Star? I’d argue that the Death Star is what I call the structured-document based EHR. What’s the alternative? The structured-workflow based EHR.

Here’s a bit more on structured-document versus structured-workflow based EHRs. A structured-workflow based EHR has a workflow engine that executes process definitions to save users mental and physical effort, and achieve higher levels of effectiveness, efficiency, and user and patient satisfaction. A structured-document based EHR, without a process-aware, workflow management systems-style foundation, forces the user to become the workflow engine. Users figuratively drag the EHR through a patient encounter, instead of the EHR anticipating user needs and fulfilling them in the background or with a minimum of user effort.

I’m *not* arguing that we need to do away with structured-documents. Just that we should not base EHRs *on* structured-document architectures. Start with structured-workflow management systems, and then implement structured-document functionality on top of that. You’ll end up with much more flexible and usable EHR workflow. And structured-documents too!

That’s having your cake and eating it too.

User-Centered vs Activity-Centered Design: A Follow Up to “User-Centered EHR Design Considered Harmful”

About six months ago I wrote User-Centered EHR Design Considered Harmful (Try Process-Centered Instead). Healthcare Scene’s EMR and EHR blog gave it a mention and it rocketed around the Twittersphere for a couple days.

I recently stumbled on this presentation about Activity-Centred Design (British spelling). I thought several of its slides (adapted) would make a nice table about benefits versus drawbacks of user-centered versus activity-centered approaches to design.

Benefits Drawbacks
User-Centered Improved usability

Fewer errors during usage

Faster learning times

Humanises software processes

Minimises guesswork

Understands user’s cognitive style

Reduces user mistakes and improves recovery

Focuses on the user

Improvements for one group of users can be detrimental to another

Users are moving targets 

Users don’t always know what they want

Research is expensive, unreliable, time consuming

Tries to fix human mistakes rather than focussing on users accomplishing a task

Activity-Centered Users can adapt better than the technologies

Active observation vs passive observation

Internal data: Statistics, heat-maps, eye-tracking

Learn about user behaviour, rather than the user

Activity has purpose. User has behaviour. Purpose is more predictable than behaviour

UI evolves over time to facilitate user activity

Uses analytic and cognitve data from users

Solves problems instead of user mistakes

[I’m sure there are disadvantages to activity-centered design, but none were listed in this presentation]

Food for thought!

Healthcare Workflow-as-a-Service? One-Minute Interview and 6 Questions for KISSFLOW CTO

(Remarkably, two days after this interview, about healthcare workflow-as-a-service, a healthcare workflow-as-a-service won KiSSFLOW’s War-of-Workflows context. Here’s the press release. Thanks for mentioning me KiSSFLOW!)

A single tweet can lead to all kinds of interesting conversations! For example, I tweeted about a new workflow-as-a-service for Google Apps.

Why? I’m always looking for new, easy-to-use and deploy workflow technology that can be useful in healthcare. (Hmmm. Momentary flight of fancy: Meaningful Use 2.0 certified EHR based on Google Apps and KISSFLOW? Let’s see, that’d require a variety of interfaces, lots of domain knowledge, etc. But just think of the scalable, customizable, clinical teamware… … [Snap out of it, man!] Sorry.)

By the way, just in case you think Google Apps are non-starters in healthcare due to HIPAA, here’s a systematic argument that Google Apps can be used in a HIPAA-compliant manner. Be sure to check out this Google Apps and HIPAA Security Standards Matrix!

Anyway, soon I’m tweeting back and forth with KISSFLOW‘s CTO Mani Doraisamy, half a world away in Chennai, India. One thing leads to another and I nabbed a One-Minute Interview via Skype (below). Afterward, by email, Mani followed up with answers to some very basic questions about how workflow, cloud, mobile, and social are coming together, plus potential implications for healthcare.

First, here’s the One-Minute Interview. Mani is so enthusiastic and charming! Be sure to stick around until the second half so you see the big room he’ll be speaking to the next day.

One-Minute Interview with Mani Doraisamy, CTO, KISSFLOW

At the very end of this post, I added Mani’s unveiling of KISSFLOW at Google I/O. Be sure to watch it too! He demos creating of a workflow in just five steps starting at 3:30.

Mani does not work specifically in healthcare, but in “off camera” conversation, before the official One-Minute Interview, Mani was remarkably well informed. He also asked astute questions about the state of business process management (BPM) and adaptive case management (ACM) in healthcare. So, my questions sought to extrapolate Mani’s expertise into healthcare IT and the EHR space, particularly regarding workflow-as-a-service.

My last question for Mani, number six below, simply scratched a personal intellectual itch: how do cloud, mobile, and social fit together? I recently tweeted about this potentially explosive (in a good way) combo.

Mani’s answer to this question is amongst the more concise and perceptive I’ve yet read. Other industries, particularly BPM and ACM, seem further along in leveraging not just cloud, mobile and social, but, intriguingly, leveraging the combination of cloud, mobile and social technology. So read Mani’s answer and ponder it’s implications for healthcare.

Email from Mani Doraisamy, CTO of KISSFLOW:

Hi Charles,

Please find my answers to your questions below:

1. First of all, to make sure everyone is starting off on the same page, what are Google Apps?

Google Apps is an office productivity software on the web. It has mail, spreadsheets and document editing capability similar to Microsoft office. But, you dont have to install it in your machine. It can be accessed on your web browser. It shines when multiple people in your team are collaborating together on a document or spreadsheet.

2. What is KISSFLOW and what value does it provide to Google Apps users?

Google Apps provides free form collaboration. KiSSFLOW is a workflow solution to streamline such collaboration. For example, treatment guidelines for patients requires the team to streamline collaboration for diagnoses. With KiSSFLOW, you can specify the sequence of diagnoses based on the treatment guidelines. Automating such process reduces errors, improves patient safety and reduces cost.

3. Can you expand a bit more on this whole idea of “Workflow-as-a-service”? What are it’s implications for IT in general? Health IT in particular?

It has 2 implications:

  • Traditionally, you buy and install workflow software in your computer and upgrade them when new version is available. With workflow-as-a-service, it is available on your web browser over the internet. New versions are automatically upgraded and fully managed by the workflow service provider.
  • You have to pay upfront for traditional workflow software. With workflow-as-a-service, you dont have to pay upfront. You can pay based on your usage month-on-month.

It benefits industries like healthcare that are not tech savy, as it moves infrastructure and software headaches back to the vendors. It lets healthcare companies to focus on their core competency instead of worrying about IT.

4. Is it really possible for a non-programmer to create and customize their own workflows with KISSFLOW?

Yes. There is a difference in the way business users (non-programmers) and programmers think:

  • Business users think only about main flow e.g: what happens when you approve.
  • Programmers think of all the alternate flow irrespective of usage (e.g: what happens when you reject or when there is a clarification).

KiSSFLOW is created with business users in mind. With KISSFLOW, business user has to define only the main flow. KiSSFLOW takes care of the alternate flows.

5. Do you know of any current/potential/planned/imagined uses or use cases for KISSFLOW in healthcare?

KiSSFLOW can be used for automating clinical processes like patient care, safety initiatives and admission processes. Administrative processes like documentation, assessment completion and consent forms are also good use cases for KiSSFLOW.

6. In healthcare, as in other industries, there’s much discussion about cloud and mobile and social, but other industries are further along using and even combining them. From your perspective, how are cloud and mobile and social coming together? Lessons for healthcare?

  • Cloud is about operational efficiency – How do you outsource your IT in a cost effective manner. It lets you focus on your core business problem.
  • Mobile is about accessibility and agility – How fast and easily can you access data from anywhere to server better.
  • Social is about customer retention and reputation management – How do we get customer feedback & improve on them and how do we spread the word from happy customers.

Together they create a powerful combination that help us to be better organized internally and customer focused externally.

Please let me know if you any more information!


And here’s Mani introducing KISSFLOW at Google I/O:
Mani creates a Google App workflow in just five steps starting at 3:30

If you’re intrigued, and do something interesting in healthcare with KISSFLOW, please come back here and tell me about it!



PS You can follow KISSFLOW on Twitter at @KISSFLOWBUZZ and Mani Doraisamy at @ManiDoraisamy.