This article first appeared in volume 16 #2 of Augmentative Communication News. The issue was written by partner Sarah W. Blackstone in collaboration with partners Janice Light, David R. Beukelman, Howard Shane, and Jeff Higginbotham. It is being reprinted with permission. For more information about, or to subscribe to ACN go to www.augcominc.com or call 831-649-3050.

Visual scene displays (VSDs), like other types of AAC displays, may be used to enhance communication on either low-tech boards or high-tech devices. The VSDs described in this issue are meant primarily to address the needs of beginning communicators and individuals with significant cognitive and/or linguistic limitations. These groups are unserved or underserved by current AAC technologies.

VSDs are designed to provide a high level of contextual support and to enable communication partners to be active and supportive participants in the communication process. Take, for example, a VSD showing a photo of picnic tables and family members eating, drinking and having fun at a reunion. If used on a low-tech board, the photo provides a shared context for interactants to converse about topics related to the reunion. When used on an AAC device, spoken messages can be embedded under related elements in the digitized photo. These are known as “hot spots.” Someone with aphasia may use a VSD to converse with a friend by touching hot spots to access speech output, e.g.,“This is my brother Ben,” or  “Do you go to family reunions?” Or, a three-year-old child with cerebral palsy may tell how “Grandpa” ate “marshmallows” at the reunion by touching the related “hot spots” on her VSD.

My thanks to David Beukelman, Kathy Drager, Jeff Higginbotham, Janice Light, and Howard Shane for their generous help in preparing this issue. These researchers are collaborators in the AAC-RERC (Rehabilitation Engineering Research Center on Communication Enhancement). They are committed to improving the design of AAC technologies so devices become more intuitive, user friendly and (bottom line) easier for people with complex communication needs to learn and to use.

Clinical News briefly explores why visual scene displays (VSDs) may be particularly useful to young children and individuals with cognitive and linguistic limitations. In addition, this section defines VSDs, their features and potential applications. Beginning Communicators summarizes completed research projects at Pennsylvania State University documenting the difficulties children with typical development have using AAC technologies. It also describes a current Penn State project that features VSDs to support the language and communication development in young children with severe communication impairments. Adults with Aphasia summarizes work underway at the University of Nebraska and describes ways in which VSDs support the social conversations of individuals with severe aphasia. Other Applications of VSDs: Beyond Conversation discusses ways VSDs can support learning, offer prompts and provide information. It highlights a research project underway at Children’s Hospital Boston that is exploring ways to use VSDs with children on the autism spectrum.

Most of the technologies discussed in this issue are still prototypes. Even so, the ideas and approaches described herein can already be creatively put to use.

Sarah W. Blackstone, Ph.D., CCC-SP, Author

Visual scene displays

Decades ago, people who were unable to speak (or write) had to rely solely on their impaired speech and other natural methods of communication. This led to frustration for both the individuals and their communication partners. During the 1970s and 1980s, the disability rights movement, new laws and new public policies brought children with severe physical, communication and cognitive disabilities into their neighborhood schools and gave adults who had been sequestered at home or living in institutions the means to move out into their communities. Concurrently, teachers, therapists, family members and advocates found ways for individuals with limited speech and language skills to communicate, by providing a growing range of AAC techniques and strategies, such as manual signs, gestures, pictures and graphic symbols. By the mid 1980s, the field of AAC had been firmly launched.

Today we have machines that “talk” and a growing cadre of professionals and advocates working to provide functional communication to people with complex communication needs. The AAC community has helped pave the roads to education, employment and community participation. AAC techniques, strategies and technologies have enabled some people with severe communication impairments to fulfill the same social roles as their nondisabled peers and to pursue their personal goals and dreams. However, many people who have complex communication needs are still not benefiting from AAC technologies.

Current AAC technologies

Simple AAC devices enable users to express a small number of messages. While useful, the messages are typically restricted to a few preprogrammed phrases. Other devices are more complex and have dictionaries of symbols to represent language, an ability to store and retrieve thousands of messages, colorful displays, intelligible voice output with male and female voices, multiple access options, environmental controls, Internet access and rate enhancing strategies.

A common complaint about complex AAC devices, however, is they are difficult to use and time consuming to learn. Research and clinical experience suggest that AAC technologies often place enormous cognitive demands on the individuals who use them. For example, current AAC devices that are available to beginning communicators and individuals with cognitive and linguistic limitations are comprised of isolated symbols, whose meaning must be learned. These symbols are typically arranged in rows and columns. Users must navigate through pages and pages of symbols to retrieve phrases or words to construct meaning, or they must learn to use coding strategies (abbreviation expansion, semantic compaction, Morse code). The cognitive demands of these approaches make learning time-consuming. Some people manage to master these systems, but many do not.

Visual scene displays

Advances in mainstream technologies (e.g., miniaturization, multi-functionality, storage capacities, processing speed, video, photo and voice capabilities, locator functions) offer options that may better address the requirements of people with complex communication needs who have, until now, been unserved or underserved by AAC technologies. One approach researchers are currently investigating—­visual scene displays (VSDs)—may be particularly helpful to beginning communicators and individuals with cognitive and linguistic limitations. VSDs offer a way to (1) capture events in the individual’s life, (2) offer interactants a greater degree of contextual information to support interaction and (3) enable communication partners to participate more actively in the communication process.

Visual scene displays (VSDs) portray events, people, actions, objects and activities against the backgrounds within which they occur or exist. These scenes are used as an interface to language and communication. A VSD may represent

• a generic context (e.g., a drawing of a house with a yard, an office with workers or a school room with a teacher and students.)
• a personalized context (e.g., a digital photo of a child playing in his bedroom or a picture of the family on a beach while on vacation.)

Note: A VSD may also have animated elements (i.e., intelligent agents) that can move around a scene.

Differences between VSDs and grid displays

Figure 1 illustrates and Table I contrasts some of the variables that differentiate traditional AAC grid displays from visual scene displays: (a) primary types of representation used (e.g., photos, symbols, letters); (b) extent to which the representations are personalized (e.g., involve the individual in the scene, are familiar or generic); (c) how much context is provided in the representation (e.g., low=isolated concepts; high=concepts provided in a personal photo); (d) type of layout (e.g., grid, full scene, partial scene, combination grid and scene); (e) how displays are managed (e.g., menu pages, navigation bars);  (f) how concepts are retrieved (e.g., by selecting a grid space, pop ups, hot spots, speech keys) and (g) primary uses of the display (e.g., communication of wants/needs, information exchange, conversational support, social interaction, social closeness).

Both VSDs and grid displays can be used on low- and high-tech devices. Traditional AAC displays are configured in grids, most often in rows and columns, with elements comprised of individual graphic symbols, text and/or pictures arranged on the display according to specific organizational strategies (i.e., by category, by activity, alphabetically, or idiosyncratically). The elements on a grid display are decontextualized so that users can construct desired messages on numerous topics using the same symbols.

In contrast, the elements of a VSD are components of a personalized or generic scene, i.e., pictured events, persons, objects and related actions, organized in a coherent manner. As such, the display provides a specific communication environment and a shared context within which individuals can tell stories, converse about a topic, engage in shared activities and so on. VSDs also enable communication partners to assume a supportive role, if needed, to make interaction easier for beginning communicators and individuals with cognitive and linguistic limitations.

Contributors to this issue (Beukelman, Light and Drager, Shane) currently are investigating various features of VSDs to determine which can best meet the needs of different population groups, including young children with complex communication needs, children with autism and adults with aphasia.   

Current applications of VSDs Depending upon the age and cognitive and linguistic needs of the individual using the display, VSDs can be configured to serve several purposes.  

• Stimulate conversation between interactants. Adults with aphasia and very young children often find it difficult to have conversations because of their limited language skills. Yet, within a shared context and with a skilled communication partner, conversations can occur. For example, a man with severe aphasia, still has much to share with his friends. Using a VSD with digitized photos of his trips and family activities, he can set topics, converse and navigate between topics when a friend comes to visit.
• Support play, share experiences and tell stories. Bill wants to play with his toy farm, so his Mom finds the page on his AAC system that has a large photo of them playing with the farm. Bill touches the part of the photo that has the bucket of feed that the farmer is carrying. Bill’s touch retrieves the speech output ‘feed.” His Mom gets the farmer and waits. Bill then touches the cow in the photo to retrieve the speech output “cow.” His Mom touches the farmer, the feed and the cow in sequence and then says, “Oh, the farmer’s going to feed the cow,” and acts it out.
• Facilitate the active participation of interactants during shared activities. In this application, the activity is “in” the VSD. For example, Joe and his mom read the scanned pages from Brown Bear, Brown Bear, or sing “Old Macdonald had a farm” using a picture of a farm with various stanzas of the song.
• Provide instruction, specific information or prompts. Many children with autism are drawn to electronic screen media (e.g., videos, DVDs, TV) and, therefore, may find VSDs engaging. An animated character, known as an intelligent agent, may be useful in teaching language concepts or prompting the use of speech to label objects, activities and attributes.

Potential benefits of VSDs

VSDs are personalized, create a shared context, provide language in context and enable partners to assume more responsibility during interactions. They also shift the focus away from the expression of wants and needs toward social interaction and the exchange of ideas and information. VSDs can be highly personalized, which is important for young children learning language and can be helpful for others as well (e.g., individuals with aphasia, autism, Down syndrome, etc.) VSDs may reduce cognitive demands, make learning easier and offer beginning communicators and individuals with limited cognitive and linguistic skills immediate success. Researchers report that very young children and individuals with aphasia and autism have required minimal, if any, instruction before using devices configured with VSDs to communicate.

Persons who rely on conventional AAC technology may also benefit from VSD support of their conversational interactions in much the same ways that typical speakers use photo albums to support their communication. For example,

Tom relies on AAC technology because his speech is largely unintelligible as a result of amyotrophic lateral sclerosis. When he takes a trip or goes to an event, his family takes digital photographs and develops a simple slide show. When meeting with friends, the family runs the slide show in the background on a portable computer. His guests often take a couple of moments to view the show before talking with Tom. If his AAC technology contained VSD capability in addition to spelling, word prediction and message retrieval, he would probably interact more easily and effectively.

AAC-RERC researchers are striving to inform the field about the ways in which VSDs and hybrid (combination of grid and VSD) displays are effective with specific populations. They are observing the impact of VSDs on early language/concept development and on social interaction. Neither the current nor the potential value of ongoing VSD research and development should be underestimated. By continuing to explore the many unanswered questions about VSDs, their configurations and their uses, these pioneering researchers are opening new windows of communication opportunities for individuals we don’t yet serve very well.