1.5. Experiment Design

1.5.1. Choosing the Experiment Location

After several years of testing auditory signage in controlled and small environments, it was highly desirable to test the system in a much larger urban transit environment with “real world” simulations and tasks.   The robust environment that was available at the San Francisco Caltrain station was chosen.  It provided the ability to test blind subjects transferring between four different transit modes in an area that had 51 Talking Signs(R) transmitters.   This rich environment was a superb place to conduct much needed research and a very broad range of questions and tasks were designed to take full advantage of the opportunity.  
 
It is quite difficult to get a group of blind travelers together at the same site for testing and it can also be quite expensive and time-consuming.   While trying to limit subject fatigue and stress, attempts were made to elicit the most information from the experiment by asking many relevant questions and performing many complex field tasks.  
 
1.5.2. San Francisco Caltrain Experiment Site
The San Francisco Caltrain station environment offered a unique opportunity to test RIAS in a realistic urban multi-modal setting.   The train station takes up the entire block face along 4 th Street.  Across one intersecting street (King Street) is the Municipal Railway Muni “N” Judah line Light Rail station.  On the other intersecting street (Townsend Street) outside the train station is a cabstand and across 4th Street are several bus shelters.   Figure 1.1 shows a diagram of the area used in this experiment and all Talking Signs ® installations.  Figure 1.2 shows a blowup of the 4th and King Street intersection installations.   Table 1.1 lists the exact verbal message heard at each of the RIAS transmitter locations.

Figure 1. 1 Talking Signs® Installation at Caltrain Station

 

Figure 1. 2   Street Corner Detail

 

Table 1. 1   Legend for Figure 1.1 and Figure 1.2 .

San Francisco Caltrain Station and Surrounding Environment

Verbatim Talking Signs® Message and Location Legend

01  Townsend Street Entrance to Caltrain Station

02  Exit to Townsend Street

03  Townsend Street Entrance to Caltrain Station

04  Exit to Townsend Street

05A Newspapers, Magazines, Snacks, and Candy

05B Flowers and Drinks

06  Refreshments, Coffee, Hot Dogs, and Doughnuts

07  Exit to Fourth Street

08  Fourth Street Entrance to Caltrain Station

09  Tickets and Information

10  Caltrain Ticket Machine and instructions

11  Waiting Room, Restrooms, Public Phones, Drinking Fountain

12  Exit to Station

13  Public Phones

14  Women’s Restroom

15  Men’s Restroom

16  Drinking Fountain

17  Exit to Fourth and King Plaza

18  Caltrain Waiting Room, Restrooms, Public Phones, Drinking Fountain

19  Exit to Fourth and King Street Plaza

20  Fourth and King Street Plaza Entrance to Caltrain Station

21  Plaza Entrance to Train Platforms

22  Exit to Fourth and King Street Plaza

23  Plaza Entrance to Train Platforms

24  Exit to Fourth and King Street Plaza

25  Plaza Entrance to Train Platforms

26  Exit to Fourth and King Street Plaza

27  Exit to Station

28  Platform Eleven (on the Right) and Platform Twelve (on the Left)

29  Exit to Station

30  Platform Nine (on the Right) and Platform Ten (on the Left)

31  Exit to Station

32  Platform Seven (on the Right) and Platform Eight (on the Left)

33  Exit to Station

34  Platform Five (on the Right) and Platform Six (on the Left)

35  Exit to Station

36  Platform Three (on the Right) and Platform Four (on the Left)

37  Exit to Station

38  Platform One (on the Right) and Platform Two (on the Left)

39  Passenger Pickup and Drop off, Taxi stand

40  Traveling east on 700 block of 4th St.  toward King Street.   For Muni Light Rail Raised Platform cross 2 south bound lanes of King Street.   Push button to activate pedestrian signal.

41  Walk Sign King Street.   Wait King Street

42  Walk sign 4th street.   Wait 4th Street

43  Traveling north on 100 block of King Street toward 4th St.  Muni bus shelter for #15 and 91 owl on north side of 4th Street.   Push button to activate pedestrian signal

44  Walk Sign King Street.   Wait King Street

45  Traveling west on 800 block of 4th St.  toward King Street.   Caltrain station on west side of King Street.   Push button to activate pedestrian signal.

46  Fare machine for Muni “N” Judah line

47  Ramp up to Muni platform

48  Traveling south on 200 block of King Street toward 4th Street.  Caltrain Station on south side of 4th Street.   Push button to activate pedestrian signal

49  Walk sign 4th Street.   Wait 4th Street

50  Pay phone and bus shelter for Muni bus line #15

1.5.3. Outline of Tasks

Making transit transfers and mode changes can be difficult barriers for many people with vision restrictions and for those with other print-reading disabilities.  The tasks in the field experiment determined if blind people would be able to use Talking Signs® to safely and easily move from one form of transit to another and to efficiently find and use amenities and cross streets with much less effort and time than when attempting these mode changes with their regular method of travel.  Hypothesis 1 is tested in these field tasks where 30 vision-impaired people make five different mode transfers, making realistic stops along the way for various amenities and ticketing or fare-paying tasks.   Altogether, subjects traveled five different routes to simulate making five transfers using four different forms of transportation (a detailed description of each route and intermediate stops, along with subject’s performance, is given in Section 3.1 ).  Travel time, errors, and requests for assistance during the experimental trials were recorded.   Because of unforeseen construction barriers and time constraints, the principal researcher acted as a sighted guide and walked the subjects to several locations where no measurements were taken.  
 
Subjects were asked classification questions to determine the characteristics of their blindness or vision loss and to reveal information about variables such as age, education, sex, and basic travel skills and abilities.   Pre-test interviews gave information on the subjects’ travel and activity behavior, and perceived difficulties while using transit and making transfers.  Following the experiment with RIAS, the same questions were asked to determine if changes in transit attitude had occurred.  Post-test interviews were conducted to measure attitudes and feelings about the possible impact of this technology.  Subjects were asked about difficulties of various transit tasks, rated their perceptions as related to the relative benefits of the technology, and data gathered about their spatial understanding of the environment.   Subjects compared their regular method of travel to their experience when using the RIAS, specifically rating street crossings, in-terminal searching and walking tasks, and making transfers in general.  
 
Other post-test questions related to subjects’ perceived trip-making behavior and difficulties of travel in environments that would be as fully served by RIAS as the test environment.   This provided data to determine if the technology was perceived as improving their ability to use transit, their frequency of using it, and whether it improved their quality of life by encouraging them to take trips that they had previously not taken.  Near the end of the post-test interview, questions were asked to evaluate how helpful RIAS was in various locations and if they should be installed there, as well as other evaluations of the system.  
 
Although blind and vision-impaired persons are the primary focus, this technology has much wider appeal for other print-handicapped people.  
 
1.5.4. Question Design
 A combination of experimental methods and techniques was used in this research.   Pre and post-test interviews were given to all 30 subjects.   Many of the questions were the same in both conditions.   They consisted of a combination of five-point rating scales for evaluating various perceptions.  Numerical data on trips made on various modes and also trips for various activities, as well as subjects’ walking and riding times for these activities, were also collected.  Other data were collected by subjects, choosing from a list of choices or “filling in the blank” responses.   Evaluations of the perceived difference between their regular method of travel and the RIAS travel with regard to street intersections, in-terminal use, making transfers, and the effects on travel behavior were collected, using non-timed, open-ended questions.  This combination of question types helped establish methodological or convergent validity and reduced experimenter bias in the phrasing of questions and responses. 
 
Since many of the same questions were asked before and after the experiment, data were collected on “within subject” variations of the impact of the system and also on “between subjects” variations.   The within subject data helps to understand differences for each individual, no matter what their degree of blindness, skills, socio-economic statue, or other characteristics.  Using within subject data, more statistical power can be gained with fewer subjects because one assumes that, except for condition, the other variables are identical.  Between subject data allows for testing the differences between wide ranges of subjects.
 
1.5.5. Field Test Design
In order to collect the most naturalistic and “real-life” data possible, a field experiment was conducted at the terminal test site.   Subjects were free to ask others for verbal assistance, if needed, and to use whatever techniques were best for them.   Except for the two conditions of using their regular method or using RIAS , no other variables were manipulated.  A time limit of four minutes was put on each walking and search sub-task in order to avoid undue stress and extreme frustration.   Subjects were given verbal information as to the locations to be visited and when to start and stop.    
 
The order of the two conditions was manipulated.   Fifteen people traveled using their regular methods first, no RIAS ( NRIAS), and then repeated the experiment using RIAS.  The other 15 subjects used RIAS on their first attempt; 10 of these 15 subjects then tried transfer tasks 1 & 2 with their regular method (5 of the 15 did not perform the tasks with their regular method because of time constraints during the study).  It is normal for people to have some learning gain on a second trial.   Previous experiments (Golledge & Marston, 1999) showed that people that used the RIAS for their first trial had travel time and error production quite similar to those who had walked the route first with their regular method and then tried the RIAS, especially in more complex routes.   Since many of the destinations have no readily accessible cues to indicate “you are here,” it is difficult to “learn” a route after only one trial, especially for those with no usable vision.   Therefore, means test between the group that used RIAS first and those that used it second allowed for between-subject testing of the same condition to determine if any learning effect took place.   The learning effect of repeated exposure in the regular method condition was also examined during the first two trials.
 
With both between and within subject experiment design in both conditions, powerful analysis should be insured, and the following comparison analysis can be made. 

Group 1

Group 2

Regular Methods (NRIAS)1st (N=15)

RIAS 1st (N=15)

RIAS 2nd (N=15)

Regular Methods (NRIAS)2nd

First two transfer tasks only ( N=10)

 
BACK TO OVERVIEW
BACK TO TABLE OF CONTENTS
NEXT SECTION