Real-time robust body part tracking for augmented reality interface

Authors:

Jinki Jung

Kyusung Cho

Hyun S. Yang

KAIST - interaction lab in korea

Link to paper

Summary:

Goal: to model the 3D body from the 2D detected body parts. a step towards more intuitive interaction in AR environment. (project NATAL)

- system deletes the background to get one single body blob.

- uses skin texture to detect face and 2 hands.

- uses lower body contour to detect leg and feet.

- users particle tracking to track the head through the frames.

The 3D pose is then estimated from the body parts recognized. An adaptive color detection is used to provide clothing and illumination independent detection of body parts. The 3D location is calculated based on the center point camera and its distance to head , and 2 feet.

Discussion:

There is one more person in MIT media lab with strange name doing the same kind of research. The skin texture detection independent of clothing color is cool.

Comments: Franck, Murat

That one there! Pointing to establish device identity

Authors:

Colin Swindells

Kori M. Inkpen

John C. Dill

Melanie Tory

Link to paper

Summary:

Goal: Facilitate Human - Computer Identification with a pointing device.

As the number of devices increase, users need to go through the cumbersome process of remembering the wireless settings and other parameters to connect to devices in the environment. The scenario can also applied to when the user wants to share documents with other users around in a wireless network. This paper provides a device which works on micro-controller with IR receivers and transmitters that can used to point at some device and connect to it easily.

A user study was conducted with 4 users each for 2 phases. First phases required users to select mobile devices and the second included selection of item from the given list. A pre and post questionnaire data were collected. The results were then compared to the performance of users with graphical list from which the users can select a device of choice.

Discussion:

The user study and the data analysis were pretty thorough. There does not seem to much preference for the device over the graphical list (UI).

Webcam Mouse Using Face and Eye Tracking in Various Illumination Environments

Summary:

Goal: environment lighting independent face detection from the webcam video stream.

The red and blue subspace is used to reduce the illumination noise. Since the skin texture for fac eis dependent on the illumination condition. The illumination condition is identified first and then face texture pattern is modeled using 10 examples for each illumination condition. Motion detection technique is used to eliminate color region similar to face in the background.

Iris gives out luminance and its detected using the sharp change in the Y component.

Discussion:

Statistics on accuracy is not given. The KNN classifier would be drastically slow in case of more examples. Each illumination setting requires a set of examples to be provided. How do we enumerate all the illumination setting ? I do not think the recognition algorithm would scale to settings that has no examples.

Comment: franck, Murat

XWand: UI for intelligent spaces

Authors:

Andrew Wilson

Stephen Shafer

Microsoft Research

Link to paper

Summary:

Goal: to build an interaction device which can be used to point and interact with multiple devices around the user. (including voice medium).

- Orientation - combination of magnetometer and accelerometer. Can be affected by the metal in the environment

- Position - uses vision techniques to find the 3D position from 2 - 2D position. Tracks 2 IR LEDs.

An average error of 6' is found in pointing tasks (pointing accuracy of device).

Dynamic bayes network is used to process the events from the wand (gestures / button events) and the speed recognition events. Speech recognition allows multimodal interaction with devices in the user environment and also provides multiple methods to perform one operation. Ignoring speech recognition results based on the pointing context helps improve the speech recognition. "Volume up" is ignored while pointing at the lights.

A user study with 10 male users was performed. Variables - Time taken to complete task, accuracy of pointing and responses to questionnaire. Scenarios - Tracking & No tracking , Audio Feedback & No Audio feedback were tested.

The users did not find the audio feedback very useful while tracking of the wand is enabled.

Discussion:

The wand is familiar to the wand simulated with Wii for Wiizards games and i do not remember another such want pointing device (mounted on mouse). Given the usage scenario, the device is novel.

Comments: franck

Online, Interactive Learning of Gestures for Human/Robot Interfaces

Authors:

Christopher Lee

Yangsheng Xu

Summary:

Goal: Provde better interactivity and control over robots (effectiveness of tele-operation).

Cyberglove is used record hand gesture input from the users. Hidden Markov model is used to recognize the gestures.

- Users perform a series of gesture

- HMM train on the gesture if they classify them into one of the existing gestures and perform the related action.

- if HMM is not able to classify,

- asks the user to disambiguate and train.

- new gesture added by the user.

Discussion:

the preprocessing procedure is very interesting. It is applicable to PowerGlove++ project by Drew & co.

Comments: Franck

RealTime HandTracking as a User Input Device

Author:

Robert Y Wang

Summary:

Goal: Easy to use and inexpensive system for user input using hands.

Design: a glove with color patterns . optimal color pattern and pose recognition algorithms have been explained in this paper.

A nearest neighbor approach is used to recognize a hand pose. every pose gives different image. So the image lookup approach is being used. The query image is normalized and down-sampled before nearest neighbor is looked up.

As the database size increased the RMS error to the nearest neighbor image decreased. To increase the retrieval rate the author compressed the image into a 128 bit binary sequence and used hamming distance to compare images.

future work:

The experiments to determine the optimal color pattern. Three dimensions of change - color, spatial frequency of blobs and icons / shapes used.

Discussion:

It is a extremely cheap solution. I would like to read about the studies on the pattern color/ spatial frequency studies to know its significance.

I think the separability / difference between the different poses is a catch. The system works well for poses for which the images are different but i do not know if that would restrict the number of the poses.

Liquids, smoke, and soap bubbles: reflections on materials for ephemeral user interfaces

Authors:

Axel Sylvester

Tanja Dring

Albrecht Schmidt

Summary:

Goal: Tangible ephemeral interface design with soap, bubble and smoke.

Setup - a round table - 20 inch in diameter, bubbles can be blown on the surface which stays on it for minutes together. The movement of the bubbles can be tracked with the camera beneath the table. The user moves the bubbles using either moist hand or slightly blowing the bubble. In a first application bubbles are used to influence the brightness and hue of the

surrounding room light. The room illumination gets brighter the bigger a recognized bubble is. Hue is set according to the position of a recognized bubble using the x and y axis to

bring up blue and red tones

Discussion:

Interesting mode of interaction. I could see applications in fun filled games for children.

Recent Developments and Applications of Haptic Devices

Authors:

Summary:

The paper talks about the present day technologies available for haptic feedback. Factors to be considered while designing a haptic device

* weight

* frictional feedback in virtual world

* Parallel vs serial mechanism - serial mechanism offer less stiffness.

* ability of the device to handle pressure exerted by the user.

* hydraulic actuators - large forces but risk of hydraulic fluid leak.

The paper discusses about a wide range of devices starting from mice to force feedback skeleton arm and their costs , applications and working

Discussion:

I dint know the abundance of such technologies. This paper introduces to lot of research being done in this field expecially research in neuromuscular stimulation.

Hand and Brain - Neurophysiology and psychology of hand movements

Summary:

Optic Ataxia. pg 18/19

- one of the types is caused by parietal damage

- deficit in spacial vision - difficult to localize an object in visual space.

- gaze, misreaching, size of grasp.

- can identify objects but cannot pick up

Visual form agnosia

- ability to reach objects but not perceive them visually.

- There are different pathways for transforming visual information in to actions and perception.

Page 22 - Experiments to analyze the precision grasp mechanism. Objects chosen that would require contour analysis.

Coming to grips with the objects we grasp: detecting interactions with efficient wrist-worn sensors

Authors:

Eugen Berlin

Jun Liu

Kristof van Laerhoven

Bernt Schiele

Summary:

contributions of this paper are threefold: First, we mention the technical procedure in optimizing a wrist-worn RFID antenna. Second, a benchmark is presented that allows evaluation of different antenna configurations. Third, an approximation algorithm is demonstrated which makes recognition of short gestures possible.

- Experiments were conducted for finding the best performance of RFID reader for oval and circular antenna. Oval antenna performed better.

- A sliding window technique is implemented to find the most probable window in which the gesture was performed.

- a series of studies were performed to evaluate the instrument. First, reading an object from a cluster of objects kept inside a box. A study with user performing different gardening tasks using 16 objects and 36 tags was performed. A long study was performed with one user performing daily activities with 29 objects and 43 tags. The user was observed for 3 consecutive days.

Discussion:

Sliding window is an interesting method to extract the most probable gesture from the continuous stream. Evaluation study was quite extensive.

The Wiimote with multiple sensor bars: creating an affordable, virtual reality controller

Authors:

Torben Sko

Henry Gardner

Summary:

Goal: Using Wiimote, nunchuk and sensor bars for interaction in multiple screen environment.

The IR camera in Wii mote can detect upto 4 IR LED source. So it can detect 2 sensor bars at a time. A software which process the data does some intelligent prediction on which bars are being detected. The dynamic model is built from the IR data received from the Wiimote. The static model is read from a configuration file and describes the mapping of IR sources to pixel positions on the displays behind them. configuration needs to comply with the following conditions: at least two IR sources should visible at most times (justifying why each sensor bar has two IR sources), two sensor bars are visible when moving from one bar to another, and no more than 4 IR sources are visible at any one time. A sensor bar / environment switching algorithm has been defined by the author in the paper.

First user testing : 12 participants , 1 female , 8 with prior experience in using Wiimote - “Involvement/Control” and “Interface Quality” were rated at 5.7 & 5.3. Physical presence sensor bars were noted as distracting.

Second user testing: 8 participants, 1 female, 6 with prior experience in using Wii mote and everyone had gaming experience. The users had to play half life 2 (FPS). Everybody stated that they enjoyed the interface. Most of them expressed frustration with the control system. Some participants reported physical discomfort due rate of rotation in the screen.

Third user testing: 4 users from second test. refined was done in seating position, slower movement rate, vertical shooting and reset rate. Positive feedbacks on the usability of the system

Discussion:

One of the few research papers where follow up studies have been conducted. The authors have conducted a good user study. I would like to know the orientation of the sensor bars that helped them restricting the number of sensors on the screen to 4 always. The dynamic modelling system would also be interesting to know.

The peppermill: a human-powered user interface device

Authors:

Nicolas Villar

Steve Hodges

Summary:

GOAL: the use of a simple circuit to enable interaction-powered devices that support rotary input as the primary interaction technique; a design for a generic wireless interface device based on this circuit. gestural interaction devices like Wiimote enable users to perform large set of gestures which have potential energy. There are lot of oppurtunities to harness this energy to use it as power source for the device.

When the user turns the knob, the microcontroller powers up and samples the inputs from the supply circuit, as well as the state of the three additional buttons. It encodes and transmits the speed of turn, direction of turn and state of the buttons (pressed/released) as a single wireless packet. As long as enough power is being generated the microcontroller continually samples and transmits packets at 5ms intervals. Some interesting extensions include providing force feedback to users by shorting the DC motor. This stops the motor and does not allow the user to turn the knob. By periodically and momentarily braking rotation in this manner, it becomes possible to dynamically generate a variety of interesting haptic effects without supplying additional power.

Discussion: Interesting field of research. Tapping potential energy from the gesture movements is awesome. The haptic feedback entensions provided by the system is interesting.

Comments:

Whack gestures: inexact and inattentive interaction with mobile devices

Authors:

Scott E. Hudson

Chris Harrison

Beverly L. Harrison

Anthony LaMarca

Summary:

Goal: Inexact & inattentive interaction. interaction that does not require visual attention and does not disrupt the current activity.

Whack gestures - a set of gestures which are combinations of whack/ tap and wiggle/ shake. An adhoc recognition engine is set up to recognize the gestures. A user study with 11 people was performed to check the accuracy of the recognition.

Discussion:

I was totally taken back by the evaluation. I was expecting qualitative evaluation of the system as such user experience evaluation. I did not expect a recognizer accuracy evaluation. It would have been more interesting if they had evaluated the user experience given a certain scenario.

Comments: Murat

Gameplay issues in the design of spatial 3D gestures for video games

Authors:

John Payne

Paul Keir

Jocelyn Elgoyhen

Mairghread McLundie

Martin Naef

Martyn Horner

Paul Anderson

Summary:

effective implementation in 2D, 3D spatial gestures present their own problems in relation to: how to present 3D gesture feedback, user performance differences,how to instruct/learn user gestures,what are familiar semiotics for 3D gestures. The authors have created a device called 3motion and created several games to evaluate the useful ness of gestures in gaming scenarios. They identified intuitive gestures, clear user feedback and effective semiotics are important.

Discussion:

I did not realize that this was a paper before Wii was in the market. elaborate research on user feedback and recognizing features of feedback would be helpful. In fact, each topic identified as important needs an elaborate study.

Comments: Drew, franck

Wiizards: 3D gesture recognition for game play input

Authors:

Louis Kratz

Matthew Smith

Frank J. Lee

Summary:

Goal : Gesture recognition from 3D accelorometer data from Wiimote. Fluid tolerance and more immersive experience in gaming.

Paper evaluated a HMM model for recognizing Wii accelerometer gestures. The effect number of training data and number of states in HMM on the recognition speed and accuracy has been analyzed. The average correctness of HMM without training data is also measured which is around 50%. The HMM performed well at 15 states and after 10 samples per gesture yielded 80% accuracy and after 20 samples per gesture training 95% accuracy. The results showed that more states in HMM meant more training time and slower recognition speed.

Discussion:

I do not see what is novel about this system. Almost all the classifiers need some form of training so why not think about making the training process interesting or making the training part of the game.

Comments: Franck, Kevin

Device agnostic 3D gesture recognition using hidden Markov models

Authors:

Anthony Whitehead

Kaitlyn Fox

Summary:

Goal: test the accuracy of the 3D gestures by dividing the 3D space into subspaces.

The 3D space is divided into 27, 64 and 127 cubes. The data collected is then used to train HMM and tested on a gesture set. The number of training examples needed, recognition speed are measured. 27 state HMM acheived 800 recognition per second while a single recognition took seconds for 125 state HMM. The number of training data needed to train 25 state HMM was lesser. More states in the HMM meant more training data to train.

Discussion:

I am confused between the states and the segmentation of the space. By states do they mean the actual number of states in the HMM or the number of segmentations. The interesting part though is dividing the 3d space into cubes.

Comments: Franck,

Sensory Hand

Author:

Vernon b Mountcastle

Summary:

* Movement is the facilitating agent for complex tactile experiences.

the ability of the brain to integrate successive patterns of input to create a perceptual whole. The intra-cortical processing times of 80-100 ms for each pattern. Mechanical oscillations can be delivered to the skin. Humans can sense over frequency range from about 5 - 600 Hz. Frequencies in the range of 5 -50 Hz evoke a localized stimulus site. Increasing the frequencies changes the sensory experience gradually to the deep, spreading and poorly localized hum.

* Weber Law - Ratio of the perceivable change in tactile stimuli and the magnitude of stimuli is constant.

* Fechner law - Extending the Weber's law, Fechner quantified the sensation as logarithmic function of the magnitude of the stimulus.

Chapter 11

Localization error is minimum at finger pad and gradually increase towards proximal and palm.

(Schady & Torebjork, 1983) - point localization and Wheat, et al - tactile localization experiment on hand - separation threshold to distinguish a point from sphere of radius to the object of same radius. The human receptiveness increases with time. Sensitivity to stimuli increased in 300ms.

Movement and direction - all the stimuli reaching hand are sensed by scanning movements combine lateral movement and skin stretch.

Flutter vibration - experiments on frequency change detections and change detection thresholds.

An architecture for gesture-based control of mobile robots

Authors:

Iba, S.

Weghe, J. M. V.

Paredis, C. J. J.

Khosla, P. K.

Summary:

Goal: Gesture recognition system to interact with Robots. More appropriately gesture spotting with HMM.

Hardware used : Cyberglove + 6DOF location sensor.

Feature set: 18 sensor data is reduced to 10 dimension vector and each dimension taken with it derivatives increases the dimension of the vector to 20. This 20 dimension vector is coded on to a 32 bit integer. This codeword is then sent into HMM for recognition. HMM is trained with 5000 postures from full hand posture space. Restricting the observation sequence to better quantification. Wait state is included to provide a method to reject invalid gestures.

Gesture set : opening, opened, closing, pointing, waving left and waving right. These gestures carry different semantics in local mode and global mode of robot control.

Discussion:

Interesting extension to HMM for rejecting invalid gestures. Are gestures better than joystick controls while interacting with robots?

Comments: Drew, Franck

Human-centered interaction with documents

Authors:

Andreas Dengel

Stefan Agne

Bertin Klein

Achim Ebert

Matthias Deller

Summary:

Goal: Combining 3D visualization techniques and hand gesture recognition to improve interaction with documents.

Visualization:

Well designed virtual reality like graphical document explorer in 3D. The documents are represented in the form of books in the book case. Search bar is invoked using a gesture and the documents related to the query are displayed with higher semantic zoom and greater detail than book case. The system provides 2 modes of visualization. Plane mode - matrix of documents. Color coding - yellowness for the oldness of the document, animating pulsing behavior to state the importance, thickness to show the size of the document and the thumbnail shows the first page of the document.

Cluster Mode - 3d visualization of the related documents. The relations are visualized by the colored flashing lines connecting the documents.

Gesture recognition - a gesture recognition system which would allow definition of gestures, take in training data and provide recognition results. The system contains 2 threads - data collection and gesture manager. Data collection thread generates events based on the glove movements - glove move, posture changed,... . Gesture manager receives the events from the data collection thread and is responsible for the gesture recognition.

Discussion:

The document moving gesture and the pointing gesture were intuitive. Other gestures for opening the document and returning the document to the grid did not seem to be intuitive. The paper claims to have several ways of interacting with a document but did not mention it how/ what those are. There is no user study. I could not understand some parts of the visualization concepts used in the system like relation visualization, occlusion due to the links and green box to solve occlusion. A systematic user study would have helped in seeing what the original problems are.

I thought i would get some ideas on the gestures users tend to use in 3D environments in the paper.

Comments: Franck

Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes

Authors:

Jacob O. Wobbrock

Andrew D. Wilson

Yang Li

Summary:

Goal :

1. be resilient to variations in sampling due to movement speed or sensing;

2. support optional and configurable rotation, scale, and position invariance;

3. require no advanced mathematical techniques (e.g., matrix inversions, derivatives, integrals);

4. be easily written in few lines of code;

5. be fast enough for interactive purposes (no lag);

6. allow developers and application end-users to “teach” it new gestures with only one example;

7. return an N-best list with sensible [0..1] scores that are independent of the number of input points;

8. provide recognition rates that are competitive with more complex algorithms previously used in HCI to recognize the types of gestures

A class library contains templates . A template/ gesture is a sequence of points. A user entered gesture is first resampled, rotated to indicative angle and scaled to match the template. A MSE score for the best angle is then calculated for each class template. N - best list is then generated based on the MSE score.

User study : Data was collected from 10 users - 4800 gestures. 16 gestures were asked to draw in slow, medium and fast speeds.

The recognizer provides 97% accuracy on 1 training sample / class and 99.5% accuracy for 3+ training sample/ class.

Limitations: $1 cannot distinguish gestures whose identities depend on specific orientations, aspect ratios, or locations. 1D gestures like horizontal and vertical lines cannot be recognized - will be affected by non uniform scaling. The gestures cannot be differentiated based on speed.

Discussion:

A pretty extensive user study which analyzes effect number of training sample/ template, execution speed, accuracy,... I do not understand how qualitative data can be collected for a classifier. But the authors have collected qualitative data on the gestures that users liked. Analysis of the limitations of the recognizer was interesting. The authors also provide solution for removing some limitations.

Comments: Peschel, Franck

The $3 recognizer: simple 3D gesture recognition on mobile devices

Authors:

Sven Kratz

Michael Rohs

Summary:

Goal: build a simple 3D gesture recognition system - simple to implement and require less training data

A gesture recognizer was build for 3D gestures extended Wobbrock's 1$ recognizer for 2D touch gestures. It is easy to implement, requires as less as 5 samples for good recognition result. The system uses raw acceleration data. The change in acceleration is calculated. A sequence of acceleration delta is used to represent a gesture trace. A class library contains a list of gesture traces. The user entered gesture is compared to the class libraries to generate a score table. A heuristic is then applied to recognize the gesture.Resampling, rotation to indicative angle and rescaling are used to normalize the user performed gesture. MSE at best angle if found for each class and a score table is generated. To reduce false positives, a heuristics is used. A threshold value is set. The score table is sorted based on the scores. A heuristic to select class is defined based on threshold. If the threshold dont hold the gesture is recognized as unclassified.

Discussion:
Very similar to the $1 recognizer. easy to implement and easy to train. Providing accuracy data would have been more compelling in defending the classifier. introducing "unclassified" tag is an improvement over the $1 recognizer.

Comment: Franck

Office Activity Recognition using Hand Posture Cues

Authors:

Brandon Paulson

Tracy Hammond

Summary:

Goal: Activity recognition based on hand postures. hand posture to determine object interaction and user dependency in interaction style.

Activity recognition can help establish context of the user interaction. Activity theory - activities have objectives and are accomplished using tools and objects. Therefore, by identifying the object that the user is interacting with information about activity can be extracted.

Previous work: recognizing movement related activities - vision based, wearable accelerometers

object interaction - RFID tags on objects with tag reader in hand.

Grasp types - vision based and glove input data

Implementation: CyberGlove 2 with 22 sensors is used in the system. 1NN classifier is used to classify between 12 different activities. An user study with 8 users was conducted. User independent testing produced a very low accuracy in activity recognition (average - 62%). An user dependent testing with 2 training and 3 testing samples produced 78 % accuracy while 4 training sample produced 94% accuracy. User independent gestures produced lot of variations. User dependent gestures were better recognized but confusion occured in typing on keyboard & phone and circular grip on objects like mug, drawer, telephone and stapler.

Discussion:

An interesting method to identify activity. It was not clear if each activity was captured separately or if it was performed in a sequence. it would be interesting to see the scalability of the system to various other activities. I would also like to see an example application where the context information is used. I am a little confused on how this data can be used.

Comments: Paul
Drew

American sign language recognition in game development for deaf children

Authors:

Helene Brashear

Valerie Henderson

Kwang-Hyun Park

Harley Hamilton

Seungyon Lee

Thad Starner

Summary:

Goal: Design a ASL gesture recognition system.

A wizard of oz user study was conducted with 5 children to collect gesture data. 541 phrase samples and 1959 sign samples were collected (22 word vocabulary). The system uses a camera and wrist mounted accelerometer to capture motion of the hand. HMM is used to train on the gestures and recognize gestures. Feature vector is combination of the vision data and the accelerometer data. x,y,z of accelerometer data for both hands, x,y change in center position of hand in frames, mass, eccentricity, length of major and minor axis, angle of major axis and direction of major axis in x,y offset. User dependent model of classifier produced 93.39% accuracy and user independent model produced 86.28% accuracy in recognizing gestures.

Discussion:

The data has pruned and the gestures which were seemed to correct were selected for testing. I donot think it is a correct practise to test a recognizer. Also, vocabulary of words used for testing was small.

Comment:

An empirical evaluation of touch and tangible interfaces for tabletop displays

Authors:

Aurlien Lucchi

Patrick Jermann

Guillaume Zufferey

Pierre Dillenbourg

Discussion:

Goal: Evaluation of touch and tangible interfaces on table top displays.

This is a comprehensive user study with 40 users and a set of activities to compare touch and tangible interfaces. Tangible interfaces are complex to measure based on Fitt's law. A top projected table top system is built for supporting both the interfaces. The table can track multiple finger interaction. The system uses a camera mounted on the top of the table to track the tagged tangible objects. The toolbar in the touch interface was replaced by the tangible objects. Users were asked to model a building using shelves and walls. The virtual wall in touch interface was rescalable while the tangible one was not. Activities like translation and rotation has natural gestures in tangible. Selection did not have any meaning in the tangible interface. Addition, removal and adjustment activities were performed physically in tangible interface while certain tool bar icons were used in touch interface. An introductory video was presented to the users to explain the different actions available. The actiions of the users were video taped and logged. The user were asked to model a warehouse. To compare the interfaces 3 variables were used - Completion time of overall experiment, completion time of each action and Accuracy. Overall completion time favored tangible interfaces. Comparing individual activities showed that scaling, translation and deletion operations favor touch interface while addition operation favors tangible interface. Touch was slightly less accurate. User preferences show that tangible interfaces were easier to learn than touch interfaces. Touch interface made user more stressed and irritated.

Summary:

Tangible interface seem to be more intuitive way to interact. But the rigidity involved or the difficulty involved in editing may affect the user experience. I am impressed by the user study. It is expensive, systematic and has both quantative and qualitative data.

It would have been interesting if the authors did an follow up study to see if the users retained what they learned.

Comments:

Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound

Authors:

Takayuki Iwamoto

Mari Tatezono

Hiroyuki Shinoda

Summary:

Goal: to provide high fidelity tactile feedback for interaction with ultrasound, acoustic radiation pressure.

An array of 91 transducers were used to form a tactile feedback surface. The sensors were activated to find the radius of the focus point , force/pressure exerted at a distance by the tactile device on the hand. The evaluation of the system was done based on comparison of the theoretical values calculated with the practical measurements. The user study was not systematic and does not have quantitative data. The user felt the tactile sensation when the feedback was in the form of vibration rather than constant pressure.

Discussion:

This mode of feedback enriches user experience. This device could change the mode of interaction for blind people. The interaction medium for blind people is predominantly voice based. The visual display can be replaced by this device to provide a better interaction for the blind.

COMPUTER VISION-BASED GESTURE RECOGNITION FOR AN AUGMENTED REALITY INTERFACE

Authors:

Moritz Störring

Thomas B. Moeslund

Yong Liu

Erik Granum

Summary:

A paper proposes a computer vision based algorithm for recognizing gestures in an augmented reality system. The hand gestures (counting 1 -5 ) is captured with Head Mounted Camera (HMC). The problem comes down to identifying each finger and finding the number of fingers in an image. The image of hand is transformed to polar transformation.

User study; Qualitative data from the user study showed the system was robust and users adapted to the system easily

Discussion:

The target of the paper was to provide a gesture recognition system but the accuracy of the system has not been reported. Not providing quantitative data raises questions about the algorithm. Why is the robustness of the AR interface analyzed in the user study?

Comments: Drew , Franck

FreeDrawer: a free-form sketching system on the responsive workbench

Author:

Gerold Wesche

Hans-Peter Seidel

Summary:

Goal - To provide a simple 3D computer aided tool for the direct transfer of design intent into a corresponding computer respresentation.

Guidelines for immersive control - hide the mathematical complexity of object representations, direct and real time interaction, full scale modeling, large working volume and intuitive, easy-to-learn.

Related work - "3 - draw" uses HMD, tablet and 2 hand interaction. The system keeps track of the 2 hands with one hand controlling the tablet and the other drawing in the tablet. and some 2D - 3D drawing tools.

Features - Drawing, creating curve network, new curve in the network, filling the surface , Curve smoothening, Curve dragging, surface sculpting. UI - the design tools spread as virtual pointers at the end of pen. This allows smooth selection than grabbing.

Forceful feedback might help in improving the system in giving feedback about constraints.

Discussion:

An user study to understand the user experience would have helped. The tool selection widget is interesting and different. By including the force feedback, there might be some advantages but it must not be invasive and hinder in the activities of the user. The kind of implementation of the force feedback would make a big difference in user experience.

Comments: Franck

Lab day 1

Eye Tracker:
It was a tiring experience. It took fair amount of time to realize that only one of my eye was being tracked. I went through the calibration procedure twice. I did not understand how it worked though. I tried doing some simple task with the device. One task was to use it as a mouse. I tried navigating around the screen and clicking on an icon. I could not navigate as i intended to. I could not select any icon on the screen. It was difficult to avoid jitter and concentrate on the same area.
I found difficulty in holding the pointer to one area. It was interesting to see the recognition of the eye from the software and i missed noticing lot of parameters it calculated.
i think the device can be used to the context or roughly the area the user is gazing at.

Head mounted display:
The device reminded me of the night vision goggles that i often see on movies. It was cool . It had displays in front of the eyes and a camera in front to capture the images in front. There was control box which controlled the display input. It can be used to relay the video captured by camera or can be enhanced I did not find much of difficulty in using it. Josh conducted an user study with the device. The tasks we were asked to do were interesting. Arranging books sitting and standing, writing, reading and walking around the room. Well the task helped me adjust to the difference in the depth perceived through goggles. May be with greater usage the user might get adopted to this difference.

Motion Editing with Data Glove

Authors:
Wai-Chun Lam
Feng Zou
Taku Komura

Summary:
Goal - Mapping hand motion to the whole body to edit human motion. Controlling human figures in real time environment such as games and virtual reality system.

Previous work in motion puppetry were focussed on facial expressions. Using mouse and keyboard controls for motion editing is difficult since the lesser degree of freedom that the devices offer.

A mapping between the hand and the body. The effectiveness of the tool is based on the mapping function. There are 2 stages in the tool - Capture stage and reproduction stage.

To match the trajectories and reduce the noise in the data, a fourier series expansion is applied to data. An ordinary walking motion was captured using this device and was used to make hopping motion with large strides and a running motion along zigzag path.

Limitation: The system works effectively if the original motion captured and the newly generated motion are related. Unrelated motion like jumping with 2 legs while original is walking does not work. Foot on ground cannot be captured exactly.

Discussion:

I would say this tool is better than the mouse + keyboard but this would still have constraints. And the animation shown here is fairly simple (no human features). There is no user study or proof that this tool works in a complex animation. Also mapping finger motion to walking is natural but how does mapping other parts of the body to the hand help.

Comments: Sashi, Drew

EyePoint: Practical Pointing and Selection Using Gaze and Keyboard

Authors:
Manu Kumar
Andreas Paepcke
Terry Winograd

Summary:
Gaze technology problems -
accuracy, eye jitter detection, distinguishing between intentional dwelling and movements to involuntary search/ scan movements ( midas touch problem).

prior work - gaze based hot spots, gaze based context awareness, dwell time activation,...

current solution for pointing and selection - look, press, look, release
- look at the point of interest for pointing to rough area
- press a hot key (click, right click, double click )
- magnified image of the rough area appears
- with the hot key pressed down, look into the magnified area to select the particular area of interest
- release the hot key to make the selection
- to escape the process, press esc or stare away from the magnified area and release the hot key

User Study:
From the pilot study, the gaze data was found to be noisy. It was difficult to distinguish whether the user was looking at the target as a whole or focusing at a point. Focus points were included in the magnified view to improve the accuracy of the system. A gaze marker was included as a visual feedback for the user. But it proved to be a distraction in the pilot study.

20 users who were not used to eye tracker technology were involved. Three variations of the study are Eyepoint with and without focus grids and Eyepoint with gaze marker. Both qualitative and quantitative data was collected. Tasks in the user study were a real world browsing, a synthetic pointing and a mixed typing and pointing task. The users were first calibrated on the eye tracker and had 5-10 minutes training.

Web study - A single link was enabled on a page and all other links were disabled. The users were instructed to click the enabled link which was highlighted in orange. The time between the introduction of the page and the link selection was measured. The error rate for eyepoint was larger than that for the mouse. The users felt mouse was accurate but they (3/4 of users) were inclined to using eyepoint. Users felt focus points were useful.

Balloon study - A balloon with different sizes (22,30,40) are placed on the screen and amount of time taken to click is determined. The error rate and time were significantly affected by the size of the balloon. The eyepoint was slower than mouse by 100ms. 3/4 of the users felt inclined to use eyepoint. Subjects also felt fatigued by using mouse.

Mixed study - Mixed typing and pointing task included selecting a constant size balloon shown on the screen and then typing the shown word in the text box. This allowed the researchers to measure the time taken to move the hand from mouse to keyboard and then back to mouse for next selection. Eye point had better performance but lacked accuracy .

The error rate was more in case of eye point than mouse due to various reasons.
- dependence on individual, calibration and posture of the subject
- some subject errors like pressing the hot key before looking at balloon.
- seeing the target using peripheral vision before getting fixated on the fovial vision.
- astigmatism , squint, glasses that reduces the accuracy eye tracker.

Discussion:
I liked the user study. They have given significance to both qualititative and quantitative data. In case of the qualitative data collected, it was reported that the user liked to use eye point. The fatigue i think is not emphasized. I feel though selection by gaze tracking is natural, concentrating on an area is a very tiring work. Interesting to see users have gone through one hour of the user study. I could not use the eye tracker for more than 10 minutes in the lab.

Comments: Drew , Murat

HoloSketch: a virtual reality sketching/animation tool

Authors:

Michael F. Deering

Link to the paper:

Summary:

Goal: Building an application that applies the Virtual reality technologies in simple 3D drawing and animation tool

Previous technologies: 2D mouse has 2DOF in a interaction space which requires 6DOF movement. HMD, during the time when the system was made, had lower visual resolution. Novelty in the Holosketch is the introduction of 6DOF of hand input device. Holosketch uses a 20 inch CRT display, head tracking device (field sequential shutter glasses) and wand 6DOF virtual world manipulation tool.

Calibration: high accuracy position and orientation tracker for tracking head movements. Corrections are made for the distortions due to the curvature of the CRT as well as the index of refraction. The individuals interocular distance is used as calibration parameter, since based on the parameter the system corrects for changes due to rotation of the viewers eyes. Holosketch interaction techniques rely upon this high accuracy.

Menu design is based on the 3D consequences of Fitt's law, screen real estate cost and the position of the menu ( not interfering with the displayed object). The fade up pie menu is used to display the set of menus. The workspace fades out in the background. Right button of the wand is clicked and pressed to pop up the pie menu and released at the desired menu - to select the menu.

Drawing Features - Features include selecting and drawing primitive 3D shapes, drawing lines by pressing the left button of the wand and sweeping on the 3D space (leaves a toothpaste line with different thickness), 3D text typed using keyboard and imported geometric shapes. Drawing attributes if the shape can be changed. The most important of which is the Color. The application provides a RGB color cube to select the color from. The tip of the wand changes to a sphere showing the current color selection. As the wand moves through the cube, the color of the sphere changes. If the wand moves out of the color cube, the cube disappears.

Selection - Last object is the default selected object. Continuous color change/ blinking is used to show the selected object. Middle button click is used to select an object. Shift key + Middle button click is used to select multiple objects.

Editing - After an object is selected, the object can be moved, scaled, grouped and various other attributes (like point size of text) can be set. Squeeze buttons on the side are used to control the movement of the object. To prevent accidental trigger, key combinations are used. Control key for positional change, Ctrl + Shift for orientation change and Shift key for Position + Orientation change. Property sheets (Attributes menu item) of the object can be used to change the parameters.

Animation operations - Operations like Rotation, Solid movie looping , Color oscillation, Scaler, shifter and flight path can be done using the application.

Environmental variables like light source can also be controlled in the application. The viewpoint for the user can be changed by moving the head. To examine particular object, the user can select the object and rotate it with the wand. In the control mode (select from fade up menu), the entire virtual universe can be considered as an object. Depressing the left wand button changes the scale of the universe. The user can mark the particular user settings with a command and get back to the settings with another command.

User Study:

Single user, a traditional artist, used the system for a month. Holding one's hand in the air for long periods of time was not hard but making fine adjustments became difficult. A new mode was added - ability to draw 2 handed. Use 2d mouse as the lever and vary dynamically the radius of the toothpaste and simultaneously drawing with 3d wand.

Limitations - Geometry modelling and complex physics simulation to give more complexity to the animation and the virtual world. Optimal placement of menu buttons, choice of button colors has been secondary.

Discussion:

The user study analyzed the experience of a single user over the period of one month. It gives extensive and detailed data about the usability of the interface and the application. But the experience of a single user can not be generalized to other users. The user study was conducted on a computer/ traditional artist rather than a novice. The author claims the system will be easy to use for a novice.

The choice of a big 3D pie menu was confusing. The idea behind careful choice of menu interface was to not interfere with displaying the object being edited and occupying less real estate. The pie is flooded with all the menus and takes up the whole screen.

The interaction technique introduced by the system seems to be interesting.

Java 3D and Virtual Portal are projects extended from the concepts of Holosketch. Virtual portal as shown in the picture below contains 3 perpendicular wall with rear projection and head tracking display. this was used in automobile simulation. The 3 wall projection to provide the natural 3D interface is better than single flat display. CAVE, a VR system, extended this and had a floor projector to provide more natural 3D interface.

Comment: Paul, Franck, Sashi