I liked it very much: the conference had good quality papers but was still reasonably small (around 150 attendants), and of course the weather and the Italian food were great We arrived on Tuesday which gave us some time to explore the city and take the ferry to Capri (a great suggestion by Robbie).
I am not going to discuss the conference program into detail this time, but will just highlight a couple of interesting papers. Possibly one of the coolest papers was “Exploring Video Streams using Slit-Tear Visualizations” by Anthony Tang (video). Another presentation I enjoyed was “TapTap and MagStick: Improving One-Handed Target Acquisition on Small Touch-screens” by Anne Roudaut (video). It seems there is lots of related work in this area (e.g. Shift, ThumbSpace, etc.). Peter Brandl presented two interesting papers: Bridging the Gap between Real Printouts and Digital Whiteboards and Combining and Measuring the Benefits of Bimanual Pen and Direct-Touch Interaction on Horizontal Interfaces. He was brave enough to do an impressive live demo for the first paper Oh, and he also covered the conference in a blog post.
The first paper in our session, titled “A Mixed-Fidelity Prototyping Tool for Mobile Devices” by Marco de Sá, introduced a tool to easily design prototypes and evaluate them in real-life situations. The system was well thought out and serves a real need. I can imagine that we could use this kind of tool in a user-centered UI design course. The second paper in our session was “Model-based Layout Generation” by Sebastian Feuerstack. I already met Sebastian at CADUI 2006. They presented a generic layout model based on constraints. It reminded me a bit of the layout model Yves worked on for our EIS 2007 paper. They used the Cassowary constraint solver, which I also used for my MSc thesis on constraint-based layouts for UIML. Sebastian told me he got the idea from my demo at CADUI 2006. I forgot to add a certain constraint (the layout of the UI was thus underconstrained), which by coincidence had no effect on the user interface everytime I tested it. Of course, when I showed the demo it did have an effect This clearly illustrated that constraint solvers are sometimes unpredictable (see Past, present, and future of user interface software tools by Myers et al.). Sebastian’s solution to this problem was to hide the constraints from the designer and generate them automatically from a graphical layout model.
Juan Manuel Gonzalez Calleros — who I met at CADUI 2006, TAMODIA 2006 and a few other occasions — presented a poster and a paper at the workshop on haptics. He took a few pictures while Jan was presenting (thanks again Juan!). Here are Juan and Jan discussing UsiXML vs UIML
Overall, the comments on our work were positive, although of course one of the biggest problems is still the lack of support for multi-screen interfaces. As Jan is actively hacking on Gummy these days, I don’t think it will take very long for this to be included in the tool
I spent some time last weekend looking into Smalltalk again. The first time I did this was somewhere around 2004, when I played around with Ruby and discovered that it was strongly influenced by Smalltalk. Back then I watched an old video by Dan Ingalls on object-oriented programming which finally made me fully understand the essence of OOP: it’s all about messaging
In my personal opinion, this video (or at least the message that Dan tries to communicate) should be better integrated in OOP courses at universities. Another invaluable resource for grasping these ideas is Design Principles Behind Smalltalk, again by Dan Ingalls. Of course, it’s difficult to understand what OOP is about if you have to learn it through a weak implementation. We learned the basics of OOP in C++ for example, which would be blasphemy to Alan Kay He once saidActually I made up the term object-oriented, and I can tell you I did not have C++ in mind. Here’s his definition of OOP:
OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP. There are possibly other systems in which this is possible, but I’m not aware of them.
Before I looked into Smalltalk, to my understanding objects just contained a bunch of methods or functions that had access to the object’s context. I did not really grasp the idea that objects just respond to messages (or method calls in my definition). The real difference about this is that in Smalltalk messages are dynamically dispatched at runtime. A method is the function or subroutine that is invoked in response to the sending of a message, which will be matched to the message name (or selector) at runtime. In contrast, method calls in C++, Java and C# are statically bound at compile-time. There is thus a distinction between the semantics (or message) and implementation strategy (or method) in Smalltalk. Decoupling these allows for more flexibility, such as objects that cache all incoming messages until their database connection is fully set up, after which they replay these messages, or objects that forward messages to other objects (which might have even been passed in at runtime). This is of one of the aspects of extreme late binding in Alan Kay’s definition of OOP.
It’s exactly this run-time lookup of methods that enables effortless polymorphism. As explained in the video, at some point the intermediate factorial result will become an instance of LargeInteger, while in previous iterations it was an instance of SmallInteger. The multiplication message (*) is sent to this object, after which the correct method in the class LargeInteger is looked up for handling the message, allowing the existing code to continue to work. Java, C# and C++ have all inherited this feature (although C++ requires explicitly declaring methods as virtual for this to work, due to efficiency reasons). Smalltalk can even realize polymorphism without inheritance (also known as duck typing), although this is not shown in this video. Smalltalk has implicit interfaces: an object’s interface is the messages it responds to. If two objects both respond to a certain message, they are interchangeable (even at runtime). Traditional languages such as Java or C++ only support inheritance-based polymorphism (although something similar to duck typing can be achieved with C++ templates). Here’s the explanation by Dan Ingalls:
Polymorphism: A program should specify only the behavior of objects, not their representation.
A conventional statement of this principle is that a program should never declare that a given object is a SmallInteger or a LargeInteger, but only that it responds to integer protocol. Such generic description is crucial to models of the real world. Consider an automobile traffic simulation. Many procedures in such a system will refer to the various vehicles involved. Suppose one wished to add, say, a street sweeper. Substantial amounts of computation (in the form of recompiling) and possible errors would be involved in making this simple extension if the code depended on the objects it manipulates. The message interface establishes an ideal framework for such an extension. Provided that street sweepers support the same protocol as all other vehicles, no changes are needed to include them in the simulation:
More details on the differences between Smalltalk and current OOP languages are explained in Smalltalk: Getting The Message. I believe that understanding the original philosophy behind OOP helps you be a better object-oriented programmer in any language. Ramon Leon discusses the common mistake of magic objects which is an interesting read.
But let’s get to the point of why I started looking into Smalltalk again. At the moment, I mostly program in C# (and sometimes in Java), but I often feel frustrated with both languages. After being exposed to Ruby and Python, I feel like static typing requires me to write too much code and helps the compiler more than it helps me. Furthermore, Java seems to be overly engineered with all the factories, manager, readers and writers, while C# is often inconsistent or lacking in its implementation (e.g. anonymous methods are not really closures). Both languages are becoming increasingly complex with the addition of more and more features. Generics for example is just not necessary in a dynamically typed language. The problem with scripting languages such as Ruby and Python however, is that they are often interpreted and slow. I experimented a bit with JRuby (a Ruby implementation in Java with full access to Java’s class library) but that didn’t satisfy my needs either. After trying to code a simple Hello World Swing application in JRuby, I was stunned that it still required me to wrap code inside an ActionListener like Java does, while I really just wanted to pass in a Ruby block.
Other people have also been struggling with languages such as Java or C# (e.g. Jamie Zawinski, Mark Miller and Steve Yegge) or are looking for alternatives (e.g. Martin Fowler and Tim Bray). I think the popularity of Ruby might motivate more people to have a look at Smalltalk. Furthermore, if you know Ruby, it’s easier to get acquainted with Smalltalk. Besides lots of similarities in the class library (the Kernel class, the times message on numbers, etc.), Ruby already introduces the notion that everything is an object, objects in Ruby communicate through messages and Ruby has blocks. However, Ruby is not really equivalent to Smalltalk yet. Ruby introduced extra syntax to be more familiar to people that were used to C-style programming languages, thereby losing part of Smalltalk’s flexibility. In fact, the beauty of Smalltalk is that its entire syntax easily fits on a postcard. If you look closely at this example, even a conditional test in Smalltalk is implemented using messaging on objects. You just send the message ifFalse to an instance of the class Boolean, and pass in a code block you want to have executed when the value is false. It’s turtles all the way down.
Another problem I came across when developing in Java or C# (or in any other OOP language I used) was the difficulty of changing class hierarchies. Very often, due to time constraints, a design is just left in its original state, and the new requirements are supported by performing a quick hack. I suspect this problem is especially prevalent in so-called “research code” It gets even worse when programming in teams. Although this problem is generally known in software engineering and several strategies have been proposed to deal with it, I wondered why the promise of OOP failed here. Wasn’t OOP supposed to improve the situation and make spaghetti code obsolete?
Jeffrey Massung asked himself a similar question: What if the philosophy (OOP) wasn’t the problem, but the implementation (language) was?, and decided to write a 2D DirectX game in Smalltalk. It seems Smalltalk did indeed allow for easier design changes. Self (a language derived from Smalltalk) tries to alleviate the aforementioned problem by specializing through cloning of existing objects instead of through class hierarchies. It’s funny to note that the problem wasn’t that bad in Smalltalk, since you could still easily change the hierarchy, unlike in languages such as Java or C++.
The real power of Smalltalk is not its syntax, but the entire environment. I believe this is also key to understanding OOP. The current languages and tools (e.g. IDEs) we use for doing object-oriented programming are just weak implementations of the original Smalltalk environment. When working in Smalltalk, you are working in a world of running objects, there are no files or applications, everything is an object. For example, version control systems in Smalltalk are actually aware of the semantics of your code, they are not just text-based. When merging code they can show you what methods have been changed, added or removed, what classes were changed, allow you to decide which changes you want to keep, etc.. Although I think Bazaar is a great, it doesn’t come close to this way of working. Smalltalk allows live debugging and code changes, which is tremendeously useful. Ever wished that you could fix a problem while you’re debugging and immediately check if your solution works without having to recompile your application and start the entire process again? In Smalltalk (and Lisp) that’s possible. If you want to find out more about why Smalltalk is way ahead of current mainstream OOP languages, have a look at Ramon Leon’s Why Smalltalk.
Update:: Scott Lewis commented that I should have emphasized that Smalltalk is mostly written in Smalltalk: So when you subclass any object, you can go back up the chain of inherited objects and see how everything works. Likewise when you hit an error/bug, the debugger lets you delve about as deeply as you could possibly want into what is going wrong, and why it is an error. This is indeed a powerful aspect of Smalltalk, and an example of how it was influenced by Lisp.
Besides reading about Smalltalk, I have also been experimenting a bit with Squeak. Squeak is an open source implementation of the Smalltalk programming language and environment, created by its original designers. Squeak runs bit-identical on many platforms (including Windows CE/PocketPC). I will leave my Squeak experiments for another blog post though
To conclude, it seems that we are very good at ignoring the past. We just take our current systems for granted, and use them as a reference frame for future innovations. Marshall McLuhan once phrased it like this: We drive into the future using only our rearview mirror. I believe this is true in HCI research as well, as people like Dan Olsen have pointed out. He argued that our existing system models are barriers to the inclusion of many of the interactive techniques that have been developed. He gave the example of the recent surge in vision-based systems and multi-touch input devices, which get forced in a standard mouse point model because that is all that our systems support:
Multiple input points and multiple users are all discarded when compressing everything into the mouse/keyboard input model. Lots of good research into input techniques will never be deployed until better systems models are created to unify these techniques for application developers.
Research on toolkits is a lot less popular these days. We try to map everything into existing models, and always feel like we have to support legacy applications, which hampers significant progress. Bill Buxton has also studied innovation in HCI, and questioned the progress we made in the last 20 years.
I think the reason why so many great work was done by the early researchers in our field (e.g. Ivan Sutherland, Douglas Engelbart and Alan Kay) is — besides that they were very creative and intelligent people — that there was not that much previous work, they just had to start from scratch. Alan Kay once asked Ivan Sutherland how it was possible that he had invented computer graphics, done the first object oriented software system and the first real time constraint solver all by himself in one year, after which Sutherland responded I didn’t know it was hard.
Some other resources I recommend are “How to give a good research talk” by Simon Peyton Jones, and the Presentation Zen blog. These should already provide you with the basics for giving a good (research) talk. Here is what I personally found useful in the Microsoft Research session:
Use animations sparingly: animations are only useful to illustrate a process in your system, or make something more clear to the audience. Don’t overdo it. In my opinion, I offended against this rule with my EIS 2007 presentation. Some animations were useful, but a lot of them were unnecessary. When I gave part of this presentation to a few other researchers some time after the conference, one of them commented that I should contact George Lucas about the effects and transitions
Use pictures for related work: Patrick argued that a lot of people remember pictures from papers they read, so using a visual representation of the related work is more useful than a list of references.
Try to demo the current status of your future work: Rick showed the future work demo of their photo tourism paper he gave during his talk at SIGGRAPH. This way you give the audience evidence that you’re actively improving upon your work.
Tactics to handle rude questions: Mary gave a few tips for dealing with rude questions such as repeating the question that was posed. This is always useful to indicate how you have understood it. Furthermore, it gives people in the audience a second chance if they did not understand the person who posed the question.
All in all an interesting seminar, might be useful to organize something similar at our institute in the future. Thanks to Lode for sharing the link on his blog.
At Ubicomp 2007, there was a book stand by Springer just outside the conference room. On the last day, the volunteer behind the stand told me that I could choose one of the books that were still lying there. I didn’t see anything interesting at first. Since a few people at our institute are working on multimodal systems, I picked the book SmartKom: Foundations of Multimodal Dialogue Systems.
During the holidays, I read the first part of the book and noticed the book was relevant for me after all. SmartKom was a large four-year project about multimodal dialogue systems. They developed a system that provides symmetric multimodality in a mixed-initiative dialogue system with an embodied conversational agent. There is also a follow-up project that should ends in 2007: SmartWeb. SmartWeb goes beyond SmartKom in supporting open-domain question answering using the entire (Semantic) Web as its knowledge base.
Symmetric multimodality means that every input mode (e.g. speech, gesture, facial expression) is also available for output, and vice versa. Multimodal interaction is one way to make interaction between humans and computers more intuitive. Human dialogue is not only based on speech but also on nonverbal communication such as gesture, gaze, facial expression, and body posture. One of the major characteristics of human-human interaction is the coordinated use of different modalities (e.g. allowing all modalities to refer to or depend upon each other). Symmetric multimodality combined with a mixed-initiative conversational agent results in more intuitive interaction. The SmartKom systems reduces recognition errors by modality fusion. By considering multiple input modalities together (e.g. speech, facial expression and gesture), the system can more correctly estimate the user’s intention.
SmartKom has been used in several application scenarios: in public telephone booths, home entertainment systems, mobile systems and in a car environment. The last part of the book discusses techniques to evaluate multimodal dialogue systems, which should be an interesting read.
We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI offers both explanatory and generative power. It provides insights for design, uncovers gaps or opportunities for future research, and leads to the development of improved evaluation techniques.
The paper discusses amongst others the results of a CHI 2006 workshop on the next generation of HCI. The authors provide a framework for classifying, comparing and evaluating new interaction styles. The framework concentrates on four themes used in these emerging interaction styles:
Naïve Physics: people have common sense knowledge about the physical world.
Body Awareness & Skills: people have an awareness of their own physical bodies and possess skills for controlling and coordinating their bodies.
Environment Awareness & Skills: people have a sense of their surroundings and possess skills for negotiating, manipulating, and navigating within their environment.
Social Awareness & Skills: people are generally aware of others in their environment and have skills for interacting with them.
These four themes are clarified by the accompanying picture:
This framework can be useful to evaluate the “intuitiveness” of new interaction methods by measuring the extent to which they use knowledge and skills from the real world.
Apparantely, even when Johnny is procrastinating, he is doing interesting work
JL: I guess I just spend a lot of time on my hobbies that I really enjoy doing and it turns out that my hobbies end up being productive. Even the Wii remote work started as a way to procrastinate working on my thesis.
I found out a bit too late that Nicolas Nova would be giving a talk at iMAL in Brussels yesterday. Luckily he always puts his slides online
The talk also explained his (seemingly random) blog title: “Pasta&Vinegar”. He states that the hybridization of digital and physical environments is explored both by academic researchers (pasta) and artists and designers (vinegar). In the talk at iMAL he talked about why vinegar is important for pasta
His slides contain lots of interesting and creative ideas, such as blogjects, augmenting animals (e.g. a dog with sensors that controls a WoW character) and a tooth implant that vibrates when you have an incoming call.
If you want to invent something that is to be used 10 years from now, who can you observe? Nicolas states that looking at new media, art and design can give us clues. He also explains that art and design can better convey desire of people for the future, and shows a typical diagram from an IT company that is not appealing to people and too much focused on the technology in the background. He finally refers to the use of technology in art. SIGGRAPH’s Emerging Technologies and Art Gallery are good examples of this and of combining pasta and vinegar.
Johnny Lee’s back again with his Wiimote interactive whiteboard. Commercial versions of these things are expensive and heavy. His technique doesn’t even need a projector, just a computer, a Wiimote and a simple IR emitting pen. The pen is just a stylus with an infrared LED in the tip.
Johnny Lee is back again indeed I posted about his method to track your fingers using a Wiimote earlier. This time he uses a the Wiimote’s infrared camera to track light pens (pens that emit an infrared light at the tip) on a surface to create an interactive whiteboard. It’s really nice that he can use any surface. You could use a projector in combination with an ordinary projection screen, a wall or a desk. If you don’t have a projector, you could turn any LCD display into a tablet surface.
Since the Wiimote can track up to four different points, these surfaces are also multi-touch. This means you can have multi-touch interaction on any projected image. It would be interesting to combine this with a steerable projector system.
The source code is available. I will definitely keep an eye on his Wii projects page.
On Tuesday I went to the talk of Adam Greenfield in Leuven, organized by the Microsoft Research Chair on Intelligent Environments. The main topic of his talk was the social and ethical implications of ubiquitous computing. Adam started his talk by saying that there are a lot of ubicomps. He uses the term everyware to cover Weiserian ubicomp, pervasive computing, tangible media and ambient intelligence. Everyware is free from the baggage of Xerox PARC, free from politics and easy to understand. He defines it as distributed, networked information processing resources that are embedded in the environment. Adam sees everyware as inherently multi-disciplinary. He works at the NYU Interactive Telecommunications Program which hosts both people with an artistic and a technical background.
Similar to Bell and Dourish Greenfield claims that the first stage of ubicomp is already here. Technologies such as RFID and NFC together with devices such as the iPod and iPhone are already ubiquitous computing devices. However, the way we interact with them has not yet changed significantly. He says we already have robust information processing in our environment today (and that this can be seen in standards such as IPv6 and in devices such as Proliphix Network Thermostats). An interesting point he made was that when he started giving these talks 18 months ago, most of the examples he used were research prototypes, but now most examples are commercial products. Furthermore, adoption of these new technologies and products is unproblematic. In Hong Kong, 95% of the people between 16 and 65 used the new Octopus RFID metro pass system.
Concerning the consequences of everyware, he referred to the Panopticon, an 18th century prison that was optimized for surveillance. The guards could see the prisoners’ cells at any time, while the prisoners could never see the guards. The prisoners’ default state was to be monitored, so they acted accordingly. So this might happen as well with everyware that is watching us and sending information out. We may get used to it, and just watch our steps more closely. Here is an example of such as prison (image courtesy of Wikipedia):
Greenfield talked about the design of everyware, and referred to “design dissolving in behavior” by Naoto Fukasawa. It comes down to closely looking at people’s everyday behavior and trying to improve it with a solution that is as simple as possible. Design has to achieve an object without thought. People shouldn’t have to think about an object when using it. This also came up during the DIPSO workshop, and is a feature that was lacking from the i_AM table (How do you use it and what can you do with it?). As an example, Greenfield referred to the Octopus transit system again which allows you to quickly pass by the metro gate with an RFID-tagged metro pass in your pocket or bag since the reader’s range is large enough to read it as you walk by.
He continued with future issues and mentioned inadvertent, unknowing and unwilling use of everyware. The first issue can occur when you mistakenly publish your location to the whole world instead of to your closest friends. The cost of inadvertent use rises with everyware. Unknowing use might occur when a user walks over a sensor on the ground that recognizes when someone is walking on it, but the user does not know that he can later be identified since we all have a unique walking pattern. Finally, unwilling use can occur when people don’t want to use everyware technology but are forced to do so, e.g. you may need to use an RFID-enabled metro pass to get on the metro in Hong Kong. He also briefly discussed security (when all objects are connected one object might trigger behavior in another object, e.g. a failure could make your automatic garage door go up and down), and the digital burden of having to deal with your digital traces (Should I postulate this query? Can it be traced?).
Finally, Greenfield said it’s time to take everyware seriously and proposed 5 principles to design everyware:
Be harmless
Be self-disclosing
Be conservative of face
Be conservative of time
Be deniable
Harmlessness refers to safety, everyware should always try to ensure users’ safety (physical, psychic, financial). It is graceful degradation taken further. The second principle refers to smart objects announcing their functionality. Greenfield proposes to use icons to indicate data collection, support for gestural interfaces or self-describing objects:
The third principle means that everyware should not necessarily embarass, humiliate or shame its users. Society has a necessary membrane of protective hypocrisy according to Greenfield. Examples include the strict categorisations used on social networking websites such as Flickr (e.g. friends or family), or what happened to Robert Scoble on Facebook a while ago.
Principle 4 refers to everyware not introducing undue complication into ordinary operations (this is what Weiser actually referred as invisible computing in my opinion). Everyware should not take over, and should assume that an adult, competent user knows what he or she is doing. Finally, users should always be given the ability to opt out (principle 5), with no penalty other than the inability to make use of the functionality that the ubicomp system offers. I believe that the last two principles cannot be realized by systems that make all the decisions for the user. An approach such as mixed-initiative interaction might be more appriopriate.
Adam also talked about the cultural differences in safety and social status in Europe, the US and Asia. For one, in Asia is it common to have talking doors and elevators (which Lode also noticed on his trip to Japan), while this would drive most European people nuts Everyware is by consequence not universal, and should take into account the cultural conventions of the country or region where it is deployed.
All in all, I found the talk very interesting. Although most principles seem obvious, I have seen a fair share of ubicomp systems that violate them. I especially liked his proposal for everyware icons. People are coming up with unique names for Wii gestures which might also help in announcing how to interact with a system.
The paper Making Bits and Atoms Talk Today I wrote together with Ruben Thys about his Master’s thesis featured a scenario on interaction with printed media called SemaNews (there is also some information available at Ruben’s page under the section “Results > Daily Newspaper”).
Here is a very short video of the SemaNews scenario that I used in my presentation at the DIPSO workshop:
Apparantely the more general idea of interactivity with printed media — in this case ads in printed media — seems to be breaking through. Saatchi and Saatchi together with HITLabNZ created an interactive, augmented-reality advertisement for the Wellington Zoo in a New Zealand newspaper. Users can use their mobile to see 3D images of some of the animals in the zoo. They first download an application from a URL by decoding a barcode on the ad. As they move their phone over a printed pattern on the ad, they can rotate the 3D objects accordingly. The 3D objects are placed over the image that the camera records. The ad was definitely effective since the zoo saw a 32 % increase in visitors
There is no real interactivity for the user here though. These barcodes just encode an URL and thereby offer the reader an easy way to go to the accompanying website while letting Google know that they came from a particular print ad. Nevertheless, it is interesting to see that barcodes and other auto-ID technologies such as RFID are getting more popular everyday.
We arrived on Tuesday which gave us some time to explore the city and take the ferry to Capri (a great suggestion by Robbie). 
