Design playground, Thoughts

Experiments with reactive & embedded media tokens


Early this year the ever inspiring team at Berg published their interesting exploration called ‘Lamps’, which was a research project they undertook for Google. Lamps speculates on the future of ambient media where Google could inhabit the world around you with the help of computer vision and projection technologies.
I was fascinated by their theory of ‘smart light’ and especially their use of Fiducial markers as switches and knobs.

It brought to my mind an interesting challenge being faced at work (SI Labs) about bringing large scale prints to life by means of adding interactions to static media. So I set about using the techniques used by Berg to create my own set of interactive tokens with media and interactions embedded in them, brought to life by projection mapping over fiducial markers.
Here is a short clip, displaying a few interactions & the character of the setup with a web camera placed over a projector.

I use a Processing sketch utilising the ‘Reactivation’ library to identify and embed media/interactions to fiducial markers. Using the projector an individual marker is brought to life on the opposite wall.

We could develop a language for the use of these markers, some of which are seen in the video. With movement, angular displacement, proximity and possibly touch, we could use the markers as switches, knobs, dials for manipulating media. As shown below:
Artboard 1 1
One could also imagine that these markers could be printed with ink visible only in the IR spectrum, easily picked up by cameras and invisible to the naked eye.(Quick update: Disney Research have used this system in their experiment linked below)
We could also think of controls like these distributed around a space, not just to control media (video, audio) but for other purposes. Like an universal physical interface, for example controlling all the electronic objects in your kitchen. A centralised hub with cameras for eyes, with/without need for projection.

There were of course difficulties with the setup.
1. Mapping the field of view of a camera with projected area is quite cumbersome and needed quite a bit of thought.
Artboard 1
My final solution was an empirical one where I had to continually displace the positions between them to find an optimal solution. As seen in the video it is still not exact. I presume that with more work this could be resolved for a particular space.
2. There are issues with hardware. Camera fidelity and resolution coupled with frame-rates is a big factor in the set-up, and need to be carefully considered. So is the projector’s resolution, but this was a lesser issue when compared to the camera.

There have been many other explorations in the same vein and I wish to highlight 2 of the most impressive, in my humble opinion:
1. Disney Research ‘Hideout’
2. Kollision’s ‘Advanced 3D projection’

This was a short 3 day exploration at SI-labs. Thanks to Ben for giving me the time and space to experiment 🙂 I hope to use the kinect in conjunction with the projector in the next stages, as I think it throws up more possibilities. Stay tuned for more!

Design playground, thinking interfaces

Thinking about the near future of interacting with audiobooks

Screen Shot 2013-06-23 at 10.51.12 PM copy

This is an experimental sketch and speculation as to the near future of audiobooks or to look even broader, text converted speech. I look to explore more interactive methods of consuming or listening to audiobooks with an eye on recent technological developments and a few pointers from informal user research. This is work in progress!

Audio books have been among us for a while now. From the earliest ideas of Edison, where he envisioned a phonograph playing recordings of books in every household to the app based consumption models of today, the basic concept of listening to an audiobook via a device has remained relatively unchanged. 
But with the advent of screen and touch based interfaces, an ever increasing sophistication in text to speech conversion (and vice versa)  and asynchronous communication between text and audio,  we can possibly explore alternate modes of interacting with audiobooks, expanding the scenarios of use and making them as rich as an actual book.

The_Papa_of_the_Phonograph,_Daily_Graphic (1)

Why do Audiobooks need enhancement?

These set of thoughts are based on my experiences with audiobooks over the years and conversations with a few people I know and also interviews with authors** who love the medium and understand it’s many limitations.
For instance, audiobooks are brilliant for listening to narrative texts, where a voice can hold you in rapt attention to a scene in a book and can actually lend to imagination, largely based on the quality of the narrator.

“An audiobook is its own thing, a unique medium that goes in through the ear, sometimes leaving you sitting in the driveway to find out how the story is going to end.” – Neil Gaiman

But there are limitations, when compared to reading…

– a loss in the sense of time and space, in terms of handling the medium. For instance  dog ears in a book, page numbers, bookmarks, specific constructions of paragraphs all act as indicators in traditional media like paper. Or even in ebooks.

– this loss in the sense of space and time seems to play an important part in memory. The construction of sentences for instance aid in associative memory with respect to passages you love in a book.

– the earlier aspect of cognition and memory make it difficult for people to read reference books or text books. Also adding to the difficulty is the loss in understanding other media if present among the text. For instance, a picture, video , graphs cannot be translated and make it absolutely impossible to rely on an audiobook. Even though some people learn more by listening rather than reading. (A lecture vs a text book for example)

So through the following explorations, as I have tried to explain via a short video clip and text, look at other modes of interacting with audiobooks, possibly with apps in touch based devices.

The elements presented in the video:

1. An animated and interactive indicator for speech, which also enables you to mark passages you wish to return to. more images 1

2. Returning to the marked passage with your scrubber which provides you aural and visual feedback on interaction. more images 2

3*. Easy switching between audio and text, which are synchronously connected. This could be especially useful while working with reference or text books, with other media present.more images 3

These were quick explorations, done over 2 days, which could possibly be extrapolated further.

*Since I wrote this post, the ever innovative Amazon have launched Whisper sync for voice, a fantastic feature which enables a user to jump between text in a book , to the corresponding space in an audiobook. Glad to see point no.3 has seen the light of day. ** The wonderful Neil Gaiman speaks about Audiobooks in this npr story.

Design playground, Thoughts

‘Frugal Digital’

During the last month, I have been doing some work with the ‘Frugal Digital’ research team at the ‘Copenhagen Institute of Interaction Design’, Denmark. I was in India for a while conducting in-context research. You can have a look at the process blog for research done as part of the development of ecosystems around mobile technologies for the screening and management of hearing impairment among people in India.

Frugal Digital focuses on researching methods and practices around creating digital solutions in low resource settings like that of developing economies.

My primary contribution is with in-context research, analysis, identification of key issues and prototyping for solutions around them.

CIID, Design playground

Project update 5 …. Experiment in symbiosis 1 :

The experiment in testing a posible symbiotic relationship (above) and what it’s supposed to do (below)

Thats the setup on my desk (below) at CIID , where I have the device talking to twitter (sending messages to the owners account and looking for his activity) via processing. which is then communicating to the arduino, controlling the feeder mechanism.  A webcam attached to the stem of the ‘Poet’ , functions as an eye and looks down at the fish in the bowl.

It took me a couple of days to craft the feeding mechanism(as seen below , above the bowl) and its a simple robust system now, which can perhaps store enough feed for a month.






CIID, Design playground, Thoughts

Project update 4 : Thoughts and Experiments…

Its week 4 of my project and I am in the process of building up the experiments I spoke about in my last post. But before that I wished to take a step back and talk about whats brewing in my head now

“My challenge in this phase would be to design and test meaningful experimental setups to explore possible symbiotic relationships between people and electronic objects in the natural and electronic climates we live in today.

These experiments would serve to see how symbiosis introduced in interactions could affect behaviour in the test subjects in terms of a more involved sense of attachment and reverence towards these objects. It would also be interesting if these experiments impact his/her awareness of the environment they are part of.


I wished to come up with experiments which would explore these relationships within the context and perspective of a single human being, his social network and the environment .

The starting point would be that of the defining experiments with the objects (as illustrated above)  and to see how relationships would radiate from that circle towards the environment and the human being.


In thinking about symbiosis and how I could design experiments , I decided to look at networked objects and the two realms we tend to reside in today.

In order to talk about what I have been thinking , I am using the illustration below , which speaks about the physical and digital space :

An essence of our physical selves permeates through the membrane towards the digital sphere where our online avatars and persona’s reside. In a similar vein it is interesting to think about digital objects whose essence permeates through the membrane too.

‘The Internet of things’ and ‘ambient intelligence’ in objects around us are a couple of terms bandied about which come to mind when this kind of permeation is talked about. It is already being done as is illustrated with twitter accounts given to bridges etc (!/imlondonbridge) , which is quite interesting.

What if these objects / devices could not only sense and perceive objects in the physical domain, but also your activity in the digital.

Could they then be perceived as digital spirits of objects ? …

This was an interesting question which popped into my head, and reminded me of the notion of Yakshas/Ganhdarvas in Indian mythology (my grandma used to tell me tales about them when I was quite little) , or the Kami in Japanese myth , who were benevolent and sensitive entities said to reside in the arts (music , literature) , elements(sun , wind , rain , certain fires) , rocks , wells etc.

There was a certain reverence attached to them , not in terms of worship , but in terms if a give and take between people and these entities which needed to be maintained in order to keep a balance. This balance was crucial in maintaining order and good fortune .

These aspects of benevolence, sensitivity and balance are something I want to bring to the table in terms of the experiments to be conducted


Crafting the first experiment:

The very first setup is of the ‘Poet’ , a fish feeding system/ device which exists and operates in both the physical and digital space, speaking to the person who owns it and also feeding off his social network , which it needs to survive and continue feeding the fish in its care.

More about this setup in the next post …

Design playground, Inspiration, Thoughts

Randomness: experiments,thoughts, inspiration and Graffiti for bots ?….

‘Robot readable world’  …… So what would graffiti for robots / seeing algorithms look like ?

Maybe like so ?


Peter Jackson on George Lucas  and Star wars in the book  ‘The making of Star wars’ :

‘What Star Wars also did ….was create a science fiction / fantasy word that felt lived in , used. Things were scratched ; there were oil marks , things broke down. ‘ – speaking about the realism portrayed in star wars and the fact that we could identify with the sets created for the reasons stated above.

So what if I came up with devices which did not function perfectly , but were programmed to break down eventually , things which required care so as to make them last longer.

I also came across this set of experiments from Cesar Harada and Tomasso Lanza , which touched upon this subject


From Sherry Turkle’s book  ‘Alone together’ :

“I’m done with smart machines. I want a machine that’s attentive to my needs. Where are the sensitive machines?”
—Tweet available at dig_natRT @tigoe via @ramonapringle

So where are they?


“…think of objects not as instruments for our use, but as entities that are effectively linked and that need care – to think of objects as plants in our garden…Think of objects that are beautiful and useful as trees in your own garden, objects that endure and have lives of their own, objects that perform services and, require care…I am thinking of criteria of quality that leads to a system of objects that have the variety, complexity, life and blend of beauty and utility of a garden but, at the same time, are a product of the real world, a world extensively and intensively artificial.”
Ezio Manzini, “Prometheus of the Everyday”, Design Issues, 1992.

Interesting ….. another phrase that Manzini uses is  ‘ Considered as an operator acting in relation to the daily environment , the designers ultimate responsibility can only be to contribute to the production of a habitable world, a world in which human beings do not merely survive but also express and expand their cultural and spiritual possibilities. ‘

‘contribute to the production of a habitable world’ …..awesome !!!


‘If we were to record all the ways in which gross technological negligence — clear-cutting of forests, pollution from factories, etc.  –  destroys the integrity of the biosphere its clear we need to keep the technium in check or we’re in dire straits environmentally.  But I don’t think this destructive tendency is inherent in the technology. The technium wants many of the same things that we do.  Clean water, for example. Most industrial processes require clean water. Some high-tech processes require water that’s cleaner than drinking water.  In this sense the technium doesn’t want pollution; it wants the same kind of pristine environment that we want, especially with regards to higher technologies……..

….Pollution is the same — the solution to pollution in most cases is better technology. All the trajectories for the technium are towards recycling materials including pollutants, energy efficiency, scarcity substitutions, the replacement of mass with information — all of which we would call green technology.
I can imagine the technium and nature being in harmony over the long term with the exception of one area where these two forces don’t seem to be in alignment: elimination of species habitat. The technium seems to be insensitive to species elimination. I think this a real problem, but it quickly became apparent that we didn’t have a very good understanding of all the species on the earth.    ‘

– Kevin Kelly on the TECHNIUM , the 7th kingdom of life and its attitude to the environment…


Inspiration : Ted Chiang’s ‘Lifecycle of Software objects’ …… a must read…