Unified Media (U n Me)
How to create concepts & technologies for AV and force displays augmented 3D reality, or Unified Media (U n Me), and address personal and community well being by advancing entertainment, health, and economy.
Intimate body-machine merging, resulting from human biomechanics studies and corresponding U n Me system integration, will create rich and compelling 3D physical embodiment in this novel digital medium. Developed concepts and technologies, focused on specific applications, will significantly affect several fields including entertainment and health.
My research will focus on: 1) whole body dynamics & U n Me system integration 2) U n Me applications to entertainment & health and 3) Business Book (business & economy).
Read more below or skip to the list of various potential projects in alphabetical order. Proposed research prospectus is authentic representation of my interest, passion and professional background (top expertise in body-environment interactions -- in particular legs; 3D whole body dynamics: capturing, modeling and simulations; body-machine merging; artificial fingers, artificial legs, complex system integration, graphics and animation; physics and most recently finance/banking). You may also want to see my '05 faculty application and addendum.
Why U n M?
Adding additional functional/natural dimension will likely exponentially augment our ability to parse through large data sets, organize and visualize data, type/write/draw/combine 3D elements and perform several simultaneous tasks = substantially improved work related performance and efficiency. U n Me augmented experience may help grow businesses more efficiently and might have valuable impact on energy/time efficiency (more trust in online meetings & less unnecessary business trips). U n Me will have significant impact on entertainment and leisure activities. Advancement of U n Me technologies may have a significant impact on health and advancement of assistive technologies for physically and mentally impaired. For those that are confined to limited physical space and therefore isolated from society (at homes, hospitals, prisons, army bases, ships, and space stations) U n Me will add important social dimension.
U n Me concepts and technologies will likely inspire "renaissance" for number of fields including: Animation, Architecture, Arts, Biomechanics & Kinesiology, Business, Computer Science and Artificial Intelligence, Design (online and physical add-ons), Economy, Education, Entertainment (Movies, Games, Storytelling etc), Global Policies, Journalism, Linguistics, Neurosonsorimotor/Neuro Sciences, Marketing, Outsourcing, Physics, Physical Rehabilitation, Politics, Robotics (Virtual and Real), Social and Behavioral Studies, Sociology, Speech processing technologies, Therapy, Trade etc and induce numerous Tele-fields and several new fields.
User(s), indoor or outdoor, could choose to "live" (in) 3D movie, build imaginary 3D castles and create stories, physically engage in mixed/virtual 3D environments for multiplayer games, have a business meeting, survey a newly designed building, walk, run, recreate, dance, practice sports and/or martial arts, learn new skills, take a medical test or physical therapy session, visit distant or imaginary places, visit a famous museum or city, explore the surface of an exotic planet, throw online party, hold hands and dance indoor/outdoor with someone thousand miles away dressed in beautiful artistic world of imagination, or play exotic instruments in a band with friends from other coast (with time lag of less than ~ 1/20 of second).
And there we are, maybe facing a new technological bubble -- it might change a lot, especially the way we think about and operate with digital world! And new opportunities seems to be almost unlimited. "Kids, turn off that ... thing and come to dinner!"
Being Physical, Social and 3D spatial. Human H3.0D. Let the FORCE be with U n Me.
Being 3D spatial, physical & social is deeply rooted in human nature. My goal is to critically address prospective technologies, their advantages and disadvantages and to create more beneficial, more natural and more nature connected media concepts and technologies that reflect and adapt to our needs and positively augment quality of our lives. Technology should not be handicapping element, instead it should improve personal and community well-being for both better today and better tomorrow.
Modern mass media devices are limited to synthetic AV "displays" (with only a few exceptions e.g. Wii gaming interface). Furthermore, beyond "type" & "move/click the mouse" control actions, these AV "displays" are independent of body state*. While watching movie or performing work related activity one typically can not walk/dive/fly through 3D scene, change point of view, orient "AV head" and in real 3D world-like manner, physically touch, grasp and manipulate media objects. Similarly one can not hold hands and dance indoor/outdoor with someone thousand miles away dressed in beautiful artistic world of imagination**. Finally, because computing V displays do not interact with us in more realistic 3D physical ways we typically limit ourselves to 2D (or even 1D) structured digital world visualizations/ thinking. Utterly unnatural and definitively unnecessary!
(*Body state=position/orientation in space, where the various parts of the body are located in relation to each other & time derivative.)
(**U n Me communications will never be and should not be a substitute for natural direct human communications. Rather, in the similar way as telephone, radio, TV, internet and other A+2D V media are benefiting our civilization U n Me will be useful, add-on world, human assistive medium.)
Natural physical interaction assumes: 1) environment generated input = AV and action-reaction forces that all depend on body state relative to 3D physical setting 2) creation of body force output that changes surrounding and defines subsequent body state via 3) set of physical laws. That determines a natural way how we move through space, stay balanced, perceive Nature, and interact with other human beings and environment. Mass media technologies need yet to explore this important natural/human dimension!
Unified media (U n Me) is envisioned as novel users connected online medium providing human-centered natural sensation of 3D media embodiment firstly via richer and more natural feeling of spatial location/AV perceptive field orientation and secondly via proprioception, vestibular sense and ability to apply "virtual" action-reaction forces to operate with abstract representations of otherwise distant or imaginary objects in physically meaningful ways.
By advancing unified media concepts and technologies I will address that important missing natural/human dimension and help create new powerful medium for creation, imagination, entertainment, learning, healing; help build a more natural technological bridge for human-human communication and induce positive personal and societal development (something that majority will feel contented with in the years to come).
Current trends and some of the potential impacts
Modern mass media are characterized by several trends including: new applications, augmented experience (larger screens, better audio systems etc), mobility, compactness (small, multifunctional, energy efficient etc), pronsumers (producers/consumers) connected networks etc.
Regarding new applications -- advancement of U n Me (including T3) concepts and technologies will likely inspire "renaissance" for number of fields: Animation, Architecture, Arts, Biomechanics & Kinesiology, Business, Computer Science and Artificial Intelligence, Design (online and physical add-ons), Economy, Education, Entertainment (Movies, Games, Storytelling etc), Global Policies, Journalism, Linguistics, Neurosonsorimotor/Neuro Sciences, Marketing, Outsourcing, Physics, Physical Rehabilitation, Politics, Robotics (Virtual and Real), Social and Behavioral Studies, Sociology, Speech processing technologies, Therapy, Trade etc and induce numerous Tele-fields and several new fields mentioned below, eg Space reconstruction (from sound & light) and Touchograph .
U n Me (including T3) augmented experience will have significant impact on various ways we think about and operate with digital world. Adding additional functional/natural dimension and appropriate interfaces* will likely exponentially augment (vs. linearly) our ability to parse through large data sets, organize and visualize data, type/write/draw/combine 3D elements and perform several simultaneous tasks. This suggests substantially improved work related performance and efficiency.
*Several examples of proposed interfaces: Exo hands may exert "virtual" forces defining intangible (distant or imaginary) objects and let user touch, grasp and operate with 3D AV objects and/or augment typing/writing and/or info-sculpting performance (for static table/chair mounted exos see Haptic exo hands, Magic Place, for mobile/wearable exos see Eyeglass VR/AR post-ed culture). Dynamic armchair could provide exploration of U n Me in more whole body proprioceptive and spatially unified way (see Magic Place ). Walk-on ground platform could give sensation of mixed/virtual 3D environment multidirectional walking/running on complex topologies (see Media Platforms). T3 3D controllers will let user navigate freely along 3D path in T3 media space (translations) and control T3 AV perceptive field orientation (rotations) (see T3 instead of TV (Tele3) ). Light-weight, private, and energy efficient eyeglasses that could be used for both VR and AR, see Eyeglass VR/AR post-ed culture.
U n Me (including T3) augmented experience will help grow businesses more efficiently and will have valuable impact on energy/time efficiency and ; e.g. E-meetings, advancing realistic experience of physical presence to boost trust (maybe even via tele - hand shake :) ) and to remove unnecessary long trips and large energy / fuel / money / time waste; see BusinessBook , T3 instead of TV (Tele3) , Haptic exo hands, Eyeglass VR/AR post-ed culture .
U n Me will have significant impact on entertainment and leisure activities. User(s), indoor or outdoor, could choose to "live" (in) 3D movie with audiovisual "field" maybe attuned with "head" movements, build imaginary 3D castles and create stories, physically engage in mixed/virtual 3D environments for multiplayer games, walk, run, recreate, dance, practice sports and/or martial arts, learn new skills, visit distant or imaginary places, throw online party or play exotic instruments with friends from other coast (with time lag of less than ~ 1/20 of second).
U n Me technologies will have significant impact on physically and mentally impaired; e.g. rehabilitation devices and vestibular prosthesis/postural reminders for movement impaired, haptic displays and rapid space reconstruction for blind, visual post-ed reminders for those with memory problems, sense of caring touch and training systems for autistic kids etc, see Assistive technology for physically and mentally impaired .
Advancement of U n Me concepts and technologies may have impact on mobility and compactness. Typing/writing systems for mobile systems -- typing/writing (or future 3D info-sculpting) requires a haptic feedback, therefore it would be valuable to create novel mobile/compact typing/writing interfaces including wearable "finger pen" and/or exo hands to provide superior performance and comfort; see Eyeglass VR/AR post-ed culture
If popular, U n Me may induce large scale pronsumers connected networks and therefore have significant impact on various industries as well as on many important societal aspects. (see BusinessBook , Community concepts & technologies)
Finally, to those that are confined to limited physical space and therefore isolated from society (at homes, hospitals, prisons, army bases, ships, and space stations) U n Me (including T3) will add important social dimension.
Formally
I plan to address many things: 1) advanced synthetic AV and Force "displays" whose output will depend on user's "relative state", 2) comfortable control interfaces that will allow user to exert input on environment, potentially change it and define own subsequent "relative state", 3) space/time independent/dependent physical and "magic" laws/parameters, 4) different space contents, representation & space topologies 5) interactions between different spaces 6) user's simultaneous (in real time) presence at different locations/times in the same space 7) user's simultaneous presence in multiple spaces 8) user/observer dependent 1)-7) that are user and/or system(s) defined 9) many users in single space 10) multiple users in multiple spaces 11) rapid "recording"/"reconstruction" of real space and creation of "simulated' space 12) compelling and useful applications to advance personal and community well-being including entertainment, health, economy, and 13) various advantages, disadvantages and commercial viability.
My goal
My goal is to create U n Me (including T3) concepts and technologies to advance personal and community well-being including entertainment, health, & economy. I will:
1) continue my work on human biomechanics, balance and control (see Movements & Forces),
2) address various advantages/disadvantages and engineer concepts and technologies that will establish pathway to large scale U n Me (including T3) rich media environments (see T3 instead of TV (Tele3), Media Platforms, Magic Place, Haptic exo hands, Eyeglass VR/AR post-ed culture etc)
3) utilize U n Me (including T3) concepts and technologies to advance entertainment (with substantial effort on concepts and technologies for younger generations, see Future, that's our kids)
4) utilize U n Me (including T3) concepts and technologies to advance health (see Assistive technology for physically and mentally impaired)
3) and last but not the least continue my work on large scale enterprises, e.g. Business Book concept (see BusinessBook), and other community related concepts and technologies (see Community concepts & technologies).
Proposed research prospectus is authentic representation of my interests and professional background (see my '05 faculty application). I will address these challenging problems in modular, step by step, approach. I hope to closely collaborate with many of my ML and MIT colleagues on number of these interesting and important research themes.
| BusinessBook | ||
| Eyeglass VR/AR post-ed culture | Future, that's our kids | Haptic exo hands |
|
Magic Place (Media installation space for new building) |
Media Platforms | Movements & Forces |
| Space reconstruction (from sound & light) | Touchograph | T3 instead of TV (Tele3) |
Assistive technologies for physically and mentally impaired
I am 100% open for collaboration on any of the following prospective projects:
For movement impaired people, I would like to continue my work on biomechanics, balance and control and apply that knowledge for systems that will intimately merge body and machine and help people walk, move and/or interact with environment.
I plan to study Parkinson disease and vestibular deficiency using standard VICON system, AMTI Force Plates, Media Platforms, Haptic exo hands and potentially engineer wearable vestibular prosthesis and/or postural reminders.
For semi-paralyzed people assistive devices and for autistic kids companionship, education and training see below Haptic exo hands (maybe together with Eyeglass VR/AR post-ed culture).
For blind people see/touch Haptic exo hands, Eyeglass VR/AR post-ed , Space reconstruction (from sound & light), and Touchograph.
For those with dementia, memory problems and for those who needs to be often reminded see Eyeglass VR/AR post-ed culture.
Also, I would like to explore general and specific strategies how these systems may monitor human physical and physiological performance, understand human needs and adapt accordingly to optimize for the best performance.
One could develop better communication technologies for peer to peer courier services and online group meetings (this will help to introduce more trust to online businesses and remove some of the unnecessary long trips and energy/fuel/money waste). This is another context in which U n ME (including T3) and human interaction language will become particularly important; e.g. imagine e-meetings where participant's activity (voice/gesture) is directly correlated with his/her representation on the screens of other participants -- different participant could have different views depending on their engagement with other participants, also when someone is talking his/her presence will be zoomed in and more central to e-meeting participants etc. Finally tele - hand shake may add important element to boost trust between business partners.
U n Me (including T3) augmented experience will have significant impact on various ways we think about and operate with digital world. Adding additional functional dimension and appropriate interfaces will likely exponentially augment (vs. linearly) our ability to parse through large data sets, organize and visualize data, type/write/draw/combine 3D elements and perform several simultaneous tasks. This suggests substantially augmented work related performance and efficiency. This fact could have particular impact on large and complex banking and financial data sets.
BusinessBook (BB) is envisioned as advanced high-tech unifying platform that could help you grow business and make money. More specifically how technology may assist the currently handicapped economy via business-augmenting, internet-based financial platform that is intermediary among traditional financial institutions (BANKS, building societies, credit unions, stock brokerages, asset management firms etc). For more details see BB.
MIT has been traditionally "technological brain" behind US military-industrial complex. Maybe it is good time to focus some of that amazing intellect toward economy and banking, recreate new concepts and technologies that could help general populi in US and worldwide to bear progressively with this crisis and recreate new better tomorrow. Therefore goal is to create concepts and technologies for an inspiring business enhancement tool that could help to remedy the currently handicapped world economy.
BB is thought of as self-organized (within its predetermined principles/rules) on-line business workplace and financial market that may grow to a large global enterprise encompassing many banks and billion of users across continents. BB may provide: business projects web hosting (with template forms), data storage (time stamped electronic agreements/contracts etc), secure financial transactions in cooperation with partner financial institutions, network visualizations, search engines, track record of members? business performance under various metrics, basic legal and business consulting, templates of business forms/agreements/contracts, content/copyrights supervision and anything else that could help its goals.
BB may utilize the project description provided by project owner, site visitors? reviews and when allowed (by project owners) project business metrics/reports in order to assist users, BANKS and other financial institutions to devise and implement, current and future, investment and other strategies. It will assist business owners to plan their businesses well in the future, notify them who are the reliable current or future investors, service providers or other partners/clients and finally assist them to succeed with more certainty and without wasted time, money and energy.
Community concepts & technologies
How to create concepts and tools for people to reach to other people, express problems, suggest resolutions and unify for common cause, personal and community well-being.
How to reach to and help those that are left behind social, economic, educational and technological landscapes of modern society.
How is technology influencing basic structure of decision making process at small and large scales? How that could be improved? Could democracy be improved in US and abroad?
Is there a need for high level global decision making, i.e. reason for democratic governing organization to exist in an online world that span across the globe (meaning many counties)? Should this be similar to UN or completely new structure or maybe no predefined structure is necessary?
These are all very important questions that I don't have the answer to but I would like to address them with my students and colleagues.
Eyeglass VR/AR post-ed culture
Based on currently available technologies and market trends I anticipate growth in various applications involving light-weight VR (Virtual Reality) and AR (Augmented reality) eyeglasses over next several years. I would like to address technologies that will advance these systems and seamlessly blend body (in particular head and hands) dynamics with synthetic visual, audio and tactile content. Also, I would like to work on novel mobile typing interfaces and/or wearable exo-hands. (For table/chair mounted exo hands see Haptic exo hands ). Finally, I plan to address novel concepts how these mobile technologies can be used in everyday life.
Imagine wearable computing embodied in your cell (that's computing "brain") maybe embedded in your VR/AR glasses with post-ed images/sounds plus "vestibular system" and finally system for typing/writing (or 3D info-sculpting) - this system will require a haptic feedback, therefore it would be valuable to create novel typing interfaces including wearable finger pen and/or exo hands that will provide superior performance and comfort.
I anticipate that lightweight, small, private, energy-efficient eyeglasses (or theatre glasses) with powerful visual display may be relatively inexpensive consumer product (in $50-$300 range) that might impact mass-media experience.
How love and technology could improve kids healthy and happy development via play, storytelling, games and entertainment, including physical and outdoor activities in novel ways. My goal is to create physically engaging and more natural and nature connected technologies for kids.
Conceptually: Children need attention and love. Children need to play and have fun. Non formal learning, learning via interactions with peers, learning through play and learning through creation are pivotal for children's development. Finally children need technologies that will stimulate physical and outdoor activities in novel ways - kids need to jump, run, kick and sweat. I wrote on April 16, 2005: "Current computer interface is inappropriate for the youngest generation. One should explore to which extent conventional technology may be exchanged with more natural interfaces. Kids could walk, run, jump, wave, shake, point toward objects, move toys, sing and scream etc. and in that way communicate with digital world while playing alone or with each other and building virtual or real castles of their imaginary worlds. Kids may also create visual and sound ambiance and compose interactive stories in which they would react to, reproduce or vary the actions of actors or generate their motions and actions from scratch. This work may help my daughters and other children to enjoy modern technologies in more natural way."
Technologically: Social computing for kids networks. Novel gaming interfaces and AR, MR, VR wearable devices. Advanced 3D multimedia= interactive toys & robots (that will have all laptops' functionalities plus much more), interactive/intelligent media distributed in 3D space, and advanced 3D tactile displays (3DTD) rapidly communicating palpable reality of otherwise intangible (imaginary or distant) objects. BTW, 3DTD may be: 1) reconfigurable dynamic surface (like 3D rapidly changing sculptures), 2) wearable vibro-tactile/pressure devices, actuated exoskeletons/skins.
I also plan to continue work on New Playgrounds with colorful, shape-changing and force-generating elements that could augment outdoor playground experience. It would be very interesting to combine New Playgrounds physicality with AR concepts, lightweight kids eyeglasses or Patie Maes' sixth sense class of devices and engage children in collective games and spatially distributed storytelling.
Finally, I anticipate popularity and therefore rapid development of T3 technologies see T3 instead of TV (Tele3) and plan to create class of T3 add-ons concepts and technologies that will be designed especially for younger generations.
I plan to study class of new "haptic exo hands", table/armchair mounted (see Media (Chair) Platform) and create compelling immersive experience of virtual force exerting U n Me Space. For wearable exo-hands see Eyeglass VR/AR post-ed culture. I am inspired/challenged to address this problem for two reasons: 1) to study biomechanics of hand interactions and 2) to investigate commercial viability of this media. Currently, there is no scientific or technological obstacle for new generation of exo hands that could (maybe one-sidedly) glue to user's hand, seamlessly follow fast hand/finger movements and finally apply appropriate forces to simulate new haptic worlds.
Imagine advanced internet based AR, MR or VR spaces where keyboards, mousses, touch screens and simple data gloves are exchanged or augmented with vibrant, force generating, modular and finally elegant interfaces that may add new dimension to workplace or open doors to new worlds of fantasy, magic, play, storytelling, social interaction, and love.
My research objectives are 1) to experiment with creation of relatively inexpensive novel human-machine interfaces, i.e. 3D force field displays that will communicate "physics", i.e. forces defining intangible (distant or imaginary) objects integrated with other existing tactile, auditory and vision interfaces; 2) to create control systems and basic media content so that compelling experiences of U n Me Space may be generated for users; and 3) to study both human and machine and their fusion into the medium, including both physical and social interactions.
Magic Place -- (Media installation space for the new ML building)
Media (Chair) Platform is envisioned as comfortable, dynamic armchair (in living room or in the new ML building's Magic Place space :) ) that will let you explore U n Me Space in more proprioceptive and spatially unified way. As you navigates through the Rich Media Space the Media (Chair) Platform will move with you in real 3D space. With exo-hands (see Haptic exo hands) attached to the Media (Chair) Platform you will be able to touch and operate U n Me content. I will create hardware and software for class of Media (Chair) Platforms.
Video content may be either projected on walls, presented on large spatially distributed screens or perceived through new generation of light-weight goggles (in which case sound audio input should be synchronized with head motion).
These systems will allow user to interact with 3D movie scene, research distant real or imaginary spaces, play multi-player games, analyze newly designed buildings or investigate vast amount of banking data situated in a large dimensional abstract space.
I anticipate that Media (Chair) Platform will be relatively inexpensive consumer product (in $1K-$2K range, clearly with various limitations) that might impact mass-media experience. Furthermore, this could be one of the elements for Media Installation Demo Space in the new ML building. Deformable wall surfaces with low & high spatial and temporal resolution, potentially motion/touch/force responsive, may be also added to this system (I'll be happy to design or help design these systems).
I plan to create class of (mostly) stand-alone force-generating indoor/outdoor systems that could augment large scale connected immersive space experience -- most notably Media (Base) Platform, Media (Chair) Platform and New Playgrounds. (see also Haptic exo hands and Eyeglass VR/AR post-ed culture )
Media (Base) Platform is envisioned as system that could provide multidirectional locomotion (locomotion on ordinary treadmill in comparison is one-dimensional) and also simulate complex topologies/stiffnesses beneath feet. Media (Chair) Platform is envisioned as comfortable dynamic armchair (for living room) that could augment exploration of U n Me Spaces in more sensorymotor, proprioceptive and spatially unified way (for more details see Magic Place). I also plan to create (kids/energy/environment friendly) New Playgrounds with colorful, shape-changing and force-generating elements that could augment playground experience (for more details see Future, that's our kids). I also plan to experiment with light-weight "Post-ed" or VR eyeglasses that could mix real and imaginary worlds to different extent (see Eyeglass VR/AR post-ed culture).
Media (Base) Platform
Current design of Media (Base) Platform allow for two modalities (multidirectional & complex-topology) to be addressed separately; it doesn't require "mirror legs" or otherwise cumbersome hardware. Furthermore, this system could be used as an exercise machine without MR (mixed reality) or VR (virtual reality) elements. I am working on its mechanical design since 2004.
I anticipate that Media (Base) Platform will be relatively inexpensive consumer product (in $1K-$2K range, clearly with various limitations) that will provide rich physical & social experience; imagine socializing and dancing at online party / playing dynamic multi-player game / living 3D movie / walking & exploring newly designed building etc. It may introduce popular pathway to highly immersive environments and inspire otherwise inactive people to engage in physical activity.
For outdoor activity (maybe in combination with Eyeglass VR/AR post-ed culture system) one will not require multidirectional "treadmill" (maybe only variable stiffness beneath feet).
I plan to continue my work on human biomechanics and study human movement/force "space" at different levels of granularity for different subjects, tasks, world conditions and finally applications.
It will be valuable to build movement/force model at given granularity level that can explain observed movement/force "landscape" and/or synthesize novel dynamic activities. This knowledge can be then applied to numerous human embodied, vision and/or sound based interfaces. This knowledge can be also applied for rapid generation of biologically inspired avatar and/or robot movements in synthetic, mixed and/or real environments.
Media Platforms and Haptic exo hands (see also Eyeglass VR/AR post-ed culture and Touchograph) will be great scientific platforms for extracting these individual and collective, granularities at different scales in order to address human interactions, "physical intelligence" and finally numerous practical applications. Since 2001 I am actively using VICON motion capture system, ordinary stereoscopic camera, AMTI force plates system and various actuated, force-generating systems.
These systems will be great scientific tools for biomechanical studies and neural control studies of human movement dynamics. And, when added with other MR or VR elements these system may become a very sophisticated laboratory for medical, socio-psychological, behavioral and other studies.
I would like to study human locomotion, perception, reflexes, adaptability to novel conditions, learning processes at the lower and higher cognitive levels, and later variety of socio-psychological behavioral phenomena at different scales in a well controlled, repeatable experimental environment that is rich in various synthetic sensory inputs. Among other I would like to study healthy people and those suffering from Parkinson disease & vestibular deficiencies. (My mother is suffering from Parkinson disease.)
Space reconstruction (from sound & light)
I would like to advance commercial systems (hardware/software) that could record space (via emitters/receivers of sound and light) and rapidly digitally reconstruct the geometry of space/content characteristics (object/pattern identification, position, speed, material properties etc). Imagine walking in the nature or exploring beautiful cathedral and digitally storing its 3D geometry and other properties. Numerous applications ranging from AR/MR/VR interfaces, architecture, meteorology, sea and space exploration to mobile systems, robotic control and health technologies.
Imagine that you are able to precisely record the physical characteristic of haptic interactions with objects or human body (various forces as function of position and speed) and create model that will allow you to recreate subsequent illusion of physical contact. Imagine that you are able to touch and feel face of your kid or loved one as it was twenty years ago.
I plan to advance novel ways to provide 3D media content assuming a) potentially novel hardware/software on the media producers side and b) already existing (or majority of) hardware technologies on the media consumer side (2D screens, pc/laptop, etc). Goal here is to provide compelling T3 experience. T3 consumer will navigate freely along 3D path in T3 media space (translations) and control T3 AV perceptive field orientation (rotations) -- this navigation will define picture on screen & sound from loud-speakers.
A) Smooth matching from multiple camera/microphone angles from the same recording point of 3D space can easily provide 4pi (i.e. entire sphere) AV perceptive field (PF) . However, more difficult task is to smoothly match 4pi PF from two or more recording points. Furthermore, it is even more difficult to obtain satisfactory matching with <4pi recordings and various sources of noise. It would be valuable to develop software that could provide better 3D AV PF matching (and therefore better T3 experience) for given set and geometry of cameras/microphones and/or solve for the most optimal recording geometries/strategies given the set of preferences/instructions by the T3 crew. It would be valuable to create state-of-the-art software that will utilize users' annotations, simultaneously analyze audio and video recordings and also utilize novel devices (see Space reconstruction (from sound & light) )
B) Advancements of T3 technologies will minimally include novel remote controllers that will allow user (potentially passively sitting in his/her armchair) to comfortably control 3D path (translations), AV PF orientations (rotations) and several other functions. I would like to address this problem and create more advanced remote controller, 3D functionality augmented mouse and/or joystick. For more advanced exo-hands that could be either wearable see Eyeglass VR/AR post-ed culture or table-chair mounted see Haptic exo hands. For more dynamic armchairs or walk-on platforms that could be used for media exploration see Media Platforms and Magic Place . In case of single screen and multiple-viewers each with remote control competing for point of view and AV PF orientation (potentially playing multiplayer physically engaging game) I would like to develop software that will automatically combine multiple inputs (maybe combined with their 3D physical locations) with e.g. T3 recommended values and obtain optimal result.
T3 media will potentially have impact on many fields. Imagine T3 impact on better communication technologies for peer to peer courier services and online group meetings (that could help introduce more trust to online businesses and remove some of the unnecessary long trips and energy/fuel/money waste). See BusinessBook Similarly T3 systems will allow users to interact with/live in 3D movie scene, engage in novel type of storytelling, research distant real or imaginary spaces, play multi-player games, analyze newly designed buildings or investigate vast amount of banking data situated in a large dimensional abstract space. I plan to address these and similar problems.