Mind’s Eye surveillance to watch, identify and predict human behavior

The future surveillance society world could have a very Orwellian no-privacy flavor.

Carnegie Mellon researchers, taking part in DARPA’s Mind’s Eye program, have created visually intelligent software to recognize human activities in video and then predict what might happen next.

networkworld.com | Oct 29, 2012

By Ms. Smith

If a person holding a gun were to walk up to you, what might you think would happen next? Researchers from Carnegie Mellon have created intelligent software that will identify human activities in videos and then predict what might happen next. It should come as little surprise that the spookily named ‘Mind’s Eye’ program is sponsored by DARPA’s Information Innovation Office.  

“A truly ‘smart’ camera would be able to describe with words everything it sees and reason about what it cannot see,” said DARPA. Visually intelligent technology previously ‘thought’ in terms of nouns to describe a scene, but Carnegie Mellon researchers have made smart software that can also think in terms of action verbs. “A video shows a woman carrying a box into a building. Later, it shows her leaving the building without it. What was she doing?” asked Carnegie Mellon University.

The Mind’s Eye software “will compare the video motion to actions it’s already been trained to recognize (such as walk, jump, and stand) and identify patterns of actions (such as pick up and carry). The software examines these patterns to infer what the person in the video is doing. It also makes predictions about what is likely to happen next and can guess at activities that might be obscured or occur off-camera.”

Carnegie Mellon University’s National Robotics Engineering Center explained the image below as: “The Mind’s Eye program will automate video analysis – recognizing current behavior, interpolating actions that occur off-camera, and predicting future behavior.”

The next step is to make the ‘Cognitive Engine’ even smarter. According to the report “Using Ontologies in a Cognitive-Grounded System: Automatic Action Recognition in Video Surveillance”, the researchers “plan to extend the system functionalities in order to support a wider range of action verbs and run tests on a large video dataset.”

DARPA explained, “In the first 18 months of the program, Mind’s Eye demonstrated fundamentally new capabilities in visual intelligence, including the ability of automated systems to recognize actions they had never seen, describe observed events using simple text messages, and flag anomalous behaviors.” Carnegie Mellon is one of twelve research teams and three commercial integrators participating in the five-year Mind’s Eye Program.

Previously, BRS Labs had the “smartest AI suspicious behavioral recognition system” with “the capability to learn from what it observes, remember activity patterns and adjust to changes in the environment, field of view and equipment – without manual interaction.” Phys reported that the Carnegie Mellon “researchers’ approach is designed to help prevent crimes or dangerous events from happening.” The newest visually intelligent software “system would eventually sound an alarm if it recognized that an action was not permitted, detecting anomalous behaviors. One example of such a scenario would be the cameras at an airport or bus station, flagging a bag abandoned for more than a few minutes.” This Army-funded AI research was disclosed “at the Semantic Technology for Intelligence, Defense, and Security conference at George Mason University.”

The Mind’s Eye system could potentially be used to analyze live drone footage. Who knows, it might even be integrated to work with the unblinking surveillance stare of the Army’s 7-story flying football field-sized blimp? It will likely be embraced in the future by the police and by the military to keep soldiers out of harm’s way. It might have home security applications, watching the surveillance video and alerting home owners with a text message before burglars break in.

On the creepy privacy invasion side of the coin, it’s one more surveillance technology hunting suspicious behavior. Let’s hope the researchers get it right because when added to social media surveillance helping the government read your mind and future TSA plans to track all ‘daily travels to work, grocery store and social events’, the future surveillance society world could have a very Orwellian no-privacy flavor.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s