Stop for a second and look out your window. You’ll likely see a tree somewhere outside. Do you know its species? How often should it be watered? Is it healthy?

Pietro Perona imagines a world where you could look at a tree—or myriad other objects—and, with the help of a smartphone, gain instant expertise about it. While you may not know anything about the item, “someone will know what it is,” says Perona, the Allen E. Puckett Professor of Electrical Engineering at the California Institute of Technology (Caltech). “And if someone knows, [your] iPhone should be able to know as well.”

He doesn’t mean that you’ll just ask Google or Siri. Rather, you’d take a photo of the object—take the tree, for example—and an algorithm used in an app or a website would be able to recognize it by comparing your image to data collected on a variety of tree species. Perona envisions this type of algorithm working for millions of other observable things, from plants and animals to geological formations—even various health-related symptoms, like skin rashes.

Perona researches computer vision, and how to “reproduce the ability of the human visual system in a machine,” he says. For the past few years, he’s been particularly interested in training computers to do fine-grained visual categorization, which is the ability to identify objects very specifically, like a tree down to its species, or a building by its style of architecture. Some companies already use technologies that can do this—Facebook’s software can recognize your friends’ visages in photographs, for example.

In 2009, Perona started exploring the idea of using machine learning to provide on-demand expertise about bird species. It was the first project in an ongoing collaboration with Cornell University computer science professor Serge Belongie called Visipedia, which is essentially a visual encyclopedia that learns from human experts how to classify objects. Their labs worked with bird experts to label thousands of internet images of birds from around the country, annotating distinguishing details about their beaks, eyes, and other morphology. They then fed this data into their algorithm, thereby “teaching” it how to identify birds through their observable traits. 

Computer visual algorithms “have strong learning components, so you [can] build a very generic piece of software that can learn from example,” Perona says.

The resulting bird algorithm can so far identify more than 550 bird species in North America, Perona says, and was recently applied to a mobile app called Merlin, which enables users to upload or take a photo of a bird to find out its species.

For his next Visipedia project, Perona is applying the algorithm to trees, having been inspired by the neighborhood conditions around Caltech. “I was somewhat concerned about the future of trees in Pasadena because California was restricting watering, and I could see that many trees were dying,” Perona says. He thought that the algorithm could provide a better street tree inventory, which public officials could draw on to improve their strategies for tending to their urban forests.

Fortunately, satellite imagery services like Google Maps provide a regularly updated and publicly available trove of images of street trees in urban areas like Pasadena. Perona recruited human volunteers to label the trees from non-tree objects in images taken around Pasadena, and then used species information from a previous tree inventory done by the city to train the algorithm’s identification skills.

In a proof-of-concept test comparing the algorithm’s output with the city survey of Pasadena trees, the algorithm correctly detected about 80 percent of the trees, and out of those, identified the correct species about 80 percent of the time, Perona says.

Perona foresees this algorithm being used primarily by city planners across the country. By precluding the need for human surveyors, the technology could potentially save tens of thousands of hours and possibly even millions of dollars, Perona estimates. Moreover, the algorithm could be used to conduct arboreal surveys more regularly, allowing for more updated information on the status of trees in a given area. 

“Most cities will survey every 20-30 years, [but] potentially we could have a survey every six months,” Perona says. “We could see how the urban forests in the U.S. are evolving over time much more accurately, how they’re responding to climate change, how they’re responding to a virus that’s spread from one county to the next. It will be more like seeing the stock market or something that changes all the time, and it gets updated rather than [retaining] this very static point of view.”