In a world where signs of global warming are becoming increasingly common, data has become a critical component in the fight against climate change. In recent years, with the help of satellites and other remote sensing platforms, researchers have been able to observe and track minute changes in the Earth’s surface, waters, and atmosphere. This course of study, called Earth observation, has been at the forefront of many papers shown at this year’s EarthVision Conference.
Computer Vision and Pattern Recognition (CVPR) Conference, the premier computer vision conference, hosts the Institute of Electrical and Electronics Engineers (IEEE) EarthVision workshop, which has become known as a prestigious venue for machine learning and remote sensing research. The event brings together experts in the fields of machine learning, environmental science, and remote sensing to discuss ways to improve the automated analysis of Earth Observation data.
At this year’s conference, Ph.D. student Ando Shah received the Best Paper Award for “Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation.” The research addresses a crucial issue in the field; there is an overabundance of vastly different satellite data and a lack of fluid, task-adaptable machine learning models that could deal with these different kinds of data.
In recent years, Shah has pursued a variety of research topics, such as illegal sand mining and water management methods in rice cultivation. In his project, Sand Mining Watch, Shah developed a suite of open-source tools to enable the production of high-resolution maps of sand-mining activity around the world. These tools rely on satellite imagery and deep learning to automate the process of detecting sand mining activity.
“I am an environmental data scientist working on a range of topics where measuring change at scale is important… While working on a difficult environmental policy problem involving illegal mining and violence over vast regions, I found it difficult to train a model that could identify such mining signatures because there were very few labels,” Shah stated.
To solve this issue, Shah and his collaborators set out to build a foundation model, a pre-trained machine learning model that uses a minimal amount of labeled data to solve difficult tasks. This model became the namesake for their paper: Panopticon.
“Panopticon” often refers to a concept first introduced by the 18th-century jurist Jeremy Bentham to describe an ideal model for prison design where one single person could watch over an entire prison. The philosopher, Michel Foucault, reinterpreted it as a metaphor for repression, control, and surveillance. Despite its controversial connotations, the research team chose to flip its meaning: rather than watch people, the model serves as a metaphor for systems that can keep a watchful and benevolent eye over our planet.
“The beauty of our model is that, like the original panopticon concept, it provides comprehensive visibility from a single vantage point. But unlike Bentham’s prison design, our goal is not control and fear, but rather understanding and monitoring,” Shah said. “There is also a technical parallel that we find fitting: the original panopticon was designed to see everything from a central position, regardless of where attention was directed. Similarly, our model can “see” through any sensor configuration without needing specific adaptations — a kind of sensor-agnostic vision that mirrors the all-seeing nature of the conceptual panopticon, but repurposed for planetary good.”
“The abilities that were unlocked by this model are being used by myself and other researchers in domains from remote sensing to climate modeling. The original environmental policy problem with illegal mining is now using Panopticon, and it seems to be working better than all models we’ve tried so far, which has been very encouraging!” he added.
Panopticon’s win in the Best Paper category ultimately cemented the research as a substantial step forward in the Earth observation field and has created new emerging questions to explore further. Shah was also invited to speak at the European Space Agency’s biennial conference, Living Planet Symposium in Vienna, Austria, as well as the IEEE Geoscience and Remote Sensing Symposium in Brisbane, Australia.
“The most important thing that I wanted to do in this paper was…to have a principled approach to research with a set of grounded hypotheses and to provide empirical evidence for them on what works, what doesn’t, and why. I think we achieved these goals, and those particular results will live on and help computer vision researchers to develop better methods that work well for earth observation,” Shah explained.
“This was my first foray in building large AI foundation models, and the experience I gained has been invaluable…Overall, this has been a tremendous experience, and I couldn't have done this without my co-first author Leonard Waldmann, and the support of my advisor John Chuang!”
