Pushing Pixels

The scientific practices that provide us images of Martian landscapes also cast seeing itself in a new light

This spring, the Curiosity rover made headlines when scientists reported new evidence of liquid water on the surface of Mars. By measuring the temperature and relative humidity in a crater over the course of a Martian year, the researchers found that salts in the soil could pull water vapor out of the atmosphere and form a brine during the overnight hours, before the moisture would evaporate again at sunrise. “We don’t really see the water,” the study’s lead author acknowledged in an interview. But other instances of seeing, like the dark markings that show up in orbital images and that appear to be water flowing downhill, were what motivated this latest set of measurements in the first place. On Mars, what we can see shapes what we can know.

Vision is commonly taken to be the most direct way of perceiving the world around us. But since the 1970s, visuality has taken a drubbing in the human sciences as feminists and scholars of colonialism have linked the gaze to projects of domination. A prominent anthropologist called for a turn toward “sensuous scholarship,” which would foreground smell, taste, and texture instead of the bloodless rationalism of sight. Quietly, though, a revival of interest in vision has been gaining momentum, one that brings specific practices of seeing into focus while emphasizing their partiality, plasticity, and rootedness in the material world. Some of the most interesting contributions have examined the screen-based graphics and data visualizations that increasingly organize contemporary seeing, at least in the wired global North.

In sociologist Janet Vertesi’s new book about the Mars Rover mission, the work of seeing is fundamental to interactions between scientists and their robot proxies. Images taken by the nine cameras mounted on each of Curiosity’s predecessors, Spirit and Opportunity, anchored a research enterprise distributed across multiple sites and disciplines. As Rover scientists engaged in what Vertesi calls “a practical process of visual construal,” they advanced and tried to commensurate different visions of the Martian surface. Some team members even developed an embodied sense of the rover’s experience on Mars, imaginatively reconstructing how it would see and move across the landscape. Drawing on two years of ethnographic research, Seeing Like A Rover brims with the texture of everyday life on the mission even as it offers fresh insight into the work that images do. The book’s lavish, full-color illustrations were made possible by subventions from Princeton University, where Vertesi teaches, and they allow us to see what she and her interlocutors saw.

I got interested in Vertesi’s work because of its resonance with my own research on unmanned pilot training: both of us write about seeing and acting at a distance. Just as flying an aircraft from a ground control station changes what it means to be a pilot, conducting planetary science from fifty million miles away means turning field geologists into, as one ruefully put it to Vertesi, “pixel pushers.” Rover scientists triangulated their digital image work with lab research and visits to terrestrial analog sites, but they were forced to contend with the constraints imposed by their physical remove from Mars. Unmanned pilots report a three-second lag between their inputs and the aircraft’s response while operating over a satellite link, but for Rover scientists this lag could last for up to twenty minutes, making real-time control inputs impossible. Vertesi devotes an entire chapter to the planning meetings in which the mission team plotted out the rover’s path for the next day, carefully calculating how much memory would be available for image storage and when the rover would need to stop and recharge its batteries.

If the nose camera on an unmanned aircraft gets spattered with mosquitoes on a summer training sortie, my informants at U.S. Customs and Border Protection were known to land the aircraft and spray the lens down with Windex. But images taken by rover cameras had to be cleaned up after the fact, by adjusting the instrument’s results rather than the instrument itself. Part of this process was the calibration target, a small sundial with colorful bands at the corners that was affixed to the back of the rover. By taking images of the target each day and comparing them to reference values recorded back on Earth, the mission team could determine and adjust for the variable effects of local atmospheric conditions. The target, even at a distance, offered the stability of a ground truth. Yet the team’s reliance on image calibration also meant that Rover scientists knew not to fetishize the raw, unmediated image: they accepted that seeing on Mars could only be digitally mediated seeing, and so they regarded the judicious use of color correction as an indispensable part of producing visual knowledge.

Through her own work as an image calibrator and through extended observation, Vertesi was able to catalog the other transformations that rover images would undergo: false color, heightened contrast, textual annotations. As these enhancements stacked up, worries about their accuracy could creep in. Overwhelmed at one point by the glut of data superimposed on a single image, one scientist snapped at a colleague: “Can you annotate this in some way to indicate what’s interpreted and what’s real?” But the keenest insight of Seeing Like A Rover is its suggestion that the ambiguity of digital images, their latent potential to mean more than one thing, may be the key to their social efficacy. To this end, Vertesi riffs on Wittgenstein’s discussion of gestalt figures like the duck/rabbit, where it is unclear which visual elements form the background and which the foreground. Seeing the figure, alternately, as duck or as rabbit means confronting the selectivity of our visual experience and the possibility of seeing otherwise. Rover scientists used digital image processing to do something similar, advancing visual arguments by drawing images as one version of reality or another. The scientists emphasized that this was not the same as making up results. Rather, they saw themselves as amplifying elements that were already present in a digital image susceptible to infinite remixing. “One person’s artifact,” Vertesi reminds us, “is another’s data.”

There’s a funny moment toward the end of the book when Vertesi pays a visit to a researcher named Ross, who was known throughout the Rover community for his image processing skills. Vertesi asks Ross to demonstrate his vaunted decorrelation stretch technique; a little perplexed, he opens the image processing suite on his computer, loads some sample images, and explains: “I just push this button.” The distinctive greens and purples that recalled, for Vertesi, the palette of Andy Warhol were the result of a software macro applying a mathematical formula. In fact, Warhol’s critique of art’s singularity and his embrace of the readymade in some ways anticipated the relationship between Ross and his images. When Vertesi wanted to include one of Ross’s signature stretches in an article she was writing, another scientist told her not to worry about tracking down permission. In a pinch, Ross’s colleague offered, he could recreate the image on the spot and give her permission to publish that one instead. Even as the aesthetics of Ross’s technique were admired, what made his images scientific was precisely the ability to reproduce them.

Where Seeing Like A Rover falls short is in its treatment of the social forms that coalesce around images. Rather than zeroing in on moments of controversy or rancor, Vertesi chose to highlight the ongoing production of consensus within the mission team. This is well and good, and Vertesi’s astute analysis shows how team meetings were structured around ritual affirmations of “happiness,” in which members either indicated that they were happy with the plan on the table or else were drawn into further discussion. But while consensus building may be a form of politics, it ought not to be confused with “radical collectivity,” as Vertesi at one point characterizes the team’s social order. What, I found myself wondering, were the criteria by which scientists were included in or excluded from this collective? Which proposals for rover research did the NASA review panel approve, and whose proposals were never submitted at all? The politics of access do surface when Vertesi explains why, as a Canadian citizen, she was not permitted to observe the engineers who were actually controlling the rovers. Under the State Department’s International Traffic in Arms Regulations, granting that kind of access to a foreign national would have resulted in draconian penalties. Vertesi stresses that she and her informants complied with the restrictions, but in doing so she passes up an opportunity to ask why a scientific mission to Mars should have been bound up with the arms trade in the first place.

These silences in the text are more than omissions; they reflect a set of orienting assumptions about how research should be carried out. Vertesi makes those assumptions explicit in an endnote to the final chapter of Seeing Like A Rover, which situates the team’s visual work in the broader social and political context of the Rover mission:

It may seem strange to reserve the final chapter in this book for a discussion of political factors, which are likely the first thing readers think about when they consider NASA. My purpose is not to intimate that macropolitics plays only an ancillary role in Rover science, but rather to follow my actors and the scope of my laboratory field site in my analysis, concordant with my embedded perspective.

Defining the “macro” of politics in opposition to the “micro” of firsthand observation is a move often associated with the research tradition of ethnomethodology, which Vertesi sums up in terms of its concern with “the construction of social order as established through everyday, practical activities, accounts, and interactions.” Ethnomethodology is at its best when it brackets abstractions and investigates how actual people behave. Yet its explanatory power can be limited unless it cinches those daily dramas up with the world beyond them. As sociologist Karin Knorr-Cetina put it more than thirty years ago, “we will not get a grasp of whatever is the whole of the matter by a microscopic recording of face-to-face interaction. However, it may be enough to begin with if we can—for the first time—hear the macro-order tick."

We are back, then, to Wittgenstein’s duck/rabbit: Vertesi is not wrong to draw the Rover mission as a local social achievement. But the image she crafts for us in the pages of Seeing Like A Rover can and must be reprocessed so that broader concerns can become visible. It’s useful, for instance, to know that lobbying from the satellite industry recently prompted a reconsideration of the restrictions that kept Vertesi out of the rover control room. Vertesi’s book does not take up this theme, although its final chapter does open out on translocal issues by identifying the three publics to which Rover scientists understood themselves to be accountable: NASA administrators (and their bosses in Congress), other scientists not formally participating in the mission, and a general public, which included a dedicated cadre of amateurs. With these publics in mind, the mission team periodically captured images that they called “glory pans,”

even though the images had no particular scientific value. Like the landscape paintings whose formal properties they shared, these images were intended to cultivate a network of patrons whose ongoing support would sustain the mission. The highly technical visual work of the previous chapters turns out to depend on the production of what Vertesi dubs the Martian picturesque.

Unmanned pilots have a less genteel way of describing the publics to which they are accountable: not as patrons, but as customers. The term routinizes aerial imagery as just another service to be purchased, and it draws a moral equivalence between different agencies consuming that imagery: whether it’s the fire department or the CIA, one customer is as good as another. It’s this flattened, market-driven mediascape, in part, that allows digital images to circulate so promiscuously. But Seeing Like A Rover shows how slowing down that swirl of images, examining their creation and manipulation one frame at a time, can afford us a glimpse of a different world as well as a clearer view of our own. Machines can do some of our seeing, but the politics of how and where and whether we look are likely to remain all too human.