Shocked by the NSA revelations? You don’t know the whole story

A review of Big Data: A Revolution That Will Transform How We Live, Work, and Think, by Viktor Mayer-Schoeneberger and Kenneth Cukier

@@@@ (4 out of 5)

While Edward Snowden bounces from one temporary refuge to another in search of safe harbor from the long arms of the U.S. government, the American public is starting to wake up to the reality of Big Data. The National Security Agency, long one of the pioneers in this burgeoning but little-appreciated field, has been teaching us — or, rather, Snowden, The Guardian, and the Washington Post have been teaching us — about the power that resides in gargantuan masses of data. Now here come Viktor Mayer-Schoeneberger and Kenneth Cukier with a new book that goes far beyond the headlines about espionage and invasion of privacy to give us an eminently readable, well-organized overview of Big Data’s origins, its characteristics, and its potential for both good and evil.

When we think of Big Data, we, or at least most of us, think of computers. However, the authors persuade us that the fundamentals of Big Data were laid down more than a century before the invention of the microprocessor. They point to a legendary American seaman named Matthew Maury. In the middle of the 19th Century, after 16 years of effort, Maury published a book based on 1.2 billion data points gleaned from old ships’ logs stored by the Navy that dramatically reduced the distances (and, hence, the time elapsed) in ocean voyages by both military and commercial ships. Maury used facts derived from decades of mariners’ observations to dispel the myths, legends, superstitions, and rumors that had long caused ocean-voyaging ships to pursue roundabout courses. Not so incidentally, Maury’s work also facilitated the laying of the first transatlantic telegraph cable.

If not the first, this was certainly an early application of Big Data, which the authors describe as follows: “big data refers to things one can do at a large scale that cannot be done at a smaller one, to extract new insights or create new forms of value, in ways that change markets, organizations, the relationship between citizens and governments, and more.” For example, if Maury had had available only a fraction of the old ships’ logs he found in the naval archives, his task would have been impractical, since each individual log doubtless included small errors (and an occasional big one). Only by amassing a huge store of data did those errors cancel out one another.

Now, in the Digital Age, the volumes of data that can be harnessed are, at times, literally astronomical. “Google processes more than 24 petabytes of data per day, a volume that is thousands of times the quantity of all printed material in the U.S. Library of Congress.” AT&T transfers about 30 petabytes of data through its networks each day. Twenty-four or 30 of something doesn’t sound like much, unless you understand that a megabyte is a million bytes, a gigabyte is a billion bytes, a terabyte is 1,000 times the size of a gigabyte, and a petabyte is 1,000 times the size of a terabyte. That’s 1,000,000,000,000,000 bytes. That’s a lot of data! But even that’s only a tiny slice of all the data now stored in the world, “estimated to be around 1,200 exabytes.” And an exabyte (I’m sure you’re dying to know) is the equivalent of 1,000 petabytes. So, 1,200 petabytes could also be stated as 1.2 zettabytes, with a zettabyte equal to 1,000 petabytes, and I’ll bet that not one person in a million has ever heard of a zettabyte before. Had you?

All of which should make clear that when we talk about Big Data today, we’re talking about really, really big numbers — so big, in fact, that almost no matter how messy or inaccurate the data might be, it’s usually possible to draw useful, on-target insights from analyzing it. That’s what’s different about Big Data — and that’s why the phenomenon is bound to change the way we think about the world.

We live in a society obsessed with causality. We often care more about why something happened than about what it was that happened. And in a world where Big Data looms larger and larger all the time, we’ll have to get used to not knowing — or even caring much — why things happen.

“At its core,” write Mayer-Schoeneberger and Cukier, “big data is about predictions. Though it is described as part of the branch of computer science called artificial intelligence, and more specifically, an area called machine learning, this characterization is misleading. Big data is not about trying to ‘teach’ a computer to ‘think’ like humans. Instead, it’s about applying math to huge quantities of data in order to infer probabilities: the likelihood that an email message is spam; that the typed letters ‘teh’ are supposed to be ‘the’; that the trajectory and velocity of a person jaywalking mean he’ll make it across the street in time [so that] the self-driving car need only slow slightly.”

The authors refer to data as “the oil of the information economy,” predicting that, as it flows into all the nooks and crannies of our society, it will bring about “three major shifts of mindset that are interlinked and hence reinforce one another.” First among these is our ever-growing ability to analyze inconceivably large amounts of data and not have to settle for sampling. Second, we’ll come to accept the inevitable messiness in huge stores of data and learn not to insist on precision in reporting. Third, and last, we’ll get used to accepting correlations rather than causality. “The ideal of identifying causal mechanisms is a self-congratulatory illusion; big data overturns this,” the authors assert.

If you want to understand this increasingly important aspect of contemporary life, I suggest you read Big Data.

Viktor Mayer-Schoeneberger and Kenneth Cukier come to the task of writing this book with unbeatable credentials. Mayer-Schoeneberger is Professor of Internet Governance at Oxford University, and Kenneth Cukier is Data Editor at The Economist.

Surprised by the news about NSA surveillance? Read this book!

A review of Top Secret America: The Rise of the New American Security State, by Dana Priest and William M. Arkin

@@@@@ (5 out of 5)

Note: This review first appeared here on September 11, 2011 (yes, 9/11/11). In view of the recent news about the NSA’s Prism program and other widespread and long-standing efforts to amass personal information about the American public, I’m posting it again. This superb book deserves a far wider audience than it received in 2011.

If you treasure your freedom as an American . . . if you’re concerned about how the U.S. Government spends your tax money . . . or if you simply want to understand how our country is managed . . . you owe it to yourself to read this brilliant book. Alternately mind-boggling and blood-curdling, Top Secret America is the most impressive piece of investigative journalism I’ve read in years. Dana Priest and Bill Arkin have written a book that, in a rational world, would usher in an orgy of housecleaning through the far reaches of the Pentagon, the CIA, the NSA, the FBI, the Department of Homeland Security, and every other department, agency, or office that pretends to be involved in strengthening our national security.

Even then — even if we somehow reined in the known alphabet agencies — we would only be scratching the surface. Here’s Priest writing about the work of her co-author: “After two years of investigating, Arkin had come up with a jaw-dropping 1,074 federal government organizations and nearly two thousand private companies involved with programs related to counterterrorism, homeland security, and intelligence in at least 17,000 locations across the United States — all of them working at the top secret classification level.” There is an additional three thousand “state and local organizations, each with its own counterterrorism responsibilities and jurisdictions.”

Perhaps there’s one saving grace in this brouhaha of activity. Priest again: “Post 9/11, government agencies annually published some 50,000 separate serialized intelligence reports under 1,500 titles, the classified equivalent of newspapers, magazines, and journals. Some were distributed daily; others came out once a week, monthly, or annually.” There is so much “information” generated by the counterterrorism establishment that senior managers frequently ignore it all and instead ask their aides to talk to people to find out what’s really meaningful.

Don’t be mollified by the belief that all this activity is carried out by designated intelligence agencies. The nation’s warriors have their own alphabet-soup of agencies, departments, and units devoted to the same ends. The Pentagon created a major new entity called the Northern Command headed by a four-star general (the military’s highest rank) to protect the “homeland.” However, the Northern Command has no troops of its own and, to take any action, must ask permission from the leaders of each state’s National Guard and other agencies on whom it depends for personnel.

Priest and Arkin clearly take a dim view of all this:

• Many, if not all, of the Federal Government’s most closely guarded secrets are vulnerable to theft through simple file-sharing software installed on 20 million computers.
• The Director of National Intelligence, a new position created in 2004 to coordinate the work of the 16 major U.S. intellgence agencies, possesses no power to do so and is frequently ignored by them. But his staff numbers in the thousands, and they hold forth from a new, 500,000-square foot office building.
• The degree of duplication in the national security world is chilling. “Each large organization [engaged in counterterrorism] started its own training centers, supply depots, and transportation infrastructure. Each agency and subagency manned its own unit for hiding the identities of undercover employees and for creating cover names and addresses for them and for their most sensitive projects. Each ecosystem developed a set of regional and local offices.”
• Duplication of effort runs so deep that there are three separate lists of “High Value Targets,” one each for the CIA, the Pentagon, and the super-secret Joint Special Operations Command (the people who killed Bin Laden). And “at least thirty-four major federal agencies and military commands, operating in sixteen U.S. cities, tracked the money flow to and from terrorist networks.”

The depth and quality of Priest and Arkin’s research is unexcelled, and their writing is brisk and easy to read. The book benefits from the straightforward, first-person approach Priest adopted. It’s written largely from her point of view, with Arkin’s contributions as a researcher noted in the third person.

Dana Priest has reported for the Washington Post for more than 20 years. She won the George Polk Award in 2005 for reporting on secret CIA detention facilities and the Pulitzer Prize in 2006 for uncovering black sites prisons. Her exposure of the deplorable conditions at Walter Reed Army Hospital helped the Washington Post win another Pulitzer in 2007. She deserves another Pulitzer for this illuminating book.

Bill Arkin served in U.S. Army intelligence in 1974 to 1978 and had worked as a consultant, political commentator, blogger, activist, and researcher for a number of progressive organizations before teaming up with Priest to write the widely-acclaimed series of Washington Post articles on which this book was based.

Before Silicon Valley, Bell Labs was America’s hub of innovation

A review of The Idea Factory: Bell Labs and the Great Age of American Innovation, by Jon Gertner

@@@@ (4 out of 5)

Ask yourself why the United States of America has remained the dominant economic and military power on the planet for nearly a century now. Is it the superior universal public education system we used to brag about? Is it the wealth of our natural resources: millions of acres of rich, arable land and bountiful mineral and petroleum wealth? Is it the peculiar American ability to build and manage efficient large enterprises? Is it the size and the demographic richness of our population, constantly renewed by the influx of resourceful people from other lands and cultures?

Jingoistic rhetoric aside, it’s most likely that your list of reasons — even, possibly, your only reason — is “American know-how,” the homegrown phrase that points to what seems an unusual national talent for creative thinking and innovation. In fact, it’s difficult to overlook the disproportionate presence of the United States on the lists of Nobel Prizewinners, industrial patents, and other markers of forward thinking in science and engineering throughout much of the 20th Century.

In The Idea Factory, Jon Gertner examines one period and one place where the evidence of American know-how was most pronounced: the time from the end of World War II to the late 1970s in Murray Hill, New Jersey, where AT&T’s Bell Laboratories was headquartered. There, an extraordinary assemblage of brilliant scientists and engineers, guided by a succession of equally brilliant managers, invented or developed into practical form the fundamental advances in science and technology that have shaped the world we live in today: the transistor, the laser, quality assurance methods, communications satellites, mobile telephony, digital photography, fiber-optic communications, and a number of much less well-known but equally important technological advances as well as a long list of innovations in weaponry and spy technology that many of us would prefer not to know about. (In fact, the relationship of Bell Labs to the Pentagon, especially its National Security Agency, remained close throughout the period studied in this book.)

It’s difficult to exaggerate the impact of the work at Murray Hill and its outlying sites following World War II. The transistor — the brainchild of three Bell scientists, John Bardeen, Walter Brattain, and William Shockley — is frequently cited as the single most important invention of the century. Certainly, the transistor lies at the heart of all things digital today. Even more fundamental to the world we inhabit is the information theory of Claude Shannon, who explained how computers might communicate with one another long before anything resembling today’s computers existed.

As Gertner explains in great detail, most of Bell Labs’ work was carried out in service of the growing AT&T telephone network. (If you’re young enough to confuse that AT&T with today’s business of the same name, be advised that AT&T was America’s government-regulated telephone monopoly from the 1920s through the 1970s.) Those familiar with the network called it the biggest and most complex machine in the world. “The system’s problems and needs were so vast that it was hard to know where to begin explaining them,” Gertner writes. “The system required that teams of chemists spend their entire lives trying to invent new, cheaper sheathing so that phone cables would not be permeated by rain and ice; the system required that other teams of chemists spend their lives working to improve the insulation that lay between the sheathing and the phone wires themselves. Engineers schooled in electronics, meanwhile, studied echoes, delays, distortion, feedback, and a host of other problems in the hope of inventing strategies, or new circuits, to somehow circumvent them.”

Gertner makes absolutely clear, however, that “this book does not focus on those tens of thousands of Bell Laboratories workers. Instead, it looks primarily at the lives of a select and representative few,” chiefly scientist-managers Mervin Kelly, Jim Fisk, and William Baker and scientists John Pierce and William Shockley. Every one of these individuals was exceptional, and Gertner does an excellent job giving us glimpses of their eccentricities and missteps as well as their extraordinary lives and character and their accomplishments.

I can fault this exhaustive study in only one way: it’s exhausting, expecially in its concluding chapters, where Gertner spends far too many pages dwelling on the eulogies offered up by the managers who ran Bell Labs when it was alive and well, before the break-up of the old AT&T that was consummated in 1983.