Big Data: An Interview with Kenneth Cukier and Viktor Mayer-Schonberger

Kenneth Cukier and Viktor Mayer-Schonberger, authors of the new book Big Data: A Revolution That Will Transform How We Live, Work, and Think, published last month, answered several questions on big data, foreign policy, and China. Questions by Sharone Tobias.

As a foreign policy think tank, what should big data mean to the Council on Foreign Relations? Should U.S. policymakers use big data in their decision-making processes? Are other governments already doing so?

Kenneth Cukier: Much of the work of international relations is based on anecdote, sentiment and judgment--- "qualities," not "quantities." We make assessments and forecasts based on our instincts and experience, and even vaunt this as our most cherished value. We prize this almost romantic notion of the diplomatic arts---we swoon over an intellectual analysis of events and people to produce a wise response in an uncertain and dangerous world. Think Kennan quoting Gibbon to understand Soviet intentions in the Long Telegram. There’s absolutely a place for this. But we ought use more tools in the toolkit---because we now can. We need to tap data, and use our discrimination alongside it. In the past, harnessing data largely wasn’t possible so we usually didn’t. But that constraint no longer exists to anything like the same extent: the data can be collected and processed at low cost. So we’d be foolish and recklessly sentimental if we didn’t add a data-driven component to our understanding and practice of foreign policy.

One interesting area where this is happening already is overseas aid. A group called AidData---backed by USAID, the College of William & Mary and Brigham Young University---scrutinizes some five trillion dollars in aid from almost 100 donor agencies to see that it is spent well. In one case study, the World Bank mapped their aid programs in Kenya to where the funds were physically spent. It discovered that the majority of programs were in the southwest, when the worst poverty is in the country’s northeast---suggesting that the bank may want to reassess its projects. As another example, there is a lot of research in the U.S. and elsewhere to use vast streams of data such as news stories and social media to predict where and when conflicts may occur. It’s even given birth to one start-up company, Recorded Future. Yes, it sounds like science fiction. But why wouldn’t policymakers tap this sort of information alongside their talent, experience and wisdom in making decisions, just as pilots use computer instrumentation to land planes as well as their own training and skill?

As for whether big data is being used today in foreign ministries around the world. If only! It is hard enough to get corporate boards to get their minds around the changes. But it’s starting to happen. In the U.S., there are officials in many agencies that understand the potential and are leading the way. Some bright lads in Britain’s foreign office are also thinking about this. We’re just at the start.

Viktor’s trip to China a few months ago was covered widely by the Chinese press. Big data revenues in China were estimated to be 310 million RMB ($49.6 million) in 2012, and Chinese companies such as Alibaba and Tencent have extraordinary potential to utilize data from hundreds of millions of consumers. What are some of the major innovations have you seen coming out of China and especially from these companies?

Viktor Mayer-Schonberger: It is still very early days with big data in China. What is remarkable however is that so many of both the very big players such as Alibaba, and the small entrepreneurial startups see big data as a huge opportunity for China. They point at (a) scale---in people and thus in data, and (b) investments in cloud infrastructure so that processing and storage prices are envisioned to plummet much like they did in the U.S. recently. And they are aware that many of the challenges of a fast growing large country, such as the build-out of physical infrastructures such as roads, rails, the electrical grid, etc., or pollution or corruption can be better tackled using big data. So there is a hope that big data can turn into a societal force of progress, not just into a differentiator among competing IT companies.

Do you think that the United States is at the forefront of big data innovation? Will it continue to be? What other countries are leading the way in new technology in this field?

KNC: Yes, the U.S. is at the forefront of big data. But that’s just because it tends to be at the forefront of most technology trends. And this says less about America’s inherent strengths and more about its vibrant educational, business and financial culture that encourages people to flit between disciplines and sectors and start businesses. Consider: Japan and France produce some of the best mathematicians and scientists, but just try seeing what happens if a PhD in engineering tries to publish in sociology or teach a course in the field. Cultures that are porous by allowing skills in one domain to apply to another will excel at big data, since this is essential. Many of the pioneers of big data have these multifaceted backgrounds: LinkedIn’s former analytics guru DJ Patil began his career crunching numbers to uncover biological weapons in central Asia for the Department of Defense. And a subtle nod is in order for Duncan Watts, an Australian physicist and former Navy officer who left his professorship at Columbia University---in Sociology---to join industry, eventually landing at Microsoft Research in New York.

So for the moment, the U.S. has a very healthy lead. But it is not inherently sustainable. Big data companies are cropping up everywhere. Britain has many, unsurprisingly. France and Germany are getting into the game---they have the skills and they have the companies that sit on oodles of data. Yet the most interesting place plowing into big data is China. We haven’t seen major businesses or research papers come out of China yet. But as one indicator, our book "Big Data" has been a best-seller there for months: the country sees big data as a way to promote national greatness and lead a key technology trend unlike in the past, when it had to be a follower for computing and the Internet. I wouldn’t be surprised if Japan and South Korea also flexed their big data muscles. At the end of the day, an important requirement for success is advanced math skills, where Asian students score very well on international comparisons.

The internet today is balkanized, with Chinese users using their own version of Google (Baidu), Facebook (Renren), Twitter (Weibo), and much more. Will this create two major big data industries (the U.S. and China) instead of a consolidated one?

VMS: It is more complicated than that. It is not just the U.S. and China---Russia and Brazil for instance also have very large and very popular homegrown Internet platform providers. In a way the Internet has been balkanized for a long time, likely since its inception. There is a push towards concentration, but innovation also creates new niche opportunities for new players. Before Facebook everyone was talking about the dominance of Google for instance; then Facebook came along.

In addition there are geographic and cultural differences that create significant market opportunities that cannot be captured by the big U.S. players. So until the big data economy matures, we’ll see many players surface, some of them becoming quite big, only to be superseded by an innovative new startup. This means multiple big and many, many small but successful players in multiple regions and geographies throughout the world.

Your overall outlook on the book is very optimistic. However, it focuses primarily on industry in the developed world. Looking at the developing world and more authoritarian states like China, do you see big data enabling these countries to exert more control over their populations?

KNC: Yes. But I don’t see it so much as a function of big data as a regrettable part of authoritarianism that exists in every other dimension, including the uses of technology. It would be very pleasing-sounding to say that, actually, big data might winnow the information asymmetries between the governors and the governed, and weaken the grip that authoritarian states have over their populations. But that strikes me as Pollyannish.

The issue raises essential questions. Does big data exacerbate things? Quite possibly, yes. Should we be concerned? Yes, very. Is there a risk that even in Western democracies that this technology may be abused to erode people’s freedom? Only someone ignorant of history, even very modern history, could answer in the negative. So the key issue will be having a serious societal conversation about this and establishing robust safeguards to guard the guardians. But here too, I deliberately reach for a classic image to stress that this is not a new problem but a timeless one.

Most of the world is not connected to the Internet, including hundreds of millions of Chinese, Indians, and others in the developing world. Do you foresee opportunities in big data analysis’ ability to reach this huge segment of the population despite this fact? Have you seen any big companies in China, India, or elsewhere making a big push on this front?

KNC: I don’t see a way that big data can be especially useful to bridge the digital divide. Perhaps it’s my lack of imagination, and I’d be pleased to be proved wrong. There are marginal effects that move things in the right direction. For example, better analysis of cell phone usage in developing countries will allow wireless operators to better engineer their networks and ensure there is adequate radio spectrum where it is most needed. But again, this is an indirect improvement to overcome the digital divide.

Likewise, the ability to do mobile health applications just as we do mobile-banking in developing countries will improve the lives of people in China, India, Africa and elsewhere. This may help them adopt the technologies faster than otherwise and narrow the digital divide. But it’s not inherent to big data per se. Ultimately, big data will do lots of things, but there are lots of things it won’t or can’t do. Reducing some of the "divides" that societies face---sanitation, health, education, and so on---is one. To be sure, big data will help in all those areas. But it’s not a silver bullet to rid the world of its inherent inequalities. The peaceable kingdom, the city on a hill, is not big data.