Living by the Numbers: Big Data Knows What Your Future Holds
Forget Big Brother. Companies and countries are discovering that algorithms programmed to scour vast quantities of data can be much more powerful. They can predict your next purchase, forecast car thefts and maybe even help cure cancer. But there is a down side.
On balmy spring evenings, Hamburg's Köhlbrand Bridge offers an idyllic postcard view of the city's harbor. The Elbe River shimmers in the reddish glow of sunset, forklifts, cranes and trucks seem to move in slow motion, and occasionally a container ship glides by. But from the standpoint of Sebastian Saxe, the area is primarily an equation with many variables. For the past four-and-a-half years, the 57-year-old mathematician has been working on his trickiest computing task yet at the behest of the company that manages the Hamburg port.
The port covers an area of 7,200 hectares (about 28 square miles). Roughly 200 trains a day traverse its 300-kilometer (186-mile) network of rails and its 130 bridges to transport goods that have arrived by ship. Saxe, as chief information officer of the Hamburg Port Authority (HPA), faces the enormous task of optimizing this logistical nightmare.
The amount of land is finite, and further expansion is not possible. Nevertheless, the Hamburg Senate has announced its goal of almost tripling container transshipment volumes in the city by 2025. This will only work if Saxe and his 60-member IT team manage to optimally exploit another resource: data. He certainly has plenty of it.
The port is already filled with sensors today. Trucks and freight trains are constantly transmitting their positions while incoming container ships report their location and speed. Sensors that constantly monitor port traffic are built into the Köhlbrand Bridge.
"Our goal is a totally interconnected, intelligent port, a Smart-port," says Saxe. He envisions a port in which, for example, a railroad drawbridge would no longer open at specific times, but rather just before a ship actually reaches it. This eliminates unnecessary delays for the railroad and at the terminal. Even the Köhlbrand Bridge would become "intelligent," in that it would report its current condition and predict future maintenance needs, all through the use of sensors. The frequency of scheduled maintenance dates was recently increased because significantly larger numbers of heavy trucks were crossing the bridge than had been planned for. This was of interest to Saxe and the HPA, but also to police and the customs agency, because some of the trucks were carrying illegal loads.
In the end, the complex harbor logistics will create a machine that controls itself. Saxe's vision of the future is a sort of port exchange, allowing shipping companies to predict, down to the minute, how quickly their containers will be moved from the water to the road.
Many other companies worldwide are in the same position as the HPA. They are rediscovering a raw material that they, their facilities and their customers produce in excess every day: data.
The expression "Big Brother" has become dated. Experts would seem to have reached consensus on the term "Big Data" to describe the new favorite topic of discussion in boardrooms, at conventions like Berlin's re:publica last week, and in a number of new books. Big Data promises both total control and the logical management of our future in all aspects of life. Authors like Oxford Professor Victor Mayer-Schönberger are calling it a "revolution." According to Mayer-Schönberger, Big Data, which is also the title of his current book on the subject, will change our working environment and even the way we think.
The most important factor is not the sheer volume of data, even though it is currently growing faster than ever. An estimated 2.8 zettabytes of data were created in 2012. One zettabyte is 1,000,000,000,000,000,000 kilobytes. Experts predict that the volume of new data could increase to 40 zettabytes by 2020. It would take about 250 million DVDs to store the amount of data being transmitted on the Internet in a single day. This volume doubles about once every two years.
New is the way companies, government agencies and scientists are now beginning to interpret and analyze their data resources. Because storage space costs almost nothing nowadays, computers, which are getting faster and faster, can link and correlate a wide variety of data around the clock. Algorithms are what create order from this chaos. They dig through, discovering previously unknown patterns and promptly revealing new relationships, insights and business models.
Though the term Big Data means very little to most people, the power of algorithms is already everywhere. Credit card companies can quickly recognize unusual usage patterns, and hence automatically warn cardholders when large sums are suddenly being charged to their cards in places where they have never been. Energy companies use weather data analyses to pinpoint the ideal locations for wind turbines down to the last meter. According to official figures, since the Swedish capital Stockholm began using algorithms to manage traffic, drive times through the city's downtown area have been cut in half and emissions reduced by 10 percent. Online merchants have recently started using the analyses to optimize their selling strategies. The widespread phrase "Customers who bought this item also bought " is only one example of the approach.
Turning Data into Dollars
Google and Facebook are pure, unadulterated Big Data. Their business models are based on collecting, analyzing and marketing information about their users, through advertising tailored as closely as possible to the individual. This gigantic database and the notion of what can be done with more than a billion individual profiles in the age of Big Data was worth at least $100 billion (78 billion) to Facebook investors.
The prospect of turning their treasure troves of data into dollars is now fueling the fantasies of businesses in many industries, from supermarkets to the automobile industry, and from aviation to banks and insurance companies. According to figures published by industry association Bitkom, global sales related to Big Data applications amounted to 4.6 billion in 2012. That number is expected to increase to about 16 billion by 2016.
Countless Big Data applications are also being tested in medicine and science. Even the public sector, especially police departments and security agencies, not always the most progressive when it comes to IT, have recognized the potential benefits in their fields.
What captivates so many people is the promise of gazing into the future, thanks to the lightning speed at which massive amounts of data can be analyzed. In fact, algorithms allow for astonishingly precise predictions of human behavior, be it in front of supermarket shelves, in traffic or when it comes to credit-card payment patterns.
In 2010, Google predicted a wave of flu outbreaks on the basis of user searches. American data specialist Nate Silver predicted the outcome of the last US presidential election well in advance and more precisely than all demographers.
'The End of Chance'
Some cities even predict the probability of crimes in certain neighborhoods. The method, known as "predictive policing," seems like something straight out of a Hollywood film, and in fact it is. In Steven Spielberg's "Minority Report," perpetrators were arrested for crimes they hadn't even committed yet.
Finding the presumed delinquents also doesn't seem to present a problem. Scientists have figured out that, with the help of our mobile phone geolocation and address book data, they can predict with some certainty where we will be tomorrow or at a certain time a year from now.
The increasing accuracy of such forecasts have led American tech guru Chris Anderson to proclaim that we are arriving at the "end of theory." Austrian media executive Rudi Klausnitzer, who has just written a book on the subject called "Das Ende des Zufalls" ("The End of Chance"), has reached a similar conclusion.
It is a prospect that is not altogether appealing to some. But many already rely on the prognostic ability of soulless algorithms in the most intimate spheres of life. The extensive questionnaires used by online dating agencies are fed into algorithms designed to increase the probability of finding a compatible partner.
A gold rush of sorts is taking shape in companies, research laboratories and some government agencies. In many places, the mantra of data is extolled as the new "oil" or "gold" of the 21st century. Some people are already benefiting financially: statisticians, physicists and so-called data scientists or data miners, who advise companies on Big Data applications. As with the classic American gold rush in the 19th century, most of the money is being made by those who sell equipment, tools and expertise, Big Data specialists like Blue Yonder, a company with 85 employees.
© SPIEGEL ONLINE 2013
All Rights Reserved
Reproduction only allowed with the permission of SPIEGELnet GmbH
Click on the links below for more information about DER SPIEGEL's history, how to subscribe or purchase the latest issue of the German-language edition in print or digital form or how to obtain rights to reprint SPIEGEL articles.
- Frequently Asked Questions: Everything You Need to Know about DER SPIEGEL
- Six Decades of Quality Journalism: The History of DER SPIEGEL
- A New Home in HafenCity: SPIEGEL's New Hamburg HQ
- Reprints: How To License SPIEGEL Articles