Big Data

Image via CrunchBaseBig Data: Big News
Facebook And Big Data

After reading this you appreciate your Facebook stream just a little more.

O'Reilly Radar: What is big data?

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn't fit the strictures of your database architectures. ..... cost-effective approaches have emerged to tame the volume, velocity and variability of massive data. Within this data lie valuable patterns and information ...... Today's commodity hardware, cloud architectures and open source software bring big data processing into the reach of the less well-resourced. ...... analytical use, and enabling new products ...... Being able to process every item of data in reasonable time removes the troublesome need for sampling ...... by combining a large number of signals from a user's actions and those of their friends, Facebook has been able to craft a highly personalized user experience and create a new kind of advertising business. It's no coincidence that the lion's share of ideas and tools underpinning big data have emerged from Google, Yahoo, Amazon and Facebook. ....... The emergence of big data into the enterprise brings with it a necessary counterpart: agility. Successfully exploiting the value in big data requires experimentation and exploration. ........ Input data to big data systems could be chatter from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data, the list goes on. ....... the three Vs of volume, velocity and variety are commonly used to characterize different aspects of big data. ........ Having more data beats out having better models ...... If you could run that forecast taking into account 300 factors rather than 6, could you predict demand better? ......... Many companies already have large amounts of archived data, perhaps in the form of logs, but not the capacity to process it. ...... data warehouses or databases such as Greenplum — and Apache Hadoop-based solutions ...... Apache Hadoop.. places no conditions on the structure of the data it can process. ...... First developed and released as open source by Yahoo, it implements the MapReduce approach pioneered by Google in compiling its search indexes. Hadoop's MapReduce involves distributing a dataset among multiple servers and operating on the data: the "map" stage. The partial results are then recombined: the "reduce" stage. ......... Hadoop is not itself a database or data warehouse solution, but can act as an analytical adjunct to one. ....... A MySQL database stores the core data. This is then reflected into Hadoop, where computations occur, such as creating recommendations for you based on your friends' interests. Facebook then transfers the results back into MySQL, for use in pages served to users. ............ the increasing rate at which data flows into an organization — has followed a similar pattern to that of volume. Problems previously restricted to segments of industry are now presenting themselves in a much broader setting. Specialized companies such as financial traders have long turned systems that cope with fast moving data to their advantage. Now it's our turn. ......... Online retailers are able to compile large histories of customers' every click and interaction: not just the final sales. Those who are able to quickly utilize that information, by recommending additional purchases, for instance, gain competitive advantage. The smartphone era increases again the rate of data inflow, as consumers carry with them a streaming source of geolocated imagery and audio data. ......... The importance lies in the speed of the feedback loop, taking data from input through to decision. ........ you wouldn't cross the road if all you had was a five-minute old snapshot of traffic location. ......... "streaming data," or "complex event processing." ...... when the input data are too fast to store in their entirety: in order to keep storage requirements practical some level of analysis must occur as the data streams in. ........ At the extreme end of the scale, the Large Hadron Collider at CERN generates so much data that scientists must discard the overwhelming majority of it — hoping hard they've not thrown away anything useful. The second reason to consider streaming is where the application mandates immediate response to the data. Thanks to the rise of mobile applications and online gaming this is an increasingly common situation. ........ The velocity of a system's outputs can matter too. The tighter the feedback loop, the greater the competitive advantage. ....... Rarely does data present itself in a form perfectly ordered and ready for processing. A common theme in big data systems is that the source data is diverse, and doesn't fall into neat relational structures. It could be text from social networks, image data, a raw feed directly from a sensor source. None of these things come ready for integration into an application. .......... the reality of data is messy. Different browsers send different data, users withhold information, they may be using differing software versions or vendors to communicate with you. And you can bet that if part of the process involves a human, there will be error and inconsistency. ....... Is this city London, England, or London, Texas? By the time your business logic gets to it, you don't want to be guessing. ...... a principle of big data: when you can, keep everything. There may well be useful signals in the bits you throw away. ....... documents encoded as XML are most versatile when stored in a dedicated XML store such as MarkLogic. Social network relations are graphs by nature, and graph databases such as Neo4J make operations on them simpler and more efficient. ....... a disadvantage of the relational database is the static nature of its schemas. In an agile, exploratory environment, the results of computations will evolve with the detection and extraction of more signals. Semi-structured NoSQL databases meet this need for flexibility: they provide enough structure to organize data, but do not require the exact schema of the data before storing it. ........ three forms: software-only, as an appliance or cloud-based. ...... IT is undergoing an inversion of priorities: it's the program that needs to move, not the data. .... Financial trading systems crowd into data centers to get the fastest connection to source data, because that millisecond difference in processing time equates to competitive advantage. ...... 80% of the effort involved in dealing with data is cleaning it up in the first place ...... data science, a discipline that combines math, programming and scientific instinct. ...... The art and practice of visualizing data is becoming ever more important in bridging the human-computer gap to mediate analytical insight in a meaningful way. ...... advice to businesses starting out with big data: first, decide what problem you want to solve.

Hawking

New York Times: Life and the Cosmos, Word by Painstaking Word: In the 1960s, with Sir Roger Penrose, he used mathematics to explicate the properties of black holes. In 1973, he applied Einstein’s general theory of relativity to the principles of quantum mechanics. And he showed that black holes were not completely black but could leak radiation and eventually explode and disappear, a finding that is still reverberating through physics and cosmology. ....... With a cheek muscle, he signals an electronic sensor in his eyeglasses to transmit instructions to the computer. In this way he slowly builds sentences; the computer transforms them
Image via Wikipedia into the metallic, otherworldly voice familiar to Dr. Hawking’s legion of fans. ....... It’s an exhausting and time-consuming process. Yet this is how he stays connected to the world, directing research at the Center for Theoretical Cosmology at the University of Cambridge, writing prolifically for specialists and generalists alike and lecturing to rapt audiences from France to Fiji. ........ At one point, he spoke of the special joys of scientific discovery. “I wouldn’t compare it to sex,” he said in his computerized voice, “but it lasts longer.” The audience roared. .......... despite the limitations, it was Dr. Hawking who wanted to do the interview in person rather than by e-mail. ....... Using e-mail, I can communicate with scientists all over the world. ....... I have traveled the world, from the Antarctic to zero gravity. (Pause.) Perhaps one day I will go into space. ....... you’ve said elsewhere that you think it’s a bad idea for humans to make contact with other forms of life ....... Previously I have said it would be a bad idea to contact aliens because they might be so greatly advanced compared to us, that our civilization might not survive the experience. .......... I don’t have much positive to say about motor neuron disease. But it taught me not to pity myself, because others were worse off and to get on with what I still could do. I’m happier now than before I developed the condition. I am lucky to be working in theoretical physics, one of the few areas in which disability is not
Image via Wikipediaa serious handicap. ........ My advice to other disabled people would be, concentrate on things your disability doesn’t prevent you doing well, and don’t regret the things it interferes with. Don’t be disabled in spirit, as well as physically. ...... the Large Hadron Collider ..... It will be two years before it reaches full power. When it does, it will work at energies five times greater than previous particle accelerators........ our experience has been that when we open up a new range of observations, we often find what we had not expected ...... I had not expected “A Brief History of Time” to be a best seller. It was my first popular book and aroused a great deal of interest. ........ I entered the health care debate in response to a statement in the United States press in summer 2009 which claimed the National Health Service in Great Britain would have killed me off, were I a British citizen. I felt compelled to make a statement to explain the error. ....... I am British, I live in Cambridge, England, and the National Health Service has taken great care of me for over 40 years. I have received excellent medical attention in Britain, and I felt it was important to set the record straight. I believe in universal health care. And I am not afraid to say so. ........ the human spirit is capable of enduring terrible hardships. ....... I would go back to 1967, and the birth of my first child, Robert. My three children have brought me great joy. ....... (After five minutes.) I hope my experience will help other people.