Data Science FAQ

Not exactly. All, and I mean all of its insights come straight out of academic statistics, computer science, and dropout hacker lore. But just because something's made of other things doesn't mean it isn't real, that it isn't a valuable permutation.

(I grant you that only half of the excitement is due to these methods, the other half being a truckful of marketing hype.)

"Data science" is a silly name for a few reasons: because all statisticians apply scientific methods to data, and have done for a hundred years; because all programming is programming on data; because the actual job has much more data munging and rote scripting in it than it has experimental science.

Distinguishing analysts and data scientists

1. The underlying cause of most of it is the data deluge - we collect millions of times more data, basically for free. This deluge leads to all the distinctive features:
   
   • Distributed computing, because the dataset doesn't fit in one computer's RAM.
   
   
   • Sudden great performance, because it turns out that scaling your sample size gives a qualitative change for some domains (“Unreasonable Effectiveness of Data”). Same algorithms as the 60s sometimes!
   
   
   • Unprecedented aggregations, means a market in data. Huge potential value in scraping and extracting unstructured sources. Acxiom is a $1bn company, for just one example; though Google is probably the pre-eminent example.
   
   
   • Data products (FitBit; restaurants, friends, jobs; credit ratings; health)
   
   (And the web and IoT are the cause of that.)


2. Unsupervised machine learning allows for cheap use of the much greater volume of data which isn't annotated with a neat output label (and so can't be used for supervised training).


3. Cognitive automation: we have recently been able to automate high-level things which used to require skilled humans: trading decisions, media recommendations, fine art, maybe even hypothesis generation: i.e. creativity!


4. Demand for real-time summary of these new datasets means a sudden investment in stream processing algorithms and frameworks (which were previously not a very efficient use of industrial computer scientists).

We can define a data scientist in a black-box manner, as follows:

• learner: an inductive program that takes in example data and outputs a model.
• model: a static program that takes examples and outputs predictions.
• data scientist: a pretentious inductive program that takes in data and a vague problem, and outputs a learner and some good hyperparameters. 9

That ^ is vague; here's a breakdown of the actual tasks involved:

Really your job is not coding or being able to read formulae or knowing where to find data without any anti-scraping measures: your job to optimise the difficult tradeoffs of using computers to optimise things. You're a computer economist, not much of a scientist.

• Bias / Variance. After a point, you can minimise underfitting or overfitting, not both.


• Accuracy / Interpretability. For some tasks, you can't have optimal performance while still being able to comprehend the final weights.


• Easy Training / Optimality. An instance of The Iron Triangle: 'You can pick two of: fast, cheap, optimal'.


• Space / time. You can use more data or spend less runtime.


• Natural signal / Balanced classes. In classification, you often want to train the model to recognise differing classes equally well, even if one is rarer. There are a few ways to do this, but one of them is just to use fewer examples of the common class.


• Full signal / Validation. You can train on all the available data or you can know if what you're doing is any good. This one we can defeat, with the bootstrap.

Hard to beat the exhaustive list in Harris et al (2013). (Though it covers the things you'd expect a whole data science team to have: )

• Classical statistics (general linear model, ANOVA)
• Bayesian statistics (MCMC, BUGS)
• Simulation (discrete, agent-based, continuous)
• Spatial statistics (geographic covariates, GIS)
• Temporal statistics (forecasting, time-series analysis)
• Surveys and Marketing (multinomial modeling)


• Visualization (statistical graphics, mapping, web-based data‐viz)


• Distributed data (Hadoop, MapReduce)
• Structured data (SQL, JSON, XML)
• Unstructured data (noSQL, text mining)
• Data manipulation (regexes, R, SAS, web scraping)
• Systems administration (*nix, DBA, cloud tech.)
• Frontend programming (JavaScript, HTML, CSS)
• Backend programming (Java/Rails/Objective C)


• Algorithms (computational complexity, CS theory)
• Graphical models (social networks, Bayes networks)
• Machine learning (decision trees, neural nets, SVM, clustering)
• Analysis (linear algebra, real analysis, calculus)
• Optimization (linear, integer, convex, global)
• Science (experimental design, technical writing)


• Business (management, business development, budgeting)
• Product development (design, project management)

Similar roles have been played in the past by jobs like expert system designer, decision support developer, knowledge discovery engineer, predictive analytics guy, business intelligence analyst, data miner, and big data engineer, with varying levels of rigour and bullshit. The quant developer, in finance, is basically the same thing but with far less variance in skill level.

A more honest name for data scientist would be analytics developer, but hey we all got to make a living. The best you can say for us is, at least we don't claim that our poky little logistic regressor is An Artificial Intelligence. (...)

By surveying people with the job title "data scientist" and then clustering the data, Harris et al. split "data scientist" into four real jobs:

• Data researcher: A postgrad statistician. Writes new algorithms, builds proof-of-concept ensembles, writes analysis libraries, disseminates.


• Data developer: Machine learning engineer. Application builder.


• Data businessperson: A knowledgeable interface between technical and non-technical people. Doesn’t code: steers modelling approach.


• Data creative: Bit of everything. Heavy on visualisations? 6

Many, many articles act as if all DS jobs were data researcher jobs; this is the root of the constant HN catfighting about whether you need to know e.g. postgrad linear algebra before starting, let alone before applying.

Answer: data researchers obviously need to; the others, not so much. The ineliminable need is for decent coding skills. Maybe people take that need as too obvious to mention, but I've seen actual DS job applications which fail to mention programming once.

The 2013 O'Reilly survey of self-described data scientists found 70% of respondents holding a Master's. 5 Unless you are highly skilled already, or unless your target company are enlightened, it could be hard. You should be able to start as an analyst and work up though.

A mathematical structure that approximates a part of the world. (Hopefully the part that caused your data.) Writing a model out in equations or programs can make it completely precise (and this allows us to computerise it). A model summarises, simplifies, unifies, and guides. Occasionally it surpasses.

Models are intentionally fake and smooth toys. This is because adding too much detail takes too much time and computing to be useful, and makes it impossible to make general claims, and actually prevents you from understanding the thing at hand. ("Can't see the wood for the trees.")

A model's abstraction allows us to see the unity of seemingly unrelated problems (for instance, magnetisation, the changes in states of matter, and urban riots are all explained by just one kind of model, criticality theory). The brain of the C. elegans worm, the US power grid, and the film collaborations are all very well described by one kind of mathematical object, a "small-world network". And many phenomena fall under the same "power law" distribution and can be said to share a deep structure.

A statistical model is one which admits that it's not going to get the exact right answer every time (or any time) but which offers a good approximation to an uncertain world, by giving a weighted spread of possible values for a given input.

In general, even when we call our model "highly predictive", we do not predict the future; rather we're reducing how wrong we are about the future.

Because

• it is always incomplete (small samples, few features, physical limits)
• it is usually an indirect reflection of the thing of interest. (proxies, latency)
• it is noisy (measurement error, data corruption, unknown processes)
• it always has some risk of being fabricated.
• it is often ambiguous.
• of the fundamental nature of inductive thought: you can never be sure via sampling.

In a great paper reflecting on the Great Recession, Lo and Mueller give a taxonomy of uncertainty (in the course of arguing that economists mistook themselves as having physicists' luck, a tractable domain:

1. "Complete certainty". You are in a Newtonian clockwork universe with no residuals, no observer effects, utterly stable parameters. So, given perfect information, you yield perfect predictions.


2. "Risk without uncertainty". You know a probability distribution for an exhaustive set of outcomes. No statistical inference needed. This is life in a hypothetical honest casino, where the rules are transparent and always followed. This situation bears little resemblance to financial markets.


3. "Fully Reducible Uncertainty". There is one probability distribution over a set of known outcomes, but parameters are unknown. Like an honest casino, but one in which the odds are not posted and must therefore be inferred from experience. In broader terms, fully reducible uncertainty describes a world in which a single model generates all outcomes, and this model is parameterized by a finite number of unknown parameters that do not change over time and which can be estimated with an arbitrary degree of precision given enough data. As sample size increases, classical inference brings this level down to level 2.


4. "Partially Reducible Uncertainty". The distribution generating the data changes too frequently or is too complex to be estimated, or it consists in several nonperiodic regimes. Statistical inference cannot ever reduce this uncertainty to risk. Four sources:
   
   

   (1) stochastic or time-varying parameters that vary too frequently to be estimated accurately;
   (2) nonlinearities too complex to be captured by existing models, techniques, and datasets;
   (3) non-stationarities and non-ergodicities that render useless the Law of Large Numbers, Central Limit Theorem, and other methods of statistical inference and approximation;
   and (4) the dependence on relevant but unknown and unknowable conditioning information.

   At this level of uncertainty, modeling philosophies and objectives in economics and finance begin to deviate significantly from those of the physical sciences... model-building in the social sciences should be much less informed by mathematical aesthetics, and much more by pragmatism in the face of partially reducible uncertainty.


5. "Irreducible uncertainty". Ignorance so complete that it cannot be reduced using data: no distribution, so no success in risk management. Such uncertainty is beyond the reach of probabilistic reasoning, statistical inference, and any meaningful quantification. This type of uncertainty is the domain of philosophers and religious leaders, who focus on not only the unknown, but the unknowable.

We can model well when we are in situations 1 to 3; and we can model a bit, defensively, in situation 4.

• summarise data
• predict new data
• simulate reality

Note that these three aims are actually super-sets of each other:

To predict new data is to use a model to summarise future data. (Since a model is also a compressed description of a dataset.)

To simulate reality requires you to infer actual structure and true parameters: to infer these is to predict that future data will confirm your estimates of them, and the inference is also a prediction that repeat experiments will find the same parameters, if you factor out noise.

Hoo boy.

First, recall that the "error" of model outputs denotes the uncertainty surrounding a number, not the blunders. Error in this sense is routine and non-blameworthy. There are two classes of error:

1. Systematic uncertainties:

• are due to problems with calibrating the data collection process, or due to the modelling approach
• are usually correlated with previous measurements
• occur when a theory is not yet mature.

2. Stochastic uncertainties 133:

• can arise from real fluctuations (e.g. random quantum processes),
• are uncorrelated with previous measurements,
• follow a well-developed theory;

We can split systematics into two, model error and parameter error 134:

• It was overfitted to an imperfectly representative training set.
• The population has changed ("Concept drift")
• Your analysis environment has changed ("Data drift")

Because of a burst of successes in previously intractable tasks (which make money, or should eventually make money).

Substantially just because we have millions of times more data now, a wonderful architecture for dealing with it quickly, and ways of using it without a human having to touch it.

Also because of steady improvements in statistical theory: e.g. nonparametric models, which don't compress away any of the information in your inputs and still don't take long to compute.

Good question, since they are both just statistical modelling approaches: it's just a tribal separation. That is, the methods were developed (and reinvented) in different university departments.

One real difference is that black-box prediction, which is so often the terminal goal in ML, was heavily disparaged in stats for a long time. So a cartoon statistician aimed to infer the data-generating mechanism, while a cartoon ML engineer aimed for optimum prediction of future data. This wall is breaking down, I'm told.

In a sense they're not different: to infer (parameters) is to predict what new data will look like, and is also to predict that repeat experiments will find similar parameters. But conventionally, inference is an attempt to find the actual way that the data were generated, not just an empirically adequate tool that gets the input-output pairs correct enough.

The question for inference is "how exactly were these data generated?", where prediction just wants a decent answer to "what output will new inputs result in?". Leaping from data to world, or from data to more data.

That's a poorly defined question, Gavin. We can classify ML systems in a few ways: by the sort of data they use; by the abstract approach their algorithm takes; by the structure of the task they're solving, or by the literal task ("speech recognisers" or "car drivers").

There exists a sweeping and powerful proof that there is no such computable thing.

In practice, the most competitive algorithm over a range of well-defined problems are neural networks (replacing the gradient boosting machines which ruled before them).

Not "knowledge" or "tasks"; each model is an instance of some computational structure, whether:

• Functions
• Rulesets (ILP)
• State machines
• Grammars
• Problem solvers

Finding a function (a pairing machine like an equation) is by far the most common aim. Learning is usually "figuring out an equation to solve a specific problem based on some example data”.

There's a boring and an exciting version. The boring reading is just that deep NNs have a big gap between the input and output layer: more hidden layers mean deep.

The exciting reading is when we take recent successes to hint at a fundamental mechanism of all learning: successive abstractions, one per layer, until a clear signal rests atop the network: a newlyborn concept.

Surprisingly, they're not totally well-defined terms. But when they are being explicitly distinguished:

• ML is pure statistical learning: you hand it input-output pairs and tweak until your function is approximated.


• DM is learning over well-understood domains, where you can design your algorithms substantially.

You need to write the output model in a language the computer understands (model representation),

you need to encode the examples in a way the computer understands (data representation),

you need an unambiguous rule for distinguishing good models from bad ones (evaluation function),

and you need a way of picking the best hypothesis in the model language (optimisation function).

First, what do you know? The representation you choose limits the knowledge you can encode into the learner. If we have a lot of knowledge about what makes examples similar, you should use 'instance-based methods'. If we have knowledge about probabilistic dependencies, graphical models. If we have knowledge about the strict conditions involved, use first-order logic.

• instance: no generalisation, just compare new to all previous, in memory (e.g. k-nearest neighbors)
• Hyperplane-based methods form a linear combination of the features per class and predict the class with the highest-valued combination.
• Decision trees test one feature at each internal node, with one branch for each feature value, and have class predictions at the leaves.

• Add a term to the eval function (Regularisation):
  • Tikhonov regularization: penalize complex functions
  • Ivanov regularization: constrain the hypothesis space, either in the functional form or by adding constraints to the
• Simple significance test before and after, before adding new features

Yes. A grid search is the ultimate in multiple testing risk. We have to control the false discovery rate (fraction of falsely accepted non-null hypotheses). Bonferroni.

Because of the combinatorial explosion, or "curse of dimensionality".

And also in the process: the size of the failure space is arguably the square of software engineering's (already enormous) failure space.

"most of the volume of a high-dimensional orange is in the skin, not the pulp." Generalizing correctly becomes exponentially harder as the dimensionality grows, because a fixed-size training set covers a dwindling fraction of the input space.

An example from Pedro Domingos:

Say you have a vast, vast training set: a trillion rows and a hundred columns (which all just say "Yes" or "No" about the feature they're representing). Your vast training set still only covers a tiny fraction, 0.0000000000000000001%, of the whole input space:

$$ \text{Number of possible inputs} = 2^{100} = 1.3 x 10^{30}.$$ $$ 1 \text{trillion} = 1 x 10^{12} $$ $$ \text{coverage} = 1 x 10^{12} / 1.3 x 10^{30} = 1^{-18}th \text{of all possible examples} $$ * blessing of non-uniformity: examples are usually not spread uniformly throughout the instance space, but are concentrated near a lower-dimensional manifold. For example, k-nearest neighbor works quite well for handwritten digit recognition even though images of digits have one dimension per pixel, because the space of digit images is much smaller than the space of all possible images.

Well - you accept you must use software. And you are accustomed to using other people's software.

But when your task is in any way unusual, or when you're a hotshot who needs more performance than the Enterprise people shovel at you, or when you need to glue together several other pieces of software, or when you want more than just one-off answers - i.e. a regular report, or millions of split-second decisions, or a giant real-time pipeline, or a whole weak AI of your very own - then you will be coding (or hiring coders).

Minimally, code is glue for systems: take input from tool #1, change the format a little, give to tool #2. We call this scripting.

Maximally, code is the heart of your comparative advantage and the reason you haven't been steamrolled into the competitive sump yet. We call this application development. An app unifies a whole process: networking, scraping and web API abuse, cleaning, crunching, graphics, mailing reports, whatever.

If you program your way through a data problem, you gain much power and efficiency. You:

• Can liberate the task from proprietary software. Done right, this yields 1) performance, 2) actual customisability, 3) massive cost savings. 201


• Horizontal scaling. The corporate IT giants (SAS, Oracle) can't touch the cost-effectiveness of open source HPC / distributed analysis: when your need grows, you can just bolt on a new server. Commercial distributed tools will come, but expensively. Hadoop is free, except for the 100 servers you have to buy.


• Utterly flexible. Many of the best tools are just code libraries. (As opposed to full apps.) Can do unprecedented tasks.


• Your programs can do without the overhead of a GUI.


• Writing your analysis as a script means that you can share the work very easily, which means division of labour, reproducible results, and easy automation of even very intense quantitative work.

But there's a larger point: coding is a novel way of thinking in general. Yes, it is like maths - but testable, causal, interactive.

A programming language is "how you tell a computer what to do". But before that it's a way to express ideas and get push back from a rational oracle. (It's not reality that's pushing back, of course. You don't know if they're true, but you know if they are clear, if they could even possibly be true, if you are not completely fooling yourself.)

Consider the Bible, or Karl Marx's work, or Sigmund Freud's work. These are rammed full of invalid and unsound ideas - but they are beautiful, unified, and powerful, so they proved persuasive to billions of people. Human language offers no easy test of consistency, no way of really precisely connecting idea to idea. We have had only hard, piecemeal, irreplicable interpretation.

To see what's added by code, here's a thought experiment: Imagine the economic value of a line-by-line description, in English, of the Linux kernel. It would be nothing compared to the billions of dollars of value the kernel has created or saved. 200

The computability of source code is a side effect of its clarity. Code is testable thought.

This decision is really not as important as it looks. Scala and Python are beautiful to write. Java is everywhere (and lets you flick between DS and trad software engineering jobs). R is academically cutting-edge.

Alright, here's a deeper take - but you must add your own weights to the columns:


Whatever you do, learn regex!

It's like, so 2013.

Practically speaking, 'big data' is any dataset which cannot fit in memory on an expensive computer. More often, 'big data' refers to the use of that data: i.e. it is cheap, mass surveillance of private citizens by businesses and governments.

Oh it is - just not in your favour. A model is often dangerous if it is wrong, and sometimes dangerous if it is right.

Cathy O'Neil writes persuasively about perverse decision systems 301; from her I distilled this list of warning signs:

Opacity

• Is the subject aware they are being modelled?
• Is the subject aware of the model's outputs?
• Is the subject aware of the model's predictors and weights?
• Is the data the model uses open?
• Is it dynamic - does it update on its failed predictions?

Scale

• Does the model make decisions about many thousands of people?
• Is the model famous enough to change incentives in its domain?
• Does the model cause vicious feedback loops?
• Does the model assign high-variance population estimates to individuals?

Damage

• Does the model work against the subject's interests?
• If yes, does the model do so in the social interest?
• Is the model fully automated, i.e. does it make decisions as well as predictions?
• Does the model take into account things it shouldn't?
• Do its false positives do harm? Do its true positives?
• Is the harm of a false positive symmetric with the good of a true positive?

After the financial meltdown in 2008, the ex-quant Emanuel Derman came up with this:

• I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
• Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
• I will never sacrifice reality for elegance without explaining why I have done so. Nor will I give the people who use my model false comfort about its accuracy. Instead, I will make explicit its assumptions and oversights.
• I understand that my work may have enormous effects on society and the economy, many of them beyond my comprehension.

Precisely because it is a bit overheated, it could catch on.

The "data life cycle", which you might be lucky enough to control, depending on your organisation size:

1. Data capture.
2. Data maintenance. Busywork: any processing that doesn't generate value itself (movement, integration, cleansing, enrichment, extract-transform-load). Also calculating derived values like (Net revenue = Gross revenue - fixed costs - variable costs x units).
3. Data synthesis. To creation values via some inductive logic: predictions or inferences.
4. Data usage. Use in object-level tasks.
5. Data publication. To send data outside the org. Includes data breaches.
6. Data archival. To backup and then remove data from production environments.
7. Data purging. To remove every copy from the org.

It's another surprisingly vague term. It means something like "data without a schema, or normalisation, or metadata, or labelling as a good or bad outcome". Anything you couldn't analyse on a spreadsheet. Text corpuses are a good example: the meaning is in there somewhere, but "structured" (nonprobabilistic) methods won't help you much.

Or: Structured data is joined-up data: relations are known, duplicates have been taken out, absolute timings are given, connections to other rows are graphed.

When performance (network latency or model training speed or model accuracy) is more important than the cost.

• If you're just adding features but not rows, then you might be damaging your model with overfit and redundancy.
• If most values are zero then you have a sparsity problem. (Yes, "zero" or "N/A" is still data.)
• If it takes you above the 64GB sweet spot, so it doesn't fit in one computer's RAM, and doesn't gain much from parallelisation. Much harder to manipulate, even in these halcyon days of Spark and Sqoop.
• Some datasets are fantastically expensive, and you don't necessarily know that it will even help before you buy.

Since 2006, the launch of MapReduce, not that much. Since 2014, the launch of Spark, only as hard as functional programming is.

One of the jarring things about Hadoop development, speaking as a C21st programmer, is the return to massive waits between edit/compile/debug cycles (like, a 5 minute wait even using test data). This is the other price (besides your $100,000 Azure bill.)

Hell mode.

If you're like me, you'll need qualitative meaning first, to get you fired up: Gleick's The Information and Nilsson's Quest for Artificial Intelligence, Hofstadter's Goedel, Escher, Bach, Floridi's Blackwell Guide to the Philosophy of Computation, Howson's Scientific Reasoning, O'Neil's Weapons of Math Destruction, Domingos' Master Algorithm. Then the real work begins:

• Start with Python: Zed Shaw's Learn the Hard Way will get you moving, if you do as he tells you. Learn Python 3; you probably won't have much Python2 legacy code to handle.
• You must read other people's code to improve. Try a gamified site like CodeWars, which shows you elegant solutions only after you sweat out a dreadful hack.
• This is the nicest series of maths videos I've ever seen. Constant clicking of concepts into place.
• Once you are comfortable with basic abstraction, work through Downey's Think Stats.
• I enjoyed Blais' Statistical Inference for Everyone too.

Then read everything in the other sections below. (:

Danger zone.

• Work through Downey's Think Stats, Think Bayes and Davidson's Probabilistic Programming for Hackers.
• Grus' Data Science from Scratch will make you implement all kinds of cool statistical machines, which gives you an edge over even the highly mathematised people.
• Efron and Hastie's Computer Age Statistical Inference is a treat. It will persuade you that statisticians are presently being plagiarised and shortchanged.
• Ross' First Course in Probability</i>
• The Cambridge Maths for the Natural Sciences workbook is a focussed fave. Similarly, Meyer's Mathematics for Computer Science.
• Tom Mitchell's Machine Learning is old but clear. And being old is good: cuts through hype and shows what is not new at all.
• Oehler's A First Course in Design and Analysis of Experiments
• Mattuck's Intro to Analysis, chapters 1 - 17.
• then Strang's Linear Algebra, chapters 1,2,3,6 at least.
• A partisan pick is Etz et al's How to Become a Bayesian.

A reach ten times your grasp.

• Cady's >Handbook is designed especially for you, in that case. I like his scepticism and choice of tools n topics.
• Learn Python the Hard Way might be too slow. Try Sargent & Stachurski's Quantitative Economics (it's more broadly applicable than it sounds) or just dive into Grus.
• Neural Networks and Deep Learning is a great read.
• Jaynes' unfinished masterwork, The Logic of Science will tie together acres of mathematical ground. Start by reading the bitchy footnotes!
• Kleinberg & Tardos is the nicest intro to algos.

Well what are you waiting for?

• Chen's 120 Data Science Interview Questions gives you a good sense of how the field actually unfolds in business settings. And how brutal it's going to be if you interview in America.
• Mackay's Information Theory, Inference, and Learning Algorithms is really very beautiful.
• Pearl's Causality is groundbreaking, exciting, and clear
• Shalizi's Advanced Data Analysis is a wonderful read.
• Blum et al, Foundations of Data Science is maybe the first heavy textbook that focusses entirely on the core problems of the mishmash that is DS: hyper dimensionality is chapter 1, complexity theory, Bonferroni ghosts...</i>
• Everybody always recommends Hastie and Tibshirani, but Mackay's my guy.
• Bishop's Pattern Recognition is similarly hyped. It is synoptic and rigorous but hard work.
• Then there are the tool books, but you'll have no problem with those