Why

Presented by: Andy Choens, MSW

According to Wikipedia:


              R is a programming language and software environment for
              statistical computing and graphics. The R language is
              widely used among statisticians and data miners for
              developing statistical software and data
              analysis. Polls, surveys of data miners, and studies of
              scholarly literature databases show that R's popularity
              has increased substantially in recent years.

This is not a WHY NOT SAS presentation. This is a Why R presentation. By the end of the presentation, I hope to have demonstrated that R is a relevant language in the work we are doing today and will be even more relevant in years to come.

URLs To Remember:

The 1st link will let you see the presentation, as it was presented here today.
The 2nd link will let you see all of the underlying code used in today's presentation.
Later in the presentation, I will show some demo output files and demo code files. These resources can be found in the examples folder on GitHub which you can access by following the 2nd link.
You are free to use any of these materials. This presentation, slides, demo files, etc. are licensed under a Creative Commons with Attribution license.
Any time you see text in this color of blue. It is a link. If you click it, you will go somewhere fun.

Bad Reasons To Learn / Use R

Some guy with a beard likes it.
Someone on the Internet said it was great.
SAS isn't doing what you want it to. (Operator error?)

Good Reasons to Learn / Use R

Ecosystem
Literate Programming
Open Source / Open Science
Job Growth
Everyone Writes About R

Advance: The ecosystem of tools around R is nothing short of AMAZING and it is getting more amazing every single day, because ANYONE can contribute to the ecosystem.
As of February 2, 2015: CRAN, the primary R repository, contains 6,391 packages. These are peer reviewed packages, developed by the community. Bioconductor contains an additional 936 packages. Just let that sink in. And while you do:
I'd like to tell you about where you can use R. You can use R from within SAS. You can use R from within SPSS. You can use R from within MATLAB, Oracle, Postgres, and Vertica. R skills are portable skills. Microsoft recently bought the R-based consulting firm Revolution Analytics, which could result in better integration of R in Microsoft produces such as SQL Server or even Office.
Advance: This presentation isn't long enough to take a deep dive into Literate Programming, but I do want to mention this idea briefly.
Literate programming is an idea from the 1980s, to interweave natural language and computer programming language. I'll give you concrete examples of this later, but the upshot is that it makes it possible to write a program that is both self-documenting AND is able to compile to a finished, publishable manuscript or report. No more cutting and pasting for an hour. No more messing up a table because you forgot a decimal point on the 9th row of the 3rd column. A self-documenting, transparent manuscript that can be published.
This is an idea from the 80s. The book, Literate Programming was published in 1984. This is an idea that is very influential in the R community and is not only considered to be current and cutting edge, it is something that people are working on today to make it easier and faster to produce.
Advance: One good reason to use R is because it is open source. Those 7,000+ packages? Only possible because R is open source and engages with the community.
IBM is unlikely to make integrate SAS into SPSS. They are competitors. Besides, owning SAS and SPSS on the same machine without a student discount is . . . prohibitively expensive. R is considered safe, common ground, because it is not a commercial entity.
Advance: A perfectly reasonable, self-interested reason to use any tool professionally is because you think it will lead to future job opportunities.
Knowing how to use SAS effectively is a good way to make yourself eligible for a large number of job opportunities. Knowing how to use both R and SAS effectively makes you relevant to even more future job opportunities. Please note: I said AND. Not or. Not instead of. I said AND.
Advance: But please. Don't take my word for it. See what some of these people have to say.

Ecosystem

Packages

The capabilities of R are extended through user-created packages, which allow specialized statistical techniques, graphical devices (ggplot2), import/export capabilities, reporting tools (knitr, Sweave), etc. These packages are developed primarily in R, and sometimes in Java, C, C++ and Fortran.

Anyone in this room can write and contribute a package. Anyone in Albany can write and contribute a package to the R ecosystem.
If others value it, it will be used. It will get blogged about. Others will use it.
To contribute a package, or to R-Core, you don't have to work in a specific office or have a specific title.
R packages are like extensions for Firefox or Apps for Google Chrome. The take the base system and enhance it. And if you have ever looked at the number of extensions and apps for a web browser, it is staggering. Now, imagine that same diversity in an analytics package.
Now, stop imagining, because you can have it and it won't cost you a dime.

Popular Package Places

CRAN: The official R repositories. Wide ranging collection of packages. The growth of packages on this repo alone is staggering, and only increasing.
Bio: Provides packages for the analysis of genomic data. Bioconductor and R are the two repos I use the most. Packages here tend to be of very high quality, are updated, and well accepted by the broader community. There are over 7,000 packages at these two sites alone.
Omegahat: Older repo. Predates CRAN. Has more bleeding edge packages. For example, there is a package to integrate R and Firefox, allowing you to run R within a webpage. Some duplication with CRAN.
R-Forge: Clone of the venerable Source Forge site. This page is mostly packages not yet approved for distribution via CRAN, or is the beta version of the next release. Use most of this at your own risk. Some very interesting packages.
GH: OMG. There are over 20,000 public repositories on GitHub, which appear to be R source code. Some of these are packages developed by the author for personal use. Some are analyses. Some are for various classes. The diversity here is incredible. I have no idea how to count only those repos on GH that are packages. I looked around, and nobody else does either.

To Infinity & Beyond!

Source: r4stats.com/articles/popularity/

I want to plug the r4stats page, which is where I got a lot of the information I used in this presentation. The amount of information there is staggering and quite frankly somewhat obsessive.
R-Core is a powerful stats package. But I don't use R because of the brilliance of R-Core. I use R because of the packages.
In 2009 John Fox wrote some code that calculates the number of R packages available on CRAN for each release of R. This image does not include Bioconductor or any other R package repositories. Just the official CRAN packages.
For those keeping track at home, the super-imposed model is a quadratic parabola. More importantly is the continuing, rapid rise.
This single image is quite possibly one of the best reasons to use R and one of the best reasons to run for the hills. With this many options available, there is duplication. Some packages are brilliant. Some are bizarre. Some are probably downright stupid. To leverage this incredible resource takes time and it takes expertise.

Packages Examples

dplyr	ggplot2	sqlutils
A fast, consistent tool for working with data frame like objects, both in memory and out of memory.	An implementation of the grammar of graphics in R. It combines the advantages of both base and lattice graphics: conditioning and shared axes are handled automatically, and you can still build up a plot step by step from multiple data sources. It also implements a sophisticated multidimensional conditioning system and a consistent interface to map data to aesthetic attributes.	This package provides utilities for working with a library of SQL files.

Package Craziness

kobe-package

Ch IP peak Anno

The tuna Regional Fisheries Management Organisations (tRFMOs) use a common framework for providing scintific advice, i.e. the Kobe II Framework. This is based on maintaining fishing mortal- ity below FMSY and stock biomass above BMSY. This package provides methods for summarising results from stock assessments and Management Strategy Evaluations in the Kobe format.

The package includes functions to retrieve the sequences around the peak, obtain enriched Gene Ontology (GO) terms, find the nearest gene, exon, miRNA or custom features such as most conserved elements and other transcription factor binding sites supplied by users. Starting 2.0.5, new functions have been added for finding the peaks with bi-directional promoters with summary statistics (peaksNearBDP), for summarizing the occurrence of motifs in peaks (summarizePatternInPeaks) and for adding other IDs to annotated peaks or enrichedGO (addGeneIDs). This package leverages the biomaRt, IRanges, Biostrings, BSgenome, GO.db, multtest and stat packages.

Commercial Support

Programming Languages

Java

JMP

Mathematica

MATLAB

Python

SAS

SPSS

Statistica

tableau

Database
Vendors

Hadoop

Oracle

PostgreSQL

Vertica

Business Intelligence

Alteryx

Jaspersoft

Oracle Business Intelligence Enterprise Edition

Pentaho

SAP (and SAP HANA)

R bills itself as the lingua franca of statistics. This is done for several reasons.
It is fun to say lingua franca. Admit it. You're rolling your eyes, but you know I'm right.
More importantly, providers of commercial products agree. You can use R from within SAS, SPSS, Oracle and Vertica. In fact, it is actually difficult to find a statistics package, database or business intelligence tool that doesn't support R.
That means R skills are portable. Even in an office that uses SPSS, it is possible to leverage R skills. If you later move to a job that uses SAS, your SPSS skills aren't very useful. Your R skills . . . are. Sorry for the pun. But I meant to do it.

Interface Choices

Most commercial statistics packages come with a default interface of questionable quality. And if you don't like it . . . you can get over it. There probably aren't very many alternative choices.
R is different. In truth, the default R interface makes the SAS interface look positively utopic. But, because R is open source and is the lingua franca of the statistics community, there are more interfaces available for R than you have time to experiment with. And more are coming out every day.
I apologize for the color theme shown in all of the screenshots. I like it. It is called Solarized and is available in most pro-grade programming environments these days. I tend to use it anywhere / everywhere I can. Makes my eyes hurt less. All screenshots are taken from one of my laptops, and show a section of code I wrote in R and Emacs org-mode to assess a Kaggle data set about the Titanic disaster.

Interface - RStudio

Interface - EMACS / ESS

Interface - Eclipse / Statet

Interface - JASP

Interface - Custom Dashboard

But what if you want to deploy a custom interface? An interface to help an end-user understand a specific idea? What if you don't want to separate your code from your presentation?
SHINY is here to help. This is an extension to R that allows you to build dashboards and complex analytic websites in R. We will explore this particular dashboard in more detail shortly.
If you want a modern, powerful interface, for programming, interactive research or presentation, R has an option that at the very least rivals anything else on the market.
Total cost for the software seen in this presentation? Nothing. At its best, science is a meritocracy. R, and the dynamic ecosystem around it epitomizes this meritocratic spirit.

Literate Programming

From Wikipedia: The literate programming paradigm, as conceived by Knuth, represents a move away from writing programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts. Literate programs are written as an uninterrupted exposition of logic in an ordinary human language, much like the text of an essay, in which macros are included to hide abstractions and traditional source code.
In other words . . . stop writing reports and papers by running an analysis and then cutting and pasting the results into Word. This is an approach to science that combines the writing and the programming as a single, cohesive piece.
Literate Programming is a vast topic and is worthy of multiple presentations in its own right. Donald Knuth's ideas have had a deep impact on how I prefer to approach science. For the next few minutes, I would like to show you some simple examples of what literate programming means, to me.
At this point, it is worth pointing out, this presentation is an example of literate programming. In the following slides, we will look at some data, and pictures. I didn't cut and paste this information into this slide show. I embedded it into the slide show. Let me show you what I mean. But, to do that, I need a data set.

MTCARS

The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973 - 1974 models).

MTCARS - Just A Glimpse

data(mtcars)
kable(head(mtcars))

	mpg	cyl	disp	hp	drat	wt	qsec	vs	am	gear	carb
Mazda RX4	21.0	6	160	110	3.90	2.620	16.46	0	1	4	4
Mazda RX4 Wag	21.0	6	160	110	3.90	2.875	17.02	0	1	4	4
Datsun 710	22.8	4	108	93	3.85	2.320	18.61	1	1	4	1
Hornet 4 Drive	21.4	6	258	110	3.08	3.215	19.44	1	0	3	1
Hornet Sportabout	18.7	8	360	175	3.15	3.440	17.02	0	0	3	2
Valiant	18.1	6	225	105	2.76	3.460	20.22	1	0	3	1

Source: http://stat.ethz.ch/R-manual/R-devel/library/datasets/html/mtcars.html

This is the first 10 lines of the mtcars data set.
We are going to focus on the mpg, cyl, and disp columns.
Here, I am displaying the code used to produce the tabular display. As we will see, I don't have to do that. Sometimes it is beneficial to do so. Usually it is not.
mpg Miles/(US) gallon
cyl Number of cylinders
disp Displacement (cu.in.)
hp Gross horsepower
drat Rear axle ratio
wt Weight (lb/1000)
qsec 1/4 mile time
vs V/S
am Transmission (0 = automatic, 1 = manual)
gear Number of forward gears
carb Number of carburetors

R Markdown


              ---

              title: Minimal Markdown Example

              author: Andy Choens, MSW

              ---

              

              This is a simple, minimal example of report written in R markdown. The
              purpose of this report is two fold:

              

              1. To enlighten you about the relationship between engine displacement
              and gas mileage.

              2. Provide an accessible introduction to Literate Programming.

              

              The source of the data is the 1974 Motor Trend USA Magazine.

              

              ```{r echo=FALSE}

              

              ## Init R and Config

              library(ggplot2)

              data(mtcars)

              

              ```

Insert a Graph


              # Scatterplot

              

              Displacement is the independent variable. Gas mileage is the dependent
              variable. Correlation does not imply causation, except for this time
              when it does.

              

              ```{r echo=FALSE}

              

              ggplot(data = mtcars, aes(x=disp, y=mpg)) + geom_point(shape=1) + geom_smooth(method=lm)

              

              ```

Transparent, Reproducible

Code
PDF: render("vignettes/minimal-markdown.Rmd", pdf_document())
Word: render("vignettes/minimal-markdown.Rmd", word_document())
HTML: render("vignettes/minimal-markdown.Rmd", html_document())

How many of you have written a report, only to discover you made an error when cutting and pasting your results into Word? Literate programming removes errors due to cutting and pasting. The output is transparent and reproducible.
It is transparent, because you can look at the data and look at the code and know where the numbers come from. It is reproducible, because I will get the exact same results EVERY time I run this "report".
And if you give me a new data set (formatted the same way), I can spit out a new, equally accurate report in seconds. You cannot make these claims when you assemble a report by hand.
Science should be reproducible and verifiable.
Alaska Project Example: Trying to reproduce another analyst's work.

The Return of SHINY

Open Source /
Open Science

Free as in freedom, not beer.

The term open source often seems to have a negative connotation in people's minds. That is unfortunate, because it is everywhere. The Internet, your cell phone, SAS. None of this would work without open source. I want to dispel some myths and show you that R's greatest strength is the open source license.
In fact, R, without the open source license would be just another failed stats package. I'm standing here talking about R, the open source implementation of S. I doubt any of us will ever use S and nobody here cares. S doesn't matter. There is no community. There is no vibrant ecosystem. R matters, and it is the license that makes it different and better than its predecessor.
People interpret Open Source to mean 'free'.
But it means much more than that. Open Source, is an ideological method for the development, distribution and use of software.
The ecosystem, the integration with other tools, the veritable multitude of user interface options available for R. . . .
All of this is made possible by the ideological choices made by the R-Core team who lead the development of R.
Science isn't about getting into a journal. Science is about understanding. Science is about collaborating. Darwin didn't pull evolution clean out of the sky. Evolution was part of a larger zeitgeist.
In Darwin's day, journals were where that conversation happened.
Today, the conversation is about data. The discovery of the Higgs Boson? Data driven. And you can download the code they used to do it. Obama Care? Data driven.
In order to have a data driven conversation, we have to have a common language. Tools like R and Python are that common language, not because they are fundamentally better, but they are fundamentally available to anyone. With these tools, code, the true lingua franca of modern science, can be understood and used by anyone. From Mozambique to Albany, NY.
R isn't competing with SAS and SPSS because it is a beautiful language. It isn't. It is competing with them because of the side affects of the decision to develop R in an open, inclusive and transparent manner. Transparency, and a tradition of sharing code results in better science.

Open Source Is Everywhere:

FOSS:
	Firefox	Adium	Java
Based On:
	Chrome	Safari	Android

Even @ DOH:

FOSS:
	Filezilla	SQL Workbench/J	7-Zip
Based On:
	Vertica	SAS

Why?

The Four Freedoms

The freedom to run the program for any purpose.
The freedom to study how the program works, and change it to make it do what you wish.
The freedom to redistribute copies so you can help your neighbor.
The freedom to improve the program, and release your improvements (and modified versions in general) to the public, so that the whole community benefits.

Wikipedia, lists the four freedoms associated with Free Software, as defined by the Free Software Foundation as:
Richard Stallman wrote these, thinking about software. But, at the time her was working at the AI lab at MIT, so science is not an unknown idea to him.
I believe these four freedoms fit in well with the scientific process. Replace the word program with analysis. And re-read it.
I just defined open science. A form of verifiable, transparent science that uses code as the common core language that can be used, modified, and critiqued by anyone.
To the extent that science today is driven by data, the language of science must be code. We are all computer programmers because we are scientists and code is our common language.
Therefore we should use a language, a code, that is available to anyone. Because science is about understanding and that fundamentally requires communications and collaboration.

Real: Code, Science, People

This ain't Fishy Science

Now it gets personal.
My wife is a Type 1 Diabetic. Managing diabetes is a full time job. Tools such as the Dexcom Continuous Glucose Monitor (CGM) are nothing short of revolutionary. This is device she wears, which can estimate her blood glucose level every 5 minutes.
Before this, she could only know her blood glucose level when she manually tested her blood by pricking her finger. Obviously, she can't do this every 5 minutes. She wouldn't have any fingers left.
The Nighscout project is a open source data system that reads the data from the CGM and uploads it to a Mongo database, making the data accessible to anyone in the world. For parents of Type 1 children, this is revolutionary. Equally revolutionary, Type 1 diabetics can contribute their CGM readings to a centralized database. This is the single largest repository of CGM data in the world.
But this data has a problem. Nobody has a good way to use it, to manage it, to analyze it. Yet. By the end of the year, there will be a package for R called Nightscout. It will be able to interface with a user's private data AND the central data warehouse. It will be able to return key statistics, such as an estimated A1C score, which is a key process measure for the management of diabetes.
I know this, because I'm going to write it. Any doctor, or researcher will be able to use this package to download data from the Nightscout project, structure it and return some basic analyses of the data. Anyone studying diabetes will be able to take advantage of this technology. This is open science. Completely transparent. Completely reproducible. By anyone on earth.

Open Data or Open Science?

Hint: The presenter wants the latter.
data.ny.gov
Open Data is more powerful when it comes with open, reproducible, transparent analysis
Added benefit: Open science comes with no additional privacy concerns.

The Cuomo administration has made great strides in publishing government data to the public.
But data without code is fairly useless. True transparency comes from a combination of the two. Open science, transparent and reproducible, like what the Nightscout project is doing, enables citizens to understand its government.
Many of these open data sets are anonymous versions of data sets used internally for measurement and evaluation. Citizens should have access to the code AND the data, because you need both to understand. And when a license to SAS and SPSS cost thousands of dollars, I argue that these tools are fundamentally inappropriate for this purpose.
As citizens we have a right to know what government is doing. Wrapping that understanding up in a tool that is inaccessible to all by the wealthiest does not lead to further citizen engagement.

Job Growth

Jobs

Who Wants A Job?

Comparative Number of Jobs

Growth over Time

Indeed.com Query


              R
              !"R D" !"A R" !"H R" !"R N" !toys !kids !" R Walgreen" !walmart !"HVAC R" !"R Bard"

              and ( "biostatistics" or "data analysis" or "data analyst"
              or "epidemiologist"

              or "healthcare analysis" or "healthcare analyst"

              or "statistical"

              )

              ,SAS
              !"system administrator" !"school age" !sata !firmware !scsi !raid !samsung !scandinavian !sonar !nurse

              and ( "biostatistics" or "data analysis" or "data analyst"
              or "epidemiologist"

              or "healthcare analysis" or "healthcare analyst"

              or "statistical"

              )

The former pictures are links to the site on indeed.com's page where you can reproduce these pictures. The code on this page is the code I used to produce the pictures. Unfortunately, indeed.com is blocked here at DOH, so these are static images, not live returns from their website.
But, here is the code I used. Feel free to reproduce it, improve it, etc.
Another plug for r4stats, where I got the idea for doing this. This is a modified version of what they publish. My modifications are an attempt to focus the query on health care, which is obviously more relevant for this audience.

Everyone Writes About R

Lots of Articles About . . . . R!

New York Times	Data Analysts Captivated by R's Power	January 6, 2009
New York Times	R You Ready For R?	January 8, 2009
Fast Company	Why The R Programming Language Is Good For Business	May 5, 2014
Fast Company	The 9 Best Programming Languages For Crunching Data	May 5, 2014
Fast Company	Where Data And Creativity Meet: Confessions Of A Quant, Madison Avenue's "Hitman".	~ 2 Years Ago
Revolution Analytics	R Is Still Hot and Getting Hotter	January 14, 2015

With the exception of the article by Revolution Analytics, all of these articles were in the mainstream press.
I should point out that the NY Times publicly uses R, so their coverage could be a little biased. I don't know if Fast Company uses it internally or not.
I included the white paper by Revolution Analytics, because it goes into a lot of great detail. I'm not saying this is an "article". This is an advocacy piece, written by some very smart people.
And I'm apparently not the only one who thinks they are clever. Earlier this year, Microsoft bought Revolution Analytics because they felt they were left out of the ongoing revolution in data analytics. Although R has its roots and history in Unix, I am curious to see what Microsoft can do with this. Enhanced interoperability with Excel, SQL Server, Access, etc. would certainly be interesting.