R on Ray Yang, Ph.D.

Interoperability in Tech. Evolution—Thoughts on “Reticulate"

Mon, 11 Mar 2019 00:00:00 +0000

“In our highly dynamic world, it’s not enough for an organization to possess a competitive advantage at a point in time; it needs an evolutionary advantage over time—a capacity to change as fast as change itself; to change before a crisis breaks.” — Gary Hamel

On March 6th, the R package “reticulate”(1.11.1) was released on Cran, which allows R users to directly call Python objects or use Python in an R interface.¹ Bloggers have been praising this development for the ground-breaking interoperability between R and Python. With “reticulate,” we no longer need to jump between different IDEs to leverage the complementarity between R and Python. This will certainly bring big ease to the programming routine of many “bilingual” data scientist using both R and Python.

As an occupational habit, I tend to think about the value creation and value capture in the dynamics of industry and technology evolution. Here are some thoughts on the “reticulate” case.

First of all, one the user level, “reticulate” by Rstudio will benefit those who program in both R and Python. So far, R and Python are the two most popular programming languages in data science. A decision for a newcomer in data science to make is always: “R vs. Python, which one should I choose?” Now this headache is significantly lessened. If we can overcome the “language barrier” easily, we won’t be stuck in the either-or any more.

Second, on the community level, there have already been quite a few early adopters who are R-Python bilinguals even before the pre-release of “reticulate.” They choose a language depending on which one is more readily usable for a specific problem. For bilinguals, there is no transition in their head, but there are some inefficient transitions on their fingertips. Now, they will leverage the power of “interoperability” to keep their hand movements in one place–Rstudio. These users will be engaged in Rstudio more often. What will also naturally happen? (a) the total number of R-Python bilinguals will grow; (b) the two languages will become more integrated, instead of being divided. We will see fewer (or slower growth of) packages/modules which have the same name and perform the same functionality in two languages, and also less “R vs Python comparisons.”

Third, thinking from the technological (and also strategic) perspective, we see the greatly increased compatibility between two programming languages. What follows compatibility will be an increase in the adoption rate on the technology who opens up its door to embrace other technologies. Rstudio is the superpower in the R domain. Such a move will draw many Python programmers to R and also attract and retain R users with the empowerment of R-Python interoperability. This “one-way” compatibility² will greatly enhance the usability and desirability of the Rstudio as a platform provider.

The release of “reticulate” is a strategic move of Rstudio. More data scientists will be stay engaged in Rstudio, or they will spend more fo their time on the platform. But this is not the end of the story. The interoperability will continuously enlarge the overlap between the R and Python communities if we draw a Venn diagram in our mind. The Python-centered IEDs are also powerful. They have great features that their users wouldn’t live without. They may match the move of Rstudio. More likely, the reverse “recuticate” in Python will emerge from the large open source community, by innumerable open source developers. Even if the one-way compatibility will continue, Python IEDs will not lose the game. As R programmers can easily operate Python in Rstudio, they are also more likely to travel and contribute to the Python world. This is similar to the case when Apple allows Microsoft’s Windows operating system to be installed on Mac and Amazon’s Kindle electronic book app installed on iPad. The competing platforms will both benefit from the one-way compatibility, because the enlarged overlap of the Venn diagram will bring “two-way” flows of traffic to both sides.³

Open source can always provide more flexibility to the users, developers, and businesses involved. Perhaps we cannot formulate the dynamics of technological evolution in a game-theoretic fashion, as the players in open source are not “competing” in the same way. For example, Anaconda, the company who distributes many Python IDEs (including JupyterLab and Spyder), has long been providing Rstudio IDE on their cloud and GUI. I see “reticulate” as a strategic, but also very harmonious move to create net benefits to the open source community for data science at large.

This can be easily done in Rstudio. Webinar by Rstudio Documentation by Rstudio ↩︎
The existing python IDEs have not provided an interface for R-Python interoperability yet. Python programmers tend to use JupyterLab, Rodeo, Spyder, Visual Studio Code, and PyCharm, none of which have such an interoperability feature right now.↩︎
For a more detailed analysis: Adner, R., Chen, J., & Zhu, F. (2016). Frenemies in Platform Markets: The Case of Apple’s iPad vs. Amazon’s Kindle. Harvard Business School Technology & Operations Mgt. Unit Working Paper, (15-087).↩︎

“MMeM”: Modeling the Multivariate Mixed-effects

Sun, 10 Mar 2019 00:00:00 +0000

I co-developed an R pakcage “MMeM” for estimating the variance-covariance matrix of random effects \(\mathbf{u}\) and \(\mathbf{e}\) on multiple dependent variables.

Statistical Intuition

The univariate mixed-effects model

In univariate mixed-effects model: \(\mathbf{y} = \mathbf{Xb} + \mathbf{Zu} + \mathbf{e}\) (1), we estimate the variance component \(\sigma_u^2\) and \(\sigma_e^2\) for \(\mathbf{u} \sim N(\mathbf{0}, \sigma_u^2\mathbf{I})\) and \(\mathbf{e} \sim N(\mathbf{0}, \sigma_e^2\mathbf{I})\).

In formula (1): \(\mathbf{y}\) is n \(\times\) 1 response vector; \(\mathbf{X}\) and \(\mathbf{Z}\) are n \(\times\) p and n \(\times\) s; \(\mathbf{b}\) is p \(\times\) 1 coefficients vector for the fixed effects; \(\mathbf{u}\) is s \(\times\) 1 matrix for the random effects, \(\mathbf{e}\) is n \(\times\) 1 vector of random errors.

The multivariate mixed-effects model

In multivariate mixed-effects model: \(\mathbf{y} = (\mathbf{I} \otimes \mathbf{X})\mathbf{b} + (\mathbf{I} \otimes \mathbf{Z} )\mathbf{u} + \mathbf{e}\) (2), in which \(\mathbf{y} = \left\{\mathbf{y}_i\right\}_c, \mathbf{b} = \left\{\mathbf{b}_i\right\}_c, \mathbf{u} = \left\{\mathbf{u}_i\right\}_c, \mathbf{e} = \left\{\mathbf{e}_i\right\}_c, i =1, \dots, q\), we estimate the variance-covariance matrix of random effects \(\mathbf{u}\) and \(\mathbf{e}\) on \(q\) response variates, namely \(\mathbf{T}\) and \(\mathbf{E}\), for \(var(\mathbf{u}) = \mathbf{G} = \mathbf{T}\otimes \mathbf{I}_s, var(\mathbf{e}) = \mathbf{R} = \mathbf{E} \otimes \mathbf{I}_n, var(\mathbf{y}) = \mathbf{V} = \mathbf{T}\otimes \mathbf{ZZ}' + \mathbf{E} \otimes \mathbf{I}_n\).

In formula (2): \(\mathbf{y}\) is nq \(\times\) 1 response vector; \(\mathbf{X}\) is n \(\times\) p design matrix for the fixed effects; \(\mathbf{b}\) is pq \(\times\) 1 coefficients vector for the fixed effects; \(\mathbf{Z}\) is n \(\times\) s design matrix for the random effects; \(\mathbf{u}\) is sq \(\times\) 1 vector of the random effects; \(\mathbf{e}\) is nq \(\times\) 1 vector of random errors.

An Implementation Example

# install.packages("MMeM")
library(MMeM)
data(simdata)

Initialization

# initialize with a positive-definiate var-cov
T.start = matrix(c(10, 5, 5, 15), 2, 2)
E.start = matrix(c(10, 1, 1, 3), 2, 2)

Estimation

# using the Henderson3 estimation mothod
results_henderson = MMeM_henderson3(fml = c(V1,V2) ~ X_vec + (1|Z_vec), data = simdata, factor_X = TRUE)

## Bivariate response: V1 and V2

print(results_henderson)

## $T.estimates
##                 T: V1   T: V1 V2     T: V2
## T.estimates  65.47395   9.969188  9.766204
## T.variance  807.56303 128.824836 20.552160
## 
## $E.estimates
##                E: V1  E: V1 V2     E: V2
## E.estimates 55.74506 11.477502 41.171927
## E.variance  11.09826  8.196899  6.054027

“regrrr”: One-stop R-toolkit for Compiling Regression Results

Wed, 06 Mar 2019 00:00:00 +0000

In strategy/management research, we always need to compile the regression results into the publishable format and sometimes plot the moderating effects. Thus, I developed this “regrrr” package to help do the job.

Here is the quickstart guide.

Installation

To install from CRAN:

install.packages("regrrr")
library(regrrr)

You can also use devtools to install the latest development version:

devtools::install_github("raykyang/regrrr")
library(regrrr)

Examples

compile the correlation table

library(regrrr)

## Registered S3 methods overwritten by 'tibble':
##   method     from  
##   format.tbl pillar
##   print.tbl  pillar

data(mtcars)
m0 <- lm(mpg ~ vs + carb + hp + wt, data = mtcars)
m1 <- update(m0, . ~ . + wt * hp)
m2 <- update(m1, . ~ . + wt * vs)
cor.table(data = m2$model)

##          Mean  S.D.     1     2    3    4    5
## 1.mpg   20.09  6.03  1.00                     
## 2.vs     0.44  0.50  0.66  1.00               
## 3.carb   2.81  1.62 -0.55 -0.57 1.00          
## 4.hp   146.69 68.56 -0.78 -0.72 0.75 1.00     
## 5.wt     3.22  0.98 -0.87 -0.55 0.43 0.66 1.00

compile the regression table

regression_table <- rbind(
combine_long_tab(to_long_tab(summary(m0)$coef),
                 to_long_tab(summary(m1)$coef),
                 to_long_tab(summary(m2)$coef)),
compare_models(m0, m1, m2))
rownames(regression_table) <- NULL
print(regression_table)

##        Variables   Model 0   Model 1   Model 2
## 1    (Intercept) 35.435*** 48.157*** 46.698***
## 2                  (2.503)   (4.097)   (9.272)
## 3             vs     1.353     1.077     2.171
## 4                  (1.382)   (1.152)   (6.320)
## 5           carb    -0.057    -0.043    -0.009
## 6                  (0.449)   (0.374)   (0.426)
## 7             hp   -0.024† -0.113***   -0.107*
## 8                  (0.014)   (0.027)   (0.044)
## 9             wt -3.792*** -8.071***   -7.594*
## 10                 (0.658)   (1.307)   (3.016)
## 11         hp:wt             0.027**    0.025†
## 12                           (0.008)   (0.015)
## 13         vs:wt                        -0.367
## 14                                     (2.081)
## 15     R_squared     0.833     0.889     0.889
## 16 Adj_R_squared     0.808     0.867     0.862
## 17       Delta_F            12.517**     0.031

plot the moderating effect

plot_effect(reg.coef = summary(m2)$coefficients, data = mtcars, model = m2,
            x_var.name = "wt", y_var.name = "mpg", moderator.name = "hp",
            confidence_interval = TRUE,  CI_Ribbon = FALSE, 
            xlab = "Weight", ylab = "MPG", moderator.lab = "Horsepower") +
ggplot2::theme(text=ggplot2::element_text(family="Times New Roman", size = 16))

plot the moderating effect with a linear spline

library(lspline)
data(mtcars)
m3 <- lm(mpg ~ vs + carb + hp + lspline(wt, knots = 4, marginal = FALSE) * hp, data = mtcars)
plot_effect(reg.coef=summary(m3)$coefficients, data = mtcars, model = m3, 
            x_var.name = "wt", y_var.name = "mpg", moderator.name = "hp",
            xlab="Weight", ylab="MPG", moderator.lab="Horsepower") +
ggplot2::theme(text=ggplot2::element_text(family="Times New Roman", size = 16))

As we can see from the last line of code, the plot is customizable using “ggplot2”. There are a couple of other functions. Please see the reference manual on R documentation for details.

I’m also aiming to expand the package’s usage around its core functions. If you have any ideas or want to report a bug, please contact me or suggest on the GitHub page.