The print method (print.noquote) prints character strings without quotes (' dots'). If right is specified in a call print(x, right=.), it takes precedence over a possible right setting of x, e.g., created by x. Startup: Initialization at Start of an R Session stop: Stop Function Execution stopifnot: Ensure the Truth of R Expressions strptime: Date-time Conversion Functions to and from Character strrep: Repeat the Elements of a Character Vector strsplit: Split the Elements of a Character Vector strtoi: Convert Strings to Integers strtrim: Trim.
In non-standard evaluation, you learned the basics of accessing and evaluating the expressions underlying computation in R. In this chapter, you’ll learn how to manipulate these expressions with code. You’re going to learn how to metaprogram: how to create programs with other programs!
Structure of expressions begins with a deep dive into the structure of expressions. You’ll learn about the four components of an expression: constants, names, calls, and pairlists.
Names goes into further details about names.
Calls gives more details about calls.
Capturing the current call takes a minor detour to discuss some common uses of calls in base R.
Pairlists completes the discussion of the four major components of an expression, and shows how you can create functions from their component pieces.
Parsing and deparsing discusses how to convert back and forth between expressions and text.
Walking the call tree with recursive functions concludes the chapter, combining everything you’ve learned about writing functions that can compute on and modify arbitrary R code.
Throughout this chapter we’re going to use tools from the
pryr package to help see what’s going on. If you don’t already have it, install it by running
Structure of expressions
To compute on the language, we first need to understand the structure of the language. That will require some new vocabulary, some new tools, and some new ways of thinking about R code. The first thing you’ll need to understand is the distinction between an operation and a result:
We want to distinguish the action of multiplying
x by 10 and assigning that result to
y from the actual result (40). As we’ve seen in the previous chapter, we can capture the action with
quote() returns an expression: an object that represents an action that can be performed by R. (Unfortunately
expression() does not return an expression in this sense. Instead, it returns something more like a list of expressions. See parsing and deparsing for more details.)
An expression is also called an abstract syntax tree (AST) because it represents the hierarchical tree structure of the code. We’ll use
pryr::ast() to see this more clearly:
There are four possible components of an expression: constants, names, calls, and pairlists.
constants are length one atomic vectors, like
ast()displays them as is.
Quoting a constant returns it unchanged:
names, or symbols, represent the name of an object rather than its value.
ast()prefixes names with a backtick.
calls represent the action of calling a function. Like lists, calls are recursive: they can contain constants, names, pairlists, and other calls.
()and then lists the children. The first child is the function that is called, and the remaining children are the function’s arguments.
As mentioned in every operation is a function call, even things that don’t look like function calls still have this hierarchical structure:
pairlists, short for dotted pair lists, are a legacy of R’s past. They are only used in one place: the formal arguments of a function.
at the top-level of a pairlist. Like calls, pairlists are also recursive and can contain constants, names, and calls.
str() does not follow these naming conventions when describing objects. Instead, it describes names as symbols and calls as language objects:
Using low-level functions, it is possible to create call trees that contain objects other than constants, names, calls, and pairlists. The following example uses
substitute() to insert a data frame into a call tree. This is a bad idea, however, because the object does not print correctly: the printed call looks like it should return “list” but when evaluated, it returns “data.frame”.
Together these four components define the structure of all R code. They are explained in more detail in the following sections.
There’s no existing base function that checks if an element is a valid component of an expression (i.e., it’s a constant, name, call, or pairlist). Implement one by guessing the names of the “is” functions for calls, names, and pairlists.
pryr::ast()uses non-standard evaluation. What’s its escape hatch to standard evaluation?
What does the call tree of an if statement with multiple else conditions look like?
ast(x + y %+% z)to
ast(x ^ y %+% z). What do they tell you about the precedence of custom infix functions?
Why can’t an expression contain an atomic vector of length greater than one? Which one of the six types of atomic vector can’t appear in an expression? Why?
Typically, we use
quote() to capture names. You can also convert a string to a name with
as.name(). However, this is most useful only when your function receives strings as input. Otherwise it involves more typing than using
quote(). (You can use
is.name() to test if an object is a name.)
(Names are also called symbols.
is.symbol() are identical to
Names that would otherwise be invalid are automatically surrounded by backticks:
There’s one special name that needs a little extra discussion: the empty name. It is used to represent missing arguments. This object behaves strangely. You can’t bind it to a variable. If you do, it triggers an error about missing arguments. It’s only useful if you want to programmatically create a function with missing arguments.
To explicitly create it when needed, call
quote() with a named argument:
You can use
formals()to both get and set the arguments of a function. Use
formals()to modify the following function so that the default value of
xis missing and
Write an equivalent to
eval(). Write an equivalent to
eval(). (Don’t worry about the multiple ways of choosing an environment; assume that the user supplies it explicitly.)
A call is very similar to a list. It has
[ methods, and is recursive because calls can contain other calls. The first element of the call is the function that gets called. It’s usually the name of a function:
But it can also be another call:
The remaining elements are the arguments. They can be extracted by name or by position.
The length of a call minus 1 gives the number of arguments:
Modifying a call
You can add, modify, and delete elements of the call with the standard replacement operators,
Calls also support the
[ method. But use it with care. Removing the first element is unlikely to create a useful call.
If you want a list of the unevaluated arguments (expressions), use explicit coercion:
Generally speaking, because R’s function calling semantics are so flexible, getting or setting arguments by position is dangerous. For example, even though the values at each position are different, the following three calls all have the same effect:
To work around this problem, pryr provides
standardise_call(). It uses the base
match.call() function to convert all positional arguments to named arguments:
Creating a call from its components
To create a new call from its components, you can use
as.call(). The first argument to
call() is a string which gives a function name. The other arguments are expressions that represent the arguments of the call.
as.call() is a minor variant of
call() that takes a single list as input. The first element is a name or call. The subsequent elements are the arguments.
The following two calls look the same, but are actually different:
What’s the difference? Which one should you prefer?
Implement a pure R version of
Concatenating a call and an expression with
c()creates a list. Implement
concat()so that the following code works to combine a call and an additional argument.
list()s don’t belong in expressions, we could create a more convenient call constructor that automatically combines lists into the arguments. Implement
make_call()so that the following code works.
mode<-work? How does it use
Read the source for
pryr::standardise_call(). How does it work? Why is
standardise_call()doesn’t work so well for the following calls. Why?
Read the documentation for
pryr::modify_call(). How do you think it works? Read the source code.
ast()and experimentation to figure out the three arguments in an
if()call. Which components are required? What are the arguments to the
Capturing the current call
Many base R functions use the current call: the expression that caused the current function to be run. There are two ways to capture a current call:
sys.call()captures exactly what the user typed.
match.call()makes a call that only uses named arguments. It’s like automatically calling
pryr::standardise_call()on the result of
The following example illustrates the difference between the two:
Modelling functions often use
match.call() to capture the call used to create the model. This makes it possible to
update() a model, re-fitting the model after modifying some of original arguments. Here’s an example of
update() in action:
update() work? We can rewrite it using some tools from pryr to focus on the essence of the algorithm.
update() has an
evaluate argument that controls whether the function returns the call or the result. But I think it’s better, on principle, that a function returns only one type of object, rather than different types depending on the function’s arguments.
This rewrite also allows us to fix a small bug in
update(): it re-evaluates the call in the global environment, when what we really want is to re-evaluate it in the environment where the model was originally fit — in the formula.
This is an important principle to remember: if you want to re-run code captured with
match.call(), you also need to capture the environment in which it was evaluated, usually the
parent.frame(). The downside to this is that capturing the environment also means capturing any large objects which happen to be in that environment, which prevents their memory from being released. This topic is explored in more detail in garbage collection.
Some base R functions use
match.call() where it’s not necessary. For example,
write.csv() captures the call to
write.csv() and mangles it to call
To fix this, we could implement
write.csv() using regular function call semantics:
This is much easier to understand: it’s just calling
write.table() with different defaults. This also fixes a subtle bug in the original
write.csv(mtcars, row = FALSE) raises an error, but
write.csv(mtcars, row.names = FALSE) does not. The lesson here is that it’s always better to solve a problem with the simplest tool possible.
Compare and contrast
write.csv(mtcars, 'mtcars.csv', row = FALSE)work? What property of argument matching has the original author forgotten?
update.formula()to use R code instead of C code.
Sometimes it’s necessary to uncover the function that called the function that called the current function (i.e., the grandparent, not the parent). How can you use
match.call()to find this function?
Pairlists are a holdover from R’s past. They behave identically to lists, but have a different internal representation (as a linked list rather than a vector). Pairlists have been replaced by lists everywhere except in function arguments.
The only place you need to care about the difference between a list and a pairlist is if you’re going to construct functions by hand. For example, the following function allows you to construct a function from its component pieces: a list of formal arguments, a body, and an environment. The function uses
as.pairlist() to ensure that the
function() has the pairlist of
args it needs.
This function is also available in pryr, where it does a little extra checking of arguments.
make_function() is best used in conjunction with
alist(), the argument list function.
alist() doesn’t evaluate its arguments so that
alist(x = a) is shorthand for
list(x = quote(a)).
make_function() has one advantage over using closures to construct functions: with it, you can easily read the source code. For example:
One useful application of
make_function() is in functions like
curve() allows you to plot a mathematical function without creating an explicit R function:
x is a pronoun.
x doesn’t represent a single concrete value, but is instead a placeholder that varies over the range of the plot. One way to implement
curve() would be with
Functions that use a pronoun are called anaphoric functions. They are used in Arc (a lisp like language), Perl, and Clojure.
alist(a = )different? Think about both the input and the output.
Read the documentation and source code for
pryr::partial(). What does it do? How does it work? Read the documentation and source code for
pryr::unenclose(). What does it do and how does it work?
The actual implementation of
curve()looks more like
How does this approach differ from
Parsing and deparsing
Sometimes code is represented as a string, rather than as an expression. You can convert a string to an expression with
parse() is the opposite of
deparse(): it takes a character vector and returns an expression object. The primary use of
parse() is parsing files of code to disk, so the first argument is a file path. Note that if you have code in a character vector, you need to use the
Because there might be many top-level calls in a file,
parse() doesn’t return just a single expression. Instead, it returns an expression object, which is essentially a list of expressions:
You can create expression objects by hand with
expression(), but I wouldn’t recommend it. There’s no need to learn about this esoteric data structure if you already know how to use expressions.
eval(), it’s possible to write a simple version of
source(). We read in the file from disk,
parse() it and then
eval() each component in a specified environment. This version defaults to a new environment, so it doesn’t affect existing objects.
source() invisibly returns the result of the last expression in the file, so
simple_source() does the same.
source() is considerably more complicated because it can
echo input and output, and also has many additional settings to control behaviour.
What are the differences between
Read the help for
deparse()and construct a call that
parse()do not operate symmetrically on.
Compare and contrast
simple_source()so it returns the result of every expression, not just the last one.
The code generated by
simple_source()Msn games. lacks source references. Read the source code for
sys.source()and the help for
srcfilecopy(), then modify
simple_source()to preserve source references. You can test your code by sourcing a function that contains a comment. If successful, when you look at the function, you’ll see the comment and not just the source code.
Walking the AST with recursive functions
It’s easy to modify a single call with
pryr::modify_call(). For more complicated tasks we need to work directly with the AST. The base
codetools package provides some useful motivating examples of how we can do this:
findGlobals()locates all global variables used by a function. This can be useful if you want to check that your function doesn’t inadvertently rely on variables defined in their parent environment.
checkUsage()checks for a range of common problems including unused local variables, unused parameters, and the use of partial argument matching.
To write functions like
checkUsage(), we’ll need a new tool. Because expressions have a tree structure, using a recursive function would be the natural choice. The key to doing that is getting the recursion right. This means making sure that you know what the base case is and figuring out how to combine the results from the recursive case. For calls, there are two base cases (atomic vectors and names) and two recursive cases (calls and pairlists). This means that a function for working with expressions will look like:
Finding F and T
We’ll start simple with a function that determines whether a function uses the logical abbreviations
F is generally considered to be poor coding practice, and is something that
R CMD check will warn about. Let’s first compare the AST for
TRUE is parsed as a logical vector of length one, while
T is parsed as a name. This tells us how to write our base cases for the recursive function: while an atomic vector will never be a logical abbreviation, a name might, so we’ll need to test for both
F. The recursive cases can be combined because they do the same thing in both cases: they recursively call
logical_abbr() on each element of the object.
Finding all variables created by assignment
logical_abbr() is very simple: it only returns a single
FALSE. The next task, listing all variables created by assignment, is a little more complicated. We’ll start simply, and then make the function progressively more rigorous.
Again, we start by looking at the AST for assignment:
Assignment is a call where the first element is the name
<-, the second is the object the name is assigned to, and the third is the value to be assigned. This makes the base cases simple: constants and names don’t create assignments, so they return
NULL. The recursive cases aren’t too hard either. We
lapply() over pairlists and over calls to functions other than
This function works for these simple cases, but the output is rather verbose and includes some extraneous
NULLs. Instead of returning a list, let’s keep it simple and use a character vector. We’ll also test it with two slightly more complicated examples:
This is better, but we have two problems: dealing with repeated names and neglecting assignments inside other assignments. The fix for the first problem is easy. We need to wrap
unique() around the recursive case to remove duplicate assignments. The fix for the second problem is a bit more tricky. We also need to recurse when the call is to
find_assign3() implements both strategies:
We also need to test subassignment:
We only want assignment of the object itself, not assignment that modifies a property of the object. Drawing the tree for the quoted object will help us see what condition to test for. The second element of the call to
<- should be a name, not another call.
Now we have a complete version:
While the complete version of this function is quite complicated, it’s important to remember we wrote it by working our way up by writing simple component parts.
Modifying the call tree
The next step up in complexity is returning a modified call tree, like what you get with
bquote() is a slightly more flexible form of quote: it allows you to optionally quote and unquote some parts of an expression (it’s similar to the backtick operator in Lisp). Everything is quoted, unless it’s encapsulated in
.() in which case it’s evaluated and the result is inserted:
This provides a fairly easy way to control what gets evaluated and when. How does
bquote() work? Below, I’ve rewritten
bquote() to use the same style as our other functions: it expects input to be quoted already, and makes the base and recursive cases more explicit:
R Add Quote In String
The main difference between this and the previous recursive functions is that after we process each element of calls and pairlists, we need to coerce them back to their original types.
Note that functions that modify the source tree are most useful for creating expressions that are used at run-time, rather than those that are saved back to the original source file. This is because all non-code information is lost:
These tools are somewhat similar to Lisp macros, as discussed in Programmer’s Niche: Macros in R by Thomas Lumley. However, macros are run at compile-time, which doesn’t have any meaning in R, and always return expressions. They’re also somewhat like Lisp fexprs. A fexpr is a function where the arguments are not evaluated by default. The terms macro and fexpr are useful to know when looking for useful techniques from other languages.
R Quotes In String Quartets
logical_abbr()use a for loop instead of a functional like
logical_abbr()works when given quoted objects, but doesn’t work when given an existing function, as in the example below. Why not? How could you modify
logical_abbr()to work with functions? Think about what components make up a function.
Write a function called
ast_type()that returns either “constant”, “name”, “call”, or “pairlist”. Rewrite
bquote2()to use this function with
switch()instead of nested if statements.
Write a function that extracts all calls to a function. Compare your function to
Write a wrapper around
bquote2()that does non-standard evaluation so that you don’t need to explicitly
bquote(). There is a subtle bug in
bquote(): it won’t replace calls to functions with no arguments. Why?
Improve the base
recurse_call()template to also work with lists of functions and expressions (e.g., as from