one page of answers

 

1. Problem statement: what kind of problem is presented by the authors and why this problem is important?
2. Approach & Design: briefly describe the approach designed by the authors
3. Strengths and Weaknesses: list the strengths and weaknesses, in your opinion
4. Evaluation: how did the authors evaluate the performance of the proposed scheme? What kind of workload was designed and used?
5. Conclusion: by your own judgement.

Experiences Building PlanetLab
Larry Peterson, Andy Bavier, Marc E. Fiuczynski, Steve Muir

Department of Computer Science
Princeton Universit

y

Abstract. This paper reports our experiences building
PlanetLab over the last four years. It identifies the re-
quirements that shaped PlanetLab, explains the design
decisions that resulted from resolving conflicts among
these requirements, and reports our experience imple-
menting and supporting the system. Due in large part
to the nature of the “PlanetLab experiment,” the discus-
sion focuses on synthesis rather than new techniques, bal-
ancing system-wide considerations rather than improving
performance along a single dimension, and learning from
feedback from a live system rather than controlled exper-
iments using synthetic workloads.

1 Introduction
PlanetLab is a global platform for deploying and eval-
uating network services [21, 3]. In many ways, it has
been an unexpected success. It was launched in mid-
2002 with 100 machines distributed to 40 sites, but to-
day includes 700 nodes spanning 336 sites and 35 coun-
tries. It currently hosts 2500 researchers affiliated with
600 projects. It has been used to evaluate a diverse set of
planetary-scale network services, including content dis-
tribution [33, 8, 24], anycast [35, 9], DHTs [26], robust
DNS [20, 25], large-file distribution [19, 1], measurement
and analysis [30], anomaly and fault diagnosis [36], and
event notification [23]. It supports the design and evalua-
tion of dozens of long-running services that transport an
aggregate of 3-4TB of data every day, satisfying tens of
millions of requests involving roughly one million unique
clients and servers.
To deliver this utility, PlanetLab innovates along two

main dimensions:

• Novel management architecture. PlanetLab ad-
ministers nodes owned by hundreds of organiza-
tions, which agree to allow a worldwide community
of researchers—most complete strangers—to access
their machines. PlanetLab must manage a complex
relationship between node owners and users.

• Novel usage model. Each PlanetLab node should
gracefully degrade in performance as the number of
users grows. This gives the PlanetLab community
an incentive to work together to make best use of its
shared resources.

In both cases, the contribution is not a new mechanism or
algorithm, but rather a synthesis (and full exploitation) of
carefully selected ideas to produce a fundamentally new
system.
Moreover, the process by which we designed the sys-

tem is interesting in its own right:

• Experience-driven design. PlanetLab’s design
evolved incrementally based on experience gained
from supporting a live user community. This is
in contrast to most research systems that are de-
signed and evaluated under controlled conditions,
contained within a single organization, and evalu-
ated using synthetic workloads.

• Conflict-driven design. The design decisions that
shaped PlanetLab were responses to conflicting re-
quirements. The result is a comprehensive architec-
ture based more on balancing global considerations
than improving performance along a single dimen-
sion, and on real-world requirements that do not al-
ways lend themselves to quantifiable metrics.

One could view this as a new model of system design, but
of course it isn’t [6, 27].
This paper identifies the requirements that shaped the

system, explains the design decisions that resulted from
resolving conflicts among these requirements, and reports
our experience building and supporting the system. A
side-effect of the discussion is a fairly complete overview
of PlanetLab’s current architecture, but the primary goal
is to describe the design decisions that went into building
PlanetLab, and to report the lessons we have learned in
the process. For a comprehensive definition of the Plan-
etLab architecture, the reader is referred to [22].

2 Background
This section identifies the requirements we understood at
the time PlanetLab was first conceived, and sketches the
high-level design proposed at that time. The discussion
includes a summary of the three main challenges we have
faced, all of which can be traced to tensions between the
requirements. The section concludes by looking at the
relationship between PlanetLab and similar systems.

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 351

2.1 Requirements

PlanetLab’s design was guided by five major require-
ments that correspond to objectives we hoped to achieve
as well as constraints we had to live with. Although we
recognized all of these requirements up-front, the follow-
ing discussion articulates them with the benefit of hind-
sight.
(R1) It must provide a global platform that supports

both short-term experiments and long-running services.
Unlike previous testbeds, a revolutionary goal of Planet-
Lab was that it support experimental services that could
run continuously and support a real client workload. This
implied that multiple services be able to run concurrently
since a batch-scheduled facility is not conducive to a
24×7 workload. Moreover, these services (experiments)
should be isolated from each other so that one service
does not unduly interfere with another.
(R2) It must be available immediately, even though

no one knows for sure what “it” is. PlanetLab faced a
dilemma: it was designed to support research in broad-
coverage network services, yet its management (control)
plane is itself such a service. It was necessary to deploy
PlanetLab and start gaining experience with network ser-
vices before we fully understood what services would be
needed to manage it. As a consequence, PlanetLab had
to be designed with explicit support for evolution. More-
over, to get people to use PlanetLab—so we could learn
from it—it had to be as familiar as possible; researchers
are not likely to change their programming environment
to use a new facility.
(R3) We must convince sites to host nodes running

code written by unknown researchers from other organi-
zations. PlanetLab takes advantage of nodes contributed
by research organizations around the world. These nodes,
in turn, host services on behalf of users from other re-
search organizations. The individual users are unknown
to the node owners, and to make matters worse, the ser-
vices they deploy often send potentially disruptive pack-
ets into the Internet. That sites own and host nodes, but
trust PlanetLab to administer them, is unprecedented at
the scale PlanetLab operates. As a consequence, we must
correctly manage the trust relationships so that the risks
to each site are less than the benefits they derive.

(R4) Sustaining growth depends on support for auton-
omy and decentralized control. PlanetLab is a world-
wide platform constructed from components owned by
many autonomous organizations. Each organization must
retain some amount of control over how their resources
are used, and PlanetLab as a whole must give geographic
regions and other communities as much autonomy as pos-
sible in defining and managing the system. Generally,
sustaining such a system requires minimizing centralized
control.

(R5) It must scale to support many users with mini-
mal resources. While a commercial variant of PlanetLab
might have cost recovery mechanisms to provide resource
guarantees to each of its users, PlanetLab must operate
in an under-provisioned environment. This means con-
servative allocation strategies are not practical, and it is
necessary to promote efficient resource sharing. This in-
cludes both physical resources (e.g., cycles, bandwidth,
and memory) and logical resources (e.g., IP addresses).
Note that while the rest of this paper discusses the

many tensions between these requirements, two of them
are quite synergistic. The requirement that we evolve
PlanetLab (R2) and the need for decentralized control
(R4) both point to the value of factoring PlanetLab’s man-
agement architecture into a set of building block compo-
nents with well-defined interfaces. A major challenge of
building PlanetLab was to understand exactly what these
pieces should be.
To this end, PlanetLab originally adopted an organiz-

ing principle called unbundled management, which ar-
gued that the services used to manage PlanetLab should
themselves be deployed like any other service, rather than
bundled with the core system. The case for unbundled
management has three arguments: (1) to allow the sys-
tem to more easily evolve; (2) to permit third-party de-
velopers to build alternative services, enabling a software
bazaar, rather than rely on a single development team
with limited resources and creativity; and (3) to permit
decentralized control over PlanetLab resources, and ulti-
mately, over its evolution.

2.2 Initial Design

PlanetLab supports the required usage model through dis-
tributed virtualization—each service runs in a slice of
PlanetLab’s global resources. Multiple slices run con-
currently on PlanetLab, where slices act as network-wide
containers that isolate services from each other. Slices
were expected to enforce two kinds of isolation: resource
isolation and security isolation, the former concerned
with minimizing performance interference and the latter
concerned with eliminating namespace interference.
At a high-level, PlanetLab consists of a centralized

front-end, called PlanetLab Central (PLC), that remotely
manages a set of nodes. Each node runs a node man-
ager (NM) that establishes and controls virtual machines
(VM) on that node. We assume an underlying virtual ma-
chine monitor (VMM) implements the VMs. Users create
slices through operations available on PLC, which results
in PLC contacting the NM on each node to create a local
VM. A set of such VMs defines the slice.
We initially elected to use a Linux-based VMM due to

Linux’s high mind-share [3]. Linux is augmented with
Vservers [16] to provide security isolation and a set of
schedulers to provide resource isolation.

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association352

2.3 Design Challenges
Like many real systems, what makes PlanetLab interest-
ing to study—and challenging to build—is how it deals
with the constraints of reality and conflicts among re-
quirements. Here, we summarize the three main chal-
lenges; subsequent sections address each in more detail.
First, unbundled management is a powerful design

principle for evolving a system, but we did not fully un-
derstand what it entailed nor how it would be shaped by
other aspects of the system. Defining PlanetLab’s man-
agement architecture—and in particular, deciding how to
factor management functionality into a set of independent
pieces—involved resolving three main conflicts:

• minimizing centralized components (R4) yet main-
taining the necessary trust assumptions (R3);

• balancing the need for slices to acquire the resources
they need (R1) yet coping with scarce resources
(R5);

• isolating slices from each other (R1) yet allowing
some slices to manage other slices (R2).

Section 3 discusses our experiences evolving PlanetLab’s
management architecture.
Second, resource allocation is a significant challenge

for any system, and this is especially true for PlanetLab,
where the requirement for isolation (R1) is in conflict
with the reality of limited resources (R5). Part of our ap-
proach to this situation is embodied in the management
structure described in Section 3, but it is also addressed
in how scheduling and allocation decisions are made on a
per-node basis. Section 4 reports our experience balanc-
ing isolation against efficient resource usage.
Third, we must maintain a stable system on behalf of

the user community (R1) and yet evolve the platform to
provide long-term viability and sustainability (R2). Sec-
tion 5 reports our operational experiences with Planet-
Lab, and the lessons we have learned as a result.
2.4 Related Systems
An important question to ask about PlanetLab is whether
its specific design requirements make it unique, or if our
experiences can apply to other systems. Our response is
that PlanetLab shares “points of pain” with three simi-
lar systems—ISPs, hosting centers, and the GRID—but
pushes the envelope relative to each.
First, PlanetLab is like an ISP in that it has many

points-of-presence and carries traffic to/from the rest of
the Internet. Like ISPs (but unlike hosting centers and the
GRID), PlanetLab has to provide mechanisms that can
by used to identify and stop disruptive traffic. PlanetLab
goes beyond traditional ISPs, however, in that it has to
deal with arbitrary (and experimental) network services,
not just packet forwarding.

Second, PlanetLab is like a hosting center in that its
nodes support multiple VMs, each on behalf of a differ-
ent user. Like a hosting center (but unlike the GRID or
ISPs), PlanetLab has to provide mechanisms that enforce
isolation between VMs. PlanetLab goes beyond hosting
centers, however, because it includes third-party services
that manage other VMs, and because it must scale to large
numbers of VMs with limited resources.

Third, PlanetLab is like the GRID in that its resources
are owned by multiple autonomous organizations. Like
the GRID (but unlike an ISP or hosting center), PlanetLab
has to provide mechanisms that allow one organization to
grant users at another organization the right to use its re-
sources. PlanetLab goes far beyond the GRID, however,
in that it scales to hundreds of “peering” organizations by
avoiding pair-wise agreements.

PlanetLab faces new and unique problems because it
is at the intersection of these three domains. For exam-
ple, combining multiple independent VMs with a single
IP address (hosting center) and the need to trace disrup-
tive traffic back to the originating user (ISP) results in
a challenging problem. PlanetLab’s experiences will be
valuable to other systems that may emerge where any of
these domains intersect, and may in time influence the
direction of hosting centers, ISPs, and the GRID as well.

3 Slice Management
This section describes the slice management architecture
that evolved over the past four years. While the discus-
sion includes some details, it primarily focuses on the de-
sign decisions and the factors that influenced them.

3.1 Trust Assumptions
Given that PlanetLab sites and users span multiple orga-
nizations (R3), the first design issue was to define the un-
derlying trust model. Addressing this issue required that
we identify the key principals, explicitly state the trust as-
sumptions among them, and provide mechanisms that are
consistent with this trust model.
Over 300 autonomous organizations have contributed

nodes to PlanetLab (they each require control over the
nodes they own) and over 300 research groups want to
deploy their services across PlanetLab (the node own-
ers need assurances that these services will not be dis-
ruptive). Clearly, establishing 300×300 pairwise trust
relationships is an unmanageable task, but it is well-
understood that a trusted intermediary is an effective way
to manage such an N×N problem.
PLC is one such trusted intermediary: node owners

trust PLC to manage the behavior of VMs that run on
their nodes while preserving their autonomy, and re-
searchers trust PLC to provide access to a set of nodes
that are capable of hosting their services. Recognizing
this role for PLC, and organizing the architecture around

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 353

Owner
1

2

3

4

PLC
Service
Developer
(User)

Node

Figure 1: Trust relationships among principals.

it, is the single most important aspect of the design be-
yond the simple model presented in Section 2.2.
With this backdrop, the PlanetLab architecture recog-

nizes three main principals:

• PLC is a trusted intermediary that manages nodes
on behalf a set of owners, and creates slices on those
nodes on behalf of a set of users.

• An owner is an organization that hosts (owns) Plan-
etLab nodes. Each owner retains ultimate control
over their own nodes, but delegates management of
those nodes to the trusted PLC intermediary. PLC
provides mechanisms that allow owners to define re-
source allocation policies on their nodes.

• A user is a researcher that deploys a service on a set
of PlanetLab nodes. PlanetLab users are currently
individuals at research organizations (e.g., universi-
ties, non-profits, and corporate research labs), but
this is not an architectural requirement. Users create
slices on PlanetLab nodes via mechanisms provided
by the trusted PLC intermediary.

Figure 1 illustrates the trust relationships between node
owners, users, and the PLC intermediary. In this figure:

1. PLC expresses trust in a user by issuing it credentials
that let it access slices. This means that the user
must adequately convince PLC of its identity (e.g.,
affiliation with some organization or group).

2. A user trusts PLC to act as its agent, creating slices
on its behalf and checking credentials so that only
that user can install and modify the software running
in its slice.

3. An owner trusts PLC to install software that is able
to map network activity to the responsible slice.
This software must also isolate resource usage of
slices and bound/limit slice behavior.

4. PLC trusts owners to keep their nodes physically se-
cure. It is in the best interest of owners to not cir-
cumvent PLC (upon which it depends for accurate
policing of its nodes). PLC must also verify that ev-
ery node it manages actually belongs to an owner
with which it has an agreement.

Given this model, the security architecture includes the
following mechanisms. First, each node boots from an
immutable file system, loading (1) a boot manager pro-
gram, (2) a public key for PLC, and (3) a node-specific
secret key. We assume that the node is physically secured
by the owner in order to keep the key secret, although a
hardware mechanism such as TCPA could also be lever-
aged. The node then contacts a boot server running at
PLC, authenticates the server using the public key, and
uses HMAC and the secret key to authenticate itself to
PLC. Once authenticated, the boot server ensures that the
appropriate VMM and the NM are installed on the node,
thus satisfying the fourth trust relationship.
Second, once PLC has vetted an organization through

an off-line process, users at the site are allowed to cre-
ate accounts and upload their private keys. PLC then
installs these keys in any VMs (slices) created on be-
half of those users, and permits access to those VMs via
ssh. Currently, PLC requires that new user accounts
are authorized by a principal investigator associated with
each site—this provides some degree of assurance that
accounts are only created by legitimate users with a con-
nection to a particular site, thus satisfying the first trust
relationship.
Third, PLC runs an auditing service that records in-

formation about all packet flows coming out of the node.
The auditing service offers a public, web-based interface
on each node, through which anyone that has received un-
wanted network traffic from the node can determine the
responsible users. PLC archives this auditing information
by periodically downloading the audit log.

3.2 Virtual Machines and Resource Pools

Given the requirement that PlanetLab support long-lived
slices (R1) and accommodate scarce resources (R5), the
second design decision was to decouple slice creation
from resource allocation. In contrast to a hosting cen-
ter that might create a VM and assign it a fixed set of
resources as part of an SLA, PlanetLab creates new VMs
without regard for available resources—each such VM is
given a fair share of the available resources on that node
whenever it runs—and then expects slices to engage one
or more brokerage services to acquire resources.
To this end, the NM supports two abstract objects: vir-

tual machines and resource pools. The former is a con-
tainer that provides a point-of-presence on a node for a
slice. The latter is a collection of physical and logical
resources that can be bound to a VM. The NM supports
operations to create both objects, and to bind a pool to a
VM for some fixed period of time. Both types of objects
are specified by a resource specification (rspec), which is
a list of attributes that describe the object. A VM can run
as soon as it is created, and by default is given a fair share
of the node’s unreserved capacity. When a resource pool

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association354

is bound to a VM, that VM is allocated the corresponding
resources for the duration of the binding.
Global management services use these per-node opera-

tions to create PlanetLab-wide slices and assign resources
to them. Two such service types exist today: slice cre-
ation services and brokerage services. These services can
be separate or combined into a single service that both
creates and provisions slices. At the same time, different
implementations of brokerage services are possible (e.g.,
market-based services that provide mechanisms for buy-
ing and selling resources [10, 14], and batch scheduling
services that simply enforce admission control for use of
a finite resource pool [7]).

As part of the resource allocation architecture, it was
also necessary to define a policy that governs how re-
sources are allocated. On this point, owner autonomy
(R4) comes into play: only owners are allowed to invoke
the “create resource pool” operation on the NM that runs
on their nodes. This effectively defines the one or more
“root” pools, which can subsequently be split into sub-
pools and reassigned. An owner can also directly allocate
a certain fraction of its node’s resources to the VM of a
specific slice, thereby explicitly supporting any services
the owner wishes to host.

3.3 Delegation
PlanetLab’s management architecture was expected to
evolve through the introduction of third-party services
(R2). We viewed the NM interface as the key feature,
since it would support the many third-party creation and
brokerage services that would emerge. We regarded PLC
as merely a “bootstrap” mechanism that could be used to
deploy such new global management services, and thus,
we expected PLC to play a reduced role over time.
However, experience showed this approach to be

flawed. This is for two reasons, one fundamental and
one pragmatic. First, it failed to account for PLC’s cen-
tral role in the trust model of Section 3.1. Maintaining
trust relationships among participants is a critical role
played by PLC, and one not easily passed along to other
services. Second, researchers building new management
services on PlanetLab were not interested in replicating
all of PLCs functionality. Instead of using PLC to boot-
strap a comprehensive suite of management services, re-
searchers wanted to leverage some aspects of PLC and
replace others.
To accommodate this situation, PLC is today struc-

tured as follows. First, each owner implicitly assigns all
of its resources to PLC for redistribution. The owner can
override this allocation by granting a set of resources to
a specific slice, or divide resources among multiple bro-
kerage services, but by default all resources are allocated
to PLC.
Second, PLC runs a slice creation service—called

pl conf—on each node. This service runs in a stan-
dard VM and invokes the NM interface without any ad-
ditional privilege. It also exports an XML-RPC interface
by which anyone can invoke its services. This is impor-
tant because it means other brokerage and slice creation
services can use pl conf as their point-of-presence on
each node rather than have to first deploy their own slice.
Originally, the PLC/pl conf interface was private as we
expected management services to interact directly with
the node manager. However, making this a well-defined,
public interface has been a key to supporting delegation.
Third, PLC provides a front-end—available either as

a GUI or as a programmatic interface at www.planet-
lab.org—through which users create slices. The PLC
front-end interacts with pl conf on each node with the
same XML-RPC interface that other services use.

Finally, PLC supports two methods by which slices
are actually instantiated on a set of nodes: direct and
delegated. Using the direct method, the PLC front-end
contacts pl conf on each node to create the correspond-
ing VM and assign resources to it. Using delegation, a
slice creation service running on behalf of a user con-
tacts PLC for a ticket that encapsulates the right to create
a VM or redistribute a pool of resources. A ticket is a
signed rspec; in this case, it is signed by PLC. The agent
then contacts pl conf on each node to redeem this ticket,
at which time pl conf validates it and calls the NM to
create a VM or bind a pool of resources to an existing
VM. The mechanisms just described currently support
two slice creation services (PLC and Emulab [34], the
latter uses tickets granted by the former), and two bro-
kerage services (Sirius [7] and Bellagio [2], the first of
which is granted capacity as part of a root resource allo-
cation decision).

Note that the delegated method of slice creation is
push-based, while the direct method is pull-based. With
delegation, a slice creation service contacts PLC to re-
trieve a ticket granting it the right to create a slice, and
then performs an XML-RPC call to pl conf on each node.
For a slice spanning a significant fraction of PlanetLab’s
nodes, an implementation would likely launch multiple
such calls in parallel. In contrast, PLC uses a polling
approach: each pl conf contacts PLC periodically to re-
trieve a set of tickets for the slices it should run.
While the push-based approach can create a slice in

less time, the advantage of pull-based approach is that
it enables slices to persist across node reinstalls. Nodes
cannot be trusted to have persistent state since they are
completely reinstalled from time to time due to unrecov-
erable errors such as corrupt local file systems. The pull-
based strategy views all nodes as maintaining only soft
state, and gets the definitive list of slices for that node
from PLC. Therefore, if a node is reinstalled, all of its
slices are automatically recreated. Delegation makes it

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 355

possible for others to develop alternative slice creation
semantics—for example, a “best effort” system that ig-
nores such problems—but PLC takes the conservative
approach because it is used to create slices for essential
management services.
3.4 Federation
Given our desire to minimize the centralized elements of
PlanetLab (R4), our next design decision was to make it
possible for multiple independent PlanetLab-like systems
to co-exist and federate with each other. Note that this
issue is distinct from delegation, which allows multiple
management services to co-exist withing a single Planet-
Lab.
There are three keys to enabling federation. First, there

must be well-defined interfaces by which independent in-
stances of PLC invoke operations on each other. To this
end, we observe that our implementation of PLC natu-
rally divides into two halves: one that creates slices on
behalf of users and one that manages nodes on behalf of
owners, and we say that PLC embodies a slice authority
and a management authority, respectively. Correspond-
ing to these two roles, PLC supports two distinct inter-
faces: one that is used to create and control slices, and
one that is used to boot and manage nodes. We claim
that these interfaces are minimal, and hence, define the
“narrow waist” of the PlanetLab hourglass.
Second, supporting multiple independent PLCs im-

plies the need to name each instance. It is PLC in
its slice authority role that names slices, and its name
space must be extended to also name slice authori-
ties. For example, the slice cornell.cobweb is implicitly
plc.cornell.cobweb, where plc is the top-level slice au-
thority that approved the slice. (As we generalize the slice
name space, we adopt “.” instead of “ ” as the delimiter.)
Note that this model enables a hierarchy of slice author-
ities, which is in fact already the case with plc.cornell,
since PLC trusts Cornell to approve local slices (and the
users bound to them).
This generalization of the slice naming scheme leads

to several possibilities:

• PLC delegates the ability to create slices to regional
slice authorities (e.g., plc.japan.utokyo.ubiq);

• organizations create “private” PlanetLab’s (e.g.,
epfl.chawla) that possibly peer with each other, or
with the “public” PlanetLab; and

• alternative “root” naming authorities come into exis-
tence, such as one that is responsible for commercial
(for-profit) slices (e.g., com.startup.voip).

The third of these is speculative, but the first two scenar-
ios have already happened or are in progress, with five
private PlanetLabs running today and two regional slice

Service Lines of Code Language
Node Manager 2027 Python

Proper 5752 C
pl conf 1975 Python
Sirius 850 Python
Stork 12803 Python

CoStat + CoMon 1155 C
PlanetFlow 5932 C

Table 1: Source lines of code for various management services

authorities planned for the near future. Note that there
must be a single global naming authority that ensures all
top-level slice authority names are unique. Today, PLC
plays that role.

The third key to federation is to design pl conf so that
it is able to create slices on behalf of many different slice
authorities. Node owners allocate resources to the slice
authorities they want to support, and configure pl conf to
accept tickets signed by slice authorities that they trust.
Note that being part of the “public” PlanetLab carries the
stipulation that a certain minimal amount of capacity be
set aside for slices created by the PLC slice authority, but
owners can reserve additional capacity for other slice au-
thorities and for individual slices.

3.5 Least Privilege
We conclude our description of PlanetLab’s management
architecture by focusing on the node-centric issue of how
management functionality has been factored into self-
contained services, moved out of the NM and isolated in
their own VMs, and granted minimal privileges.

When PlanetLab was first deployed, all management
services requiring special privilege ran in a single root
VM as part of a monolithic node manager. Over time,
stand-alone services have been carved off of the NM
and placed in their own VMs, multiple versions of some
services have come and gone, and new services have
emerged. Today, there are five broad classes of manage-
ment services. The following summarizes one particular
“suite” of services that a user might engage; we also iden-
tify alternative services that are available.

Slice Creation Service: pl conf is the default slice cre-
ation service. It requires no special privilege: the
node owner creates a resource pool and assigns it to
pl conf when the node boots. Emulab [34] offers
an alternative slice creation service that uses tickets
granted by PLC and redeemed by pl conf.

Brokerage Service: Sirius [7] is the most widely used
brokerage service. It performs admission control on
a resource pool set aside for one-hour experiments.

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association356

Sirius requires no special privilege: pl conf allo-
cates a sub-pool of resources to Sirius. Bellagio [2]
and Tycoon [14] are alternative market-based bro-
kerage services that are initialized in the same way.

Monitoring Service: CoStat is a low-level instrumen-
tation program that gathers data about the state of
the local node. It is granted the ability to read
/proc files that report data about the underlying
VMM, as well as the right to execute scripts (e.g.,
ps and top) in the root context. Multiple ad-
ditional services—e.g., CoMon [31], PsEPR [5],
SWORD [18]—then collect and process this infor-
mation on behalf of users. These services require no
additional privilege.

Environment Service: Stork [12] deploys, updates, and
configures services and experiments. Stork is
granted the right to mount the file system of a client
slice, which Stork then uses to install software pack-
ages required by the slice. It is also granted the right
to mark a file as immutable, so that it can safely be
shared among slices without any slice being able to
modify the file. Emulab and AppManager [28] pro-
vide alternative environment services without extra
privilege; they simply provide tools for uploading
software packages.

Auditing Service: PlanetFlow [11] is an auditing ser-
vice that logs information about packet flows, and
is able to map externally visible network activity to
the responsible slice. PlanetFlow is granted the right
to run ulogd in the root context to retrieve log in-
formation from the VMM.

The need to grant narrowly-defined privileges to cer-
tain management services has led us to define a mecha-
nism called Proper (PRivileged OPERation) [17]. Proper
uses an ACL to specify the particular operations that can
be performed by a VM that hosts a management service,
possibly including argument constraints for each opera-
tion. For example, the CoStat monitoring service gath-
ers various statistics by reading /proc files in the root
context, so Proper constrains the set of files that can be
opened by CoStat to only the necessary directories. For
operations that affect other slices directly, such as mount-
ing the slice’s file system or executing a process in that
slice, Proper also allows the target slice to place addi-
tional constraints on the operations that can be performed
e.g., only a particular directory may be mounted by Stork.
In this way we are able to operate each management ser-
vice with a small set of additional privileges above a nor-
mal slice, rather than giving out coarse-grained capabili-
ties such as those provided by the standard Linux kernel,
or co-locating the service in the root context.

Finally, Table 1 quantifies the impact of moving func-
tionality out of the NM in terms of lines-of-code. The
LOC data is generated using David A. Wheeler’s ’SLOC-
Count’. Note that we show separate data for Proper and
the rest of the node manager; Proper’s size is in part a
function of its implementation in C.
One could argue that these numbers are conservative,

as there are additional services that this list of manage-
ment services employ. For example, CoBlitz is a large
file transfer mechanism that is used by Stork and Emulab
to disseminate large files across a set of nodes. Similarly,
a number of these services provide a web interface that
must run on each node, which would greatly increase the
size of the TCB if the web server itself had to be included
in the root context.

4 Resource Allocation
One of the most significant challenges for PlanetLab has
been to maximize the platform’s utility for a large user
community while dealing with the reality of limited re-
sources. This challenge has led us to a model of weak
resource isolation between slices. We implement this
model through fair sharing of CPU and network band-
width, simple mechanisms to avoid the worst kinds of in-
terference on other resources like memory and disk, and
tools to give users information about resource availability
on specific nodes. This section reports our experiences
with this model in practice, and describes some of the
techniques we’ve adopted to make the system as effec-
tive and stable as possible.

4.1 Workload
PlanetLab supports a workload mixing one-off experi-
ments with long-running services. A complete character-
ization of this workload is beyond the scope of this paper,
but we highlight some important aspects below.

CoMon—one of the performance-monitoring services
running on PlanetLab—classifies a slice as active on a
node if it contains a process, and live if, in the last five
minutes, it used at least 0.1% (300ms) of the CPU. Fig-
ure 2 shows, by quartile, the number of active and live
slices across PlanetLab during the past year. Each graph
shows five lines; 25% of PlanetLab nodes have values
that fall between the first and second lines, 25% between
the second and third, and so on.

Looking at each graph in more detail, Figure 2(a) illus-
trates that the number of active slices on most PlanetLab
nodes has grown steadily. The median active slice count
has increased from 40 slices in March 2005 to the mid-
50s in April 2006, and the maximum number of active
slices has increased from 60 to 90. PlanetLab nodes can
support large numbers of mostly idle slices because each
VM is very lightweight. Additionally, the data shows that
75% of PlanetLab nodes have consistently had at least 4

0

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 357

0

20

40

60

80

100

05/May 05/Jul 05/Sep 05/Nov 06/Jan 06/Mar

S
lic

es
w

ith
a

pr
oc

es
s

Min

1st Q Median 3rd Q Max

(a) Active slices, by quartile

0

5

10

15

20

25

30

05/May 05/Jul 05/Sep 05/Nov 06/Jan 06/Mar

S
lic

es
us

in
g

>0
.1

%

C

P
U

Min 1st Q Median 3rd Q Max

(b) Live slices, by quartile

Figure 2: Active and live slices on PlanetLab

active slices during the past year.
Figure 2(b) shows the distribution of live slices. Note

that at least 50% of PlanetLab nodes consistently have a
live slice count within two of the median. Additional data
indicates that this is a result of three factors. First, some
monitoring slices (like CoMon and PlanetFlow) are live
everywhere, and so create a lower bound on the number
of live slices. Second, most researchers do not appear
to greedily use more nodes than they need; for example,
only 10% of slices are deployed on all nodes, and 60% are
deployed on less than 50 nodes. We presume researchers
are self-organizing their services and experiments onto
disjoint sets of nodes so as to distribute load, although
there are a small number of popular nodes that support
over 25 live slices. Third, the slices that are deployed on
all nodes are not live on all of them at once. For instance,
in April 2006 we observed that CoDeeN was active on
436 nodes but live on only 269. Robust (and adaptive)
long-running services are architected to dynamically bal-
ance load to less utilized nodes [33, 26].

Of course we did not know what PlanetLab’s work-
load would look like when we made many early design
decisions. As reported in Section 2.2, one such decision
was to use Linux+Vservers as the VMM, primarily be-
cause of the maturity of the technology. Since this time,
alternatives like Xen have advanced considerably, but we
have not felt compelled to reconsider this decision. A key
reason is that PlanetLab nodes run up to 25 live VMs,

and up to 90 active VMs, at at time. This is possible
because we could build a system that supports resource
overbooking and graceful degradation on a framework of
Vserver-based VMs. In contrast, Xen allocates specific
amounts of resources, such as physical memory and disk,
to each VM. For example, on a typical PlanetLab node
with 1GB memory, Xen can support only 10 VMs with
100MB memory each, or 16 with 64MB memory. There-
fore, it’s not clear how a PlanetLab based on Xen could
support our current user base. Note that the management
architecture presented in the previous section is general
enough to support multiple VM types (and a Xen proto-
type is running in the lab), but resource constraints make
it likely that most PlanetLab slices will continue to use
Vservers for the foreseeable future.

4.2 Graceful Degradation

PlanetLab’s usage model is to allow as many users on a
node as want to use it, enable resource brokers that are
able to secure guaranteed resources, and gracefully de-
grade the node’s performance as resources become over
utilized. This section describes the mechanisms that sup-
port such behavior and evaluates how well they work.

4.2.1 CPU

The earliest version of PlanetLab used the standard Linux
CPU scheduler, which provided no CPU isolation be-
tween slices: a slice with 400 Java threads would get

400

times the CPU of a slice with one thread. This situation
occasionally led to collapse of the system and revealed
the need for a slice-aware CPU scheduler.
Fair share scheduling [32] does not collapse under

load, but rather supports graceful degradation by giving
each scheduling container proportionally fewer cycles.
Since mid-2004, PlanetLab’s CPU scheduler has per-
formed fair sharing among slices. During that time, how-
ever, PlanetLab has run three distinct CPU schedulers: v2
used the SILK scheduler [3], v3.0 introduced CKRM (a
community project in its early stages), and v3.2 (the cur-
rent version) uses a modification of Vserver’s CPU rate
limiter to implement fair sharing and reservations. The
question arises, why so many CPU schedulers?
The answer is that, for the most part, we switched CPU

schedulers for reasons other than scheduling behavior.
We switched from SILK to CKRM to leverage a com-
munity effort and reduce our code maintenance burden.
However, at the time we adopted it, CKRM was far from
production quality and the stability of PlanetLab suffered
as a result. We then dropped CKRM and wrote another
CPU scheduler, this time based on small modifications to
the Vservers code that we had already incorporated into
the PlanetLab kernel. This CPU scheduler gave us the ca-
pability to provide slices with CPU reservations as well
as shares, which we lacked with SILK and CKRM. Per-

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association358

0

20

40

60

80

100

05/May 05/Jul 05/Sep 05/Nov 06/Jan 06/Mar

A
va

ila
bl

e
C

P
U

%

Min 1st Q Median 3rd Q Max

Figure 3: CPU % Available on PlanetLab

haps more importantly, the scheduler was more robust, so
PlanetLab’s stability dramatically improved, as shown in
Section 5. We are solving the code maintenance problem
by working with the Vservers developers to incorporate
our modifications into their main distribution.

The current (v3.2) CPU scheduler implements fair
sharing and work-conserving CPU reservations by over-
laying a token bucket filter on top of the standard Linux
CPU scheduler. Each slice has a token bucket that accu-
mulates tokens at a specified rate; every millisecond, the
slice that owns the running process is charged one token.
A slice that runs out of tokens has its processes removed
from the runqueue until its bucket accumulates a mini-
mum amount of tokens. This filter was already present
in Vservers, which used it to put an upper bound on the
amount of CPU that any one VM could receive; we sim-
ply modified it to provide a richer set of behaviors.

The rate that tokens accumulate depends on whether
the slice has a reservation or a share. A slice with a reser-
vation accumulates tokens at its reserved rate: for exam-
ple, a slice with a 10% reservation gets 100 tokens per
second, since a token entitles it to run a process for one
millisecond. The default share is actually a small reser-
vation, providing the slice with 32 tokens every second,
or 3% of the total capacity.

The main difference between reservations and shares
occurs when there are runnable processes but no slice has
enough tokens to run: in this case, slices with shares are
given priority over slices with reservations. First, if there
is a runnable slice with shares, tokens are given out fairly
to all slices with shares (i.e., in proportion to the number
of shares each slice has) until one can run. If there are
no runnable slices with shares, then tokens are given out
fairly to slices with reservations. The end result is that the
CPU capacity is effectively partitioned between the two
classes of slices: slices with reservations get what they’ve
reserved, and slices with shares split the unreserved ca-
pacity of the machine proportionally.

CoMon indicates that the average PlanetLab node has
its CPU usage pegged at 100% all the time. However,
fair sharing means that an individual slice can still ob-

0

20

40

60

80

100

0 500 1000 1500 2000

P
er

ce
nt

of
sl

ic
es

MB used when reset

Figure 4: CDF of memory consumed when slice reset

tain a significant percentage of the CPU. Figure 3 shows,
by quartile, the CPU availability across PlanetLab, ob-
tained by periodically running a spinloop in the CoMon
slice and observing how much CPU it receives. The data
shows large amounts of CPU available on PlanetLab: at
least 10% of the CPU is available on 75% of nodes, at
least 20% CPU on 50% of nodes, and at least 40% CPU
on 25% of nodes.
4.2.2 Memory
Memory is a particularly scarce resource on PlanetLab,
and we were faced with with chosing between four de-
signs. One is the default Linux behavior, which either
kernel panics or randomly kills a process when memory
becomes scarce. This clearly does not result in grace-
ful degradation. A second is to statically allocate a fixed
amount of memory to each slice. Given that there are up
to 90 active VMs on a node, this would imply an imprac-
tically small 10MB allocation for each VM on the typical
node with 1GB of memory. A third option is to explic-
itly allocate memory to live VMs, and reclaim memory
from inactive VMs. This implies the need for a control
mechanism, but globally synchronizing such a mecha-
nism across PlanetLab (i.e., to suspend a slice) is prob-
lematic at fine-grained time scales. The fourth option is
to dynamically allocate memory to VMs on demand, and
react in a more predictable way when memory is scarce.
We elected the fourth option, implementing a simple

watchdog daemon, called pl mom, that resets the slice
consuming the most physical memory when swap has al-
most filled. This penalizes the memory hog while keep-
ing the system running for everyone else.
Although pl mom was noticably active when first

deployed—as users learned to not keep log files in mem-
ory and to avoid default heap sizes—it now typically re-
sets an average of 3-4 VMs per day, with higher rates dur-
ing heavy usage (e.g., major conference deadlines). For
example, 200 VMs were reset during the two week run-
up to the OSDI deadline. We note, however, that roughly
one-third of these resets were on under-provisioned nodes
(i.e., nodes with less than 1GB of memory).

Figure 4 shows the cumulative distribution function of

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 359

0

20

40

60

80

100

05/Dec 06/Jan 06/Feb 06/Mar 06/Apr

A
va

ila
bl

e
M

B
M

em
or

y

Min 1st Q Median 3rd Q Max

Figure 5: Memory availability on PlanetLab

how much physical memory individual VMs were con-
suming when they were reset between November 2004
and April 2006. We note that about 10% of the resets
(corresponding largely to the first 10% of the distribution)
occurred on nodes with less than 1GB memory, where
memory pressure was tighter. Over 80% of all resets had
been allocated at least 128MB. Half of all resets occurred
when the slice was using more than 400MB of memory,
which on a shared platform like PlanetLab indicates ei-
ther a memory leak or poor experiment design (e.g., a
large in-memory logfile).

Figure 5 shows CoMon’s estimate of how many MB of
memory are available on each PlanetLab node. CoMon
estimates available memory by allocating 100MB, touch-
ing random pages periodically, and then observing the
size of the in-memory working set over time. This serves
as a gauge of memory pressure, since if physical memory
is exhausted and another slice allocates memory, these
pages would be swapped out. The CoMon data shows
that a slice can keep a 100MB working set in memory
on at least 75% of the nodes (since only the minimum
and first quartile line are really visible), so it appears that
there is not as much memory pressure on PlanetLab as we
expected. This also reinforces our intuition that pl mom
resets slices mainly on nodes with too little memory or
when the slice’s application has a memory leak.
4.2.3 Bandwidth
Hosting sites can cap the maximum rate at which the
local PlanetLab nodes can send data. PlanetLab fairly
shares the bandwidth under the cap among slices, using
Linux’s Hierarchical Token Bucket traffic filter [15]. The
node bandwidth cap allows sites to limit the peak rate at
which nodes send data so that PlanetLab slices cannot
completely saturate the site’s outgoing links.

The sustained rate of each slice is limited by the
pl mom watchdog daemon. The daemon allows each
slice to send a quota of bytes each day at the node’s cap
rate, and if the slice exceeds its quota, it imposes a much
smaller cap for the rest of the day. For example, if the
slice’s quota is 16GB/day, then this corresponds to a sus-
tained rate of 1.5Mbps; once the slice sends more than

100

1000

10000

100000

06/Jan/21 06/Feb/04 06/Feb/18 06/Mar/04 06/Mar/18 06/Apr/01 06/Apr/15

T
x

B
an

dw
id

th
(K

b/
s)

1st Q Median 3rd Q Max

(a) Transmit bandwidth in Kb/s, by quartile

100

1000

10000

100000

06/Jan/21 06/Feb/04 06/Feb/18 06/Mar/04 06/Mar/18 06/Apr/01 06/Apr/15

R
x

B
an

dw
id

th
(K

b/
s)

1st Q Median 3rd Q Max

(b) Receive bandwidth in Kb/s, by quartile

Figure 6: Sustained network rates on PlanetLab

16GB, it is capped at 1.5Mbps until midnight GMT. The
goal is to allow most slices to burst data at the node’s cap
rate, but prevents slices that are sending large amounts of
data from badly abusing the site’s local resources.
There are two weaknesses of PlanetLab’s bandwidth

capping approach. First, some sites pay for bandwidth
based on the total amount of traffic they generate per
month, and so they need to control the node’s sustained
bandwidth rather than the peak. As mentioned, pl mom
limits sustained bandwidth, but it operates on a per-slice
(rather than per-node) basis and cannot currently be con-
trolled by the sites. Second, PlanetLab does not currently
cap incoming bandwidth. Therefore, PlanetLab nodes
can still saturate a bottleneck link by downloading large
amounts of data. We are currently investigating ways to
fix both of these limitations.

Figure 6 shows, by quartile, the sustained rates at
which traffic is sent and received on PlanetLab nodes
since January 2006. These are calculated as the sums of
the average transmit and receive rates for all the slices
of the machine over the last 15 minutes. Note that the
y axis is logarithmic, and the Minimum line is omit-
ted from the graph. The typical PlanetLab node trans-
mits about 1Mb/s and receives 500Kb/s, corresponding
to about 10.8GB/day sent and 5.4GB/day received. These
numbers are well below the typical node bandwidth cap
of 10Mb/s. On the other hand, some PlanetLab nodes do
actually have sustained rates of 10Mb/s both ways.

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association360

0

20

40

60

80

100

05/May 05/Jul 05/Sep 05/Nov 06/Jan 06/Mar

D
is

k
F

ul
l%

Min 1st Q Median 3rd Q Max

Figure 7: Disk usage, by quartile, on PlanetLab

4.2.4 Disk

PlanetLab nodes do not provide permanent storage: data
is not backed up, and any node may be reinstalled without
warning. Services adapt to this environment by treating
disk storage as a cache and storing permanent data else-
where, or else replicating data on multiple nodes. Still,
a PlanetLab node that runs out of disk space is essen-
tially useless. In our experience, disk space is usually ex-
hausted by runaway log files written by poorly-designed
experiments. This problem was mitigated, but not en-
tirely solved, by the introduction of per-slice disk quotas
in June 2005. The default quota is 5GB, with larger quo-
tas granted on a case-by-case basis.
Figure 7 shows, by quartile, the disk utilization on

PlanetLab. The visible dip shortly after May 2005 is
when quotas were introduced. We note that, though disk
utilization grows steadily over time, 75% of Planetlab
nodes still have at least 50% of the disks free. Some
PlanetLab nodes do occasionally experience full disks,
but most are old nodes that do not meet the current sys-
tem requirements.

4.2.5 Jitter

CPU scheduling latency can be a serious problem for
some PlanetLab applications. For example, in a packet
forwarding overlay, the time between when a packet ar-
rives and when the packet forwarding process runs will
appear as added network latency to the overlay clients.
Likewise, many network measurement applications as-
sume low scheduling latency in order to produce pre-
cisely spaced packet trains. Many measurement applica-
tions can cope with latency by knowing which samples to
trust and which must be discarded, as described in [29].
Scheduling latency is more problematic for routing over-
lays, which may have to drop incoming packets.
A simple experiment indicates how scheduling latency

can affect applications on PlanetLab. We deploy a packet
forwarding overlay, constructed using the Click modular
software router [13], on six Planetlab nodes co-located
at Abilene PoPs between Washington, D.C. and Seattle.
Our experiment then uses ping packets to compare the

0

20

40

60

80

100

75 80 85 90 95 100

P
er

ce
nt

Millisecond RTT from ping

Network
Overlay w/SCHED_RR

Overlay

Figure 8: RTT CDF on network, overlay with and without SCHED RR

RTT between the Seattle and D.C. nodes on the network
and on the six-hop overlay. Each of the six PlanetLab
nodes running our overlay had load averages between 2
and 5, and between 5 and 8 live slices, during the ex-
periment. We observe that the network RTT between the
two nodes is a constant 74ms over 1000 pings, while the
overlay RTT varies between 76ms and 135ms. Figure 8
shows the CDF of RTTs for the network (leftmost curve)
and the overlay (rightmost curve). The overlay CDF has
a long tail that is chopped off at 100ms in the graph.
There are several reasons why the overlay could have

its CPU scheduling latency increased, including: (1) if
another task is running when a packet arrives, Click must
wait to forward the packet until the running task blocks
or exhausts its timeslice; (2) if Click is trying to use more
than its “fair share”, or exceeds its CPU guarantee, then
its token bucket CPU filter will run out of tokens and
it will be removed from the runqueue until it acquires
enough tokens to run; (3) even though the Click process
may be runnable, the Linux CPU scheduler may still de-
cide to schedule a different process; and (4) interrupts
and other kernel activity may preempt the Click process
or otherwise prevent it from running.
We can attack the first three sources of latency using

existing scheduling mechanisms in PlanetLab. First, we
give the overlay slice a CPU reservation to ensure that
it will never run out of tokens during our experiment.
Second, we use chrt to run the Click process on each
machine with the SCHED RR scheduling policy, so that
it will immediately jump to the head of the runqueue
and preempt any running task. The Proper service de-
scribed in Section 3.5 enables our slice to run the privi-
leged chrt command on each PlanetLab node.

The middle curve in Figure 8 shows the results of re-
running our experiment with these new CPU scheduling
parameters. The overhead of the Click overlay, around
3ms, is clearly visible as the difference between the two
left-most curves. In the new experiment, about 98% of
overlay RTTs are within 3ms of the underlying network
RTT, and 99% are within 6ms. These CPU scheduling
mechanisms are employed by PL-VINI, the VINI (VIr-

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 361

tual Network Infrastructure) prototype implemented on
PlanetLab, to reduce latency in an overlay network as an
artifact of CPU scheduling delay [4].
We note two things. First, the obstacle to making

this solution available on PlanetLab is primarily one of
policy—choosing which slices should get CPU reserva-
tions and bumps to the head of the runqueue, since it
is not possible to reduce everyone’s latency on a heav-
ily loaded system. We plan to offer this option to short-
term experiments via the Sirius brokerage service, but
long-running routing overlays will need to be handled on
a case-by-case basis. Second, while our approach can
provide low latency to the Click forwarder in our exper-
iment 99% of the time, it does not completely solve the
latency problem. We hypothesize that the remaining CPU
scheduling jitter is due to the fourth source of latency
identified earlier, i.e., kernel activity. If so, we may be
able to further reduce it by enabling kernel preemption, a
feature already available in the Linux 2.6 kernel.
4.2.6 Remarks
Note that only limited conclusions can be drawn from the
fact that there is unused capacity available on PlanetLab
nodes. Users are adapting to the behavior of the system
(including electing to not use it) and they are writing ser-
vices that adapt to the available resources. It is impossible
to know how many resources would have been used, even
by the same workload, had more been available. How-
ever, the data does document that PlanetLab’s fair share
approach is behaving as expected.

5 Operational Stability
The need to maintain a stable system, while at the same
time evolving it based on user experience, has been a ma-
jor complication in designing PlanetLab. This section
outlines the general strategies we adopted, and presents
data that documents our successes and failures.
5.1 Strategies
There is no easy way to continually evolve a system that
is experiencing heavy use. Upgrades are potentially dis-
ruptive for at least two reasons: (1) new features intro-
duce new bugs, and (2) interface changes force users to
upgrade their applications. To deal with this situation, we
adopted three general strategies.
First, we kept PlanetLab’s control plane (i.e., the ser-

vices outlined in Section 3) orthogonal from the OS. This
meant that nearly all of the interface changes to the sys-
tem affected only those slices running management ser-
vices; the vast majority of users were able to program to
a relatively stable Linux API. In retrospect this is an ob-
vious design principle, but when the project began, we
believed our main task was to define a new OS interface
tailored for wide-area services. In fact, the one example
where we deviated from this principle—by changing the

socket API to support safe raw sockets [3]—proved to be
an operational diaster because the PlanetLab OS looked
enough like Linux that any small deviation caused dis-
proportionate confusion.
Second, we leveraged existing software whereever

possible. This was for three reasons: (1) to improve the
stability of the system; (2) to lower the barrier-to-entry
for the user community; and (3) to reduce the amount of
new code we had to implement and maintain. This last
point cannot be stressed enough. Even modest changes
to existing software packages have to be tracked as those
packages are updated over time. In our eagerness to reuse
rather than invent, we made some mistakes, the most no-
table of which is documented in the next subsection.
Third, we adopted a well-established practice of rolling

new releases out incrementally. This is for the obvious
reason—to build confidence that the new release actually
worked under realistic loads before updating all nodes—
but also for a reason that’s somewhat unique to Planet-
Lab: some long-running services maintain persistent data
repositories, but doing so depends on a minimal number
of copies being available at any time. Updates that reboot
nodes must happen incrementally if long-running storage
services are to survive.
Note that while few would argue with these

principles—and it is undoubtedly the case that we would
have struggled had we not adhered to them—our experi-
ence is that many other factors (some unexpected) had a
significant impact on the stability of the system. The rest
of this section reports on these operational experiences.

5.2 Node Stability
We now chronicle our experience operating and evolving
PlanetLab. Figure 9 illustrates the availability of Plan-
etLab nodes from September 2004 through April 2006,
as inferred from CoMon. The bottom line indicates the
PlanetLab nodes that have been up continuously for the
last 30 days (stable nodes), the middle line is the count
of nodes that came online within the last 30 days, and
the top line is all registered PlanetLab nodes. Note that
the difference between the bottom and middle lines repre-
sents the “churn” of PlanetLab over a month’s time; and
the difference between the middle and top lines indciates
the number of nodes that are offline. The vertical lines in
Figure 9 are important dates, and the letters at the top of
the graph let us refer to the intervals between the dates.
There have clearly been problems providing the com-

munity with a stable system. Figure 9 illustrates several
reasons for this:

• Sometimes instability stems from the community
stressing the system in new ways. In Figure 9, inter-
val A is the run-up to the NSDI’05 deadline. During
this time, heavy use combined with memory leaks in

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association362

GFEDCBA

A: Runup to NSDI ’05 deadline
B: After NSDI ’05 deadline
C: 3.0 rollout begins
D: 3.0 stable release

E: 3.1 stable release
F: 3.2 rollout begins
G: 3.2 stable release

0

100

200

300

400

500

600

700

04/Sep 04/Nov 05/Jan 05/Mar 05/May 05/Jul 05/Sep 05/Nov 06/Jan 06/Mar

N
od

e
co

un
t

Stable nodes (up > 30 days)
Active in last 30 days

Registered nodes

Figure 9: Node Availability

some experiments caused kernel panics due to Out-
of-Memory errors. This is the common behavior
of Linux when the system runs out of memory and
swap space. pl mom (Section 4.2.2) was introduced
in response to this experience.

• Software upgrades that require a reboot obviously
effect the set of stable nodes (e.g., intervals C and
D), but installing buggy software has a longer-term
affect on stability. Interval C shows the release of
PlanetLab 3.0. Although the release had undergone
extensive off-line testing, it had bugs and a relatively
long period of instability followed. PlanetLab was
still usable during this period but nodes rebooted at
least once per month.

• The pl mom watchdog is not perfect. There is a
slight dip in the number of stable core nodes (bottom
line) in interval D, when about 50 nodes were re-
booted because of a slice with a fast memory leak; as
memory pressure was already high on those nodes,
pl mom could not detect and reset the slice before
the nodes ran out of memory.

We note, however, that the 3.2 software release from late
December 2005 is the best so far in terms of stability:
as of February 2006, about two-thirds of active Planet-
Lab nodes have been up for at least a month. We at-
tribute most of this to abandoning CKRM in favor of the
Vservers native resource management framework and a
new CPU scheduler.

One surprising fact to emerge from Figure 9 is that a
lot of PlanetLab nodes are dead (denoted by the differ-
ence between the top and middle lines). Research orga-
nizations gain access to PlanetLab simply by hooking up
two machines at their local site. This formula for growth

has worked quite well: a low barrier-to-entry provided
the right incentive to grow PlanetLab. However, there
have never been well-defined incentives for sites to keep
nodes online. Providing such incentives is obviously the
right thing to do, but we note that that the majority of
the off-line nodes are at sites that no longer have active
slices—and at the time of this writing only 12 sites had
slices but no functioning nodes—so it’s not clear what
incentive will work.

Now that we have reached a fairly stable system, it be-
comes interesting to study the “churn” of nodes that are
active yet are not included in the stable core. We find it
useful to differentiate between three categories of nodes:
those that came up that day (and stayed up), those that
went down that day (and stayed down), and those that
have rebooted at least once. Our experience is that, on
a typical day, about 2 nodes come up, about 2 nodes go
down, and about 4 nodes reboot. On 10% of days, at least
6 nodes come up or go down, and at least 8 nodes reboot.

Looking at the archives of the support and
planetlab-users mailing lists, we are able to iden-
tify the most common reasons nodes come up or go down:
(1) a site takes its nodes offline to move them or change
their network configuration, (2) a site takes its nodes
offine in response to a security incident, (3) a site acci-
dently changes a network configuration that renders its
nodes unreachable, or (4) a node goes offline due to a
hardware failure. The last is the most common reason
for nodes being down for an extended period of time; the
third reason is the most frustrating aspect of operating a
system that embeds its nodes in over 300 independent IT
organizations.

Understanding the relative frequency of different sorts
of site events may be important for designers of other
large-scale distributed systems; this is a topic for further
study.

5.3 Security Complaints

Of the operational issues that PlanetLab faces, respond-
ing to security complaints is perhaps the most interesting,
if only because of what they say about the current state of
the Internet. We comment on three particular types of
complaints.

The most common complaints are the result of IDS
alerts. One frequent scenario corresponds to a perceived
DoS attack. These are sometimes triggered by a poorly
designed experiment (in which case the responsible re-
searchers are notified and expected to take corrective ac-
tion), but they are more likely to be triggered by totally
innocent behavior (e.g., 3 unsolicited UDP packets have
triggered the threat of legal action). In other cases, the
alerts are triggered by simplistic signitures for malware
that could not be running on our Linux-based environ-
ment. In general, we observe that any traffic that devi-

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 363

ates from a rather narrow range of acceptable behavior is
increasingly viewed as suspect, which makes innovating
with new types of network services a challenge.

An increasingly common type of complaint comes
from home users monitoring their firewall logs. They see
connections to PlanetLab nodes that they do not recog-
nize, assume PlanetLab has installed spyware on their
machines, and demand that it be removed. In reality,
they have unknowingly used a service (e.g., a CDN) that
has imposed itself between them and a server. Receiving
packets from a location service that also probes the client
to select the most appropriate PlanetLab node to service
a request only exacerbates the situation [35, 9]. The take-
away is that even individual users are becoming increas-
ingly security-senstive (if less security-sophisticated than
their professional counterparts), which makes the task of
deploying alternative services increasingly problematic.
Finally, PlanetLab nodes are sometimes identified as

the source or sink of illegal content. In reality, the con-
tent is only cached on the node by a slice running a CDN
service, but an overlay node looks like an end node to
the rest of the Internet. PlanetLab staff use PlanetFlow
to identify the responsible slice, which in turn typically
maintains a log that can be used to identify the ultimate
source or destination. This information is passed along
to the authorities, when appropriate. While many hosting
sites are justifiably gun-shy about such complaints, the
main lesson we have learned is that trying to police con-
tent is not a viable solution. The appropriate approach
is to be responsive and cooperative when complaints are
raised.

6 Discussion
Perhaps the most fundamental issue in PlanetLab’s de-
sign is how to manage trust in the face of pressure to de-
centralize the system, where decentralization is motivated
by the desire to (1) give owners autonomous control over
their nodes and (2) give third-party service developers the
flexibility they need to innovate.

At one end of the spectrum, individual organizations
could establish bilateral agreements with those organiza-
tions that they trust, and with which they are willing to
peer. The problem with such an approach is that reaching
the critical mass needed to foster a large-scale deploy-
ment has always proved difficult. PlanetLab started at the
other end of the spectrum by centralizing trust in a sin-
gle intermediary—PLC—and it is our contention that do-
ing so was necessary to getting PlanetLab off the ground.
To compensate, the plan was to decentralize the system
through two other means: (1) users would delegate the
right to manage aspects of their slices to third-party ser-
vices, and (2) owners would make resource allocation de-
cisions for their nodes. This approach has had mixed suc-
cess, but it is important to ask if these limitations are fun-

damental or simply a matter of execution.
With respect to owner autonomy, all sites are allowed

to set bandwidth caps on their nodes, and sites that have
contributed more than the two-node minimum required
to join PlanetLab are allowed to give excess resources
to favored slices, including brokerage services that redis-
tribute those resources to others. In theory, sites are also
allowed to blacklist slices they do not want running lo-
cally (e.g., because they violate the local AUP), but we
have purposely not advertised this capability in an effort
to “unionize” the research community: take all of our ex-
periments, even the risky ones, or take none of them. (As
a compromise, some slices voluntarily do not run on cer-
tain sites so as to not force the issue.) The interface by
which owners express their wishes is clunky (and some-
times involves assistance from the PlanetLab staff), but
there does not seem to be any architectural reason why
this approach cannot provide whatever control over re-
source allocation that owners require (modulo meeting
requirements for joining PlanetLab in the first place).
With respect to third-party management services, suc-

cess has been much more mixed. There have been some
successes—Stork, Sirius, and CoMon being the most no-
table examples—but this issue is a contentious one in the
PlanetLab developer community. There are many pos-
sible explanations, including there being few incentives
and many costs to providing 24/7 management services;
users preferring to roll their own management utilities
rather than learn a third-party service that doesn’t exactly
satisfy their needs; and the API being too much of a mov-
ing target to support third-party development efforts.
While these possibilities provide interesting fodder for

debate, there is a fundamental issue of whether the cen-
tralized trust model impacts the ability to deploy third-
party management services. For those services that re-
quire privileged access on a node (see Section 3.5) the an-
swer is yes—the PLC support staff must configure Proper
to grant the necessary privilege(s). While in practice such
privileges have been granted in all cases that have not vio-
lated PlanetLab’s underlying trust assumptions or jeopar-
dized the stability of the operational system, this is clearly
a limitation of the architecture.
Note that choice is not just limited to what manage-

ment services the central authority approves, but also
to what capabilities are included in the core system—
e.g., whether each node runs Linux, Windows, or Xen.
Clearly, a truly scalable system cannot depend on a sin-
gle trusted entity making these decisions. This is, in fact,
the motivation for evolving PlanetLab to the point that it
can support federation. To foster federation we have put
together a software distribution, called MyPLC, that al-
lows anyone to create their own private PlanetLab, and
potentially federate that PlanetLab with others (including
the current ”public” PlanetLab).

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association364

This returns us to the original issue of centralized ver-
sus decentralized trust. The overriding lesson of Plan-
etLab is that a centralized trust model was essential to
achieving some level of critical mass—which in turn al-
lowed us to learn enough about the design space to define
a candidate minimal interface for federation—but that it
is only by federating autonomous instances that the sys-
tem will truly scale. Private PlanetLabs will still need bi-
lateral peering agreements with each other, but there will
also be the option of individual PlanetLabs scaling inter-
nally to non-trivial sizes. In other words, the combination
of bilateral agreements and trusted intermediaries allows
for flexible aggregation of trust.

7 Conclusions
Building PlanetLab has been a unique experience. Rather
than leveraging a new mechanism or algorithm, it has re-
quired a synthesis of carefully selected ideas. Rather than
being based on a pre-conceived design and validated with
controlled experiments, it has been shaped and proven
through real-world usage. Rather than be designed to
function within a single organization, it is a large-scale
distributed system that must be cognizant of its place in
a multi-organization world. Finally, rather than having to
satisfy only quantifiable technical objectives, its success
has depended on providing various communities with the
right incentives and being equally responsive to conflict-
ing and difficult-to-measure requirements.

Acknowledgments
Many people have contributed to PlanetLab. Timothy
Roscoe, Tom Anderson, and Mic Bowman have provided
significant input to the definition of its architecture. Sev-
eral researchers have also contributed management ser-
vices, including David Lowenthal, Vivek Pai and Ky-
oungSoo Park, John Hartman and Justin Cappos, and Jay
Lepreau and the Emulab team. Finally, the contributions
of the PlanetLab staff at Princeton—Aaron Klingaman,
Mark Huang, Martin Makowiecki, Reid Moran, Faiyaz
Ahmed, Brian Jones, and Scott Karlin—have been im-
measurable.
We also thank the anonymous referees, and our shep-

herd, Jim Waldo, for their comments and help in improv-
ing this paper.
This work was funded in part by NSF Grants CNS-

0520053, CNS-0454278, and CNS-0335214.

References
[1] ANNAPUREDDY, S., FREEDMAN, M. J., AND MAZIERES, D. Shark: Scal-

ing File Servers via Cooperative Caching. In Proc. 2nd NSDI (Boston, MA,
May 2005).

[2] AUYOUNG, A., CHUN, B., NG, C., PARKES, D., SHNEI-
DMAN, J., SNOEREN, A., AND VAHDAT, A. Bellagio: An
Economic-Based Resource Allocation System for PlanetLab.
http://bellagio.ucsd.edu/about.php.

[3] BAVIER, A., BOWMAN, M., CULLER, D., CHUN, B., KARLIN, S., MUIR,
S., PETERSON, L., ROSCOE, T., SPALINK, T., AND WAWRZONIAK, M.
Operating System Support for Planetary-Scale Network Services. In Proc.
1st NSDI (San Francisco, CA, Mar 2004).

[4] BAVIER, A., FEAMSTER, N., HUANG, M., PETERSON, L., AND REX-
FORD, J. In VINI Veritas: Realistic and Controlled Network Experimenta-
tion. In Proc. SIGCOMM 2006 (Pisa, Italy, Sep 2006).

[5] BRETT, P., KNAUERHASE, R., BOWMAN, M., ADAMS, R., NATARAJ,
A., SEDAYAO, J., AND SPINDEL, M. A Shared Global Event Propaga-
tion System to Enable Next Generation Distributed Services. In Proc. 1st
WORLDS (San Francisco, CA, Dec 2004).

[6] CLARK, D. D. The Design Philosophy of the DARPA Internet Protocols.
In Proc. SIGCOMM ’88 (Stanford, CA, Aug 1988), pp. 106–114.

[7] DAVID LOWENTHAL. Sirius: A Calendar Service for PlanetLab.
http://snowball.cs.uga.edu/dkl/pslogin.php.

[8] FREEDMAN, M. J., FREUDENTHAL, E., AND MAZIERES, D. Democratiz-
ing content publication with Coral. In Proc. 1st NSDI (San Francisco, CA,
Mar 2004).

[9] FREEDMAN, M. J., LAKSHMINARAYANAN, K., AND MAZIERES, D. OA-
SIS: Anycast for Any Service. In Proc. 3rd NSDI (San Jose, CA, May
2006).

[10] FU, Y., CHASE, J., CHUN, B., SCHWAB, S., AND VAHDAT, A. SHARP:
An Architecture for Secure Resource Peering. In Proc. 19th SOSP (Lake
George, NY, Oct 2003).

[11] HUANG, M., BAVIER, A., AND PETERSON, L. PlanetFlow: Maintaining
Accountability for Network Services. ACM SIGOPS Operating Systems
Review 40, 1 (Jan 2006).

[12] JUSTON CAPPOS AND JOHN HARTMAN. Stork: A Soft-
ware Packagement Management Service for PlanetLab.
http://www.cs.arizona.edu/stork.

[13] KOHLER, E., MORRIS, R., CHEN, B., JANNOTTI, J., AND KAASHOEK,
M. F. The Click Modular Router. ACM Transactions on Computer Systems
18, 3 (Aug 2000), 263–297.

[14] LAI, K., RASMUSSON, L., ADAR, E., SORKIN, S., ZHANG, L., AND
HUBERMAN, B. A. Tycoon: An Implemention of a Distributed Market-
Based Resource Allocation System. Tech. Rep. arXiv:cs.DC/0412038, HP
Labs, Palo Alto, CA, USA, Dec. 2004.

[15] LINUX ADVANCED ROUTING AND TRAFFIC CONTROL.
http://lartc.org/.

[16] LINUX VSERVERS PROJECT.
http://linux-vserver.org/.

[17] MUIR, S., PETERSON, L., FIUCZYNSKI, M., CAPPOS, J., AND HART-
MAN, J. Privileged Operations in the PlanetLab Virtualised Environment.
SIGOPS Operating Systems Review 40, 1 (2006), 75–88.

[18] OPPENHEIMER, D., ALBRECH, J., PATTERSON, D., AND VAHDAT, A.
Distributed Resource Discovery on PlanetLab with SWORD. In Proc. 1st
WORLDS (San Francisco, CA, 2004).

[19] PARK, K., AND PAI, V. S. Scale and Performance in the CoBlitz Large-File
Distribution Service. In Proc. 3rd NSDI (San Jose, CA, May 2006).

[20] PARK, K., PAI, V. S., PETERSON, L. L., AND WANG, Z. CoDNS: Im-
proving DNS Performance and Reliability via Cooperative Lookups. In
Proc. 6th OSDI (San Francisco, CA, Dec 2004), pp. 199–214.

[21] PETERSON, L., ANDERSON, T., CULLER, D., AND ROSCOE, T. A
Blueprint for Introducing Disruptive Technology into the Internet. In Proc.
HotNets–I (Princeton, NJ, Oct 2002).

[22] PETERSON, L., BAVIER, A., FIUCZYNSKI, M., MUIR, S., AND ROSCOE,
T. PlanetLab Architecture: An Overview. Tech. Rep. PDN–06–031, Plan-
etLab Consortium, Apr 2006.

[23] RAMASUBRAMANIAN, V., PETERSON, R., AND SIRER, E. G. Corona: A
High Performance Publish-Subscribe System for the World Wide Web. In
Proc. 3rd NSDI (San Jose, CA, May 2006).

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and ImplementationUSENIX Association 365

[24] RAMASUBRAMANIAN, V., AND SIRER, E. G. Beehive: O(1) Lookup Per-
formance for Power-Law Query Distributions in Peer-to-Peer Overlays. In
Proc. 1st NSDI (San Francisco, CA, Mar 2004), pp. 99–112.

[25] RAMASUBRAMANIAN, V., AND SIRER, E. G. The Design and Imple-
mentation of a Next Generation Name Service for the Internet. In Proc.
SIGCOMM 2004 (Portland, OR, Aug 2004), pp. 331–342.

[26] RHEA, S., GODFREY, B., KARP, B., KUBIATOWICZ, J., RATNASAMY,
S., SHENKER, S., STOICA, I., AND YU, H. OpenDHT: A Public DHT
Service and its Uses. In Proc. SIGCOMM 2005 (Philadelphia, PA, Aug
2005), pp. 73–84.

[27] RITCHIE, D. M., AND THOMPSON, K. The UNIX Time-Sharing System.
Communications of the ACM 17, 7 (Jul 1974), 365–375.

[28] RYAN HUEBSCH. PlanetLab Application Manager.
http://appmanager.berkeley.intel-research.net/.

[29] SPRING, N., BAVIER, A., PETERSON, L., AND PAI, V. S. Using PlanetLab
for Network Research: Myths, Realities, and Best Practices. In Proc. 2nd
WORLDS (San Francisco, CA, Dec 2005).

[30] SPRING, N., WETHERALL, D., AND ANDERSON, T. Scriptroute: A Public
Internet Measurement Facility. In Proc. 4th USITS (Seattle, WA,Mar 2003).

[31] VIVEK PAI AND KYOUNGSOO PARK. CoMon: A Monitoring Infrastruc-
ture for PlanetLab. http://comon.cs.princeton.edu.

[32] WALDSPURGER, C. A., AND WEIHL, W. E. Lottery Scheduling: Flexible
Proportional-Share Resource Management. In Proc. 1st OSDI (Monterey,
CA, Nov 1994), pp. 1–11.

[33] WANG, L., PARK, K., PANG, R., PAI, V. S., AND PETERSON, L. L. Reli-
ability and Security in the CoDeeN Content Distribution Network. In Proc.
USENIX ’04 (Boston, MA, Jun 2004).

[34] WHITE, B., LEPREAU, J., STOLLER, L., RICCI, R., GURUPRASAD, S.,
NEWBOLD, M., HIBLER, M., BARB, C., AND JOGLEKAR, A. An Inte-
grated Experimental Environment for Distributed Systems and Networks.
In Proc. 5th OSDI (Boston, MA, Dec 2002), pp. 255–270.

[35] WONG, B., SLIVKINS, A., AND SIRER, E. G. Meridian: A Lightweight
Network Location Service without Virtual Coordinates. InProc. SIGCOMM
2005 (Philadelphia, PA, Aug 2005).

[36] ZHANG, M., ZHANG, C., PAI, V. S., PETERSON, L. L., AND WANG,
R. Y. PlanetSeer: Internet Path Failure Monitoring and Characterization in
Wide-Area Services. In Proc. 6th OSDI (San Francisco, CA, Dec 2004),
pp. 167–182.

OSDI ’06: 7th USENIX Symposium on Operating Systems Design and Implementation USENIX Association366