Table Of ContentA Peer-to-Peer Browsable File Index using a
Popularity Based Global Namespace
Thomas Jacobs and Aaron Harwood
Department of Computer Science & Software Engineering
7 University of Melbourne
0
Victoria, 3010, AUSTRALIA.
0
2
February 1, 2008
n
a
J
Abstract
0 ing [29].
3
] The distribution of files using decentralized, peer-to- The P2P paradigmis loosely characterizedby an ap-
C peer(P2P)systems,hassignificantadvantagesovercen- plication network in which a significant proportion of
tralizedapproaches. Itishowevermoredifficulttosettle
D the application’s functionality is implemented by peers
on the best approach for file sharing. Most file sharing
. in a decentralized way, rather than being implemented
s systems are based on query string searches, leading to
c bycentralized servers[2]. P2Pfile sharing systemscon-
a relatively simple but inefficient broadcastor to an ef-
[ sist of program(s)that are used to create and maintain
ficient but relatively complicated index in a structured
P2P networks to facilitate the sharing of files between
1
environment. In this paper we use a browsable peer-
v users; they allow users to designate a set of files from
to-peer file index consisting of files which serve as di-
0 their PC’sfile systemto be sharedandthey allowusers
9 rectory nodes, interconnecting to form a directory net- to download shared files from other users of the P2P
1 work. We implementedthe systembasedonBitTorrent
network.
1 andKademlia. Thedirectorynetworkinheritsallofthe
0
advantages of decentralization and provides browsable,
7 There are two key parts of a P2P file sharing sys-
efficient searching. To avoid conflict between users in
0
tem. The first partis the file distribution systemwhich
/ the P2P system while also imposing no additional re-
s provides the means to transmit files between peers; it
strictions, we allow multiple versions of each directory
c
dictates how peers in the system should behave in or-
: node to simultaneously exist – using popularity as the
v dertodownloadanduploadfiles. Thesecondpartisthe
basis for default browsing behavior. Users can freely
i
X add files and directory nodes to the network. We show, filediscovery systemwhichisthemeansforuserstofind
thefilesthatareavailableontheP2Pnetwork. P2Pfile
r usingasimulationofuserbehaviorandfilequality,that
a sharing systems typically provide the file discoverysys-
thepopularitybasedsystemconsistentlyleadsuserstoa
tembymaintainingsomeformofindexofthefiles. P2P
high quality directorynetwork;abovethe averagequal-
filesharingsystemsdifferinhowandwheretheyimple-
ity of user updates.
ment these two parts. Some maintain the file index in
a centralized way, and others in a decentralized way;
1 Introduction some indexes are structured,i.e. provideefficient query
processing, and some are unstructured with inefficient
query processing. P2P file sharing systems implement
Peer-to-peer (P2P) file sharing systems have steadily
the file distribution system in a decentralized way; this
grown in usage – the Internet traffic generated in total
is unequivocally the original defining trait.
by only seven P2P file sharing systems was reported to
haveoutgrownwebtrafficin2002andincreasedtoover
half of all Internet traffic by the end of 2004 [15, 4]. In this paper we describe our experimental file shar-
There are more than one hundred P2P file sharing sys- ing system, called Localhost, that combines a directory
tems listed online and file sharing is the most widely node approach for file indexing with a novel popular-
used application among other emerging applications of ity based namespace. We show how the popularity
P2P including Internet telephony [26], instant messag- basednamespaceprovidesawayfordecentralizedmain-
ing [13], grid computing [28], and decentralized gam- tenance to lead to a high quality directory network.
1
1.1 Query strings and browsing mous with sharing or adding a file to the network.
Query string search is characterized as a process in
which the user describes a request by forming a query 1.2 Adding files to the network
string that consists of one or more keywords and the
system presents a set of filenames that match or sat- A widely used method for adding files to the network
isfy the query string. The kind of index is transparent is unstructured sharing, where users designate a folder
to the user. Keyword search suffers from a vocabulary in their local file system and have all of its contents
differences barrier,also referredto as the semantic bar- shared. The user shares files obliviously to what other
rier (Nadis, 1996),betweenthe publisher of the file and files are being shared and in many cases the files that
the user wishing to download the file. Keyword search are downloaded by the user from another peer are also
is most suitable when the user has an idea of what files put into the user’s local shared files folder. There is no
theywantfromthesystembeforehand. Keywordsearch notion of a global namespace or index of all files in the
islesssuitableforpresentingnewthingstousers,asthey network.
have to enter a specific query before being presented Two major problems that occur in P2P file sharing
with a list of available files. systemsthatuseunstructuredsharingarepollution and
Browsing is an alternate approach to find files. The poisoning [8]. Pollution of a P2P network is the acci-
user is presented with a list of available files to select dental injection of unusable copies of files into the net-
from. The index is seen by the user and typically pro- work,bynon-malicioususers. Poisoningiswherealarge
vides some categorization that allows the user to make numberoffake files aredeliberatelyinjectedintoaP2P
amoreefficientselection. Thisapproachdoesnotsuffer network by malicious users or groups. Fake files are
from the semantic barrier because browsing presents a specificallycreatedbymalicioususersorgroupstoseem
list of all available files. However the user may spend like certain files, but consist of rubbish data or are un-
moretimemakingaselection,especiallywhenthelistis usable in some way. Both of these problems reduce the
large and the index is flat, as compared to using query perceived availability of files to users and reduce the
strings. usefulness of the system to users, because discovering
The majority of P2P file sharing systems use query usable files is more difficult. A study [17] found that a
string search. Napster [5], Gnutella-based [24] systems, significantproportionoffilesontheKaZaAnetworkare
eMule [14], and KaZaA [17] currently use query string unusable, due to poisoning and pollution. A number of
search as their only means for finding files in their net- P2P file sharing systems employ a file rating system in
works. Some P2P file sharing systems allow the user responsetotheseproblems. Fileratingsystemsletusers
to browse each individual peer’s shared files. However, rate each file’s quality - the theory is that enough users
these systems do not support a browsable namespace will find the fake/unusable files and rate them poorly,
that is global among all peers because they do not di- allowing other users to identify them before download-
rectly provide a way of collaboratively organizing files ing them. These file ratingsystems havebeenshownto
into a single, integrated, coherent categorical or hier- be largely ineffective [17].
archical structure. Consequently, over 25 terabytes of The BitTorrent protocol specifies only file download-
files are fragmented across more than 8000 individual ing, but a file sharing system is nonetheless being used
listings,witheachlistinghavingitsownwayoforganiz- which is supported by the protocol. Any user can sub-
ing its files [21]. mit files to an index website, and the file is checked by
The Freenet system [9] takes an unusual, but neces- themoderatorsofthewebsitebeforebeingaddedtothe
sary due to the anonymity property, approach of pro- website’s index. If the file is found to be fake or of un-
viding directory nodes that serve, like hyper-text doc- usable quality, it is not added to the index. Although
uments, to point to other files in the peer-to-peer net- pollution and poisoning levels are difficult to measure,
work, thereby forming a browsable structure called a sources indicate that the the effective BitTorrent file
directory network, like the web. Leavesor end points of sharing system is virtually pollution and poisoning free
the directory network are regular files. In our work,we because of this scheme [22]. While this system is work-
use this concept in a more general way, transparently able, it relies on a central server and it is difficult to
applying it to an existing P2P file distribution proto- decentralized the moderation process, i.e. to allow all
col. Freenetdoesnotallowdifferentusersofthesystem users to participate as moderators.
to write to the same name in the namespace and this In our work we make use of a global namespace to
leadsto a “namerace”situationwhere the firstto pub- store the directory network of shared files. The global
lishunder a name willownthatname. We considerthe namespace is a set of names which are consistently re-
casewhenmultiplevaluesofthesamekeycanexist,and ferred to by all peers in the network; each directory
what kind of directory structure would result. In this node and file has a name in the namespace. A number
context, writing to a name in the namespace is synony- of structured peer-to-peer protocols are available that
2
maintain a global namespace. The Kademlia protocol 2.1.1 BitTorrent protocol
is used by Azureus which we make use of in our imple-
The BitTorrent protocol is designed and used for P2P
mentation. While we considered a number of existing
file distribution [3, 10]. Following the BitTorrent pro-
shared access control methods, such as web based own-
ership(thenamespaceisassociatedwithIPaddressesof tocol, a file, F = {f1,f2,...}, is broken up into pieces
which are transmitted between peers. Piece size is usu-
userswhocanmodifyonlythosepartsofthenamespace
ally between 32 kilobytes and 128 kilobytes. A torrent
that they control)anddelegatedauthority (like the do-
file, T, is used to publish a file or collection of files and
main name system where access is administered and
contains:
delegated down through a hierarchical authority), we
proposedanewmethodbasedonpopularity. Inourim-
• the name(s) of the file(s), F1,F2,...,
plementation, all users are allowed to submit their own
version of the content for a given name in the names- • the SHA-1 hash, H(•), of every piece of every file,
pace;thismethodisappealingbecausetheusersarestill
effectively acting obliviously to each other. The system • the torrentfile’s infohash which is the SHA-1 hash
naturally displays the most popular version to the user of the concatenation of the file(s), H(F1F2···) (or
(in the case when the user has no version viewing pref- just H(F) for a single file) and
erence) when the content for that name is requested,
• the web address for one or more trackers.
which is computed as the version that most users are
currentlyviewing. Theusercanoptionallyviewallver-
The infohash is used to uniquely identify a torrent file.
sions and make a different selection at their discretion.
The term torrent refers to the collection of file(s) that
the torrent file was created from. From now on, with-
1.3 Our contribution out loss in generality, we will assume that the torrent
contains only a single file. A tracker is a server that
In our work, we apply the concept of a decentralized maintains a list of IP addresses of peers in the swarm.
directorynetwork,usingdirectorynodesthataretrans- Theswarmisthesetofpeerscurrentlyinvolvedintrans-
parently distributed by an existing P2P file distribu- mitting pieces of the file to each other. The torrent file
tion protocol. We further show that a popularity based is distributed in full between users by some means ex-
namespace can be used to provide a way for decentral- ternaltotheBitTorrentpeer,suchasviawebsites. The
izedmaintenancetoleadtoahighqualitydirectorynet- torrentfileisinputtotheBitTorrentprotocolandisre-
work. quired for the protocol to download the file; given the
torrent file the peer contacts one or more of the listed
trackers to obtain the IP address and port numbers of
2 The Localhost system
other peers that are seeding the file. The user publish-
ing the file acts as the initial seed and initially, there is
In this section we describe the details of the Localhost one seed in the swarm. As peers in the swarm obtain
system, depicted in Figure 1, to provide a context for pieces of the file, they become seeds for these pieces as
the popularity based namespace concept. At the high- well.
est level, the Localhost peer is a modification of the
Azureuspeer;the modificationsinclude additionaldata
2.1.2 Kademlia protocol
operations and an embedded HTTP server to provide
web browser based interaction. We use the term Lo- Kademlia [18] is one of many DHT based protocols,in-
calhost distributed system (LDS) to refer to the system cluded among some of the most well known such as
that is created by the interconnection of a number of Chord [27], CAN [23], Pastry [25], and Tapestry [31].
Localhost peers. A DHT is globalnamespace where eachpeer maintains
somepartofthespace. ThetwomajorDHToperations
that we consider are:
2.1 Underlying protocols
• put(k,v) – stores the data string v under key k in
Our work builds from a number of technologies and in
the DHT.
this section we abstract the details that are sufficient
to understand our modifications that were applied to • v ← get(k) – retrieves the data string v from the
build the Localhost system. The technologies include
DHT that is stored under the key k.
the BitTorrent file distribution protocol, the Kademlia
distributedhashtable(DHT),andAzureus–auserap- Note that some DHTs, including Kademlia, allow
plication which combines both BitTorrent and Kadem- multiple data strings to be stored under, and retrieved
lia. from,asinglekey. Inthiscase,v isasetofdatastrings.
3
ts Localhost (modified Azureus) peer
2. get(H(nodename))
current directory node
3. list of versions
Kademlia 1. node name
4. get(H(version))
protocol contents of current
cache embedded
5. torrent file 8. node display directory node
subsystem HTTP server
6. torrent file Localhost control logic
BitTorrent
7. file content list of other versions
protocol
with viewing counters
Figure 1: Localhost system overview,showing example process and salient parts of the web browser interface.
DHT based systems operate in a completely decen- Version 2.3 of Azureus also introduces a torrent file
tralizedway. DHTprotocolsareabletoprovidethetwo downloadfunctionwhichallowstorrentfilestobedown-
operations described above by making the peers form a loaded from remote peers: T ← get(H(F)). The tor-
DHT overlay network. The DHT overlay network is rent file T is then given to the BitTorrent protocol.
formed by each peer maintaining a set of contacts. A TheKademliaimplementationinAzureusallowseach
contact is the peer ID and IP addresses of a remote peer to store a single value only, under each key. The
peer in the DHT. Each peer has a peer ID, which is a peer associates each stored value with the associated
number chosen from the namespace of keys. The set peer’s network address. When a peer executes the se-
of contacts each peer maintains does not include every quence
possiblecontactintheDHT.ThespecificDHTprotocol
useddictateswhichcontactseachpeermaintains. Using put(k,v1),put(k,v2),v ←get(k)}
thesecontacts,DHToverlaynetworkssuchasKademlia
alloweachpeertolocatetheremotepeerresponsiblefor then v =v2 is the result. In other words multiple peers
a certain key in O(logn) time. Once the correct peer can each store a different value under the same key. A
has been located, the put and get operations can be singlepeercanstoredifferentvaluesonlyunderdifferent
done by contacting that peer. keys.
2.1.3 Azureus application 2.2 Localhost concepts
AzureusisaJavaimplementationoftheBitTorrentpro- In the following sections we describe the concepts that
tocol while allows a number of files to be downloaded we proposed and implemented using the previous tech-
and seeded concurrently. From version 2.3, Azureus nologies.
also includes an implementation of the Kademlia pro-
tocol. All Azureus peers join the same DHT, by con- 2.2.1 Interpreting files as directory nodes
tactingacertainpeerthatissetupforthepurposethat
aims to always be online. Azureus uses the Kademlia We adopted a directory node approach similar to the
DHT to implement a feature called decentralized track- onetakenbyFreenet. Thebasicideaistohavethepeer
ing. Decentralized tracking is an optional replacement interpret some files as directory nodes, these files con-
for BitTorrenttrackers. When decentralizedtracking is tainanindexofdirectorynodenamesand/orfilenames.
enabled, an Azureus peer executes the operation: The peer displays this directory to the user and allows
further selection. Interestingly, because the directory
put(H(F),(IP,port)) nodes are distributed in a decentralized way using the
underlying file distribution protocol, the directory net-
for each file, F, that it is seeding; where H(F) is the workinheritsthisproperty,alsobecomingdistributedin
infohash for the file and (IP,port) is the peer’s IP ad- a decentralized way. Another benefit of this approach
dressanditsBitTorrentprotocolportnumber. Multiple for indexing files is that it can be appliedobliviously to
peerscanstoretheir (IP,port) informationatthe same the existing file distribution protocol.
key. Given H(F) for a file, a peer can then execute Unlike web files which are served from centralized
v ← get(H(F)) to obtain a list of other peers in the servers, P2P files are served potentially from multiple
swarm for that file. The set of values in v is given to peers. Also, for files to be shared via the directory
the BitTorrent protocol. This use of a DHT replaces network the peers must be able to add to or modify
the use of trackercommunicationdone by the standard the directory nodes. In our case, the use of a tor-
BitTorrent protocol. rent file proved problematic because of its use of hash
4
functions. Consider a torrent file that contains, among operate independently of each other.
other things, the infohash of the file to be downloaded, We considered the use of delegated authority, where
T = {H(F)}; T is required to download F. It would the entire namespace is initially owned by a single au-
be desirable to define a directory node by inclusion of thorityandpermissiontomodifypartsofitisdelegated
the torrent files for the files that are indexed by that onrequest;e.g. likethedomainnamesystem. Wecould
directory node. However, we cannot define a directory implement this approach by using a decentralized web
node to contain torrent files because then for two di- of trust model. However, this approach does not com-
rectory nodes, F1 and F2, we would have a circular ref- pletely absolve peers from each other.
erence, where F1 = {T2}, F2 = {T1}, T2 = {H(F1)} To adhere to the P2P paradigm, we proposed and
and T1 = {H(F2)}; hence T1 = {H({T1})}. This prob- implemented the notion of versions. Each peer writes
lem exists for any P2P protocol that identifies files by its own version of a given name in the namespace. We
using hash functions. Let us restrictthe network struc- make use of the DHT ability for multiple peers to each
ture to that of tree. The problem is then reduced to store a value at the same name in the namespace. A
one of efficiency, since for a chain of directory nodes, version of a file is uniquely identified by the infohash
F1,F2,...,Fl, a modification to Fl requires a modifica- of that file. For the purpose of user selection, we also
tiontoFl−1 andsoonbacktoF1. Thusarootdirectory store a textual description of each version along with
node would be modified for every modification to a di- its infohash. So for a given file version, F, and a node
rectory node beneath it in the tree. name for that file, a peer executes
Due to these relationships we separated the textual
names of nodes, callednode names,from their infohash put(H(nodename),(description,H(F)))
andmakeuseofatwostepprocess. Adirectorynodeis
defined as containing the node names of the nodes that to register this version to the DHT. The peer of course
it indexes. The process of getting a node is numbered must be seeding F. A get(H(nodename)) will proceed
in Figure 1. A user request for a node name generates as discussedin the previous section to return the list of
aget(H(nodename))whichreturnsalistofversions of all versions; and a selected version can be downloaded.
the node with that name. Versions are discussed in the Becausethelistofversionscouldbeaslargeasthenum-
next section. Assuming a version is selected the peer ber of peers, we use a downloadtime limit to download
executes a get(H(version)) to obtain a list of other only a portion of the list relative to the speed of the
peers that are seeding that file; any of these peers can download. If the peer has not viewed that node before,
be contactedforthetorrentfile. Thetorrentfile isthen then its viewing preference is automatically set to the
given to the BitTorrent protocol to obtain the file con- versions which is most popular (inferred from the sam-
tent which is returned to the user. If the file is a direc- ple of versions collected in the download time limit). If
tory node then the directory structure is displayed and the peer has viewed that node before, then the viewing
the user continues to make selections. preferenceiswhateverversionofthatnodethepeerlast
viewed. A cache is used to maintain previously viewed
versions. This mechanism is the essential aspect of the
2.2.2 Node versions and popularity
popularity based namespace.
Notethatregisteringaversioniseffectivelya“prefer-
Systemsthatuseaglobalnamespaceshouldspecifyhow
ence” for that version of the file or name in the names-
users can modify the namespace in order to add the
files that they wish to share; this immediately poses pace. As a consequence of the DHT allowing each peer
to write only one value for a given key, each peer can
theproblemsofsharedaccesscontrolwhentwoormore
users want to modify content with the same name in setapreferencefor atmostoneversionofeachname in
the namespace.
the namespace. In our case the namespace is the DHT
space provide by Kademlia. The Localhost peer provides appropriate web forms
to the web browser for the user to edit any currently
Interestingly the web uses a kind of global names-
viewed version of a directory or file node, providing a
pace, the set of uniform resource locations, consisting
new version to the system. When a peer downloads a
of an IP address and file name; users can only modify
versionitalsoregisterstheversion,sothatitcontributes
contentwiththenamesthattheyownontheirlocalfile
to the swarm of peers that share that version.
system. Outgoing connections can be easily made but
incomingconnectionsrequireexistinguserstoagreeand
to modify their existing files. Because of this, new con-
2.2.3 The user interface
tentmaynotbe linkedto forsometime ifthe publisher
of that content does not have agreements with existing The dynamics of a popularity based system are in part
publishers. We wanted to avoid this situation for pub- influencedbytheuserinterface,andwehaveconsidered
lishers of P2P files; peers should be able to effectively e.g. displaying the list of the most popular versions, a
5
list of all versions and a list of recently registered ver- We usually consider the case when the directory
sions. Discussion of the affect of the user interface on tree starts with V0 = {v1,1,v2,1,v3,1,v4,1}, E0(v1,1) =
the system is beyond the scope of this paper. {v2,1,v3,1,v4,1}, all nodes are directories (there are no
files yet) and the leaf directories have no connections.
Figure 2(c) (bottom) depicts the initial condition and
3 Simulations of popularity
an update, as explained in the next section.
based namespace In [7] page quality is defined as the fraction of to-
tal users that would like a page the first time they see
the page; page quality is an intrinsic property of the
In this section we show our simulation analysis of the
page. In our work, each node version v has a quality,
Localhost system with respect to use of a popularity
Q ∈[0,1). Thequalitydeterminestheprobabilitythat
based namespace. The goal of the analysis is to under- v
auserwillcontinue toview thatversionratherthanse-
stand the efficacy of the system, where we can loosely
lecting a new version to view; i.e. the probability of
saythatthesystemisintendedtoprovidepeerswithan
selecting a new version is 1−Q and this test is made
ability to efficiently share files; we further refine this to v
each time the user views that version. The quality of
mean high quality files. Broadly speaking, the system
a node is determined by a random variable and is set
should be malleable in the sense that it canboth admit
whenthe node is createdbyauser. Inouranalysis,the
a large number of peer updates while remain coherent
quality is independent of all other nodes (including the
or stable in structure.
version it was derived from) and is independent of the
In our analysis, we generated a user model that rep-
user creating the version. Thus, any changes to a node
resents user behavior. For this approach it is necessary
mayinduce anarbitraryincreaseordecreaseinquality.
to make assumptions about user behavior and also to
We use the cumulative distribution function
adopt a clear definition of quality with respect to user
updates, e.g. the quality of a file or directory node up- P Qv <q =q1s,
date. The main simplification is that we consider the
(cid:2) (cid:3)
case when directory nodes can form connections only wheres>0isaconstantthatdescribesthefrequencyof
in such a way that a tree network is formed, i.e. new high quality updates compared to low quality updates;
versions of directory nodes can contain additional con- if s = 1 then all values of quality are equally likely.
nections only to new directories or files, not to existing In this work, we use p to represent a source of random
directories or files. From now on we talk about the di- numbersintherange[0,1). Thus,wechoosethequality
rectory tree. Our user model determines how a user of a node v using
i,j
traverses the directory tree, how they make selections
of possible versions and connections, how they choose Q =ps .
i,j i,j
to make updates and what quality those updates have.
To assess the malleability of the system we applied Figure2(a)givesexamplesforvariousvaluesofs. Note
our user model to a starting directory tree containing the expected number of nodes with quality in (a,b],
only a few nodes and measured properties of the re-
sulting directory tree that is evolved by user updates. E |{v |a≤Qv <b}| =b1s −as1,
(cid:2) (cid:3)
We measured the ability for high quality updates to be
and the average quality of the nodes:
“seen”byotherusersinthesystemandforhighquality
nodes to be viewed by the majority of users. In this 1
1
aconnaslyidseisrwtheecoevnosilduetirotnhaintuteprdmasteosfatrheesueqpudeantteianluamnbdewr.e E(cid:2)Qi,j(cid:3)=Z0 psdp= 1+s.
3.2 User behavior model
3.1 Directory tree and node quality
Sinceweareinterestedonlyintheevolutionofdirectory
The directory tree at time t consists of a set of node tree, we model the user process as an update. In this
versions, V where v = v ∈ V is the j-th version for work,the word user and peer is synonymous since each
t i,j t
the i-th node, were i = 1,2,...,n and j = 1,2,...,n . peeriscontrolledbyauniqueuser. Theuserstartsfrom
i
The time t represents the t-th update to the tree. The the root of the tree and navigates to a leaf. The user
type ofa node determines if itmaycontainconnections then chooses a node along the navigatedpath, to make
to other nodes or not; only directory nodes can contain an update yielding a new version of that node. User
connections. Each directory node has a set of connec- behavior is described by the parameters listed in Table
tions to other nodes, E (v) ⊂ V ; where E (v) = {} for 1. There are N users. The function γ is set to the
t t t i,u
all t if v is a file. File nodes naturally represent leaves useru’sversionviewingpreferencefornodei,itisunde-
in the structure. finedifthe useruhasnotyetviewednode i. Eachtime
6
⇒
1 4
p
update
0.8 0.25 3 4 no update padd
uality 0.6 0.5 1 hoice 2 1.5 addedleatedairelicntkoryaddpfailefile
q 0.4 2 c
0.5
4 1
0.2 v1,1 v1,1 v1,2
⇒
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
random value random value v2,1 v3,1 v4,1 v2,1 v3,1 v4,1 v5,1
(a) Quality versus random value (b) User choice versus random (c) Userdecisionprocess(top)and (d) Examplemaintreeofsize87
forvariouss valuefork=4andvariousδˆ exampleupdatefromtheinitialdi- nodesafter105 timesteps
rectorynetwork(bottom)
Figure 2: User model functions and examples
To model peer churn, we use the parameter p ;
leave
Table 1: User state and actions
which is the probability that a user leaves the network.
notation description
When u ∈N is chosen for a traversal, with probability
N the number of peers
p it will be “reset”. The reset erases all popularity
γ theversionofnodeithatuseruisviewing leave
i,u information in the directory tree (i.e. any versions that
p the probability that a user makes an up-
update u was viewing). After a reset the (new) peer continues
date for a given traversal
with the traversal. Thus, the number of available peers
p the probability that a user adds rather
add remainsataconstantN,butpeerseffectivelycomeand
than deletes a link in a directory
go (once left, they do not return).
p the probability that a user adds a file
file
In Algorithm 1, user u generates a path, P, starting
ratherthanadirectorywhenaddingalink
from the root and proceeding down to a leaf. The pur-
p theprobabilitythatauserleavestheP2P
leave
pose of generating the path is to model the behavior
network
of a user who is browsing the structure to either down-
load a file or to make an update. It is not sufficient
that user u traverses the tree, the decision to update to simply pick at random from the set of all nodes be-
a node on the path is taken with probability p ; causethenodeprobabilitiesarepartiallydeterminedby
update
a new link is added with probability p (otherwise which users are viewing which nodes, the quality of the
add
an existing link is deleted) and a new link points to a nodes, and the connections from one directory node to
filewithprobabilityp (otherwiseitpointstoadirec- another. Basically, the current location of the peer is
file
tory). Notethatanexistingfilecanonlybemodifiedby kept in l = vi,j and at each step, the peer checks Ql
deleting the link to it in one traversal and then adding to determine if a new version of node i should be se-
a new link in another traversal. Figure 2(c) shows the lected or deviated to. A selected version becomes the
decisionprocess(top)andtheresultofanupdatetothe peer’s viewing preference γi,u for node i. The default
root node (creating versionv1,2) adding a new link and versionto view is a randomversionfromthe mostpop-
hence a new node. ular versions. After traversing to a leaf, a decision is
Note that p essentially allows us to model the made whether to update a node or not.
update
update frequency, i.e. the fraction of time that is spent The model to determine which node in P to update
byusersupdatingthedirectorytree,ratherthansimply also requires consideration. Choosing uniformly at ran-
browsing and downloading. This is important because dom would cause excessive updates to the root node
the affects of popularity require time for users to group which is unrealistic. We model the user choice by com-
together on the popular directory nodes, before being putinganestimatesizeofthetotaldirectorytree,based
effective,andtheseaffectsinturnaffecttheupdateout- on the outgoing degree of directory nodes along P and
comes. thetotallengthofthepath,andthengeneratingastair-
Evolution of the directory tree is done by repeatedly case probability distribution that provides an approx-
calling Algorithm 1,using a peer u chosenuniformly at imate uniform random distribution over all accessible
random from N, increasing the time t by 1 after each nodes. This model says that users are more likely to
call; hence the number of “time steps” is the number of makeupdatestowardstheleavesofthetreeratherthan
times that the algorithm has been called. towards the root.
7
Algorithm 1 Traverse(useru, time t) Algorithm 3 Select(node index i)
P ←{} v ←v withprobabilityλ(v )/ λ(v )Returnv
i,j i,j j i,j
l=v1,j ←Viewing(1,u) P
δ ←|E (l)|
t
if p≥Ql then Algorithm 4 Update(node vi,j, user u)
l ←Select(1) j′ ←n ←n +1
i i
end if Et(vi,j′)←Et(vi,j)
while l is a directory node and |Et(l)|>0 do Qi,j′ ←psi,j′
l =vi,j ←Viewing Random(Et(l)),u if p>padd and |Et(vi,j′)|>0 then
δ ←δ+|Et(l)| (cid:0) (cid:1) delete the connection Random(Et(vi,j′))
if p≥Ql then else if p>pfile then
l ←Select(i) n←n+1
end if node vn,1 becomes a directory node
if l is not a file then Et(vi,j′)←Et(vi,j′)∪{vn,1}
P ←P ∪{l} Qn,1 ←psn,1
end if else
end while n←n+1
δˆ←δ/|P| node vn,1 becomes a file node
c← the C(δˆ,|P|)-th entry in P Et(vi,j′)←Et(vi,j′)∪{vn,1}
if p<pupdate then Qn,1 ←psn,1
Update(c,u) end if
end if
3.3 Simulation control parameters
FromAlgorithm1,δisthetotaldegreeofthenodesin
thepath(notincludingnodeversionsthatweredeviated We used the control parameters N = 100, s = 1.0,
frombecauseofaqualitydecision,andnotincludingthe padd = 0.75, pfile = 0.5, pupdate = 0.5, pleave = 0.0.
last node if it is a file). Then δˆ = δ/k where k = |P| The initial directory tree is given in Figure 2(c) (bot-
and the choice of which node to update is given by: tom left) and is viewed by node 0, all nodes are direc-
tories with quality 0.5. We ranallthe simulations until
C(δˆ,k)= logˆ(1+(δˆk−1)p) (1) time t=105 andthe resultsare the averageoftenreal-
δ
(cid:4) (cid:5) izations. The control parameters correspond to a fixed
wherepis chosenuniformly atrandomin[0,1). Exam- numberofdedicatedpeersthatarevigorouslyupdating
ples are shown in Figure 2(b); the choice function is in the directory tree, adding new nodes equally likely to
an integer in {0,1,...,k −1} and is never equal to k be files or directories.
because p is never 1. Note that: When examining the results we often consider the
main tree, which we define as the tree that a new peer,
limC(δˆ,k)=⌊pk⌋ having no initial viewing preferences, would browse.
δˆ→1
This tree is computed by tracing the most popular
which is the case when the tree appears to be a linear paths. In the case that more than one such tree ex-
list. ists then we pick at random. An example main tree is
shown in Figure 2(d), for the control parameters. El-
To make random selections with probability pro-
lipses are directories,diamonds are nodes and there are
portional to the number of viewers of a version, and
4 shades of grey, from dark grey which indicates qual-
to select randomly among the most popular versions,
ity > 0.75 to light grey which indicates quality < 0.25.
we let λ(v ) = {u | j = γ } , and λ (i) =
i,j i,u max
max {λ(v )}. The(cid:12) function Random(cid:12)(set X) returns an In the example notice that quality is generally higher
j i,j (cid:12) (cid:12)
towards the root, because those nodes are visited more
element x ∈ X, chosen uniformly at random. Algo-
oftenleadingtoabetterefficacyofthepopularityeffect.
rithms 2, 3 and 4 are called by Algorithm 1.
The average quality ofthe main treeis definedas the
averageofthequalityofallnodesinthemaintree. The
Algorithm 2 Viewing(node index i, peer u)
outcomeisgoodifthe averagequalityexceeds1/(1+s)
if j =γ is undefined then
i,u and bad if it does not.
j =γ(i,u)←Random {v |λ(v )=λ (i)}
i,j i,j max
We independently varied each of the parameters and
end ifReturn v (cid:0) (cid:1)
i,j
report the most interesting results in the following sec-
tions.
8
umber of nodes in main tree 111 648024000000 0000....9521 node frequency 1 0e 0.− 00.000 .11141 0000....5912 node frequency 1 0e .0− 00.000 .11114 00..12 viewers per node 2222331 .......322468248 0000....5921
average n 2 00 0 20 40 60 80 100 11ee−−0065 1 10 100 11ee−−0065 1 00..59 10 100 111...642−1 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0
time (x1000) degree of node number of viewers quality of node
(a) Average nodes in main tree (b) Nodefrequencyversusdegree (c) Node frequency versus num- (d) Viewers per node versus
versustime att=105 bersofviewersatt=105 qualityofnodeatt=105
y 30
average quality of main tree 0000 00000....5678.....567895555 0000....1259 umber of nodes that reach majorit 1122 50505 0000....1295 average time to reach majority 122334455505050505000000000000000000000000000 0000....9512 average quality of main tree 00000 00000.....45678.....5678955555 00..255 124...000
0.45 n 0 10000 0.4
0 20 40 60 80 100 −1 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 −1 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 0 20 40 60 80 100
time (x1000) quality of node quality of node time (x1000)
(e) Average quality of the main (f) Number of nodes that reach (g) Average timeto reacha ma- (h) Average quality of the main
treeversustime amajorityversusquality jorityversusquality treeversustimewithvaryings
Figure 3: Simulation results, (a) to (g) with p ={0.1,0.2,0.5,0.9},(h) with s={0.25,0.5,1.0,2,4}
update
3.4 Update frequency become more distinguished than low quality nodes.
Figure 3(e) shows the averagequality of the nodes in
The update frequency is given by pupdate and deter- the main tree versus time. A small update frequency
mines the proportion of traversals of the directory tree leads to a significantly better average quality, though
that result in updates. A low update frequency means there is no difference from p = 0.2 to 0.1; so re-
update
thatusersarebrowsingthetreemoreoftenthanupdat- ducing the update frequency further than this does not
ing, and vice versa. Clearly the total nodes in the sys- help. Note that the average quality of an update is
tem increases to become roughly 105pupdate (e.g. with 1/(1+s)=0.5;sothesystemisworkingwelltoimprove
pupdate = 0.9 it increased to nearly 90,000; nodes with the averagequality of files found. Compare Figure 3(e)
noviewersarenotcounted),howeverthe averagenodes to Figure 3(h), which shows the average quality of the
inthemaintreeisaround0.1−0.2%ofthe totalnodes, maintreewhenonlysvariesfrom0.25to4. Inallcases
as shown in Figure 3(a). Figure 3(b) provides the total the average quality of the main tree is above the mean
node frequency versus degree. A smaller update fre- quality, 1/(1+s), over all nodes.
quency leads to nodes of smaller degree.
Continuing, Figure 3(f) shows the number of nodes
The node frequency versus number of viewers, Fig- that reach a majority (more than half the users) and
ure 3(c), shows that almost all of the nodes have only Figure 3(g) shows the average time it takes from the
1 viewer (the creator of that node) and that this distri- time of creation, for a node of given quality to reach
bution becomes more negatively sloped as the update a majority. Interestingly there is no apparent trend in
frequency increases. This is because new versions, re- Figure 3(g). This is due to the fact that users select a
gardless of quality, are selected with probability 1/N version to view based only on popularity and not node
and if the new version is contained in a tree outside of quality. Consider a low quality node that gains moder-
the main tree then its probability of selection is further ate popularity; while, with high probability, users mi-
reduced. At the same time, Figure 3(d) shows that the grate away from the low quality node, users are likely
number of viewers per high quality node (Q ∈ [0.9,1)) to choose the node of moderate popularity over a high
is almost twice as much as that for a low quality node quality node that has little popularity at that time. A
(Q ∈ [0,0.1)) when p = 0.1, but this difference lowqualitynodecouldgainmildpopularityviarandom
update
is reduced as p increases. Clearly, a small update fluctuations. It could also quickly gain high popularity
update
frequencyallowsthepopularityofhighqualitynodesto if it is the only node on a path. Even if a new version
9
2.4
1 0.1 2.3 0.1 0.75 0.8 1
node frequency 1 0e 0.− 00.000.1141 0000....752955 viewers per node 111122... ...7682912 00..525 0.9 ge quality of main tree 000 0..0.675..75565 1010000 10 node frequency 1 0e 0.−0 .0000.1411 11110000000000
1e−05 11..54 avera 0.5 10000 1e−05
1e−06 1.3 0.45 1e−06
1 10 100 −1 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 0 20 40 60 80 100 1 10 100 1000
degree of node quality of node time (x1000) degree of node
(a) Nodefrequencyversusdegree (b) Viewers per node versus (c) Average quality of the main (d) Nodefrequencyversusdegree
att=105 qualityofnodeatt=105 treeversustime att=105
y 35
node frequency 11 0ee 0.−− 00.0000 .111541 0012....52005 mber of nodes that reach majorit 21123 505050 42100.....000525 average quality of main tree 000 0000...456....4567555 000...125
1e−06 1 4.0 10 100 nu 0−1 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1.0 0.35 0 20 40 60 800.9 100
number of viewers quality of node time (x1000)
(e) Node frequency versus num- (f) Number of nodes that reach (g) Average quality of the main (h) Visualdepictionofmaintree
berofviewersatt=105 amajorityversusquality treeversustimewiths=4 for N = 10 at t= 105, the dark
center istheresultofexcessively
largedegree
Figure 4: Simulation results, (a) - (b) for p = {0.1,0.25,0.5,0.75,0.9}, (c) - (d) for N = {10,100,1000,10000},
file
(e) - (f) for s={0.25,0.5,1.0,2.0,4.0},(g) for s=4 and p ={0.1,0.2,0.5,0.9},(h) for N =10
update
is created with high quality, the already popular low to a slight decrease in averagequality of the main tree,
quality node remains popular for some time. the number of low quality nodes that reach a majority
increasestothepointthatthereisverylittledistinction
between low quality nodes and high quality nodes; the
3.5 Add frequency
numberofviewersperlowqualitynodebecomesroughly
We set the add frequency, p , to be ≥0.5 so that the equaltothenumberofviewersperhighqualitynodeas
add
growthofthedirectorytreewaspositive. Asp →0.5 shown in Figure 4(b).
add
theaveragequalityofthemainnodeincreases,similarly
toreducingp . Howeverthe totalnumberofnodes
update 3.7 Number of peers and churn
decreases because in some cases a deletion event is in-
effectual, e.g. a node cannot be deleted outright in the Increasingthenumberofpeersshowsaninterestingout-
sense of removing it from the system, it can only be come which is seen in Figure 4(c), the quality of the
contendedwitha new version;a node canbe effectively main tree versus time. For N =10 the average quality
removed from the main tree by creating a high quality fallstoavaluelessthanthatforN =100. ForN =1000
version of the parent node that does not contain the it rises to match the case when N = 100 though more
child connection. slowly. For N = 10000 it sits slightly above 0.5 which
is the initial quality of the initial nodes; in a separate
simulation over twice the time interval we observe the
3.6 File/Directory frequency
average quality to rise to almost 0.6, hence as N be-
Thefile/directoryfrequency,p ,determineshowoften comes large it takes longer for the main tree to grow.
file
filesareaddedasopposedtodirectories. Asfilesbecome Forasmallnumberofusers,anodecanquicklybecome
more likely, the number of directories decreases. This popular, but it does not necessarily stay popular for
naturallyincreasesthedegreeofthedirectoriesasshown very long. For a large number of users, it takes longer
in Figure 4(a); furthermore it tends to push the degree for a node to become popular, and popular nodes re-
distributiontowardsa powerlaw. While thisleadsonly main popular for longer. Hence the average size of the
10
