Comet: An active distributed key-value store RoxanaGeambasu,AmitA.Levy,TadayoshiKohno, ArvindKrishnamurthy,HenryM.Levy UniversityofWashington Abstract A significant limitation of such systems for generic application support, however, is that different applica- Distributed key-value storage systems are widely used in tionshavedifferentneeds. Asanexample,eachDynamo corporations and across the Internet. Our research seeks to greatlyexpandtheapplicationspaceforkey-valuestoragesys- application inside of Amazon runs its own Dynamo in- temsthroughapplication-specificcustomization. Wedesigned stance[17], eventhougha singleinstancemight belog- and implemented Comet, an extensible, distributed key-value ically better and more resource efficient. In our own store. Each Comet node stores a collection of active storage workonVanish[25]–asecurity-orientedDHTapplica- objects(ASOs)thatconsistofakey,avalue,andasetofhan- tion – we needed to make application-specific parame- dlers. Comethandlersrunasaresultoftimersorstorageoper- terandpolicychangestoVuze(amillion-nodecommer- ations,suchasgetorput,allowinganASOtotakedynamic, cial DHT) in order to harden it against attack. While application-specificactionstocustomizeitsbehavior.Handlers these changes were conceptually simple, e.g., modi- arewritteninasimplesandboxedextensionlanguage,provid- fying the storage replication algorithm, deploying our ingpropertiesofsafetyandisolation. changestookmonthsofworkwithVuze’sDHTdesigner. WeimplementedaCometprototypefortheVuzeDHT,de- Other Vuze applications may wish to make their own ployed Comet nodes on Vuze from PlanetLab, and built and evaluated over a dozen Comet applications. Our experience application-specificchangesorenhancements,butdoing demonstratesthatsimple, safe, andrestrictedextensibilitycan soisneitherfeasiblenorsupportable,anditdoesn’tscale. significantlyincreasethepowerandrangeofapplicationsthat We believe that with the huge consolidation benefits of canrunondistributedactivestoragesystems.Thisapproachfa- sharedcloudstorageservices, eitherinsideoroutsideof cilitatesthesharingofasinglestoragesystembyapplications the enterprise, supporting specialization of storage ser- withdiverseneeds,allowingthemtoreaptheconsolidationben- vicescanhavehighpayoffsinthefuture. efitsinherentintoday’smassiveclouds. This paper presents Comet, a next-generation, flexi- 1 Introduction ble, distributed storage system, which opens the world ofdistributedstoragetoanewsetofmorecomplexstor- The last decade has seen the rise of distributed storage applications. In particular, Comet permits multiple age systems built on loosely coupled collections of au- applications to share a single Comet instance, while en- tonomous computers. For example, Amazon’s S3 [3] ablingeachapplicationtochangethebehaviorofitsstor- provides a key-value storage service for external Web ageelementstosuititsownrequirements. Forexample, clients. Amazon’sDynamo[17], ApacheCassandra[5], astorageelementcanmakedecisionsbasedonitsaccess andProjectVoldemort[38]providereliableandscalable history, its current number of replicas, the time of day, key-value stores for company-internal applications (for etc. Therefore Comet can easily support different stor- Amazon,Facebook,andLinkedIn,respectively). Onthe agelifetimes,accessmethods,accesscontrolschemes,or globalInternet,DHTsprovidedbyBitTorrent-basedsys- replication schemes for different storage-element types, tems,suchasVuze[58]anduTorrent[56],storemetadata inawaythatmakesthemeasytodeployandtest. Using formillionsofclientsusingpeer-to-peerfile-sharingap- Comet, we can also carry out interesting measurement- plications. Andfinally,researchershavedevelopedcom- basedexperimentsfromwithintheDHT. plete file systems on top of untrusted clients in widely distributedP2Penvironments[2,14,44]. Cometimplementsactivestorageobjects(ASOs). An Distributed storage systems offer many advantages activestorageobjectconsistsofakey,anassociatedvalue over their centralized counterparts. For example, a de- (an untyped blob), and optionally, a set of simple han- centralizedstructuresupportsscalability;thelackofcen- dlers. An ASO’s handlers execute as a result of com- tralizedmanagementenhancesautomaticloadbalancing; mon storage events on the object (such as get and put) andtheuseofreplicationinahighlydistributedenviron- orfromtimereventsthatitshandlersrequest. Asaresult, ment can improve reliability and data availability. We an ASO can modify its environment, monitor its execu- thereforeexpectDynamo-likestoragesystemstobecome tion,andmakedynamicdecisionsaboutitsstate. commonplaceasgenericapplicationinfrastructuresinthe The design of an extensible system for this environ- future,bothinsideoftheenterpriseandassharedservices ment presents a set of interesting design questions. For ontheInternet. example,whatfeaturesshouldthesystemprovideforap- plicationsandwhichcan(andshould)beleftout? What traditional put/get interface, but many others (e.g., Mer- is the proper tradeoff between power and safety? How cury[8],CoralCDN[21],Scribe[45],andBayeux[64]) can client nodes be confident that active storage objects require customized interfaces and are implemented by willnotcausedamageorinterference? Howcanwepre- altering the underlying DHT mechanisms in significant venttheuseofactivestorageobjectstomountaDDoSat- ways. Our work provides the ability to extend a DHT tack? Andoverall,howcanweextendthestoragesystem without requiring a substantial investment of effort to without losing its principal characteristics? Our Comet modifyitsimplementation. designconsiderstheseandotherissues. Deployed DHTs don’t currently offer good semantics The remainder of this paper describes our goals, ar- and security. However, people do know how to make chitecture,experience,andevaluationofComet. Topro- them consistent [32, 34] and harden then against at- vide concrete insight into Comet’s design and potential, tacks[11,16,48,26,59]. ThereasonDHTsdonotcur- we implemented a Comet prototype and used it to cre- rentlyimplementthesetechniquesisthattherehasnotyet ateanddeployasetofoveradozenCometapplications. been a deployed application that truly needed strong se- Our prototype leverages Vuze: each Comet instance is manticsandsecurity. Forexample,theVuzedesignper- an extended Vuze client that can execute Comet active ceived many threats as irrelevant [23] and deployed few storage objects while also serving as a full participant defensesagainstthem. However,afterthenew,morede- in the million-node Vuze DHT. Comet applications are mandingVanishapplicationwasproposed[25],theVuze written in Lua – a common application-extension lan- DHTrespondedbyembracingavarietyofeffectivesecu- guage. We modified the Lua runtime to meet our iso- rity measures. In addition to enabling new applications lationandsafetyrequirements,providingasafesandbox atopDHTs,wehopetodrivethedesignofthesesystems forhandlerexecution. Totestourapplicationsweranour towardswell-understood,yetunadoptedlevelsofsecurity CometclientsfromseveralhundredPlanetLabnodesand andconsistency. measuredtheirbehavior. Overall,ourexperiencedemon- 3 Goals stratesthatahighlyrestrictivebutactivedistributedstor- age system can provide significant power to simultane- Comet is a distributed key-value storage system. Like ouslysupportapplicationswithdiversestorageneeds. other such systems, a Comet storage object is a pair. Unlike previous systems, however, 2 RelatedWork Comet’s design facilitates extensible, active storage ob- The concept of extensible systems has been widely ex- jects. A Comet application performing a put can there- ploredinthepastinseveraldomains. Extensibleoperat- fore include, along with a key and value, a small set of ingsystemshavebeenproposedthatsupportapplication- handlers for that object. The node receiving the put specific needs [6, 46, 28]. Active networks allow code storesthehandlersalongwiththekeyandvalue,registers tobedownloadedalongwithnetworkdataandexecuted thehandlersforeventsthattheyspecify,andexecutesthe withinthenetworkinfrastructure(e.g.,onrouters)toex- handlerswhentheirrespectiveeventsoccur. tendnetworkservices[60,54]. Activemessagesexecute Comet’ssystemgoalsare: a small amount of user code with each message recep- 1. Flexibility. Comet should be easily customizable to tion [57]. Click explored the design of an extensible achieveourtargetfunctionsdescribedbelow. router[30].Databasetriggersallowapplicationstodefine 2. Isolation and safety. A client node running Comet procedural codethat isexecuted inresponse todatabase should be protected from the execution of handlers operations[35]. (e.g.,anexecutinghandlercannotcorruptthenodeor In the context of storage systems, Watchdogs [7] ex- useunlimitedresources).Handlersshouldnotbeable tendstheUnixfilesystem,allowingauser-modeprocess tomountmessagingattacksonothernodes. tointerposeonfileoperationsforspecificfilestochange 3. Performance. The performance of gets/puts on a access semantics. Several projects have proposed the Comet ASO with null handlers should be the same integration of CPUs and disks to create intelligent disk asonanon-activesystem,andexecutionofhandlers storage systems that can provide on-board application- shouldhaveonlynegligibleperformanceimpact. specificfunctions,e.g.,fordecisionsupportsystems,data mining,andimageprocessing[29,41,1]. Isolation and safety are particularly important to our DHTsareincreasinglyusedtosupportavarietyofdis- architecture. WhileCometcanbeusedindifferentenvi- tributedapplications,suchasfile-sharing,distributedre- ronments, we designed it to enable wide-scale, outside- sourcetracking,end-systemmulticast,publish-subscribe the-firewalldeploymentonautonomousnodes,similarto systems,distributedsearchengines,andevendata-center P2PsystemsandDHTs. UsersdownloadingCometmust applications. Some of these systems (e.g., as CFS [14], trust it and have guarantees about its behavior. For this i3 [52], and PAST [44]) can be implemented using the reason,Cometenforcesfourimportantrestrictions: 1. Limitedknowledge:anASOisnotawareofotherob- 4 ArchitectureandImplementation jectsorresourcesstoredonthesamenodeandhasno This section describes Comet’s active storage architec- directwaytolearnaboutthem. ture and prototype implementation. One could imagine 2. Limited access: an object handler can manipulate running Comet in various environments, e.g., an inside- only its own value and cannot modify the values of the-firewall corporate deployment or a distributed envi- otherobjectsonitsstoragenode. ronmentwithautonomousuntrustednodes. Wefocusour 3. Limitedcommunication:anactivestorageobjectcan- currentarchitectureandprototypeonthelatter. notsendarbitrarymessagesoverthenetwork. 4.1 Architecture 4. Limited resource consumption: an ASO’s resource Figure 1(a) shows the high-level architecture of our usage is strictly bounded, e.g., the system limits the Comet distributed storage system. The Comet storage amountofcomputationandmemoryitcanconsume. system consists of three basic components. First is the routing substrate (Figure 1(a) bottom), which imple- We are specifically not attempting to build a general- ments the value/node mapping, allowing a client to find purposedistributedprogrammingsystem,suchasPlanet- nodesthatstorespecificdataitems. InthecaseofaDHT, Lab[4,36];suchasystemwouldbeunacceptableinour forexample,theroutingsubstratetypicallyappliesahash target environment and inappropriate (and unnecessary) functiontothekeytocomputetheIDsofnodesthatstore for our needs. Rather, our goal is to support relatively the associated value. However, other routing substrates simple specializations or actions on simple storage ob- maylocatevaluesinotherways. jects. Evenverysimplespecializationscanprovideasig- The second component is the key-value store, which nificantlymorepowerfulstoragesystemthatenablesnew maintainsasetofkey-valuepairsoneachnode. Akey- typesofapplications.Wethereforetakealightweightand valuestoragesystemtypicallyexportsasimpleget/put limited approach. As examples, an ASO should be able interface. Whileexistingstoragesystemsstorearbitrary, toperformthefollowingfunctions: untypedbytestrings,theCometstoragesystemstoresac- • Statistics gathering. Collect statistics about its use, tive storage objects (ASOs). An ASO consists of a key e.g.,bycountingthenumberofgetsandputs. anditsassociatedstate(i.e.,avalue,storedasanuntyped byte string), along with optional code that operates on • Informationtracking. Loginformation,suchasalist thatstate. Thecodeisstructuredasasetofhandlersthat ofIPsthatperformedgetoperationsonitsvalueora specify how the object behaves, i.e., how it modifies its recenthistoryofthevaluesitstored. statewhencertaineventsoccur. Forexample, anASO’s • Time awareness. Take time-based actions, e.g., to onGet handler is invoked whenever a remote client per- makeperiodicchangestoitsstateorself-destructaf- forms a get operation to access an object. This handler teratimerhaselapsed. mightperformsomesimpleoperation,suchasincrement- ing a counter for the number of gets or appending the • Locationawareness. Makelocation-baseddecisions, client’sIPaddresstoalogstructure. Thecounterorthe e.g.,choosingwheretostorebasedonnodes’network log structure would be stored as part of the ASO’s state locations. thatcanbeaccessedbythehandler. • Access control. Implement simple access control Thethirdarchitecturalcomponentistheactiveruntime policiesonitsown. system. The runtime system handles ASO invocations • Replication. Implementdifferentreplicationpolicies. and provides the security policy and execution environment.Anapplicationrunningonaremoteclientspecifies • Storagesystemmeasurement.Provideinsightintothe the initial state and handlers for an ASO when initially behaviorofthedistributedstoragesystemasseenby storingtheobjectviaaputoperation. Whenaclientper- clientsexecutingwithinthesystemitself. forms a get or a put, it can optionally request a cryp- tographic checksum of the code associated with the tar- Asweshallsee,theonlylong-termstateavailabletoa get ASO. This can serve as an integrity check that the handler is its object’s value; therefore, any logs, counts, client’s initial put is to a key with no associated ASO etc.,mustbestoredaspartofthatvalue. However,anac- and that subsequent operations are performed on ASOs tiveobjectcanchoosetoreportonlyasubsetofitsstored created by the application. In most implementations, a valuerecordonaget,oritcanselectivelyreportdifferent Comet node distrusts remote nodes and client applica- valuestodifferentcallersbasedoncallparameters. tions;therefore,theruntimecomponentoftheactivesub- ThefollowingsectionsdescribeComet’sarchitecture. systemimplementsandenforcesanASOexecutionsand- Inparticular,wediscussthetradeoffsrequiredtoprovide box(Figure1(a), top). OurCometprototypeusesalan- flexibilitywhilealsoachievingisolationandsafety. guage sandbox based on Lua [43] to prevent a handler Storage Node x o ASO b 1 nd - state ASO2 Application / getSystemTime()→UTC aS - code (handlers) User getIP()→node’sexternalIP getID()→node’sDHTID ASO API ASO Handlers Remote getKey()→ASO’skey Storage Node deleteSelf():deallocateASO External Security ASO Interaction Policies Runtime get(key,[args])→value,nodesstoringcopies Active Subsystem put(key,value[,nodes]) lookup(key)→nodesclosesttoakey K1 ASO1 put/get/delete K ASO 2 2 Key-Value Store Routing Substrate (a) Architecture. (b) ASOAPI. Figure 1: CometArchitectureandAPIs. (a)depictsthedecompositionofaCometnodeintotwoverticalcomponents-the coreCometcode,whichistrustedfromthenode’sperspective,andtheASOcodewhichisarbitraryand,therefore,untrusted. (b) detailstheAPIexposedtoASOs. from accessing outside state and to constrain the ASO andrunningoveradozenapplicationsonourCometpro- fromconsumingtoomanycomputationalandmemoryre- totype. Interestingly, we were able to build a relatively sourcesonthehost.TheASOruntimeconsultsasecurity diverse set of applications with a surprisingly small in- policymodule,whichspecifiesallexecutionlimits. terface,whichhasremainedrelativelystablethroughthe While some applications may be satisfied by an en- project. Thissuggeststhatasmallinterface,liketheone tirelysandboxedexecution,manywouldbenefitfroman showninFigure1(b),cansupportawidevarietyofappli- ASO’s limited ability to interact with or “sense” its en- cations. Naturally,therearelimitations. Forexample,we vironment. For example, to implement the conditional explicitly prohibit any direct network-level interactions replicationschemeweaddedtoVuzeforVanish,anASO with remote nodes on the Internet. While this feature requiresknowledgeofthenumberofreplicasintheDHT might be desirable to certain measurement applications, and the time of day (to enforce the desired minimum itsDDoSimplicationswouldbeunacceptable. replication interval). For this reason, the active subsystem exposes a small API (called the ASO API) to the onGet(caller[,callbackID,payload]) InvokedwhenagetisperformedontheASO.Returnsavaluewhichwill handlers. bepassedbacktothecaller. Insteadofreturningavalueimmediately,the handlercouldalsoperformaputattheoptionalcallbackIDsometimeinthe 4.2 ActiveStorageObjectAPI future. Thehandleralsotakesanoptionalpayloadargumentofarbitrary type. onPut(caller) Table1andFigure1(b)showthehandlerandASOrun- Invokeduponinitialputwhentheobjectiscreated.Returnsthevaluethat timeAPIs,respectively. ThehandlerAPIsupportsinvo- shouldbestoredbythenode(e.g.,itselfornil). cationsbasedontheprimarystoragefunctions–put,get onUpdate(newvalue,caller) InvokedonanASOwhenaputoverwritesanexistingvalue. Returnsthe – as well as an onTimer handler to be executed period- valuethatshouldbestored,e.g.,newvalueifitshouldbereplaced,oritself ifnot. ically (e.g., once every 10 minutes) during the object’s onTimer() lifetime. Forexample,anASOcoulddirectlyimplement Invokedperiodically.Thishandlerhasnoreturnvalue.Itisusedtoperform acustomreplicationpolicyinitsonTimerhandler. periodicmaintenancesuchasreplication. The ASO runtime API is the only way for an ASO Table1: ASOHandlerCalls. to interact with its environment outside of the sandbox. Ourdesignsupportstwotypesofusefulinteractions: (1) 4.3 LanguageBasedSandbox obtaining information about the local node, and (2) ex- ecuting various storage system operations. The former Our Comet prototype focuses on a DHT environment categoryincludesfunctionstoobtainthetimeofday,the composed of a large number of untrusted autonomous hostingmachine’sexternalIPaddress,etc. Thelatterin- nodesthatcooperatetosupportthedistributedactivestor- cludesfunctionstointeractwithotherstoragesystemob- age system. In this environment, the key challenges in- jects. TheASOAPIwasnotdesignedtobeentirelygen- clude providing a strong sandbox and limiting ASO re- eral;rather,ourgoalwastoprovideaminimalinterface, sourceconsumption. Webrieflydescribehowoursystem informedinpartbyourrequirementsofsecurity,privacy, addressesthesechallengesusingalanguagebasedsand- and isolation. We tested this interface by implementing box. The Comet prototype required an ASO programming other nodes. In the API presented in Section 4.2, these environment that reflected our needs for simple extensi- operationsreturnvaluesorneighboringnodeIDs. How- bility, flexibility, performance, isolation, and safety. To ever, since these operation are slow in the DHT setting meet these needs, we chose Lua [43], a lightweight and andmayblockforsecondsorevenminutes,wechoseto easily constrained scripting language. A dynamically implementthemusingfunctioncallbacks. Eachsuchop- typed,imperativeandfunctionalprogramminglanguage, erationtakesanoptionalparameter,afunctionwhichac- Lua is most commonly used for coding application ex- ceptstheresultasitsparameter. Forexample,insteadof tensions. Inthiscontext,itletsusersaddormodifyfea- returningavalue,agetoperationtakesafunctionwhich turesinvideogameengines,Webservers,versioncontrol is eventually passed the result of the operation. The op- systemsandotherapplications(specificexamplesinclude eration returns immediately with no value, and the get World of Warcraft, SimCity 4, Adobe Photoshop Light- is actually performed after the ASO execution has com- room,andSqueezeboxJivePlatform). Severalproperties pleted. Whilethispresentsaslightlydifferentparadigm make Lua well suited for implementing ASOs. First, it totheuser,wethinkthisprovidesagreaterabilitytoop- employesasmallsetofprogrammingconstructs(includ- timizetheperformanceofComet-basedapplications. ingfirst-orderfunctions)andasmallnumberofdatatypes 4.4 CometPrototypeImplementation (includingtables,whichareheterogeneousassociativear- rays). Second, Lua compiles to simple bytecode, which We built the Comet prototype on the Vuze DHT, which makesitrelativelyeasytosandbox. Finally,ASOswrit- supports the widely used Vuze BitTorrent client. The teninLuaareconciseandsmallwhenserialized;theLua DHT is used mainly for distributed tracking of torrents; ASOsweimplementedareallunder1.5KB,aboutfiveto howeverithasbeenusedinresearchaswell[27,25]. tentimessmallerthanJavaequivalents. Vuze implements the Kademlia routing protocol, in CometrepresentsASOsasLuatablesthatencapsulate which each node is assigned a 160-bit ID based on the both persistent state and the handlers to be invoked on SHA1hashofitsIPaddressandport. BasicDHTopera- that state. Lua tables can implement basic arrays, asso- tions(get,put,andremove)takea160-bitkey,perform ciative arrays, or both. While an associative array can alookuptofindnodeswhoseIDisclosetothatkey,and containanyname-valuemappings,wetreatcertainasso- thensendareadorstoreRPCtothosenodes. ciationsashandlers. Inparticular,iftheASOtablecon- WeminimallyextendedtheVuzeinterfacetoconform tains an associative array with the names “onGet,” “on- to Comet’s abstract operations. For example, we aug- Put,” “onUpdate,” or “onTimer” – and those names are mented get to allow a caller to pass an arbitrary byte- associated with values that are Lua functions – then the string argument. This supports a parameterized get op- runtimeinvokesthosefunctionswhenthecorresponding eration, where the ASO can return different values de- events occur. Our runtime system serializes Lua tables pendingontheparameter(analogoustothesemanticsfor intoabytestreamfortransmissiontoastoragenodeona GETinHTTP). putrequest. Allowing extensibility in a DHT environment creates WemadeseveralmodificationstothestandardLuain- challenges,e.g.,ithasthepotentialtoprovideaplatform terpreter for the Comet runtime system. We sandbox for DDoS attacks. Therefore, in addition to the Lua re- ASOs by removing all but the core libraries from the source restrictions described previously, we limit DHT runtime, leaving only a math package, string manipula- communicationsthatASOscanperformintwoways. tion, and table manipulation. As a result, handlers are First,wedonotallowanASOtoperformoperationson extremelyrestricted: theyhavenodirectnetworkaccess, arbitraryDHTkeysornodes,butratheronlyonspecific nosystemexecutioncapabilities,nothreadcreationcapa- key-nodepairs. AnASOmaycommunicatewithanyof bilities,andnofilesystemaccess. Wealsostrictlybound itsneighboringnodesthatareresponsibleforreplicasof theamountofresourcesthatahandlercanconsume. For the ASO. We also allow the ASO to communicate with example,theruntimelimitsboththenumberofbytecode key-node pairs that have interacted with it in the past, instructionsthatahandlercanexecuteandtheamountof once for each such interaction. To enable this function- memory it can consume. If a handler exceeds either of ality, we extended Comet requests to include the ID of theselimits,theruntimeterminatesitsexecution. the requesting node and the ID of a local key contained TheCometruntimeexposesaDHTwrapperobjectto within the node. If an ASO receives a get request with handlers,whichallowsanASOtocommunicatewithits a key ID specified, it gains the capability for a one-time environment. The ASO can learn information about the operationonthatkeytothenodethatissuedtherequest. hosting node, including the external IP address and the TheASOcantheneitherreturnavalueimmediatelyand current system time. It can also perform a restricted set exhaust its one-time capability, or save that capability ofDHToperations. Forexample,itcanperformgetand for future use. This mechanism allows applications to putoperationsonreplicatedcopiesofitsvaluestoredat respond to DHT requests at a future point in time, es- (cid:7) (cid:4) pecially if the requested data is not currently available. functionaso:handleLookup(nodes) WedonotallowASOstopassthesecapabilitiesbetween nodes=self.selectGoodNodes(nodes) each other as doing so would enable a malicious node dht.put(dht.getKey(),self,nodes) to mount DDoS attacks. In Section 5 we discuss signed end ASOs,whichdonothavetheserestrictions. functionaso:onTimer() Second, Comet imposes rate limits on the number of dht.lookup(dht.getKey(),self.handleLookup) end messages generated by an ASO, either to neighboring (cid:6) (cid:5) nodesstoringreplicasortoarbitrarykey-nodepairsthat Listing1: SmartReplication have interacted with it in the past. This prevents misbe- 5.2 ControllingDataAccess havingASOsfromexhaustingthebandwidthresourcesof Comet objects can implement various policies that con- theCometnodeshostingthem. Wediscussthesesecurity trolhowdatastoredintheobjectsisaccessed. Weillus- issuesfurtherinSection7. trateafewsuchexamples. 5 Applications Timeouts and Limited-read values: ASOs can be Thissectionseekstodemonstrateboththerangeofstor- usedtoimplementobjectsthatwillbeaccessibleforonly age behaviors that Comet can support and the ease with a limited, application-specified time. Such objects are which those behaviors can be implemented. To do this, meaningfulforsecurityapplicationssuchasVanish[25], we describe several of the active storage applications whichprovidesupportforself-destructingdigitaldataby we have implemented, deployed, and measured on our storingcryptographickeysinaDHT. CometPlanetLabprototype. Weprovidecodesnippetsto Listing 2 shows the handler code required to imple- showhowsimplytheseactionscanbeprogrammedinour mentapplication-specifictimeouts. Eachreplicastoresa Lua-based ASO environment. In Section 6, we present timestamp when the object is created (stored) and then measurementsfromsomeoftheseexamples. deletestheobjectafter60minutesusingatimerhandler. Inaddition,theonGethandlerpreventstheobject’scon- 5.1 CustomizableReplication tents from being accessed after the timeout but before it isdeletedbyatimerhandler. Most DHTs specify a fixed replication policy for stored values, requiring applications to conform to that pol- (cid:7) (cid:4) icy. In contrast, Comet ASOs can provide their own functionaso:onPut(value) application-specific replication mechanisms, e.g., con- self.timeout=dht.getSystemTime()+60∗MINUTES trollingthereplicationfactor,thereplicationinterval,and returnself the choice of nodes on which the object will be repli- end cated. Thisflexibilityisusefulforapplicationsthatplace functionaso:onTimer() if(dht.getSystemTime()>self.timeout)then varying degrees of emphasis on performance, availabil- −−deletelocalASO ity, locality, and security. For instance, a security sensi- dht.deleteSelf() tiveapplication(suchasVanish)mightuseasmallnum- end berofreplicasandlongreplicationintervals,limitingthe end dispersionofitsobjectsstoredintheDHT.Ontheother functionaso:onGet() hand, an application that values availability might repli- if(dht.getSystemTime()>self.timeout)then catefrequentlytoalargenumberofnodes. −−deletelocalASO Listing1showshowanASOcandefineacustomized dht.deleteSelf() replication policy. In this example, the onTimer han- returnnil dler wakes up periodically, invokes lookup to deter- end returnself mine a list of nodes closest to the ASO’s key, executes end selectGoodNodes1toidentifyasubsetofnodesthatwill (cid:6) (cid:5) serve as replicas, and then stores a copy of itself on the Listing2: Timeouts selected nodes using put. We have also implemented a An ASO can also choose to delete itself after it has timerhandlerthatreplicatesonlywhenthenumberofex- been read – providing a “limited-read value” – where istingreplicasfallsbelowacertainthreshold;thislowers each replica can be read at most once. In addition to communication overhead and mitigates data harvesting itsuseforself-destructingdata,limited-readvaluescould attacks for security sensitive applications, reflecting the beusedinsettingswhereobjectsrepresenttasksandare changeswemadetoVuzeafterwepublishedVanish[25]. deleted once they have been claimed by worker nodes. 1TheLuacodeforselectGoodNodesisomittedforbrevity. Itim- The object then serves as a synchronizing construct be- plementsanapplication-specificpolicyforchoosingreplicas. tweenthetask’sproducerandconsumer. (cid:7) (cid:4) Listing3implementslimited-readvalues. Whenaget aso.pending={} isperformed,thenoderecordsthefactthatthevaluehas functionaso:onGet(callerNode,callbackKey) been read. It then propagates the request to every other if(self.value)then replicabyoverwritingthemwithnil. Notethattheob- returnself.value ject does not delete itself immediately, but rather stays end around for a while and periodically attempts to delete self.pending[callerNode]=callbackKey other replicas to ensure that copies on nodes with tran- returnnil end sientconnectivityissues[22]areeventuallydeleted.Note functionaso:onUpdate(callerNode,value) alsothatconcurrentgetsissuedtodifferentreplicanodes self.value=value might successfully read the value. In general, as with forcallerNode,keyinpairs(self.pending)do other distributed storage systems, consistent update of dht.put(key,value,{callerNode}) replicated values would require the use of heavy-weight end consensusoperations. Cometdoesnotcurrentlyprovide self.pending={} such primitives. ASO handlers do however provide the end (cid:6) (cid:5) ability for replicas to detect and correct inconsistencies, Listing4: Pub-sub e.g., ASOs can compare and reconcile replica contents throughperiodicinvocationsoftheonTimerhandler. request. There are a few issues with the code provided above, (cid:7) (cid:4) especiallyifitweretobeextendedtosupportpassword- functionaso:onGet() protectedupdates. Amaliciousnodecouldclaimtostore if(self.read)thenreturnnilend theobjectbutsimplyserveasaproxyforclients’requests self.read=dht.getSystemTime()+30∗MINUTES and thereby implement man-in-the-middle attacks. This dht.put(dht.getKey(),nil)−−deletesreplicas couldbesolvedbyexposingbasicencryptionprimitives returnself end totheASO,likeasecurehashfunctionand/orpublickey functionaso:onTimer() cryptographicprimitives.Forexample,insteadofpassing if(self.read)then theplaintextpasswordtotheASO,theclienthashesthe dht.put(dht.getKey(),nil)−−deletesreplicas concatenation of the password with its IP/port, thus the if(dht.getSystemTime()>self.read)then ASO can verify that the request is not being forwarded dht.deleteSelf() byamaliciousnode. TheASO’ssecuritycanbefurther end strengthened by public/private key pairs, with the ASO end storing the public key and clients authenticating them- end (cid:6) (cid:5) selves by presenting a message signed with the corre- Listing3: Limited-ReadValues sponding private key. With these enhancements, a ma- liciousnodestoringacopyoftheobjectcannotoverwrite DataSubscription: AnASOcanallowclientsto“sub- thecontentsofotherreplicassinceitdoesn’tpossessthe scribe” so that they will be notified when the ASO re- privatekey. ceives a new value. In Listing 4, when the subscriber (cid:7) (cid:4) performsaget,theASOsavesthesubscriber’snetwork functionaso:onGet(caller,callerId,password) location (callerNode) and a key that will serve as the if(password==‘‘mypass1234’’)then subscriber’srecipientofthevalue(callbackKey).When return‘‘Wellkeptsecret’’ a value update occurs, the ASO distributes the value to end returnnil all registered subscribers – the runtime ensures that the end ASO distributes these values only to clients who have (cid:6) (cid:5) actually performed a get on the ASO. In the example Listing5: Password shown,theASOclearsitssubscriberlistafteritsputop- An application could use multiple mechanisms for erations;subscribersmustthenre-subscribeifthey’restill controllingdataaccess,e.g.,itcouldusetimeoutsincon- interested. Laterwewilldescribeanimplementationofa junction with password-protected access. While Comet scalablepublish-subscribeschemebasedonthisdesign. does not allow ASOs to register multiple handlers for a Sensitive values: ASOs can implement various forms givenstorageoperation,thedevelopercancombineallof of access control policies. For instance, Listing 5 pro- the desired mechanisms into a single handler. Though vides read access to the object’s value only if the client this might increase programming complexity, it allows can present a predetermined password akin to a feature theapplicationdevelopertocontrolhowdifferentmech- alreadyprovidedbysomeDHTs,likeOpenDHT[40]. A anismsinteractwitheachotherandprovidesthebasisfor client provides the password as an argument to the get apredictableanddeterministicexecutionmodel. (cid:7) (cid:4) 5.3 MeasurementsandMonitoring replicaIps,hostIps,accessorIps={} DHT Measurements: ASOs provide a platform for functionaso:onGet(callerIp) table.insert(self.accessorIps,callerIp) instrumenting and measuring the DHT using the DHT returnself nodesthemselves.Thisenablesamoredetailedandcom- end prehensive view of the DHT and helps provide accurate functionaso:onPut(caller) estimatesofDHTpropertiessuchaschurn,nodelifetime table.insert(self.accessorIps,caller.getIP()) distribution,transientinconsistencies,etc. table.insert(self.hostIps,dht.localNode.getIP()) Forinstance,Listing6tracksthekclosestnodestothe returnself ASO and stores the information it learns as part of the end object state. A measurement application can create ob- functionaso:handlePut(nodes) jectsofthistype,storethematmultiplelocationswithin fori,vinipairs(nodes)do the DHT, and obtain snapshots of DHT membership by table.insert(self.replicaIps,v.getIp()) retrievingtheobjects’contentsusinggetoperations. end end (cid:7) (cid:4) functionaso:onTimer() aso.neighbors={} dht.put(dht.getKey(),self,20,self.handlePut) functionaso:handleLookup(nodes) end (cid:6) (cid:5) self.neighbors[dht.getSystemTime()]=nodes end Listing7: Monitoring functionaso:onTimer() 5.4 SmartRendezvous dht.lookup(dht.getKey(),self.handleLookup) DHTs are used for rendezvous in many distributed sys- end (cid:6) (cid:5) tems. InP2Pfile-sharingsystemssuchasBitTorrent,the Listing6: Lifetime DHT is used as a distributed tracker either with or as a replacementforacentralizedtracker. Thatis, peersthat Whilethismeasurementcouldbeperformedbynodes want to download a particular file use the DHT to iden- thatarenotpartoftheDHT(asinearlierwork[20,50]), tifyotherpeerswhoaredownloadingorsharingthefile. measurementsfromwithintheDHTcanprovidemoreac- ThedownsidewithcurrentDHT-baseddistributedtrack- curatedata.Forexample,thelifetimemeasurementcould ers, however, is that they result in random overlay con- be carried out by a client that interactively crawls the nections,asthereisnomechanismtoenforcemoreintel- routingtablesoftheDHTnodesandthenusesheartbeat ligentpeer-matchingtechniques. messages to monitor the uptimes of the nodes it learns With Comet we can address this limitation by using about. Thisapproachcouldprovidefaultydata,however, ASOs to track participating nodes, as well as construct iftheDHTcontainsfirewallednodesthatdonotreceive peerliststhatareoptimizedforarequestingnode. Peers or respond to such heartbeat messages.2 On the other could be matched in order to lower inter-node laten- hand,firewallednodesstillcommunicatewithneighbors, cies [33], maximize reciprocation probability based on forexampletoreplicatevalues.Therefore,measurements peer bandwidths [37], or lower ISP costs [62, 12]. We performed from ASOs within the DHT can be more ac- have implemented one such matching scheme that uses curate,aswewilldemonstratelater. the nodes’ network coordinates to predict inter-node la- Monitoring uses: An ASO can also maintain audit tenciesandprovidesalistofnearbypeerstoeachjoining trails, e.g., indicating where it has been stored thus far, node. WedescribethisindepthinSection6.3.2. who has read or updated the object, etc. Such tasks are 5.5 SignedASOs particularlyusefulfordebuggingandaidinrapidproto- typing. For example, this may help a developer to learn Theexamplesdiscussedsofaradheretothestrictsecurity whether a new ASO replication mechanism is operating policywesetout:ASOscannotperformoperationsonar- properly. Alternately,logscanalsobeusedforforensics. bitraryDHTkeysornodes. Wenowconsideruseswhere Listing 7 illustrates a monitoring application that tracks werelaxthisassumption, butrequirethattheASOcode thenodesstoringandaccessingavalue. besignedbytheDHTadministratoraftermanualverifi- This specific implementation comes with a few cation of its security properties. As we will see below, caveats. Each replica may have a different view of the thisallowstheDHTtodeploynewfunctionalityandser- listofnodesthathavestoredorreadthevalue.Toaddress vicesbyusingsignedASOsthataccessarbitraryDHTlo- this, the experimenter needs to get the union of the lists cations,butaresafe(i.e.,theydonotenableDoSattacks stored in all the replicas, consolidating them as a post- of targeted DHT nodes).3 We have considered signed processingstep. 3Insomecases,thesafetyoftheASOcodecouldpresumablybever- 2Infact,abouthalfthenodesinP2PDHTsarefirewalled[23]. ifiedautomatically,e.g.,byusingsophisticatedcompile-timeanalysis; ASOs in particular as a mechanism that a DHT’s devel- 5.6 Summary operoradministratorcouldusefortestingandevaluation This section described a set of example storage objects of new features, before they are added to the main-line that we have implemented using Comet. Through these DHTcode. examples, it should be clear that with very small exten- sions (on the order of a few lines or a few tens of lines Recursive Get: Vuze and many other DHTs support ofcode),aCometapplicationcancreateawiderangeof iterative routing for key lookups. In this approach, the powerfulstorageobjectbehaviorsthatwouldbeimpossi- node performing the lookup is involved in every step of bleinexistingdistributedstoragesystemsorDHTs. theroutingoperation,i.e.,itidentifiesthetargetnodeby repeatedly querying DHT nodes to find other nodes that 6 Evaluation are closer to the target key. An alternative is to perform We deployed Comet on approximately 200 PlanetLab recursiverouting,whereintermediatenodesontheroute hosts and evaluated our design in three steps. First, we pass the lookup directly to nodes that are closer to the characterizetheresourceutilizationofthevariousappli- key. Iterativelookupprovidesgreatercontroltothenode cationsthatwedeveloped. Second,wemeasuredmicro- performingthelookup(e.g., itcancontrollookupparal- benchmarkstounderstandtheoverheadsassociatedwith lelism), but it comes at the cost of increased latency. If active storage objects. Lastly we report on our experi- bothformsoflookupareavailable,anapplicationwould enceswithprototypingapplicationsusingComet. userecursivelookupsbydefault,butfallbackoniterative lookupsafterpersistentfailures[15]. 6.1 ApplicationCharacteristics WithsignedASOsitispossibletoimplementrecursive Table 2 shows resource consumption requirements for lookupseventhoughtheunderlyingDHTsupportsitera- our Comet applications. The Max Instructions column tivelookupbydefault(asisthecasewithChord,Kadem- gives the number of dynamic Lua instructions required lia, and Vuze). The node initiating the lookup creates a to execute the most expensive handler, while Execution query ASO,whichcontainsareferencetoitself,andalo- Time gives the execution time for that handler. Where calcallbackIDwhereitwouldliketoreceivetheanswer. thisvalueisdatasensitive,weprovideanestimatebased WhenthesignedASOiscreateditsonPuthandlerisin- on the expected maximum value. Code Size shows the voked; thehandlerqueriesthelocalroutingtabletofind sizeofeachASOwiththeminimumamountofdataand alivenodethatisclosesttothetargetkey,storesacopy MaxSizeisthemaximumsizetowhichtheASOsmight of the signed ASO on this node, and deletes itself from growforthatapplication.Fromthetableweseethatmost the current node. This process is repeated until one of ASOs execute fewer than 100 Lua instructions and are thenodesstoringthetargetisreached,andtheonUpdate smallerthan1KBinsize. handler of the target ASO sends the object’s value back totheoriginalnode,whichinitiatedtherequest. Application Max Execution Code Max Instructions Time Size Size Replication <10 4µs 0.223K <1K Caching and Hierarchical Publish-Subscribe: This SmartReplication <100 6µs 0.444K <1K ideacanbeextendedtoaccomplishbothcachingandhi- Timeouts ≈10 4µs 0.434K <1K Limited-ReadValue ≈10 4µs 0.553K <1K erarchical publish-subscribe data delivery. For caching, SensitiveValue <10 4µs 0.230K <1K the onUpdate handler can be modified to communicate PubSub 10,000s 54µs 0.498K 100K HierarchicalPubSub 100s 6µs 0.673K 1K theobjectnotonlytotherequestingnodebutalsotothe Lifetime(External) 100s 6µs 1K 6K/hr intermediatenodethatconveyedtherequest.Thenumber Lifetime(Internal) <100 6µs 1.776K ≈3K ofintermediatenodestowhichtheobjectisreplicatedcan Monitoring ≈10 4µs 0.971K 3K/hr SmartRendezvous 1,000s 14µs 1.107K 10K be determined by gathering and analyzing statistics on RecursiveGet ≈50 6µs 0.714K ≈1K object popularity (also accomplished using simple han- Table2: ExpectedApplicationResourceConsumption dler code), so that only popular objects are replicated at multiplenodes(asinBeehive[39]).Toimplementhierar- chicalpublish-subscribe,intermediatenodespropagatea 6.2 PerformanceandOverheads subscriptioneventtothenextnodeinthelookupprocess only if they haven’t done so before and maintain state We report on simple microbenchmark measurements to for subsequent queries routed to them. When a value is comparetheCPUandmemorycostsofVuzeandComet. published, it is propagated through a dissemination tree These experiments were run on an quad-core machine so that the communication load is distributed across all withXeonprocessorsclockedat2.67GHz. intermediatenodes(asinScribeandBayeux[45,64]). Single-Node Throughput. In this experiment, concur- rentgetoperationsareperformedonmanyvaluesstored studyingthisispartoffuturework. in the target node. We measure the throughput of get 1e+07 800 Comet - Null values Log(Throughput) (operations per second) 1 011 0e01 0+001 0000010 060001 1V01u100zMeKK0 ( iiiinNnnnssosstttttrrrr uuauuccccctttttiiiiioovoonennn)ssss Memory consuption in MB 12345670000000 00000000 Vuze - Null values 0 20 40 60 80 100 10 100 1000 10000 100000 1e+06 Offered load (concurrent operations) Log(Number of values stored) (a) Single-NodeThroughput. (b) MemoryFootprint. Figure2: Microbenchmarks. requeststhatreturnsuccessfullyusingaclosedfeedback Comet nodes as we increase the number of stored ob- loop. All operations are issued locally on the node, so jects. Again using the median number of values stored thatnetworklatencydoesnotaffectthroughput. per Vuze node (around 400), the difference in memory Figure 2(a) compares the throughput of objects with consumption at this level is negligible (about 36MB for different ASO execution costs, expressed as the number bothCometandVuze). LonglivedDHTnodescanstore ofLuabytecodeinstructionsexecutedperhandler. Both 10,000s of values, and the highest observed is around Comet and Vuze experience peak throughput when the 30,000values[49].Intheserarecases,ouroverheadrela- number of concurrent operations is equal to the num- tivetoVuzeisabout27%,buteventhenthetotalmemory ber of cores (eight). ASOs with zero instructions per footprintisstillreasonable. handler are functionally equivalent to Vuze values as We next consider a workload where Comet object they simply return themselves. The peak throughput of sizes are exponentially distributed with an average size CometASOsisabout60%smallerthanthepeakthrough- of 10KB. In this case, a node with 500MB can store on put of Vuze (1.4M operations per second as opposed to average 50,000 values. If we assume an order of mag- 3.5M operations per second). This shows the cost of nitudemorevaluespernodethaninVuze(4,000instead the Comet/Lua execution environment. Previous mea- of 400), and an order ofmagnitude larger values (10KB surements[49]showthatthetypicalDHTloadonVuze insteadof1KBlimitimposedbyVuze),themediannode clients in the wild is at most a few hundred operations would consume about 80MB (40MB of startup memory persecond,whichmakestheadditionalCometoverhead costsandanother40MBfortheASOs)inmemory.4 relatively insignificant in this context. As we increase 6.3 ApplicationExperience thecomputationalcomplexityoftheaverageASO(1Kto 1M instructions per handler), the throughput decreases, Wenowreportonourexperiencesinprototypingandde- but still remains well above the maximum current Vuze ployingsomeoftheapplicationsdescribedinSection5. workload. 6.3.1 MeasuringNodeLifetimes Operation Latency. At the 90th percentile, with maxi- WerevisittheexperimentperformedbyFalkneretal.[20] mum throughput (8 concurrent operations in our exper- tomeasurethelifetimesofnodesintheVuzeDHT.This iments), a request involving 100 Lua instructions has a experiment was done by performing random get opera- latency of about 300 microseconds. For handlers with tionsfromseveralVuzeclientsinordertogatherapproxi- 1Minstructions(twoordersofmagnitudemorethanour mately300KIPsparticipatingintheDHT.Thecollection most compute-intensive handlers), it is 13 milliseconds. ofnodeswasthenpingedevery2.5minutestocheckfor The latency for a Vuze DHT lookup is on the order of liveness. Theauthorsobservedthatnearlyhalfthenodes seconds,thereforethelatencyimposedbyevenextremely were immediately unavailable after first being detected. computationallyintensiveASOsisnotsignificant. One weakness of the methodology employed is that the Memory Footprint. In this experiment, we store in- clientscouldnotdifferentiatenodesthatareunreachable creasing numbers of values in the nodes. For the Vuze becauseofNATsfromthosethathavelefttheDHT.Us- nodes,thestring“helloworld”isstoredatdifferentkeys, ing measurement nodes that have active communication whileforCometnodeswestoreanequivalentLuaASO channels with NATed DHT nodes would help minimize which returns the same string upon a get request. Fig- ure2(b)comparesthememoryfootprintoftheVuzeand 4VuzeandCometconsumeabout40MBwithoutstoringanyvalues.

Comet: An active distributed key-value store - Amit Levy

Similar Comet: An active distributed key-value store - Amit Levy