Searching and Browsing Linked Data with SWSE: the Semantic

55 Pages·2011·1.23 MB·English

SearchingandBrowsingLinkedDatawithSWSE:
theSemanticWebSearchEngine
AidanHogana,AndreasHarthb,Ju¨rgenUmbricha,SheilaKinsellaa,AxelPolleresa,
StefanDeckera
aDigitalEnterpriseResearchInstitute,NationalUniversityofIreland,Galway
bAIFB,KarlsruheInstituteofTechnology,Germany
Abstract
Inthispaper,wediscussthearchitectureandimplementationoftheSemanticWebSearchEngine(SWSE).Following
traditional search engine architecture, SWSE consists of crawling, data enhancing, indexing and a user interface for
search,browsingandretrievalofinformation;unliketraditionalsearchengines,SWSEoperatesoverRDFWebdata–
looselyalsoknownasLinkedData–whichimpliesuniquechallengesforthesystemdesign,architecture,algorithms,
implementation and user interface. In particular, many challenges exist in adopting Semantic Web technologies for
Web data: the unique challenges of the Web – in terms of scale, unreliability, inconsistency and noise – are largely
overlookedbythecurrentSemanticWebstandards.Herein,wedescribethecurrentSWSEsystem,initiallydetailing
thearchitectureandlaterelaboratinguponthefunction,design,implementationandperformanceofeachindividual
component.Insodoing,wealsogiveaninsightintohowcurrentSemanticWebstandardscanbetailored,inabest-
effort manner, for use on Web data. Throughout, we offer evaluation and complementary argumentation to support
ourdesignchoices,andalsoofferdiscussiononfuturedirectionsandopenresearchquestions.Later,wealsoprovide
candid discussion relating to the difficulties currently faced in bringing such a search engine into the mainstream,
andlessonslearntfromroughlysixyearsworkingontheSemanticWebSearchEngineproject.
Keywords: Websearch,semanticsearch,RDF,SemanticWeb,LinkedData
1. Introduction model,fastresponsetimes,andastuteprioritisation
ofresults,Google[18]hasbecometheyard-stickfor
Offering a minimalistic and uncluttered user in- Web-search, servicing approximately 64.6% of tra-
terface, a simple keyword-based user-interaction ditional Web search queries2 over billions of Web
documents.Arguably,Googlereachestheimminent
limit of providing the best possible search over the
Emailaddresses: [email protected](AidanHogan), largely HTML data it indexes. However, from the
[email protected](AndreasHarth), userperspective,thecoreGoogleengine(hereserv-
[email protected](Ju¨rgenUmbrich), ing as the archetype for traditional HTML search
[email protected](SheilaKinsella),
engines,suchasYahoo,MSN/Bing,AOL,Ask,etc.)
[email protected](AxelPolleres),
is far from the consummate Web search solution:
[email protected](StefanDecker).
1 The work presented in this paper has been funded
in part by Science Foundation Ireland under Grant No. 2 Statistics taken from Nielsen MegaView Search for
SFI/08/CE/I1380(Lion-2)andbyanIRCSETpostgraduate ∼11 g searches recorded in Aug. 2009: cf. http://
scholarship. searchenginewatch.com/3634991
PreprintsubmittedtoElsevier 14June2011
Googledoesnottypically producedirectanswersto Times,Flickr,Last.fm,scientificpublishingindexes,
queries, but instead typically recommends a selec- biomedicalinformationandgovernmentalagencies.
tion of related documents from the Web. We note This precedent raises an obvious question: assum-
thatinmorerecentyears,Googlehasbeguntopro- ing large-scale adoption of high-quality RDF pub-
vide direct answers to prose queries matching cer- lishing on the Web, could a search engine indexing
tain common templates – for example, “population RDF feasibly improve upon current HTML-centric
of china” or “12 euro in dollars” – but again, such engines? Theoretically at least, such a search en-
functionalityislimitedtoasmallsubsetofpopular ginecouldofferadvancedqueryingandbrowsingof
userqueries.Furthermore,Googlenowprovidesin- structured data with search results automatically
dividualandfocussedsearchinterfacesoverimages, aggregated from multiple documents and rendered
videos, locations, news articles, books, research pa- directlyinacleanandconsistentuser-interface,thus
pers, blogs, and realtime social media – although reducing the manual effort required of its users.
thesetoolsareinarguablypowerful,theyarelimited Indeed, there has been much research devoted to
totheirrespectivedomain. thistopic,withvariousincarnationsof(mostlyaca-
Inthegeneralcase,Googleisnotsuitableforcom- demic) RDF-centric Web search engines emerging
plex information gathering tasks requiring aggre- – e.g., Swoogle, Falcons, Watson, Sindice – and in
gation from multiple indexed documents: for such this paper, we present the culmination of over six
tasks,usersmustmanuallyaggregatetidbitsofper- yearsresearchonanothersuchengine:the“Seman-
tinent information from various recommended het- ticWebSearchEngine”(SWSE)3.
erogeneous sites,eachsuchsitepresentinginforma- Indeed, the realisation of SWSE has implied two
tioninitsownformattingandusingitsownnaviga- majorresearchchallenges:
tionsystem.Ineffect,Google’slimitationsarepred- (i) the system must scale to large amounts of
icatedonthelackofstructureinHTMLdocuments, data;and
whosemachineinterpretabilityislimitedtotheuse (ii) the system must be robust in the face of
ofgenericmarkup-tagsmainlyconcernedwithdoc- heterogeneous, noisy, impudent, and possibly
ument rendering and linking. Although Google ar- conflictingdatacollectedfromalargenumber
guablymakesthebestofthelimitedstructureavail- ofsources.
able in such documents, most of the real content is SemanticWebstandardsandmethodologiesarenot
contained in prose text which is inherently difficult naturallyapplicableinsuchanenvironment;inpre-
for machines to interpret. Addressing this inherent sentingthedesignandimplementationofSWSE,we
problem with HTML Web data, the Semantic Web show how standard Semantic Web approaches can
movement provides a stack of technologies for pub- be tailored to meet these two challenging require-
lishingmachine-readabledataontheWeb,thecore ments, often taking cues from traditional informa-
ofthestackbeingtheResourceDescriptionFrame- tionretrievaltechniques.
work(RDF). As such, we present the core of a system which
Using URIs to name things – and not just doc- we demonstrate to provide scale, and which is dis-
uments – RDF offers a standardised and flexible tributed over a cluster of commodity hardware.
framework for publishing structured data on the Throughout, we focus on the unique challenges of
Web(i)suchthatdatacanbelinked,incorporated, applying standard Semantic Web techniques and
extendedandre-usedbyotherRDFdataacrossthe methodologies, and show why the consideration of
Web, (ii) such that heterogeneous data from inde- the source of data is an integral part of creating
pendentsourcescanbeautomaticallyintegratedby a system which must be tolerant to Web data –
software-agents, and (iii) such that the meaning of in particular, we show how Linked Data principles
datacanbewell-definedusinglightweightontologies can be exploited for such purposes. Also, there are
described in RDF using the RDF Schema (RDFS) many research questions still very much open with
andWebOntologyLangauge(OWL)standards. respecttothedirectionoftheoverallsystem,aswell
Thanks largely to the “Linked Open Data” asimprovementstobemadeintheindividualcom-
project [14] – which has emphasised more prag- ponents; we discuss these as they arise, rendering
matic aspects of Semantic Web publishing – a rich a road-map of past, present and possible future re-
lode of open RDF data now resides on the Web: searchintheareaofWebsearchoverRDFdata.
this“WebofData”includescontentexportedfrom,
for example: Wikipedia, the BBC, the New York 3 http://swse.deri.org/
2
Morespecifically,inthispaperwe: turnsearchresultsindifferentdomain-specificcate-
– present the architecture and modus-operandi of gories (Web, Images, Videos, Shopping, etc.), data
our system for offering search and browsing over ontheSemanticWebisflexiblytypedanddoesnot
RDFWebdata(Section2); need to follow pre-defined categories. Returned ob-
– presenthigh-levelrelatedworkinRDFsearchen- jects can represent people, companies, cities, pro-
gines(Section3); teins–anythingpeoplecaretopublishdataabout.
– present core preliminaries required throughout InamannerfamiliarfromtraditionalWebsearch
therestofthepaper(Section4); engines, SWSE allows users to specify keyword
– detailthedesign,distributedimplementationand queriesinaninputboxandrespondswitharanked
evaluation of the offline index building compo- list of result snippets; however, the results refer to
nents, including crawling (Section 5), consolida- entities not documents. A user can then click on
tion (Section 6), ranking (Section 7), reasoning an entity snippet to derive a detailed description
(Section8),andindexing (Section9); thereof. The descriptions of entities are automati-
– summarise the runtimes for each of the offline callyaggregatedfromarbitrarilymanysources,and
tasks(Section10); userscancross-checkthesourceofparticularstate-
– detailthedesign,distributedimplementationand ments presented; descriptions also include inferred
evaluation of the runtime components, includ- data – data which has not necessarily been pub-
ingour(lightweight)query-processor(Section11) lished, but has been derived from the existing data
anduser-interface(Section12); throughreasoning.Userscansubsequentlynavigate
– concludewithdiscussionoffuturedirections,open to related entities, as such, browsing the Web of
researchchallengesandcurrentlimitationsofWeb Data.
searchoverRDFdata(Sections13–14). Along these lines, Figure 1 shows a screen-
shot containing a list of entities returned as a
result to the keyword search “bill clinton”
2. SystemOverview
– such results pages are familiar from HTML-
centric engines, with the addition of re-
In this section, we outline the functionality of
sult types (e.g., DisbarredAmericanLawyers,
the SWSE system. We begin with an overview of
AmericanVegitarians, etc.). Results are aggre-
the functionality offered to end users (Section 2.1).
gated from multiple sources. Figure 2 shows a
Thereafter we present the high-level SWSE archi-
screenshot of the focus (detailed) view of the
tecture (Section 2.2) and describe the distributed
Bill Clinton entity, with data aggregated from
framework upon which our components operate
54 documents spanning six domains (bbc.co.uk,
(Section 2.3). Wrapping up this section, we detail
dbpedia.org, freebase.com, nytimes.com,
the software and hardware environments used for
rdfize.com and soton.ac.uk), as well as novel
theexperimentsdetailedherein(Section2.4).
datafoundthroughreasoning.
2.1. ApplicationOverview
2.2. SystemArchitecture
Toputlaterdiscussionintocontext,wenowgive
a brief overview of the lightweight functionality of The high-level system architecture of SWSE
the SWSE system; please note that although our looselyfollowsthatoftraditionalHTMLsearchen-
methodsandalgorithmsaretailoredforthespecific gines [18]. Figure 3 details the pre-runtime archi-
needsofSWSE,manyaspectsoftheirimplementa- tecture of our system, showing the components in-
tion, design and evaluation apply to more general volved in achieving a local index of RDF Web data
scenarios. amenableforsearch.Liketraditionalsearchengines,
Unlike prevalent document-centric Web search SWSE contains components for crawling, ranking
engines, SWSE operates over structured data and and indexing data; however, there are also compo-
holdsanentity-centricperspectiveonsearch:incon- nents specifically designed for handling RDF data,
trast to returning links to documents containing namely the consolidation component and the rea-
specified keywords [18], SWSE returns data rep- soning component. The high-level index building
resentations of real-world entities. While current processisasfollows:
search engines such as Google, Bing and Yahoo re- – the crawler accepts a set of seed URIs and re-
3
Fig.1.ResultsviewforkeywordqueryBillClinton Fig.2.FocusviewforentityBillClinton
trievesalargesetofRDFdatafromtheWeb; example,see[54]).
– theconsolidationcomponenttriestofindsynony- Wewilldetailthedesignandoperationofeachof
mous(i.e.,equivalent)identifiersinthedata,and thecomponentsinthefollowingsections,butbefore-
canonicalises the data according to the equiva- hand, we present the distribution framework upon
lencesfound; whichallofourcomponentsareimplemented.
– the ranking component performs links-based
analysis over the crawled data and derives scores
indicating the importance of individual elements
2.3. DistributionAbstraction
in the data (the ranking component also consid-
ers URI redirections encountered by the crawler
In order to scale, we deploy each of our compo-
whenperformingthelinks-basedanalysis);
nents over a distributed framework which we now
– the reasoning component materialises new data
briefly describe; Figure 4 illustrates the distributed
whichisimpliedbytheinherentsemanticsofthe
operations possible in our framework. The frame-
inputdata(thereasoningcomponentalsorequires
workisbasedonasharednothingarchitecture[116]
URIredirectioninformationtoevaluatethetrust-
and consists of one master machine which orches-
worthinessofsourcesofdata);
trates the given tasks, and several slave machines
– theindexingcomponentpreparesanindexwhich
whichperformpartsofthetaskinparallel.
supports the information retrieval tasks required
The master machine can instigate the following
bytheuserinterface.
distributedoperations:
Subsequently, the query-processing and user-
– scatter: partition a file into chunks given some
interfacecomponentsservicequeriesovertheindex
local split function, and send the chunks to indi-
builtintheprevioussteps.
vidual machines – usually only used to initialise
Our methods follow the standards relating to
ataskandseedtheslavemachineswithaninitial
RDF [92], RDFS [65] and OWL [115], and leverage
setofdataforprocessing;
theLinkedDataprinciples[9]whichstatehowRDF
– run:requesttheparallelexecutionofataskbythe
shouldbepublishedontheWeb.Assuch,ourmeth-
slave machines – such a task either involves pro-
ods should be 100% precise (aka. sound) with re-
cessingofsomelocaldata(embarrassinglyparal-
specttodatacorrectly publishedaccordingtothese
lel execution), or execution of the co-ordinate
documents, but we note that oftentimes, the noise
methodbytheslaveswarm;
inherentinheterogenousRDFWebdatamaycreate
– gather: gathers chunks of output data from the
unintendedresults.Wecharacterisetheseproblems
slaveswarmandperformssomelocalmerge func-
as they occur, although we know of no method for
tion over the data – this is usually performed to
accurately determining the amount of incorrect or
createasingleoutputfileforatask,ormoreusu-
unwantedresultsgeneratedbythesetasks–wenote
ally to gather global knowledge required by all
thatsuchconsiderationsmayalsobesubjective(for
slavemachinesforafuturetask;
4
Pre-Runtime Runtime
Seed Crawl Consolidate Rank Reason Index
URIs
Query
RDF Data / Consolidated Identifier Reasoned Processing
Redirects Data Ranks Data
Intermediary Results
Fig.3.SystemArchitecture
– flood:broadcastglobalknowledgerequiredbyall a convenient means of development given our Java
slavemachinesforafuturetask. code-base.
The master machine is intended to disseminate in- All of our evaluation is based on nine machines
put data to the slave swarm, to provide the control connectedbyGigabitethernet4,eachwithuniform
logicrequiredbythedistributedtask(commencing specifications; viz.: 2.2GHz Opteron x86-64, 4GB
tasks,co-ordinatingtiming,endingtasks),togather main memory, 160GB SATA hard-disks, running
andlocallyperformtasksonglobalknowledgewhich Java1.6.0 12onDebian5.0.4.Pleasenotethatmuch
the slave machines would otherwise have to repli- of the evaluation presented in this paper assumes
cate in parallel, and to transmit globally required that the slave machines have roughly equal specifi-
knowledge.Themastermachinecanalsobeusedto cationsinordertoensurethattasksfinishinroughly
computethefinalresultforagivendistributedtask; thesametime,assumingevendatadistribution.
however,theendgoalofourdistributedframework Wecurrentlydonotconsidermoreadvancedtop-
istoproduceadistributedindexovertheslavema- icsinourarchitecture–suchasload-balancing(with
chines,thusthistaskisneverrequiredinoursystem. the exception of evenly distributing data), replica-
Theslavemachines,aswellasperformingtasksin tion, uptime and counteracting hardware failure –
parallel, can perform the following distributed op- and discussion of these fall outside of the current
eration(onthebehestofthemastermachine): scope.
– co-ordinate: local data on each machine is par-
titioned according to some split function, with 3. RelatedWork
thechunkssenttoindividualmachinesinparallel;
eachmachinealsogatherstheincomingchunksin
In this section, we give an overview of related
parallelusingsomemerge function.
work, firstly detailing distributed architectures for
Theaboveoperationallowsslavemachinestopar-
Web search (Section 3.1), then discussing related
tition and disseminate intermediary data directly
systems in the field of “Hidden Web” and “Deep
tootherslavemachines;theco-ordinateoperation
Web” (Section 3.2), and finally describing current
could be replaced by a pair of gather/scatter op-
systems that offer search and browsing over RDF
erations performed by the master machine, but we
Webdata(Section3.3)–forafurthersurveyofthe
wish to avoid the channelling of all such intermedi-
latter,cf.[127].Pleasenotethatwewillgivefurther
arydatathroughonemachine.
detailed related work in the context of each compo-
We note that our framework resembles the
nentthroughoutthepaper.
MapReduce framework [31], with scatter loosely
corresponding to the Map operation, and gather
loosely corresponding to the Reduce operation; 3.1. DistributedWebSearchArchitectures
similarly, the split function corresponds loosely to
thePartitionfunction,andtheco-ordinatefunc- Distributed architectures have long been com-
tion loosely corresponds to the Shuffle operation monintraditionalinformation-retrievalbasedWeb
intheMapReducesetting. search engines, incorporating distributed crawl-
ing,ranking,indexingandquery-processingcompo-
nents. Although all mainstream search engines are
2.4. SoftwareEnvironment/Hardware
based on distributed architectures, details are not
We instantiate this architecture using the stan- 4 We observe, e.g., a max FTP transfer rate of 38MB/sec
dard Java Remote Method Invocation libraries as betweenmachines.
5
s0 s0 s0
scatter0 s1 run(args0) s1 flood s1
scatter1 init’ed run(args1) flood
... ... ready ...
split scatter... run(...) flood
m scattern sn m run(argsn) sn m flood sn
ready hed send
s0 split split s1 finis s0
merge merge gather0 s1
coordinate0...n gather1 ...
gather
merge merge prepare merge
... split split sn m gathern sn
Fig.4.DistributionMethodsArchitecture
commonly published. Again, one of the most well- ing techniques since content is usually generated
knownsearchenginearchitecturesisthatpreviously by means of bespoke flexible queries; thus, tradi-
described for the Google search engine [18]. More tional search engines can only skim the surface of
recent publications relating to the Google architec- suchinformation[66].Infact,suchdata-richsources
turerelatetotheMapReduceframeworkpreviously haveleadtoearlyspeculativeworkonentity-centric
alludedto[31],andtotheunderlyingBigTable[23] search[28].
distributeddatabasesystem. Approaches to exploit such sources heavily rely
Similar system architectures have been defined on manually constructed, site-specific wrappers to
in the literature, including WebBase [70] which in- extract structured data from HTML pages [21],
cludesanincrementalcrawler,storagemanager,in- or to communicate directly with the underlying
dexerandqueryprocessor;inparticular,theauthors database of such sites [24]. Some works have also
focus on hash- and log-based partitioning for stor- looked into automatically crawling such hidden-
ing incrementally-updated vast repositories of Web Web sources, by interacting with forms found dur-
documents. The authors of [94] also describe a sys- ing traditional crawls [112]; however, this approach
tem for building a distributed inverted-index over is “task-specific” and not appropriate for general
a large corpus of Web pages, for subsequent analy- crawling.
sisandquery-processing:theyemployanembedded The Semantic Web may represent a future direc-
distributeddatabasesystem. tion for bringing deep-Web information to the sur-
Much of the work presented herein is loosely in- face,leveragingRDFasacommonandflexibledata-
spired by such approaches, and thus constitutes model for exporting the content of such databases,
an adaptation of such works for the purposes of leveragingRDFSandOWLasameansofdescribing
searchoverstructureddata.Sinceweconsiderrepli- the respective schemata, and thus allowing for au-
cation, fault tolerance, incremental indexing, etc., tomaticintegrationofsuchdatabyWebsearchen-
currently out of scope, many of our techniques are gines.EffortssuchasD2R(Q)[13]seemanaturalfit
morelightweightthanthosediscussed. forenablingRDFexportsofsuchonlinedatabases.
3.2. HiddenWeb/DeepWebApproaches 3.3. RDF-centricSearchEngines
So called “Hidden Web” or “Deep Web” ap- Earlyprototypesusingtheconceptsofontologies
proaches [21] are predicated on the premise that a and semantics on the Web include Ontobroker [32]
vastamountoftheinformationavailableontheWeb and SHOE [67], which can be seen as predecessors
is veiled behind sites with heavy dynamic content, tostandardisationeffortssuchasRDFSandOWL,
usually backed by relational databases. Such infor- describing how data on the Web can be given in
mation is largely impervious to traditional crawl- structured form, and subsequently crawled, stored,
6
inferencedandqueriedover. exploiting RDF for the purposes of domain-specific
Swoogle5 offers search over RDF documents by querying;forexample,therecentGoWebsystem10
meansofaninvertedkeywordindexandarelational demonstrates the benefit of searching structured
database[38].Swooglecalculatesmetricsthatallow data for the biomedical domain [36]. However, in
ontologydesignerstocheckthepopularityofcertain catering for a specific domain, such systems do not
properties and classes. In contrast to SWSE, which targetthesamechallengesanduse-casesaswedo.
ismainlyconcernedwithentitysearchoverinstance
data,Swoogleismainlyconcernedwithmoretradi-
4. Preliminaries
tionaldocument-searchoverontologies.
Watson6 provides a similar effort to provide
Before we continue, we briefly introduce some
keyword search facilities over Semantic Web docu-
standard core notation used throughout the paper
ments, but additionally provides search over enti-
– relating to RDF terms (constants), triples and
ties [114,29]. However, they do not include compo-
quadruples – and also discuss Linked Data princi-
nents forconsolidationor reasoning,andseemingly
ples. Note that in this paper, we will generally use
insteadfocusonprovidingAPIstoexternalservices.
bold-face to refer to infinite sets: e.g., G refers to
Sindice7 isaregistryandlookupserviceforRDF
the set of all triples; we will use calligraphy font to
files based on Lucene and a MapReduce frame-
denoteasubsetthereof:e.g.,G isaparticularsetof
work [103]. Sindice originally focussed on provid-
triples,whereG ⊂G.
ing an API for finding documents which reference
a given RDF entity or given keywords – again,
4.1. ResourceDescriptionFramework
document-centric search. More recently however,
Sindicehasbeguntoofferentitysearchintheform
ofSig.Ma8 [120].However,Sig.mamaintainsaone- The Resource Description Framework provides a
to-onerelationshipbetweenkeywordsearchandre- structuredmeansofpublishinginformationdescrib-
sults, representing a very different user-interaction ing entities through use of RDF terms and RDF
modeltothatpresentedherein. triples, and constitutes the core data model for our
The Falcons Search engine9 offers entity-centric searchengine.Inparticular,RDFallowsforoption-
searching for entities (and concepts) over RDF ally defining names for entities using URIs and al-
data [27]. They map certain keyword phrases to lows for subsequent re-use of URIs across the Web;
query relations between entities, and also use class using triples, RDF allows to group entities into
hierarchies to quickly restrict initial results. Con- named classes, allows to define named relations be-
ceptually,thissearchenginemostcloselyresembles tween entities, and allows for defining named at-
our approach. However, there are significant differ- tributes of entities using string (literal) values. We
ences in how the individual components of SWSE nowbrieflygivesomenecessarynotation.
andFalconsaredesignedandimplemented.Forex-
ample, like us, they also rank entities, but using a RDF Constant Given a set of URI references U,
logarithm of the count of documents in which they a set of blank nodes B, and a set of literals L, the
are mentioned – we employ a links-based analysis setofRDFconstants isdenotedbyC=U∪B∪L.
ofsources.Also,Falconssupportsreasoninginvolv- The set of blank nodes B is a set of existensially
ing class hierarchies, whereas we apply a more gen- quantified variables. The set of literals is given as
eralrulebasedapproach,applyingascalablesubset L = L ∪L , where L is the set of plain literals
p d p
of OWL 2 RL/RDF rules. Such differences will be and L is the set of typed literals. A typed literal is
d
discussedfurtherthroughoutthispaper,andinthe thepairl =(s,d),wheres isthelexicalformofthe
contextoftheindividualcomponents. literalandd∈U isadatatypeURI.ThesetsU,B,
Aside from the aforementioned domain-agnostic L andL arepairwisedisjoint.
p t
searchsystems,wenotethatothersystemsfocuson Please note that in this paper, we treat blank
nodesastheirskolemversions:i.e.,notasexistential
variables,butasdenotingtheirownsyntacticform.
5 http://swoogle.umbc.edu/
We also ensure correct merging of RDF graphs [65]
6 http://watson.kmi.open.ac.uk/WatsonWUI/
7 http://sindice.com/ byusingblank-nodelabelsuniqueforagivensource.
8 http://sig.ma
9 http://iws.seu.edu.cn/services/falcons/ 10http://gopubmed.org/goweb/
7
ForURIs,weusenamespaceprefixesinthispaper the set of data sources S ⊂ U as the set of URIs
ascommonintheliterature–thefullURIscanbere- S = {s ∈ U | get(s) (cid:54)= ∅}. We define the reference
trievedfromtheconvenienthttp://prefix.ccser- functionrefs:C→2SasthemappingfromanRDF
vice. For space reasons, we sometimes denote owl: termtothesetofdatasourcesthatmentionit.
asthedefaultnamespace.
RDFTripleinContext/RDFQuadruple Apair(t,
RDFTriple Atriplet =(s,p,o)∈(U∪B)×U×
c)withatriplet =(s,p,o),c ∈Sandt ∈get(c)is
(U∪B∪L) is called an RDF triple. In a triple (s,
calledatripleincontext c.Wemayalsoreferto(s,p,
p,o),s iscalledsubject,p predicate,ando object.
o,c)asanRDFquadruple orquadq withcontextc.
RDFGraph WecallafinitesetoftriplesanRDF
graph G ⊂GwhereG=(U∪B)×U×(U∪B∪L). HTTP Dereferencing We define dereferencing as
thefunctionderef:U→UwhichmapsagivenURI
totheidentifierofthedocumentreturnedbyHTTP
RDF Entity We refer to the referent of a URI
lookupoperationsuponthatURIfollowingredirects
or blank-node as an RDF entity, or commonly just
(for a given finite and non-cyclical path) [47], or
entity.
which maps a URI to itself in the case of failure.
Notethatwedonotdistinguishbetweenthediffer-
4.2. Linked Data, Data Sources, Quadruples, and
ent 30x redirection schemes, and that this function
Dereferencing
wouldinvolve,e.g.,strippingthefragmentidentifier
of a URI [11]. Note that all HTTP level functions
In order to cope with the unique challenges of {get,refs,deref}aresetatthetimeofthecrawl,and
handling diverse and unverified Web data, many of are bounded by the knowledge of our crawl: for ex-
ourcomponentsandalgorithmsrequireinclusionof ample,refswillonlyconsiderdocumentsaccessedby
anotionofprovenance:considerationofthesourceof thecrawl.
RDFdatafoundontheWeb.Tightlyrelatedtosuch
notions are the Linked Data Best Practices (here
5. Crawling
paraphrasing [9]):
LDP1 useURIstonamethings;
We now begin the discussion of the first compo-
LDP2 useHTTPURIssothatthosenamescanbe
nent required for building the index, and thus for
lookedup;
retrieving the raw RDF documents from the Web:
LDP3 provide useful structured information when
thatis,thecrawler.Ourcrawlerstartswithasetof
alook-uponaURIismade–loosely,calledderef-
seedURIs,retrievesthecontentofURIs,parsesand
erencing;
writes content to disk in the form of quads, and re-
LDP4 includelinksusingexternalURIs.
cursivelyextractsnewURIsforcrawling.Welever-
In particular, within SWSE, these best-practices
age Linked Data principles (see Section 4.2) to dis-
form the backbone of various algorithms designed
tointeractandbetoleranttoWebdata. covernewsources,wherefollowingLDP2andLDP3,
We must thus extend RDF triples with context weconsiderallhttp:protocolURIsextractedfrom
anRDFdocumentascandidatesforcrawling.
to denote the source thereof [53,58]. We also de-
Like traditional HTML crawlers, we identify the
finesomerelationsbetweentheidentifierforadata
followingrequirementsforcrawling:
source, and the graph it contains, including a func-
– Politeness: The crawler must implement po-
tion to represent HTTP redirects prevalently used
liteness restrictions to avoid hammering remote
inLinkedDataforLDP3[9].
servers with dense HTTP GET requests and
to abide by policies identified in the provided
DataSource Wedefinethehttp-download function
robots.txtfiles12.
get:U→2GasthemappingfromaURItoanRDF
– Throughput:Thecrawlershouldcrawlasmany
graph it may provide by means of a given HTTP
URIs as possible in as little time as is possible
lookup [47] which directly returns status code 200
withintheboundsofthepolitenesspolicies.
OKanddatainasuitableRDFformat.11 Wedefine
112G referstothepowersetofS. 12http://www.robotstxt.org/orig.html
8
– Scale: The crawler should employ scalable tech- Algorithm1Algorithmforcrawling
niques,andon-diskindexingasrequired. Require: Seeds, Rounds, Pld-Limit, Min-
– Quality: The crawler should prioritise crawling Delay
URIsitconsiderstobe“highquality”. 1: frontier ←Seeds
Thus, the design of our crawler is inspired by re- 2: pld0...n ←newqueue
lated work from traditional HTML crawlers. Addi- 3: stats←newstats
tionally–andspecifictocrawlingstructureddata– 4: whilerounds+1<Roundsdo
weidentifythefollowingrequirement: 5: putfrontier intopld0...n
– Structured Data: The crawler should retrieve 6: whiledepth+1<Pld-Limitdo
a high percentage of RDF/XML documents and 7: fori=0tondo
avoid wasted lookups on unwanted formats: e.g., 8: prioritise(pldi,stats)
HTMLdocuments. 9: endfor
Currently, we crawl for RDF/XML syntax docu- 10: start←current time()
ments–RDF/XMLisstillthemostcommonlyused 11: fori=0tondo
syntaxforpublishingRDFontheWeb,andweplan 12: curi =calculate cur(pldi,stats)
infuturetoextendthecrawlertosupportotherfor- 13: if curi >random([0,1])then
matssuchasRDFa,N-TriplesandTurtle. 14: geturifrompldi
The following algorithm details the operation of 15: urideref =deref(uri)
thecrawler,andwillbeexplainedindetailthrough- 16: if urideref =urithen
outthissection. 17: G =get(uri)
18: outputG
19: UG ←URIsinG
5.1. High-levelApproach
20: UG ←pruneblacklistedfromUG
21: addunseenURIsinUG tofrontier
Ourhigh-levelapproachistoperformbreath-first 22: updatestatswrt.UG
crawling,followingprecedentsetbytraditionalWeb 23: else
crawlers (cf. [15] [69]): the crawl is conducted in 24: if urideref isunseenthen
rounds, with each round crawling a frontier. On a 25: addurideref tofrontier
high-level,Algorithm1representsthisround-based 26: updatestatsforurideref
approachapplyingRoundsnumberofrounds.The 27: endif
frontier comprises of seed URIs for round 0 (Algo- 28: endif
rithm 1, Line 1), and thereafter with novel URIs 29: endif
extracted from documents crawled in the previous 30: endfor
round (Algorithm 1, Line 21). Thus, the crawl em- 31: elapsed←current time()-start
ulates a breadth-first traversal of inter-linked Web 32: if elapsed<Min-Delaythen
documents. (Note that the algorithm is further tai- 33: wait(Min-Delay−elapsed)
lored according to requirements we will describe as 34: endif
thesectionprogresses.) 35: endwhile
As we will see later in the section, the round- 36: endwhile
basedapproachfitswellwithourdistributedframe-
work, allowing for crawlers to work independently
proach to crawling according to observations that
foreachround,andco-ordinatingnewfrontierURIs
writing/reading concurrently and dynamically to a
attheendofeachround.Additionally,[99]showthat
single queue can become the bottleneck in a large-
a breadth-first traversal strategy tends to discover
scalecrawler.
high-qualitypagesearlyoninthecrawl,withthejus-
tification that well-linked documents (representing
higherqualitydocuments)aremorelikelytobeen- 5.1.1. IncorporatingPoliteness
countered in earlier breadth-first rounds; similarly, The crawler must be careful not to bite the
breadthfirstcrawlingleadstoamorediversedataset hands that feed it by hammering the servers of
earlieron,ratherthanadepth-firstapproachwhich data providers or breaching policies outlined in the
may end up traversing deep paths within a given provided robots.txt file [118]. We use pay-level-
site. In [88], the authors justify a rounds-based ap- domains [88] (PLDs; a.k.a. “root domains”; e.g.,
9
bbc.co.uk) to identify individual data-providers, efit of maintaining a persistent state for the crawl
and implement politeness on a per-PLD basis. and thus offering a “continuation point” useful for
Firstly, when we first encounter a URI for a PLD, extensionofanexistingcrawl,orrecoveryfromfail-
wecross-checktherobots.txtfiletoensurethatwe ure.
arepermittedtocrawlthatsite;secondly,weimple- We implement the on-disk storage of URIs using
menta“minimumPLDdelay”toavoidhammering Berkeley DB which comprises of two indexes – the
servers, viz.: a minimum time-period between sub- first provides lookups for URI strings against their
sequent requests to a given PLD. This is given by status (polled/unpolled); the second offers a key-
Min-Delay inAlgorithm1. sortedmapwhichcaniterateoverunpolledURIsin
In order to accommodate the min-delay policy decreasing order of inlink count. The inlink count
withminimaleffectonperformance,wemustrefine reflects the total number of documents from which
ourcrawlingalgorithm:largesiteswithalargeinter- the URI has been extracted thus far; we deem a
nalbranchingfactor(largenumbersofuniqueintra- higher count to roughly equate to a higher priority
PLDoutlinksperdocument)canresultinthefron- URI.
tier of each round being dominated by URIs from The crawler utilises both the on-disk index and
asmallselectionofPLDs.Thus,na¨ıvebreadth-first the in-memory queue to offer similar functionality
crawlingcanleadtocrawlershammeringsuchsites; as above. The on-disk index and in-memory queue
conversely, given a politeness policy, a crawler may aresynchronisedatthestartofeachround:
spendalotoftimeidlewaitingforthemin-delayto (i) links and respective inlink counts extracted
pass. from the previous round (or seed URIs if the
Onesolutionistoreasonablyrestrictthebranch- firstround)areaddedtotheon-diskindex;
ing factor [88] – the maximum number of URIs (ii) URIs polled from the previous round have
crawled per PLD per round – which ensures that theirstatusupdatedon-disk;
individualPLDswithlargeinternalfan-outarenot (iii) an in-memory PLD queue is filled using an
hammered;thus,ineachroundofthecrawl,weim- iteratorofon-diskURIssortedbydescending
plementacut-offforURIsperPLD,givenby Pld- inlinkcount.
LimitinAlgorithm1. Mostimportantly,theaboveprocessensuresthat
Secondly, to ensure the maximum gap between onlytheURIsactive(currentPLDqueueandfron-
crawlingsuccessiveURIsforthesamePLD,weim- tier URIs) for the current round must be stored in
plement a per-PLD queue (given by pld in Al- memory. Also, the process ensures that the on-disk
0...n
gorithm 1) whereby each PLD is given a dedicated indexstoresthepersistentstateofthecrawlerupto
queue of URIs filled from the frontier, and during the start of the last round; if the crawler unexpect-
the crawl, a URI is polled from each PLD queue in edlydies,thecrawlcanberesumedfromthestartof
around-robinfashion.IfallofthePLDqueueshave the last round. Finally, the in-memory PLD queue
been polled before the min-delay is satisfied, then isfilledwithURIssortedinorderofinlinkcount,of-
the crawler must wait: this is given by Lines 31–34 feringacheapformofintra-PLDURIprioritisation
inAlgorithm1.Thus,theminimumcrawltimefora (Algorithm1,Line8).
round–assumingasufficientlyfullqueue–becomes
Min-Delay *Pld-Limit.
5.1.3. Multi-threading
Thebottle-neckforasingle-threadedcrawlerwill
5.1.2. On-diskQueue be the response times of remote servers; the CPU
As the crawl continues, the in-memory capacity load, I/O throughput and network bandwidth of a
of the machine will eventually be exceeded by the crawling machine will not be efficiently exploited
capacity required for storing URIs [88]. Perform- by sequential HTTP GET requests over the Web.
ing a stress-test, we observed that with 2GB of Thus,crawlersarecommonlymulti-threadedtomit-
JAVA heap-space, the crawler could crawl approx. igatethisbottleneckandperformconcurrentHTTP
199kURIs(additionallystoringtherespectivefron- lookups. At a certain point of increasing the num-
tier URIs) before throwing an out-of-memory ex- ber of lookup threads operating, the CPU load,
ception.Inordertoscalebeyondtheimpliedmain- I/O load, or network bandwidth becomes an im-
memory limitations of the crawler, we implement mutablebottleneck;thisbecomestheoptimalnum-
on-disk storage for URIs, with the additional ben- berofthreads.
10