classifiers against adversarial examples

38 Pages·2017·3.85 MB·English

Workshoptrack-ICLR2017
A THEORETICAL FRAMEWORK FOR ROBUSTNESS OF (DEEP)
CLASSIFIERS AGAINST ADVERSARIAL EXAMPLES
BeilunWang,JiGao,YanjunQi
DepartmentofComputerScience
UniversityofVirginia
Charlottesville,VA22901,USA
{bw4mw,jg6yd,yanjun}@virginia.edu
ABSTRACT
Mostmachinelearningclassifiers,includingdeepneuralnetworks,arevulnerable
to adversarial examples. Such inputs are typically generated by adding small
butpurposefulmodificationsthatleadtoincorrectoutputswhileimperceptibleto
human eyes. The goal of this paper is not to introduce a single method, but to
maketheoreticalstepstowardsfullyunderstandingadversarialexamples. Byusing
conceptsfromtopology,ourtheoreticalanalysisbringsforththekeyreasonswhy
anadversarialexamplecanfoolaclassifier(f )andaddsitsoracle(f ,likehuman
1 2
eyes)insuchanalysis. Byinvestigatingthetopologicalrelationshipbetweentwo
(pseudo)metricspacescorrespondingtopredictorf andoraclef , wedevelop
1 2
necessaryandsufficientconditionsthatcandetermineiff isalwaysrobust(strong-
1
robust)againstadversarialexamplesaccordingtof . Interestinglyourtheorems
2
indicatethatjustoneunnecessaryfeaturecanmakef notstrong-robust,andthe
1
rightfeaturerepresentationlearningisthekeytogettingaclassifierthatisboth
accurateandstrong-robust.
1 INTRODUCTION
DeepNeuralNetworks(DNNs)canefficientlylearnhighlyaccuratemodelsandhavebeendemon-
stratedtoperformexceptionallywell(Krizhevskyetal.,2012;Hannunetal.,2014). However,recent
studiesshowthatintelligentattackerscanforcemanymachinelearningmodels,includingDNNs,to
misclassifyexamplesbyaddingsmallandhardlyvisiblemodificationsonaregulartestsample.
Themaliciouslygeneratedinputsarecalled“adversarialexamples”(Goodfellowetal.,2014;Szegedy
et al., 2013) and are commonly crafted by carefully searching small perturbations through an
optimizationprocedure. Severalrecentstudiesproposedalgorithmsforsolvingsuchoptimizationto
foolDNNclassifiers. (Szegedyetal.,2013)firstlyobservethatconvolutionDNNsarevulnerable
to small artificial perturbations. They use box-constrained Limited-memory BFGS (L-BFGS) to
createadversarialexamplesandfindthatadversarialperturbationsgeneratedfromoneDNNnetwork
can also force other networks to produce wrong outputs. Then, (Goodfellow et al., 2014) try to
clarifythattheprimarycauseofsuchvulnerabilitiesmaybethelinearnatureofDNNs. Theythen
proposethefastgradientsignmethodforgeneratingadversarialexamplesquickly. Subsequentpapers
(Fawzietal.,2015;Papernotetal.,2015a;Nguyenetal.,2015)haveexploredotherwaystoexplore
adversarial examples for DNN (details in Section 2.1). The goal of this paper is to analyze the
robustnessofmachinelearningmodelsinthefaceofadversarialexamples.
Inresponsetoprogressingeneratingadversarialexamples,researchersattempttodesignstrategiesfor
makingmachine-learningsystemsrobusttovariousnoise,intheworstcaseasadversarialexamples.
Forinstance,denoisingNNarchitectures(Vincentetal.,2008;Gu&Rigazio,2014;Jinetal.,2015)
candiscovermorerobustfeaturesbyusinganoise-corruptedversionofinputsastrainingsamples
1. Amodifieddistillationstrategy(Papernotetal.,2015b)isproposedtoimprovetherobustnessof
DNNsagainstadversarialexamples,thoughithasbeenshowntobeunsuccessfulrecently(Carlini&
Wagner,2016a). Themostgenerallysuccessfulstrategytodateisadversarialtraining(Goodfellow
1Though(Gu&Rigazio,2014)pointedoutsuchastrategyisvulnerabletoattackerswhoconsiderthemodel
asastackeddenoiserplusaclassifierandcomputethegradientsoftheoverallarchitecture
1
Workshoptrack-ICLR2017
Table1: Alistofimportantnotationsusedinthepaper.
f Alearnedmachinelearningclassifierf =c ◦g .
1 1 1 1
f Theoracleforthesametask(seeDefinition(2.1))f =c ◦g .
2 2 2 2
g Partoff includingoperationsthatprogressivelytransforminputintoanew
i i
formoflearnedrepresentationsinX .
i
c Partoff includingsimpledecisionfunctions(likelinear)forclassifying.
i i
X Inputspace(e.g.,{0,1,2,...,255}32×32×3forCIFAR-10data(Krizhevsky
&Hinton,2009)).
Y Outputspace(e.g.,{1,2,3,...,10}forCIFAR-10data(Krizhevsky&Hinton,
2009)).
X Featurespacedefinedbythefeatureextractionmoduleg ofpredictorf .
1 1 1
X Featurespacedefinedbythefeatureextractionmoduleg oforaclef .
2 2 2
d (·,·) ThemetricfunctionformeasuringsampledistancesinfeaturespaceX with
1 1
respecttopredictorf .
1
d (·,·) ThemetricfunctionformeasuringsampledistanceinfeaturespaceX with
2 2
respecttooraclef .
2
d(cid:48)(·,·) The Pseudometric function with respect to predictor f , d(cid:48)(x,x(cid:48)) =
1 1 1
d (g (x),g (x(cid:48))).
1 1 1
d(cid:48)(·,·) The Pseudometric function with respect to oracle f , d(cid:48)(x,x(cid:48)) =
2 2 2
d (g (x),g (x(cid:48))).
2 2 2
a.e. almosteverywhere(Folland,2013);(definedbyDefinition(8.2)in Section8.1)
(cid:15),δ ,δ ,δ,η smallpositiveconstants
1 2
etal.,2014;Szegedyetal.,2013)whichinjectsadversarialexamplesintotrainingtoimprovethe
generalizationofDNNmodels2. Morerecenttechniquesincorporateasmoothnesspenalty(Miyato
etal.,2016;Zhengetal.,2016)oralayer-wisepenalty(Carlini&Wagner,2016b)asaregularization
terminthelossfunctiontopromotethesmoothnessoftheDNNmodeldistributions.
Recentstudies(reviewedby(Papernotetal.,2016b))aremostlyempiricalandprovidelittleunder-
standingofwhyanadversarycanfoolmachinelearningmodelswithadversarialexamples. Several
importantquestionshavenotbeenansweredyet:
• Whatmakesaclassifieralwaysrobusttoadversarialexamples?
• Whichpartsofaclassifierinfluenceitsrobustnessagainstadversarialexamplesmore,compared
withtherest?
• Whatistherelationshipbetweenaclassifier’sgeneralizationaccuracyanditsrobustnessagainst
adversarialexamples?
• Why(many)DNNclassifiersarenotrobustagainstadversarialexamples? Howtoimprove?
Thispapertriestoanswerabovequestionsandmakesthefollowingcontributions:
• Section2pointsoutthatpreviousdefinitionsofadversarialexamplesforaclassifier(f )have
1
overlookedtheimportanceofanoraclefunction(f )ofthesametask.
2
• Section4formallydefineswhenaclassifierf isalwaysrobust("strong-robust")againstadversarial
1
examples. Itprovesfourtheoremsaboutsufficientandnecessaryconditionsthatmakef always
1
robustagainstadversarialexamplesaccordingtof . Ourtheoremsleadtoanumberofinteresting
2
insightsinSection5,likethatthefeaturerepresentationlearningcontrolsifaDNNisstrong-robust
ornot.
• Section6isdedicatedtoprovidepracticalandtheoreticallygroundeddirectionsforunderstanding
andhardeningDNNmodelsagainstadversarialexamples.
Table1providesalistofimportantnotationsweuseinthepaper.
2Though(Papernotetal.,2016b)pointedoutthatadversarialtrainingisvulnerabletoblack-boxattacksusing
modeltransferability
2
Workshoptrack-ICLR2017
Figure1: Exampleofamachine-learningclassifier(predictor)andahumanannotator(oracle)for
classifyingimagesofhand-written“0”. Bothincludetwosteps: featureextractionandclassification.
Theupperhalfisaboutthelearnedmachineclassifierf andthelowerhalfisabouttheoraclef . f
1 2 1
transformssamplesfromtheoriginalspaceX toanembeddedmetricspace(X ,d )usingitsfeature
1 1
extractionstep. Here,d isthesimilaritymeasureonthefeaturespaceX . Classificationmodels
1 1
likeDNNcoverthefeatureextractionstepintheirmodel,thoughmanyothermodelslikedecision
treeneedhard-craftedordomain-specificfeatureextraction. Thenf canusealinearfunctionto
1
decidetheclassificationpredictiony ∈Y. Similarly,humanoraclef transformsdatasamplesfrom
(cid:98) 2
theoriginalspaceX intoanembeddedmetricspace(X ,d )byitsfeatureextraction. Here,d is
2 2 2
thecorrespondingsimilaritymeasure. Thentheoraclegettheclassificationresulty ∈Y usingthe
featurerepresentationofsamples(X ,d ).
2 2
2 RE-DEFINE ADVERSARIAL EXAMPLES
Thissectionprovidesageneraldefinitionofadversarialexamples,byincludingthenotionofan
oracle. Foraparticularclassificationtask,alearnedclassifierisrepresentedasf :X →Y,where
1
X representstheinputsamplespaceandY istheoutputspacerepresentingacategoricalset.
2.1 PREVIOUSFORMULATIONS
Various definitions of “adversarial examples” exist in the recent literature, with most following
Eq.(2.1). SeemoredetailedreviewsinSection9. Thebasicideaistogenerateamisclassifiedsample
x(cid:48)by“slightly”perturbingacorrectlyclassifiedsamplex,withanadversarialperturbation∆(x,x(cid:48)).
Formally,whengivenx∈X
Findx(cid:48)
s.t. f (x)(cid:54)=f (x(cid:48)) (2.1)
1 1
∆(x,x(cid:48))<(cid:15)
Herex,x(cid:48) ∈X. ∆(x,x(cid:48))representsthedifferencebetweenxandx(cid:48),whichdependsonthespecific
datatypethatxandx(cid:48)belongto3. Table2summarizesdifferentchoicesoff and∆(x,x(cid:48))usedin
1
therecentliterature,inwhichnormfunctionsontheoriginalspaceX aremostlyusedtocalculate
3Forexample,inthecaseofstrings,∆(x,x(cid:48))representsthedifferencebetweentwostrings.
3
Workshoptrack-ICLR2017
∆(x,x(cid:48)). MultiplealgorithmshavebeenimplementedtosolveEq.(2.1)asaconstrainedoptimization
(summarized by the last column of Table 2). More details are included for three such studies in
Section3.2.
Whensearchingforadversarialexamples, oneimportantpropertyhasnotbeenfullycapturedby
Eq.(2.1). Thatis,anadversarialexamplehasbeenmodifiedveryslightlyfromitsseedandthese
modificationscanbesosubtlethat,forexampleinimageclassification,ahumanobserverdoesnot
evennoticethemodificationatall. Wedefinetheroleof“humanobserver”moreformallyasfollows:
Definition2.1. An“Oracle”representsadecisionprocessgeneratinggroundtruthlabelsforatask
ofinterest. Eachoracleistask-specific,withfiniteknowledgeandnoise-free4.
The goal of machine learning is to train a learning-based predictor function f : X → Y to
1
approximateanoracleclassifierf :X →Y forthesameclassificationtask. Forexample,inimage
2
classificationtasks,theoraclef isoftenagroupofhumanannotators. Addingthenotationoforacle,
2
wereviseEq.(2.1)into:
Findx(cid:48)
s.t. f (x)(cid:54)=f (x(cid:48))
1 1
(2.2)
∆ (x,x(cid:48))<(cid:15)
2
f (x)=f (x(cid:48))
2 2
2.2 MEASURINGSAMPLEDIFFERENCEINWHICHSPACE? MODELING&DECOMPOSINGf2
∆ (x,x(cid:48)) < (cid:15)reflectsthatadversarialexamplesadd“smallmodifications”thatarealmostimper-
2
ceptibletooracleofthetask. Clearlycalculating∆ (x,x(cid:48))needstoaccordtooraclef . Formost
2 2
classification tasks, an oracle does not measure the sample difference in the original input space
X. Wewanttoemphasizethatsampledifferenceiswithregardstoitsclassificationpurpose. For
instance,whenlabelingimagesforthehand-writtendigitalrecognition,humanannotatorsdonot
needtoconsiderthosebackgroundpixelstodecideifanimageis“0”ornot.
Illustrated in Figure 1, we denote the feature space an oracle uses to consider difference among
samples for the purpose of classification decision as X . The sample difference uses a distance
2
functiond inthisspace. Anoraclefunctionf :X →Y canbedecomposedasf =c ◦g where
2 2 2 2 2
g : X → X represents the operations for feature extraction from X to X and c : X → Y
2 2 2 2 2
denotes the simple operation of classification in X . Essentially g includes the operations that
2 2
(progressively)transforminputrepresentationsintoaninformativeformofrepresentationsX . c
2 2
appliesrelativelysimplefunctions(likelinear)inX forthepurposeofclassification. d isthemetric
2 2
function(detailsinSection4)anoracleusestomeasurethesimilarityamongsamples(byrelyingon
representationslearnedinthespaceX ). WeillustratethemodelinganddecompositioninFigure1.
2
In Section4ourtheoreticalanalysisuses(X ,d )tobringforththefundamentalcausesofadversarial
2 2
examplesandleadstoasetofnovelinsightstounderstandsuchexamples. Tothebestoftheauthors’
knowledge,thetheoreticalanalysismadebythispaperhasnotbeenuncoveredbytheliterature.
ModelingOraclef : Onemayarguethatitishardtomodelf and(X ,d )forrealapplications,
2 2 2 2
since if such oracles can be easily modeled machine-learning based f seems not necessary. In
1
Section 9.1, we provide examples of modeling oracles for real applications. For many security-
sensitiveapplicationsaboutmachines,oraclesf doexist5. Forartificialintelligencetaskslikeimage
2
classification,humansaref . Asillustratedbycognitiveneurosciencepapers(DiCarlo&Cox,2007;
2
DiCarloetal.,2012),humanbrainsperformvisualobjectrecognitionusingtheventralvisualstream,
andthisstreamisconsideredtobeaprogressiveseriesofvisualre-representations,fromV1toV2
toV4toITcortex (DiCarlo&Cox,2007). Experimentalresultssupportthathumanvisualsystem
makesclassificationdecisionatthefinalITcortexlayer. Thisprocessiscapturedexactlybyour
decompositionf =c ◦g .
2 2 2
4Weleavealldetailedanalysisofwhenanoraclecontainsnoiseasfuturework.
5Oraclesf doexistinmanysecurity-sensitiveapplicationsaboutmachines.Butmachine-learningclassifiers
2
f areusedpopularlyduetospeedorefficiency
1
4
Workshoptrack-ICLR2017
Table2: Summaryofthepreviousstudiesdefiningadversarialexamples.
Previousstudies f ∆(x,x(cid:48)) Formulationoff (x)(cid:54)=f (x(cid:48))
1 1 1
(Goodfellowetal.,2014) Convolutionalneuralnetworks (cid:96) argmaxLoss(f (x(cid:48)),f (x))
∞ 1 1
x(cid:48)
(Szegedyetal.,2013) Convolutionalneuralnetworks (cid:96) argminLoss(f (x(cid:48)),l),subjectto:l(cid:54)=f (x(cid:48))
2 1 1
x(cid:48)
(Biggioetal.,2013) Supportvectormachine(SVM) (cid:96) argminLoss(f (x(cid:48)),−1),subjectto:f (x)=1
2 1 1
x(cid:48)
(Kantchelianetal.,2015) DecisiontreeandRandomforest (cid:96) , (cid:96) , argminLoss(f (x(cid:48)),−1),subjectto:f (x)=1
2 1 1 1
(cid:96) x(cid:48)
∞
(Papernotetal.,2016a) Convolutionalneuralnetworks (cid:96) argmaxLoss(f (x(cid:48)),f (x))
0 1 1
x(cid:48)
(Grosseetal.,2016) Convolutionalneuralnetworks (cid:96) argmaxLoss(f (x(cid:48)),f (x))
0 1 1
x(cid:48)
(Xuetal.,2016) RandomforestandSVM (cid:96) ,(cid:96) argminLoss(f (x(cid:48)),−1),subjectto:f (x)=1
1 ∞ 1 1
x(cid:48)
2.3 REVISEDFORMULATION
Now we use the decomposition of f to rewrite ∆ (x,x(cid:48)) as d (g (x),g (x(cid:48))) in Eq. (2.2) and
2 2 2 2 2
obtainourproposedgeneraldefinitionofadversarialexamples:
Definition2.2. adversarialexample: Supposewehavetwofunctionsf andf . f :X →Y isthe
1 2 1
classificationfunctionlearnedfromatrainingsetandf :X →Y istheclassificationfunctionofthe
2
oraclethatgeneratesground-truthlabelsforthesametask. Givenasamplex∈X,anadversarial
examplex(cid:48) ∈X. (x,x(cid:48))satisfiesEq.(2.3).
Findx(cid:48)
s.t. f (x)(cid:54)=f (x(cid:48))
1 1
(2.3)
d (g (x),g (x(cid:48)))<δ
2 2 2 2
f (x)=f (x(cid:48))
2 2
Mostpreviousstudies(Table2)havemadeanimportantandimplicitassumptionaboutf (through
2
using∆(x,x(cid:48)) < (cid:15)): f isalmosteverywhere(a.e.) continuous. Weexplainsthea.e. continuity
2
assumptionanditsindicationinSection8. Basically,whenf isassumedcontinuousa.e.,
2
P(f (x)=f (x(cid:48))|d (g (x),g (x(cid:48)))<δ )=1
2 2 2 2 2 2
Therefore,whenf iscontinuousa.e. Eq.(2.3)canbesimplifiedintothefollowingEq.(2.4).
2
Findx(cid:48)
s.t. f (x)(cid:54)=f (x(cid:48)) (2.4)
1 1
d (g (x),g (x(cid:48)))<δ
2 2 2 2
3 BACKGROUND: ADVERSARIAL EXAMPLES
3.1 BACKGROUND: PREVIOUSDEFINITIONSOFADVERSARIALEXAMPLES
For the purpose of "fooling" a classifier, naturally, the attacker wants to control the size of the
perturbation∆(x,x(cid:48))toensuretheperturbedsamplex(cid:48)stillstayscloseenoughtotheoriginalsample
xtosatisfytheintended"fooling"purpose. Forexample,intheimageclassificationcase,Eq.(2.1)
canusethegradientinformationtofinda∆(x,x(cid:48))thatmakeshumanannotatorsstillrecognizex(cid:48)as
almostthesameasx,thoughtheclassifierwillpredictx(cid:48)intoadifferentclass. Inanotherexample
withmoreobvioussecurityimplicationsaboutPDFmalware(Xuetal.,2016),x(cid:48)in Eq.(2.1)isfound
bygeneticprogramming. AmodifiedPDFfilefromamaliciousPDFseedwillstillberecognizedas
maliciousbyanoraclemachine(i.e.,avirtualmachinedecidesifaPDFfileismaliciousornotby
actuallyrunningit),butareclassifiedasbenignbystate-of-artmachinelearningclassifiers(Xuetal.,
2016).
5
Workshoptrack-ICLR2017
Thefollowing Eq.(3.1)hasbeenpopularaswell. Eq.(3.1)isaspecialcaseof Eq.(2.1).
argmin∆(x,x(cid:48))
x(cid:48)∈X (3.1)
Subjectto: f (x)(cid:54)=f (x(cid:48))
1 1
Eq.(3.1)triestofindthex(cid:48) byminimizing∆(x,x(cid:48))undersomeconstraints. Eq.(2.1)isamore
generalformulationthan Eq.(3.1)andcansummarizemostrelevantstudies. Forexample,in(Xu
etal.,2016)"adversarialexamples"arethosegeneratedPDFsthatcanfoolPDFRate(alearning-based
classifierfordetectingmaliciousPDFs)toclassifythemasbenign. Thedistancesofthesevariant
PDFstotheseedPDFarenotnecessarilyminimal. Forsuchcases, Eq.(2.1)stillfits,while Eq.(3.1)
doesnot.
Besides, in the field of computer security, machine learning has been popular in classifying the
malicious (y = 1) behavior versus benign behavior (y = −1). For such a context, two different
definitionsofadversarialexamplesexistintheliterature:
Forinstance,(Biggioetal.,2013)usesaformulaasfollows:
argmin(f (x(cid:48)))
1
x(cid:48)
s.t. ∆(x,x(cid:48))<d (3.2)
max
f (x)>0
1
Differently,(Lowd&Meek,2005)usesthefollowingformula:
argmin(∆(x,x(cid:48)))
x(cid:48)
s.t. f (x(cid:48))<0 (3.3)
1
f (x)>0
1
Hered isasmallpositiveconstant. Thesedefinitionsof“adversarialexamples”arespecialcases
max
of Eq.(3.1)and Eq.(2.1).
3.2 BACKGROUND: PREVIOUSALGORITHMSGENERATING“ADVERSARIALEXAMPLES”
Tofoolclassifiersattesttime,severalapproacheshavebeenimplementedtogenerate“adversarial
perturbations”bysolvingEq.(2.2). AccordingtoEq.(2.2),anadversarialexampleshouldbeable
tochangetheclassificationresultf (x),whichisadiscretevalue. TosolveEq.(2.2),weneedto
1
transformtheconstraintf (x)(cid:54)=f (x(cid:48))intoanoptimizableformulation. Thenwecaneasilyusethe
1 1
LagrangianmultipliertosolveEq.(2.2). AllthepreviousstudiesdefinealossfunctionLoss(·,·)to
quantifytheconstraintf (x)(cid:54)=f (x(cid:48)). Thislossfunctioncanbethesamewiththetrainingloss,or
1 1
itcanbechosendifferently,suchashingelossorcrossentropyloss.
Wesummarizefourcommonattackingstudiesasfollows:
Gradientascentmethod(Biggioetal.,2013) Machinelearninghasbeenpopularinclassifying
malicious(y =1)versusbenign(y =−1)incomputersecuritytasks. Forsuchcontexts,asimple
way to solve Eq. (2.2) is through gradient ascent. To minimize the size of the perturbation and
maximize the adversarial effect, the perturbation should follow the gradient direction (i.e., the
directionprovidingthelargestincreaseoffunctionvalue,herefromy = −1to1). Therefore,the
perturbationrineachiterationiscalculatedas:
r =(cid:15)∇ Loss(f (x+r),−1)Subjectto: f (x)=1 (3.4)
x 1 1
Byvarying(cid:15),thismethodcanfindasamplex(cid:48)withregardtod (x,x(cid:48))suchthatf (x)(cid:54)=f (x(cid:48)).
2 1 1
Box L-BFGS adversary (Szegedy et al., 2013) This study views the adversarial problem as a
constrainedoptimizationproblem,i.e.,findaminimumperturbationintherestrictedsamplespace.
TheperturbationisobtainedbyusingBox-constrainedL-BFGStosolvethefollowingequation:
argmin(c×d (x,x+r)+Loss(f (x+r),l)),x+r ∈[0,1]p (3.5)
2 1
r
Herepisthetotalnumberoffeatures,cisatermaddedfortheLagrangemultiplier. (foranimage
classificationtask,itis3timesthetotalnumberofpixelsofanRGBimage)lisatargetlabel,which
isdifferentfromtheoriginallabel. Theconstraintx+r ∈[0,1]pmeansthattheadversarialexample
isstillintherangeofsamplespace.
6
Workshoptrack-ICLR2017
Fast gradient sign method (Goodfellow et al., 2014) The fast gradient sign method proposed
by(Goodfellowetal.,2014)viewsd asthe(cid:96) -norm. Inthiscase,anaturalchoiceistomakethe
2 ∞
attackstrengthateveryfeaturedimensionthesame. Theperturbationisobtainedbysolvingthe
followingequation:
argmin(c×d (x,x+r)−Loss(f (x+r),f (x))),x+r ∈[0,1]p (3.6)
2 1 1
r
Thereforetheperturbationcanbecalculateddirectlyby:
r =(cid:15)sign(∇ Loss(f (z),f (x))) (3.7)
z 1 1
Herethelossfunctionisthefunctionusedtotraintheneuralnetwork. Arecentpaper(Kurakinetal.,
2016)showsthatadversarialexamplesgeneratedbyfastgradientsignmethodaremisclassifiedeven
aftertheseimageshavebeenrecapturedbycameras.
Jacobian-based saliency map approach (Papernot et al., 2015a) (Papernot et al., 2015a) pro-
posedtheJacobian-basedsaliencymapapproach(JSMA)tosearchforadversarialsampleswhile
limiting the number of pixel to modify in the image. As a targeted attack, JSMA iteratively per-
turbspixelsinaninputthathavelargeadversarialsaliencyscores. Theadversarialsaliencymap
iscalculatedfromtheJacobian(gradient)matrix∇ f (x)oftheDNNmodelatthecurrentinput
x 1
x. The (i,j)th component in Jacobian matrix ∇ f (x) describes the derivative of output class j
x 1
withrespecttofeaturepixeli. Foreachpixeli,itsadversarialsaliencyscoreiscalculatedtoreflect
howthispixelwillincreasetheoutputscoreofclassj versuschangingthescoreofotherpossible
outputclasses. Theprocessisrepeateduntilmisclassificationinthetargetclassisachievedorthe
maximumnumberofperturbedpixelshasbeenreached. Essentially,JSMAoptimizesEquation2.1
bymeasuringperturbation∆(x,x(cid:48))throughthe(cid:96) -norm.
0
3.3 BACKGROUND: RELATEDWORKSINABROADERCONTEXT
Investigatingthebehaviorofmachinelearningsystemsinadversarialenvironmentsisanemerging
topic (Huang et al., 2011; Barreno et al., 2006; 2010; Globerson & Roweis, 2006; Biggio et al.,
2013; Kantchelian et al., 2015; Zhang et al., 2015). Recent studies can be roughly categorized
intothreetypes: (1)Poisoningattacksinwhichspeciallycraftedattackpointsareinjectedintothe
trainingdata. Multiplerecentpapers(Alfeldetal.,2016;Mei&Zhu,2015b;Biggioetal.,2014;
2012;Mei&Zhu,2015a)haveconsideredtheproblemofanadversarybeingabletopollutethe
trainingdatawiththegoalofinfluencinglearningsystemsincludingsupportvectormachines(SVM),
autoregressivemodelsandtopicmodels. (2)Evasionattacksareattacksinwhichtheadversary’s
goalistocreateinputsthataremisclassifiedbyadeployedtargetclassifier. Relatedstudies(Szegedy
etal.,2013;Goodfellowetal.,2014;Xuetal.,2016;Kantchelianetal.,2015;Rndic&Laskov,2014;
Biggioetal.,2013;Papernotetal.,2016b;Sinhaetal.,2016)assumetheadversarydoesnothave
anopportunitytoinfluencethetrainingdata, butinsteadfinds“adversarialexamples”toevadea
trainedclassifierlikeDNN,SVMorrandomforest. (3)Privacy-awaremachinelearning(Duchietal.,
2014)isanotherimportantcategoryrelevanttodatasecurityinmachinelearningsystems. Recent
studieshaveproposedvariousstrategies(Xieetal.,2014;Bojarskietal.,2014;Stoddardetal.,2014;
Li & Zhou, 2015; Rajkumar & Agarwal, 2012; Dwork, 2011; Nock et al., 2015) to preserve the
privacyofdatasuchasdifferentialprivacy. Thispaperfocusesonevasionattacksthataremostly
usedtoattackingclassifiersthattrytodistinguishmaliciousbehaviorsfrombenignbehaviors. Here
weextendittoabroadermeaning–adversarialmanipulationoftestsamples. Evasionattacksmaybe
encounteredduringsystemdeploymentofmachinelearningmethodsinadversarialsettings.
Inthebroadersecuremachinelearningfield,researchersalsomakeattemptsforhardeninglearning
systems. Forinstance: (1) (Barrenoetal.,2010)and (Biggioetal.,2008)proposeamethodto
introducesomerandomnessintheselectionofclassificationboundaries;(2)Afewrecentstudies
(Xiaoetal.,2015;Zhangetal.,2015)considertheimpactofusingreducedfeaturesetsonclassifiers
underadversarialattacks.(Xiaoetal.,2015)proposesanadversary-awarefeatureselectionmodelthat
canimproveaclassifier’srobustnessagainstadversarialattacksbyincorporatingspecificassumptions
abouttheadversary’sdatamanipulationstrategy. (3)Anotherlineofworks,namedasadversarial
training(Goodfellowetal.,2014),designsanewlossfunctionfortrainingneuralnetworks,whichisa
linearinterpolationofthelossfunctionoftheoriginalsampleandthelossfunctionoftheadversarial
examplegeneratedbytheoriginalsample. Ascalableversionofadversarialtraining(Kurakinetal.,
2016)wasrecentlyproposed. Byapplyingseveraltricks,theauthorcanapplytheadversarialtraining
todeepernetworktrainedbytheimagenetdataset. (4)Multiplestudiesmodeladversarialscenarios
withformalframeworksrepresentingtheinteractionbetweentheclassifierandtheadversary. Related
7
Workshoptrack-ICLR2017
effortsincludeperfectinformationassumptions(Dalvietal.,2004),assumingapolynomialnumberof
membershipqueries (Lowd&Meek,2005),formalizingtheattackprocessasatwo-personsequential
Stackelberggame(Brückner&Scheffer,2011;Liu&Chawla,2010),amin-maxstrategy(training
aclassifierwithbestperformanceundertheworstperturbation)(Dekeletal.,2010;Globerson&
Roweis, 2006), exploring online and non-stationary learning (Dahlhaus, 1997; Cesa-Bianchi &
Lugosi,2006),andformalizingasanadversarialreinforcementlearningproblem(Uther&Veloso,
1997). (5)APACmodelstudyaboutlearningadversarybehaviorinasecuritygamesalsoinvestigated
thesolutionofcomputingthebestdefenderstrategyagainstthelearnedadversarybehavior. Ithasa
similarconclusionasours(Section4)thattheextremecasesthatthedefenderdoesn’tworkonlyhas
zeroprobability(Sinhaetal.,2016).
4 DEFINE STRONG-ROBUSTNESS
Withamoreaccuratedefinitionof“adversarialexamples”,nowweaimtoanswerthefirstcentral
question: “What makes a classifier always robust against adversarial examples?”. Section 4.2
definestheconcept“strong-robust”describingaclassifieralwaysrobustagainstadversarialexamples.
Section 4.3 and Section 4.4 present sufficient and necessary conditions for “strong-robustness”.
Section5thenprovidesasetoftheoreticalinsightstounderstand“strong-robustness”.
4.1 MODELINGANDDECOMPOSINGf1
AsshowninFigure1, wedecomposef inasimilarwayasthedecompositionoff . Thisisto
1 2
answeranotherkeyquestion: “whichpartsofalearnedclassifierinfluenceitsrobustnessagainst
adversarialexamplesmore,comparedwiththerest?”. Amachine-learningclassifierf =c ◦g ,
1 1 1
whereg :X →X representsthefeatureextractionoperationsandc :X →Y performsasimple
1 1 1 1
operation(e.g., linear)ofclassification. Section9.2providesmultipleexamplesofdecomposing
state-of-the-artf 6. d denotesthedistancefunctionf usestomeasuredifferenceamongsamples
1 1 1
inX .
1
Almostallpopularmachinelearningclassifierssatisfythea.e. continuityassumption. Itmeans:
P(f (x)=f (x(cid:48))|d (g (x),g (x(cid:48)))<δ )=1
1 1 1 1 1 1
When f is not continuous a.e., it is not robust to any types of noise. See Section 8 for detailed
1
discussions.
For the rarecases that f is notcontinuous a.e., Section13 discusses "boundarypoints" of f 7.
1 1
Roughlyspeaking,whenf isnotcontinuousa.e.,8
1
P(f (x)(cid:54)=f (x(cid:48))|d (g (x),g (x(cid:48)))<δ )>0
1 1 1 1 1 1
Thereforethefollowingprobabilityof“boundarypointsbasedadversarialexamples”mightnotbe0
forsuchcases9:
P(f (x)(cid:54)=f (x(cid:48))|f (x)=f (x(cid:48)),
1 1 2 2
(4.1)
d (g (x),g (x(cid:48)))<δ ,d (g (x),g (x(cid:48)))<δ )
1 1 1 1 2 2 2 2
ThevalueofthisprobabilityiscriticalforouranalysisinTheorem(4.3)andinTheorem(4.5).
4.2 {δ2,η}-STRONG-ROBUSTAGAINSTADVERSARIALEXAMPLES
Wethenapplyreverse-thinkingonDefinition(2.2)andderivethefollowingdefinitionof strong-
robustnessforamachinelearningclassifieragainstadversarialexamples:
Definition 4.1. {δ ,η}-Strong-robustness of a machine-learning classifier: A machine-learning
2
classifier f (·) is {δ ,η}-strong-robust against adversarial examples if: ∀x,x(cid:48) ∈ X a.e., (x,x(cid:48))
1 2
satisfiesEq.(4.2).
∀x,x(cid:48) ∈X
P(f (x)=f (x(cid:48))|f (x)=f (x(cid:48)), (4.2)
1 1 2 2
d (g (x),g (x(cid:48)))<δ )>1−η
2 2 2 2
6Noticethatg mayalsoincludeimplicitfeatureselectionstepslike(cid:96) regularization.
1 1
7Boundarypointsarethosepointssatisfyingf (x)(cid:54)=f (x(cid:48))andd (g (x),g (x(cid:48)))<δ )
1 1 1 1 1 1
8Whenf iscontinuousa.e.,P(f (x)(cid:54)=f (x(cid:48))|d (g (x),g (x(cid:48)))<δ )=0.
1 1 1 1 1 1 1
9“Boundarypointsbasedadversarialexamples”onlyattackseedsampleswhoareboundarypointsoff .
1
8
Workshoptrack-ICLR2017
Whenf iscontinuousa.e.,Eq.(4.2)simplifiesintoEq.(4.3):
2
∀x,x(cid:48) ∈X, P(f (x)=f (x(cid:48))|
1 1
(4.3)
d (g (x),g (x(cid:48)))<δ )>1−η
2 2 2 2
Eq.(4.2)definesthe“{δ ,η}-strong-robustness”asaclaimwiththehighprobability. Tosimplify
2
notations,intherestofthispaper,weuse“strong-robust”representing“{δ ,η}-strong-robust”. Also
2
intherestofthispaperweproposeandprovetheoremsandcorollariesbyusingitsmoregeneral
formbyEq.(4.2). Forallcases,iff iscontinuousa.e.,allproofsandequationscanbesimplified
2
byusingonlythetermd (g (x),g (x(cid:48)))<δ (i.e. removingthetermf (x)=f (x(cid:48)))accordingto
2 2 2 2 2 2
Eq.(4.3)).
The“strong-robustness”definitionleadstofourimportanttheoremsinnexttwosubsections.
4.3 TOPOLOGICALEQUIVALENCEOFTWOMETRICSPACES(X1,d1)AND(X2,d2)IS
SUFFICIENTINDETERMININGSTRONG-ROBUSTNESS
Intheappendix,Section10.1brieflyintroducestheconceptofmetricspaceandthedefinitionof
topologicalequivalenceamongtwometricspaces. AsshowninFigure1,heref definesametric
1
space(X ,d )onX withthemetricfunctiond . Similarlyf definesametricspace(X ,d )on
1 1 1 1 2 2 2
X withthemetricfunctiond .
2 2
Ifthetopologicalequivalence(Eq.(10.1))existsbetween(X ,d )and(X ,d ),itmeansthatfor
1 1 2 2
allpairofsamplesfromX,wehavethefollowingrelationship:
∀x,x(cid:48) ∈X,
(4.4)
d (g (x),g (x(cid:48)))<δ ⇔d (g (x),g (x(cid:48)))<δ
1 1 1 1 2 2 2 2
When f is continuous a.e., this can get us the following important theorem, indicating that the
1
topological equivalence between (X ,d ) and (X ,d ) is a sufficient condition in determining
1 1 2 2
whetherornotf isstrong-robustagainstadversarialexamples:
1
Theorem 4.2. When f is continuous a.e., if (X ,d ) and (X ,d ) are topologically equivalent,
1 1 1 2 2
thenthelearnedclassifierf (·)isstrong-robusttoadversarialexamples.
1
Proof. SeeitsproofsinSection10.3.4
Thistheoremcanactuallyguaranteethat:
∀x,x(cid:48) ∈X,
P(f (x)=f (x(cid:48))|f (x)=f (x(cid:48)), (4.5)
1 1 2 2
d (g (x),g (x(cid:48)))<δ )=1
2 2 2 2
ClearlyEq.(4.5)isaspecial(stronger)caseofthe“strong-robustness”definedbyEq.(4.2).
Formoregeneralcasesincludingf mightnotbecontinuousa.e.,weneedtoconsidertheprobability
1
oftheboundarypointattacks(Eq.(4.1)). Therefore,wegetamoregeneraltheoremasfollows:
Theorem4.3. If(X ,d )and(X ,d )aretopologicallyequivalentandP(f (x)(cid:54)=f (x(cid:48))|f (x)=
1 1 2 2 1 1 2
f (x(cid:48)),d (g (x),g (x(cid:48))) < δ ,d (g (x),g (x(cid:48))) < δ ) < η, then the learned classifier f (·) is
2 1 1 1 1 2 2 2 2 1
strong-robusttoadversarialexamples.
Proof. SeeitsproofsinSection10.3.3.
4.4 FINERTOPOLOGYOF(X,d(cid:48))THAN(X,d(cid:48))ISSUFFICIENTANDNECESSARYIN
1 2
DETERMININGSTRONG-ROBUSTNESS
Nowweextendthediscussionfromtwometricspacesintotwopseudometricspaces. Thisextension
findsthesufficientandnecessaryconditionthatdeterminesthestrong-robustnessoff . Therelated
1
twopseudometricsared(cid:48) (forf )andd(cid:48) (forf ),bothdirectlybeingdefinedonX. AppendixSec-
1 1 2 2
tion10.2includesdetaileddescriptionsofpseudometric,pseudometricspaces,topologyandafiner
topologyrelationshipbetweentwopseudometricspaces.
9
Workshoptrack-ICLR2017
Table3: Summaryoftheoreticalconclusionsthatwecanderive. HereX1 = Rn1 andX2 = Rn2.
Thestrong-robustnessisdeterminedbyfeatureextractionfunctiong . Theaccuracyisdeterminedby
1
boththeclassificationfunctionc andthefeatureextractionfunctiong .
1 1
Cases: d &d arenorms Canbeaccurate? Basedon Illustration
1 2
(cid:84)
(I) X \(X X )(cid:54)=∅, NotStrong-robust cannotbeaccurate Theorem(4.4) Figure2
1 1 2
X (cid:54)⊂X
2 1
(II) n >n ,X (cid:40)X Notstrong-robust canbeaccurate Corollary(5.1) Figure2
1 2 2 1
(III) n =n ,X =X Strong-robust canbeaccurate Corollary(5.2) Figure4
1 2 1 2
(IV) n <n ,X ⊂X Strong-robust cannotbeaccurate Theorem(4.4) Figure5
1 2 1 2
Essentially, thetopologyinpseudometricspace(X,d(cid:48)))isafinertopologythanthetopologyin
1
pseudometricspace(X,d(cid:48))means:
2
∀x,x(cid:48) ∈X,d(cid:48)(x,x(cid:48))<δ ⇒d(cid:48)(x,x(cid:48))<δ (4.6)
2 2 1 1
Becaused(cid:48)(x,x(cid:48))=d (g (x),g (x(cid:48)))andd(cid:48)(x,x(cid:48))=d (g (x),g (x(cid:48))),theaboveequationequals
1 1 1 1 2 2 2 2
to:
∀x,x(cid:48) ∈X,
(4.7)
d (g (x),g (x(cid:48)))<δ ⇒d (g (x),g (x(cid:48)))<δ
2 2 2 2 1 1 1 1
UsingEq.(4.7)andthecontinuitya.e. assumption,wecanderivethefollowingTheoremaboutthe
sufficientandnecessaryconditionforf beingstrong-robust:
1
Theorem4.4. Whenf iscontinuousa.e.,f isstrong-robustagainstadversarialexamplesifand
1 1
onlyifthetopologyin(X,d(cid:48))isafinertopologythanthetopologyin(X,d(cid:48)).
1 2
Proof. SeeitsproofinappendixSection10.3.1.
Actuallytheabovetheoremcanguaranteethatwhenf iscontinuousa.e.:
1
∀x,x(cid:48) ∈X,P(f (x)=f (x(cid:48))|d (g (x),g (x(cid:48)))<δ )=1 (4.8)
1 1 2 2 2 2
Eq.(4.8)clearlyisaspecial(stronger)caseofstrong-robustnessdefinedbyEq.(4.2).
Whenf isnotcontinuousa.e.,weneedtoconsidertheprobabilityoftheboundarypointsbased
1
adversarial examples (Eq. (4.1)). For such a case, we get a sufficient condition 10 for the strong-
robustness:
Theorem 4.5. When f is not continuous a.e., if the topology in (X,d(cid:48))) is a finer topol-
1 1
ogy than the topology in (X,d(cid:48)) and P(f (x) (cid:54)= f (x(cid:48))|f (x) = f (x(cid:48)),d (g (x),g (x(cid:48))) <
2 1 1 2 2 1 1 1
δ ,d (g (x),g (x(cid:48)))<δ )<η,thenf isstrong-robustagainstadversarialexamples.
1 2 2 2 2 1
Whenf isnotcontinuousa.e.,itsstrong-robustnessissignificantlyinfluencedbyitsboundarypoints
1
andthereforerelatestothec function. Section13.2providessomediscussionandweomitcovering
1
suchcasesintherestofthispaper.
5 TOWARDS PRINCIPLED UNDERSTANDING
Thefourtheoremsproposedaboveleadtoasetofkeyinsightsaboutwhyandhowanadversarialcan
foolamachine-learningclassifierusingadversarialexamples. Oneofthemostvaluableinsightsis:
featurelearningstepdecideswhetherapredictorisstrong-robustornotinanadversarialtestsetting.
Allthediscussionsinthesubsectionassumef iscontinuousa.e..
1
5.1 UNNECESSARYFEATURESRUINSTRONG-ROBUSTNESS
Theorem(4.2)and Theorem(4.4)indicatethatwhenf iscontinuousa.e.,thetwofeaturespaces
1
(X ,d )and(X ,d )orthefunctionsg andg determinethestrong-robustnessoff . Basedon
1 1 2 2 1 2 1
Theorem(4.4),wecanderiveacorollaryasfollows(proofinSection10.3.1):
10When f is not continuous a.e., it is difficult to find the necessary and sufficient condition for strong-
1
robustnessoff .Weleavethistofutureresearch.
1
10