Towards Generalization and Simplicity in Continuous Control

12 Pages·2006·0.78 MB·English

Towards Generalization and Simplicity
in Continuous Control
AravindRajeswaran∗ KendallLowrey∗ EmanuelTodorov ShamKakade
UniversityofWashingtonSeattle
{ aravraj, klowrey, todorov, sham } @ cs.washington.edu
Abstract
ThisworkshowsthatpolicieswithsimplelinearandRBFparameterizationscan
betrainedtosolveavarietyofwidelystudiedcontinuouscontroltasks,including
thegym-v1benchmarks. Theperformanceofthesetrainedpoliciesarecompetitive
withstateoftheartresults,obtainedwithmoreelaborateparameterizationssuchas
fullyconnectedneuralnetworks. Furthermore,thestandardtrainingandtesting
scenariosforthesetasksareshowntobeverylimitedandpronetoover-fitting,thus
givingrisetoonlytrajectory-centricpolicies. Trainingwithadiverseinitialstate
distributioninducesmoreglobalpolicieswithbettergeneralization. Thisallows
for interactive control scenarios where the system recovers from large on-line
perturbations;asshowninthesupplementaryvideo.
1 Introduction
Deep reinforcement learning (deepRL) has recently achieved impressive results on a number of
hard problems, including sequential decision making in game domains [1, 2]. This success has
motivatedeffortstoadaptdeepRLmethodsforcontrolofphysicalsystems,andhasresultedinrich
motorbehaviors[3,4]. ThecomplexityofsystemssolvablewithdeepRLmethodsisnotyetatthe
levelofwhatcanbeachievedwithtrajectoryoptimization(planning)insimulation[5,6,7],orwith
hand-craftedcontrollersonphysicalrobots(e.g. BostonDynamics). However,RLapproachesare
excitingbecausetheyaregeneric,model-free,andhighlyautomated.
Recent success of RL [2, 8, 9, 10, 11] has been enabled largely due to engineering efforts such
aslargescaledatacollection[1,2,11]orcarefulsystemsdesign[8,9]withwellbehavedrobots.
Whenadvancesinafieldarelargelyempiricalinnature,itisimportanttounderstandtherelative
contributions of representations, optimization methods, and task design or modeling: both as a
sanitycheckandtoscaleuptohardertasks. Furthermore,inlinewithOccam’srazor,thesimplest
reasonableapproachesshouldbetriedandunderstoodfirst.Athoroughunderstandingofthesefactors
isunfortunatelylackinginthecommunity.
Inthisbackdrop,weaskthepertinentquestion: "Whatarethesimplestsetofingredientsneeded
tosucceedinsomeofthepopularbenchmarks?"Toattemptthisquestion,weusetheGym-v1[12]
continuouscontrolbenchmarks,whichhaveacceleratedresearchandenabledobjectivecomparisons.
Sincethetasksinvolveunder-actuation,contactdynamics,andarehighdimensional(continuous
space),theyhavebeenacceptedasbenchmarksinthedeepRLcommunity. Recentworkstesttheir
algorithmseitherexclusivelyorprimarilyonthesetasks[13,4,14],andsuccessonthesetaskshave
beenregardedasdemonstratinga“proofofconcept”.
Ourcontributions: Ourresultsandtheirimplicationsarehighlightedbelowwithmoreelaborate
discussionsinSection5:
∗Equalcontributions.Projectpage:https://sites.google.com/view/simple-pol
31stConferenceonNeuralInformationProcessingSystems(NIPS2017),LongBeach,CA,USA.
1. ThesuccessofrecentRLeffortstoproducerichmotorbehaviorshavelargelybeenattributed
totheuseofmulti-layerneuralnetworkarchitectures. Thisworkisamongthefirsttocarefully
analyzetheroleofrepresentation,andourresultsindicatethatverysimplepoliciesincluding
linearandRBFparameterizationsareabletoachievestateoftheartresultsonwidelystudied
tasks. Furthermore,suchpolicies,particularlythelinearones,canbetrainedsignificantlyfaster
duetoordersofmagnitudefewerparameters. Thisindicatesthatevenfortaskswithcomplex
dynamics,therecouldexistrelativelysimplepolicies. Thisopensthedoorforstudyingawide
rangeofrepresentationsinadditiontodeepneuralnetworks,andunderstandtrade-offsincluding
computationaltime,theoreticaljustification,robustness,samplecomplexityetc.
2. Westudytheseissuesnotonlywithregardstotheperformancemetricathandbutwealsotake
thefurtherstepinexaminingtheminthecontextofrobustness. Ourresultsindicatethat,with
conventionaltrainingmethods,theagentisabletosuccessfullylearnalimitcycleforwalking,
butcannotrecoverfromanyperturbationsthataredeliveredtoit. Fortransferringthesuccessof
RLtorobotics,suchbrittlenessishighlyundesirable.
3. Finally, we directly attempt to learn more robust policies through using more diverse train-
ing conditions, which favor such policies. This is similar in spirit to the model ensemble
approaches[15,16]anddomainrandomizationapproaches[17,18],whichhavesuccessfully
demonstratedimprovedrobustnessandsimulationtorealworldtransfer. Underthesenewand
morediversetrainingscenarios,weagainfindthatthereisnocompellingevidencetofavorthe
useofmulti-layerarchitectures,atleastforthebenchmarktasks. Onasidenote,wealsoprovide
interactivetestingoflearnedpolicies,whichwebelieveisbothnovelandwhichshedslighton
therobustnessoftrainedpolicies.
2 ProblemFormulationandMethods
WeconsiderMarkovDecisionProcesses(MDPs)intheaveragerewardsetting,whichisdefinedusing
thetuple: M={S,A,R,T,ρ }. S ⊆Rn,A⊆Rm,andR:S×A→Rarea(continuous)setof
0
states,setofactions,andrewardfunctionrespectively,andhavetheusualmeaning. T :S×A→S
isthestochastictransitionfunctionandρ istheprobabilitydistributionoverinitialstates. Wewish
0
tosolveforastochasticpolicyoftheformπ :S×A→R ,whichoptimizestheobjectivefunction:
+
(cid:34) T (cid:35)
1 (cid:88)
η(π)= lim E r . (1)
T→∞ T π,M t
t=1
Since we use simulations with finite length rollouts to estimate the objective and gradient, we
approximateη(π)usingafiniteT. Inthisfinitehorizonrolloutsetting,wedefinethevalue,Q,and
advantagefunctionsasfollows:
(cid:34) T (cid:35)
(cid:88) (cid:104) (cid:105) (cid:104) (cid:105)
Vπ(s,t)=E r Qπ(s,a,t)=E R(s,a) +E Vπ(s(cid:48),t+1)
π,M t(cid:48) M s(cid:48)∼T(s,a)
t(cid:48)=t
Aπ(s,a,t)=Qπ(s,a,t)−Vπ(s,t)
Notethateventhoughthevaluefunctionsaretime-varying,westilloptimizeforastationarypolicy.
Weconsiderparametrizedpoliciesπ ,andhencewishtooptimizefortheparameters(θ). Thus,we
θ
overloadnotationanduseη(π)andη(θ)interchangeably.
2.1 Algorithm
Ideally, a controlled scientific study would seek to isolate the challenges related to architecture,
taskdesign,andtrainingmethodsforseparatestudy. Inpractice,thisisnotentirelyfeasibleasthe
resultsarepartlycoupledwiththetrainingmethods. Here,weutilizeastraightforwardnaturalpolicy
gradientmethodfortraining. Theworkin[19]suggeststhatthismethodiscompetitivewithmost
stateoftheartmethods. Wenowdiscussthetrainingprocedure.
UsingthelikelihoodratioapproachandMarkovpropertyoftheproblem,thesamplebasedestimate
ofthepolicygradientisderivedtobe[20]:
T
∇ ηˆ(θ)=g = 1 (cid:88)∇ logπ (a |s )Aˆπ(s ,a ,t) (2)
θ T θ θ t t t t
t=0
2
Algorithm1PolicySearchwithNaturalGradient
1: Initializepolicyparameterstoθ0
2: fork =1toK do
3: Collecttrajectories{τ(1),...τ(N)}byrollingoutthestochasticpolicyπ(·;θk).
4: Compute∇θlogπ(at|st;θk)foreach(s,a)pairalongtrajectoriessamplediniterationk.
5: ComputeadvantagesAπ basedontrajectoriesiniterationkandapproximatevaluefunction
k
Vπ .
k−1
6: Computepolicygradientaccordingto(2).
7: ComputetheFishermatrix(4)andperformgradientascent(5).
8: Update parameters of value function in order to approximate Vπ(s(n)) ≈ R(s(n)), where
k t t
R(s(n)) is the empirical return computedas R(s(n))=(cid:80)T γ(t(cid:48)−t)r(n). Here n indexes
t t t(cid:48)=t t
overthetrajectories.
9: endfor
Gradientascentusingthis“vanilla”gradientissub-optimalsinceitisnotthesteepestascentdirection
inthemetricoftheparameterspace[21,22]. Thesteepestascentdirectionisobtainedbysolvingthe
followinglocaloptimizationproblemarounditerateθ :
k
maxiθmize gT(θ−θk) subjectto (θ−θk)TFθk(θ−θk)≤δ, (3)
whereF istheFisherInformationMetricatthecurrentiterateθ . WeestimateF as
θk k θk
T
Fˆ = 1 (cid:88)∇ logπ (a |s )∇ logπ (a |s )T, (4)
θk T θ θ t t θ θ t t
t=0
asoriginallysuggestedbyKakade[22]. ThisyieldsthesteepestascentdirectiontobeFˆ−1g and
θk
correspondingupdaterule: θ =θ +αFˆ−1g. Hereαisthestep-sizeorlearningrateparameter.
Empirically,weobservedthatkc+h1oosinkgafixeθdkvalueforαoranappropriatescheduleisdifficult[23].
Thus,weusethenormalizedgradientascentprocedure,wherethenormalizationisundertheFisher
metric. This procedure can be viewed as picking a normalized step size δ as opposed to α, and
solvingtheoptimizationproblemin(3). Thisresultsinthefollowingupdaterule:
(cid:115)
δ
θ =θ + Fˆ−1g. (5)
k+1 k gTFˆ−1g θk
θk
A dimensional analysis of these quantities reveal that α has the unit of return−1 whereas δ is
dimensionless. Thoughunitsofαareconsistentwithageneraloptimizationsettingwherestep-size
hasunitsofobjective−1,intheseproblems,pickingagoodαthatisconsistentwiththescalesofthe
rewardwasdifficult. Ontheotherhand,aconstantnormalizedstepsizewasnumericallymorestable
andeasiertotune: foralltheresultsreportedinthispaper,thesameδ =0.05wasused. Whenmore
thanonetrajectoryrolloutisusedperupdate,theaboveestimatorscanbeusedwithanadditional
averagingoverthetrajectories.
For estimating the advantage function, we use the GAE procedure [13]. This requires learning
a function that approximates Vπ, which is used to compute Aπ along trajectories for the update
k k
in (5). GAE helps with variance reduction at the cost of introducing bias, and requires tuning
hyperparameterslikeadiscountfactorandanexponentialaveragingterm. Goodheuristicsforthese
parametershavebeensuggestedinpriorwork. Thesamebatchoftrajectoriescannotbeusedforboth
fittingthevaluefunctionbaseline,andalsotoestimategusing(2),sinceitwillleadtooverfitting
andabiasedestimate. Thus,weusethetrajectoriesfromiterationk−1tofitthevaluefunction,
essentiallyapproximatingVπ ,andusetrajectoriesfromiterationkforcomputingAπandg.Similar
k−1 k
procedureshavebeenadoptedinpriorwork[19].
2.2 PolicyArchitecture
Linearpolicy: Wefirstconsideralinearpolicythatdirectlymapsfromtheobservationstothe
motortorques. Weusethesameobservationsasusedinpriorworkwhichincludesjointpositions,
3
jointvelocities, andforsometasks, informationrelatedtocontacts. Thus, thepolicymappingis
a ∼N(Ws +b,σ),andthegoalistolearnW,b,andσ. Formostofthesetasks,theobservations
t t
correspondtothestateoftheproblem(inrelativecoordinates). Thus,weusethetermstatesand
observationsinterchangeably. Ingeneral,thepolicyisdefinedwithobservationsastheinput,and
henceistryingtosolveaPOMDP.
RBFpolicy: Secondly,weconsideraparameterizationthatenrichestherepresentationalcapacity
using random Fourier features of the observations. Since these features approximate the RKHS
featuresunderanRBFKernel[24],wecallthispolicyparametrizationtheRBFpolicy. Thefeatures
areconstructedas:
(cid:32)(cid:80) P s(j) (cid:33)
y(i) =sin j ij t +φ(i) , (6)
t ν
whereeachelementP isdrawnfromN(0,1),ν isabandwidthparameterchosenapproximatelyas
ij
theaveragepairwisedistancesbetweendifferentobservationvectors,andφisarandomphaseshift
drawnfromU[−π,π). Thusthepolicyisa ∼ N(Wy +b,σ),whereW,b,andσ aretrainable
t t
parameters. This architecture can also be interpreted as a two layer neural network: the bottom
layerisclampedwithrandomweights,asinusoidalactivationfunctionisused,andthetoplayeris
finetuned. Theprincipalpurposeforthisrepresentationistoslightlyenhancethecapacityofalinear
policy,andthechoiceofactivationfunctionisnotverysignificant.
3 ResultsonOpenAIgym-v1benchmarks
Asindicatedbefore,wetrainlinearandRBFpolicieswiththenaturalpolicygradientonthepopular
OpenAIgym-v1benchmarktaskssimulatedinMuJoCo[25]. Thetasksprimarilyconsistoflearning
locomotiongaitsforsimulatedrobotsrangingfromaswimmertoa3Dhumanoid(23dof).
Figure1presentsthelearningcurvesalongwiththeperformancelevelsreportedinpriorworkusing
TRPOandfullyconnectedneuralnetworkpolicies. Table1alsosummarizesthefinalscores,where
“stoc”referstothestochasticpolicywithactionssampledasa ∼π (s ),while“mean”refersto
t θ t
usingmeanoftheGaussianpolicy,withactionscomputedasa =E[π (s )]. Weseethatthelinear
t θ t
policyiscompetitiveonmosttasks,whiletheRBFpolicycanoutperformpreviousresultsonfive
ofthesixconsideredtasks. Thoughwewereabletotrainneuralnetworkpoliciesthatmatchthe
resultsreportedinliterature,wehaveusedpubliclyavailablepriorresultsforanobjectivecomparison.
VisualizationsofthetrainedlinearandRBFpoliciesarepresentedinthesupplementaryvideo. Given
thesimplicityofthesepolicies,itissurprisingthattheycanproducesuchelaboratebehaviors.
Table2presentsthenumberofsamplesneededforthepolicyperformancetoreachathresholdvalue
forreward. Thethresholdvalueiscomputedas90%ofthefinalscoreachievedbythestochastic
linearpolicy. Wevisuallyverifiedthatpolicieswiththesescoresareproficientatthetask,andhence
thechosenvaluescorrespondtomeaningfulperformancethresholds. WeseethatlinearandRBF
policiesareabletolearnfasteronfourofthesixtasks.
Allthesimulatedrobotsweconsideredareunder-actuated,havecontactdiscontinuities,andcon-
tinuous action spaces making them challenging benchmarks. When adapted from model-based
control [26, 5, 27] to RL, however, the notion of “success” established was not appropriate. To
shapethebehavior,averynarrowinitialstatedistributionandterminationconditionsareusedinthe
benchmarks. Asaconsequence,thelearnedpoliciesbecomehighlytrajectorycentric–i.e. theyare
goodonlywheretheytendtovisitduringtraining,whichisaverynarrowregion. Forexample,the
walkercanwalkverywellwheninitializeduprightandclosetothewalkinglimitcycle. Evensmall
perturbations,asshowninthesupplementaryvideo,altersthevisitationdistributionanddramatically
degrades the policy performance. This makes the agent fall down at which point it is unable to
getup. Similarly,theswimmerisunabletoturnwhenitsheadingdirectionisaltered. Forcontrol
applications,thisisundesirable. Intherealworld,therewillalwaysbeperturbations–stochasticity
intheenvironment,modelingerrors,orwearandtear. Thus,thespecifictaskdesignandnotionof
successusedforthesimulatedcharactersarenotadequate. However,thesimulatedrobotsthemselves
arerathercomplexandhardertaskscouldbedesignedwiththem,aspartlyillustratedinSection4.
4
Figure1: LearningcurvesfortheLinearandRBFpolicyarchitectures. Thegreenlinecorresponding
totherewardachievedbyneuralnetworkpoliciesontheOpenAIGymwebsite,asof02/24/2017
(trained with TRPO). It is observed that for all the tasks, linear and RBF parameterizations are
competitivewithstateoftheartresults. Thelearningcurvesdepictedareforthestochasticpolicies,
wheretheactionsaresampledasa ∼π (s ). Thelearningcurveshavebeenaveragedacrossthree
t θ t
runswithdifferentrandomseeds.
Table1:Finalperformancesofthepolicies Table2:Numberofepisodestoachievethreshold
Task Linear RBF NN Task Th. Linear RBF TRPO+NN
stoc mean stoc mean TRPO
Swimmer 362 366 361 365 131 Swimmer 325 1450 1550 N-A
Hopper 3466 3651 3590 3810 3668 Hopper 3120 13920 8640 10000
Cheetah 3810 4149 6477 6620 4800 Cheetah 3430 11250 6000 4250
Walker 4881 5234 5631 5867 5594 Walker 4390 36840 25680 14250
Ant 3980 4607 4297 4816 5007 Ant 3580 39240 30000 73500
Humanoid 5873 6440 6237 6849 6482 Humanoid 5280 79800 96720 87000
4 ModifiedTasksandResults
UsingthesamesetofsimulatedrobotcharactersoutlinedinSection3,wedesignednewtaskswith
twogoalsinmind: (a)topushtherepresentationalcapabilitiesandtestthelimitsofsimplepolicies;
(b)toenabletrainingof“global"policiesthatarerobusttoperturbationsandworkfromadiverseset
ofstates. Tothisend,wemakethefollowingbroadchanges,alsosummarizedinTable3:
1. Widerinitialstatedistributiontoforcegeneralization. Forexample,inthewalkertask,some
fractionoftrajectorieshavethewalkerinitializedproneontheground. Thisforcestheagentto
simultaneouslylearnaget-upskillandawalkskill,andnotforgetthemasthelearningprogresses.
Similarly,theheadinganglefortheswimmerandantarerandomized,whichencourageslearning
ofaturnskill.
2. Rewardshapingappropriatewiththeabovechangestotheinitialstatedistribution. Forexample,
whenthemodifiedswimmerstartswitharandomizedheadingangle,weincludeasmallrewardfor
adjustingitsheadingtowardsthecorrectdirection. Inconjunction,wealsoremovealltermination
conditionsusedintheGym-v1benchmarks.
3. Changestoenvironment’sphysicsparameters,suchasmassandjointtorque. Iftheagenthas
sufficient power, most tasks are easily solved. By reducing an agent’s action ability and/or
increasingitsmass,theagentismoreunder-actuated. Thesechangesalsoproducemorerealistic
lookingmotion.
5
Figure2: Hoppercompletesaget-upsequencebeforemovingtoitsnormalforwardwalkingbehavior.
Thegetupsequenceislearnedalongsidetheforwardhoppinginthemodifiedtasksetting.
Table3: ModifiedTaskDescription
v isforwardvelocity;θistheheadingangle;p istheheightoftorso;andaistheaction.
x z
Task Description Reward(des=desiredvalue)
Agentswimsinthedesireddirection.
Swimmer(3D) v −0.1|θ−θdes|−0.0001||a||2
Shouldrecover(turn)ifrotatedaround. x
Agenthopsforwardasfastaspossible.
Hopper(2D) v −3||p −pdes||2−0.1||a||2
Shouldrecover(getup)ifpusheddown. x z z
Agentwalksforwardasfastaspossible.
Walker(2D) v −3||p −pdes||2−0.1||a||2
Shouldrecover(getup)ifpusheddown. x z z
Agentmovesinthedesireddirection.
Ant(3D) v −3||p −pdes||2−0.01||a||2
Shouldrecover(turn)ifrotatedaround. x z z
Combined, these modifications require that the learned policies not only make progress towards
maximizingthereward,butalsorecoverfromadverseconditionsandresistperturbations.Anexample
of this is illustrated in Figure 4, where the hopper executes a get-up sequence before hopping to
makeforwardprogress. Furthermore,attesttime,ausercaninteractivelyapplypushingandrotating
perturbationstobetterunderstandthefailuremodes. Wenotethattheseinteractiveperturbationsmay
notbetheultimatetestforrobustness,butasteptowardsthisdirection.
Representationalcapacity Thesupplementaryvideodemonstratesthetrainedpolicies. Wecon-
centrate on the resultsof the walker task in the main paper. Figure 3 studies theperformance as
wevarytherepresentationalcapacity. IncreasingtheFourierfeaturesallowsformoreexpressive
policiesandconsequentlyallowforachievingahigherscore. Thepolicywith500Fourierfeatures
performsthebest,followedbythefullyconnectedneuralnetwork. Thelinearpolicyalsomakes
forwardprogressandcangetupfromtheground,butisunabletolearnasefficientawalkinggait.
(a) (b)
Figure3: (a)Learningcurveonmodifiedwalker(diverseinitialization)fordifferentpolicyarchi-
tectures. Thecurvesareaveragedoverthreerunswithdifferentrandomseeds. (b)Learningcurves
whenusingdifferentnumberofconjugategradientiterationstocomputeFˆ−1gin(5). Apolicywith
θk
300Fourierfeatureshasbeenusedtogeneratetheseresults.
6
Figure4: Wetestpolicyrobustnessbymeasuringdistancedtraveledintheswimmer,walker,and
hoppertasksforthreetrainingconfigurations: (a)withterminationconditions;(b)notermination,
andpeakedinitialstatedistribution;and(c)withdiverseinitialization. Swimmerdoesnothavea
terminationoption,soweconsideronlytwoconfigurations.Forthecaseofswimmer,theperturbation
ischangingtheheadinganglebetween−π/2.0andπ/2.0,andinthecaseofwalkerandhopper,an
externalforcefor0.5secondsalongitsaxisofmovement. Allagentsareinitializedwiththesame
positionsandvelocities.
Perturbationresistance Next,wetesttherobustnessofourpoliciesbyperturbingthesystemwith
anexternalforce. Thisexternalforcerepresentsanunforeseenchangewhichtheagenthastoresist
orovercome,thusenablingustounderstandpushandfallrecoveries. Fallrecoveriesofthetrained
policiesaredemonstratedinthesupplementaryvideo. Inthesetasks,perturbationsarenotappliedto
thesystemduringthetrainingphase. Thus,theabilitytogeneralizeandresistperturbationscome
entirelyoutofthestatesvisitedbytheagentduringtraining. Figure4indicatesthattheRBFpolicy
is more robust, and also that diverse initializations are important to obtain the best results. This
indicatesthatcarefuldesignofinitialstatedistributionsarecrucialforgeneralization,andtoenable
theagenttolearnawiderangeofskills.
5 SummaryandDiscussion
Theexperimentsinthispaperwereaimedattryingtounderstandtheeffectsof(a)representation;(b)
taskmodeling;and(c)optimization. Wesummarizetheresultswithregardtoeachaforementioned
factoranddiscusstheirimplications.
Representation The finding that linear and RBF policies can be trained to solve a variety of
continuouscontroltasksisverysurprising. Recently,anumberofalgorithmshavebeenshowntosuc-
cessfullysolvethesetasks[3,28,4,14],butalloftheseworksusemulti-layerneuralnetworks. This
suggestsawidespreadbeliefthatexpressivefunctionapproximatorsareneededtocaptureintricate
detailsnecessaryformovementslikerunning. Theresultsinthisworkconclusivelydemonstratesthat
thisisnotthecase,atleastforthelimitedsetofpopulartestbeds. Thisraisesaninterestingquestion:
whatarethecapabilitylimitsofshallowpolicyarchitectures? Thelinearpolicieswerenotexemplary
inthe“global”versionsofthetasks,butitmustbenotedthattheywerenotterribleeither. TheRBF
policyusingrandomFourierfeatureswasabletosuccessfullysolvethemodifiedtasksproducing
globalpolicies,suggestingthatwedonotyethaveasenseofitslimits.
Modeling WhenusingRLmethodstosolvepracticalproblems,theworldprovidesuswithneither
theinitialstatedistributionnorthereward. Bothofthesemustbedesignedbytheresearcherand
mustbetreatedasassumptionsabouttheworldorprescriptionsabouttherequiredbehavior. The
qualityofassumptionswillinvariablyaffectthequalityofsolutions,andthuscaremustbetakenin
thisprocess. Here,weshowthatstartingthesystemfromanarrowinitialstatedistributionproduces
7
elaboratebehaviors,butthetrainedpoliciesareverybrittletoperturbations. Usingamorediverse
statedistribution,inthesecases,issufficienttotrainrobustpolicies.
Optimization Inlinewiththethemeofsimplicity,wefirsttriedtouseREINFORCE[20],which
wefoundtobeverysensitivetohyperparameterchoices,especiallystep-size. Thereareaclassof
policygradientmethodswhichusepre-conditioningtohelpnavigatethewarpedparameterspaceof
probabilitydistributionsandforstepsizeselection. Mostvariantsofpre-conditionedpolicygradient
methodshavebeenreportedtoachievestateoftheartperformance,allperformingaboutthesame[19].
Wefeelthattheusednaturalpolicygradientmethodisthemoststraightforwardpre-conditioned
method. To demonstrate that the pre-conditioning helps, Figure 3 depicts the learning curve for
differentnumberofCGiterationsusedtocomputetheupdatein(5). Thecurvecorrespondingto
CG = 0 is the REINFORCE method. As can be seen, pre-conditioning helps with the learning
process. However,thereisatrade-offwithcomputation,andhenceusinganintermediatenumberof
CGstepslike20couldleadtobestresultsinwall-clocksenseforlargescaleproblems.
WechosetocomparewithneuralnetworkpoliciestrainedwithTRPO,sinceithasdemonstrated
impressive results and is closest to the algorithm used in this work. Are function approximators
linearwithrespecttofreeparameterssufficientforothermethodsisaninterestingopenquestion
(inthissense, RBFsarelinearbutNNsarenot). Foralargeclassofmethodsbasedondynamic
programming(includingQ-learning,SARSA,approximatepolicyandvalueiteration),linearfunction
approximationhasguaranteedconvergenceanderrorbounds,whilenon-linearfunctionapproximation
isknowntodivergeinmanycases[29,30,31,32]. Itmayofcoursebepossibletoavoiddivergence
inspecificapplications,oratleastslowitdownlongenough,forexampleviatargetnetworksor
replaybuffers. Nevertheless,guaranteedconvergencehasclearadvantages. Similartorecentwork
using policy gradient methods, recent work using dynamic programming methods have adopted
multi-layernetworkswithoutcarefulside-by-sidecomparisonstosimplerarchitectures. Coulda
globalquadraticapproximationtotheoptimalvaluefunction(whichislinearinthesetofquadratic
features)besufficienttosolvemostofthecontinuouscontroltaskscurrentlystudiedinRL?Given
thatquadraticvaluefunctionscorrespondtolinearpolicies,andgoodlinearpoliciesexistasshown
here,thismightmakeforinterestingfuturework.
6 Conclusion
Inthiswork,wedemonstratedthatverysimplepolicyparameterizationscanbeusedtosolvemany
benchmarkcontinuouscontroltasks. Furthermore,thereisnosignificantlossinperformancedueto
theuseofsuchsimpleparameterizations. Wealsoproposedglobalvariantsofmanywidelystudied
tasks,whichrequiresthelearnedpoliciestobecompetentforamuchlargersetofstates,andfound
thatsimplerepresentationsaresufficientinthesecasesaswell. Theseempiricalresultsalongwith
Occam’srazorsuggeststhatcomplexpolicyarchitecturesshouldnotbeadefaultchoiceunlessside-
by-sidecomparisonswithsimpleralternativessuggestotherwise. Suchcomparisonsareunfortunately
not widely pursued. The results presented in this work directly highlight the need for simplicity
andgeneralizationinRL.Wehopethatthisworkwouldencouragefutureworkanalyzingvarious
architecturesandassociatedtrade-offslikecomputationtime,robustness,andsamplecomplexity.
Acknowledgements
ThisworkwassupportedinpartbytheNSF.TheauthorswouldliketothankVikashKumar,Igor
Mordatch,JohnSchulman,andSergeyLevineforvaluablecomments.
References
[1] V.Mnihetal. Human-levelcontrolthroughdeepreinforcementlearning. Nature,518,2015.
[2] D.Silveretal. Masteringthegameofgowithdeepneuralnetworksandtreesearch. Nature,
529,2016.
[3] J.Schulman,S.Levine,P.Moritz,M.Jordan,andP.Abbeel. Trustregionpolicyoptimization.
InICML,2015.
8
[4] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.
Continuouscontrolwithdeepreinforcementlearning. ArXive-prints,September2015.
[5] Y.Tassa,T.Erez,andE.Todorov. Synthesisandstabilizationofcomplexbehaviorsthrough
onlinetrajectoryoptimization. InternationalConferenceonIntelligentRobotsandSystems,
2012.
[6] I.Mordatch,E.Todorov,andZ.Popovic. Discoveryofcomplexbehaviorsthroughcontact-
invariantoptimization. ACMSIGGRAPH,2012.
[7] M.AlBorno,M.deLasa,andA.Hertzmann.TrajectoryOptimizationforFull-BodyMovements
withComplexContacts. IEEETransactionsonVisualizationandComputerGraphics,2013.
[8] SergeyLevine,ChelseaFinn,TrevorDarrell,andPieterAbbeel. End-to-endtrainingofdeep
visuomotorpolicies. JMLR,17(39):1–40,2016.
[9] V.Kumar,E.Todorov,andS.Levine. Optimalcontrolwithlearnedlocalmodels: Application
todexterousmanipulation. InICRA,2016.
[10] V.Kumar,A.Gupta,E.Todorov,andS.Levine. Learningdexterousmanipulationpoliciesfrom
experienceandimitation. ArXive-prints,2016.
[11] LerrelPintoandAbhinavGupta. Supersizingself-supervision: Learningtograspfrom50ktries
and700robothours. InICRA,2016.
[12] G.Brockman,V.Cheung,L.Pettersson,J.Schneider,J.Schulman,J.Tang,andW.Zaremba.
OpenAIGym,2016.
[13] J.Schulman,P.Moritz,S.Levine,M.Jordan,andP.Abbeel. High-dimensionalcontinuous
controlusinggeneralizedadvantageestimation. InICLR,2016.
[14] ShixiangGu,TimothyLillicrap,ZoubinGhahramani,RichardE.Turner,andSergeyLevine.
Q-Prop: Sample-EfficientPolicyGradientwithAnOff-PolicyCritic. InICLR,2017.
[15] I.Mordatch,K.Lowrey,andE.Todorov. Ensemble-CIO:Full-bodydynamicmotionplanning
thattransferstophysicalhumanoids. InIROS,2015.
[16] A. Rajeswaran, S. Ghotra, B. Ravindran, and S. Levine. EPOpt: Learning Robust Neural
NetworkPoliciesUsingModelEnsembles. InICLR,2017.
[17] FereshtehSadeghiandSergeyLevine. (CAD)2RL:RealSingle-ImageFlightwithoutaSingle
RealImage. ArXive-prints,2016.
[18] JoshTobin,RachelFong,AlexRay,JonasSchneider,WojciechZaremba,andPieterAbbeel.
Domainrandomizationfortransferringdeepneuralnetworksfromsimulationtotherealworld.
ArXive-prints,2017.
[19] Y.Duan,X.Chen,R.Houthooft,J.Schulman,andP.Abbeel.Benchmarkingdeepreinforcement
learningforcontinuouscontrol. InICML,2016.
[20] RonaldJ.Williams. Simplestatisticalgradient-followingalgorithmsforconnectionistreinforce-
mentlearning. MachineLearning,8(3):229–256,1992.
[21] S.Amari. Naturalgradientworksefficientlyinlearning. NeuralComputation,10:251–276,
1998.
[22] S.Kakade. Anaturalpolicygradient. InNIPS,2001.
[23] Jan Peters. Machine learning of motor skills for robotics. PhD Dissertation, University of
SouthernCalifornia,2007.
[24] A.RahimiandB.Recht. RandomFeaturesforLarge-ScaleKernelMachines. InNIPS,2007.
[25] E.Todorov, T.Erez, andY.Tassa. MuJoCo: Aphysicsengineformodel-basedcontrol. In
InternationalConferenceonIntelligentRobotsandSystems,2012.
9
[26] TomErez,YuvalTassa,andEmanuelTodorov. Infinite-horizonmodelpredictivecontrolfor
periodictaskswithcontacts. InRSS,2011.
[27] T.Erez,K.Lowrey,Y.Tassa,V.Kumar,S.Kolev,andE.Todorov. Anintegratedsystemfor
real-timemodelpredictivecontrolofhumanoidrobots. InHumanoids,pages292–299,2013.
[28] Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa.
Learningcontinuouscontrolpoliciesbystochasticvaluegradients. InNIPS,2015.
[29] AlborzGeramifard,ThomasJWalsh,StefanieTellex,GirishChowdhary,NicholasRoy,and
JonathanPHow. Atutorialonlinearfunctionapproximatorsfordynamicprogrammingand
reinforcementlearning. FoundationsandTrends(cid:13)R inMachineLearning,6(4):375–451,2013.
[30] JennieSi. Handbookoflearningandapproximatedynamicprogramming, volume2. John
Wiley&Sons,2004.
[31] DimitriPBertsekas. Approximatedynamicprogramming. 2008.
[32] Residualalgorithms: Reinforcementlearningwithfunctionapproximation. InICML,1995.
10