A Decentralized Optimization Framework for Energy Harvesting Devices

0.91 MB·

by  Alessandro Biason

1
A Decentralized Optimization Framework
for Energy Harvesting Devices
Alessandro Biason, Student Member, IEEE, Subhrakanti Dey, Senior Member, IEEE,
and Michele Zorzi, Fellow, IEEE
Abstract—Designingdecentralizedpoliciesforwirelesscommunicationnetworksisacrucialproblem,whichhasonlybeenpartially
solvedintheliteraturesofar.Inthispaper,weproposetheDecentralizedMarkovDecisionProcess(Dec-MDP)frameworktoanalyzea
wirelesssensornetworkwithmultipleuserswhichaccessacommonwirelesschannel.Weconsiderdeviceswithenergyharvesting
capabilities,sothattheyaimatbalancingtheenergyarrivalswiththedatadeparturesandwiththeprobabilityofcollidingwithother
nodes.Randomlyovertime,anaccesspointtriggersaSYNCslot,whereinitrecomputestheoptimaltransmissionparametersofthe
wholenetwork,anddistributesthisinformation.Everynodereceivesitsownpolicy,whichspecifieshowitshouldaccessthechannelin
7 thefuture,and,thereafter,proceedsinafullydecentralizedfashion,withoutinteractingwithotherentitiesinthenetwork.Weproposea
1 multi-layerMarkovmodel,whereanexternalMDPmanagesthejumpsbetweenSYNCslots,andaninternalDec-MDPcomputesthe
0 optimalpolicyinthenearfuture.Wenumericallyshowthat,becauseoftheharvesting,afullyorthogonalscheme(e.g.,TDMA-like)is
2 suboptimalinenergyharvestingscenarios,andtheoptimaltrade-offliesbetweenanorthogonalandarandomaccesssystem.

n
a IndexTerms—admissioncontrol,energyconsumption,energyharvesting,energymanagement,optimalscheduling,powercontrol,
J telecommunicationnetworks.
9 (cid:70)

]
T 1 INTRODUCTION
I
.
s ENERGY Harvesting (EH) has been established as one of the from the environment (e.g., a solar panel, or a rectenna) and
c
most prominent solutions for prolonging the lifetime and to store it (e.g., miniaturized batteries). Clearly, the amount of
[
enhancingtheperformanceofWirelessSensorNetworks(WSNs) harvested energy strongly depends on the EH source, therefore

1 and Internet of Things (IoT) scenarios. Although this topic has manydifferentoptionshavebeenproposedsofar,includingpiezo-
v
been widely investigated in the literature so far, finding proper electric[3],temperaturegradient[4],dailytemperaturevariation,
1
energy management schemes is still an open issue in many cases vibrations [5], solar energy [6], indoor lights [7], and Radio-
8
of interest. In particular, using decentralized policies, in which Frequency(RF)energy[8].
0
2 everynodeinthenetworkactsautonomouslyandindependentlyof From a communication perspective, many analytical studies
0 theothers,isamajorproblemofpracticalinterestinWSNswhere aimed at maximizing the performance of an energy harvesting
. a central controller may not be used all the time. Many decen- network in terms of throughput [9], [10], [11], [12], delay [13],
1
0 tralized communication schemes (e.g., Aloha-like) can be found qualityofservice[14],orothermetrics.Inparticular,theproblem
7 in the literature; however, most of them were designed without a of maximizing the average value of the reported data using a
1 principleofoptimality,i.e.,withoutexplicitlytryingtomaximize device with a rechargeable battery was formulated in [15]. An
: thenetworkperformance.Instead,inthisworkwecharacterizethe information theoretic analysis of an EH system can be found
v
i optimal decentralized policy in a WSN with EH constraints and in [12], where the authors presented the “save-and-transmit” and
X
describetherelatedcomputationalissues.Althoughthisapproach the“best-effort-transmit”schemestoachievethechannelcapacity
r intrinsically leads to a more complex protocol definition, it also in the long run. Delay-aware strategies were presented in [13],
a
characterizes the maximum performance a network can achieve, where a single energy harvesting node is equipped with a data
and may serve as a baseline for defining quasi-optimal low- queue. Some researchers focused on batteryless devices [16],
complexityprotocols. [17]. In particular, [16] considered a traditional EH system with
Energy harvesting related problems in WSNs have been ad- amplitudeconstraintsandfoundthechannelcapacityundercausal
dressedbyseveralpreviousworks(e.g.,see[2]andthereferences channelinformationknowledge.Otherkindsofbatterylessdevices
therein), which tried to redefine many aspects of the communi- are presented in [17], in which a more innovative use of EH that
cation devices, from both hardware and software perspectives. exploitstheRFwavesasenergysourceisemployed.Deviceswith
Indeed,EHdevicesneedspecializedequipmenttoharvestenergy finite batteries were studied in our previous works [18], [19], but
also in [11], [20]. A common technique to model the batteries is
to approximate them with finite energy queues, in which energy
• AlessandroBiasonandMicheleZorziarewiththeDepartmentofInfor-
mationEngineering,UniversityofPadua,35131Padua,Italy. arrivesanddepartsovertime.Markovmodelsaresuitableforthese
E-mail:[email protected];[email protected] cases [21] and were largely adopted in the literature so far [9],
• SubhrakantiDeyiswiththeDepartmentofEngineeringScience,Uppsala [10],[15],[18],[19],[20],[22].Formorereferencesaboutenergy
University,Uppsala,Sweden.
harvesting,wereferthereaderstothesurveys[2],[8].
E-mail:[email protected]
However, differently from this paper, most of the protocols
ApreliminaryversionofthispaperhasbeenacceptedforpresentationatIEEE
proposed in the literature considered isolated nodes and did
WCNC2017[1].
ThisworkwaspartiallysupportedbyIntel’sCorporateResearchCouncil. not account for the interactions among devices, or focused on
2
centralized policies, in which a controller coordinates all nodes
and knows the global state of the system over time. [23] ana-
lyzeddecentralizedpolicieswithaparticularfocusonsymmetric
systems, and proposed a game theoretic approach for solving the
problem.Instead,inthispaperweuseadifferentframeworkbased
on decentralized Markov decision processes, which can handle
asymmetricscenarios.Decentralizedtheorywasalsousedin[24]
for a wireless powered communication network. However, the
scenario proposed therein is different from ours and the authors
onlyfocusedonanarrowsubclassofpolicies.
Figure1:Timeevolutionofthesystem.AftertheSYNCslotseveryuseracts
To model our energy harvesting system, we use the results independentlyoftheothers.
about decentralized control theory recently developed by Diban-
goyeetal.[25],[26],[27].Inparticular,[27]presentedadetailed solvedusingtheMarkovPolicySearchalgorithm[25].Becauseof
study of Decentralized-Partially Observable Markov Decision thecomplexityoftheoptimalapproach,weintroducetwosimpler
Processes (Dec-POMDPs) and proposed different approaches to schemes, which still exploit the structure of the optimal policy
solve them. The notion of occupancy state was introduced as a but can be computed in practice. In our numerical results we
fundamental building block for Dec-POMDPs, and it was shown comparecentralizedanddecentralizedapproaches,anddiscussthe
that, differently from classic statistical descriptions (e.g., belief performancelossofusingadecentralizedscheme.
states), it represents a sufficient statistic for control purposes.
Ourmaincontributionscanbesummarizedasfollows
Usingtheoccupancystate,wecanconverttheDec-POMDPinan
equivalentMDPwithacontinuousstatespace,namelyoccupancy- 1) We present a decentralized random access transmission
MDP. Then, standard techniques to solve POMDPs and MDPs scheme derived using a principle of optimality, and
can be applied; for example, an approach to solve a continuous discuss which are the computational pitfalls of this ap-
state space MDP is to define a grid of points (see Lovejoy’s grid proach;
approximation[28])andsolvetheMDPonlyinasubsetofstates. 2) Weintroducetwosuboptimalpolicies,whichareclosely
Althoughseveral papersintroduced moreadvanced techniquesto related to the structure of the optimal policy but can
refine the grid [29], this approach may still be inefficient and be numerically computed with reasonable complexity.
difficulttoapply.Instead,inthispaperweuseadifferentscheme, These can be used as a baseline for developing heuristic
namely the Learning Real Time A∗ (LRTA∗) algorithm [30], schemesandreal-timeprotocols.Moreover,althoughwe
whichhasthekeyadvantageofexploringonlythestateswhichare present these approaches for an EH scenario, they may
actuallyvisitedbytheprocess,withoutthedifficultyofdefininga alsobeusedinothercontexts;
gridofpoints. 3) We show that a decentralized scheme, if correctly de-
ConvertingtheDec-POMDPtoanoccupancy-MDPproduces signed,mayachievehighperformance,comparablewith
a simpler formulation of the problem, which however does not that of centralized solutions, while greatly reducing the
reduce its complexity. Indeed, for every occupancy state, it is signalinginthenetwork;
still required to perform the exhaustive backup operation, i.e., 4) Finally, our most important contribution is to show that,
to compute a decentralized control policy. This is the most differentlyfromtraditionalnetworks(i.e.,withoutenergy
critical operation in decentralized optimization, since it involves constraints), where orthogonal resource allocation is op-
solvinganon-convexproblemwithmanyvariables.Dibangoyeet timal,thebesttransmissionpolicywithenergyharvesting
al. proposed an alternate formulation of the exhaustive backup is a hybrid approach between random and orthogonal
operation as a Constraint Program [31], which can be solved, access.
e.g., using the bucket-elimination algorithm [32]. The problem
can be further simplified by imposing a predefined structure to The paper is organized as follows. Section 2 presents the
thepolicy[24],sothatonlyfewparametersneedtobeoptimized. system model. The internal layer is described in Section 3,
While this may lead to suboptimal solutions, it greatly simplifies whereastheexternallayerandtheoptimizationproblemareshown
thenumericalevaluationand,ifcorrectlydesigned,producesclose in Section 4. Optimal and suboptimal solutions are derived in
tooptimalresults.Inourpaper,weexploreandcompareboththese Sections 5 and 6, respectively. The numerical results are shown
directions. inSection7.Finally,Section8concludesthepaper.
ProblemStatementandContributions.Weconsideradecen- Notation. Throughout this paper, superscripts indicate the
tralized network with multiple devices and an access point that node indices, whereas subscripts are used for time indices. Bold-
computesanddistributestoallnodestherandomizedtransmission face letters indicate global quantities (i.e., vectors referred to all
policy. A multi-layer Markov model, in which an internal Dec- users),e.g.,e(cid:44)(cid:104)e1,...,eN(cid:105).
MDP is nested inside an external MDP, is proposed and solved.
2 SYSTEM MODEL
Theexternallayermodelsthetimeinstants,namelySYNCslots,at
whichAPcomputesthepolicy,whereastheinternallayermodels The network is composed of one Access Point (AP) and N
the system evolution between consecutive SYNC slots. To solve harvesting nodes (see Figure 1 for a graphical illustration). We
theexternallayer,weusetheValueIterationAlgorithm(see[33]) focus on an infinite time horizon framework, where a time slot
as in [19], [20]. However, differently from these papers, in our k corresponds to the time interval [kτ,(k+1)τ), k = 0,1,...,
modelthetransitionprobabilitiesbetweenstatesarederivedfrom whereτ isthecommondurationofallslots.Duringaslot,every
theoptimizationoftheinternallayer;moreover,thesojourntimes nodeindependentlydecideswhethertoaccesstheuplinkchannel
in every state are not deterministic. Instead, the internal layer is and transmit a message to AP, or to remain idle. We adopt an
3
on/offcollisionmodelinwhichoverlappingpackettransmissions slots k < k are of no interest in this section. Note that,
arealwaysirrecoverable. since β is a probability, the position of the next SYNC
k
In slot k, node i harvests energy from the environment ac- slotisunknownapriori,thereforethetimehorizonofthe
cording to a pdf B(k) (e.g., similarly to [34], [35], [36], in this Dec-MDPMbeginsatkandmayextendto+∞;
i
paper we will use a Bernoulli energy arrival process) and we • Battery Level. E = E1 ×···×EN is the set of global
assume independent arrivals among nodes. However, the model battery levels ek = (cid:104)e1k,...,eNk (cid:105), with eik ∈ Ei (cid:44)
canbeextendedtothemoregeneral,correlatedcase(e.g.,viaan {0,...,ei } (device i can store up to ei discrete
max max
underlyingcommonMarkovmodelasin[9]). energyquantaaccordingtoEquation(3)).Throughout,the
Every node is equipped with a rechargeable battery, so that terms “battery level” or “state” will be used interchange-
the energy stored in slot k can be used in a later slot. The global ably;
energy level vector in slot k is ek = (cid:104)e1k,...,eNk (cid:105). Vector ek is • Action. A = A1 × ··· × AN is the set of global
notknowntothedevices,whichonlyseetheirownbatterystatus, actions ak = (cid:104)a1k,...,aNk (cid:105), where aik ∈ Ai (cid:44) [0,1]
andisknowntoAPonlyincertainslots,namely“SYNCslots”.In denotes the transmission probability. Although ai should
k
particular, in slot k, with probability (w.p.) βk ∈ (0,1], AP may assumecontinuousvalues,weonlyconsiderSauniformly
trigger a SYNC slot and request all nodes to share their energy distributed samples of the interval [0,1] for numerical
levelssothatitcanacquiree .Weneglecttheenergycostsofthese tractability. Action ai is chosen by user i in slot k ≥ k
k k
synchronization messages, which however, if considered, would throughout a function σi : Ei → Ai, and depends only
k
make the benefits of using decentralized policies even higher. on the local state ei. Finding σi will be the objective of
k k
For example, when β = 1, ∀k, AP has full knowledge of the theoptimizationproblem;
k
batterylevelsateverytimeslot,andourmodeldegeneratesto[10]; • Transition Probability. pint is the probability transition
instead, if βk = β0, ∀k, where β0 is a constant value in (0,1], function pint : E ×A×E → [0,1] which defines the
APusesafullystochasticapproachandasksforek withthesame probability pint(e¯|e,a) of moving from a global battery
probabilityineveryslot. level e = (cid:104)e1,...,eN(cid:105) ∈ E to a global battery level
APusestheinformationaboute toinitializethetransmission e¯ = (cid:104)e¯1,...,e¯N(cid:105) ∈ E under the global action a ∈ A.
k
parameters of the whole network. Therefore, every time e is Whenatransmissionisperformed,mi ≥1energyquanta
k
acquired(aSYNCslot),acoordinationphaseisperformedandAP areconsumed;2
disseminatesthepolicytoallnodes(thepolicyisdecentralized,so • Reward. r is the reward function r : E ×A → R+ that
everynodereceivesonlyitsownpolicy).1Thereafter,everydevice mapstheglobalactiona∈Atotherewardr(e,a)when
actsindependentlyoftheothersuntilthenextSYNCslot. theglobalstateise∈E;
Although the proposed framework is very simple, modeling • Initial State Distribution. ηk is the initial state distribu-
andsolvingitformallyrequiresacomplexmathematicalstructure. tion.Inourscenariowetake
(cid:40)
Inparticular,wedecomposethesystemintwonestedlayers: 1, ife=e ,
η (e)= k (1)
k
• The external layer considers the jumps between consec- 0, ife∈E \{ek},
utive SYNC slots. Indeed, since the global battery level
wheree istheglobalstateincorrespondenceoftheinitial
k
e is completely known at every SYNC state, the system
k SYNCslotandisfullyknownbyAP;
followsaMarkovevolution;
• SYNC Probability. β represents the sequence βk,
• The internal layer models the actions to take between β ,..., which are the probabilities that a SYNC slot
k+1
two SYNC slots and requires to compute a decentralized
occurs.
policy given e . This will be modeled as a Dec-MDP,
k
In Section 4.1 we will describe the optimization problem
since multiple devices indirectly collaborate to achieve a
relatedtoM.Itssolutionprovidesadecentralizedcontrolpolicy,
commongoal.
whichwillbediscussedinSections5and6.
Thetwolayerswillbeanalyzedinthefollowingsections. Before presenting in more detail the previous bullet points, it
isimportanttoemphasizethefollowingkeycharacteristicsofthe
3 INTERNAL LAYER
Dec-MDPunderinvestigation:
Wefirstconsidertheinternallayerandpresentamathematicaltool
• Misjointlyfullyobservable,i.e.,ifallnodescollaborated
to model the actions of the devices between two SYNC slots. In
andsharedtheirlocalenergylevels,theglobalstatewould
particular, we adopt the decentralized-Markov Decision Process
be completely known (actually, this is what differentiates
(Dec-MDP) framework [26], which in our context is formally
Dec-MDPsfromDec-POMDPs[37]);
definedasfollows.
• M is a transition independent Dec-MDP, i.e., the action
3.1 Dececentralized–MDPsforEHSystems taken by node i influences only its own battery evolution
in that slot and not the others. Formally, the transition
An N-user Dec-MDP M = (k,E,A,p ,r,η ,β) is specified
int 0
probabilityfunctionp canbedecomposedas
by int
N
• InitialIndex.krepresentstheindexoftheSYNCslotthat p (e¯|e,a)= (cid:89)pi (e¯i|ei,ai). (2)
int int
triggers the beginning of the internal layer. Thus, all the
i=1
1.We note that, although a user may also receive the policy of other 2.Quantitymiisassociatedwiththephysicalparametersofthedevices,and
devices, this information would not be useful. Indeed, the decentralized relatestheamountofharvestedenergywiththeenergyusedfortransmission.
transmissionpolicyisjointlydesignedbyAP,thereforeitimplicitlyconsiders Therefore, to transmit a single packet, in general, more than one energy
thecontributionsofallnodes. quantummayberequired.
4
This feature is important to develop compact represen- On average, the reward of user i in a single slot when the
tations of the transmission policies, and in particular to batterylevelisei willbe
deriveMarkovianpoliciesasdiscussedinourSection3.5 (cid:90) ∞
andin[25,Theorem1]. g(νi (ei))(cid:44)E[χ(Vi ≥νi (ei))Vi]= νfi (ν)dν,
th th V
νi (ei)
th
3.2 BatteryLevel (5)
We adopt a discrete model for the energy-related quantities, so where χ(·) is the indicator function and fi (·) is the pdf of
V
that every battery can be referred to as an energy queue, in the potential reward, Vi. It is now clear that the transmission
which arrivals coincide with the energy harvesting process, and probabilityai isinherentlydependentonthebatterylevelas
departures with packet transmissions. In particular, the battery
(cid:90) ∞
levelofnodeiinslotkiseik andevolvesas ai =σi(ei)= fVi (v)dv =F¯Vi(νtih(ei)), (6)
νi (ei)
ei =min{ei ,ei −si +bi}, (3) th
k+1 max k k k
where we explicitly introduced the function σi(ei), which maps
wheretheminaccountsforthefinitebatterysize,sik istheenergy localobservations(ei)tolocalactionsσi(ei)=ai.Notethatthe
euqseudalftoor0trawn.spm.1iss−ioania,nadndbiktoismthiewe.pn.eragiy.arrivedinslotk.sik is complementary cumulative distribution function F¯Vi(·) is strictly
k k decreasingandthuscanbeinverted.Therefore,thereexistsaone-
Note that this model has been widely used in the EH litera- to-onemappingbetweenthethresholdvaluesandthetransmission
ture [20], [23], [35], and represents a good approximation of a probabilities.Inthefollowing,wewillalwaysdealwithaiinstead
realbatterywheneimax issufficientlyhigh. ofνi (·),andwriteg(ai)withaslightabuseofnotation.
th
3.3 Action It can be proved that g(ai) is increasing and concave in ai,
i.e., transmitting more often leads to higher rewards, but with
Node i in slot k can decide to access the channel w.p. ai =
k diminishing returns. Finally, note that this model is quite general
σi(ei), or to remain idle w.p. 1−ai. When a transmission is
k k k and,dependingonthemeaningofVi,canbeadaptedtodifferent
performed, mi energy quanta are drained from the battery, and
scenarios. For example, in a standard communication system in
a corresponding reward g(ai) is obtained. When ei < mi, no
k k which the goal is the throughput maximization, Vi can be inter-
transmissioncanbeperformedandai =0.
k pretedasthetransmissionratesubjecttofadingfluctuations[34].
3.4 TransitionProbability Global Reward. The global reward is zero when multiple
The transition probability function of user i, namely pi (see nodestransmitsimultaneously,whereasitisequaltowiνki ifonly
int nodeitransmitsinslotk(wiistheweightofnodei).Onaverage,
Equation (2)), is defined as follows (for presentation simplicity,
assumee¯i <ei andei ≥mi) sincethepotentialrewardsareindependentamongnodes,wehave
max
(cid:20) N
piint(e¯i|ei,ai) r(ν (e ))=E (cid:88)wiViχ(Vi ≥νi (ei))
th,k k k k th,k k
=((11−−ppiBiB))a(1i,−ai)+piBai, iiffee¯¯ii ==eeii−−mmii,+1, (4) ×(cid:89)i=1χ(Vkj <νtjh,k(ejk))(cid:21), (7)
p0,iB(1−ai), iofthe¯eir=wiseei,+1, j(cid:54)=i
whichcanberewrittenas
wherepi istheprobabilitythatuseriharvestsoneenergyquan- N
tum. MoBre sophisticated models, in which an arbitrary number r(ak)=r(σk(ek))=(cid:88)wig(aik)(cid:89)(1−ajk), (8)
of energy quanta can be simultaneously extracted, are described i=1 j(cid:54)=i
in[18],andcanbeintegratedintoourmodel(involving,however, where we used a instead of ν (e ) for ease of notation,
k th,k k
highercomputationalcosts). and we introduced the vector function σ (cid:44) (cid:104)σ1,...,σN(cid:105). We
k k k
3.5 Reward highlightthatσk summarizestheactionsofallusersgivenevery
batterylevelinslotk,i.e.,itspecifiesallthefollowingquantities
We will use the term “global reward” to indicate the overall
performance of the system in a slot, and simply “single-user σ1(0) ... σ1(e1 ),
k k max
reward”torefertotheperformanceofindividualusers. .. (9)
.
Single-User Reward. Assume to study isolated users, which
σN(0) ... σN(eN ).
do not suffer from interference, as in [9]. Data messages are k k max
associatedwithapotentialreward,describedbyarandomvariable Finding σ ,σ ,... represents the biggest challenge when
k k+1
Vi whichevolvesindependentlyovertimeandamongnodes.The solvingaDec-MDP.
realizationνi isperfectlyknownonlyatatimet≥kτ andonly An important observation is that the reward (8) is not neces-
k
to node i; for t < kτ, only a statistical knowledge is available. sarilyincreasingnorconvexina,whichsignificantlycomplicates
Everynodecandecidetotransmit(andaccruethepotentialreward the solution. An example of r(a) for the two-user case can be
νi) or not in the current slot k according to its value νi. In seen in Figure 2. Note that the maximum is achieved when only
k k
particular, it can be shown that a threshold transmission model one device transmits w.p. 1 and the other does not transmit. This
isoptimalforthissystem[9];thus,nodeialwaystransmitswhen impliesthat,whenthedevicesarenotenergyconstrained(i.e.,they
νi ≥ νi (ei) and does not otherwise. Note that νi (ei) depends haveenoughenergyfortransmittingandthecurrenttransmission
k th th
ontheunderlyingstate(batterylevel)ofuseributnotonthetime policy does not influence the future), the optimal user allocation
indexk(thusastationaryschedulercanbedeveloped). should follow an orthogonal approach so as to avoid collisions
5
state space of the occupancy-MDP is the occupancy simplex, the
transition rule is given by (11), the action space is A, and the
instantaneousrewardcorrespondingtodecentralizeddecisionrule
σ is
k
(cid:88)
ρ(η ,σ )= η (e¯)r(σ (e¯)), (12)
k k k k
e¯∈E
Notethatρ(η ,σ ) ≤ max r(σ (e¯)),i.e.,thelossofinforma-
Figure2:Globalrewardr(a)whenN =2. k k e¯ k
tioncorrespondstoalowerreward.Moreover,notethatifk were
(the corner points (cid:104)a1,a2(cid:105) = (cid:104)1,0(cid:105) and (cid:104)a1,a2(cid:105) = (cid:104)0,1(cid:105) aSYNCslot,wewouldhaveρ(ηk,σk)=r(σk(ek)).
achievethemaximumreward).However,aswewilldiscusslater, The complete structure of the occupancy-MDP will be given
this observation does not hold in EH scenarios, in which an inSection5.1.
action in the current slot influences the future energy levels and,
4 EXTERNAL LAYER AND OPTIMIZATION PROB-
consequently,thefuturerewards.
LEM
Note that, in the previous expressions, we have implicitly
restricted our study to Markovian policies, which map local So far, we have described how the system evolves between two
observations to local actions (i.e., σk(ek) = ak). In general SYNCslots.WenowintroducetheexternalMarkovChain,which
decentralized frameworks, tracking previous observations can be modelsthelong-termevolutionofthenetworkbyconsideringthe
used to optimally decide the current action (i.e., ak depends on subsetofallslotscomposedonlybytheSYNCslots.
ek,...,ek).However,itcanbeproved[25]thatundertransition Assume that, without loss of generality, the first SYNC slot
independentconditions(whichholdinourcase,seeSection3.1), occursatk =0,andthatthestateofthesystemise .According
0
Markovianpoliciesareoptimalandthuskeepingtrackofprevious toSection3,APusese tocomputeanddistributetoallnodesa
0
statesisnotnecessary. decentralized policy σ ,σ ,.... Moreover, the initial occupancy
0 1
Finally,wenotethatusingastationaryschedulerissuboptimal stateη isdefinedin(1),whereastheoccupancystatesη ,η ,...
0 1 2
inthemulti-usercase(thiswillbeveryclearfromournumerical areevaluatedasinSection3.6.
evaluation, e.g., see Figure 4); that is, even if we had eik(cid:48) = eik(cid:48)(cid:48) Assume now that the first SYNC slot after k = 0 is slot k(cid:48);
for k(cid:48) (cid:54)= k(cid:48)(cid:48) (same energy level in two different slots), using thus,whenthenewSYNCslotoccurs,weknowthatthetransition
σki(cid:48)(eik(cid:48)) = σki(cid:48)(cid:48)(eik(cid:48)(cid:48)) (same policy in two different slots) would probabilityfromtheinitialglobalstatee0tothefinalglobalstatee¯
not be optimal. Unfortunately, this implies that the decentralized isηk(cid:48)(e¯)(i.e.,itisgivenbytheoccupancystatebyitsdefinition),
optimization is much more challenging than the centralized one, withη =η =χ{e=e }.
k 0 0
sincetheoptimalpolicyshouldbecomputedforeveryslotk ≥k.3 Using ηk(cid:48)(e¯), we can compute the probability of going from
3.6 OccupancyState e0 to e¯. Since k(cid:48) is a random quantity (β represents a sequence
of probabilities), we provide the expression of the probability
Before formulating the optimization problem in the next section,
averaged over k(cid:48) (this models the jumps of the external MC
wefirstintroducetheconceptofoccupancystate.
betweenSYNCstates)
Theoccupancystateη isdefinedas
k
∞ (cid:18)k(cid:48)−1 (cid:19)
η (e¯)(cid:44)P(e =e¯|η ,σ ,...,σ ), k ≥k (10) (cid:88) (cid:89)
k k k k k−1 pext(e¯|e0)= βk(cid:48) (1−βk(cid:48)(cid:48)) ηk(cid:48)(e¯). (13)
and represents a probability distribution over the battery levels k(cid:48)=1 k(cid:48)(cid:48)=1
given the initial distribution ηk (introduced in Equation (1)) and In the previous expression, k(cid:48) represents the index > 0 of the
alldecentralizeddecisionrulespriortok. first SYNC slot; βk(cid:48) is the probability that k(cid:48) is a SYNC slot;
statiIstticcanfobrecsohnotwronlthpautrpthoeseosccinupDaneccy-MstDatPesr,e4praensdenctasnabsueffiecaiseinlyt theproduct(cid:81)kk(cid:48)(cid:48)(cid:48)−=11(1−βk(cid:48)(cid:48))istheprobabilitythatnoslotsprior
updatedateveryslotusingoldoccupancystates(k>k): to k(cid:48) are SYNC slots; ηk(cid:48)(e¯) is the transition probability, which
implicitly depends on e . We remark that, to evaluate (13), we
0
ηk(e¯)=ω(ηk−1,σk−1)=(cid:88)p(e¯|e,σk−1(e))ηk−1(e), need the sequence σ0,σ1,... in order to compute all the future
occupancy states η ,η ,.... In Section 4.1, we will specify how
e 1 2
(11) todefineσ ,σ ,....
0 1
whereω istheoccupancyupdatefunction. Observation 1. ThesequenceofSYNCslotssatisfiestheMarkov
Occupancy-MDP.Dibangoyeetal.[27]developedatechnique property,i.e.,ifk(cid:48) isaSYNCslot,thesystemevolutionfork ≥k(cid:48)
to solve Dec-MDPs by recasting them in equivalent continuous isconditionallyindependentofpaststatesgivene .
k(cid:48)
state MDPs. Similarly to the reduction techniques of POMDPs,
Wenowformallydefinetheoptimizationproblemandlinkthe
in which the belief is used as the state in anequivalent MDP, for
internalandexternallayers.
Dec-MDPs the occupancy state will represent the building block
oftheequivalentMDP(calledoccupancy-MDP).Inparticular,the 4.1 OptimizationProblem
3.Intheory,thepolicyshouldbecomputedforaninfinitenumberofslots. Define ρk|k (cid:44) ρ(ηk,σk) as the decentralized reward given the
However, as will be explained in Equation (19), when k (cid:29) k the rewards SYNC slot k. Then, prior to the next SYNC slot, the reward
obtainedwillbeverysmall;therefore,itwillnotbenecessarytocomputethe
of the system will be: ρ w.p. β , ρ + ρ w.p.
policyoptimallyforeveryk≥k. k|k k+1 k|k k+1|k
(1−β )β ,ρ +ρ +ρ w.p.(1−β )(1−
4.Intuitively, the occupancy state replaces the state of the system in k+1 k+2 k|k k+1|k k+2|k k+1
centralizedMDPs,orthebeliefinPOMDPs. βk+2)βk+3, and so forth. Summing together the previous terms,
6
and taking the average over the energy harvesting processes, we 5 OPTIMAL SOLUTION OF THE INTERNAL LAYER
obtainthenormalizedaveragereward
In the previous section we discussed the external optimization
(cid:20) ∞ ∞ k(cid:48)−1 (cid:21) problem and its solution via the value iteration algorithm. How-
(cid:88) (cid:88) (cid:89)
Rk (cid:44)E ρk|k βk(cid:48) (1−βk(cid:48)(cid:48)) . (14) ever, every iteration of VIA requires to solve the max in Equa-
k=k k(cid:48)=k+1 k(cid:48)(cid:48)=k+1 tion(18).ThisisequivalenttosolvingtheDec-MDPoftheinternal
layer,since(18)dependsonthedecentralizedpolicysequenceπ .
The final goal of the system is to maximize the cumulative e
weightedundiscountedlong-termreward,5 definedas In this section, we discuss how to do that optimally, whereas in
Section6wediscusssuboptimalsolutions.
1 K(cid:88)−1
G(Π,e )=liminf β ·R (π ,e ). (15) 5.1 BellmanEquation
0 K→∞ K k k k k
k=0 TheBellmanEquation(18)canberewrittenusingthedefinitions
Rk(πk,ek)isgivenin(14)whentheinitialstateofthesystemis ofpext(e¯|e0)(Equation(13))andR0(·)(Equation(14)):
e and a policy π (cid:44) (σπk,σπk ,...) is employed. Policy π
iskdecentralized, ankd it iskdrawnk+fr1om Π, which includes all thke max(cid:26)β E(cid:20)(cid:88)T φ (η ,σπe)(cid:12)(cid:12)e(cid:21)(cid:27), (19)
decentralizedtransmissionpoliciesπk,πk+1,.... πe 0 k=0 k k k (cid:12)
Since the sequence β is a design parameter and its choice is (cid:16) (cid:88) (cid:17)
φ (η ,σ)(cid:44)(1−β )k ρ(η ,σ)+ η (e¯)ziter−1(e¯) .
arbitrary,werestrictourattentiontothefollowingcase. k k 0 k k+1
e¯∈E
Assumption 1. The SYNC probability sequence is periodic with (20)
periodq (i.e.,β =β , ∀k).
k k+q Equation(19)representstheoccupancy-MDPunderinvestigation.
For example, the simplest case is q = 1, and βk = β0 for every For ease of notation, we used ηk+1 (cid:44) ω(ηk,σ). Note that
k.UnderAssumption1,itcanbeshownthat(15)isequivalentto (1 − β0)k decreases with k, whereas all the other terms are
bounded. Consequently, φ (·) decreases with k; thus, for large
q−1 k
(cid:88) (cid:88) k,itscontributionwillbenegligible.Becauseofthis,weapprox-
G(Π)= β ·R (π ,e)·ssp (e), (16)
k k e k imated the infinite sum with a finite sum from 0 to T, where T
k=0 e∈E
is a sufficiently large natural number. We now specify how the
wheresspk(e)isthesteady-stateprobabilityoftheglobalenergy system behaves between 0 and T, whereas the policy for k > T
level e associated with βk, and, instead of iterating over all k, can be arbitrarily chosen without degrading the performance of
we take the sum over the energy levels (i.e., we iterate over the thesystem.
states of the external MC). Note that, in this case, the long-term Wecansolvethemaxbyrewritingitinarecursiveform:
undiscounted reward does not depend on the initial state of the
(cid:40)
system, therefore G(Π,e0) = G(Π) for every e0. The optimal v (η )=max φk(ηk,σ)+vk+1(ω(ηk,σ)), ifk <T,
solutionoftheexternalproblemwillbe k k σ φT(ηT,σ), ifk =T,
Π(cid:63) =arg maxG(Π), (17) (21)
Π
wherev (·)isthecost-to-gofunction.Equation(19)isequivalent
k
which is a Markov Decision Process (MDP). The underlying
toβ ·v (η ).Thetrivialsolutiontofindπ istoapplyVIAinthe
0 0 0 e
MC states are all the elements of E, whereas the actions, which
finitehorizon;however,thiswouldrequire,foreveryk,tospecify
influencethetransitionprobabilities(Equation(13)),aregivenby
v (η )foreveryη ,whichisimpossibleinpractice.
k k k
the evolution of the internal Dec-MDP. In the following, for the
Analternatesolutionistousetechniquesoriginallydeveloped
sake of presentation simplicity, we impose β = β , ∀k (i.e.,
k 0 for POMDPs which were later used for Dec-POMDPs. In partic-
q =1).However,theresultscanbestraightforwardlyextendedto ular, the Learning Real Time A∗ (LRTA∗) algorithm is suitable
themoregeneralcase.
for our case, since it explores only the occupancy states which
Value Iteration Algorithm (VIA). The optimization problem
are actually visited during the planning horizon and avoids grid-
of Equation (17) can be solved using VIA [33, Vol. 1, Sec. 7.4].
based approaches (e.g., as used in [28]). In [25], the Markov
Sincewefocusonq =1,thankstoEquation(16)weonlyexamine
Policy Search (MPS) algorithm was introduced as an adaptation
k =0(inthemoregeneralcaseq >1,theprocedureisanalogous ofLRTA∗ todecentralizedscenarios.
but with q different equations). The Bellman equation (see [33])
Insummary,MPSoperatesasfollows
toiterativelysolveis
(cid:110) (cid:88) (cid:111) 1) It starts at k = 0 and, for every k ≥ 0, it computes the
ziter(e)←max β0R0(πe,e)+ pext(e¯|e)ziter−1(e¯) , LHSof(21)withLRTA∗,i.e.,themaximizationproblem
πe
e¯∈E issolvedonlyfortheoccupancystateswhichareactually
(18)
visitedandnotforeveryη ;
k
where “iter” is the index of VIA; β0R0(πe,e) represents the 2) Itreplacesvk+1 intheRHSwithanupperbound,which
initialreward,whereastheothertermistheexpectedfuturereward can be computed using the convexity of the cost-to-go
(this is derived from the old values of the Bellman equation, function.InSection5.2wewillfurtherdiscussthispoint;
ziter−1(·)). After a number of iterations (typically only a few), 3) When k = T is reached, a lower bound of the optimal
VIA converges, and the Bellman equations yield the optimal cost-to-go function is evaluated in a backward direction
solutionG(Π(cid:63)). (see[25,Sec.5.1]).
The procedure is repeated until upper and lower bounds
5.WeconsidertheundiscountedthroughputinthelongrunbecauseWSNs
typicallyreachthesteady-stateconditions.Theweightsareusedforfairness. converge to the optimal solution. We refer the readers to [25],
7
[26] for a full description of the algorithm. In the following, we
discuss how to find the upper bound of the cost-to-go function,
whichwillbeusedasabuildingblockinSection5.3.
5.2 UpperBoundoftheCost-to-goFunction
Itcanbeshownbyinductionthattheoptimalcost-to-gofunction
v(cid:63) is a convex function of the occupancy states and can be
k
approximated by piecewise linear functions [26, Theorem 4.2].
Theupperboundv¯ ofv(cid:63) canbewrittenas
k k
v¯ (η )=max{φ (η ,σ)+C(Υ ,ω(η ,σ))}, (22)
k k k k k k
σ
Figure3:Sawtoothprojectioninthetwo-dimensionalcase.Thearrowidenti-
where C interpolates the occupancy state ω(ηk,σ) using the fiesthequantitymax(cid:96)∈L{(y0(η(cid:96))−v(cid:96))ξ(cid:96)}whenη=(cid:104)0.5,0.5(cid:105).
point set Υ , which contains the visited occupancy states along
k
with their upper bound values. Every time (22) is solved, a (=d)minswt(cid:96)(Υ ,η). (27)
k
new point (η ,v¯ (η )) is added to Υ . The first points to be (cid:96)∈L
k k k k
put in Υ are the corners of the occupancy simplex (i.e., the Step(a)coincideswithDefinition(24);step(b)holdsbecausev(cid:96)
k
|E|points[1,0,...,0],...,[0,...,0,1])withtheirupperbound is not greater than y0(η(cid:96)), since y0(η(cid:96)) represents the interpola-
valuesobtainedsolvingthefollowingfullknowledgeMDP: tion using only the corner points (see Figure 3); in step (c), we
movey0(η)insidethemin ,sinceitdoesnotdependon(cid:96);finally,
R0(πe,e)(cid:44)E(cid:20)(cid:88)∞ r(σkπe(ek))(1−β0)k(cid:12)(cid:12)(cid:12)e(cid:21), (23) wWeedaelsfioneinstwrotd(cid:96)(uΥcekd,ηsw)ti(cid:96)n(Υs(cid:96)tep,η(d),).which will be used in the next
k=0 k
subsection, as the sawtooth projection obtained using the (cid:96)-th
whichisequivalentto(14)butwithr(·)insteadofρ andwith
k|0 elementofΥ .
q = 1. Expression (23) implicitly assumes that the state of the k
The sawtooth projection produces higher (i.e., worse) upper
systeme isgloballyknowninslotk,i.e.,acentralized-oriented
k bounds than the convex hull projection and thus MPS may
network.SincethisisastandardMDP,itcanbeeasilysolvedwith
require more iterations to converge (however, a single iteration
VIA.
can be performed much more quickly), but convergence is still
Sawtooth Projection. Ideally, we could use a linear interpo-
guaranteed[27].
lation as the function C (i.e., map η on the convex hull of
k
point set Υ ), but this would incur high complexity. A faster 5.3 ConstraintProgrammingFormulation
k
approach, which however has shown good performance in many The key step to perform to find the policy is solving the max
applications,istoreplaceC withthesawtoothprojection:6 in (22). Although this would be possible by performing the
swt(Υ ,η)=y0(η)−max{(y0(η(cid:96))−v(cid:96))ξ(cid:96)}, (24) exhaustive backup, i.e., by inspecting all possible choices of σ,
k
(cid:96)∈L it is more practical to introduce faster solutions. Constraint Pro-
where η is the occupancy state to interpolate, (η(cid:96),v(cid:96)) is the (cid:96)-th gramming[31]isatechniquetoexpresshardandsoftconstraints
elementofΥ ,ListhesetofindicesofΥ ,ξ(cid:96)istheinterpolation as an optimization problem. We now use it to reformulate (22)
k k
coefficient, and y0(·) is the upper bound computed using the withC =swt.
cornerpointsofΥ ,i.e., UsingthenotationofDibangoyeetal.[27],wedefine
k
y0(η)= (cid:88)η(e)Υ (e). (25) Wk(ηk,σ,(cid:96))(cid:44)φk(ηk,σ)+swt(cid:96)(Υk,ω(ηk,σ)). (28)
k
e∈E Combining(27)and(28),wecanrewriteEquation(22)as
Inthepreviousexpression,withaslightabuseofnotation,Υk(e) v¯k(ηk)=max{φk(ηk,σ)+swt(Υk,,ω(ηk,σ))}
indicates the upper bound value at the corner e of the simplex. σ
Theinterpolationcoefficientisdefinedas =max{φk(ηk,σ)+minswt(cid:96)(Υk,ω(ηk,σ))}
σ (cid:96)∈L
η(e) =maxminW (η ,σ,(cid:96)). (29)
ξ(cid:96) (cid:44) min , (26) σ (cid:96)∈L k k
e:η(cid:96)(e)>0η(cid:96)(e)
To solve the previous equation, we split it in |L| separate
andcanbederivedgeometrically(seeFigure3).Notethatweuse
mixed-integerprograms:
themaxin(24)soastoobtainthelowest(i.e.,best)upperbound.
Wenowrewritethesawtoothprojectioninasimplerform: v¯k(ηk)=maxv¯k(cid:96)(ηk), (30)
(cid:96)∈L
swt(Υ ,η)(=a)y0(η)+min(cid:110)(v(cid:96)−y0(η(cid:96))) min η(e) (cid:111) with
k (cid:96)∈L e:η(cid:96)(e)>0η(cid:96)(e) v¯(cid:96)(η )(cid:44)maxW (η ,σ,(cid:96)), (31a)
(=b)y0(η)+min max (cid:110) η(e) (v(cid:96)−y0(η(cid:96)))(cid:111) k k σ k k
(cid:96)∈Le:η(cid:96)(e)>0 η(cid:96)(e) subjectto: W(ηk,σ,(cid:96))≤Wk(ηk,σ,l), ∀l∈L. (31b)
(=c)min(cid:110)y0(η)+ max (cid:110) η(e) (v(cid:96)−y0(η(cid:96)))(cid:111)(cid:111) Weighted Constraint Satisfaction Problem (WCSP). Focus
(cid:96)∈L e:η(cid:96)(e)>0 η(cid:96)(e) now on the optimization maxσW(ηk,σ,(cid:96)) without any con-
straints. This can be formulated as a WCSP as follows. First, we
6.Theterm“sawtooth”comesfromtheshapeoftheinterpolatingfunction rewriteW(η ,σ,(cid:96))using(12),(20),(25),(27),and(28):
k
in the two-dimensional case (see Figure 3). The idea of the approach is to
(cid:88)
interpolateapointηusing|E|−1cornerpointsofthesimplex,andonepoint W (η ,σ,(cid:96))=(1−β )k η (e)r(σ(e)) (32)
k k 0 k
takenfromΥk((cid:96)inEquation(24)). e∈E
8
(cid:88)
+(1−β )k η (e(cid:48))ziter−1(e(cid:48)) the definition of the WCSP. If a constraint were violated, then
0 k+1
e(cid:48)∈E the solution of the WCSP would be infeasible and should be
+(cid:88)η (e(cid:48))Υ (e(cid:48))+ max (cid:110)ηk+1(e(cid:48)(cid:48))(v(cid:96)−y0(η(cid:96)))(cid:111), discarded. In this case, Dibangoye et al. (see [26, Section 3.4.2])
k+1 k e(cid:48)(cid:48):η(cid:96)(e(cid:48)(cid:48))>0 η(cid:96)(e(cid:48)(cid:48)) proposetoformulateanewWCSPinwhichthepreviousinfeasible
e(cid:48)∈E solutionresultsinaveryhighcostconstraint(andthusitisnever
whereη =ω(η ,σ),andweusede,e(cid:48)ande(cid:48)(cid:48)todifferentiate choseninthesolutionprocess).Ifthenewsolutionalsoturnsout
k+1 k
the indices. Note that, since (cid:96) is given, the term y0(η(cid:96)) is fixed. tobeinfeasible,theprocedureisrepeated.Theiterationsstopifa
UsingtheoccupancyupdateformulaofEquation(11),weget feasiblesolutionisfoundorifalldecentralizedactionshavebeen
(cid:88) examined.
W(η ,σ,(cid:96))=(1−β )k η (e)r(σ(e)) (33)
k 0 k Although the previous method formally leads to the opti-
e∈E mal solution, it may often degenerate in an exhaustive search
(cid:88) (cid:88)
+(1−β0)k ηk(e)p(e(cid:48)|e,σ(e))ziter−1(e(cid:48)) (i.e, examining all the decentralized policies). The corresponding
e(cid:48)∈Ee∈E complexity would be O((Sa)emax×N) if all users had the same
+ (cid:88) (cid:88)ηk(e)p(e(cid:48)|e,σ(e))Υk(e(cid:48)) batterysizeemax (seethestructureofσ(e)inEquation(9)),i.e.,
exponential in N. This operation is computationally infeasible
e(cid:48)∈Ee∈E
+ max (cid:110)(cid:88) ηk(e)p(e(cid:48)(cid:48)|e,σ(e))(v(cid:96)−y0(η(cid:96)))(cid:111). when lots of possibilities are involved. Thus, optimally solving
a Dec-MDP with guarantees on the worst case performance is
e(cid:48)(cid:48):η(cid:96)(e(cid:48)(cid:48))>0 η(cid:96)(e(cid:48)(cid:48))
e∈E stillanopenissue.Inthenextsection,weproposetwosuboptimal
Sincethefirsttermsof(33)donotdependone(cid:48)(cid:48),wemovethem approachesforhandlingtheproblembasedonthepreviousresults.
inside the maxe(cid:48)(cid:48) and take the common sum over e; then, we 6 SUB-OPTIMAL SOLUTIONS OF THE INTERNAL
notethatalltermsmultiplyη (e).Thus,byintroducingavariable
k LAYER
w (·),weobtain
(cid:96)
Since the main issue of the exhaustive search is that the space
(cid:88)
W(ηk,σ,(cid:96))= max ηk(e)w(cid:96)(e,σ(e),e(cid:48)(cid:48)). (34) of variables in Problem (30)-(31) is exceedingly large, we aim
e(cid:48)(cid:48):η(cid:96)(e(cid:48)(cid:48))>0
e∈E at reducing this space, so that σ cannot take all possible values
For every fixed (cid:96), the WCSP is formally defined as follows. but is constrained to lie in a smaller subset. The problem now
The variables are defined by σ as in (9) (i.e., the actions a ∈ A is to define such a subset. In the next subsection we present an
foreverye∈E)plustheindexe(cid:48)(cid:48).Thedomainsarethesameas approach based on WCSPs, whereas in Section 6.2 we introduce
in the original problem, i.e., A for σ and E for e(cid:48)(cid:48). A WCSP is adifferentschemebasedonparametricpolicies.
fullyspecifiedbyitsconstraints,whichareoftheform 6.1 WCSP-BasedPolicies
constraint (e)=M −η (e)w (e,σ(e),e(cid:48)(cid:48)). (35) In this case, we exploit the results about WCSPs presented in
(cid:96) k (cid:96)
Section5.3tofindasuboptimalpolicy.Theproposedalgorithmis
The total number of constraints is at most |E|, one for every
asfollows.
possible battery level. M is a large number used to cast the
WCSPinitsstandardform.StandardWCSPsolverscomputethe Algorithm1(SuboptimalpolicyusingWCSPs)
followingquantity:
1: for(cid:96)∈Ldo
min (cid:88)constraint (e), (36) 2: σ(cid:96) ←SolveWCSPfor(cid:96)givenηk
(cid:96)
σi(ei),∀ei,∀i,ande(cid:48)(cid:48)∈E 3: forl∈Ldo
k e∈E
4: EvaluateW(ηk,σ(cid:96),l)
whose solution is equal to the solution of (31a). In practice, for
every e ∈ E, the quantity constraint (e) is evaluated for all the 5: l(cid:63) ←argminl∈LW(ηk,σ(cid:96),l)
combinationsofσ(e)∈Aande(cid:48)(cid:48) ∈(cid:96)E independentlyoftheoth- 6: v¯k(cid:96)(ηk)←W(ηk,σ(cid:96),l(cid:63))
ersconstraints.Then,theyaresummedtogetherandtheminimum 7: (cid:96)(cid:63) ←max(cid:96)v¯k(cid:96)(ηk)
among all the solutions is chosen. However, a solution referred 8: returnσ(cid:96)(cid:63)
to a constraint constraint (e) may be related to other solutions.
(cid:96)
For example, different states e have some common entries (e.g., As required by the max in (30), we look at every (cid:96) ∈ L
both (cid:104)0,0(cid:105) ∈ E and (cid:104)0,1(cid:105) ∈ E have the first entry equal to 0); (Line 1) and, at the end of the algorithm, we return the solution
in this case, the corresponding actions must have some common withthemaximumvalue(Lines7-8).Lines2-6solveProblem(31)
element even if they are referring to different constraints (e.g., suboptimallyasdescribedinthefollowing.
σ((cid:104)0,0(cid:105)) = (cid:104)σ0(0),σ1(0)(cid:105) and σ((cid:104)0,1(cid:105)) = (cid:104)σ0(0),σ1(1)(cid:105) First, we solve the WCSP for a fixed (cid:96) (i.e., we solve (31a)),
musthavethefirstentryincommon). and find the corresponding solution σ . Then, using σ , we
(cid:96) (cid:96)
The main advantage of using a WCSP solver is that the evaluate W(η ,σ ,l) for every index l. Two cases should now
k (cid:96)
decentralizedpolicydoesnotneedtobecomputedasawhole,but beexamined,whichcanbehandledinafullyequivalentway,but
canbedividedinconstraintswhicharelatercombinedtogether. havedifferentmeanings:
So far, we have only focused on (31a) (i.e., on a single (cid:96)). • IfW(ηk,σ(cid:96),(cid:96)) ≤ W(ηk,σ(cid:96),l), ∀l ∈ L,thenσ(cid:96) would
To solve (30), we need to compute |L| different WCSPs (one be an optimal solution of (31), since it maximizes (31a)
for every (cid:96) ∈ L) and take the maximum among all solutions. (Line2)andsatisfies(31b).Inthiscase,(cid:96) ≡ l(cid:63) (Line5),
Howeverthisapproachpresentsonemajordrawback:forafixed(cid:96), andv¯(cid:96)(η )isthesolutionof(31);
k k
solvingasingleWCSPmaynotbesufficient.Indeed,thesolution • Instead,whenthereexistsl(cid:54)=(cid:96)suchthatW(ηk,σ(cid:96),l)<
must also satisfy (31b), which has been completely neglected in W(η ,σ ,(cid:96)), then σ is not optimal for index (cid:96). In
k (cid:96) (cid:96)
9
this case, the optimal approach would require the ex- a transmission m1 = m2 = 2 energy quanta are drawn from
ecution of a new WCSP discarding the previous solu- the battery; finally, β has period q = 1 (i.e., it is constant over
tion (in the new WCSP, σ would become a very high time). All the numerical evaluations were written in C++ and,
(cid:96)
cost solution). Instead, in Algorithm 1, we implicitly for solution of the weighted constraint satisfaction problems, we
make the following observation: σ is a feasible solution used ToulBar2 [38], a highly efficient solver of WCSPs. We first
(cid:96)
(i.e., it satisfies (31b)) of v¯l(cid:63)(η ), where l(cid:63) is such focus on the solution of the internal layer (i.e., we only look at
k k
that W(η ,σ ,l(cid:63)) ≤ W(η ,σ ,l), ∀l ∈ L (Line (5)). R (π ,e )),anddiscusslaterhowtheexternallayerperforms.
k (cid:96) k (cid:96) 0 e0 0
Therefore,solutionW(η ,σ ,l(cid:63))isfeasible;forsimplic- Transmission Probabilities. In Figures 4 (low energy arrival
k (cid:96)
ity, we improperly save its value in v¯(cid:96)(η ) (Line 6). By rates) and 5 (high energy arrival rates) we show the transmission
k k
doing so, at the end of Line 6, v¯(cid:96)(η ) does not represent probabilitiesoftheparametricdecentralizedpolicyofSection6.2,
k k
thesolutionof(31)associatedtoindex(cid:96),butitcontainsa where f is a linear function, Θi = {θi} and θi is such that
par
feasiblesolutionforsomeindexl(cid:63). θiei ∈ Ai. The dashed lines have been slightly manually
max
shiftedtotherightonlyforgraphicalpurposes.Inthesefigures,we
Inpractice,whileexecutingAlgorithm1,thespaceofvariables
onlyfocusontheinternallayer,thuswestudythesystembehavior
of Problem (30) is defined by the solutions of WCSPs at every
betweentwoconsecutiveSYNCslots.
iteration(thus,itisnotdeterminedapriori).
The main difference between Figures 4 and 5 is that, after
Theproposedapproachisfasterthantheoptimalone,sinceit
many slots, the transmission probabilities are both greater than
completelyavoidstheexhaustivesearch;however,ingeneral,itis
zero in the first case, whereas an almost pure time-orthogonal
suboptimalandthusachievesworseperformance.
approachisusedinthelatter,regardlessoftheinitialenergylevels.
6.2 ParametricPolicies Thus, when the energy resources are scarce (i.e., low energy ar-
rivals)then,ifkislarge,anorthogonalschemeinwhichcollisions
Another possibility to avoid the exhaustive search step is to use
are avoided is suboptimal. The trade-off between orthogonal and
parametric policies and thus reduce the number of optimization
randomaccessschemescanbeintuitivelyexplainedasfollows.As
variablestofewparameters.Inparticular,weforcetheactionsof
timegoeson,nodesloseinformationabouttheglobalstateofthe
useritofollowapredeterminedstructure:
system,thusadevicedoesnotknowtheenergyleveloftheother.
σi(ei)=fi (Θi,ei) (37) In this case, an orthogonal scheme might be highly inefficient:
par
if a slot were assigned only to Node 1, but this did not have
whereei istheindependentvariableandΘi isasetofparameters enough energy for transmission, then the slot would be unused.
which specify the structure of fpiar. For example, if we used Sincetheenergyresourcesarescarce,itislikelythatsuchacase
Θi = {θi}, and a simple linear function fpiar(Θi,ei) = θiei, happens. Instead, when both have a lot of energy, it is easier to
the only optimization variable of user i would be θi, and not estimatetheenergylevelofthenodes,andthuspre-assigningthe
σi(0),...,σi(eimax)asintheoriginalproblem.Inthiscase,fora slotsispossible(andinfactoptimal).Thismeansthat,inthelow
symmetric scenario, the complexity of the exhaustive search step energycase,itisbettertotransmitevenifthecollisionprobability
goes from O((Sa)emax×N) to O((Sθ)N), therefore it remains is non-negligible, to compensate the (likely) lack of energy of
exponential in N but with a much smaller coefficient in the the other user. This represents the main novelty of EH scenarios
exponent.Sθ isthenumberofvaluesthatθi canassume.7 over traditional ones in which, as explained in Section 3.5, an
In our scenario we force fpiar(Θi,ei) to be a non-decreasing orthogonaldecentralizedaccessschemeisoptimal.
function of ei as in [9], which implies that higher energy levels The same effect can be observed in Figure 4 when the initial
cannotcorrespondtolowertransmissionprobabilities. energy levels are high. Indeed, when k is small, nodes still have
information about the state of the other device; therefore, an
7 NUMERICAL EVALUATION
orthogonal approach is optimal in this case also, and collisions
The numerical evaluation is performed using two nodes, since are avoided since it is very likely that a node has enough energy
the complexity grows super-exponentially with the number of toperformthetransmission.
users.Indeedthesizeoftheoccupancystateevolvesexponentially Note that when an orthogonal access scheme is employed,
with N, and the exhaustive search operation (exponential in Node1isadvantagedwithrespecttoNode2(thiscanbeclearly
N), or a suboptimal approach, is to be performed for every seeninFigure5,wheremoreslotsareallocatedtothefirstnode).
element of the occupancy state. If not otherwise stated, we Indeed, the SNR of the first node (Λ1) is greater than the SNR
adopt the following parameters: the batteries can contain up to of the other (Λ2), and therefore higher returns are obtained by
e1 = e2 = 8 energy quanta; the energy arrival processes Node1whenatransmissionisperformed.Inthiscase,fairnessis
max max
are i.i.d. over time and the probability of receiving one energy not achieved because of the near-far effect (a node with a better
quantum is p1 = p2 in every slot; when a transmission is channelisadvantagedovertheother);however,thenetworkcould
B B
performed a reward Vi = ln(1+ΛiHi) is accrued, where Vi berebalancedbychangingtheweightsw1,w2.
representsthenormalizedtransmissionrateinaslot,andHi isan Finally, note that in Figure 4 the average transmission proba-
exponentiallydistributedrandomvariablewithmean1(see[34]); bilitiesinthelongrunalmostcoincidewiththeenergyarrivalrate
the transmission probabilities in [0,1] are uniformly quantized dividedbymi,soastoachieveenergyneutrality.
withSa =19samples;theaveragenormalizedSNRsareΛ1 =6 Energy Levels. Figure 6 shows another interesting, though
and Λ2 = 3; both devices have the same weight; to perform predictable,result:despitetheinitialenergylevel,inthelongrun
alltheenergylevelsofthesamedeviceconverge,approximately,
7.Wenotethat,althoughwedidnotreducethetheoreticalcomplexityofthe tothesamevalue.Thisisbecausealltheinitialfluctuationshave
exhaustivesearch(whichisstillexponential),usingsmallercoefficientsallows
been absorbed by the batteries. Note that the energy levels of
muchfasternumericalevaluations.Themoregeneralproblemofdeveloping
heuristicschemeswithlowercomplexityisstillopen. Node 2 are higher because Λ2 < Λ1, thus Node 2 transmits less
10
Figure4:Transmissionprobabilitiesasafunctionoftimefortwouserswhenβ0=0.05andp1B =p2B =0.1fordifferentinitialbatterylevels.
Figure5:Transmissionprobabilitiesasafunctionoftimefortwouserswhenβ0=0.05andp1B =p2B =0.9fordifferentinitialbatterylevels.
frequentlythantheotherandconsumeslessenergy,onaverage. since there is not a lot of energy to exploit. Therefore, most of
Internal Reward. In Figure 7 we show the long-term dis- the gain is obtained for higher k, which in turn leads to a less
counted reward as a function of the energy arrival rate for the informative situation about the state of the other device. In this
decentralized scheme solved using WCSPs (Section 6.1), the case, the centralized scheme may perform much better than the
decentralizedparametricscheme(Section6.2),afullyorthogonal decentralizedonebecauseofthislackofinformation.
approach,andafullysymmetricscheme.Thecurvesarenormal- Although it is not visible from Figure 7 because of the
ized with respect to the upper bound, given by the centralized normalization, the curves increase with p1 = p2, because
B B
scheme of Equation (23) (which, however cannot be achieved in more energy can be harvested (see [1, Section V]). The lowest
decentralizedscenarios,ingeneral).Tounderstandthetrendofthe normalizedrewardfortheWCSP-basedpolicyisobtainedaround
curves,itisimportanttoremarkthatthefirstslotsaftertheinitial p1 =p2 =0.5;indeed,thiscorrespondstotheleastinformative
B B
SYNCslotarethemostimportantonesfortworeasons: case,sincethebatteryfluctuationsarenotpredictableatall.
Finally, note that the decentralized policy obtained using
1) Thedecentralizedrewardφ (·)definedin(20)decreases
k
WCSP outperforms the parametric policy in almost every case.
with k. Therefore, the initial slots have higher weights
However, this is also strongly influenced by the number of pa-
andcontributemoretotheglobalreward;
rameters Θi we used, and using more parameters would lead to
2) There is more information about the state of the other
betterperformance.Moreover,itcanbeclearlyseenthatthefully
deviceintheinitialslots.
orthogonalandthesymmetricpoliciesarestronglysuboptimalin
Therefore, when the initial batteries are fully charged, the thisscenario,thereforeusinganoptimizedapproachsignificantly
centralized and decentralized schemes are closer. Indeed, since improves the throughput of the network. An interesting outcome
e = e , decentralized and centralized rewards in the first of Figure 7 is that the gap between centralized and decentralized
0 max
slots are almost equivalent. Instead, if the batteries are initially schemes may be small (< 15%), when the decentralized policy
fullydischarged,thegapbetweencentralizedanddecentralizedis is correctly designed. Therefore, using low values of β (e.g.,
wider. In this case, the first slots do not play a fundamental role, β = 0.05) we are able to achieve the twofold goal of greatly re-