Table Of ContentEMC CONFIDENTIAL
EMC® Avamar® 7.3
Technical Addendum
P/N 302-002-864
REV 01
EMC CONFIDENTIAL
Copyright © 2001- 2016 EMC Corporation. All rights reserved. Published in the USA.
Published April, 2016
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without
notice.
The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect
to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular
purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries.
All other trademarks used herein are the property of their respective owners.
For the most up-to-date regulatory document for your product line, go to the technical documentation and advisories section on the
EMC online support website.
2 EMC Avamar 7.3 Technical Addendum
EMC CONFIDENTIAL
CONTENTS
Preface
Chapter 1 Advanced Technical Information
Checkpoints................................................................................................ 24
Full checkpoints.................................................................................... 24
Valid checkpoints................................................................................. 24
Checkpoint verification............................................................................... 24
Important terms and concepts............................................................... 25
Operational overview............................................................................ 25
Common hfscheck error codes.............................................................. 26
HFS check throttling.............................................................................. 28
Types of checks performed.................................................................... 29
Automatic stripe repair.......................................................................... 30
Rolling HFS checks................................................................................ 33
Event throttling........................................................................................... 35
Retention types........................................................................................... 36
Server dynamic load balancing................................................................... 38
Storage file types........................................................................................ 39
Timezone notes.......................................................................................... 40
Maintenance window compatibility....................................................... 40
Consult the local site administrator....................................................... 40
Daylight savings time considerations.................................................... 40
Avoid local timezones that ignore local DST laws.................................. 40
Do not use GMT as local timezone for England...................................... 40
Select capital city or nearest large city for small countries..................... 41
Additional timezone resources.............................................................. 41
Chapter 2 Command Reference
Run locations.............................................................................................. 47
Boolean option behavior and syntax........................................................... 47
Wildcards.................................................................................................... 47
Numeric plug-in descriptors........................................................................ 48
5minute_cron_run...................................................................................... 51
10minute_cron_run.................................................................................... 52
ascd............................................................................................................ 53
asktime...................................................................................................... 54
avacl........................................................................................................... 60
avagent....................................................................................................... 61
avidbmaint.pl............................................................................................. 67
avinstaller.pl............................................................................................... 68
avldap........................................................................................................ 69
avlockboxcfg............................................................................................... 71
avmaint...................................................................................................... 73
avmaint access..................................................................................... 81
avmaint atrestencryption...................................................................... 82
avmaint autorestart............................................................................... 84
avmaint cancelremovebadhashes......................................................... 86
avmaint cat........................................................................................... 86
EMC Avamar 7.3 Technical Addendum 3
EMC CONFIDENTIAL
Contents
avmaint checklogs................................................................................ 87
avmaint checkpoint............................................................................... 88
avmaint config...................................................................................... 89
avmaint conversion............................................................................... 98
avmaint crunch..................................................................................... 99
avmaint decommission....................................................................... 101
avmaint ducp...................................................................................... 102
avmaint findbadhashes...................................................................... 102
avmaint garbagecollect....................................................................... 104
avmaint gcstatus................................................................................. 105
avmaint gendata................................................................................. 106
avmaint getclientmsgs........................................................................ 107
avmaint geterrors................................................................................ 108
avmaint getrefby................................................................................. 109
avmaint hfscheck................................................................................ 110
avmaint hfscheckstatus...................................................................... 114
avmaint hfscheckthrottle.................................................................... 117
avmaint infomessage.......................................................................... 119
avmaint kill......................................................................................... 120
avmaint logscan.................................................................................. 121
avmaint lookup................................................................................... 124
avmaint ls........................................................................................... 125
avmaint lscp....................................................................................... 126
avmaint networkconfig........................................................................ 129
avmaint nodelist................................................................................. 130
avmaint perf........................................................................................ 132
avmaint ping....................................................................................... 136
avmaint poolcheck.............................................................................. 137
avmaint rebuildstripe.......................................................................... 139
avmaint removebadhashes................................................................. 140
avmaint rmcp...................................................................................... 141
avmaint sched.................................................................................... 143
avmaint sessions................................................................................ 150
avmaint shutdown.............................................................................. 150
avmaint stats...................................................................................... 151
avmaint stripels.................................................................................. 153
avmaint suspend................................................................................ 154
avmaint test........................................................................................ 155
avmaint testintegrity........................................................................... 156
avmaint timesync................................................................................ 157
avmaint timing.................................................................................... 158
avmaint tracehash.............................................................................. 159
avmgr....................................................................................................... 160
avregister.................................................................................................. 171
avregister.bat............................................................................................ 171
avscc........................................................................................................ 172
avsetup_avamarbin.................................................................................. 174
avsetup_avi.pl.......................................................................................... 174
avsetup_connectemc.pl............................................................................ 174
avsetup_ems............................................................................................ 175
avsetup_mccli........................................................................................... 176
avsetup_mcs............................................................................................ 177
avsetup_mds............................................................................................ 180
avsetup_snmp.......................................................................................... 181
avsetup_webstart..................................................................................... 182
4 EMC Avamar 7.3 Technical Addendum
EMC CONFIDENTIAL
Contents
avsetup_wrapper...................................................................................... 182
avsysconfig............................................................................................... 183
avsysreport............................................................................................... 184
avtar......................................................................................................... 193
avtar.bin................................................................................................... 217
axion_install............................................................................................. 217
axionfs...................................................................................................... 220
backendreport.......................................................................................... 223
backup_upgrade_files............................................................................... 225
btfix.......................................................................................................... 227
capacity.sh............................................................................................... 228
change-passwords.................................................................................... 229
change_nodetype..................................................................................... 231
check.dpn................................................................................................. 232
check.mcs................................................................................................. 235
checklib.pm.............................................................................................. 235
clean_db.pl............................................................................................... 236
clean.dpn................................................................................................. 239
convert-probe........................................................................................... 240
copy-ata-drive........................................................................................... 241
copy-checkpoint........................................................................................ 243
cp_cron..................................................................................................... 245
cplist........................................................................................................ 246
cps........................................................................................................... 249
create_newconfigs.................................................................................... 250
cron_env_wrapper.................................................................................... 250
dbcreate_mds........................................................................................... 251
dbload.sh................................................................................................. 251
dbmaint.sh............................................................................................... 252
dbpurge.sh............................................................................................... 253
dbUpdateActivitiesExt.pl........................................................................... 255
ddrmgrcli.................................................................................................. 255
decommission.node................................................................................. 255
delete-snapups......................................................................................... 257
disktest.pl................................................................................................ 259
dpn.pm..................................................................................................... 260
dpncron.pm.............................................................................................. 260
dpnctl....................................................................................................... 260
dpnfsctl.................................................................................................... 270
dpnnetutil................................................................................................. 271
dpnsummary............................................................................................. 279
dpn-time-config........................................................................................ 282
dt and dtsh............................................................................................... 284
dump_accounts........................................................................................ 286
dumpmaintlogs........................................................................................ 287
emserver.sh.............................................................................................. 288
emwebapp.sh........................................................................................... 290
errchk....................................................................................................... 291
evening_cron_run..................................................................................... 292
expire-snapups......................................................................................... 293
gathergsankeydata................................................................................... 295
gcblitz....................................................................................................... 296
gc_cron..................................................................................................... 298
gc_report.................................................................................................. 300
gen-ssl-cert............................................................................................... 301
EMC Avamar 7.3 Technical Addendum 5
EMC CONFIDENTIAL
Contents
gethostbyname......................................................................................... 302
getlogs...................................................................................................... 303
getnodelogs.............................................................................................. 303
getsnapupstats......................................................................................... 304
health_check.pl........................................................................................ 306
hfscheck_cron........................................................................................... 308
hfscheck_kill............................................................................................. 312
hfsclean.................................................................................................... 314
hfssetup................................................................................................... 314
initacnt..................................................................................................... 315
initialize_connectemc............................................................................... 316
java_update.sh......................................................................................... 317
java_update_avi.sh................................................................................... 318
lm............................................................................................................. 318
load_accounts.......................................................................................... 320
logmrg...................................................................................................... 321
mapall...................................................................................................... 322
mcdbmaint.sh........................................................................................... 325
mcdbsql.sh............................................................................................... 326
mcconfigfirewall_snmp............................................................................. 327
mcddrcopy_sshkey................................................................................... 327
mcddrcreate_sshkey................................................................................. 327
mcddrsetup_sshkey.................................................................................. 328
mcddrsnmp.............................................................................................. 329
mcfeature................................................................................................. 330
mcflush.pl................................................................................................. 331
mcgui.bat................................................................................................. 332
mcgui.sh................................................................................................... 333
mcgui_login.bat........................................................................................ 334
mcserver.sh.............................................................................................. 336
mcsmon_run............................................................................................. 340
mcsnmp.................................................................................................... 340
mcsnmp_cron........................................................................................... 340
mds_ctl..................................................................................................... 341
metadata_cron.......................................................................................... 341
mktime..................................................................................................... 342
modify-snapups........................................................................................ 344
mondo...................................................................................................... 346
monmcs.................................................................................................... 346
monthly_cron_run..................................................................................... 347
morning_cron_run.................................................................................... 347
msgbrokerctl.pl......................................................................................... 348
nodedb..................................................................................................... 349
nodedb add if..................................................................................... 353
nodedb add node................................................................................ 354
nodedb create..................................................................................... 357
nodedb delete if.................................................................................. 358
nodedb delete module........................................................................ 358
nodedb delete nat............................................................................... 359
nodedb delete node............................................................................ 359
nodedb print....................................................................................... 360
nodedb update if................................................................................ 362
nodedb update module....................................................................... 364
nodedb update node........................................................................... 365
nodenumbers........................................................................................... 366
6 EMC Avamar 7.3 Technical Addendum
EMC CONFIDENTIAL
Contents
opstatus.dpn............................................................................................ 368
permctl..................................................................................................... 369
pingmcs.................................................................................................... 370
probe........................................................................................................ 370
probeaux.................................................................................................. 371
probedump............................................................................................... 372
probesingle............................................................................................... 373
propagate-gsan......................................................................................... 374
psregrep................................................................................................... 375
pull-checkpoint......................................................................................... 376
rebuild.node............................................................................................. 377
repl_cron.................................................................................................. 381
replicate................................................................................................... 381
resite........................................................................................................ 389
restart.dpn................................................................................................ 399
resume_crons........................................................................................... 403
rollback.dpn............................................................................................. 404
rununtil..................................................................................................... 405
sched.sh................................................................................................... 406
scn........................................................................................................... 408
securedelete............................................................................................. 411
showperfhistory........................................................................................ 413
shutdown.dpn.......................................................................................... 415
site_inventory........................................................................................... 417
ssn........................................................................................................... 420
start.dpn................................................................................................... 423
start.nodes............................................................................................... 430
stats.sh.................................................................................................... 432
status.dpn................................................................................................ 433
store-checkpoint....................................................................................... 435
stunctl...................................................................................................... 436
suspend_crons......................................................................................... 438
swraidctl................................................................................................... 439
timedist.................................................................................................... 440
timerange................................................................................................. 442
timesyncmon............................................................................................ 443
tomcatctl.................................................................................................. 445
transfer-key............................................................................................... 447
truncate.................................................................................................... 447
ugcheck.................................................................................................... 448
ugcopy...................................................................................................... 449
ugprep...................................................................................................... 450
wait.crunch............................................................................................... 455
wait.dpn................................................................................................... 457
website..................................................................................................... 458
zzdpn........................................................................................................ 460
Chapter 3 Environment Variables
AVAMAR_INSTALL_BASEDIR_PATH............................................................. 462
AVAMAR_INSTALL_VARDIR_PATH............................................................... 462
SYSNODEDB.............................................................................................. 462
SYSPROBEDIR........................................................................................... 462
SYSPROBEUSER........................................................................................ 463
EMC Avamar 7.3 Technical Addendum 7
EMC CONFIDENTIAL
Contents
Chapter 4 Important Files
.avamar.................................................................................................... 466
avagent.cfg............................................................................................... 466
AVAMAR-MCS-MIB.txt................................................................................ 466
avinstaller.xml.......................................................................................... 466
com.avamar.asn.module.datastore..................................................... 467
com.avamar.avi................................................................................... 467
com.avamar.avi.repo.......................................................................... 468
com.avamar.avigui.............................................................................. 468
com.avamar.mc.compatibility............................................................. 468
com.avamar.mc.dpn........................................................................... 468
com.avamar.mc.mcsvars..................................................................... 469
avw_start_dpn_options.txt....................................................................... 470
axionfs.cfg................................................................................................ 470
checkpoints.xml........................................................................................ 470
config_info................................................................................................ 471
Multi-node servers.............................................................................. 471
Single-node servers............................................................................ 471
Avamar NDMP Accelerator................................................................... 472
CAT servers......................................................................................... 472
Spare nodes........................................................................................ 472
emclient.xml............................................................................................. 473
emserver.xml............................................................................................ 474
com.avamar.asn................................................................................. 474
com.avamar.asn.module.datastore..................................................... 475
com.avamar.asn.module.mail............................................................. 475
com.avamar.asn.service...................................................................... 475
com.avamar.mc.................................................................................. 476
com.avamar.mc.ca.............................................................................. 476
com.avamar.mc.compatibility............................................................. 478
com.avamar.mc.dashboard................................................................. 478
com.avamar.mc.dpn........................................................................... 478
com.avamar.mc.event......................................................................... 479
com.avamar.mc.mcvars...................................................................... 479
/etc/hosts................................................................................................ 480
File format and syntax......................................................................... 480
Default /etc/hosts files....................................................................... 480
gsankeydata.xml....................................................................................... 481
gsan.log.................................................................................................... 481
license.xml............................................................................................... 482
Login manager configuration files............................................................. 483
logs.DATE.tar............................................................................................. 484
mccli.xml.................................................................................................. 484
com.avamar.mc.cli.............................................................................. 484
mcclient.xml............................................................................................. 485
com.avamar.mc.gui............................................................................. 485
mcclimcs.xml............................................................................................ 486
Default command parameters............................................................. 486
Profiles............................................................................................... 486
Behavior............................................................................................. 486
Default file settings............................................................................. 486
Practical examples.............................................................................. 488
mcserver.xml............................................................................................ 489
com.avamar.asn................................................................................. 489
com.avamar.asn.module.datastore..................................................... 490
8 EMC Avamar 7.3 Technical Addendum
EMC CONFIDENTIAL
Contents
com.avamar.asn.module.mail............................................................. 493
com.avamar.asn.service...................................................................... 494
com.avamar.mc.................................................................................. 494
com.avamar.mc.burm......................................................................... 495
com.avamar.mc.connectemc............................................................... 496
com.avamar.mc.cr............................................................................... 496
com.avamar.mc.datadomain............................................................... 498
com.avamar.mc.dpn........................................................................... 500
com.avamar.mc.dpn.users.................................................................. 503
com.avamar.mc.dtlt............................................................................ 503
com.avamar.mc.event......................................................................... 504
com.avamar.mc.ldap........................................................................... 505
com.avamar.mc.lm.............................................................................. 505
com.avamar.mc.mcsdk....................................................................... 506
com.avamar.mc.mcsm........................................................................ 506
com.avamar.mc.mcsvars..................................................................... 507
com.avamar.mc.mon........................................................................... 508
com.avamar.mc.repl........................................................................... 511
com.avamar.mc.rpt............................................................................. 511
com.avamar.mc.security..................................................................... 512
com.avamar.mc.um............................................................................. 512
com.avamar.mc.vmware..................................................................... 512
com.avamar.mc.wo............................................................................. 513
mktime.custom......................................................................................... 515
mktime.out............................................................................................... 515
ntp.conf*.................................................................................................. 515
probe.out.................................................................................................. 516
Reading a probe.out file...................................................................... 516
Parsing and naming............................................................................ 516
Physical versus logical node numbers................................................. 517
Manually creating a probe.out file....................................................... 518
probe.xml................................................................................................. 520
Elements and attributes used by the probe.xml file............................. 520
Processing precedence....................................................................... 522
Example script for creating probe.xml for a single-node server............ 523
Example probe.xml file for multi-node server....................................... 523
Example script for returning storage node addresses.......................... 524
repl_cron.cfg............................................................................................. 525
step-tickers*............................................................................................. 525
usersettings.cfg........................................................................................ 525
web.xml.................................................................................................... 525
Chapter 5 Important Directories
Multi-node server utility nodes.................................................................. 528
Multi-node server storage nodes............................................................... 528
Single-node servers.................................................................................. 528
Client........................................................................................................ 529
EMC Avamar 7.3 Technical Addendum 9
EMC CONFIDENTIAL
Contents
10 EMC Avamar 7.3 Technical Addendum
Description:documentation, advisories, and downloads, as well as how-to and troubleshooting information. This information may enable you to resolve a product issue before you contact EMC Customer Support. To access the Avamar support page: 1. Go to https://support.EMC.com/products. 2. Type a product name