Appendix C for Request for Correction 14002

337 Pages·2012·15.61 MB·English

For assistance in accessing this document, please contact [email protected].
APPENDIX C
EXHIBITS INCORPORATED BY REFERENCE INTO
INFORMATION QUALITY ACT REQUESTS FOR CORRECTION
TABLE OF CONTENTS
1. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of
Information Disseminated by Federal Agencies; Republication, 67 Fed. Reg. 8452
(February 22, 2002)
2. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of
Information Disseminated by the Environmental Protection Agency, EPA, EPA/260R-02-
008 October 2002 (Excerpts)
3. EPA website, available at http://www.epa.gov/IRIS/
4. Correspondence: February 21, 2012 and October 24, 2012 letters from Monica Jones,
Director, Quality Staff, Office of Environmental Information, EPA, to Gregory Dolan,
Executive Director-Americas/Europe, Methanol Institute (regarding IRIS Toxicological
Review of Methanol); and October 24, 2012 and June 8, 2011 letters from Monica Jones,
EPA, to Lynn Bergeson, Managing Director, Bergeson & Campbell, P.C. (regarding IRIS
Toxicological Review of Inorganic Arsenic)
5. “Review of the Environmental Protection Agency’s Draft IRIS Assessment of
Formaldehyde,” National Research Council of the National Academies, April 2011
(“NAS Formaldehyde Peer Review Report”) (Excerpts)
6. Toxicological Review of Libby Amphibole Asbestos, August 2011 External Review
Draft (Excerpts)
7. http://www.epa.gov/risk/health-risk.htm and
http://epa.gov/riskassessment/basicinformation.htm#arisk
8. December 16, 2013, A. Schmitt, U.S. Dept. of Justice, email
9. Treasury and General Government Appropriations Act for Fiscal Year 2001, Public Law
106–554, H.R. 5658, Section 515(a)
10. Testimony of EPA Administrator McCarthy Before the House Committee on Science,
Space, and Technology, November 14, 2013
11. Methods for Derivation of Inhalation Reference Concentrations and Application of
Inhalation Dosimetry, EPA, EPA/600/8-90/066F, October 1994 (Excerpts)
12. Risk Assessment Guidance for Superfund, Volume I: Human Health Evaluation Manual,
Part F (Supplemental Guidance for Inhalation Risk Assessment) Final,
EPA/540/R/070/002, January 2009 (Excerpts)
13. A Review of the Reference Dose and Reference Concentration Processes, Final Report,
EPA/630/P-02/002F, December 2002 (Excerpts)
14. Science Advisory Board Review of EPA’s Draft Assessment entitled Toxicological
Review of Libby Amphibole Asbestos, January 30, 2013 (Excerpts)
15. May 1,2012transcriptofSABpanel(Excerpts)
16. June27,2012,SABemail
17. July28,2012,SABemail
18. February6,2012transcriptofSABpanel(Excerpts)
19. July25,2012transcriptofSABpanel(Excerpts)
20. DraftScience Advisory Board Report,April2012(Excerpts)
21. EPA Materials Submittedto the NationalResearch CouncilPartI: Status of
ImplementationofRecommendations,January30,2013(Excerpts)
22. October 12,2012,SABemail
23. "DoAsbestos-Induced PleuralPlaquesCauseLungFunction Deficits?" LEKerper,HN
Lynch,LCMohr,JEGoodman,SocietyofToxicologyPoster,to bepresented 2014
24. U.S.DOT/RITA/Volpe Center Contract,May4,2009(Excerpts)
25. EPAResponseto DraftOffice ofInspector GeneralReport,September5,2012
(Excerpts)
26. Appendix(p.A-74)to EPA'sResponseto Selected MajorInteragency Commentsonthe
Interagency Science Consultation DraftIRIS ToxicologicalReviewofLibby Amphibole
Asbestos,August25,2011(OfficeofManagementandBudgetquestions related to the
proposed RfCbeing below background levels)
27. EPA website,available athttp://www.epa.gov/risk/exposure.htm,Step3
28. DraftPhase V Sampling and AnalysisPlanfor Operable Unit3,Libby Asbestos
Superfund Site,Working Draft,March20,2012(Excerpts)
29. May30,2013,emailfrom Bill Brattin(SRCInc.)to T.Hilbertcopying BobBenson
(EPA)and David Berry(EPA);earlier May17,2013emailsbetween Bill Brattin(SRC
Inc.)and T.Hilbert,copyingBobBenson;November9,2010,emailfrom T.Hilbertto
BobBenson
30. April23,2010,emailfromBobBensonto J.Hilbertand David Berry(EPA)and
transmitted commentsbyBobBenson
2
APPENDIX C-1
8452 Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices
OFFICEOFMANAGEMENTAND guidanceto Federalagenciesfor followed in drafting the guidelines that
BUDGET ensuring and maximizingthequality, wepublished on September28,2001
objectivity,utility,andintegrity of (66FR49719),are also applicable to the
Guidelinesfor Ensuring and information(including statistical amended guidelines thatwepublish
Maximizingthe Quality,Objectivity, information)disseminated byFederal today.
Utility,and Integrity ofInformation agencies * *"Section 515(b)goeson In accordance with section 515,OMB
Disseminated by Federal Agencies; to state thattheOMBguidelines shall: has designed theguidelinesto help
Republication "(1)apply to the sharing byFederal agencies ensure and maximize the
agencies of,and accessto,information quality,utility, objectivity and integrity
Editorial Note:Dueto numerouserrors, disseminated byfederal agencies;and oftheinformation thatthey disseminate
this documentis being reprinted in its "(2)require thateach Federalagency (meaningto share with,orgive access
entirety.It wasoriginally printed in the to whichtheguidelines apply— to,the public).It is crucialthat
FederalRegister onThursday,January 3, "(A)issue guidelines ensuring and information Federalagencies
2002at67FR369-378and wascorrected on maximizingthe quality,objectivity, disseminate meetstheseguidelines.In
Tuesday,February5,2002at67FR5365. utility,and integrity ofinformation this respect,the factthattheInternet
AGENCY:Office ofManagementand (including statistical information) enablesagencies to communicate
Budget,Executive Office ofthe disseminated bytheagency,bynotlater information quickly and easily to a wide
President. than 1 year after the date ofissuance of audience notonlyoffers great benefits to
ACTION:Finalguidelines. the uidelines undersubsection(a); society,butalso increasesthe potential
"~B)establish administrative harm thatcan resultfrom the
SUMMARY:Thesefinal guidelines mechanismsallowing affected persons dissemination ofinformationthat does
implementsection 515 ofthe Treasury to seekandobtain correction of not meetbasicinformation quality
and GeneralGovernment information maintained and guidelines.Recognizingthe widevariety
Appropriations Actfor Fiscal Year 2001 disseminated bythe agencythat does ofinformation Federalagencies
(PublicLaw106-554;H.R.5658). notcomply withthe guidelinesissued disseminate and the widevariety of
Section 515directs the Office of undersubsection(a);and dissemination practices thatagencies
Managementand Budget(OMB)to issue "(C)reportperiodically to the have,OMBdeveloped theguidelines
government-wideguidelinesthat Director— with several principlesin mind.
"provide policy and procedural "(i)the numberand nature of First,OMBdesigned the guidelinesto
guidancetoFederalagenciesfor complaintsreceived bythe agency applyto a wide variety ofgovernment
ensuring and maximizingthe quality, regarding the accuracy ofinformation information dissemination activities
objectivity,utility,andintegrity of disseminated bythe agencyand; that mayrangein importanceand scope.
information(including statistical "(ii)howsuchcomplaints were OMBalso designedtheguidelines to be
information)disseminated byfederal handled bytheagency." genericenough to fit all media,bethey
agencies."ByOctober 1,2002,agencies Proposed guidelines were published printed,electronic,orin other form.
mustissue their ownimplementing in theFederalRegister onJune 28,2001 OMBsoughtto avoid the problemsthat
guidelines thatinclude"administrative (66FR34489).Finalguidelines were would be inherentin developing
mechanismsallowing affected persons published intheFederalRegister on detailed,prescriptive,"one-size-fits-all"
to seek and obtain correction of September 28,2001(66FR49718).The government-wideguidelinesthatwould
information maintained and SupplementaryInformation to thefinal artificially require differenttypesof
disseminated bytheagency"thatdoes guidelines published in September2001 dissemination activities to betreated in
notcomplywiththeOMBguidelines. provides background,theunderlying thesame manner.Throughthis
Thesefinal guidelines also reflectthe principles OMBfollowed in issuing the flexibility,each agency willbeable to
changesOMBmadeto the guidelines final guidelines,andstatementsof incorporate therequirementsofthese
issuedSeptember 28,2001,asaresult intentconcerning detailed provisionsin OMBguidelines into the agency's own
ofreceiving additionalcommentonthe thefinal guidelines. information resource managementand
"capableofbeing substantially In thefinalguidelilnes published in administrative practices.
September 2001,OMBalso requested Second,OMBdesigned the guidelines
reproduced"standard(paragraphs
additionalcommentonthe"capableof so thatagencies will meetbasic
V.3.B,V.9,and V.10),whichOMB
previouslyissued on September 28, being substantially reproduced" information quality standards.Given the
standard and therelated definition of administrative mechanismsrequired by
2001,onaninterim final basis.
"influential scientific or statistical section 515 as wellas the standardsset
DATES:EffectiveDate:January 3,2002,
information"(paragraphs V.3.B,V.9, forth in thePaperworkReduction Act,it
FOR FURTHERINFORMATION CONTACT: and V.10),which wereissued on an is clear thatagenciesshould not
BrookeJ. Dickson,Office ofInformation interim final basis.Thefinal guidelines disseminate substantiveinformation
andRegulatory Affairs,Office of published today discuss the public thatdoes notmeetabasic level of
Managementand Budget,Washington, commentsOMBreceived,the OMB quality.Werecognize thatsome
DC20503.Telephone[phone] or response,and amendmentsto thefinal governmentinformation mayneed to
bye-mailto guidelines published inSeptember meethigher or morespecific
informationqualityQomb.eop.gov. 2001. information quality standardsthan
SUPPLEMENTARYINFORMATION:In section In developing agency-specific those that would applyto other typesof
515(a)ofthe Treasuryand General guidelines,agenciesshould refer both to governmentinformation.The more
GovernmentAppropriations Actfor the SupplementaryInformation to the importanttheinformation,the higher
Fiscal Year2001(PublicLaw106-554; final guidelines published in the the qualitystandardsto which it should
H.R.5658),Congress directed the Office FederalRegister on September 28,2001 be held,forexample,in those situations
ofManagementand Budget(OMB)to (66FR49718),and also to the involving"influential scientific,
issue,bySeptember30,2001, SupplementaryInformation published financial,orstatistical information"(a
government-wideguidelines that today.Westress thatthethree phrase defined inthese guidelines).The
"provide policy and procedural "UnderlyingPrinciples"that OMB guidelinesrecognize,however,that
Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices 8453
information quality comesatacost. confidential.In suchcases,while presented in an accurate,clear,
Accordingly,theagenciesshould weigh agencies'implementation ofthe complete,and unbiased manner,and as
the costs(for example,including costs guidelines maydiffer,theessence ofthe a matterofsubstance,is accurate,
attributable to agency processing effort, guidelines will apply.Thatis,these reliable,and unbiased."Integrity"refers
respondentburden,maintenance of agencies mustmaketheir methods to security—theprotection of
needed privacy,and assurances of transparentby providing informationfrom unauthorized access
suitable confidentiality)and the benefits documentation,ensure quality by orrevision,to ensurethat the
ofhigherinformation qualityin the reviewingthe underlying methodsused information is notcompromised
developmentofinformation,and the in developingthe dataand consulting through corruption orfalsification.OMB
level ofquality to whichtheinformation (as appropriate)withexperts and users, modeled the definitions of
disseminated will beheld. and keep usersinformed about "information,""government
Third,OMBdesigned the guidelines corrections andrevisions. information,""information
so thatagencies can applythemin a dissemination product,"and
SummaryofOMBGuidelines
common-sense and workable manner.It "dissemination"onthe longstanding
is importantthatthese guidelines do not Theseguidelines apply to Federal definitions ofthosetermsin OMB
imposeunnecessary administrative agenciessubjectto thePaperwork Circular A-130,buttailored them to fit
burdensthat would inhibitagencies Reduction Act(44U.S.C.chapter 35). into the contextofthese guidelines.
from continuingto take advantage ofthe Agencies are directed to develop In addition,Section 515imposestwo
Internetand othertechnologiesto information resources management reporting requirementsonthe agencies.
disseminate information thatcan be of proceduresfor reviewing and Thefirst report,to be promulgated no
greatbenefitand valueto the public.In substantiating(by documentation or later than October 1,2002,must.provide
this regard,OMBencouragesagenciesto other meansselected bytheagency)the the agency's information quality
incorporate the standardsand quality(including the objectivity, guidelines that describe administrative
proceduresrequired bythese guidelines utility,and integrity)ofinformation mechanismsallowing affected persons
into their existinginformationresources before it is disseminated.In addition, to seekand obtain,where appropriate,
managementand administrative agencies are to establish administrative correction ofdisseminated information
practices rather than create newand mechanismsallowing affected persons thatdoes notcomplywiththe OMBand
potentially duplicative or contradictory to seekand obtain,where appropriate, agencyguidelines.Thesecond reportis
processes.Theprimaryexample ofthis correction ofinformation disseminated an annualfiscal yearreportto OMB(to
is thatthe guidelines recognize that,in bythe agencythat does notcomply with befirst submitted onJanuary 1,2004)
accordance with OMBCircular A-130, the OMBoragencyguidelines. providinginformation(both quantitative
agencies already havein place well- Consistent withtheunderlying and qualitative,where appropriate)on
established information quality principles described above,these the number,nature,andresolution of
standardsand administrative guidelines stress theimportanceof complaintsreceived bythe agency
mechanismsthat allow personsto seek having agencies applythesestandards regarding its perceived orconfirmed
and obtain correction ofinformation and develop their administrative failure to comply withthese OMBand
thatis maintained and disseminated by mechanismssotheycanbe agency guidelines.
the agency.UndertheOMBguidelines, implemented in acommonsense and PublicCommentsand OMBResponse
agencies need onlyensurethattheir workable manner.Moreover,agencies
ownguidelines are consistent with mustapplythesestandardsflexibly,and ApplicabilifyofGuidelines.Some
these OMBguidelines,andthen ensure in a mannerappropriateto the nature commentsraised concernsaboutthe
thattheir administrative mechanisms and timeliness ofthe informationto be applicability ofthese guidelines,
satisfy the standardsand procedural disseminated,and incorporate them into particularly in thecontextofscientific
requirementsinthe newagency existing agencyinformation resources research conducted byFederally
guidelines.Similarly,agencies mayrely managementand administrative employed scientists orFederalgrantees
ontheirimplementation ofthe Federal practices. whopublish andcommunicatetheir
Government'scomputersecuritylaws Section 515 denotesfoursubstantive research findingsin the same manneras
(formerly,the ComputerSecurity Act, termsregarding information their academiccolleagues.OMB
and nowthe computersecurity disseminated byFederal agencies: believes thatinformation generated and
provisions ofthePaperworkReduction quality,utility,objectivity,and disseminated in these contexts is not
Act)to establish appropriate security integrity.It is notalways clear howeach covered bythese guidelines unlessthe
safeguardsfor ensuringthe"integrity" substantive term relates—orhowthe agencyrepresents the information as,or
oftheinformation thatthe agencies fourtermsin aggregate relate—tothe usestheinformationin supportof,an
disseminate. widely divergenttypes ofinformation official position ofthe agency.
In addition,in responseto concerns thatagencies disseminate.The Asageneral matter,these guidelines
expressed bysome ofthe agencies,we guidelines provide definitions that applyto"information"thatis
wantto emphasizethatOMBrecognizes attemptto establish aclear meaningso "disseminated"byagencies subjectto
thatFederalagencies provide a wide thatboth the agencyand the publiccan thePaperworkReduction Act(44U.S.C.
variety ofdataand information. readilyjudge whethera particulartype 3502(1)).See paragraphsII, V.5and V.B.
Accordingly,OMBunderstandsthatthe ofinformation to bedisseminated does Thedefinitions of"information"and
guidelines discussed below cannotbe or does notmeetthese attributes. "dissemination"establish thescope of
implementedin thesamewaybyeach In the guidelines,OMBdefines the applicability ofthese guidelines.
agency.Insomecases,forexample,the "quality"asthe encompassingterm,of "Information"means"any
data disseminated byanagency are not which"utility,""objectivity,"and communication orrepresentation of
collected bythatagency;rather,the "integrity"are the constituents. knowledgesuch asfacts or data * *"
information the agency mustprovidein "Utility"refers to the usefulness ofthe This definition ofinformation in
atimely manneris compiledfrom a informationto theintended users. paragraph V.5 does"notinclude
variety ofsourcesthat are constantly "Objectivity"focuseson whetherthe opinions,wherethe agency's
updated and revised and maybe disseminated information is being presentation makesit clear that whatis
8454 Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices
being offered is someone'sopinion andeven iftheFederalagencyretains Mostcommentsapproved ofthe
ratherthanfact ortheagency's views." ownership or other intellectual property prominentrole that peerreview playsin
"Dissemination"is defined to mean rights becausetheFederalgovernment theOMBguidelines.Somecomments
"agencyinitiated orsponsored paid for theresearch.Toavoid contended that peerreview wasnot
distribution ofinformation to the confusionregarding whetherthe agency accepted asauniversalstandardthat
public." Asusedin paragraph V.8, is sponsoringthe dissemination,the incorporates anestablished,practiced,
"agencyINITIATED * * *distribution researchershould include an and sufficientlevel ofobjectivity.Other
ofinformationto the public"refers to appropriate disclaimerinthe commentsstated thattheguidelines
informationthat the agency publication orspeechto the effectthat would bebetter clarified by making peer
disseminates,e.g.,arisk assessment the"viewsare mine,and do not review one ofseveralfactors thatan
prepared bythe agencyto inform the necessarily reflectthe view"ofthe agencyshould considerin assessing the
agency'sformulation ofpossible agency.Onthe otherhand,subsequent objectivity(and qualityin general)of
regulatory or other action.In addition, agency dissemination ofsuch original research.In addition,several
ifan agency,as an institution, information requiresthatthe commentsnoted that peerreview does
disseminatesinformation prepared by information adhereto theagency's notestablish whether analyticresults
an outside partyin a mannerthat information qualityguidelines.Insum, are capable ofbeingsubstantially
reasonablysuggeststhatthe agency these guidelines govern an agency's reproduced.In light ofthe comments,
agrees withtheinformation,this dissemination ofinformation,but thefinal guidelinesin new paragraph
appearanceofhaving theinformation generally donotgovern athird-party's V.3.b.i qualifythe presumptioninfavor
representagency views makesagency dissemination ofinformation(the ofpeer-reviewed information as follows:
dissemination oftheinformation subject exception being wherethe agencyis "However,this presumption is
to these guidelines.Bycontrast,an essentially using the third-party to rebuttable based ona persuasive
agency does not"initiate"the disseminate information onthe agency's showingbythe petitionerin a particular
dissemination ofinformation whena behalf.Agencies,particularly those that instance."
federally employedscientist or Federal fund scientific research,are encouraged Webelieve thattransparencyis
grantee orcontractor publishesand to clarify the applicability ofthese importantfor peerreview,andthese
communicateshis or herresearch guidelinesto the varioustypesof guidelines set minimumstandardsfor
findingsin thesame manneras his or information they and theiremployees the transparency ofagency-sponsored
her academiccolleagues,even ifthe andgrantees disseminate. peerreview.Aswestate in new
Federalagency retains ownership or Paragraph V.8also states that the paragraph V.3.b.i:"Ifdataand analytic
otherintellectual propertyrights definition of"dissemination"doesnot results have beensubjected to formal,
becausetheFederalgovernmentpaid for include"* *distribution limited to independent,external peerreview,the
theresearch.Toavoid confusion correspondence with individuals or information maygenerally be presumed
regarding whetherthe agencyagrees persons,pressreleases,archivalrecords, to be ofacceptable objectivity.However,
with theinformation(and is therefore publicfilings,subpoenasor adjudicative this presumption is rebuttable based on
disseminating it through theemployee processes."Theexemptionfrom the a persuasive showing bythe petitioner
or grantee),theresearchershould definition of"dissemination"for in aparticular instance.Ifagency-
include an appropriate disclaimerin the "adjudicative processes"is intended to sponsored peerreview is employedto
publication orspeechto the effect that exclude,from thescope ofthese help satisfy the objectivitystandard,the
the"viewsare mine,and do not guidelines,thefindings and review processemployed shall meetthe
necessarily reflectthe view"ofthe determinationsthatan agency makesin general criteria for competentand
agency. thecourse ofadjudicationsinvolving credible peerreview recommended by
Similarly,as used in paragraph V.8., specific parties.Thereare well- OMB–OIRAto thePresidents
"agency* *SPONSORED established proceduralsafeguards and ManagementCouncil(9/20/01)(http://
distribution ofinformation to the rights to addressthe quality of www.whitehouse.gov/omb/inforeg/
public"refers to situations wherean adjudicatory decisions andto provide oira_review-process.html),namely,`that
agency has directed athird-party to persons with anopportunityto contest (a)peerreviewersbeselected primarily
disseminateinformation,or wherethe decisions.Theseguidelines do not on the basis ofnecessarytechnical
agency hasthe authorityto review and imposeanyadditional requirements on expertise,(b)peerreviewersbeexpected
approvetheinformation before release. agencies during adjudicative to disclose to agencies prior technical/
Therefore,for example,ifanagency proceedingsand do notprovide parties policy positions they mayhavetakenon
through a procurementcontract ora to such adjudicative proceedingsany the issues at hand,(c)peerreviewersbe
grantprovidesfor a person to conduct additionalrights ofchallenge orappeal. expected to disclose to agenciestheir
research,andthen the agency directs ThePresumption FavoringPeer- sources ofpersonalandinstitutional
the person to disseminate the results(or ReviewedInformation.Asageneral fiznding(private or publicsector),and
the agencyreviewsand approvesthe matter,in the scientific and research (d)peerreviewsbeconducted in an
results before they maybe context,weregard technicalinformation open and rigorous manner."'
disseminated),then theagencyhas thathas beensubjected to formal, Theimportance ofthese general
"sponsored"the dissemination ofthis independent,external peerreview as criteria for competentand credible peer
information.Bycontrast,ifthe agency presumptively objective.Asthe review hasbeen supported bya number
simply providesfundingto support guidelines state in paragraph V.3.b.i:"If ofexpertbodies.Forexample,"the
research,andit the researcher(notthe data and analyticresults have been workoffullycompetentpeer-review
agency)whodecides whetherto subjected to formal,independent, panelscan be undermined by
disseminate theresults and—ifthe external peerreview,theinformation allegations ofconflictofinterest and
results are to bereleased—who maygenerally be presumed to be of bias.Therefore,thebestinterests ofthe
determinesthe contentand presentation acceptable objectivity." Anexample ofa Board are served byeffective policies
ofthe dissemination,thenthe agency formal,independent,external peer and proceduresregarding potential
has not"sponsored"the dissemination review is thereview process used by conflicts ofinterest,impartiality,and
eventhough it hasfunded theresearch scientific journals. panel balance."(EPA'sScienceAdvisory
Federal Register/Vol. 67, No, 36/Friday, February 22, 2002/Notices 8455
BoydPanels:ImprovedPolicies and Although such cases offalsification guidelines,andthe differences in the
ProceduresNeededto Ensure are presumablyrare,there is a nature oftheinformationthey
IndependenceandBalance,GAO-01- significant scholarly literature disseminate,wealso believe it will be
536,General Accounting Office, documenting quality problems with helpfulifagencieselaborate onthis
Washington,DC,June 2001,page 19.) articles published in peer-reviewed definition of"influential"inthecontext
As another example,"risk analyses research."Ina[peer-reviewed]meta- oftheir missionsand duties,with due
should be peer-reviewed and analysis that surprised many—andsome consideration ofthe nahire ofthe
accessible—both physically and doubt—researchersfound little evidence information they disseminate. Aswe
intellectually—sothat decision-makers that peerreview actuallyimprovesthe state in amended paragraph V.9,"Each
at all levels will be able to respond quality ofresearch papers."(See,e.g., agencyis authorized to define
critically torisk characterizations.The Science,Vol.293,page 2187(September `influential'in waysappropriatefor it
intensity ofthe peerreviewsshould be 21,2001.))In partfor this reason,many given the nature and multiplicity of
commensuratewith the significance of agencies have already adopted peer issues for whichtheagencyis
the risk or its management review and science advisory practices
implications."(SettingPriorities, thatgo beyondjournal peerreview.See, rReeprsodupciboilinty.sAsieveb stlatee in .new"
GettingResults:A NewDirectionfor e.g.,SheilaJasanoff, TheFifth Branch: paragraph V.3.b.ii:"Ifanagencyis
EPA,SummaryReport,National Science AdvisersasPolicyMakers, responsible for disseminating influential
AcademyofPublic Administration, Cambridge,MA,Harvard University scientific,financial,orstatistical
Washington,DC,April1995,page 23.) Press,1990;MarkR.Powell,Scienceat information,agencyguidelinesshall
These criteria for peerreviewers are EPA:Information in the Regulatory include a high degreeoftransparency
generallyconsistentwith the practices Process.Resourcesfor theFuture, about data and methodsto facilitate the
nowfollowed bythe NationalResearch Washington,DC.,1999,pages 138-139; reproducibility ofsuch information by
Council ofthe National Academyof 151-153;Implementation ofthe qualified third parties."OMBbelieves
Sciences.In considering these criteria EnvironmentalProtection Agency'sPeer that areproducibilitystandardis
for peerreviewers,wenotethatthere ReviewProgram:AnSABEvaluation of practical and appropriate for
are manytypes ofpeerreviews and that ThreeReviews,EPA–SAB–RSAC-01- information thatis considered
agencyguidelines concerningthe use of 009,AReviewofthe Research Strategies "influential",as defined in paragraph
peerreviewshould tailor therigor of AdvisoryCommittee(RSAC)oftheEPA V.9—that"willhave or doeshave a
peerreview totheimportance ofthe Science AdvisoryBoard(SAB), clear and substantialimpacton
information involved.Moregenerally, Washington,DC.,September 26,2001. important public policies orimportant
agenciesshould define their peer-review Forinformation likely to havean private sector decisions."The
standardsin appropriate ways,giventhe important public policy or private sector reproducibilitystandard applicable to
nature andimportanceofthe impact,OMBbelievesthatadditional influential scientific,financial,or
informationthey disseminate. quality checks beyond peerreview are statistical information isintended to
IsJournalPeer1?eviewAlways appropriate. ensurethatinformation disseminated by
Sufficient?Somecommentsargued that Definition of"Influential".OMB agenciesis sufficiently transparentin
journal peerreview should beadequate guidelines applystricter quality termsofdataand methodsofanalysis
to demonstrate quality,evenfor standardsto the disseminationof thatit wouldbefeasible for areplication
influential information thatcan be information thatis considered to beconducted.Thefactthatthe use
"influential."Commentsnoted that the oforiginal andsupporting data and
expected to have majoreffects on public
breadth ofthe definition of"influential" analyticresults havebeen deemed
policy.OMBbelievesthatthis position
in interim final paragraph V.9requires "defensible"bypeer-review procedures
overstates the effectiveness ofjournal
muchspeculation onthe partof does notnecessarily implythatthe
peerreview as aquality-control
results are transparentandreplicable.
mAelthcouhghjaounrnail pesermrev.ie w is aWgebeelinevce thiatethiss cr.iti cism has ReproducibilityofOriginaland
meritand havetherefore narrowedthe SupportingData.Several ofthe
clearly valuable,there are caseswhere
definition.Inthis narrower definition, commentsobjected to the exclusion of
flawedscience hasbeen published in
"influential",whenusedin the phrase original andsupporting datafrom the
respected journals.Forexample,the
"influential scientific,financial,or
NIHOffice ofResearchIntegrity recently statistical information",is amendedto Cromemepntrsionstderadu esucgqgeisutebdi tihratlOeMimB teyn ts.
reported thefollowing caseregarding
meanthat"the agencycan reasonably should apply thereproducibility
environmental health research:
determinethat dissemination ofthe standard to original data,andthatOMB
"Based onthereportofaninvestigation information will have or does have a should provideflexibility to the
conducted by(XX]University,dated July 16, clear and substantialimpacton agencies in determining what
1999,and additional analysisconducted by importantpublic policies orimportant constitutes"original and supporting"
ORIin its oversightreview,the USPublic private sector decisions."Theintentof data.OMBagrees and asksthatagencies
Health Servicefound thatDr.[X]engaged in
scientific misconduct.Dr.[X]committed the new phrase"clear andsubstantial" consider,in developingtheir own
scientific misconductbyintentionally is to reducethe needfor speculation on guidelines,whichcategories oforiginal
falsifying theresearch results published in the partofagencies.Weaddedthe and supporting datashould besubjectto
thejournalSCIENCEandby providing presenttense—"ordoes have"—tothis the reproducibilitystandard and which
falsified andfabricated materialsto narrower definition because on should not.Tohelp in resolving this
investigating officials at[XX)Universityin occasion,an information dissemination issue,wealso ask agenciesto consult
responseto arequestfor original datato mayoccursimultaneously with a directly with relevantscientific and
supportthe research results and conclusions particular policy change.In responseto technicalcommunitiesonthefeasibility
reportin theSCIENCEpaper.In addition, a publiccomment,weadded an explicit ofhavingtheselected categories of
PHSfindsthatthere is nooriginal data or reference to"financial"information as original and supporting data subject to
othercorroborating evidenceto supportthe
research results and conclusionsreported in consistentwith our originalintent. thereproducibility standard.Agencies
the SCIENCEpaperas a whole."(66FR Giventhe differences in the many are encouraged to addressethical,
52137,October12,2001). Federalagenciescovered bythese feasibility,and confidentialityissues
8456 Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices
with care. Aswestate in newparagraph berequired toreproduceeach analytical Harvard Six Cities Studyand the
V.3.b.ii.A,"Agencies mayidentify,in result before it is disseminated.While American CancerSociety Studyof
consultation with therelevantscientific severalcommentscommendedOMBfor Particulate AirPollution and Mortality,"
andtechnicalcommunities,those establishing an appropriate balancein ASpecial Reportofthe Health Effects
particulartypesofdatathatcan the"capable ofbeingsubstantially Institute's Particle Epidemiology
practicably besubjected to a reproduced"standard,others ReanalysisProject,Cambridge,MA,
reproducibility requirement,given considered this standard to be 2000.
ethical,feasibility,or confidentiality inherently subjective.There were also Theprimary benefit ofpublic
constraints."Further,as westate in our commentsthatsuggested thestandard transparencyis notnecessarilythat
expanded definition of would cause moreburdenfor agencies. errors in analytic results will be
"reproducibility"in paragraph V.10,"If It is notOMB'sintentthateach detected,although error correction is
agencies applythe reproducibilitytest agency mustreproduceeach analytic clearly valuable.The moreimportant
to specific typesoforiginal or result before it is disseminated.The benefit oftransparencyis that the public
supporting data,the associated purpose ofthereproducibilitystandard will be able to assess how muchan
guidelinesshall providerelevant is to cultivate a consistentagency agency'sanalytic result hinges on the
definitionsofreproducibility(e.g., commitmentto transparencyabouthow specific analytic choices madebythe
standardsfor replication oflaboratory analyticresults are generated:the agency.Concretenessaboutanalytic
data)."OMBurgescaution in the specific data used,the various choicesallows,for example,the
treatmentoforiginal andsupporting assumptionsemployed,the specific implications ofalternative technical
data because it mayoften beimpractical analytic methodsapplied,and the choicesto bereadily assessed.Thistype
or evenimpermissible orunethical to statistical procedures employed.If ofsensitivity analysis is widely
applythereproducibilitystandard to sufficienttransparencyis achieved on regarded asanessentialfeature ofhigh-
such data.Forexample,it maynotbe each ofthese matters,then an analytic quality analysis,yetsensitivity analysis
ethical to repeata"negative" resultshould meetthe"capable ofbeing cannotbe undertaken byoutside parties
(ineffective)clinical(therapeutic) substantially reproduced"standard. unlessa high degree oftransparencyis
experimentand it maynotbefeasible to Whilethere is much variation in types achieved.TheOMBguidelines donot
replicate theradiation exposures ofanalytic results,OMBbelievesthat compelsuch sensitivity analysisas a
studied after the Chernobylaccident. reproducibilityis apracticalstandard to necessary dimension ofquality,butthe
Whenagenciessubmittheir draft agency applyto mosttypesofanalyticresults. transparency achieved by
guidelinesforOMBreview,agencies Aswestate in new paragraph V.3.b.ii.B, reproducibility will allow the publicto
should includea description ofthe "Withregard to analyticresults related undertake sensitivity studies ofinterest.
extentto whichthereproducibility [to influential scientific,financial,or Weacknowledgethat confidentiality
standard is applicable andreflect statistical information),agency concerns willsometimes preclude
consultations with relevantscientific guidelines shall generallyrequire public access as an approachto
and technicalcommunitiesthat were sufficient transparency about data and reproducibility.Inresponseto public
used in developing guidelinesrelated to methodsthat anindependentreanalysis comment,wehave clarified thatsuch
applicability ofthereproducibility could be undertaken bya qualified concernsdoinclude interestsin
standard to original andsupporting memberofthe public.These "intellectual property."Toensurethat
data. transparencystandardsapplyto agency theOMBguidelines havesufficient
It is also importantto emphasizethat analysis ofdatafrom a single study as flexibility with regard to analytic
thereproducibilitystandard does not wellasto analyses thatcombine transparency,OMBhas,in new
apply to all original and supporting data informationfrom multiple studies."We paragraph V.3.b.ii.B.i,provided agencies
disseminated byagencies.Aswestate in elaborate uponthis principle in our an alternative approachfor classes or
newparagraph V.3.b.ii.A,"Withregard expanded definition of types ofanalyticresultsthatcannot
to original andsupporting datarelated "reproducibility"in paragraph V.10: practically besubject to the
[to influential scientific,financial,or "Withrespectto analyticresults, reproducibility standard."[In those
statistical information],agency `capable ofbeing substantially situations involving influential
guidelines shall notrequire that all reproduced'meansthatindependent scientific,financial,orstatistical
disseminated data besubjected to a analysis ofthe original orsupporting information * *]makingthe data and
reproducibilityrequirement."In data using identical methodswould methods publicly available will assist in
addition,weencourageagenciesto generate similar analytic results,subject determining whether analytic results are
address howgreatertransparencycan be to anacceptable degreeofimprecision reproducible.However,the objectivity
achieved regarding original and or error." standard does notoverride other
supporting data.Aswealso state in new Even in asituation wherethe original compellinginterests such as privacy,
paragraph V.3.b.ii.A,"Itis understood andsupporting data are protected by trade secrets,intellectual property,and
thatreproducibility ofdata is an confidentiality concerns,orthe analytic other confidentiality protections."
indication oftransparencyabout computer modelsor otherresearch Specifically,in cases where
research design and methodsandthus methods maybekeptconfidentialto reproducibility will notoccur dueto
areplication exercise(i.e.,anew protectintellectual property,it maystill othercompelling interests,weexpect
experiment,test,orsample)shall notbe befeasible to havethe analytic results agencies(1)to perform robustness
required prior to each dissemination." subjectto thereproducibilitystandard. checksappropriate to the importance of
Agencyguidelines need to achievea Porexample,a qualified party, theinformation involved,e.g.,
high degree oftransparency aboutdata operating underthesame determining whetheraspecific statistic
even whenreproducibility is not confidentiality protections asthe is sensitive to the choice ofanalytic
required. original analysts,maybeasked to use method,and,accompanyingthe
ReproducibilityofAnalyticResults. thesamedata,computer modelor information disseminated,to document
Manypublic comments were critical of statistical methodsto replicate the their efforts to assure the needed
thereproducibilitystandard and analytic results reported in the original robustnessin information quality,and
expressed concernthat agencies would study.See,e.g.,"Reanalysis ofthe (2)addressintheir guidelinesthe
Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices 8457
degree to whichtheyanticipate the public dissemination ofdataand robustnesschecksrequired bythese
opportunity for reproducibility to be modelsperse does not meanthatthe guidelines.Otherwise,theagency
limited bythe confidentiality of analyticresult hasbeenreproduced.It should notdisseminate anyofthe
underlying data.Aswestate in new meansonlythattheresultshould be studies that did notmeetthe applicable
paragraph V.3.b.ii.B.ii,"In situations CAPABLEofbeing reproduced.The standardsin theguidelines atthe time
where public access to data and transparency associated with this it publishesthe notice ofproposed
methods will notoccur dueto other capability ofreproduction is whatthe rulemaking.
compelling interests,agencies shall OMBguidelines are designed to Somecommentssuggested thatOMB
apply especially rigorousrobustness achieve. considerreplacing thereproducibility
checksto analytic results and document Wealso wantto build onageneral standard with astandard concerning
whatchecks were undertaken.Agency observation that we madein ourfinal "confirmation"ofresults forinfluential
guidelines shall,however,in all cases, guidelines published in September scientific and statistical information.
require a disclosure ofthe specific data 2001.In those guidelines westated:"... Although weencourage agencies to
sourcesthat havebeen used andthe in those situations involvinginfluential consider"confirmation"as arelevant
specific quantitative methodsand scientific[,financial,]or statistical standard—atleastin somecases—for
assumptionsthat havebeenemployed." information,the substantial assessing the objectivity oforiginal and
Given the differences inthe many supporting data,webelievethat
reproducibilitystandard is added as a
Federalagencies covered bythese "confirmation"is too stringent a
quality standard above and beyond
guidelines,and the differencesin standard to applyto analytic results.
some peerreview quality standards"(66
robustnesschecksandthelevel ofdetail Often theregulatory impactanalysis
FR49722(September28,2001)).A
for documentation thereofthat mightbe prepared byan agencyfora majorrule,
hypotheticalexample mayserve to
appropriate for differentagencies,we for example,will bethe onlyformal
illustrate this point.Assumethattwo
also believe it will be helpfulifagencies analysis ofan importantsubject.It
Federalagenciesinitiated orsponsored
elaborate onthese mattersin the context would be unlikelythattheresults ofthe
the dissemination offive scientific
oftheir missionsand duties,with due regulatoryimpactanalysis hadalready
consideration ofthe nature ofthe studies after October 1,2002(see beenconfirmed byother analyses.The
information they disseminate.Aswe paragraphIII.4)thatwere,before "capable ofbeing substantially
dissemination,subjected to formal,
state in new paragraph V.3.b.ii.B.ii, reproduced"standard is less stringent
independent,external peerreview,i.e.,
"Each agencyis authorized to define the than a"confirmation"standardbecause
that metthe presumptivestandard for
type ofrobustness checks,and thelevel it simplyrequiresthatanagency's
ofdetailfor documentationthereof,in "objectivity"under paragraph V.3.b.i. analysis besufficiently transparentthat
waysappropriate for it given the nature Further assume,atthetimeof anotherqualified partycouldreplicate it
and multiplicity ofissuesfor whichthe dissemination,thatneither agency through reanalysis.
agencyis responsible." reasonablyexpected thatthe Health,Safety,andEnvironmental
Weleave the determination ofthe dissemination ofanyofthese studies Information.Wenote,inthescientific
appropriate degree ofrigor to the would have"aclear andsubstantial context,thatin 1996the Congress,for
discretion ofagencies and therelevant impact"onimportantpublic policies, health decisions underthe Safe
scientific and technicalcommunities i.e.,thatthesestudies were not Drinking Water Act,adopted abasic
that workwiththe agencies.Wedo, considered"influential"under standard ofqualityforthe use ofscience
however,establish ageneralstandard paragraph V.9,andthusnotsubjectto in agency decisionmaking.Under42
for theappropriate degreeofrigor in our the reproducibilitystandardsin U.S.C.300g-1(b)(3)(A),an agencyis
expanded definition of paragraphs V.3.b.ii.A or B.Then directed,"tothe degree thatan Agency
"reproducibility"in paragraph V.10: assume,two years later,in 2005,that action is based on science,"to use"(i)
"`Reproducibility'meansthatthe oneoftheagencies decidesto issue an the bestavailable,peer-reviewed
information is capable ofbeing importantand far-reachingregulation science and supporting studies
substantiallyreproduced,subjectto an based clearly and substantially onthe conductedin accordance withsound
acceptable degree ofimprecision.Por agency's evaluation ofthe analytic and objective scientific practices;and
informationjudged to have more(less) results setforth in these five studiesand (ii) datacollected byaccepted methods
importantimpacts,the degree of thatsuch agencyreliance onthese five or bestavailable methods(ifthe
imprecisionthatis tolerated is reduced studies as published in theagency's reliabilitq ofthe method andthe nature
(increased)."OMBwillreview each notice ofproposed rulemaking would ofthe decision justifies useofthe
agency'streatmentofthis issue when constitute dissemination ofthesefive data)."
reviewing the agencyguidelines asa studies.Theseguidelines wouldrequire Wefurther notethatinthe1996
whole. therulemaking agency,prior to amendmentsto the Safe Drinking Water
Commentsalso expressed concerns publishingthe notice ofproposed Act,Congressadopted a basic quality
regarding interim final paragraph rulemaking,to evaluate thesefive standardforthe dissemination ofpublic
V.3.B.iii,"makingthe dataand models studiesto determineifthe analytic information aboutrisks ofadverse
publicly available will assistin results stated therein would meetthe health effects.Under42U.S.C.300g-
determining whether analyticresults are "capable ofbeing substantially 1(b)(3)(B),the agencyis directed,"to
capable ofbeing substantially reproduced"standardsin paragraph ensurethatthe presentation of
reproduced,"and whetherit could be V.3.b.ii.B and,ifnecessary,related information[risk]effects is
interpreted to constitute public standardsgoverning original and comprehensive,informative,and
dissemination ofthese materials, supporting datain paragraph V.3.b.ii.A. understandable."Theagencyis further
rendering mootthereproducibilitytest. Iftheagencywereto decidethatanyof directed,"in adocumentmadeavailable
(Porthe equivalent provision,see new thefive s#udies would notmeetthe to the publicin supportofaregulation
paragraph V.3.b.ii.B.i.)TheOMB reproducibilitystandard,the agency [to]specify,to the extent practicable—
guidelines do notrequire agenciesto maystill rely onthem butonlyifthey (i)each population addressed byany
reproduceeach disseminated analytic satisfy thetransparencystandard and— estimate[ofapplicable risk effectsJ;(ii)
resultbyindependentreanalysis.Thus, as applicable—thedisclosure of theexpected risk orcentral estimate of