Distributed Logging for Transaction Processing

176 Pages·2003·9.78 MB·English

Distributed Logging
for Transaction Processing
Dean Spencer Daniels
December 1988
CMU-CS-89-114
SubmittedtoCarnegieMellonUniversityinpartialfulfillmentofthe
requirementsforthedegreeofDoctorofPhilosophy
Department ofComputer Science
Carnegie Mellon University
Pittsburgh, Pennsylvania
Copyright© 1988DeanSpencerDaniels
This workwas supported by IBMandthe DefenseAdvanced Research ProjectsAgency, ARPA OrderNo.4976,
monitoredbythe Air ForceAvionics LaboratoryunderContractF33615-87-K-C-1499.
The views andconclusions contained inthisdocument arethoseof the author andshould not be interpretedas
representingthe official policies, either expressed or implied, of any of the sponsoring agencies or of the United
StatesGovernment.

Abstract
This dissertation defends the thesis that recovery logs for transaction processing can be
efficiently and reliably provided by a highly available network service. Recovery logs are a
special implementation of stable storage that transaction processing systems use to record
information essential to their operation. Recovery logs must be very reliable and have fast
access. Typically, mirrored magnetic disks are dedicated to log storage in high performance
transaction systems. Dedicated mirrored disks are an expensive resource for small processors
like workstations or nodes in a non-shared memory multiprocessor. However, it is these types of
processors that participate in many distributed programs and benefit from the availability of a
general purposed distributed transaction facility. Distributed logging promotes reliable distributed
computing by addressing the problem of the resources needed by the recovery logfor a general
purpose distributed transaction processing facility.
The distributed logging thesis is defended by discussion of the design, implementation, and
evaluation of distributed logging services. The design issues considered include the global
representation of distributed logs, communication, security, log server data structures, log space
management, and load assignment. A new distributed algorithm for replicating recovery logs and
a new data structure for direct access to log records in append-only storage are presented. The
dissertation explores the use of uninterruptible power supplies to implement low-latency non-
volatile virtual memory disk buffers.
The implementation and evaluation of the Distributed Log Facility for the Camelot Distributed
Transaction Facility is described. The Camelot DLF uses the new distributed replication
algorithm for representing distributed logs and uses uninterruptible power supplies to implement
non-volatile virtual memory. Specially designed protocols are used for communication between
clients and log servers inthe Camelot DLF. The performance of the Camelot DLF is competitive
with the Camelot local log implementation for a variety of benchmarks. The throughput capacity
of log servers is reported.

Acknowledgments
This thesis, and my entire graduate career, would not have been possible without support,
guidance, encouragement, and friendship from Alfred Spector. I doubt that Iwill ever participate
incollaborations as successful as my eight years (so far) of working with Alfred.
My readers, Maurice Herlihy, Bruce Lindsay, and Rick Rashid, are to be commended for
promptly producing helpful comments on (often very rough) drafts ofthis thesis. Bruce's efforts at
quality control are particularly appreciated.
Dean Thompson collaborated inthe development of many of the new ideas presented here and
implemented portions of the Camelot Distributed Log Facility. The system described here should
really be called Deans' logger.
The entire Camelot group, especially Dean, Alfred, Josh Bloch, Dan Duchamp, Jeff Eppinger
and Randy Pausch, is to be thanked for providing an exciting family in which to work and play.
Thanks to Steve Berman for procuring the UPSs used for the Camelot DLF.
My graduate career and life in Pittsburgh was a really great experience because of many fine
friends. There are too many to list them all, but some deserve special mention, including: Alan,
Mike, Ann, Jumpin, Chris, Archie, Steve, David, Dan, Sherri, Doug, Penn, and of course, Bill.

For W.A.B.

Table of Contents
1.Introduction 1
1.1. Distributed Logging: Thesis and Motivation 1
1.2. Goals 3
1.3. Outline 4
2. Background 5
2.1. Distributed Systems 5
2.2. What a Log Is 7
2.2.1.Transactions andTransaction ProcessingFacilities 8
2.2.1.1.TheTransactionConcept 8
2.2.1.2.TransactionSystemApplications 9
2.2.1.3. DistributedTransaction Processing Facilities 13
2.2.2.Log Definition,Use,and Implementation 19
2.2.2.1.A SimpleLog Definition 19
2.2.2.2.Log-basedTransaction RecoveryAlgorithms 20
2.2.2.3.Practical LogInterfaces 25
2.2.2.4.Log ImplementationIssues 29
3. DesignIssuesand Alternatives 33
3.1. Representation of Distributed Log Data 34
3.1.1.Mirroring 34
3.1.2.DistributedReplication 35
3.1.2.1.ReplicatedTime-OrderedUnique Identifiers 36
3.1.2.2.ReplicatedLogAlgorithm 38
3.1.2.3.FormalProofof RestartProcedure 45
3.1.3.HybridSchemes 51
3.1.4.Comparison Criteria 52
3.1.4.1.Reliability 52
3.1.4.2.Availability 55
3.1.4.3.Performance 57
3.2. Client/Server Communication 60
3.2.1.Stream Protocols 61
3.2.2.RPCProtocols 62
3.2.3.The ChannelModeland LU6.2 64
3.2.4.ParallelCommunicationsand Multicast Protocols 66
3.2.5.ComparisonCriteria 67
3.2.5.1.LoadModels 67
3.2.5.2.Forceand RandomRead Times 68
3.2.5.3.Streaming Rates 69
3.2.5.4.Resilience 69
3.2.5.5.ComplexityandSuitability 70
3.3. Security 70
3.3.1.AlternativeMechanisms 71
3.3.1.1.Authentication Mechanisms 71
3.3.1.2. PhysicalSecurity 72
3.3.1.3. End-to-EndEncryption 72
3.3.1.4. Encryption-Based Protocols 73
3.3.2.Policiesand ComparisonCriteria 74
3.3.2,1.Threat ModelsandSecurity Requirements 74
3,3.2.2.Cost 75
3.4. Log Representation on Servers 75
3.4.1.Disk RepresentationAlternatives 76
3.4.1.1.Files 76
3.4.1.2. PartitionedSpace 77
3.4.1,3.Single DataStream 79
3.4.2. Low-LatencyBufferAlternatives 82
3.4.3.Comparison Criteria 84
3.4.3.1.Low-LatencyBuffer Costs 84
3.4.3,2.DiskUtilization 85
3.4,3.3.Performance 85
3.5. Log Space Management 86
3.5.1.Mechanisms 87
3.5.1.1.ServerControlled Mechanisms 87
3.5.1.2.ClientControlled Mechanisms 89
3,5.1.3.Compression 92
3.5.2.PolicyAlternatives 93
3.5.3.Comparison Criteria 93
3.5.3.1, UseModels 94
3.5.3.2,Costs 94
3.5.3.3. Performance 95
3.6. Load Assignment 95
3.6.1. Mechanisms 96
3.6.1.1.LoadAssessment 96
3.6.1.2.LoadAssignmentMechanisms 97
3.6.2.Policies 98
3.7. Summary (and Perspective) 99
4. The Design of the Camelot Distributed Log Facility 101
4.1. Camelot Distributed Log Design Decisions 101
4.1.1.Distributed LogRepresentation 102
4.1.2.Communication 102
4.1.3.Security 103
4.1.4.Server DataRepresentation 103
4.1.5.Log SpaceManagement 103
4.1.6.LoadAssignment 104
4.2. Communication Design 104
4.2.1.Transport Protocol 104
4,2.2.MessageProtocols 105
4.2.2.1. DataMessagePackingand ReadLogBuffering 105
4.2.2.2. RPCSubprotocols 106
4.2.2.3.WriteLogSubprotocol 108
4.2.2.4.CopyLogSubprotocol 110
4.3. Log Client Structure 111