Firmware development and intergration for ALICE TPC and PHOS Front End Electronics

199 Pages·2008·6.18 MB·English

by  Johan Alme

Firmware Development and
Integration for ALICE TPC and
PHOS Front-end Electronics
A Trigger Based Readout and Control System operating in a
Radiation Environment
Johan Alme

Dissertation for the degree philosophiae doctor (PhD)
at the University of Bergen

10 August 2008
2
3
Acknowledgements
Since I started on the PhD program in May 2004 there have been many big events in
my life, both privately and professionally. I am now a married man, father to a
beautiful daughter (soon there are two!) and owns a house and a station wagon.
Brann has won the Norwegian championship for the first time in 44 years, Liverpool
FC won the Champions League and Kiss and Black Sabbath - two of my all time
favourite bands since I was a teenager - have played live in Bergen. Professionally I
have finally come to the point where I can hand in my PhD-thesis. It has been four
years where I have learnt a lot and got to know a whole lot of nice people from all
parts of the world. I have travelled more than ever before and even got to visit some
exotic places like for instance Wuhan, Chicago, Sardinia and of course CERN. It has
been a great time!
First of all I would like to thank my supervisors Kjetil Ullaland and Dieter Röhrich
for giving me this opportunity and for having belief in me. If you hadn’t talked me
into it, I wouldn’t have tried to apply for a stipend for the fourth time in a row. Kjetil
has guided and helped me in everything from the art of electronics to makeover of old
houses. We have had long and valuable discussions where I might have disagreed
with him in the start but normally not in the end. Dieter has stood up for me during
heated debates and e-mail exchanges and has always been helpful whenever help was
needed. In addition he has an amazing way of explaning the unexplainable in simple
words. The two of you have really contributed into making the 4 years that I have put
into this thesis an enjoyable and educational journey. The same goes to Bernhard
Skaali at the University of Oslo who’ve formally been my boss during these years.
Without you I would never have got this opportunity. I would also like to thank the
Norwegian Research Council for financial support.
A very special thank also goes to my friend and colleague Ketil Røed with whom I
shared an enormous amount of coffees and crackers as well as having generally
interesting discussions about nothing and everything. We have worked closely
together and I have found it to be very rewarding. Thanks also to Torsten Alt,
Matthias Richter, Per Thomas Hille and Gerd Tröger for always being willing to
answer my at times stupid questions and for making life at CERN enjoyable during
my stays there (by for instance nighttime swimming activities with Per Thomas and
Torsten). All the rest of the people in the Microelectronics group and the Nuclear
Physics group at the University of Bergen, as well as the people involved from the
Bergen University College, also deserve to be thanked. Some times for giving
4
valuable professional input, and all the time for being part of a great social
environment that have made me look forward to go to the office every day (ok, I have
to be honest – maybe not on Sundays).
My good friend Vegard Øvrelid deserves my highest appreciation for reading through
my thesis and helping me getting my English grammar in a presentable shape.
Special thanks must go to my parents for always believing in me and supporting me,
and for being loving grandparents to Liv. Liv deserves the biggest hug of all for
filling my life with joy and laughter, even if she is an earlybird…
Finally, the deepest gratitude goes to my wife, Elen Siglen, for always supporting me,
for her patience, and for keeping up with my long and endless stories of people she
have never met and topics that don't interest her. You are an even better wife and
better mother to our daughter that I ever could have dreamt of having. I love you.

Johan
Bergen, August 2008
5
Abstract
The readout electronics in PHOS and TPC – two of the major detectors of the ALICE
experiment at the LHC – consist of a set of Front End Cards (FECs) that digitize,
process and buffer the data from the detector sensors. The FECs are connected to a
Readout Control Unit (RCU) via two sets of custom made PCB backplanes. For
PHOS, 28 FECs are connected to one RCU, while for TPC the number is varying
from 18 to 25 FECs depending on location. The RCU is in charge of the data readout,
including reception and distribution of triggers and in moving the data from the FECs
to the Data Acquisition System. In addition it does low level control tasks. The RCU
consists of an RCU Motherboard that hosts a Detector Control System (DCS) board
and a Source Interface Unit. The DCS board is an embedded computer running Linux
that controls the readout electronics.
All the mentioned devices are implemented in commercial grade SRAM based Field
Programmable Gate Arrays (FPGAs). Even if these devices are not very radiation
tolerant, they are chosen because of their cost and flexibility, and most importantly
the possibility to easily do future upgrades of the electronics. Since physical shielding
of the electronics is not possible in ALICE due to the architecture of the detector, the
radiation related errors need to be handled with other techniques such as firmware
mitigation techniques.
The main objective of this thesis has been to make firmware modules for the FPGAs
reciding in different parts of the readout electronics. Because of the flexibility of the
designs, some of them have, with minor adaptations, been applied in different devices
surrounding the readout electronics. Additionally, effort has been put into testing and
integration of the system. In detail, the work presented in this thesis can be
summarized as follows:
• Firmware design for radiation environments. All firmware modules that are
designed are to be used in a radiation environment, and then special
precautions need to be taken. Additionally, a state-of-the-art solution has been
designed for protecting the main FPGA on the RCU Motherboard against
radiation induced functional failures.
• Implementation of Trigger Handling for the TPC/PHOS Readout Electronics.
The triggers are received from the global trigger system via an optical link and
are handled by an Application Spesific Integrated Circuit (ASIC) on the DCS
board. The problem is that the DCS board might have occasional down time
6
due to radiation related errors, so a special interface module is designed for the
main FPGA on the RCU Motherboard. This module decodes and verifies the
information received from the trigger system. As it is a generic design it has
also been implemented as part of the BusyBox. The BusyBox is an important
device in the trigger path of the TPC and PHOS sub-detectors.
• Implementation of the TPC/PHOS Front-end Electronics Detector Control
System. This point includes several parts that are listed from bottom up:
o PHOS FEC Board Controller. The PHOS FEC Board Controller is the
lowest level of the DCS. It is an SRAM based FPGA that is in charge of
monitoring the health status of the given FEC and configure vital parts
of the PHOS Front-end. The design is based on the TPC Board
Controller design, but it is functionally extended and has increased
robustness against radiation related errors and other external errors.
o Active Partial Reconfiguration Solution: A vital part of the Fee DCS is
the Active Partial Reconfiguration Solution that deals with radiation
related errors. This state-of-the-art design enables the possibility to
clear any radiation related error in the configuration memory of the
main FPGA on the RCU motherboard while not interfering with the
operation of the design.
o RCU main FPGA DCS interface. A small module translating the DCS
bus protocol to an FPGA internal RCU bus protocol.
o TPC/PHOS DCS board firmware. The DCS board firmware
implements the communication path down to the registers on the RCU,
and upwards towards the ARM CPU. As the DCS board might, from
time to time, reboot because of radiation related errors, effort has been
put into ensuring that this does not affect the data flow on the RCU
Motherboard. Because of the flexibility of the DCS board, this
firmware has been adapted to several other motherboards in the system.
• Testing and Verification of all firmware modules. All firmware modules have
been extensively verified with computer simulation before being tested in real
hardware.
• Maintenance of the DCS board for TPC/PHOS and of the different Fee
firmware modules in general.
7
• System Integration and System Level Tests. A big contribution has been done
integrating and testing all the modules and sub-systems. This concern both
locally on the RCU and the BusyBox, as well as making all the devices play
together on a larger scale.
As the presented electronics are located in a radiation environment and are physically
unavailable after commissioning, effort has been put into making designs that are
reliable, scalable and possible to upgrade. This has been ensured by following a
systematic design approach where testability, version management and
documentation are key elements. Some parts of the work described in this thesis have
been published and presented in international peer reviewed publications and
conferences.
8
9
Contents
ACKNOWLEDGEMENTS.................................................................................................................3
ABSTRACT..........................................................................................................................................5
CONTENTS..........................................................................................................................................9
CHAPTER 1 THE ALICE EXPERIMENT................................................................................15
1.1 INTRODUCTION......................................................................................................................15
1.2 LARGE HADRON COLLIDER....................................................................................................17
1.3 ALICE PHYSICS.....................................................................................................................18
1.4 ALICE DETECTOR.................................................................................................................19
1.4.1 Overview.....................................................................................................................19
1.4.2 Time Projection Chamber (TPC)................................................................................20
1.4.3 PHOton Spectrometer (PHOS)...................................................................................21
1.4.4 Other Detectors...........................................................................................................22
1.5 RADIATION ENVIRONMENT OF THE FEE.................................................................................22
1.6 ONLINE SYSTEMS FOR THE ALICE TPC/PHOS ELECTRONICS..............................................23
1.6.1 Experiment Control System.........................................................................................23
1.6.2 Trigger, Timing and Control System..........................................................................24
1.6.3 Data Acquisition System.............................................................................................26
1.6.4 High Level Trigger......................................................................................................28
1.6.5 Detector Control System.............................................................................................28
CHAPTER 2 TPC AND PHOS FRONT-END ELECTRONICS..............................................31
2.1 INTRODUCTION......................................................................................................................31
2.2 TPC FEE TOPOLOGY..............................................................................................................33
2.3 PHOS FEE TOPOLOGY...........................................................................................................34
2.4 FRONT END ELECTRONICS DEVICES.......................................................................................35
2.4.1 TPC Front End Card..................................................................................................35
2.4.2 PHOS Front End Card...............................................................................................36
2.4.3 Readout Control Unit..................................................................................................38
2.4.4 PHOS Trigger Generation Hardware........................................................................42
10
2.4.5 BusyBox......................................................................................................................43
CHAPTER 3 DETECTOR CONTROL SYSTEM FOR THE FEE.........................................47
3.1 INTRODUCTION......................................................................................................................47
3.2 DCS BOARD SOFTWARE.......................................................................................................49
3.2.1 Communication Software on the DCS board.............................................................49
3.2.2 The DCS Board Logical Architecture........................................................................50
3.3 DCS BOARD FIRMWARE........................................................................................................51
3.3.1 Overview and Specification........................................................................................51
3.3.2 DCS board Flavours..................................................................................................52
3.4 RCU COMMUNICATION MODULE..........................................................................................53
3.4.1 Overview....................................................................................................................53
3.4.2 Modes of Operation...................................................................................................54
3.4.3 Definition of the interconnecting DCS-RCU bus lines...............................................56
3.4.4 RCU Interrupt Handling............................................................................................56
3.4.5 MessageBuffer Functional Overview.........................................................................57
3.4.6 MessageBuffer Configuration Block..........................................................................57
3.4.7 DCS Bus Protocol......................................................................................................58
3.4.8 Flash Bus Protocol.....................................................................................................59
3.4.9 SelectMAP Bus Protocol............................................................................................60
3.4.10 Radiation Tolerance Measures..................................................................................60
3.4.11 BusyBox and Trigger-OR Flavour.............................................................................61
3.5 RCU MAIN FPGA IN THE FEE DCS.......................................................................................63
3.5.1 Introduction................................................................................................................63
3.5.2 DCS Interface.............................................................................................................64
3.5.3 Monitoring and Safety Module...................................................................................66
3.6 PHOS FEC BOARD CONTROLLER.........................................................................................68
3.6.1 Introduction and Requirements..................................................................................68
3.6.2 TPC Board Controller................................................................................................69
3.6.3 PHOS Board Controller Introduction........................................................................70
3.6.4 Registers.....................................................................................................................72
3.6.5 Health Monitoring......................................................................................................74
3.6.6 High Voltage APD Bias Settings................................................................................76