Build Your Own Programming Language: A programmer's guide to designing compilers, interpreters, and DSLs for solving modern computing problems

495 Pages·2021·5.76 MB·

by  Clinton L. Jeffery

Build Your Own
Programming
Language
A programmer's guide to designing compilers,
interpreters, and DSLs for solving modern
computing problems
Clinton L. Jeffery
BIRMINGHAM—MUMBAI
Build Your Own Programming Language
Copyright © 2021 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
or transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and
distributors, will be held liable for any damages caused or alleged to have been caused directly
or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies
and products mentioned in this book by the appropriate use of capitals. However, Packt
Publishing cannot guarantee the accuracy of this information.
Group Product Manager: Richa Tripathi
Publishing Product Manager: Alok Dhuri
Senior Editor: Storm Mann and Nisha Cleetus
Content Development Editor: Nithya Sadanandan
Technical Editor: Karan Solanki
Copy Editor: Safis Editing
Project Coordinator: Deeksha Thakkar
Proofreader: Safis Editing
Indexer: Subalakshmi Govindhan
Production Designer: Nilesh Mohite
First published: November 2021
Production reference: 1231121
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-80020-480-5
www.packt.com
This book is dedicated to Susie, Curtis, Cary, and everyone who builds their
own programming language.
– Clinton L. Jeffery
Contributors
About the author
Clinton L. Jeffery is Professor and Chair of the Department of Computer Science and
Engineering at New Mexico Institute of Mining and Technology. He received his B.S.
from the University of Washington, and M.S. and Ph.D. degrees from the University of
Arizona, all in computer science. He has conducted research and written many books and
papers on programming languages, program monitoring, debugging, graphics, virtual
environments, and visualization. With colleagues, he invented the Unicon programming
language, hosted on the Unicon website.
About the reviewers
Phillip Lee is a Peace Corps Volunteer in Sarawak, Malaysia. He has a BA from the
University of Oregon, MA, a PhD from the University of Washington, a graduate
degree in Malay/Indonesian literature from Universiti Sains Malaysia, and a graduate
degree in computing from Murdoch University, Perth. He has taught undergraduate
and graduate classes at Auckland University and Murdoch University. Philip has
publications on Latin, Greek, Malay, and Indonesian literatures. He is a co-programmer
of the Library of Congress thomas.loc.gov Congressional search system and a
co-programmer of the National Library of Medicine's toxnet.nlm.nih.gov system.
Additionally, he works as a developer of text analysis programs for the Tun Jugah
Foundation's English-Iban dictionary.
Steve Wampler has a PhD in Computer Science from the University of Arizona. After
which, he was the Associate Prof. of Computer Science from 1981 to 1993. Steve has
worked as a software designer for several major telescope projects, including the Gemini
8m Telescopes Project and the Daniel K Inouye Solar Telescope, under the Association
for Research in Astronomy. Along with that, he was a software reviewer for a number of
major telescopes, including LSST, TMT, GMT, Keck, ESO's VLT, and Spain's GTC. Steve
was the technical reviewer for the first edition of Mark Sobell's A Practical Guide to the
Linux Operating System, 1997.

Table of Contents
Preface
Section 1: Programming Language
Frontends
1
Why Build Another Programming Language?
So, you want to write your own Case study – requirements that
programming language… 4 inspired the Unicon language 13
Types of programming language Unicon requirement #1 – preserve
implementations 5 what people love about Icon 13
Organizing a bytecode language Unicon requirement #2 – support
implementation 6 large-scale programs working on
Languages used in the examples 8 big data 14
Unicon requirement #3 – high-level
Language versus library –
input/output for modern applications 14
what's the difference? 9
Unicon requirement #4 – provide
Applicability to other software universally implementable system
engineering tasks 10 interfaces 15
Establishing the requirements
Summary 16
for your language 11
Questions 16
2
Programming Language Design
Determining the kinds of words Specifying the control flow 21
and punctuation to provide in Deciding on what kinds of
your language 18 data to support 23
viii Table of Contents
Atomic types 23 Case study – designing graphics
Composite types 24 facilities in Unicon 28
Domain-specific types 25 Language support for 2D graphics 29
Adding support for 3D graphics 31
Overall program structure 26
Completing the Jzero Summary 32
language definition 27 Questions 33
3
Scanning Source Code
Technical requirements 36 Expanding our example to
construct tokens 48
Lexemes, lexical categories,
and tokens 37
Writing a scanner for Jzero 51
Regular expressions 38
The Jzero flex specification 51
Regular expression rules 38 Unicon Jzero code 54
Regular expression examples 39 Java Jzero code 57
Running the Jzero scanner 62
Using UFlex and JFlex 40
Header section 41 Regular expressions are not
Regular expressions section 42 always enough 63
Writing a simple source code scanner 42 Summary 67
Running your scanner 45 Questions 68
Tokens and lexical attributes 47
4
Parsing
Technical requirements 70 Putting together the yacc context-free
grammar section 77
Analyzing syntax 71
Understanding yacc parsers 78
Understanding context-free
Fixing conflicts in yacc parsers 81
grammars 71
Syntax error recovery 82
Writing context-free grammar rules 72
Putting together a toy example 82
Writing rules for programming
constructs 74 Writing a parser for Jzero 88
The Jzero lex specification 88
Using iyacc and BYACC/J 76
The Jzero yacc specification 88
Declaring symbols in the header section 76
Unicon Jzero code 93
Table of Contents ix
Java Jzero parser code 96 Adding detail to Java syntax
Running the Jzero parser 97 error messages 100
Using Merr to generate better
Improving syntax error
syntax error messages 100
messages 99
Summary 101
Adding detail to Unicon syntax
error messages 99 Questions 102
5
Syntax Trees
Technical requirements 104 Building internal nodes from
Using GNU make 104 production rules 117
Learning about trees 107 Accessing tree nodes on the value stack 117
Using the tree node factory method 119
Defining a syntax tree type 108
Parse trees versus syntax trees 110 Forming syntax trees for the
Jzero language 121
Creating leaves from
terminal symbols 112 Debugging and testing your
syntax tree 129
Wrapping tokens in leaves 112
Avoiding common syntax tree bugs 129
Working with YACC's value stack 113
Printing your tree in a text format 130
Wrapping leaves for the parser's
value stack 114 Printing your tree using dot 132
Determining which leaves you need 116
Summary 139
Questions 139
Section 2: Syntax Tree Traversals
6
Symbol Tables
Establishing the groundwork Creating and populating
for symbol tables 144 symbol tables for each scope 147
Declarations and scopes 144 Adding semantic attributes to
Assigning and dereferencing variables 146 syntax trees 148
Choosing the right tree traversal for Defining classes for symbol tables
the job 146 and symbol table entries 150
Creating symbol tables 152