Dragon

Table of Contents

High Level Specification

•
What is Dragon?
•
System and Project Goals
•
Introduction to FPGAs
•
Target Hardware Platforms
•
System Components
•
Performance Targets

Primary CPU Specification

•
RISCV Core
•
Instruction and Data Caches

GPU Specification

•
Dragon GPU Frontend
•
Dragon Control Unit
•
Vector Processing Unit
•
Dragon VPU: Core Architecture
•
Dragon VPU: Instruction Set

Design Lessons

•
Design Lessons: High-Level Emulation and Throughput Planning
•
Design Lessons: FPGA Technology
•
Design Lessons: The Dragon GPU

What is Dragon?

Dragon is a "fantasy console", where we imagine that shortly after the Playstation and the Nintendo 64 both came out, Dragon Corp, a new developer, put out a new console. Dragon is purposefully intended to be technologically feasible (mostly) so that it could have existed in the late 1990's, but borrows ideas and concepts which wouldn't become popular for another 5-10 years, primarily general purpose programmable GPUs and shaders.

Dragon is designed by myself (sh4) and uchinokitsune. The project itself is an exploration of what hardware design involves, the complexity and realities of developing a full stack of hardware design, firmware, software, debug tooling, etc. and how all of these fit together.

System and Project Goals

•
Provide 'semi-realistic' hardware which could have existed during the late 1990's
•
Fully-programmable GPU shaders both for compute and graphics workloads.
•
Flexible tile-based rendering.
•
Able to rasterize O(thousands) of triangles at 60 FPS at 320p.
•
Targets execution on real FPGA developer kits, both for developers and some hypothetical user.
•
Zero hardware IP dependencies. We design and implement every UART, Memory controller, GPU Core. Why? Because the main point of this project is to learn and fully understand how hardware really works and what kinds of constraints FPGA environments
•
We implement the full firmware, SDK, and Software environment.
•
Where possible, we can somewhat gracefully scale to faster execution on FPGAs which have more resources, better clock rates, etc.

Introduction to FPGAs

We are never going to afford to produce an ASIC (It's sadly still ridiculously expensive). However, Dragon does run on real physical hardware via Field Programmable Gate Arrays (FPGAs). If you are not familiar with FPGAs, here's a super short introduction to what is inside them.

FPGAs boil down to a giant 2D grid of Cells (many names, another is Programmable Logic Block). What exactly is inside a cell depends on the FPGA manufacturer, but typically there is a Look-up Table (LUT), some single-bit register(s), and some "hard" adder logic. Anything "hard" ("Hard IP") means that literally the silicon in the FPGA has that thing implemented, so it's generally pretty fast. In addition to cells, scattered throughout your FPGA you will typically have some Digital Signal Processor (DSP) elements and "Block RAM" (sometimes called "BRAM(s)"). DSPs can serve a really important function of giving you some hard multiplier capabilities which we obviously need for many things, but especially for doing 3D graphics work. Block RAM is like little islands of hard memory that you can read and write to. Amongst this sea of cells, DSPs, Block RAM, etc. is a huge number of "wires".

Now, something to realize is that whether a LUT has 4 inputs versus 6, the size and modes of access for the block RAMs, the number of Cells, the amount of Block RAM, etc. etc. -- All of these particulars depend both on the manufacturer and the specific FPGA part. Even the naming of these things varies by manufacturer. You can get smaller ($) and larger ($$$) FPGAs which will usually have more DSPs, more cells, .

Now what makes FPGA programmable is that when an FPGA is powered on, it reads its 'configuration' from some other chip, and takes on that design that was loaded. What that means is that every look-up table, every DSP configuration, etc. will be loaded from a file. Because every cell can be loaded with arbitrary data and the wiring allows you to effectively connect cells together in nearly arbitrarily complex ways, FPGAs are capable of turning a hardware design into a physical thing.

The flow for an FPGA developer is something like:

•
Design: A developer implements a logic design in some Hardware Description Language (HDL) such as Verilog, SystemVerilog, VHDL, Chisel, etc. This is a language that can describe the logic without referring (too much) to how exactly this is implemented in a particular "technology".
•
Synthesis: You use a "synthesis" tool which will take your design written in whatever language and transform it into a set of cells, DSPs, memories, etc. that can actually work on your target FPGA platform. Note that this is like a list of resources that a design uses and how they need to be 'connected' but nothing about how they're actually physically arranged in the FPGAs 'grid' of resources.
•
PNR: You now use a "Place and Route" ("PNR") tool which takes the description of things and how they're supposed to be connected, takes a description of an actual FPGA and where the resources are physically located on the chip, and tries to find out how to hook up those real resources in such a way it matches what the synthesis tool created.
•
Clock Rates: The PNR tool will hopefully find some solution, but depending on the solution it could find, the wires and arrangement of cells will have some physical delay which limits how fast signals can travel in the design, which effectively determines the maximum clock rate your design can run. If this is less than what you need/expected, at this point you need to spend a lot of time to figure out where the bottleneck/limitation is in your design to help speed up the design.
•
Debugging: Much like software debugging, you can easily create a lot of bugs in FPGA hardware, and it can be extremely hard to debug sometimes. So there is some debugging-and-fix "loop" just like in software development. I will note that real development of hardware leans heavily into more serious formal verification methods which can literally prove that certain error cases are impossible in your design, etc.

There is so much more to FPGAs, but hopefully this tells you enough to understand the later parts of this document where we reference how/why we made certain design decisions.

Target Hardware Platforms

Today the design runs and 'targets' two different FPGA parts:

•
ECP5-85F - This FPGA is our "smaller" variant. This part from Lattice Semiconductor has 85 thousand cells. The ECP5 family uses 4-input LUT cells. In reality this part will likely become more like a "reduced" version, with software potentially running at a reduced resolution or reduced framerate.
•
Artix 7 A200T - This is our "larger" variant. This part from Xilinx has roughly 200 thousands cells and these are 6-input LUTs.

Developer Notes
From a system developer perspective, the synth+PNR workflow for both of these is different. There are also some slight differences in how memories and DSPs work between these two parts which means we must be careful in how we expose a single hardware design which makes good use of the available hardware.

System Components

Dragon is made up of several key components. Each of these is mentioned here and described in more detail later in the document.

Primary CPU
The Primary CPU is a RISCV-32IM core. We selected this core because it is reasonably simple to implement, reason about, fairly simple to get a good clock rate, because of the RISCV standard it is intentionally very customizable (which we have leveraged), and perhaps most importantly, it meant that we could make use of existing developer toolchains for compiling high level languages into working firmware and software. Originally we experimented on this design utilizing the picorv32 HDL core but have since migrated to our own design. On Dragon, the Primary CPU is nominally executing at 100 MHz.

System Memory
System memory represents the single shared pool of memory which is shared between CPU, GPU, Display Controller, and Audio system. The actual amount of memory depends on the target FPGA development board on which the system is being built for, but as a minimum configuration we expect at least 32MB with SDRAM-like access semantics. Because all of these devices may contend for access at once, there is a priority mechanism in CXB (see below) to control access. Note that in the case of our ECP5 boards, we have implemented our own SDRAM controller with the proper refresh timing etc. but for the A200T target which utilizes much more complex DDR3, handling this memory controller is taken as a later exercise once the rest of the project is already working.

Display Controller
We target GDPI (which is amazingly/suspiciously similar to HDMI without the branding/licensing requirements) for display output. The Display controller is responsible for feeding that display output with proper signal so that a framebuffer sitting in system memory will actually display properly on a display.

GPU: Dragon Control Unit (DCU)
The Dragon GPU is composed of two "sides", a control side and a work side. The Dragon Control Unit (DCU) is a second core identical to the primary CPU and after some setup from the Primary CPU, it is commanded to asynchronously execute a "Control Program" which operates the rest of the GPU which only the DCU can talk to. The Control Program is responsible for looking at the compute and rendering work which has been enqueued from the Primary CPU and ensuring the work side is kept busy operating on those tasks.

GPU: Vector Processing Units
TODO

CXB Fabric
Between the various components of Dragon we need to route requests from one device to another. There is one I/O request/routing system which is somewhat akin to Network-on-chip (NOC). It is called the Cheshire Bus (CXB).

Performance Targets

TODO

RISCV Core

TODO

Instruction and Data Caches

TODO

Dragon GPU Frontend

TODO

Dragon Control Unit

TODO

Vector Processing Units

TODO

Dragon VPU: Core Architecture

TODO

Dragon VPU: Instruction Set

TODO

Design Lessons: High-Level Emulation and Throughput Planning

"Throughput and Latency are Everything"
The most important lesson of all: Almost every design decision in this project has in the end come down to data latency and throughput. As an example "How many triangles can we render at 60 FPS?" becomes a sequence of questions which are ultimately bottlenecked either by the FPGA resources or external IO. Below is an example of how that question is answered. We've performed exercises like this many many times because almost all questions of "will this be fast enough" or "how much geometry can we render" etc. will all come down to bottlenecks in the physical hardware.

•
Based on a basic SDRAM controller we've implemented and tested for the ECP5 that we want to run on, we know that we can sustain ~180MiB/s read or write for contiguous access. If I/O is random then bandwidth is effectively cut in half do to bank switching etc. This is a physical limit and a number that is shared by the entire system.
•
A 320x240 framebuffer that we want to put on the TV has 76,800 pixels. Assuming 16-bits per pixel (ARGB1555 color), that's exactly 150 KiB per frame, or ~8.79 MiB/second of memory bandwidth consumed from system memory, no matter what we do. If we do 640x480, it's 4x that. And this is just for a read, so 2x to account for the write from the GPU assuming a single render pass flushed to system memory.
•
Dragon has a single system memory. Not 4 memories each a quarter of the size or something similar. The implication is that we cannot efficiently parallelize random reads/writes over the framebuffer. You can do some quick napkin math to show that there is no way Dragon can use immediate mode rendering. As the rasterizer is working on a string of e.g. 4 pixels, it would need to read from RAM, do any ROP/compositing, then write back to the framebuffer. Almost every triangle will need to change bank/row as the triangle rasterizer goes to the next line and the next line which will tank efficiency with SDRAM. This strategy also can be arbitrarily wasteful depending on coverage. If 100 objects end up writing to the same pixel, 99 of these writes will be wasted. For these reasons we ended up deciding Dragon must be a Tile-based GPU.
•
We need to send at the absolute minimum 2 or 3 coordinates and some bit of data to describe color. In reality there is more, but let's assume we need the CPU to send the GPU 3x float16 coordinates and maybe 16bits of material information per vertex (64b total). Just to make an argument, assume all vertices are unique, then every triangle is 3x64b=24 bytes which needs to be read by whatever component will do rasterization. Assuming we have 1000 triangles sitting in system RAM that will be 24,000 bytes to be read for vertex data. Triangle index data will be more, say 8 bytes total per triangle. That's (24 vertex + 8 index)*(1000 triangles)*(60 FPS) = ~1.88MiB/sec which is not bad at all from a system bandwidth perspective. Given that the VPU will be doing these operations we also consider the total latency. A single VPU has a 64b port to the CXB to system memory. (32*1000*60)/8 accesses will be required, which equates to approximately 2 milliseconds of time at 100MHz, again assuming constant access is possible (it will not). A single VPU has this kind of small "lightspeed" for data in and out which is another pressure for there to be multiple VPUs on a single GPU when possible.
•
Assume that any 32x32 hardware tile has average coverage of 5 polygons. Hardware tile access is very fast BRAM (effectively SRAM). 32x32x5 is 5120 writes to a tile memory. Assuming we can issue these writes perfectly back-to-back and the VPU core is running at 100MHz, that's 10 nanosecond per clock * (32*32*5 total writes to tile) * (80 total tiles) = 4.096 milliseconds. Let's say we're able to 'fit' 2 VPUs on the target FPGA, then the effective number of tiles done by any VPU will be half, so the total time is then 2.048 milliseconds.
•
From this exercise, we can tell that reading data in and out of VPUs is critical to minimize. When we do, it needs to be DMA of large contiguous regions for best efficiency. Within a VPU we can do quite a lot. Much of the above assumed optimal I/O patterns were possible. Without running the memory access through the simulator/emulator it can be difficult to give exact numbers. Clearly the coverage to any particular tile and average coverage are critical. On a related note, the binner will consume additional time to pass over all geometry, but the outsized benefit is that most tiles will have a much-reduced set of geometry to render.

This will sound obvious, but it's really clear after doing many calculations like this, that chip and interconnections decisions for systems of this era were key decision points that architects had to make. The above calculation also elludes to a very obvious reason why GPUs commonly had dedicated VRAM (avoiding contention with CPU, RAM port sizes optimized to the common I/O size, along with several other reasons).

"Quickly figuring out what is possible"
When we began designing Dragon, before diving into SystemVerilog HDL, we wanted some sense of whether the rasterization, memory access, VPU instructions, etc. could be done fast enough to do interesting rendering on FPGAs like the ones we have. We created some quick C++-based high-level emulator which would simulate performaing various aspects of the system and output a Chrome Profiler trace JSON which would be easily loaded into Chrome or Perfetto for visualization 1. This exercise was critical for convincing ourselves that we had enough ALU and other operations to hide memory accesses, to find out how important binning/tile-based rendering was, etc.

Design Lessons: FPGA Technology

•
Leverage simulation, fuzzing, and formal verification wherever possible. TODO (verilator testing)

Design Lessons: The Dragon GPU

TODO