Disclaimer. This is an investigation into what is possible. Not a fully-blown solution into how to reload modules without restarting the whole server.

In a previous article, I was investigating how to use require.cache to refresh a module by deleting it and requiring it again. eg.

require('./some-module')

// use module

delete require.cache[require.resolve('./some-module')]

// use reloaded module

However, on its own, I can’t really think of a good use case to delete a cached module like that. My aim was to attempt to reload a module without restarting the server – mainly to reduce the wait time. And it works! To a degree.

fs.watchFile

So I have a child.js module that doesn’t export anything.

// child.js
console.log('hi', Date.now())

I then have a main.js that imports the child.js.

const fs = require('fs')
require('./child')

fs.watchFile(require.resolve('./child.js'), () => {
    console.log('refresh')
    delete require.cache[require.resolve('./child')]
    require('./child')
})

Running node main.js with fs.watchFile kept the server running on its own. So every time I made a change to the file, it would refresh the child module as I changed the logged output.

The only downside is that I noticed a little bit of lag when using fs.watchFile. It didn’t pick up all the changes. So I tried fs.watch.

Using fs.watch instead of fs.watchFile

The child.js file was the same. The main.js was the same with everything except fs.watchFile was replaced with fs.watch.

And the lag was gone. Changes were reflected instantaneously.

Now the reason for this is because fs.watchFile uses polling. And the Node docs recommend using fs.watch instead.

Using fs.watch() is more efficient than fs.watchFile and fs.unwatchFile. fs.watch should be used instead of fs.watchFile and fs.unwatchFile when possible. - https://nodejs.org/docs/latest/api/fs.html#fswatchfilefilename-options-listener

Caveats

The fs.watch API is not 100% consistent across platforms, and is unavailable in some situations.

An ExpressJS server

So now, how could I potentially get module reloading when a file changes without restarting the entire server?

First. Here’s a very basic Express.js server.

// -- child.js
module.exports = (req, res) => {
    res.send('Hello, World!');
}

// -- main.js
const express = require('express');
const handle = require('./child');

const app = express();

app.get('/', handle);

const PORT = 3000;
app.listen(PORT, () => {
    console.log(`Server is running on http://localhost:${PORT}`);
});

Now, when I run the server doing node main.js, and make a request to the root path, http :3000 (using HTTPIE), I get my “Hello, World!” response.

But without Nodemon or some other way of watching the file, changes to the file won’t be reflected. So let’s change that.

1. The main.js file has to be adjusted so that the child.js module is lazily-loaded when the root path is requested.

2. A file watcher was added to watch changes to the child.js.

const express = require('express');
const fs = require('fs');

fs.watch(require.resolve('./child.js'), () => {
    delete require.cache[require.resolve('./child')]
    require('./child')
})

const app = express();

app.get('/', (...args) => {
    require('./child')(...args)
});

const PORT = 3000;
app.listen(PORT, () => {
    console.log(`Server is running on http://localhost:${PORT}`);
});

And voila 🎉!

I made a change to the file, it was reloaded without restarting the server!

And for development, this is fine. We can use a variable to have reloading in a development only environment running it like MODULE_RELOAD=true node main.js .

const express = require('express');
const fs = require('fs');
const handle = require('./child');

const app = express();

if (process.env.MODULE_RELOAD) {
    fs.watch(require.resolve('./child.js'), () => {
        console.log('Server reloaded')
        delete require.cache[require.resolve('./child.js')]
        require('./child')
    })

    app.get('/', (...args) => {
        require('./child')(...args)
    });
} else {
    app.get('/', handle);
}

const PORT = 3000;
app.listen(PORT, () => {
    console.log(`Server is running on http://localhost:${PORT}`);
});

Of course, loading one file route path isn’t all that helpful either. But, I did say this was an investigation 😉.

So my problem: modelling visa checklists

I built the frontend to look like this:

So users can:

1. Create own checklists for visa documents.

2. They can add and remove documents required for the visa.

3. And then for each document, they can outline the steps needed for it to be completed.

Thinking this through, in terms of tables, you’d need:

1. A table for the visa itself.

2. A table for the documents.

3. Then a table for the steps.

4. And potentially a pivot table if I wanted to really normalise things.

Now three or four tables is nothing to complain about. But I was starting to question whether it’s worth modelling a very simple feature this way in the database.

• But then, if users can add steps and remove them at will, and add documents and remove them at will, now, imagine the database queries to do that.

• We’d need a transaction to first add or remove documents from the table, and then add or remove documents from the steps, especially if multiple people can edit that.

• And while I’m definitely not an expert, hard deleting from a database has to be treated carefully so as to not cause gap locks. Gap locks then cause concurrency problems. Now we have to handle retries. It’s just a pain.

Realising I was creating configuration data

So, naturally, I thought, that’s a lot of work for the database to do for basically what is configuration. And there’s literally no benefit because I don’t any indices on configuration data that I’d need. I don’t need to normalise it. Yes, I’d be repeating data to some degree, but not enough for me to worry about space issues.

Even more so, when it came to the UI, it was super easy to just build the documents using pure Vue.js and that naturally created a JSON data structure.

I was able to send the JSON to the backend endpoint, validate the shape and required step types using PHP enums, and then simply save that data in the backend as is, pure JSON!

And then, sending the JSON object back to the front end meant that Vue.js, simply hydrated the component that managed the visa documents with zero fuss. No translation or restructuring of data was needed between the front end or the backend. The backend saved JSON. The front end hydrated it. Voila!

What if indices become a problem?

Now, if indices were to somehow become a problem, there are ways around it.

For example, we can use generated columns in MySQL and thankfully in SQLite. I learnt that from Aaron Francis’ article on Planetscale. If you’re using Postgres, then apparently you can create indices more easily with JSONB and Gin.

So hopefully I’m forgiven for using JSON in relational SQL. And if you’re interested, you can find the functional for visa documents on Melimundo.com where the plan is to help people prepare for things like the Digital Nomad Visa in Spain. It’s a work in progress but I’m happy about that functionality.

Thanks for reading!

I love Nuxt probably just as much.

And Vue.js ❤️.

And client-side apps in general.

And Serverless. And Containers. And Kubernetes!

And the list goes on.

...so why go back to basics? Really, do I need all of that, for calculators?

Some reasons for a minimal stack

Preventing dependency drift

Reinventing the wheel is a great learning exercise. However, it's better to leverage dependencies so we have more time to work on our core business problems.

Now, saying that, frameworks usually come with a tonne of dependencies that we may not even need. Now that was the case for my project. I just didn't need all these dependencies.

Maintaining dependencies isn't noticeable when you're continually working on a code base. Every 30 minutes here and there seems negligible. But it adds up.

Usually, with dependency heavy projects, when I come back to a code base after some time, I have to inevitably run a build or install the packages afresh, at which point, something is broken. Now there are ways to try and mitigate this but it doesn't always work, like using version locks, or Nix for the OS.

Focusing on logic

It's enjoyable. Learning and becoming proficient at a framework is truly one of life's joys as a developer. But when time is limited and a new feature needs to be created, I had to decide. What do I want to challenge myself with? The framework? Or the logic?

So for this particular project, most files are just .mjs files, ran with the Bun runtime.

Why? Financial calculators are just that. Calculators.

I had to write tests. Also, using bun:test instead of the usual test library, minimising dependencies again. Although the outputs with Vitest or Jest are a lot prettier.

So whether I do FIFO, LIFO or weighted average cost methods for calculating realised gains and losses, all I had to create were blackboxes of logic. No framework required. Just pure functionality.

Building quickly

I love client side frameworks, especially as I started off a front end developer. But I found working with HTMX weirdly efficient as most of my logic lives on the server for this project. When I needed a tiny bit of interactivity and dynamic data, I didn't see any reason to reach for Vue.js or even React!

Now if I needed the power of a JavaScript framework, I would certainly have had fun building with those. Or if I was developing a project with mostly third-party APIs, then of course. But for this project, I had no reason for it.

Being a solo dev

Now there's a downside to being a solo dev. I can build something that only I will ever understand. And so using frameworks would give me a standard way for doing things. Without a framework, the onus is then on me to make sure I structure and document the code in a way that's easy for my future self to understand as well as other developers and that is extra time and thought.

So to get around building a confusing code base, my simple gotos were:

1. Modularising the code into small replaceable/deletable chunks that use the Unix philosophy. And sorry about the swearing at the beginning of the video. This might even mean some duplication.

   %[https://www.youtube.com/watch?v=1FPsJ-if2RU]

2. Documenting inputs and outputs that leveraged IntelliSense and type checks. That meant for me, writing code was super simple as IntelliSense would highlight what I'd need to write.

3. Keeping app and infrastructure separate from core logic. This is a principle found in the hexagonal architecture, and the aim is to be able to replace the framework in future, which I've never had to do. But the main reason is it makes the core logic easy to test. If there's not much pure logic, there's no need to do this.

4. Writing contextual code. My calculators are not pure mathematical calculators. They're contextual to order objects in my project. I also don't worry about input validation for every function because I believe the boundaries between user input and the core functions should validate what comes in.

Now am I against frameworks and dependencies?

No. I'm building another project using Laravel Inertia which makes use of Vue.js, because the ecosystem is so rich, and I didn't want to re-invent the wheel. I knew I'd need to integrate subscriptions, notifications, websockets or at least, server sent events, background queues, migrations – the list goes on.

I'd be crazy to recreate a bad version of a framework myself when I could just leverage something out there already, even if that means running into dependency issues.

The only thing I'd try to do, is limit the number of dependencies I add on top of Laravel to make it easier to maintain.

And what's the stack?

• Node.js

• HTMX

• Alpine.js

Maybe some would say this more complicated. But given I didn't know HTMX and Alpine.js when I started the project,

So the financial calculators

So I've built a very basic free fifo calculator for stocks and options as well as using other accounting methods. And literally, I didn't have to fight a framework or any dependency.

So, frameworkless forever?

While modern frameworks and tools offer immense power and convenience, they can also introduce unnecessary complexity, especially for projects with straightforward requirements. By returning to a minimal stack, we can focus on the core logic and deliver value without getting bogged down by dependencies. This has been a liberating project for me, and I hope you get to enjoy a project with minimal dependencies too. But 100% frameworkless isn't realistic either for all projects.

So first, I'll do a quick overview and then share the explanations afterwards.

Quick Overview

There are several ways to dump and restore data in Postgres if you have more specialist needs. I've chosen one almost analogous to how I would do it with MySQL.

Requirements

You need the correct privileges to both dump the database and recreate all the database objects.

Set the user credentials

Instead of passing options to pg_dump and psql, we can also set up our access credentials as environment variables. There are more secure alternatives to handling database access secrets though.

export PGHOST=<host>
export PGUSER=<user>
export PGPASSWORD=<password>

Dumping the data

Notice how we dump the data with the --no-privileges and --no-owner. Don't include this if you want to keep the ownership as is.

pg_dump \                                                                          
  --no-privileges \
  --no-owner \ 
  dbname > dbname.sql

Creating the new database

Create a new database from a template.

psql \
    -c "CREATE DATABASE newdb TEMPLATE template0;"

Restoring the data

Notice I used --single-transaction to import the data. If you don't mind importing the data where there are some errors, remove that option.

psql \
    --single-transaction \
    new_db < dbname.sql

Changing table ownership

Finally, when working with MySQL, I don't have to worry about table ownership. I just use grants and privileges. But when you import data in Postgres, it assumes the owner is the user associated with the connection. One way I've seen how to change table ownership to one we want is to generate the ALTER commands and then run that.

psql newdb

Then within the psql shell, we create the ALTER commands.

## :psql >>

select 'ALTER TABLE ' || t.tablename || ' OWNER TO new_owner;' 
 from  pg_tables t
 where t.tableowner != 'rdsadmin';

It will print out something like the following, which you will need to copy-paste, and run within the shell again.

## :psql >>

ALTER TABLE groups OWNER TO new_owner;
ALTER TABLE images OWNER TO new_owner;
ALTER TABLE jobs OWNER TO new_owner;

And that's it.

• We've dumped the database.

• We've restored it.

• And we've changed the table ownership to what we want.

Now, I'll share a few things I found in the documentation.

Notes on how the dump and restore work

Libpq

Libpq is the "C application programmer's interface to PostgreSQL". So I needed to install Libpq before being able to run some commands on my local environment. For me on MacOS, it was not installed with Postgres and had to be done separately.

Table locks

With MySQL, dumping the database with table locks means users may experience slow responses or even downtime. To prevent table locks in MySQL, we can do the following:

mysqldump \
    --single-transaction \
    --skip-lock-tables \
    dbname > dbname.sql

The --single-transaction tells the mysqldump to "put everything into a transaction", and "read the database in the current state and create a consistent data dump". But it will still lock the tables unless we provide --skip-lock-tables. That way, we get a consistent data output and no table locks.

↗️ Run mysqldump without locking the tables - https://mysqldump.guru/run-mysqldump-without-locking-the-tables.html

In Postgres, this is not a problem as "dumps created by pg_dump are internally consistent, that is, updates to the database while pg_dump is running will not be in the dump. pg_dump does not block other operations on the database while it is working." Therefore, no extra options are needed. Of course, there are exceptions.

↗️ Chapter 9. Backup and Restore - https://www.postgresql.org/docs/7.2/backup.htm

Using options instead of environment variables for connection settings

Similar to MySQL, we have -h host, -p port , but we have -W password for the prompt and a capital U for -U user.

Removing ownership

If you want the table ownership and privileges to remain the same, providing --no-privileges and --no-owner is not necessary. However, without this, you get the following SQL in your SQL dump:

CREATE SCHEMA abc;

ALTER SCHEMA abc OWNER TO existing_user;

Database Templates

Database Templates in Postgres is a foreign concept to me in MySQL. However, when we do, CREATE DATABASE in Postgres, it's an alias for CREATE DATABASE dbname TEMPLATE template1.

Now, there are two templates by default. template0 and template1. In the earlier example, I used template0, but why?

• template1 can be amended so you can create a new database with your customisations.

• template0 doesn't have any customisations. So it's a pure, unadulterated database that you can clone and copy.

All-or-nothing imports

Something interesting we can do in Postgres is ensure our imported data is consistent. If there are any errors when importing the dumped file, it will roll back all changes. To do this, we use --single-transaction on the import:

psql \
    --single-transaction \
    < dumpfile.sql

Note: "When using this mode, be aware that even a minor error can rollback a restore that has already run for many hours. However, that might still be preferable to manually cleaning up a complex database after a partially restored dump." - postgresql.org

That wasn't so bad

MySQL and Postgres are similar but they're not the same. Hopefully dumping and restoring in Postgres will be simpler (even if it's just for me) in future.

However, I've discovered Hledger in strict mode makes things easier.

Catching typos

I can make lots of silly mistakes in plain text. For example, when I finally came to closing the accounts, I had inconsistencies due to typos in my accounts. It was simple enough to fix them, but imagine handling hundreds of transactions with these errors!

Entering correct information the first time is far easier than trying to fix things after the fact. Thankfully with strict mode, we can't have little accidents like this.

By adding -s, we can get Hledger to check our accounts for us. Here's an example of an error message when I made a typo under strict mode doing hledger web -s:

hledger-web: Error: main.journal:15:
   | 2023-09-02 Advertising
15 |     expense:advertising             10 GBP
   |     ^^^^^^^^^^^^^^^^^^^
   |     assets:bank                    -10 GBP

Strict account checking is enabled, and
account "expense:advertising" has not been declared.
Consider adding an account directive. Examples:

account expense:advertising
account expense:advertising    ; type:A  ; (L,E,R,X,C,V)

Changing account order in Web mode

By default in Hledger, accounts in web mode are ordered as:

1. Assets

2. Equity

3. Expense

4. Income

5. Liabilities

However, I noticed my expenses account was enormous in comparison to all the other accounts. With strict mode, as we have to declare accounts upfront, Hledger will use whatever order you declare.

So, I changed the order to this:

1. Liabilities

2. Income

3. Equity

4. Assets

5. Expenses

How to order accounts in Hledger

You have to declare the top-level account eg. account expenses and not just account expenses:manufacturing before any transaction that uses them. I did it top-level accounts with each of their subaccount types together.

; Liabilities
account liabilities 
account liabilities:loan:director

; Income
account income
account income:bank-coupon

; Equity
account equity
account equity:opening
account equity:share-capital

; Assets
account assets
account assets:bank
account assets:equipment

; Expenses
account expenses
account expenses:manufacturing

; Commodities
commodity 1.00 GBP

But nothing is stopping you from grouping the top-level accounts first, and then all your sub-accounts afterwards.

account liabilities
account income
account equity
account assets
account expenses

; Liabilities 
account liabilities:loan:director

; Income
account income:bank-coupon

...etc

Managing account declarations in a separate file

If all of these accounts are too verbose for the main transactions file, you can move it out into a separate file and include it back. That keeps it nice and organised.

include accounts.journal

Always keeping Strict Mode on

If you have a new journal, adding strict mode will probably be a lot easier than having to retrofit it into an old journal. It might not be worth the effort. But for new journals, I've decided to always have strict mode by creating a bash alias. So that whenever I run Hledger, I catch silly issues upfront. As I use zsh, I added the alias to my zsh profile:

echo 'alias h="hledger -s"' >> ~/.zshrc

Callbacks are 4.8x faster when running them parallel over async/await in parallel. And only 1.9x faster when we run sequential callbacks.

I've modified this article somewhat after I got some helpful and kind comments about my dodgy test. 😂🙏

Thank you to Ricardo Lopes and Ryan Poe for taking the time to steer the benchmarks in the right direction. My first faux pas was I wasn't actually waiting for the execution of the code to finish, which crazily skewed the results. The second was I was comparing parallel to sequential runtimes, which make the benchmarks worthless.

So this is round 2 which addresses my initial errors. Previously, I said:

NodeJS: 34.7x faster if we go back to callbacks! I shouldn't have believed it. 🤣

Not as impressive as my bad benchmarks before (and see comments for context), but still a sizeable difference.

So what exactly did I test?

I compared callbacks to promises and async/await when reading a file, 10,000 times. And maybe that's a silly test but I wanted to know, which is faster at I/O.

Then I finally compared callbacks in Node.js to Go!

Now, guess who won?

I won't be mean. TLDR. Node.js callbacks Golang!

Lower is better. Results are in ms.

Now, true benchmarkers? Go easy on me on this one. But please do leave your comments to make me a better person.

Everyone keeps saying that Node.js is slow!

And it bugs me out.

Because what does slow mean? As with all benchmarks, mine is contextual.

I started reading about the Event Loop, just to even begin to understand how it works.

But the main thing I've understood is that Node.js passes I/O tasks onto a queue that sits outside the main Node.js executable thread. This queue runs on pure C. A number of threads could potentially handle these I/O operations. And that's where Node.js can shine, handling I/O.

Promises, however, get handled in the main, single executable thread. And async/await, is well, promises but now with blocking added.

So are callbacks faster than promises?

Let's put it to the test.

First off. My machine! Complements of working with Kamma. It's important to note what resources we're working with. Plenty memory and CPU.

MacBook Pro (14-inch, 2021)
Chip      Apple M1 Pro
Memory    32 GB
Cores     10
NodeJS    v20.8.1
Go        1.21.0

So we have a text.txt file with an original message, Hello, world.

echo "Hello, world" > text.txt

And we'll read this text file using native Node.js, which means, zero node module dependencies because we don't want to drag node modules down with the heaviest objects in the universe.

Callbacks

Parallel callbacks

First, let's start with parallel callbacks. I'm interested in how quickly the same file can be read as quickly as possible, all at once. And what's faster than parallel?

// > file-callback-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs";

test('reading file 10,000 times with callback parallel', (t, done) => {
    let count = 0;
    for (let i = 0; i < 10000; i++) {
        fs.readFile("./text.txt", { encoding: 'utf-8'}, (err, data) => {
            assert.strictEqual(data, "Hello, world");
            count++
            if (count === 10000) {
                done()
            }
        })
    }
});

Sequential callbacks

Second, we have callbacks again, but sequential (or rather blocking). I'm interested in how quickly the same file can be read sequentially. Having not done callbacks calling callbacks for ages, this was fun to try again. Albeit, it doesn't look pretty.

// > file-callback-blocking.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs";

let read = (i, callback) => {
    fs.readFile("./text.txt", { encoding: 'utf-8'}, (err, data) => {
        assert.strictEqual(data, "Hello, world");

        i += 1

        if (i === 10000) {
            return callback()
        }

        read(i, callback)
    })
}

test('reading file 10,000 times with callback blocking', (t, done) => {
    read(0, done)
});

Async/Await

Then we have async/await. My favourite way of working with Nodejs.

Parallel async/await

It's as parallel as I can get with async/await. I load all the readFile operations into an array and await them all using Promise.all.

// > file-async-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with async parallel', async (t) => {
    let allFiles = []
    for (let i = 0; i < 10000; i++) {
        allFiles.push(fs.readFile("./text.txt", { encoding: 'utf-8'}))
    }

    return await Promise.all(allFiles)
        .then(allFiles => {
            return allFiles.forEach((data) => {
                assert.strictEqual(data, "Hello, world");
            })
        })
});

Sequential Async/Await

This was the easiest and most concise one to write.

// > file-async-blocking.test.mjs

import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with async blocking', async (t) => {
    for (let i = 0; i < 10000; i++) {
        let data = await fs.readFile("./text.txt", { encoding: 'utf-8'})
        assert.strictEqual(data, "Hello, world");
    }
});

Promises

Finally, we have promises without async/await. I've long stopped using them in favour of async/await but I was interested in whether they were performant or not.

Parallel promises

// > file-promise-parallel.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with promise parallel', (t, done) => {
    let allFiles = []

    for (let i = 0; i < 10000; i++) {
        allFiles.push(fs.readFile("./text.txt", { encoding: 'utf-8'}))   
    }

    Promise.all(allFiles)
        .then(allFiles => {
            for (let i = 0; i < 10000; i++) {
                assert.strictEqual(allFiles[i], "Hello, world");
            }

            done()
        })
});

Sequential promises.

Again, we want to wait for the execution of all readFile operations.

This doesn't seem quite right as it's technically still not waiting for each to finish.

// > file-promise-blocking.test.mjs
import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

test('reading file 10,000 times with promises blocking', (t, done) => {
    let count = 0;
    for (let i = 0; i < 10000; i++) {
        let data = fs.readFile("./text.txt", { encoding: 'utf-8'})
            .then(data => {
                assert.strictEqual(data, "Hello, world")
                count++
                if (count === 10000) {
                    done()
                }
            })
    }
});

How I ran the tests

And voila! Results 🎉! I even ran it a few times to get a better reading.

I ran each test by doing:

node --test <file>.mjs

Reading a file 10,000 times with callbacks is over 34x 5.8x faster than with async/await in parallel! It's also 4.7x faster than with promises in parallel!

So, in Node.js land, callbacks are more performant!

Now is Go faster than Node.js?

Well, I don't write in Go, so this may be truly terrible code because I asked ChatGPT to help me and yet, it seems pretty decent.

Hey ho. Let's go. Our Golang code.

package main

import (
    "fmt"
    "io/ioutil"
    "time"
)

func main() {
    startTime := time.Now()

    for i := 0; i < 10000; i++ {
        data, err := ioutil.ReadFile("./text.txt")
        if err != nil {
            fmt.Printf("Error reading file: %v\n", err)
            return
        }
        if string(data) != "Hello, world" {
            fmt.Println("File content mismatch: got", string(data), ", want Hello, world")
            return
        }
    }

    duration := time.Since(startTime)
    fmt.Printf("Test execution time: %v\n", duration)
}

And we run it as so:

go run main.go

And the results?

Test execution time: 58.877125ms

🤯 Go is 4.9x faster than Node.js using sequential callbacks. Node.js only comes close with parallel execution.

Node.js Async/await is 9.2x slower than Go.

So yes. Node.js is slower. Still, 10,000 files in sub 300ms isn't to be scoffed at. But I've been humbled by Go's speediness!

Now just a side note. Do I have bad benchmarks?

I really did have terrible Benchmarks. Thank you again to Ricardo and Ryan.

Yes, I did. Hopefully now they're better.

But you may ask, who's really going to read the same file, over and over again? But for a relative test between things, I hope it's a helpful comparison.

I also don't know how many threads Node.js is using.

I don't know how my CPU cores affect Go vs Node.js performance.

I could just rent an AWS machine with one core and compare.

Is it because I'm on Mac M1?

How would Node.js perform on a Linux or...Windows? 😱

And there's the practicality of, yes, reading a file is one thing, but at some point, you have to wait anyway for the file to be read to do something with the data in the file. So, speed on the main thread is still pretty important.

Now, do you really want to use callbacks?

I mean, do you really, really want to?

I don't know. I definitely don't want to tell anyone what to do.

But I like the clean syntax of async/awaits.

They look better.

They read better.

I know better is subjective here but I remember callback-hell, and I was grateful when promises came into existence. It made Javascript bearable.

Now, while Golang in this instance is slower Golang is clearly faster than Node.js at its optimum, with callbacks, and with async/await, by 9.2x! So if we want good readability and performance, Golang is the winner. Although, I'd love to learn how Golang looks under the hood.

Anywho. This was fun. It was more of an exercise to help me understand how callbacks and I/O work in the Event Loop.

So to sign out

Is Node.js slow? Or are we just using Node.js on slow mode?

Probably where performance matters, Golang is worth the jump. I'll certainly be looking more at using Golang in future.

Updates

Sequential promises

If I rewrite it as below, I'm still using callbacks 😬 but at least it's sequential:

import test from 'node:test';
import assert from 'node:assert';
import fs from "node:fs/promises";

let read = (i, callback) => {
    let data = fs.readFile("./text.txt", { encoding: 'utf-8'})
        .then(data => {
            assert.strictEqual(data, "Hello, world")
            i += 1

            if (i === 10000) {
                return callback()
            }

            read(i, callback)
        })
}

test('reading file 10,000 times with promises blocking', (t, done) => {
    read(0, done)
});

I didn't include the above in the benchmarks but it is slower and not surprisingly so at 532.777667ms.

That's it.

Time blocking might work for you. That's great, genuinely. If it doesn't, here's my alternative.

Firstly, why do I care about this?

I'm going to deviate a little bit from talking about software engineering. It's how I manage my time and get things done inside and outside of work. At least, I've been doing this since September 2023.

This may change, but for now, I'm railing against the idea that productivity requires time-blocking.

I also note, that my aim is to reduce stress while achieving my goals, and for the first time, I'm managing to inch towards things I've been dreaming about for years, like moving to another country, and doing so without trying to skimp on sleep and relaxation.

I actively avoid time-blocking as much as I can.

Natural time-blocks are necessary

While I try to avoid time-blocking unnecessarily, I'd be delusional to say that it's not needed. It's absolutely essential.

Time blocks are a natural part of life.

Anytime we need to be somewhere at the same time as someone else, that's a time block.

Think lectures at school.

Think about being at work, whether remote or in the office.

Think about synchronous meetings.

Think about weddings, funerals, the World Cup and meals with friends.

Think about sleep, even.

If you want to enjoy life in real-time with others; time block.

If it's just you doing work, by yourself, why time block?

Natural time blocks aren't bad. They're about congregating people around an event at a specific time.

The event starts. The event ends. The time is done.

But if it's just me working, what am I time-blocking for?

What do I mean by the time blocking?

So something else I need to clarify is what I mean by time blocking.

I don't just mean, setting a generic block of time in which to do anything. I mean, time-blocking to achieve a certain thing.

For example:

Now this has a lot of things going for it:

1. It looks very organised.

2. It's obvious what I'm supposed to be working on and when.

3. There is a clear structure to my day.

But there are a few problems with this.

• Stress!

• It assumes I can complete certain tasks by a certain time.

• It assumes I won't be required at several points in the day to assist team members and customers with their queries.

• It assumes my meeting will only last 30 minutes.

• It assumes that I can't finish my task way within time and move on to something else.

• It assumes I'm mentally and physically ready and optimised to work on those tasks at those times.

So let me go into those problems a little more.

Time blocks aren't flexible enough to work with my changes in energy

Time blocking just doesn't allow for the ebbs and flows of the day.

Everyone's day in many ways is the same. Yet, we're all different.

Sometimes I'm full of energy first thing. Other times I'm groggy. I might want a light and simple set of todos to get going. I might be ready to get my head down and tackle something hard that's been on my mind from the night before.

But time blocks are just too inflexible to take advantage of this natural ebb and flow of energy. It doesn't care if you're optimised to deliver 100% creative performance now. You have to wait for that time-block later.

An example from Mohammed Ali

Now this seems like a deviation from time blocks, but it's a good reminder for me. When asked about how many push-ups he could do, see his answer below (or here if the embed doesn't work):

Notice, he only started counting the push-ups when they began to hurt.

It reminds me of non-personalised training advice. You see, some may say, do 2 minutes of running followed by 1 minute of walking, and repeat for 30 minutes and then increase running time until you get good at it. But what if that's too hard to start at that point? What if that's too easy? Who knows.

Another trainer instead, may check your heart rate has reached a certain maximum, then he'll make you walk until it you heart rate is steady at a certain level again. Your recovery rate might be 1 minute, it might be 15 minutes. But it's based on your energy levels and recovery rate, and not just blocks of time.

Time blocks cause unnecessary stress

Deadlines cause stress.

If a deadline is from a customer, from investors, from the tax man, from our boss, fair enough, deadlines are inevitable. Stress is also good for growth. A little bit of stress during exercise makes us fitter and stronger. But stress isn't great for our health when it's chronic.

We've heard people say "I need to have a car by the time I'm 18." I need to be earning $1,000,000 by the time I'm 30. I need to own my house by the time I'm 35! Self-imposed deadlines cause stress. I need to fit into that dress before the wedding! It can be a great motivator for the big things.

But why increment unnecessary stress throughout your day, weeks and years for little tiny tasks you won't even remember tomorrow, minute by minute, and hour by hour?

If I've said, between 10:00-12:00, I do task A, then from 12:00-14:00, I do task B, what happens when it's 12:00, and I haven't finished task A? Stress!

You could say, well, maybe just work faster.

But what does faster mean?

Do I need to? Is there a deadline? Do I want to do a shoddy job? Do I want to cut corners? Is it not important, so it doesn't matter if the quality is not good or not? Do I just deliver part of the work and be done with it?

Sometimes deadlines are important, but if the work needs to be done, and it needs to be done.

Time blocks add stress. Why not just not time block?

Other people say so too:

Time blocks limit creativity

For knowledge work, it limits our creativity. Not every piece of work needs a genius idea. A lot of work is a matter of just getting things done. But sometimes, you need that spark of imagination.

For example, maybe you've heard this quote:

It's not that I'm so smart, it's just that I stay with problems longer. - Albert Einstein

Time-blocking a task to get something done that you may not be able to achieve in that time, will limit creativity. Instead of trying to imagine a solution, I'll be scouring the internet for someone who has already solved my problem.

Unless the time-block is to come up with ideas, and not complete the work, time-blocks do not help me here.

Also, when my mind is fixated on a problem, I can't time-block my imagination. My brain just keeps going back to it. I've learned not to time-block that either.

I'm a software engineer, and estimating is really hard.

Take my example of having task A between 10:00-12:00 and task B between 12:00-14:00.

What happens if I finish task A by 10:05?

Do I do nothing for almost 2 hours? Do I start on task B instead? Then do I finish the day early? Maybe.

Again, time blocks require great estimates. Rarely is it easy to estimate so accurately about getting things done. Hopefully, our estimates aren't completely off but rarely are they accurate.

I need a break from working on something big

And that big thing, even when broken down into smaller things can take days or longer. I can't just throw myself at it for days at a time. I want little breaks in between, and so smaller todos make for little productive breaks rather than stopping work completely.

For example, something simple, like when writing this article, I reached a point where I wanted a break. So I had a small to-do, which was to cross-post another article elsewhere. I also needed to go through my todos as a whole and reprioritise them. 20 minutes later, I was ready to start back on this again.

Slowing down for more meaningful work

Ecclesiastes 3:1-11 NIV

There is a time for everything, and a season for every activity under the heavens...He has made everything beautiful in its time.

Time blocks don't care about the natural speed at which we do things.

But going slower, not so slow as to procrastinate and waste time, but slower to do something properly, not rushing, means having more pride in my work. If I rush, I do a bad job. We all do. Now there's a difference between been naturally fast, and rushing. Time blocks do not allow for that extra time to polish unless you time-block it in. I'd rather just do the thing, rather than having to plan to do the thing.

So what's my alternative?

A. I need to know how much available time I have for todos.

Here's an example of my Tuesday (which is my day off paid work where I volunteer instead). Anything that's not dead time (0.75 hours for transitioning between tasks) or isn't a non-negotiable activity eg. sleep, grooming or eating, gives me time for my Todos.

And that's it, at best, I'll have 2.5 hours for todos unless I'm happy to skimp on time spent elsewhere. If I plan a social, that will eat into my todo-time. If I decide to relax and just watch YouTube, that'll also eat into my todo-time.

But knowing I have 2.5 hours maximum of todo-time gives me a realistic view of what I can achieve and get done. Over a period of weeks and months, 2.5 hours every Tuesday will add up. But on its own, I'll only achieve so much.

B. Having a list of todos

Just having a list of todos, ideally categorised in some cognitively contextual way is more than all I need to be productive. For example, here are the highest priority todos on my list for today:

If they're not important to me, they shouldn't be on my todo-list or at least, they shouldn't be high priority on my todo-list either.

C. Prioritise my todo-list based on my goals

My goals for the next year or so will determine what I spend my time on. It helps prioritise my todos so that I'm not just doing things for the sake of doing them, but I'm doing things that get me ever closer to my goal.

Sometimes, I'm working on just one todo that spans multiple days. If the lack of progress seems to bother me, I'll break that todo down into smaller tasks. Some days, I'll whizz through a dozen little todos.

And that's it.

I don't know if that's simpler or not, but I feel less stressed working this way and more productive.

Then there was Jasmine. Or perhaps it was the other way around. But Mocha was first for me.

There was Jest.

Now there's Vitest.

If you want to write tests for your Nodejs app, you're spoilt for choice. You just run npm install for the pleasure.

Some worked well with Grunt and Gulp.

Others worked great with Webpack.

Now there are those that work perfectly with Vite.

But unless we're doing some compilation for the front end or we're working with Typescript, why aren't we using NodeJS's native test runner? It's been around since version 18! That's April 2022. Almost 2 years in!

So what does it look like? 💯 zero node modules required:

And straight from the docs.

// > node test.mjs
import test from 'node:test';
import assert from 'node:assert';

test('synchronous passing test', (t) => {
    assert.strictEqual(1, 1);
});

That's it!

Assertions? They're native.

Test runners? Also native.

The output? Delightful. 🤩

✔ synchronous passing test (0.618667ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 4.6255

We can even run tests spanning multiple files. As long as the file ends with an acceptable glob pattern like *.test.mjs , you can run the following:

node --test

Default globs are:

• **/*.test.?(c|m)js

• **/*-test.?(c|m)js

• **/*_test.?(c|m)js

• **/test-*.?(c|m)js

• **/test.?(c|m)js

• **/test/**/*.?(c|m)js

We even have watch mode for goodness' sake. How did I not know about this?

node --test --watch

Code coverage? Yes. 😯

# node --test --experimental-test-coverage

ℹ tests 6
ℹ suites 2
ℹ pass 6
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 54.586833
ℹ start of coverage report
ℹ --------------------------------------------------------------
ℹ file          | line % | branch % | funcs % | uncovered lines
ℹ --------------------------------------------------------------
ℹ test.mjs      | 100.00 |   100.00 |  100.00 | 
ℹ test.test.mjs | 100.00 |   100.00 |  100.00 | 
ℹ --------------------------------------------------------------
ℹ all files     | 100.00 |   100.00 |  100.00 |
ℹ --------------------------------------------------------------
ℹ end of coverage report

describe/it blocks? Yes, too! 🤯

describe('A thing', () => {
    it('should work', () => {
        assert.strictEqual(1, 1);
    });

    it('should be ok', () => {
        assert.strictEqual(2, 2);
    });

    describe('a nested thing', () => {
        it('should work', () => {
            assert.strictEqual(3, 3);
        });
    });
});

And it looks beautiful may I add.

▶ A thing
  ✔ should work (0.082667ms)
  ✔ should be ok (0.051292ms)
  ▶ a nested thing
    ✔ should work (0.05575ms)
  ▶ a nested thing (0.126541ms)

▶ A thing (0.512584ms)

So why aren't we using more of Nodejs natively?

One reason is that the test runner didn't exist until April 2022, probably when I was already super excited about Vite. Another reason is we rarely just work with pure Nodejs. Normally we work with a frontend framework, be it React or Vue, Typescript, Purescript, or a fully-fledged framework like AdonisJS to make our life easier.

Reducing our dependencies and relying on native

Having recently fallen foul of dependency hell with an old project, I came to the realisation, that reducing dependencies makes it easier to update a project. Try and fix a Node version 8 script with a dozen node modules that are no longer maintained, and you realise it's probably easier to just write the whole thing again.

The downside is native Nodejs will be more verbose to write, it will lack the Typescript types that I love so much, and it will be missing the prettier documentation that comes with beautiful frameworks like Nuxt.

However, if I want my projects to last a long time with a minimum of fuss and maintenance, it might be worth forgoing these shinier tools that abstract away NodeJS's NodeJSness, and just use it with its fullest purity.

It's not just a NodeJS problem. Every language has this problem. We've developed tools to make our lives easier because changing and improving the underlying language isn't always possible.

Yet, Ricardo Lopes proved you can have a sane zero-dependency mini web application without even a package.json written in NodeJS:

"There's a place for complex frameworks and architectures, sure. But for many projects, they may be an overkill."

So if we're developing a project that's 1) small, 2) not really going to change much, and 3) only needs NodeJS, why not avoid using any dependencies at all, and go native with node --test?

I'll sign off with a Flavio Tweet in favour of reducing dependencies.

https://twitter.com/flaviocopes/status/1754784031307084015

Despite that, for debugging, I'll be using console.dir() more in future. Here's why?

Let's take a deeply nested object.

let hello = {
    world: true,
    nested: {
        obj: {
            with: "some text",
            and: {
                numbers: {
                    count: [1,2,3,4,5,6,7,8,9,10],
                    symbol: Symbol("1"),
                }
            }
        }
    }
}

console.log() to debug

Now, if you do a pure console.log:

1. You'll be missing some of the properties that are past a certain depth.

2. The output is on a single line and not structured in an easily readable way.

# console.log(hello)
{ world: true, nested: { obj: { with: 'some text', and: [Object] } } }

console.log() to get the whole object

How I'd normally get around expanding that [Object] is using trusty old JSON.stringify. And voila 🎉! We have the entire object displayed.

# console.log(JSON.stringify(hello))
{"world":true,"nested":{"obj":{"with":"some text","and":{"numbers":{"count":[1,2,3,4,5,6,7,8,9,10]}}}}}

console.log() but pretty

While we could see the entire object before, it's not easy to read. We can fix that by using the JSON.stringify formatting argument:

# console.log(JSON.stringify(hello, null, 2))
{
  "world": true,
  "nested": {
    "obj": {
      "with": "some text",
      "and": {
        "numbers": {
          "count": [
            1,
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10
          ]
        }
      }
    }
  }
}

And this is much better, but it still isn't great. Look at that array! That's equally impractical.

console.dir() for better readability

Now, compare all that hard work with console.dir. You have to add the depth you want to be printed to get the entire object out.

console.dir(hello, { depth: 99 })

And it looks like this:

{
  world: true,
  nested: {
    obj: {
      with: 'some text',
      and: {
        numbers: {
          count: [
            1, 2, 3, 4,  5,
            6, 7, 8, 9, 10
          ],
          symbol: Symbol(1)
        }
      }
    }
  }
}

It has nice terminal colours because it's not JSON. It's Javascript! It benefits from the conciseness of javascript and is more compact and yet more readable.

Getting a little more out console.dir()

There are three options you can pass into the formatting argument with console.dir(). We've already covered depth so we'll skip that.

console.dir(hello, {
    showHidden: true,
    depth: 99,
    colors: true
})

• showHidden allows you to see Javascript annotations such as the length of an array.

{
  ...
        count: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, [length]: 10 ],
  ...
}

• colors by default is true (for me) in the terminal, but you can turn it off if you want. I find the colouring helpful though:

There's nothing more to say really. As I'm reading the Nodejs and finding little gems to make life a little easier. Maybe it's helpful to you.

PARTITION BY has been around since version MySQL 5.7! That's yonks ago!

I'll be the first to admit; that while MySQL's documentation is really helpful at times, most of the time, it's like an ocean, full of information, and I just need a drop of it to solve my problem. So I find myself reaching for blog posts, forums and videos to solve my issue.

And that's been a problem for me; instead of becoming comfortable with the docs, I reach for Google.

So why did I get RT*Med at work? Partly, because my colleagues are kind enough not to curse. But mainly, we were trying to find a use case, hidden in our data, not knowing if we had it or not for QA purposes. The issue was that I had no idea how to even write the query.

What use case was I looking for?

Company IP aside, imagine we had a table with data that could be grouped by a non-unique key, external_id. I wanted to find a grouping where the last row grouped by date had a nullable field but the row dated before did not. eg. I want it to pick out grouping by x and not grouping by y.

Now of course, GROUP BY with HAVING came to mind immediately because that's what I know. But that would merely find rows with a random mixture of nullable and non-nullable columns.

My naive approach at best was:

select
    *,
    sum(isnull(field_a)) as has_null,
    sum(! isnull(field_a)) as has_not_null
from
    my_table
group by
    external_id
having
    sum(isnull(field_a)) = 1
    and sum(! isnull(field_a)) > 0;

This would have led to a useless answer:

A decade-plus of MySQL, shamefully, I didn't think this problem could be solved with MySQL alone. And yet, our database master (a.k.a, our CTO) solved the conundrum using it and his quick quip was (to paraphrase), "well, that's what happens when you RTM".

I managed to get this far as a developer while being squeamish about reading through documentation. It was his gain, and my loss in the end. I missed out on a fundamental piece of knowledge that would have helped create a simple MySQL query; MySQL Window Functions.

Discovering Window Function in MySQL

Now looking at MySQL's docs rather than Googling around, I found a surprisingly helpful example to copy. As I tend to learn by example, here's what MySQL shared:

       SELECT
         year, country, product, profit,
         SUM(profit) OVER() AS total_profit,
         SUM(profit) OVER(PARTITION BY country) AS country_profit
       FROM sales
       ORDER BY country, year, product, profit;

Now, I still don't fully understand how OVER works and it's a little hazy, but having a little think about things, and how to use PARTITION BY, I came up with this:

select
    *
from
    (
        select
            *,
            -- Get total in group without using GROUP BY
            count(external_id) over (
                partition by external_id
            ) as total_in_group,
            -- Ordered by row number
            row_number() over(
                partition by external_id
                order by
                    external_id,
                    date
            ) as rownum
        from
            my_table
        order by
            external_id,
            rownum
    ) t1
where total_in_group = rownum
and field_a is null
and total_in_group > 1;

Okay, so it looks a bit much, and there's probably a simpler way, but for a first pass, I'm pretty happy. And it was put together with some trial and error. So let's break it down a little.

For repetition, we have this data:

field_a is sometimes null, which is important to us.

Counting groups of data without GROUP BY

Now, what I want is to find the group of data, where the last dated field_a is null but the other fields are not null. So that means, I want to return group x.

To do this, I start by knowing the total number of items within a group but without using GROUP BY.

count(external_id) over (
    partition by external_id
) as total_in_group,

Ordering data within groups without GROUP BY

Next, I want to order the items in the group by date using the row_number() function.

row_number() over(
    partition by external_id
    order by
        external_id,
        date
) as rownum

And now, it's like everything has fallen into place. All I need to do to find ID 3, is find where total_in_group = rownum and field_a IS NULL. That's it! 🎉. Throw in a subquery. Life is good.

Well, life may not be performant. I did do this on just 6 rows of data. At work, we were dealing with millions of rows, and Postgres, where isnull does not exist 😒. Anywho. Can't complain.

Although, it's still not quite correct. It would still work if the row before it contained a null field. But I was satisfied I was on the right track to solve the problem with a little more digging through the manual.

Knowing where to find solutions

I learn best by doing. I can't just read snippets and start writing code.

I need to understand what I'm reading in the context of building something tangible.

However, my faux pas is once I've grappled with the basics, I never look at the docs again. Where I've gone wrong, is I've read docs to solve a specific problem I had at the time.

Instead, Aaron's perspective is to read the docs to build a mental map of where to find solutions to problems we have in the future, not to understand everything in the docs today.

One example recently was at some point last year (2023) I came across an AWS article about Point in Time Recovery. I didn't even read it all.

While having a chat with our engineer, I brought up this article. As an alternative to another solution that worked in the past, it seemed both faster and simpler to implement! Now I couldn't tell him how it worked under the hood or how to implement it in Terraform or even via the GUI, purely knowing it was possible and where to find it was enough to make the suggestion!

Knowing where to find solutions, or that a solution exists, is a good reason to read the docs.

Learning to read the docs for leisure

Now, I just can't see myself, doing some pre-bed reading of Nodejs, albeit, for this exercise I did.

The recommendation of printing documentation and reading it through? I'm not sure about that. I love the visceral nature of physical books and writing by pen, but I also hate having unbounded paper everywhere.

Yet, searching for a solution under time pressure limits our creativity (as deeply explained in Hyperfocus by Chris Bailey). With a deadline, we narrow our focus to get the task done in the quickest time possible. So with a deadline in place...

Yes. I get things done.

No. It didn't look pretty.

Yes. I've produced many-a-bad solution while under a deadline, even if the thing works on the surface.

In comparison, sitting, leisurely reading, I'm just learning for fun.

Reading the docs for leisure is a very different way of absorbing information. It makes me think, the reason why many, including myself, shy away from reading the docs is because we read it when we need it, and so we don't have time to wade our way through the superfluous abundance of information. We need a solution, now!

Putting that to the test, I went back to read the docs just a little more, and I discovered that my query could be corrected.

Remember, I want to select a group, where the last row in the group by date, has a nullable field, but the row before is not null.

Here's a more complete set, where we don't want group x or y, we want z.

And voila 🥳.

select
    *
from
    (
        select
            *,

            count(external_id) over (
                partition by external_id
            ) as total_in_group,

            row_number() over(
                partition by external_id
                order by
                    external_id,
                    date
            ) as rownum,

            lead(isnull(field_a)) over(
                partition by external_id
                order by
                    external_id,
                    date
            ) as nullable
        from
            test.my_table
        order by
            external_id,
            rownum
    ) t1
where
    (total_in_group - 1) = rownum
    and nullable = 1
    and field_a is not null;

The addition was to use the LEAD function which gets the next record along. So I could then check if the next record after is nullable, and thus select ID 2. I could also find the second to last field using (total_in_group - 1) = rownum.

Yes, it's a little more complex, but it's great there are a few ways to solve this problem. And I've banked yet another MySQL function.

When reading the docs is bad!

So, reading the docs is great! Or is it?

Time is precious. And so we thank the developer who took the time to distil information in a format that could be read, over and over again to make his or her time scale rather than having to explain things constantly.

However, too much documentation might be a problem. In fact, in Fernando Villalba's article, I still can't believe the video he shared is genuine, but here's an example of verbose documentation gone wrong.

Fernando pointed out:

"If reading the manual is the only way for you to understand every functionality of your application, you have failed at design."

Too much documentation may be hiding bad design. Of course, with something like learning a programming language to solve a specific problem, it's rare for the language to be intuitive to just write it.

Intellisense helps.

Perhaps Copilot does too.

But outside of this, you have to read the manual. However, the tools we use, should be as intuitive as possible, so their function doesn't require having to read the manual to get started.

If they're not, we have an opportunity to help the maintainers:

1. improve their tools so they are intuitive to use. Too much documentation might be hiding bad design.

2. improve their docs by documenting contextually vs exhaustively, where we cannot make the tool more intuitive.

3. write external tutorials where we cannot improve the docs for whatever reason.

And what do we get out of it?

• More skill[man/woman]ship.

• More proficiency.

• Known for mastery of a tool or technology which is great for our careers.

• Better productivity.

• Better design.

• Better use of creativity.

And we can't read everything

Or can we?

Maybe if we just use a few tools instead of many, we can read all the docs, no problem.

But where I work, with so many tools and features being developed, I could spend all my time reading the docs, PRs, release notes and RFCs. I'd worry I'd get little to no work done.

So I need to be selective about what docs I read through.

What docs am I going to start reading?

I didn't want to do a MySQL tutorial, but quite the opposite. I was inspired by "Read the Docs Like a Book" article (another Aaron Francis special) which recommends becoming well acquainted with a set of documentation. The idea is that as you become more acquainted with your tools through the docs, you become a more skilled developer.

So, I decided I would "Read the Docs", and share notes on anything interesting. By interesting, it's mostly going to be things that are "new to me" or quirky. My current list to read contains:

• Nodejs because I still don't understand Buffers and I'm surprised there's a native way to run tests. I also think I should try and master it in at least one language, seeing as I love it so much.

• Nuxt because I love how it makes working with Vue easy but I couldn't tell you what a Composable is.

• JSON Schema because I'm building Tin and it'll help with standardising data formats.

• Rust because I have a goal to learn it this year.

• C++ because I've been a developer long enough, I should eventually learn it, and it's a stretch goal.

• K8s because I use it almost daily and still don't know how to use labels and metadata properly.

• JQ because it's powerful and I love working on the CLI when testing APIs with HTTPie. HTTPie all day over Postman!

• AWS because it is the Cloud!

Pre-requisites

• You will need a Nuxt application hosted on a live domain name. It won't work locally.

Step 1. Getting set up with Pirsch

Sign up with Pirsch through https://pirsch.io/ and verify your email.

They'll take you through an onboarding process where you can add your domain name. Make sure to include www if that's where your naked domain redirects to.

They'll show you a Script tag that you can copy that looks like this:

<script defer type="text/javascript" src="https://api.pirsch.io/pirsch.js"
    id="pirschjs"
    data-code="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"></script>

Step 2. Loading their script in Nuxt

With Nuxt, they use Vue Meta, the HTML Metadata manager for Vue.js. This allows you to inject scripts or stylesheets amongst other things into the header of your SPA, universal or static app.

Add the following script block through the nuxt.config.js .

export default {
  ...
  head: {
    script: [{
      src: 'https://api.pirsch.io/pirsch.js',
      'data-code': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
      type: 'text/javascript',
      id: 'pirschjs',
      defer: true
    }],
   ...
}

Change the data-code to match the code from Pirsch.

Step 3. Deploy and wait

Within minutes of deploying, we had cookieless, privacy-first analytics. It took longer to write this article than it took to make the change and get set up with Pirsch 😂.

Disclaimer: Why not Google Analytics?

Firstly, I just have to make sure I'm not mistaken; I have no hatred for Google or for Google Analytics. Yes, the mass data collection is a little creepy, but more importantly for me, I can't stomach the cookie (🍪) bombardment (💣) after GDPR came in. I don't want that experience for users at embodimentshop.com either.

Webpack helps make frontend development more streamlined and recently, I wanted to transform YAML into CSS as part of the build process using the Preons library.

By the end of this article:

• you will create a Webpack loader that converts a YAML configuration file into CSS
• you will manually test the loader against a basic HTML page
• you will know how to create your own Webpack loader in future

Background

Preons (and disclaimer, a project of mine) is a functional CSS utility library. You configure the utility classes in YAML.

Webpack is a popular module bundler, and one helpful thing it does is transform Sass into CSS with a few useful community loaders.

Pre-requisites

To follow this tutorial, you need:

• Node 12+. Install Node's command-line tool.
• Yarn. Install Yarn's a command-line tool.
• Have a basic knowledge of javascript

Step 1: Creating a new project

You will create a basic empty project and initialise npm to begin installing local node packages.

Execute the following command to create a new project folder:

mkdir custom-webpack-loader && cd custom-webpack-loader

Initialise Node's package manager:

yarn init

You will be asked a series of questions:

yarn init v1.22.4
question name (custom-webpack-loader): 
question version (1.0.0): 
question description: 
question entry point (index.js): 
question repository url: 
question author: 
question license (MIT): 
question private: 
success Saved package.json

By the end, you will have a basic package.json without any dependencies. But you'll add them in a few step.

Step 2 - Configuring Webpack

In this step, you will create a webpack configuration file and tweak it to load a YAML file. It will allow you to test the loader as if you were using it in a project.

Install Webpack and Webpack's CLI to give you access to the Webpack builder:

yarn add webpack webpack-cli

Next, you'll create the webpack.config.js:

touch webpack.config.js

In webpack.config.js, add:

const path = require("path");

module.exports = {
  //...
  mode: "development",
  entry: path.resolve(__dirname, "fixtures") + "/preons.js",
  module: {
    rules: [
      {
        test: /\.ya?ml$/,
        use: [
          "style-loader",
          "css-loader",
          {
            loader: path.resolve(__dirname, "lib") + "/loader.js"
          },
          "yaml-loader",
        ],
      },
    ],
  },
};

• The mode property tells Webpack to make optimisations based on environments. In production, however, you would change the mode to production. However, for this use case, the default will be development.
• Notice how the .module.rules[0].test catches .yaml and .yml files based on regex rules.
• The entry file points to a fixtures directory as that directory has files for you to test.
• There are four loaders, 3 of which are existing npm packages. Webpack will apply rules from bottom to top, so first, the yaml-loader will take YAML and convert it into JSON. Your loader will transform Javascript into CSS; afterwards, the css-loader will apply its transformation and finally, the style-loader takes css and puts it into style blocks that are inserted into your page through Webpack's script.

Install other dependency loaders using yarn:

yarn add style-loader css-loader yaml-loader

Now you have a basic Webpack config for testing the loader you're going to build.

Step 3 - Importing YAML from our input js file

When initially learning about Webpack, one of the strange conventions is not to deal with non-javascript files through the configuration. Instead, you import them as if they are javascript modules themselves. You'll do the same here.

First. Create the fixtures directory:

mkdir fixtures

Next, you need a Preons config file. To save you from learning the syntax and configuring one yourself, install the Preons utility globally.

npm install preons -g

Generate the Preons YAML config:

preons config > fixtures/preons.yaml

You will see a YAML file containing CSS class rules such as theme-colors, scales and percentages. The file contents should start like this:

preons:
  baseline: 1.6rem
  rules:
    theme-colors:
      black: "#000000"
      white: "#ffffff"
      transparent: transparent
      hotpinkxl: "#f188bc"

Create fixtures/preons.js.

touch fixtures/preons.js

Add a require expression to it, so you import the YAML through it.

require("./preons.yaml");

Now you have configured the webpack loader to take the fixtures/preons.js file and run any YAML imports through the lib/loader.js, and it will spit that back out inside a dist folder. But before that, you'll need the loader.

Step 4 - Creating the loader

Now you can build the loader.js. The minimum is one exported default function that returns a module.exports = <> string.

First, create the loader.js file:

mkdir -p lib && touch lib/loader.js

Add the basic method to test the loader is working:

module.exports = function (source) {
  console.log(source)
  return `module.exports = {}`;
};

At this point, you can run webpack, and you will see the YAML source load in:

npx webpack

The output will be a lot of JSON, because of the console.log(source). You'll also find a dist folder in your project with a main.js file.

With the source containing JSON, transforming using the Preons library is only a few more steps. Install Preons locally into the project to access its lib functions to do the transformation of the preons.yaml into CSS:

yarn add preons

Change the lib/loader.js to parse the YAML JSON into CSS.

const preons = require("preons/src/lib/stylesheet");

module.exports = function (source) {
  let set = JSON.parse(source);
  let css = preons({ set });
  return `module.exports = ${css}`;
};

• let set = JSON.parse(source) converts JSON into a javascript object required by the Preons stylesheet function.
• let css = preons({ set }); is a Preons function that transforms the YAML into CSS.
• returnmodule.exports = ${css}; returns the css as a module to be used by the rest of the Webpack pipeline ruleset.

Now test your webpack pipeline by running:

npx webpack

If you look inside dist/main.js, you will find the entirety of Preons CSS generated. To check the generation works, do a grep for the css Property bg-black:

cat dist/main.js | grep "bg-black"

Step 5 - Testing the webpack loader with HTML

Now you are generating CSS from the Preons YAML configuration file. You will want to test you can develop with it.

Create a raw HTML index.html file in the root of the project:

touch index.html

Then add a simple starter HTML5 file:

<!DOCTYPE html>
<html class="bg-black">
    <head>
        <link href="https://unpkg.com/browse/preons@0.4.5/dist/reset.css" rel="stylesheet" />
        <script src="dist/main.js" ></script>
    </head>
    <body class="bg-red">
        <h1>Hi</h1>
    </body>
</html>

• The index.html references Preons reset package as well as your dist/main.js file.
• The markup has Preons utility classes such as bg-black and bg-red. Open the index.html, and it will look similar to this:

Finally, you want to test that we can make changes to our YAML and it produces CSS instantly.

Start the Webpack dev server to watch for changes using:

npx webpack --watch

Go into the preons.yaml and look for the theme-color -> black. Change it from black: "#000000" to black: "yellow". The file should have this:

#....
  rules:
    theme-colors:
      black: "yellow"

Refresh the page, and the bg-black rule is now yellow.

Ok. You will want to keep bg-black as it was, but it's just to prove that the generation runs through Webpack. You can now add more colours, change colours or remove them. Of course, you can add any utilities you want using the configuration file. The documentation on how to use the configuration file is still under development, but I'll remember to add it here once it's done.

Conclusion

You built a Webpack loader to ingest YAML and create CSS to be used as part of a development and production build. You used pre-existing modules (e.g. yaml-loader) to keep your module loader concise.

If you want to continue using Preons, look out for the Preons webpack loader package, so you won't have to maintain it yourself.

Do you have any ideas for your own for a webpack loader?

Also, I hope you enjoyed this article. I'd love feedback on how to improve as a writer and as a developer, so feel free to leave a comment or DM me on Twitter.