Tim Disney

Sweetening syntactic abstractions

Version 3.0 of Sweet.js has just been released! 🎉

To set expectations, keep in mind that I consider Sweet to still be an experiment and under heavy work. The version number is a reflection of semver (we've made breaking changes) not project maturity. Since the big redesign last year tons of progress has been made, but I expect at least one or two more breaking changes (major version bumps) before things start to get baked and really ready to use. That said, if you are excited about the idea of true syntactic abstractions in JavaScript please dive in!

So what's new?

Custom Operators #

Custom operators are back! In the old pre-1.0 days we had the ability to define new operators with custom precedence and associativity but it was dropped in the redesign.

Operators are defined with the new operator keyword:

operator >>= left 1 = (left, right) => {
  return #`${left}.then(${right})`;
}

fetch('/foo.json') >>= resp => { return resp.json() }
                   >>= json => { return processJson(json) }

That expands to:

fetch("/foo.json").then(resp => {
  return resp.json();
}).then(json => {
  return processJson(json);
});

The implementation of custom operators is pretty experimental at the moment but give it a whirl and let us know if you run into any problems. More details are in the tutorial.

Note: the technical underpinnings for custom operators comes out of Jon Rafkind's dissertation on the Honu language.

Modules #

We've been steadily adding ES module support over the past few releases. The interaction between macros and modules is fairly complex so this is an ongoing process.

We currently have the ability to import macros from another module:

// foo.js
'lang sweet.js';
export syntax m = // ...

// main.js
'lang sweet.js';
import { m } from './foo';

m // ...

Note the use of the 'lang sweet.js' directives. These directives are currently required in any module that uses macros. It allows the Sweet compiler to avoid needlessly expanding modules that don't contain any macros. At present the directive is "just" an optimization but soon we'll be using it for some pretty cool stuff.

You can now also import modules into compiletime code (macro definitions) by using the for syntax form of import:

// log.js
'lang sweet.js';

export function log(msg) {
  console.log(msg);
}

// main.js
import { log } from './log.js' for syntax;

syntax m = ctx => {
  log('doing some Sweet things');
  // ...
}

We're taking the Racket approach of dividing everything up into phases. Runtime syntax is in phase 0 and compiletime macro definitions are in phase 1. Importing for syntax allows you to phase shift your code around. Phases greater than phase 1 happen when you import for syntax a macro that uses another macro that was imported for syntax. This gives rise to an infinite "tower of phases" which sounds complicated but turns out to be pretty straightforward in practice.

Still to come are better support for implicit runtime imports, finer grain support for phases that let you import for a specific phase, and an equivalent to Racket's begin-for-syntax.

Note: the technical underpinnings of modules and macros comes out of the Racket approach set forward by Matthew Flatt in his "You want it when?" paper.

Readtables #

While macros allow you to extend how syntax is parsed, sometimes you also need to extend how source text is lexed. The lexing extension approach we are taking is called readtables and @gabejohnson has been doing some amazing design and implementation work. Sweet now uses readtables internally and will soon be exposing them to users.

Internals and helpers #

During expansion Sweet constructs several intermediate representations of syntax that can be manipulated and eventually turned into a Shift AST. The exact representation we want to use is under flux but unfortunately it is exposed to macro authors inside macro definitions. Exposing what should be internal details is bad so to move away from that Sweet now provides a helper library for macro authors:

import * as H from 'sweet.js/helpers' for syntax;

syntax m = ctx => {
  let v = ctx.next().value;
  if (H.isIdentifier(v, 'foo') {
    return H.fromString(v, 'bar');
  }
  return H.fromString(v, 'baz');
}
m foo; // expands to 'bar'

Macro authors should only use the helper library to inspect and manipulate syntax objects rather than rely on the current representation of syntax. Eventually we will document and freeze a intermediate syntax representation but until then just use the helpers.

What's next? #

The current plan is to get Sweet to a solid and stable place where we can start building declarative conveniences on top its foundation. In particular, the current macro definition syntax is intentionally low-level and not convenient to work in. We've got some ideas about what this might look like but first we're going to make sure the base is solid.

If any of this excites you, please jump in! We'd love to have you!