Accessing skipped whitespace

[pdubroy]

In earlier versions of Ohm there was a way to access the implicitly-skipped whitespace. This got removed at some point and wasn't added back.

We should add this back, but it's not (yet) obvious what the right way to do it is.

[pdubroy]

Randomly ran across this on the Roslyn wiki:

How are comments stored in the syntax tree (and how to use the Syntax Visualizer)?

Roslyn trees store comments as trivia on tokens. The heuristic used is to attach trivia to the token following it (that is, as
leading trivia on the node containing the token). However, if the trivia is after the last token on a line, it attaches to the preceding token (that is, as trailing trivia).

In the following example, the first comment attaches as LeadingTrivia to the ‘int’ token in the declaration following it. You can also see the leading trivia on the LocalDeclarationStatementSyntax node containing the ‘int’ token. The first comment trivia attaches to the first token following it even though ‘int’ is on a different line because there are no tokens on the line with the comment.

 int method1 () {
  int a = 1;
  // comment1
  int b = 1; // comment 2
}

The second comment attaches as TrailingTrivia to the semi-colon, and you can also see it on the LocalDeclarationStatementSyntax node.

[pdubroy]

Current plan that @alexwarth and I discussed is to making leading whitespace available via a property/method on the node wrappers. Something a little like this:

semantics.addOperation('interpret', {
  // …
  AddExp_plus(x, _, y) {
    if (x.leadingSpaces.sourceString.length > 0) { // 👈
      console.log('x has leading spaces!');
    }
    return x.interpret() + y.interpret();
  },
});

One tricky thing is that when you call match('MyRule') in Ohm, it implicitly means "match MyRule followed by end", and if MyRule is syntactic, we skip spaces before the end. But there's currently no way to write a semantic action for that implicit top-level expression, and no way to get access to the node corresponding to end.

A few options:

1. We discussed adding a special _root action, that would allow you to write a semantic action for the top-level expression:

semantics.addOperation('interpret', {
  _root(myRule, end) {
    if (end.leadingSpaces.sourceString.length > 0) { // 👈
      console.log('founding trailing spaces!');
    }
  },
  // …
});

• This keep uniformity; in this view, all skipped spaces are leading spaces, and are accessed the same way.
• We have to introduce a new, specially-named _root action, and understanding it requires understand the implicit top-level expression. Currently, our documentation says "If the start rule is a syntactic rule, both leading and trailing spaces are skipped around the top-level application", which is (imo) a simpler way to describe it.
• I'm having trouble coming up with another thing you'd use the _root action for…which makes me think there may be a simpler solution.

2. We could also have a way of getting trailing spaces from the MatchResult:

const matchResult = g.match('MyRule');
if (matchResult.trailingSpaces.sourceString.length > 0) {
  console.log('founding trailing spaces!');
}

• We lose uniformity — everything else is leading space, this is trailing space.
• It seems to preclude using any kind of operation on the node, because we're not in the context of a semantics. It would work if we just want to expose spaces as raw characters though.

3. We could expose a trailingSpaces property on all nodes, and it would only be non-null on the root node. So you could write an action like this:

  semantics.addOperation('interpret', {
    MyRule(a, b) {
      if (this.trailingSpaces.sourceString.length > 0) { // 👈
        console.log('founding trailing spaces!');
      }
    },
    // …
  });

[pdubroy]

In the v18 beta I've implemented the leadingSpaces property on CST nodes. However, there is a minor corner case that I've discovered, that I'm not sure how to deal with.

When we use the lexification operator (#), it doesn't affect the structure of the CST. That is, there's nothing in the CST that says that we didn't try to skip spaces at that position.

The current implementation of the leadingSpaces property simply checks the memo table for spaces the given position. But, in certain scenarios, we might have skipped spaces at that position, but it's incorrect to attribute those spaces to a node inside a lex. For example:

G {
  Start = ">" "a" digit
        | ">" #(" a" letter)
}

…with the input > ab.

Here's what happens:

• In attempting to match the first alternative, we'll skip spaces at position 1, but then backtrack.
• Then, the 2nd alternative will succeed.
• When walking the CST, if we ask for leadingSpaces on the terminal node associated with the expression " a", it should be NULL (but isn't currently, as shown by the failing test added in 1ef6131).

I've experimented with few possible ways of dealing with this:

• Introduce a new Lex match record — i.e., record it as a kind of pseudo-CST node in Wasm linear memory, that is never actually exposed as a standalone node during the walk phase. This would add complexity to all of the walking code (in every language) — mostly because we'd then have a difference between the raw childCount and the visible childCount.
• Introduce a proper CST node for lex. Then the walking code is simple, but it's another node type that needs to be handled by user code.
• Record "edge metadata" — a single leadingSpacesAllowed bit on every child pointer that would allow us to mark lexical nodes (see wasm: implement "edge metadata" to mark where leadingSpaces can appear #607). It's doable but it's a lot of state that needs to be threaded through everywhere, and it complicates both the parsing process, and walking (because we have to mask all pointer accesses). We could avoid complicating the walk process by storing the bit elsewhere, but it's tricky, because we need to keep track of it when we push the binding.