Meta Programming Feature Requests

Topics: C# Language Design, VB Language Design
May 15, 2014 at 6:26 PM
Edited May 15, 2014 at 6:30 PM

Meta Programming Feature Request

I would like to request the addition of several features to C#/VB that I believe will enable a wide variety of meta-programming capabilities without necessarily affecting backwards compatibility or compromising the design of the languages.

Features
  1. Allow the compiler pipeline to be extensible at build time.
  2. Add Quasi-Quotes to the syntax of both languages.
  3. Add support for syntactic macros.
  4. Enable grammar extensions.
Extensible Compiler Pipelines
The simplest way to enable this would be to support a <PipelineType> parameter to be set into the csc build task. This would allow a developer to redirect the csc build task to use a pipeline of their own instead of the default csc compiler pipeline. You would then have an api where a developer could add, remove and re-order steps in the compiler pipeline.

This would enable a developer to apply similar transformations to their code during a build as they might in an IDE extension, except the original source remains unchanged. This is preferable over chaining multiple build tasks together as it is far simpler, faster and is superior at retaining proper tracing information back to the original source.

Specifically it enables AOP but can also enable a host of other automated code inspection or consistent transformation techniques.

Quasi Quotes
Quasi quotes are necessary to make the lives of any developer working on syntax tree transformations easier. Rather than directly using the static factory members on SyntaxFactory one could use a more friendly syntax to their source language.

For example, instead of writing:
var statement = SyntaxFactory.ParseStatement(string.Format("var {0} = 0;", name));
One would be able to write:
var statement = [| var $name = 0; |]
Quasi-Quotes in this fashion would allow you to have syntax highlighting and other IDE features in your quasi-statement while supporting meaningful binding to values and other SynaxNodes in scope without needing to resort to string generation.

This becomes increasingly important as you do more syntax transformation in general.

Syntactic Macros
A user should be able to inherit from a class, such as AbstractSyntaxMacro, that when implemented will perform a transformation on the input Syntax and return the result.

Then the default language parsers for C# and VB should support recognizing the placement of macros in certain key spots such as the beginning of a Type declaration, Member declaration, Statement or Expression. A syntactic macro is recognize when what appears to be an unrecognized keyword in one of these key places matches, by convention, the name of a Type in any referenced code inheriting from AbstractSyntaxMacro. When this macro is recognized and matched, the previously unknown syntax should be passed into the macro and be transformed instead.

For example:
// examplemacro.cs
class ExampleMacro : AbstractSyntaxMacro {
  override Transform(SyntaxNode syntax) {
    return [| Console.WriteLine("Hello World!") |];
  }
}

// main.cs
example; // unrecognized symbol

// output:
> Hello World!
The unrecognized symbol "example" does match, by convention, the name of the syntactic macro "ExampleMacro". Thus the compiler passes in the entire unrecognized statement to the macro and replaces it with the output of that transformation. It's necessary to run the macro expansion step again on the output of a macro expansion, detecting infinite cycles.

Grammar Extensions
This last feature is the hardest and should be considered independently from the previous as I don't believe that it is necessary to achieve the others or vice-versa. This feature would likely require a complex redesign of the parser itself so I understand that it's an expensive ask.

But that being said this features would allow you to extend the syntax of the language beyond merely transforming certain keywords. It would allow you to create custom operators, recognition of complex patterns and perhaps just as importantly, it would let you remove unwanted syntax when desired.

This last point is important for creators of DSL's who may want a restricted domain language while still being able to leverage complex patterns in key places, such as Expressions. There are a several examples of how to achieve this technique in .net such as:
  • Boo Ometa
  • Nitra
  • Nemerle
And I believe that they each prove the feasibility of the technique while simultaneously showing the power. This or similar features exist in myriad other platforms of course and I think they are generally regarded as extremely powerful features, especially when coupled with IDE support.

I personally have been working on a project called meta# which is an expression of this final idea. I would be happy to expand on details of how this could work, as well as what kind of usecases it could help solve. Here is teaser of Grammar Workbench in visual studio for example.