Meta-programming with attributes

Topics: C# Language Design
Apr 4, 2014 at 1:43 AM
Edited Apr 4, 2014 at 1:56 AM
Hi C# Team,

First of all, thanks four the great work. Having Roslyn released AND open sourced has put a big smile in my face :)

I'm a developer of a Signum Framework, a framework for building LOB applications with a full LINQ provider wrote for it.

After seeing Anders commenting about meta programming and attributes I've realized that this could potentially solve three places where we have a lot of redundancy on our day to day code:

1. Property getters / setters that implement INotifyPropertyChanged

We require that the properties are set using using a special Set method, that modifies the value but also does other things, like raising a PropertyChanged event, setting the entity as dirty, etc... Because of this we can not use auto-properties.
  string name;
  public string Name
  {
       get { return name; }
       set { Set(ref name, value, () => Name); }
   }
Note: I'm aware that we can use CallerInfoAttributes to avoid the lambda, just waiting to get rid of the Windows XPs out there.

I think this is a common problem, and counting all the projects we have tons of properties like this (500-1000)

I would like to avoid writing the getter and setter, but tell the C# compiler somehow that this property should be expanded in a different way:
  string name;
  [MyPropertyExpander] 
  public string Name { get; set; } = name;
Note: I'm also stretching the new 'read-only auto properties' to 'auto properties with explicit field'.

2. Expression methods that generate expressions lambdas

Often we want to factor-out some reusable method using extension methods:
public static bool AssignedToMe(this TaskDN task)
{
    return task.Worker.RefersTo(EmployeeDN.Current)
}
Because we want to use this method also when querying the database, we had teach our LINQ provider to recognize the following pattern:
static Expression<Func<TaskDN, bool>> AssignedToMeExpression =
     t => t.Worker.RefersTo(EmployeeDN.Current); //for the LINQ provider
public static bool AssignedToMe(this TaskDN task)
{
     return AssignedToMeExpression.Evaluate(task); //for in-memory calls
 }
Then we can write something like this:
Database.Query<TaskDN>().Where(t=>t.AssignedToMe()).First();
And our LINQ provider understands the pattern and expands the expression.

Works OK but there's some also some redundancy involved. Also it gets worst when you want to declare two overloads of the same extension method in the same class, then you need to disambiguate the field using an ExpressionFieldAttribute... looks like a hack.

Using the new 'method expressions' will look better
public static bool AssignedToMe(this TaskDN task) => task.Worker.RefersTo(EmployeeDN.Current)
But unfortunately, I assume that the method expressions won't generate an expression tree.

I will like to write:
[MyExpressionTreeExpander]
public static bool AssignedToMe(this TaskDN task) => task.Worker.RefersTo(EmployeeDN.Current)
And then get it compiled to:
static Expression<Func<TaskDN, bool>> AssignedToMeExpression1 =
     t => t.Worker.RefersTo(EmployeeDN.Current);
[FieldExpression("AssignedToMeExpression1")]
public static bool AssignedToMe(this TaskDN task) => task.Worker.RefersTo(EmployeeDN.Current)
3. Pseudo-Enums

Finally, and this is a little bit harder to explain: we are currently using untyped enums (Enum) to implement what we called MultiEnums.

What we want is an expandable family of enums, so other libraries can add more elements, but you still have the nice properties of enums:
  • Compile time checking (instead of strings)
  • Nice Run-time reflection capabilities (ToString, GetType)
We use this to represent on the database concepts that are hard coded in the application, like entity operations, scheduled task algorithms, process algorithms, etc... but that can be expanded by third-party libraries.
public enum TaskOperation
{
     Create,
     Save,
     Start,
     Stop,
     Cancel,
     Delete,
}

public enum MessageOperation
{
     Create,
     Save,
     MarkAsReaded,
     Delete,
}
The problem with our approach is that we can not make each enum inherit from their corresponding family (i.e: Operation) so many of our API ask for an untyped Enum value.

We have considered creating an abstract Symbol class, from witch Operation will inherit, but that will make declaring the operations too unconvenient and error-prone due to the duplicated name:
public static class TaskOperation
{
     public static Operation Create = new Operation("Create", typeof(TaskOperation));
     public static Operation Save = new Operation("Save", typeof(TaskOperation));
     public static Operation Start = new Operation("Start", typeof(TaskOperation));
     public static Operation Stop = new Operation("Stop", typeof(TaskOperation));
     public static Operation Cancel = new Operation("Cancel", typeof(TaskOperation));
     public static Operation Delete = new Operation("Delete", typeof(TaskOperation));
}
But this will get much better if we could write:
public static class TaskOperation
{
     [ExpandSymbol]public static Operation Create;
     [ExpandSymbol]public static Operation Save;
     [ExpandSymbol]public static Operation Start;
     [ExpandSymbol]public static Operation Stop;
     [ExpandSymbol]public static Operation Cancel;
     [ExpandSymbol]public static Operation Delete;
}
Or even
[ExpandSymbol]
public static class TaskOperation
{
     public static Operation Create;
     public static Operation Save;
     public static Operation Start;
     public static Operation Stop;
     public static Operation Cancel;
     public static Operation Delete;
}
Conclusion: Meta-programming with attributes

I understand that all those examples (or at least the two last ones) are too specific to come into C# features, but there's a fundamental feature that could dramatically remove lots of redundancy in C# code: Some controlled way of meta programming.

Now that Roslyn is open source, we need to be able to intercept the compiler at some stage (after Syntax tree?) and be able to make transformations on the code,

By using attributes, we keep the meta-programming capabilities inside of the C# standard syntax, so we should have a safe playground that does not interfere with the rest of the language and tooling. I think Scala does something similar expanding methods.

Ideally, the implementation of this transformation will be triggered by calling an abstract method ExpandTree in an abstract CodeExpanderAttribute or something like this. So we should be able to write the code in C# using Roslyn and provide this transformations with the rest of the library, without danger of fragmenting C#, and making it more straight forward and official than using PostSharp.

To complete my naive feature specification/wish list, I suppose that the class library that provides this CodeExpanders should be independent and not use the code expanders itself, to avoid insane cyclic problems.

Sorry for such a long message, and thanks again for the nice work!

Olmo
Apr 4, 2014 at 9:23 AM
Edited Apr 4, 2014 at 9:27 AM
Hi all,

My name's Eugene Burmako. I'm Martin Odersky's PhD student at EPFL and the lead of the Scala macros project. It is very exciting to witness the public launch of Roslyn and the fact that it's sparked a compile-time metaprogramming discussion!

Since you mentioned Scala, I'd like to share some of our experience with macros. In a nutshell, we have two big categories of macros: def macros (methods whose applications expand at compile-time) and macro annotations (something very similar to what you are proposing). Now some more details.

Def macros are part of official Scala distribution starting from Scala 2.10 (released in January 2013). Def macros have been well received in our community, having become popular in both industrial and academic projects. We've discovered a number of interesting techniques related to def macros, including materialization, macro-powered string interpolation and anonymous type providers. If you're interested, here are some very recent slides that showcase the most important use cases: http://scalamacros.org/paperstalks/2014-02-04-WhatAreMacrosGoodFor.pdf.

Macro annotations are implemented in a compiler plugin called Macro Paradise that is available for all macro-enabled Scala versions. Macro annotations are very much in line with this proposal - programmer writes a custom annotation with a macroTransform method, which takes abstract syntax trees, and then its usages are expanded at compile-time. Here's a quick example from a today's StackOverflow question: http://stackoverflow.com/questions/22850340/dynamicly-creating-case-classes-with-macros. At the moment, we are still considering whether macro annotations are going to make it into future versions of Scala, because there's a number of issues that we need to handle first. The most prominent one is how to allow typechecking during expansion of macro annotations. Here's (quite a convoluted) discussion about this: http://stackoverflow.com/questions/19379436/cant-access-parents-members-while-dealing-with-macro-annotations.

There are also a few philosophical questions that we've uncovered when experimenting with macros in a statically-typed language such as Scala. It would be interesting to discuss those:

1) Macros vs types. In Scala there's a strong tradition of doing metaprogramming with types. Thanks to the work of our type astronauts (here's one example: https://github.com/milessabin/shapeless), our community has learned how to use type-level programming in non-scary ways to tackle interesting real-world use cases. Very much like types, macros also operate at compile time, so areas of applicability of macros and types naturally intersect. In our recent talk, we summarize our findings on this matter: http://scalamacros.org/paperstalks/2014-03-01-MacrosVsTypes.pdf.

2) Blackbox vs whitebox. I think the best idea of the entire Scala macros project was to make macros integrated with the typechecker. This way, we can do advanced code analysis, can perform type-drived code generation, and a lot of other type-powered things. Also, requiring arguments to def macros to pass the typechecker first, before the macro expansion is allowed to start, has been very helpful in limiting the power of macros to manageable levels. However, sometimes we need more power (e.g. to allow macros generate new definitions that the program won't typecheck without). How do we do that and is it worth it? We don't have a definite answer yet, but here are some ideas from our recent StrangeLoop talk: http://scalamacros.org/paperstalks/2013-09-19-PhilosophyOfScalaMacros.pdf.

3) Infrastructure. As practice has shown, it's not enough to just expose a metaprogramming API (trees, symbols, type, etc), provide people with a way to write functions against it, and then just call into these functions from inside the compiler (this is exactly what we did in Scala 2.10). Numerous infrastructural problems need to be solved in order to ensure optimal developer experience. Is the API really easy to understand for people who don't have experience with compilers? Can macros be used in the same project that defines them? How do we report errors in macro-generated code? How do we debug such code? What about incremental compilation? This is something we're working on right now, and here's a brief plan of our experiment: http://scalamacros.org/paperstalks/2014-03-02-RethinkingScalaMacros.pdf.

I think this is it for a short summary of our experience, though I would certainly be glad to elaborate more. Congratulations for the release and looking forward to see what future brings now that there's an official metaprogramming API for the .NET platform!

Cheers,
Eugene
Apr 4, 2014 at 11:18 AM
Hi Eugene,

It's a pleasure to find you here. I don't have any experience with Scala and Macros, but I watched your talk on InfoQ (http://www.infoq.com/presentations/scala-macros) and it really make and impact on me. Nice work!.

I understand that the main issue is for the tooling to be able to work with the generated code. It's not similar to the code generated for lambda clousures/iterators than can be hidden away from the environment as an implementation detail. Some question arise:
  • Is there auto completion for generated code?
  • Can you debug generated code?
  • Can you go to definition?
I think even if the answer is no for all this questions, there are really useful scenarios. In my three examples I don't really need any auto-completion/go to definition for the generated code, and I will be happy if the compiler shows 'external code' in the stack trace if there's an exception.
I think this is it for a short summary of our experience, though I would certainly be glad to elaborate more. Congratulations for the release and looking forward to see what future brings now that there's an official metaprogramming API for the .NET platform!
I think you confuse it here, there's an open source compiler (Roslyn) that is now written in C# and has a beautiful API to write language services on top of them, but I think Roslyn has no extension points to allow meta programming without changing the compiler itself and forking C#.

I will read all the information you provided to have better arguments to try to persuade the C# team to make those changes

Thanks!

Olmo
Apr 4, 2014 at 11:39 AM
Depending on who you ask, definitions of metaprogramming might be different. For intance, wikipedia says that metaprogramming is writing programs that write or manipulate other programs as data. In that sense, I think, Roslyn API can be viewed as a metaprogramming API, even without the lack of a compile-time host for it.

Speaking of issues. Yes, tool support is one of them. However, there's at least one more important issue, related to integrity of APIs. From what I understand, Roslyn provides a way to explore entire programs and to typecheck abstract syntax trees. Having it working in a read-only setting is one thing, but having it working in situations when there are macros that generate new definitions is a different story. E.g. should a macro see definitions introduced by other macros? Or how do we make sure that program semantics is independent of macro expansion order?
Apr 5, 2014 at 10:59 PM
Hi again Eugene,

Scala macros

After reading your slides, and with my limited knowledge of Scalar, I've understood that there are the following type of macros:

def macros (method expansion) are type-checked after and before expansion, this makes devs intuition work and have been proved successful implementing many patterns, among them:
* Async/Await
* Slick  (LINQ for Scala, to generate expressions for IQueryable<T>)
* Trick to make performance optimizations using code generation (example with mathematical Rings)
* Serialization (is that what Pickle is about?)
* ...
type macros: Allow generating types by inheriting from a magic base class that will expand the child classes by running the macro. Are white boxed macros because the generated type is then available for the programmer (I.E. Generating database schema classes like F# type providers).
Proved less successful. Too magical?. Are now deprecated in favor of Macro Annotations

untyped macros: Similar to typed macros but the code inside of the class itself doesn't have to be valid Scala. (I.E. Implement enums). Also deprecated (in favor of macro annotations?)

macro annotations: Generalize the type macros by triggering the macro expansion in any (any?) member that has the annotation, allowing more flexibility than type macros and preparing the developer for the magic (they are used to magic when there are annotations involved).

Back to C#

While I find def macros a really beautiful and general solution, most of the use cases are already covered in C# one way or another:
  • By using an ad-hock syntax (linq, async / await, iterators..)
  • Are not a pressing topic because here we have value types (mathematical Rings)
  • Can be implemented by generating, compiling and catching lambdas at run-time (serialization).
While I'm sure there are still good scenarios for method expansion in C#, I can not come up with any just yet. Some ideas?

My proposal of using meta-programming with attributes in C#, as you said, is really similar to your implementation of macro annotations, and while macro annotations are (if i understood it correctly), an improved version of the not-that-successful type annotations, I think in C# will be different:

I think C#, with lambdas, expression trees, async/await etc... it's quite expressive at the expression/statement level. There are some limitations, like pattern matching, but I'm not sure if it could implemented without new syntax anyway. And at the expression/statement level the developer can easily factor out method to remove redundancy.

On the other hand, on the class/member declaration level, there's much more clutter. All this public/private/protected/virtual/abstract/sealed/override/readonly attributes add a lot of boilerplate code. And there's no way to factor out things like the declaration of a property/method/class etc...

An initial implementation could be quite conservative: without allowing the macros to declare any visible member I can cover my three use cases, with a low risk of interfering other language features and services.

Also, your answer on http://stackoverflow.com/questions/19379436/cant-access-parents-members-while-dealing-with-macro-annotations suggest that the immutable design of Roslyn could also simplify writing safe annotations to end users.

Let's see that the C# team thinks before attempting to write an initial attempts. Possibly it's way to complicated and I should start with something simpler :)
Apr 6, 2014 at 4:23 PM
Nice discussion here!

I just want to add that I really like your second proposition. The first time I saw Expression-bodied members I thought exactly about what you wrote.
Apr 6, 2014 at 5:03 PM
Thanks! We implement ExpressionMethod (as we call it) today, but with some meta-programming we could remove redundancy.

Let's see it this thread catch the attention of the C# team :)
Apr 7, 2014 at 8:02 PM
Have you looked at Nemerle and how one does meta-programming in that language? Anything that can be adopted by C# and VB?

http://nemerle.org/
Apr 7, 2014 at 8:54 PM
Note: I'm aware that we can use CallerInfoAttributes to avoid the lambda, just waiting to get rid of the Windows XPs out there.
What does one have to do with the other?

The C# 5.0 compiler just needs the attribute to exist. If it doesn't exit on your targeting framework, just create one yourself. I've implemented extension methods and a good part of LINQ for the 2.0 RTM framework just by using the C#3.0 and/or C#4.0 compilers.
Apr 7, 2014 at 9:38 PM
The C# 5.0 compiler just needs the attribute to exist. If it doesn't exit on your targeting framework, just create one yourself.
It's true! I'd forget that all the compiler features that depend on the BCL do so just by class names and conventions.

Thanks a lot! I will start applying it ASAP.
Apr 8, 2014 at 9:26 PM
Hi Olmo,

It looks that attribute-based macros might indeed be a better fit for C# than something like def macros. A lot of power of def macros in Scala comes from the fact that a lot of language features are desugared to / can be represented with method calls (even type providers, for example, can be emulated with def macros: https://github.com/travisbrown/type-provider-examples), so this part of def macros might be less relevant to C#.

As for your summary of the macro landscape in Scala, it's all correct. At the moment, there are only def macros and macro annotations. Type macros and untyped macros were experimented with and then abandoned. Type macros seemed interesting, but then the only real usage that they found was really an abuse (inheriting from a type macro allowed that type macro to see and update declarations of the child class), so we moved on to macro annotations which could do the same and more, but in a much more natural way. Untyped macros provided a lot of additional power, but surprisingy there were no real killer features that utilized such power, so we decided to live without untyped macros in order not to risk losing control over the language.

Cheers,
Eugene
Apr 12, 2014 at 7:32 AM
@Core team, maybe you could comment on this one? It would be interesting to learn about your plans wrt compile-time metaprogramming with Roslyn.
Apr 12, 2014 at 3:59 PM
+1 for more Core team involvement. We understand you have schedules and deadlines but would be nice to hear your opinion.
Apr 15, 2014 at 10:07 PM
I'm watching this discussion trying to find time to jump in, but I'm too snowed with a conference this week where I am talking about Roslyn (metaprogramming workshop on Thursday, which will include Roslyn). I will try for next week. The depth of the conversation is fantastic and I do not wish to toss out lightweight thoughts.
Apr 16, 2014 at 12:20 AM
Olmo, why didn't you use class inheritance? or Interface Inheritance?
namespace example
{
    interface ISymbol { };
    interface IOperation        : ISymbol { };
    interface ITaskOperation    : IOperation { };
    interface ICreaateOperation : ITaskOperation { };
    interface ISaveOperation    : ITaskOperation { };
    interface IStartOperation   : ITaskOperation { };
    interface IStopOperation    : ITaskOperation { };
    interface ICancelOperation  : ITaskOperation { };
    interface iDeleteOperation  : ITaskOperation { };
}

Apr 16, 2014 at 6:44 AM
AdamSpeight2008:

Looks like an idea... let's see:

We use operations as strongly-typed key to refer to a method in the server. They can be implemented (associated to an action) and invoked.

If they would have been interfaces, all the methods that take a operation should be generic, complicating things a little, and the operations are sent through a web service, complicating things a lot.

A similar possibility will be that the operations will be classes, this way we could instantiate them and send them. But there will have to be marked as serializable.

Additionally Symbols, as well as Enums, have the nice property of being identified by [Namespace].[Type].[Value] while with classes will need more complicated names, or make them inner classes.

Adding all up and this is how the code will be:
namespace example
{
    //This we already have
    [Serializable]
    public class Symbol {... };
    [Serializable]
    public class Operation : Symbol { ... };

    //new code
    [Serializable]    
    public class TaskOperation    : Operation 
    {
        [Serializable]public class Create: TaskOperation { }
        [Serializable]public class Save: TaskOperation { };
        [Serializable]public class Start: TaskOperation { };
        [Serializable]public class Stop: TaskOperation { };
        [Serializable]public class Cancel: TaskOperation { };
        [Serializable]public class Delete: TaskOperation { };
    }
     

     //or maybe this one, because they don't have to inherit from TaskOperation
    public static class TaskOperation 
    {
        [Serializable]public class Create: Operation  { }
        [Serializable]public class Save: Operation { };
        [Serializable]public class Start: Operation { };
        [Serializable]public class Stop: Operation { };
        [Serializable]public class Cancel: Operation { };
        [Serializable]public class Delete: Operation { };
    }
}
The declaration is actually not that bad, is worth considering, but there are other problems:
  • Using the new syntax, refering to a operation will mean writing new TaskOperation.Save() instead of just TaskOperaton.Save. A little bit more annoying.
  • By using static read-only fields, we don't create new objects every time, so there's less memory pressure.
  • More important, we make some tricks at start time to associate the current Id in the database to the instance in the operation field, so they are just normal entities that can be saved as part of other entities (OperationLog for example, that associates who invoked what and when). This will be impossible with types and we will need to start making conversions again, this is the reason we have just invented symbols instead of using enums. https://github.com/signumsoftware/framework/commit/8fb3a1f975a204e8625ce23ffd69506e68f359f6
Finally, thanks to the CallerInfoAttribute trick from PauloMorgado, and other trick involving StackFrames we can now write:
public static class TaskOperation
{
     public static Operation Create = new Operation();
     public static Operation Save = new Operation();
     public static Operation Start = new Operation();
     public static Operation Stop = new Operation();
     public static Operation Cancel = new Operation();
     public static Operation Delete = new Operation();
}
So it's not that bad, but a little bit slower, than if we used meta-programming.
Apr 18, 2014 at 2:48 AM
Olmo. thank you for the rationale. Few coders will give that to why they've not doing something a particular way.

Why would it need to be generic? Since inheritance is being used (derived from ITaskOperation.)

I think you do something like

void TheMethod( ITaskOperation o )
{
 if      ( o is ISaveOperation   ) { }
 else if ( o is ICreateOperation ) { }
 else if ( o is IStartOperation  ) { }
 else if ( o is IStopOperation   ) { }
 else if ( o is ICancelOperation ) { }
 else if ( o is IDeleteOperation ) { }
 else
 {
  // unknown ITaskOperation //
 }
}
Apr 18, 2014 at 7:43 AM
Why would it need to be generic? Since inheritance is being used (derived from ITaskOperation.)
Because you actually have to call the method, and you can not instantiate interfaces:
TheMethod(new IStopOperaton()) //Doesn't compile
So you either:
TheMethod(new StopOperaton()) //  make it an object instead

TheMethod<IStopOperation>()  // make it generic

TheMethod(typeof(IStopOperation))  // of pass the typeof
Aug 22, 2014 at 3:28 PM
Edited Aug 22, 2014 at 10:53 PM
Metaprogramming with extended syntax: https://metah.codeplex.com/wikipage?title=Metah.W%3a%20A%20Workflow%20Metaprogramming%20Language&referringTitle=Home
Image
Sorry, the docs is in Chinese, I'm learning English :)
Aug 24, 2014 at 7:12 AM
Knat wrote:
Metaprogramming with extended syntax: https://metah.codeplex.com/wikipage?title=Metah.W%3a%20A%20Workflow%20Metaprogramming%20Language&referringTitle=Home
Image
Sorry, the docs is in Chinese, I'm learning English :)
Hi Kant. Your project looks interesting but I'm not sure if it's the same meta programming we are talking about.

Here we are talking about the ability to write c# attributes that modify c# expression trees at compile time.

Your project looks more like an API for writing some kind of visual/component block programming. Something like workflow foundation?
Aug 24, 2014 at 8:37 AM
Hi Olmo,
It's a discussion of meta-programming with attributes, but my project is about meta-programming with extended syntax, I only saw the word "meta-programming", sorry again :)
BTW, Roslyn doesn't have any abilities of meta-programming, we have to do some "dark magic".