Open source infrastructure work going forward

Topics: General
Oct 20, 2014 at 6:04 PM
Hi, all,
As I've mentioned elsewhere, we're starting to work on improvements to our infrastructure -- both our internal infrastructure as well as our interaction with the open source world. This is something that's been on our agenda for awhile, and finally we're at a point where we can apply some real thought and time to it.

The TL;DR version of this post is:
  • We want to address pain points that have become apparent since we went OSS, including bug status transparency, fixing coverage of scenarios that only affect the OSS community (as opposed to internal usage adjacent to closed code), and getting more feedback from Roslyn team members.
  • I'd like to use this discussion to bring together issues that have been discussed elsewhere in this project.
  • I (or others) will do my best to respond to all of them. We won't always be able to act on the suggestions, but we'll be sure to comment on them. It's likely that my own responses will be "batched" to something like once/day, but others from my team will probably jump in as well.
OK, so on to the main show.

OSS has been a real learning experience for us, despite having a few folks on the team who were pretty knowledgeable about it to begin with. We'd always intended to regularly revisit the infrastructure as we discovered areas for improvement, but we got unexpected slowed down on doing that, and I thought folks might be curious as to why. It's actually a Good Thing(tm) in the long run, although it's caused us to have our attention focused elsewhere in the short term. Basically, we completely re-org'd the team earlier this summer to all be software engineers, as opposed to separate devs and testers. This is pretty cool, as it allowed us to bring all of those learnings together, and in particular made sure that both coding and the testing of it were as close together as possible.

Anyway, we all had to learn new roles -- former devs had to learn how to write good tests, and former testers had to write production code. We decided that we'd plunge right in and get all of that figured out from the beginning, so that we didn't end up having people stick with their "safe school." This, combined with a big build infrastructure change related to how we build into Visual Studio (which I won't go into), took some time to sort itself out.

Now, constant OSS goodness is certainly a big goal, but we also had another big goal which we couldn't miss: getting the APIs locked down. Until we did that, no one could take a bet on us without being randomized constantly -- not in the OSS community, and not within VS. It had to get done. And that's where much of the resources went while we were learning these new jobs.

Now that things have let up a bit on the API front, we're having discussions on infrastructure improvements. We've had two meetings on this already, and the plans we created will start being implemented in the next sprint (which starts October 27th). The changes will probably go well into the new year, but some goodness should start to roll out immediately. Here are the OSS-specific ones. (There are VS-specific ones as well, but they don't affect this discussion.)

First of all, we want to get more of our folks onto the discussions, and there's nothing blocking that -- that should just happen, and we're driving that already.

Second, we want to implement a way to ensure more status is being driven back to you on bugs, keeping you informed while still protecting internal discussions touching on closed code.

Third, we are going to investigate making more of our Roslyn-specific code open in the next year, which will help eliminate some of the weird boundaries we have to work around currently. (Our F# folks have already done this, and it's helped a lot.) I'd already touched on this direction vaguely in the OSS announcement blog earlier this year, but until now we actually haven't moved much on this yet. We'll start thinking about this now, so stay tuned.

Fourth, we are investigating a move to GitHub. I won't make a promise on a date yet -- there are still some important things we need to figure out about such a move -- but it's very much an active conversation, and something we'd like to do. I couldn't see any valid reason to withhold this information, given the community interest in this issue, so there it is.

Fifth, we want to invest more into using what the community uses where possible, for example exploring Git usage for us internally on everything (closed and open), or converging on a test framework that works on both sides for everything. That will just make us more agile when dealing with two worlds. And where we explicitly can't do this for internal reasons (yeah, those will always exist), we want more robust automation around any infrastructure interfaces between our closed and open worlds.

Sixth -- well, you tell me what six, seven, and so on should be. (But let's keep it focused on infrastructure and interaction, and not feature requests, please!) Bear in mind that all of the above (except the first point) are directions we are taking, and not (yet) promises, though obviously it's in our best interests to make good on them where we can.

"Roslyn" Manager
Oct 22, 2014 at 12:45 AM
Edited Oct 22, 2014 at 12:50 AM
This is all great news, and I'm really glad to hear it.

Responding to your call for suggestions. It would be nice to clear up the pull request infrastructure to allow willing people to contribute to Roslyn more easily. I'm sure it will help a lot in the long run, since there are many things on your backlog that the community could help with ("up-for-grabs"), but the existing infrastructure doesn't really make it easy. I also believe there will be many more willing contributors on GitHub than on CodePlex if you actually decide to move there — this is something that the TypeScript team has already experienced.

Currently there are two main hurdles in contribution:
  1. There is no transparency in your timeline. Clearly, VS has a fixed release schedule, and you have to align any features with it (although it has been mentioned earlier that you are considering shipping small Roslyn updates in the future instead of holding off all of the features until the next VS release). However, since the OSS community doesn't know the schedule of your sprints, we don't know whether a particular PR has a chance of acceptance at this point of time, or is it going to be postponed since your current sprint only focuses on bug fixing.
  2. Contributing features has an unpredictable response. If I decided to implement any of the low-hanging fruits in C# that you are postponing at the moment (e.g. digit separators, binary literals, etc.), I wouldn't know whether this PR even has a chance of consideration. I understand that accepting a feature PR requires (a) a dedicated reviewer on the team, (b) time to make the review and integrate the changes, and (c) resources to sync the changes with the closed code, such as the IDE layer. However, I believe that allocating some resources towards (a) and (b) will actually raise a load off your shoulders, even if you still have to deal with (c) on your own.
    F# team has actually done a great job of categorizing small features and reviewing the progress on it. They have integrated several dozens of pull requests by now, if I'm not mistaken.
It would be nice to hear some of your thoughts on this. And, once again, thank you for all of the great work!
Oct 22, 2014 at 5:10 AM
Thanks, Skiminok. Yeah, I should have brought that up in the original points, it's a good topic.

On timeline issues, we sort of get into murky ground because of the weirdness generated between open code and closed code. I'll explain: I can't, for example, broadcast or even hint at VS ship dates (or even preview dates) for the simple reason that This Would Be Frowned Upon by folks smarter than I about our business. (I was about to make a joke that I'd like find my stuff waiting for me outside of the building in a cardboard box, but in fact what would probably happen is that I'd be forced to wear the "Clippy" costume as punishment for some undetermined amount of time.) But it's those same VS timelines that drive our ability to take significant pulls. Furthermore, like my peers over in .NET land, we have to shut ourselves down earlier than the rest of VS so that they can stabilize on us, which means that any pulls that we take in the interim have to be limited in scope and not something that (for example) would either change the language or the structure of trees returned by the APIs.

There's also a psychological component for the first release of Roslyn, which boils down to the fact that we've been working on this for six years (or eleven, depending if you count some blue-sky hallway conversations), and so for sanity's sake we need to tie it up with a ribbon and put a v1 stamp on it before we start the next phase and bring in meatier pulls. (That's a sorry defense, to be sure -- I realize that there's nothing really magical about a version number, which is really just a point-in-time thing-- but the VS point stands nevertheless.)

To be fair, there are plenty of pulls that don't cause those complications, and -- mea culpa -- we can do better there, and will. But those are the same sort of pulls that aren't affected by timelines, and by corollary maybe aren't quite as exciting, albeit still very much appreciated.

So, what can we do about this? Well, one thing we can do is finish open-sourcing the rest of Roslyn. It's probably no surprise that the most tightly-bound consumer of Roslyn here in MS currently is (wait for it) the rest of Roslyn, so by tying those pieces together so that they can update to the OSS world at the same time, we make significant inroads into what I call the "dissonance" problem. This is one of two reasons, for example, why our F# team can take on so many pulls (as you note, those guys are like machines these days) -- it's easier to get relevant pieces working together because the community can take care of all of that with the pull, as opposed us taking pulls and then having to painstakingly patch the rest of Roslyn and VS if required, depending on the pull type. (The second reason is that VB and C# have more internal usage in VS.) And this sort of thing really happens as we make changes -- we have an SLA with the rest of VS which essentially boils down to "if we break it, we buy it," but the impact to ourselves is already high as we are trying to shut down. There will still be potential patching up around VS in that future full-Roslyn-OSS world, but rather less of it.

To your second point: right now, the rule-of-thumb we'll be following as we start to engage more will be something like:
  • Infrastructure patch: sounds great, let's talk.
  • Bug fix which doesn't alter language, APIs, or the return values from the APIs: cool, let's look at it.
  • Got a new static analysis component: definitely, we are very interested in that.
(Now, of course, we've got to do that, which is part of this re-engagement phase. We also have to categorize and communicate the "whys," as you point out.)

Anything more fundamental (like a new language feature or an API alteration) would have to wait until we fork the v1 codebase, which will be after we really and truly lock down the APIs & language. Frankly, we are actually very interested in turning the community loose on some meatier stuff we're interested in for v2, as well as stuff that the community comes up with itself -- in fact, my management has brought this up several times as well, so there is a lot of support for that.

Apologies, long response for a simple couple of questions, but this is a topic that we think about a lot.

Oct 22, 2014 at 8:30 AM
Edited Oct 22, 2014 at 8:32 AM

My name is Kevin Ransom, I work on the Managed Languages team, normally I work on F# and am also a member of the Roslyn development team. Our goal is to make each of the managed languages open source projects productive places for collaborative development. We want to enable developers such as yourself to be able to improve your working environment by enabling contributions to the tools you use every day.

Working with a Microsoft Dev team is tough, we have a ton of processes and experiences that make a lot of sense to us and seem nuts to people who don't experience what we do on a daily basis.

For example, our worst fear is that we introduce an undetected subtle change that cost the Windows division or Office division a couple of days of work. It never happens because we obsess about the possibility and are super conservative about our process, but it is still a fear.

Right at the moment we are laser focused on shipping out V1.0, what that means to us internally is that we are fixing bugs, improving testing, and nailing the code for our approved features. You will notice that recently we cut a bunch of features that have been in the codebase for the last 9 months because we couldn't nail down the exact semantics and commit to the behaviors implemented. Even low hanging fruit features need a lot of thinking about right now. Bugs on the other hand are a different thing, if you can provide great fixes to the very many bugs that we have, then we will consider them for inclusion in V1.0. There are no guarantees, but they will be considered. Take the issues that look like bugs not features, and see whether you can fix it. Take a look at our tests create new ones, and better ones, if it looks safe and useful I will personally do what I can to advocate for us to take the fix.

We are right at the beginning of open sourcing Roslyn, small targeted and safe contributions will be considered I will ensure that they are. Stay away from new language features, the bar for them is just too high this late in the game, fix bugs, improve tests, documentation and infrastructure.

If you need a chat you can mail me at kevinr at microsoft dot com.

I look forward to working with you on making specific improvements to Roslyn.

Oct 22, 2014 at 12:22 PM
I'm not part of this project and haven't even been following it that closely, but this discussion has been linked a great deal over the last couple of days, so here are my thoughts:

I think there's a need for pragmatism/realism when dealing with an open source project on the scale of Roslyn. By scale I mean not just the number of modules, files, or lines of code, but the vast array of users of all kinds who can be affected by tiny changes to the core code. Most of us just never have to deal with this sort of thing.

I see the open sourcing of Roslyn more as transparency than anything else: a way for the community closely to follow what's happening day-to-day; to download, build, test and experiment with the code; and to contribute in particular ways.

One of the best ways to begin to contribute to any open source project as a newcomer is to submit tests - either failing ones, or potential missed regressions - as a kind of asynchronous, remote pairing exercise with the core team. From there you can move on to submitting patches along with the tests.

And of course every open source project loves it when somebody helps with the documentation :)
Oct 22, 2014 at 1:42 PM
There's also a psychological component for the first release of Roslyn, which boils down to the fact that we've been working on this for six years (or eleven, depending if you count some blue-sky hallway conversations), and so for sanity's sake we need to tie it up with a ribbon and put a v1 stamp on it before we start the next phase and bring in meatier pulls. (That's a sorry defense, to be sure -- I realize that there's nothing really magical about a version number, which is really just a point-in-time thing-- but the VS point stands nevertheless.)
This isn't "psychological." At this point, shipping Roslyn is a feature we all badly need.

To date (20 years) I think I've seen only one visible feature go into the languages because of a single person's need (an expansion of event definition in VB). Feature changes, even minor ones like nameof, take a massive amount of thought and effort. That shouldn't change, can't change! I think it's very hard for people to understand how hard language and API design work is and just how much sideways "how is this going to bite us later" kind of thinking. We've set aside dozens of potentially cool features because the side cases are just too nasty, the impact of something I really want (like better attributes) cuts to the core of the .NET framework, or is fundamentally at odds with the language vision (XML literals in C# and JSON literals in both languages). It's so cool that we can steal patterns of OSS as well as technical patterns from F#, but C#/VB aren't F#. C# and VB have an enormous, enormous audience whose primary, perhaps only, need is stability.

The solo player and small external group can do many things - prototype features without ego (which means you may contribute without a pull being accepted), get involved in discussions, tests, bug fixes, documentation and now analyzers. Adam's string format analyzer comes to mind!

I believe we should work at better respect for help in the areas you most need. I mean us as a community - treating these actions by the community as super-cool big time contributions. I do see action items as making it easy to identify area for small scale contribution, a sane path for documentation inclusion, a path for prototype submission that might be separate from the pull request model, and recognition for these efforts, such as easy inclusion in MVP status.

Oct 22, 2014 at 3:43 PM
@markrendle: I love the idea of submitting tests for consideration -- in fact, testing as often as not reveals additional little holes that need to be plugged, and fixes for such could go along as well. Documentation is also an area where we already benefit from community feedback in many forms (lectures at UGs, blogs, books, etc.) and which I'd like us to make more inroads into on the actual OSS site.

@KathleenDollard: Many thanks, that's exactly it (and using far less words than I did). The tension created by the community pushing for more is healthy and helps us narrow down on the proper future direction for the language, but at some point it leads to tough decisions about what's in and what's not. For our part, we strive to create and grow languages and platforms where each piece has a specific and high-impact part to play, set out fair and square in a logical plan, with as little ambiguity as possible between the pieces -- and with a support cost going forward that's commensurate with the benefit of the feature. The cool bit is that we now get to have these conversations in the open and get feedback faster -- and furthermore, with the prototyping evolving visibly in real-time. (OTOH, it means we have to be even better about labeling what's experimental and potentially cuttable vs. what's set!)
Oct 22, 2014 at 5:27 PM
Wow, thanks for all the responses! It was certainly much more than I anticipated.
I actually want to apologize if I accidentally made an impression that I'm accusing you of something, or pushing something — because I absolutely don't. I wanted to propose some discussions points based on the issues that worried me in the past. Of course I understand the complexity that comes with you trying to juggle both closed-source world and open-source world at once. Hopefully the further release of more Roslyn layers with help with that.

Kathleen actually raised an interesting point. The model you guys are pursuing is different from the "typical GitHub OSS workflow". Since Roslyn is a project of huge complexity, and nobody can comprehend the consequences of any single change better than the MS team, it makes sense that the goal of open-sourcing it is more along the lines of getting feedback, rather than contributions. However, it also means that we, as a community, can get a chance to experiment with the workflow. For example, prototyping a feature or a spec fix is certainly not going to be simply accepted as a pull request. However, there should be a place for this (maybe with a different name rather than PR), to give you something to start off a discussion. I believe that the discussion the community raises is beneficial nonetheless, even if the feature is not going to be included in the current development cycle.
Oct 22, 2014 at 5:34 PM
@Skiminok: Oh, no worries -- I totally got what you meant and I think we are agreeing. I have a character flaw in that I tend to use any innocent question as an opportunity to address additional related but unasked questions as well, just to tie everything up neatly -- that's a dangerous thing when combined with my pronounced tendency towards verbosity. :-) (Writing is one of the two vices I indulge myself with -- the other involves cardigans, but I'm not sure it's relevant here.)
Oct 26, 2014 at 12:57 PM
What is the focus around performance on that v1.0 launch? We are running into a few performance issues in compilation at Stack Overflow especially around PDB generation and aren't sure how much time we should invest there. We're currently using Roslyn for compilation so that we can do localization at compile time (both extracting the strings and replacing them with a localized, high runtime performance replacement).

We're incredibly obsessive about our deploy times over here and the biggest part of that is the actual code compilation - taking 2 minutes to dump a PDB and such are blockers to adopting certain things there. The dev most directly working on that piece is working on a minimal-case repro so that we can submit an issue, but are these things concerns for V1, or just core compilation performance?

FWIW, We're very good with source control here...but Github is still a much friendlier interface around the pull requests. Half the time we're not getting email when feedback happens on already submitted contributtions on Codeplex (for example, this contribution to the ASP.Net stack has sat simply because Marc was not alerted - I've experienced the same...we'll fix that up this week now that we know). Needless to say, we definitely support a Github move for this and other reasons (e.g. issue search).

I also echo @markrendle's sentiment - we'd like to submit tests for issues we find rather quickly then dig into fixes and such after. It's a very quick way to alert you to the issue and ensure it's "on the list" so to speak. We currently have one break around bit shifts that I'm not sure if it's been communicated since there are several channels inside and outside of Microsoft - and some of them are restricted (e.g. MVP) channels. I'd love it if we could submit an issue and a test directly. Microsoft could also add any issues to (what I hope would be) a single Github issue list that is all-inclusive of known bugs. Right now (at least it seems to us) there's a bit of a split of internal/external known issues.

We love what you guys are doing; it allows us go nuts over here and implement what were insane solutions before Roslyn was available. Keep it up.
Oct 26, 2014 at 1:51 PM
@Nick -- thanks for the feedback! Right now, for the perf metrics we are tracking (which is quite a few), Roslyn is meeting or beating VS2013 except in one case -- VM usage for very large solutions, which is still something we are taking a look at. We are not, to my knowledge, tracking PDB generation specifically (except insofar as we are tracking build all-up, and like I say that's definitely in line). A couple points -- first, we've made a lot of perf enhancements recently, and second, the individual times of, say, build are not necessarily equivalent version to version even if the overall time is the same.

I'm going to ask my resident perf guru, Paul, to follow up here on your specific question. (Note that I am on a business trip in Serbia this week and so my responses will be delayed and at odd hours.)

Oct 26, 2014 at 8:29 PM
@Nick, we're not tracking PDB writing in isolation, but 2 minutes to dump a PDB is high. How big are the resulting PDBs and binaries? We would definitely be interested in seeing the minimal repro.
  • Paul Harrington (MSFT). Roslyn performance lead.
Oct 26, 2014 at 9:22 PM
@pharring Thanks for the quick reply.

The PDBs in question (for the Stack Overflow main project) is about 15.7MB, but I don't have details just yet. We found the issue just before our meetup last week and it hasn't gotten additional love yet. We plan to attach a profiler this week and see what's happening there. The intent here is to replace the current csc.exe in Stack Overflow's build process and do the localization magic in one pass. Samo (our main dev working on this) or I will follow up as soon as we have more info.
Oct 27, 2014 at 6:32 AM
Edited Oct 27, 2014 at 6:32 AM
I'd just like to say the recent response from the team has been great. Over the last two weeks I've seen a lot more engagement from Roslyn team members on the Issues page, which is really awesome.

Everything you guys have touched on has been really reassuring and I'm excited to see things continue to move forward. I can certainly imagine how overwhelming it must be to be nearing the end of a six-year journey and I really think it was worth it. Roslyn is a fantastic tool for the "everyman" programmer such as myself. :)
Oct 29, 2014 at 12:19 PM
Thanks, Josh -- appreciate the feedback! We're enjoying the interaction a lot as well.

Nov 12, 2014 at 3:07 PM
Just wanted to add a quick note based on all of the announcements today at Connect() -- in concert with the new VS preview that's been announced, we will also be creating a fork of the Roslyn code matching that preview as usual. This will go live later on today.

Nov 12, 2014 at 5:48 PM
The source code matching the new VS preview has been published. We need to update some scripting so that people wishing to push their own modified version of that code into that VS can do so (for various complicated reasons that I won't go into, it's not something that we could do proactively), and we are actively working on that today.