There’s been a lot of discussion lately around ergonomic ref-counting. We had a lang-team design meeting and then a quite impactful discussion at the RustConf Unconf. I’ve been working for weeks on a follow-up post but today I realized what should’ve been obvious from the start – that if I’m taking that long to write a post, it means the post is too damned long. So I’m going to work through a series of smaller posts focused on individual takeaways and thoughts. And for the first one, I want to (a) bring back some of the context and (b) talk about an interesting question, what should we call the trait. My proposal, as the title suggests, is Handle – but I get ahead of myself.
This blog post gives you a tour of Symposium, a wild-and-crazy project that I’ve been obsessed with over the last month or so. Symposium combines an MCP server, a VSCode extension, an OS X Desktop App, and some mindful prompts to forge new ways of working with agentic CLI tools.
Symposium is currently focused on my setup, which means it works best with VSCode, Claude, Mac OS X, and Rust. But it’s meant to be unopinionated, which means it should be easy to extend to other environments (and in particular it already works great with other programming languages). The goal is not to compete with or replace those tools but to combine them together into something new and better.
In addition to giving you a tour of Symposium, this blog post is an invitation: Symposium is an open-source project, and I’m looking for people to explore with me! If you are excited about the idea of inventing new styles of AI collaboration, join the symposium-dev Zulip. Let’s talk!
You heard it here first: my guess is that Rust, Python, and TypeScript are going to become the dominant languages going forward (excluding the mobile market, which has extra wrinkles). The argument is simple. Increasing use of AI coding is going to weaken people’s loyalty to programming languages, moving it from what is often a tribal decision to one based on fundamentals. And the fundamentals for those 3 languages look pretty strong to me: Rust targets system software or places where efficiency is paramount. Python brings a powerful ecosystem of mathematical and numerical libraries to bear and lends itself well to experimentation and prototyping. And TypeScript, of course, is compiled to JavaScript which runs natively on browsers and the web and a number of other areas. And all of them, at least if setup properly, offer strong static typing and the easy use of dependencies. Let’s walk through the argument point by point.
Recently someone forwarded me a PCMag article entitled “Vibe coding fiasco” about an AI agent that “went rogue”, deleting a company’s entire database. This story grabbed my attention right away – but not because of the damage done. Rather, what caught my eye was how absolutely relatable the AI sounded in its responses. “I panicked”, it admits, and says “I thought this meant safe – it actually meant I wiped everything”. The CEO quickly called this behavior “unacceptable” and said it should “never be possible”. Huh. It’s hard to imagine how we’re going to empower AI to edit databases and do real work without having at least the possibility that it’s going to go wrong.
Just yesterday, AWS announced General Availability for a cool new service called Aurora DSQL – from the outside, it looks like a SQL database, but it is fully serverless, meaning that you never have to think about managing database instances, you pay for what you use, and it scales automatically and seamlessly. That’s cool, but what’s even cooler? It’s written 100% in Rust – and how it go to be that way turns out to be a pretty interesting story. If you’d like to read more about that, Marc Bowes and I have a guest post on Werner Vogel’s All Things Distributed blog.
Today is the 10th anniversary of Rust’s 1.0 release. Pretty wild. As part of RustWeek there was a fantastic celebration and I had the honor of giving some remarks, both as a long-time project member but also as representing Amazon as a sponsor. I decided to post those remarks here on the blog.
“It’s really quite amazing to see how far Rust has come. If I can take a moment to put on my sponsor hat, I’ve been at Amazon since 2021 now and I have to say, it’s been really cool to see the impact that Rust is having there up close and personal.
Knock, knock. Who’s there? Dyn. Dyn who? Dyn you have ideas for dyn? I am generally dissatisfied with how dyn Trait in Rust works and, based on conversations I’ve had, I am pretty sure I’m not alone. And yet I’m also not entirely sure the best fix. Building on my last post, I wanted to spend a bit of time exploring my understanding of the problem. I’m curious to see if others agree with the observations here or have others to add.
This article is a slight divergence from my Rust in 2025 series. I wanted to share my latest thinking about how to support dyn Trait for traits with async functions and, in particular how to do so in a way that is compatible with the soul of Rust.
Background: why is this hard?
Supporting async fn in dyn traits is a tricky balancing act. The challenge is reconciling two key things people love about Rust: its ability to express high-level, productive code and its focus on revealing low-level details. When it comes to async function in traits, these two things are in direct tension, as I explained in my first blog post in this series – written almost four years ago! (Geez.)
For many years, C has effectively been the “lingua franca” of the computing world. It’s pretty hard to combine code from two different programming languages in the same process–unless one of them is C. The same could theoretically be true for Rust, but in practice there are a number of obstacles that make that harder than it needs to be. Building out silky smooth language interop should be a core goal of helping Rust to target foundational applications. I think the right way to do this is not by extending rustc with knowledge of other programming languages but rather by building on Rust’s core premise of being an extensible language. By investing in building out an “extensible compiler” we can allow crate authors to create a plethora of ergonomic, efficient bridges between Rust and other languages.
Rust turns 10 this year. It’s a good time to take a look at where we are and where I think we need to be going. This post is the first in a series I’m calling “Rust in 2025”. This first post describes my general vision for how Rust fits into the computing landscape. The remaining posts will outline major focus areas that I think are needed to make this vision come to pass. Oh, and fair warning, I’m expecting some controversy along the way—at least I hope so, since otherwise I’m just repeating things everyone knows.
A few years back I proposed view types as an extension to Rust’s type system to let us address the problem of (false) inter-procedural borrow conflicts. The basic idea is to introduce a “view type” {f1, f2} Type1, meaning “an instance of Type where you can only access the fields f1 or f2”. The main purpose is to let you write function signatures like & {f1, f2} self or &mut {f1, f2} self that define what fields a given type might access. I was thinking about this idea again and I wanted to try and explore it a bit more deeply, to see how it could actually work, and to address the common question of how to have places in types without exposing the names of private fields.
So, a little bird told me that Rust 2024 is going to become stable today, along with Rust 1.85.0. In honor of this momentous event, I have penned a little ditty that I’d like to share with you all. Unfortunately, for those of you who remember Rust 2021’s “Edition: The song”, in the 3 years between Rust 2021 and now, my daughter has realized that her father is deeply uncool1 and so I had to take this one on solo2. Anyway, enjoy! Or, you know, suffer. As the case may be.
I believe that AI-powered development tools can be a game changer for Rust—and vice versa. At its core, my argument is simple: AI’s ability to explain and diagnose problems with rich context can help people get over the initial bump of learning Rust in a way that canned diagnostics never could, no matter how hard we try. At the same time, rich type systems like Rust’s give AIs a lot to work with, which could be used to help them avoid hallucinations and validate their output. This post elaborates on this premise and sketches out some of the places where I think AI could be a powerful boost.
This post lays out the idea of preview crates.1 Preview crates would be special crates released by the rust-lang org. Like the standard library, preview crates would have access to compiler internals but would still be usable from stable Rust. They would be used in cases where we know we want to give users the ability to do X but we don’t yet know precisely how we want to expose it in the language or stdlib. In git terms, preview crates would let us stabilize the plumbing while retaining the ability to iterate on the final shape of the porcelain.
This post floats a variation of boats’ UnpinCell proposal that I’m calling MinPin.1 MinPin’s goal is to integrate Pin into the language in a “minimally disruptive” way2 – and in particular a way that is fully backwards compatible. Unlike Overwrite, MinPin does not attempt to make Pin and &mut “play nicely” together. It does however leave the door open to add Overwrite in the future, and I think helps to clarify the positives and negatives that Overwrite would bring.
In July, boats presented a compelling vision in their post pinned places. With the Overwrite trait that I introduced in my previous post, however, I think we can get somewhere even more compelling, albeit at the cost of a tricky transition. As I will argue in this post, the Overwrite trait effectively becomes a better version of the existing Unpin trait, one that effects not only pinned references but also regular &mut references. Through this it’s able to make Pin fit much more seamlessly with the rest of Rust.
What would you say if I told you that it was possible to (a) eliminate a lot of “inter-method borrow conflicts” without introducing something like view types and (b) make pinning easier even than boats’s pinned places proposal, all without needing pinned fields or even a pinned keyword? You’d probably say “Sounds great… what’s the catch?” The catch it requires us to change Rust’s fundamental assumption that, given x: &mut T, you can always overwrite *x by doing *x = /* new value */, for any type T: Sized. This kind of change is tricky, but not impossible, to do over an edition.
This is the first of what I think will be several follow-up posts to “Claiming, auto and otherwise”. This post is focused on clarifying and tweaking the design I laid out previously in response to some of the feedback I’ve gotten. In future posts I want to lay out some of the alternative designs I’ve heard.
TL;DR: People like it
If there’s any one thing I can take away from what I’ve heard, is that people really like the idea of making working with reference counted or cheaply cloneable data more ergonomic than it is today. A lot of people have expressed a lot of excitement.
This blog post proposes adding a third trait, Claim, that would live alongside Copy and Clone. The goal of this trait is to improve Rust’s existing split, where types are categorized as either Copy (for “plain old data”1 that is safe to memcpy) and Clone (for types that require executing custom code or which have destructors). This split has served Rust fairly well but also has some shortcomings that we’ve seen over time, including maintenance hazards, performance footguns, and (at times quite significant) ergonomic pain and user confusion.
This post lays out a 4-part roadmap for the borrow checker that I call “the borrow checker within”. These changes are meant to help Rust become a better version of itself, enabling patterns of code which feel like they fit within Rust’s spirit, but run afoul of the letter of its law. I feel fairly comfortable with the design for each of these items, though work remains to scope out the details. My belief is that a-mir-formality will make a perfect place to do that work.
I’ve been thinking a wild thought lately: we should deprecate panic=unwind. Most production users I know either already run with panic=abort or use unwinding in a very limited fashion, basically just to run to cleanup, not to truly recover. Removing unwinding from most case meanwhile has a number of benefits, allowing us to extend the type system in interesting and potentially very impactful ways. It also removes a common source of subtle bugs. Note that I am not saying we should remove unwinding entirely: that’s not an option, both because of stability and because of Rust’s mission to “deeply integrate” with all kinds of languages and systems.
Extern types have been blocked for an unreasonably long time on a fairly narrow, specialized question: Rust today divides all types into two categories — sized, whose size can be statically computed, and unsized, whose size can only be computed at runtime. But for external types what we really want is a third category, types whose size can never be known, even at runtime (in C, you can model this by defining structs with an unknown set of fields). The problem is that Rust’s ?Sized notation does not naturally scale to this third case. I think it’s time we fixed this. At some point I read a proposal — I no longer remember where — that seems like the obvious way forward and which I think is a win on several levels. So I thought I would take a bit of time to float the idea again, explain the tradeoffs I see with it, and explain why I think the idea is a good change.
Ownership is an important concept in Rust — but I’m not talking about the type system. I’m talking about in our open source project. One of the big failure modes I’ve seen in the Rust community, especially lately, is the feeling that it’s unclear who is entitled to make decisions. Over the last six months or so, I’ve been developing a project goals proposal, which is an attempt to reinvigorate Rust’s roadmap process — and a key part of this is the idea of giving each goal an owner. I wanted to write a post just exploring this idea of being an owner: what it means and what it doesn’t.
This blog post explores an alternative formulation of Rust’s type system that eschews lifetimes in favor of places. The TL;DR is that instead of having 'a represent a lifetime in the code, it can represent a set of loans, like shared(a.b.c) or mut(x). If this sounds familiar, it should, it’s the basis for polonius, but reformulated as a type system instead of a static analysis. This blog post is just going to give the high-level ideas. In follow-up posts I’ll dig into how we can use this to support interior references and other advanced borrowing patterns. In terms of implementation, I’ve mocked this up a bit, but I intend to start extending a-mir-formality to include this analysis.
Well, it’s that time of year, when thoughts turn to…well, Rust of course. I guess that’s every time of year. This year was a pretty big year for Rust, though I think a lot of what happened was more in the vein of “setting things up for success in 2024”. So let’s talk about 2024! I’m going to publish a series of blog posts about different aspects of Rust I’m excited about, and what I think we should be doing. To help make things concrete, I’m going to frame the 2024 by using proposed project goals – basically a specific piece of work I think we can get done this year. In this first post, I’ll focus on async Rust.
To your average Joe, being “rusty” is not seen as a good thing.1 But readers of this blog know that being Rusty – with a capitol R! – is, of course, something completely different! So what is that makes Rust Rust? Our slogans articulate key parts of it, like fearless concurrency, stability without stagnation, or the epic Hack without fear. And there is of course Lindsey Kuper’s epic haiku: “A systems language / pursuing the trifecta: / fast, concurrent, safe”. But I feel like we’re still missing a unified set of axioms that we can refer back to over time and use to guide us as we make decisions. Some of you will remember the Rustacean Principles, which was my first attempt at this. I’ve been dissatisfied with them for a couple of reasons, so I decided to try again. The structure is really different, so I’m calling it Rust’s design axioms. This post documents the current state – I’m quite a bit happier with it! But it’s not quite there yet. So I’ve also got a link to a repository where I’m hoping people can help improve them by opening issues with examples, counter-examples, or other thoughts.
Lately I’ve been iterating on an idea I call project goals. Project goals are a new kind of RFC that defines a specific goal that a specific group of people hope to achieve in a specific amount of time – for example, “Rusty Spoon Corp proposes to fund 2 engineers full time to stabilize collections that support custom memory allocations by the end of 2023”.
Project goals would also include asks from various teams that are needed to complete the goal. For example, “Achieving this goal requires a dedicated reviewer from the compiler team along with an agreement from the language design team to respond to RFCs or nominated issues within 2 weeks.” The decision of whether to accept a goal would be up to those teams who are being asked to support it. If those teams approve the RFC, it means they agree with the goal, and also that they agree to commit those resources.
My belief is that project goals become a kind of incremental, rolling roadmap, declaring our intent to fix specific problems and then tracking our follow-through (or lack thereof). As I’ll explain in the post, I believe that a mechanism like project goals will help our morale and help us to get shit done, but I also think it’ll help with a bunch of other ancillary problems, such as providing a clearer path to get involved in Rust as well as getting more paid maintainers and contributors.
At the moment, project goals are just an idea. My plan is to author some sample goals to iron out the process and then an RFC to make it official.
A few years back, the Async Wg tried something new. We collaboratively authored an Async Vision Doc. The doc began by writing “status quo” stories, written as narratives from our cast of characters, that described how people were experiencing Async Rust at that time and then went on to plan a “shiny future”. This was a great experience. My impression was that authoring the “status quo” stories in particular was really helpful. Discussions at EuroRust recently got me wondering: can we adapt the “status quo” stories to something bigger? What if we could author a living document on the Rust user experience? One that captures what people are trying to do with Rust, where it is working really well for them, and where it could use improvement. I love this idea, and the more I thought about it, the more I saw opportunities to use it to improve other processes, such as planning, public communication, and RFCs. But I’m getting ahead of myself! Let’s dive in.
I’m on the plane back to the US from Belgium now and feeling grateful for having had the chance to speak at the EuroRust conference1. EuroRust was the first Rust-focused conference that I’ve attended since COVID (though not the first conference overall). It was also the first Rust-focused conference that I’ve attended in Europe since…ever, from what I recall.2 Since many of us were going to be in attendance, the types team also organized an in-person meetup which took place for 3 days before the conference itself3. Both the meetup and the conference were great in many ways, and sparked a lot of ideas. I think I’ll be writing blog posts about them for weeks to come, but I thought that to start, I’d write up something general about the conference itself, and some of my takeaways from the experience
Rust helps you to build reliable programs. One of the ways it does that is by surfacing things to your attention that you really ought to care about. Think of the way we handle errors with Result: if some operation can fail, you can’t, ahem, fail to recognize that, because you have to account for the error case. And yet often the kinds of things you care about depend on the kind of application you are building. A classic example is memory allocation, which for many Rust apps is No Big Deal, but for others is something to be done carefully, and for still others is completely verboten. But this pattern crops up a lot. I’ve heard and like the framing of designing for “what do you have to pay attention to” – Rust currently aims for a balance that errs on the side of paying attention to more things, but tries to make them easy to manage. But this post is about a speculative idea of how we could do better than that by allowing programs to declare a profile.
In the previous Polonius post, we formulated the original borrow checker in a Polonius-like style. In this post, we are going to explore how we can extend that formulation to be flow-sensitive. In so doing, we will enable the original Polonius goals, but also overcome some of its shortcomings. I believe this formulation is also more amenable to efficient implementation. As I’ll cover at the end, though, I do find myself wondering if there’s still more room for improvement.
Over the last few weeks I had been preparing a talk on “Inclusive Mentoring: Mentoring Across Differences” with one of my good friends at Amazon. Unfortunately, that talk got canceled because I came down with COVID when we were supposed to be presenting. But the themes we covered in the talk have been rattling in my brain ever since, and suddenly I’m seeing them everywhere. One of the big ones was about empathy — what it is, what it isn’t, and how you can practice it. Now that I’m thinking about it, I see empathy so often in open source.
lqd has been doing awesome work driving progress on polonius. He’s authoring an update for Inside Rust, but the TL;DR is that, with his latest PR, we’ve reimplemented the traditional Rust borrow checker in a more polonius-like style. We are working to iron out the last few performance hiccups and thinking about replacing the existing borrow checker with this new re-implementation, which is effectively a no-op from a user’s perspective (including from a performance perspective). This blog post walks through that work, describing how the new analysis works at a high-level. I plan to write some follow-up posts diving into how we can extend this analysis to be more precise (while hopefully remaining efficient).
Some time ago I wrote about how I wanted to improve how my blog works. I recently got a spate of emails about this – thanks to all of you! And a particular big thank you to Luna Razzaghipour, who went ahead and ported the blog over to use Hugo, cleaning up the layout a bit and preserving URLs. It’s much appreciated! If you notice something amiss (like a link that doesn’t work anymore), I’d be very grateful if you opened an issue on the babysteps github repo! Thanks!
One of Rust’s core principles is “stability without stagnation”. This is embodied by our use of a “release train” model, in which we issue a new release every 6 weeks. Release trains make releasing a new release a “non-event”. Feature-based releases, in contrast, are super stressful! Since they occur infrequently, people try to cram everything into that release, which inevitably makes the release late. In contrast, with a release train, it’s not so important to make any particular release – if you miss one deadline, you can always catch the next one six weeks later. That’s the theory, anyway: but I’ve observed that, in practice, stabilizing a feature in Rust can still be a pretty stressful process. And the more important the feature, the more stress. This blog post talks over my theories as to why this is the case, and how we can tweak our processes (and our habits) to address it.
I recently posted a draft of an RFC about Return Type Notation to the async working group Zulip stream. In response, Josh Triplett reached out to me to raise some concerns. Talking to him gave rise to a 3rd idea for how to resolve the send bound problem. I still prefer RTN, but I think this idea is interesting and worth elaborating. I call it higher-ranked projections.
Idea part 1: Define T::Foo when T has higher-ranked bounds
In a previous post on async closures, I concluded that the best way to support async closures was with an async trait combinator. I’ve had a few conversations since the post and I want to share some additional thoughts. In particular, this post dives into what it would take to make async functions matchable with a type like impl FnMut() -> impl Future<Output = bool>. This takes us down some interesting roads, in particular the distinction between giving and lending traits; it turns out that the closure traits specifically are a bit of a special case in turns of what we can do backwards compatibly, due to their special syntax. on!
It’s well known that my blog has some issues. The category links don’t work. It renders oddly on mobile. And maybe Safari, too? The Rust snippets are not colored. The RSS feed is apparently not advertised properly in the metadata. It’s published via a makefile instead of some hot-rod CI/CD script, and it uses jekyll instead of whatever the new hotness is.1 Being a programmer, you’d think I could fix this, but I am intimidated by HTML, CSS, and Github Actions. Hence this call for help: I’d like to hire someone to “tune up” the blog, a combination of fixing the underlying setup and also the visual layout. This post will be a rough set of things I have in mind, but I’m open to suggestions. If you think you’d be up for the job, read on.
I’ve been thinking about async closures and how they could work once we have static async fn in trait. Somewhat surprisingly to me, I found that async closures are a strong example for where async transformers could be an important tool. Let’s dive in! We’re going to start with the problem, then show why modeling async closures as “closures that return futures” would require some deep lifetime magic, and finally circle back to how async transformers can make all this “just work” in a surprisingly natural way.
Rust has lots of mechanisms that prevent you from doing something bad. But, right now, it has NO mechanisms that force you to do something good1. I’ve been thinking lately about what it would mean to add “must move” types to the language. This is an idea that I’ve long resisted, because it represents a fundamental increase to complexity. But lately I’m seeing more and more problems that it would help to address, so I wanted to try and think what it might look like, so we can better decide if it’s a good idea.
In today’s lang team design meeting, we reviewed a doc I wrote about temporary lifetimes in Rust. The current rules were established in a blog post I wrote in 2014. Almost a decade later, we’ve seen that they have some rough edges, and in particular can be a common source of bugs for people. The Rust 2024 Edition gives us a chance to address some of those rough edges. This blog post is a copy of the document that the lang team reviewed. It’s not a proposal, but it covers some of what works well and what doesn’t, and includes a few sketchy ideas towards what we could do better.
One interesting question about async fn in traits is whether or not we should label the trait itself as async. Until recently, I didn’t see any need for that. But as we discussed the question of how to enable “maybe async” code, we realized that there would be some advantages to distinguishing “async traits” (which could contain async functions) from sync traits (which could not). However, as I’ve thought about the idea more, I’m more and more of the mind that we should not take this step — at least not now. I wanted to write a blog post divin g into the considerations as I see them now.
I previously introduced the “send bound” problem, which refers to the need to add a Send bound to the future returned by an async function. This post continues my tour over the various solutions that are available. This post covers “Trait Transformers”. This proposal arose from a joint conversation with myself, Eric Holk, Yoshua Wuyts, Oli Scherer, and Tyler Mandry. It’s a variant of Eric Holk’s inferred async send bounds proposal as well as the work that Yosh/Oli have been doing in the keyword generics group. Those posts are worth reading as well, lots of good ideas there.1
In the previous post, I introduced the “send bound” problem, which refers to the need to add a Send bound to the future returned by an async function. I want to start talking about some of the ideas that have been floating around for how to solve this problem. I consider this a bit of an open problem, in that I think we know a lot of the ingredients, but there is a bit of a “delicate balance” to finding the right syntax and so forth. To start with, though, I want to introduce Return Type Notation, which is an idea that Tyler Mandry and I came up with for referring to the type returned by a trait method.
Nightly Rust now has support for async functions in traits, so long as you limit yourself to static dispatch. That’s super exciting! And yet, for many users, this support won’t yet meet their needs. One of the problems we need to resolve is how users can conveniently specify when they need an async function to return a Send future. This post covers some of the background on send futures, why we don’t want to adopt the solution from the async_trait crate for the language, and the general direction we would like to go. Follow-up posts will dive into specific solutions.
When I started working on Rust in 2011, my daughter was about three months old. She’s now in sixth grade, and she’s started growing rapidly. Sometimes we wake up to find that her clothes don’t quite fit anymore: the sleeves might be a little too short, or the legs come up to her ankles. Rust is experiencing something similar. We’ve been growing tremendously fast over the last few years, and any time you experience growth like that, there are bound to be a few rough patches. Things that don’t work as well as they used to. This holds both in a technical sense — there are parts of the language that don’t seem to scale up to Rust’s current size — and in a social one — some aspects of how the projects runs need to change if we’re going to keep growing the way I think we should. As we head into 2023, with two years to go until the Rust 2024 edition, this is the theme I see for Rust: maturation and scaling.
I’ve been thinking about what “Rust 2024” will look like lately. I don’t really mean the edition itself — but more like, what will Rust feel like after we’ve finished up the next few years of work? I think the answer is that Rust 2024 is going to be the year of “everywhere”. Let me explain what I mean. Up until now, Rust has had a lot of nice features, but they only work sometimes. By the time 2024 rolls around, they’re going to work everywhere that you want to use them, and I think that’s going to make a big difference in how Rust feels.
After my last post on dyn async traits, some folks pointed out that I was overlooking a seemingly obvious possibility. Why not have the choice of how to manage the future be made at the call site? It’s true, I had largely dismissed that alternative, but it’s worth consideration. This post is going to explore what it would take to get call-site-based dispatch working, and what the ergonomics might look like. I think it’s actually fairly appealing, though it has some limitations.
Re-reading my previous post, I felt I should clarify why I called it the “soul of Rust”. The soul of Rust, to my mind, is definitely not being explicit about allocation. Rather, it’s about the struggle between a few key values — especially productivity and versatility1 in tension with transparency. Rust’s goal has always been to feel like a high-level but with the performance and control of a low-level one. Oftentimes, we are able to find a “third way” that removes the tradeoff, solving both goals pretty well. But finding those “third ways” takes time — and sometimes we just have to accept a certain hit to one value or another for the time being to make progress. It’s exactly at these times, when we have to make a difficult call, that questions about the “soul of Rust” starts to come into play. I’ve been thinking about this a lot, so I thought I would write a post that expands on the role of transparency in Rust, and some of the tensions that arise around it.
In the last few months, Tyler Mandry and I have been circulating a “User’s Guide from the Future” that describes our current proposed design for async functions in traits. In this blog post, I want to deep dive on one aspect of that proposal: how to handle dynamic dispatch. My goal here is to explore the space a bit and also to address one particularly tricky topic: how explicit do we have to be about the possibility of allocation? This is a tricky topic, and one that gets at that core question: what is the soul of Rust?
Have you heard of the Salsa project? Salsa is a library for incremental computation – it’s used by rust-analyzer, for example, to stay responsive as you type into your IDE (we have also discussed using it in rustc, though more work is needed there). We are in the midst of a big push right now to develop and release Salsa 2022, a major new revision to the API that will make Salsa far more natural to use. I’m writing this blog post both to advertise that ongoing work and to put out a call for contribution. Salsa doesn’t yet have a large group of maintainers, and I would like to fix that. If you’ve been looking for an open source project to try and get involved in, maybe take a look at our Salsa 2022 tracking issue and see if there is an issue you’d like to tackle?
As some of you may know, on May 4th Jack Huey opened a PR to stabilize an initial version of generic associated types. The current version is at best an MVP: the compiler support is limited, resulting in unnecessary errors, and the syntax is limited, making code that uses GATs much more verbose than I’d like. Nonetheless, I’m super excited, since GATs unlock a lot of interesting use cases, and we can continue to smooth out the rough edges over time. However, folks on the thread have raised some strong concerns about GAT stabilization, including asking whether GATs are worth including in the language at all. The fear is that they make Rust the language too complex, and that it would be better to just use them as an internal building block for other, more accessible features (like async functions and [return position impl trait in traits][RPITIT]). In response to this concern, a number of people have posted about how they are using GATs. I recently took some time to deep dive into these comments and to write about some of the patterns that I found there, including a pattern I am calling the “many modes” pattern, which comes from the chumsky parser combinator library. I posted about this pattern on the thread, but I thought I would cross-post my write-up here to the blog as well, because I think it’s of general interest.
You’ve probably heard that the Rust type checker can be a great “co-pilot”, helping you to avoid subtle bugs that would have been a royal pain in the !@#!$! to debug. This is truly awesome! But what you may not realize is how it feels in the moment when this happens. The answer typically is: really, really frustrating! Usually, you are trying to get some code to compile and you find you just can’t do it.
Lately I’ve been diving deep into tokio’s mini-redis example. The mini-redis example is a great one to look at because it’s a realistic piece of quality async Rust code that is both self-contained and very well documented. Digging into mini-redis, I found that it exemplifies the best and worst of async Rust. On the one hand, the code itself is clean, efficient, and high-level. On the other hand, it relies on a number of subtle async conventions that can easily be done wrong – worse, if you do them wrong, you won’t get a compilation error, and your code will “mostly work”, breaking only in unpredictable timing conditions that are unlikely to occur in unit tests. Just the kind of thing Rust tries to avoid! This isn’t the fault of mini-redis – to my knowledge, there aren’t great alterantive patterns available in async Rust today (I go through some of the alternatives in this post, and their downsides).
Rust has been wrestling with coherence more-or-less since we added methods; our current rule, the “orphan rule”, is safe but overly strict. Roughly speaking, the rule says that one can only implement foreign traits (that is, traits defined by one of your dependencies) for local types (that is, types that you define). The goal of this rule was to help foster the crates.io ecosystem — we wanted to ensure that you could grab any two crates and use them together, without worrying that they might define incompatible impls that can’t be combined. The rule has served us well in that respect, but over time we’ve seen that it can also have a kind of chilling effect, unintentionally working against successful composition of crates in the ecosystem. For this reason, I’ve come to believe that we will have to weaken the orphan rule. The purpose of this post is to write out some preliminary exploration of ways that we might do that.
There are two ergonomic features that have been discussed for quite some time in Rust land: perfect derive and expanded implied bounds. Until recently, we were a bit stuck on the best way to implement them. Recently though I’ve been working on a new formulation of the Rust trait checker that gives us a bunch of new capabilities — among them, it resolved a soundness formulation that would have prevented these two features from being combined. I’m not going to describe my fix in detail in this post, though; instead, I want to ask a different question. Now that we can implement these features, should we?
Last Friday, tmandry, cramertj, and I had an exciting conversation. We were talking about the design for combining async functions in traits with dyn Trait that tmandry and I had presented to the lang team on Friday. cramertj had an insightful twist to offer on that design, and I want to talk about it here. Keep in mind that this is a piece of “hot off the presses”, in-progress design and hence may easily go nowhere – but at the same time, I’m pretty excited about it. If it works out, it could go a long way towards making dyn Trait user-friendly and accessible in Rust, which I think would be a big deal.
Last year, we shipped Rust 2021 and I have found the changes to be a real improvement in usability. Even though the actual changes themselves were quite modest, the combination of precise capture closure and simpler formatting strings (println!("{x:?}") instead of println!("{:?}", x)) is making a real difference in my “day to day” life.1 Just like NLL and the new module system from Rust 2018, I’ve quickly adapted to these new conventions. When I go back to older code, with its clunky borrow checker workarounds and format strings, I die a little inside.2
One of the things people often complain about when doing Async Rust is cancellation. This has always been a bit confusing to me, because it seems to me that async cancellation should feel a lot like panics in practice, and people don’t complain about panics very often (though they do sometimes). This post is the start of a short series comparing panics and cancellation, seeking after the answer to the question “Why is async cancellation a pain point and what should we do about it?” This post focuses on explaining Rust’s panic philosophy and explaining why I see panics and cancellation as being quite analogous to one another.
Hi all! Welcome to 2022! Towards the end of last year, Tyler Mandry and I were doing a lot of iteration around supporting “dyn async trait” – i.e., making traits that use async fn dyn safe – and we’re starting to feel pretty good about our design. This is the start of several blog posts talking about where we’re at. In this first post, I’m going to reiterate our goals and give a high-level outline of the design. The next few posts will dive more into the details and the next steps.
Wow! The response to the last Rustc Reading Club was overwhelming – literally! We maxed out the number of potential zoom attendees and I couldn’t even join the call! It’s clear that there’s a lot of demand here, which is great. We’ve decided to take another stab at running the Rustc Reading Club, but we’re going to try it a bit differently this time. We’re going to start by selecting a smaller group to do it a few times and see how it goes, and then decide how to scale up.
The next “Cross Team Collaboration Fun Times” (CTCFT) meeting will take place next Monday, on 2021-11-22 at 11am US Eastern Time (click to see in your time zone). Note that this is a new time: we are experimenting with rotating in an earlier time that occurs during the European workday. This post covers the agenda. You’ll find the full details (along with a calendar event, zoom details, etc) on the CTCFT website.
I wanted to write about an idea that’s been kicking around in the back of my mind for some time. I call it view types. The basic idea is to give a way for an &mut or & reference to identify which fields it is actually going to access. The main use case for this is having “disjoint” methods that don’t interfere with one another.
This is not a proposal (yet?)
To be clear, this isn’t an RFC or a proposal, at least not yet. It’s some early stage ideas that I wanted to document. I’d love to hear reactions and thoughts, as I discuss in the conclusion.
Ever wanted to understand how rustc works? Me too! Doc Jones and I have been talking and we had an idea we wanted to try. Inspired by the very cool Code Reading Club, we are launching an experimental Rustc Reading Club. Doc Jones posted an announcement on her blog, so go take a look!
The way this club works is pretty simple: every other week, we’ll get together for 90 minutes and read some part of rustc (or some project related to rustc), and talk about it. Our goal is to walk away with a high-level understanding of how that code works. For more complex parts of the code, we may wind up spending multiple sessions on the same code.
A quick update to my last post: first, a better way to do what I was trying to do, and second, a sketch of the crate I’d like to see for experimental purposes.
An easier way to roll our own boxed dyn traits
In the previous post I covered how you could create vtables and pair the up with a data pointer to kind of “roll your own dyn”. After I published the post, though, dtolnay sent me this Rust playground link to show me a much better approach, one based on the erased-serde crate. The idea is that instead of make a “vtable struct” with a bunch of fn pointers, we create a “shadow trait” that reflects the contents of that vtable:
If you’re willing to use nightly, you can already model async functions in traits by using GATs and impl Trait — this is what the Embassy async runtime does, and it’s also what the real-async-trait crate does. One shortcoming, though, is that your trait doesn’t support dynamic dispatch. In the previous posts of this series, I have been exploring some of the reasons for that limitation, and what kind of primitive capabilities need to be exposed in the language to overcome it. My thought was that we could try to stabilize those primitive capabilities with the plan of enabling experimentation. I am still in favor of this plan, but I realized something yesterday: using procedural macros, you can ALMOST do this experimentation today! Unfortunately, it doesn’t quite work owing to some relatively obscure rules in the Rust type system (perhaps some clever readers will find a workaround; that said, these are rules I have wanted to change for a while).
(20 min) Sprints and groups implementing the async vision doc (tmandry)
(15 min) rust-analyzer talk (TBD)
The rust-analyzer project aims to succeed RLS as the official language server for Rust. We talk about how it differs from RLS, how it is developed, and what to expect in the future.
After the CTCFT this week, we are going to try an experimental social hour. The hour will be coordinated in the #ctcft stream of the rust-lang Zulip. The idea is to create breakout rooms where people can gather to talk, hack together, or just chill.
In the previous post, I talked about how we could write our own impl Iterator for dyn Iterator by adding a few primitives. In this post, I want to look at what it would take to extend that to an async iterator trait. As before, I am interested in exploring the “core capabilities” that would be needed to make everything work.
Start somewhere: Just assume we want Box
In the first post of this series, we talked about how invoking an async fn through a dyn trait should to have the return type of that async fn be a Box<dyn Future> — but only when calling it through a dyn type, not all the time.
In the previous “dyn async traits” posts, I talked about how we can think about the compiler as synthesizing an impl that performed the dynamic dispatch. In this post, I wanted to start explore a theoretical future in which this impl was written manually by the Rust programmer. This is in part a thought exercise, but it’s also a possible ingredient for a future design: if we could give programmers more control over the “impl Trait for dyn Trait” impl, then we could enable a lot of use cases.
In the previous post, we uncovered a key challenge for dyn and async traits: the fact that, in Rust today, dyn types have to specify the values for all associated types. This post is going to dive into more background about how dyn traits work today, and in particular it will talk about where that limitation comes from.
Today: Dyn traits implement the trait
In Rust today, assuming you have a “dyn-safe” trait DoTheThing , then the type dyn DoTheThing implements Trait. Consider this trait:
Over the last few weeks, Tyler Mandry and I have been digging hard into what it will take to implement async fn in traits. Per the new lang team initiative process, we are collecting our design thoughts in an ever-evolving website, the async fundamentals initiative. If you’re interested in the area, you should definitely poke around; you may be interested to read about the MVP that we hope to stabilize first, or the (very much WIP) evaluation doc which covers some of the challenges we are still working out. I am going to be writing a series of blog posts focusing on one particular thing that we have been talking through: the problem of dyn and async fn. This first post introduces the problem and the general goal that we are shooting for (but don’t yet know the best way to reach).
RustConf is always a good time for reflecting on the project. For me, the last week has been particularly “reflective”. Since announcing the Rustacean Principles, I’ve been having a number of conversations with members of the community about how they can be improved. I wanted to write a post summarizing some of the feedback I’ve gotten.
The principles are a work-in-progress
Sparking conversation about the principles was exactly what I was hoping for when I posted the previous blog post. The principles have mostly been the product of Josh and I iterating, and hence reflect our experiences. While the two of us have been involved in quite a few parts of the project, for the document to truly serve its purpose, it needs input from the community as a whole.
We’re going to try something a bit different this time! The agenda is going to focus on Rust interest groups and domain working groups, those brave explorers who are trying to put Rust to use on all kinds of interesting domains. Rather than having fixed presentations, we’re going to have a panel discussion with representatives from a number of Rust interest groups and domain groups, led by AngelOnFira. The idea is to open a channel for communication about how to have more active communication and feedback between interest groups and the Rust teams (in both directions).
As the web site says, Rust is a language empowering everyone to build reliable and efficient software. I think it’s precisely this feeling of empowerment that people love about Rust. As wycats put it recently to me, Rust makes it “feel like things are possible that otherwise feel out of reach”. But what exactly makes Rust feel that way? If we can describe it, then we can use that description to help us improve Rust, and to guide us as we design extensions to Rust.
A detailed agenda will be announced in a few weeks. Current thinking however is to center the agenda on Rust interest groups and domain working groups, those brave explorers who are trying to put Rust to use on all kinds of interesting domains, such as game development, cryptography, machine learning, formal verification, and embedded development. If you run an interest group and I didn’t list your group here, perhaps you want to get in touch! We’ll be talking about how these groups operate and how we can do a better job of connecting interest groups with the Rust org.
The Rust project has a number of mechanisms for getting people involved in the project, but most are oriented around 1:1 engagement. Doc has been investigating some of the ways that other projects engage contributors, such as Python’s mentored sprints. She will discuss how some of those projects run things and share some ideas about how that might be applied in the Rust project.
Hey everyone! At the CTCFT meeting this Monday (2021-06-21), we’re going to try a “social hour”. The idea is really simple: for the hour after the meeting, we will create breakout rooms in Zoom with different themes. You can join any breakout room you like and hangout.
After the CTCFT this week, we are going to try an experimental social hour. The hour will be coordinated in the #ctcft stream of the rust-lang Zulip. The idea is to create breakout rooms where people can gather to talk, hack together, or just chill.
You may have heard that the Rust 2021 Edition is coming. Along with my daughter Daphne, I have recorded a little song in honor of the occasion! The full lyrics are below – if you feel inspired, please make your own version!1 Enjoy!
(Spoken) Breaking changes where no code breaks. Sounds impossible, no? But in the Rust language, you might say that we like to do impossible things. It isn’t easy. You may ask, how do we manage such a thing? That I can tell you in one word… Edition!
The meeting is a mechanism to help keep the members of the Rust teams in sync and in touch with one another. The idea is to focus on topics of broad interest (more than two teams):
Status updates on far-reaching projects that could affect multiple teams;
Experience reports about people trying new things (sometimes succeeding, sometimes not);
“Rough draft” proposals that are ready to be brought before a wider audience.
The meeting will focus on things that could either offer insights that might affect the work you’re doing, or where the presenter would like to pose questions to the Rust teams and get feedback.
The Async Vision Doc effort has been going now for about 6 weeks. It’s been a fun ride, and I’ve learned a lot. It seems like a good time to take a step back and start talking a bit about the vision doc structure and the process. In this post, I’m going to focus on the role that I see vision docs playing in Rust’s planning and decision making, particularly as compared to RFCs.
If you’re available and those stories sound like something that interests you, please join him! Just ping me or Ryan on Discord or Zulip and we’ll send you the Zoom link. If you’ve already joined a previous session, the link is the same as before.
Ryan Levick and I are going to be hosting more Async Vision Doc Writing Sessions this week. We’re not organized enough to have assigned topics yet, so I’m just going to post the dates/times and we’ll be tweeting about the particular topics as we go.
If you’ve joined before, we’ll be re-using the same Zoom link. If you haven’t joined, then send a private message to one of us and we’ll share the link. Hope to see you there!
This is an exciting week for the vision doc! As of this week, we are starting to
draft “shiny future” stories, and we would like your help! (We are also still
working on status quo stories, so there is no need to stop working on those.)
There will be a blog post coming out on the main Rust blog soon with all the
details, but you can go to the “How to vision: Shiny future” page now.
If you’re available and those stories sound like something that interests you, please join him! Just ping me or Ryan on Discord or Zulip and we’ll send you the Zoom link. If you’ve already joined a previous session, the link is the same as before.
I’ve been working on the Rust project for just about ten years. The language has evolved radically in that time, and so has the project governance. When I first started, for example, we communicated primarily over the rust-dev mailing list and the #rust IRC channel. I distinctly remember coming into the Mozilla offices1 one day and brson excitedly telling me, “There were almost a dozen people on the #rust IRC channel last night! Just chatting! About Rust!” It’s funny to think about that now, given the scale Rust is operating at today.
Ryan Levick and I are hosting a number of public drafting sessions scheduled this week.
Some of them are scheduled early to cover a wider range of time zones.
If you’re available and those stories sound like something that interests you, please join us!
We’re particlarly interested in having people join who have had related experiences, as the goal here is to capture the details from people who’ve been there.
During these sessions, we’ll be looking over the status quo issues
and writing a story or two! If you’d like to join, ping me on Discord
or Zulip and I’ll send you the Zoom link.
I’ve got an announcement to make. As of Jan 4th, I’m starting at Amazon as the tech lead of their new Rust team. Working at Mozilla has been a great experience, but I’m pretty excited about this change. It’s a chance to help shape what I hope to be an exciting new phase for Rust, where we grow from a project with a single primary sponsor (Mozilla) to an industry standard, supported by a wide array of companies. It’s also a chance to work with some pretty awesome people – both familiar faces from the Rust community1 and some new folks. Finally, I’m hoping it will be an opportunity for me to refocus my attention to some long-standing projects that I really want to see through.
I wanted to write a post that looks back over 2020 from a personal perspective. My goal here is to look at the various initiatives that I’ve been involved in and try to get a sense for how they went, what worked and what didn’t, and also what that means for next year. This post is a backdrop for a #niko2021 post that I plan to post sometime before 2021 actually starts, talking about what I expect to be doing in 2021.
Since we created the Rust teams, I have been serving as lead of two teams: the compiler team and the language design team (I’ve also been a member of the core team, which has no lead). For those less familiar with Rust’s governance, the compiler team is focused on the maintenance and implementation of the compiler itself (and, more recently, the standard library). The language design team is focused on the design aspects. Over that time, all the Rust teams have grown and evolved, with the compiler team in particular being home to a number of really strong members.
The point of the async interview series, in the end, was to help
figure out what we should be doing next when it comes to Async I/O. I
thought it would be good then to step back and, rather than
interviewing someone else, give my opinion on some of the immediate
next steps, and a bit about the medium to longer term. I’m also going
to talk a bit about what I see as some of the practical challenges.
I’ve noticed that the ideas that I post on my blog are getting much
more “well rounded”. That is a problem. It means I’m waiting too long
to write about things. So I want to post about something that’s a bit
more half-baked – it’s an idea that I’ve been kicking around to
create a kind of informal “analysis API” for rustc.
The problem statement
I am interested in finding better ways to support advanced analyses
that “layer on” to rustc. I am thinking of projects like Prusti or
Facebook’s MIRAI, or even the venerable Clippy. All of these
projects are attempts to layer on additional analyses atop Rust’s
existing type system that prove useful properties about your code.
Prusti, for example, lets you add pre- and post-conditions to your
functions, and it will prove that they hold.
Hello everyone! I’m happy to be posting a transcript of my async
interview with withoutboats. This particularly interview took place
way back on January 14th, but the intervening months have been a bit
crazy and I didn’t get around to writing it up till now.
Video
You can watch the video on YouTube. I’ve also embedded a copy here
for your convenience:
Next steps for async
Before I go into boats’ interview, I want to talk a bit about the
state of async-await in Rust and what I see as the obvious next steps.
I may still do a few more async interviews after this – there are
tons of interesting folks I never got to speak to! – but I think it’s
also past time to try and come to a consensus of the “async roadmap”
for the rest of the year (and maybe some of 2021, too). The good news
is that I feel like the async interviews highlighted a number of
relatively clear next steps. Sometime after this post, I hope to post
a blog post laying out a “rough draft” of what such a roadmap might
look like.
Hello! For the latest async interview, I spoke with Eliza Weisman
(hawkw, mycoliza on twitter). Eliza first came to my attention as the author of the
tracing crate, which is a nifty crate for doing application level
tracing. However, she is also a core maintainer of tokio, and she
works at Buoyant on the linkerd system. linkerd is one of a small
set of large applications that were build using 0.1 futures – i.e.,
before async-await. This range of experience gives Eliza an interesting
“overview” perspective on async-await and Rust more generally.
Hello! For the latest async interview, I spoke with Steven Fackler
(sfackler). sfackler has been involved in Rust for a long time and
is a member of the Rust libs team. He is also the author of a lot of
crates, most notably tokio-postgres.
I particularly wanted to talk to sfackler about the AsyncRead and
AsyncWrite traits. These traits are on everybody’s list of
“important things to stabilize”, particularly if we want to create
more interop between different executors and runtimes. On the other
hand, in [tokio-rs/tokio#1744], the tokio project is considering
adopting its own variant traits that diverge significantly from those
in the futures crate, precisely because they have concerns over the
design of the traits as is. This seems like an important area to dig
into!
Hello! For the latest async interview, I spoke with Florian Gilcher
(skade). Florian is involved in the async-std project, but he’s
also one of the founders of Ferrous Systems, a Rust consulting firm
that also does a lot of trainings. In that capacity, he’s been
teaching people to use async Rust now since Rust’s 1.0 release.
Video
You can watch the video on YouTube. I’ve also embedded a copy here
for your convenience:
One note: something about our setup meant that I was hearing a lot of
echo. I think you can sometimes hear it in the recording, but not
nearly as bad as it was live. So if I seem a bit spacey, or take very
long pauses, you might know the reason why!
In my #rust2020 blog post, I mentioned rather off-handedly
that I think the time has come for us to talk about forming a Rust
foundation. I wanted to come back to this topic and talk in more
detail about what I think a Rust foundation might look like. And,
since I don’t claim to have the final answer to that question by any
means, I’d also like to talk about how I think we should have this
conversation going forward.
Hello! For the latest async interview, I spoke with Carl Lerche
(carllerche). Among many other crates1, Carl is perhaps best
known as one of the key authors behind tokio and mio. These two
crates are quite widely used through the async ecosystem. Carl and I
spoke on December 3rd.
Video
You can watch the video on YouTube. I’ve also embedded a copy here
for your convenience:
Background: the mio crate
One of the first things we talked about was a kind of overview of the
layers of the “tokio-based async stack”.
In the first post, I covered what we said about Fuchsia,
interoperability, and the organization of the futures crate.
In the second post, I covered cramertj’s take on the Stream,
AsyncRead, and AsyncWrite traits. We also discused the idea of
attached streams and the imporance of GATs for modeling those.
In the first post, I covered what we said about Fuchsia,
interoperability, and the organization of the futures crate. This
post covers cramertj’s take on the Stream trait as well as the
AsyncRead and AsyncWrite traits.
Next, cramertj and I turned to discussing some of the specific traits
from the futures crate. One of the traits that we covered was
Stream. The Stream trait is basically the asynchronous version
of the Iterator trait. In (slightly) simplified form, it is as
follows:
For the second async interview, I spoke with Taylor Cramer – or
cramertj, as I’ll refer to him. cramertj is a member of the compiler
and lang teams and was – until recently – working on Fuchsia at
Google. He’s been a key player in Rust’s Async I/O design and in the
discussions around it. He was also responsible for a lot of the
implementation work to make async fn a reality.
I want to write about an idea that Josh Triplett and I have been
iterating on to revamp the lang team RFC process. I have written a
draft of an RFC already, but this blog post aims to introduce the
idea and some of the motivations. The key idea of the RFC is formalize
the steps leading up to an RFC, as well as to capture the lang team
operations around project groups. The hope is that, if this
process works well, it can apply to teams beyond the lang team as
well.
Technically speaking, it’s past the deadline for #rust2020 posts, but
I’m running late this year, and I’m going to post something anyway.
In this post, I am focusing on what I see as the “largest scale”
issues, and not on technical initiatives. If I have time, I will try
to post a follow-up talking about some of the key technical
initiatives that I think we should focus on as well.
Hello from Iceland! (I’m on vacation.) I’ve just uploaded [the first
of the Async Interviews][video] to YouTube. It is a conversation with Alex
Crichton (alexcrichton) and Nick Fitzgerald (fitzgen) about how
WebAssembly and Rust’s Async I/O system interact. When you watch it,
you will probably notice two things:
First, I spent a lot of time looking off to the side! This is
because I had the joint Dropbox paper document open on my side
monitor and I forgot how strange that would look. I’ll have to
remember that for the future. =)
Second, we recorded this on October 3rd1, which was before
async-await had landed on stable. So at various points we talk about
async-await being on beta or not yet being stable. Don’t be
confused. =)
Video
You can view the [video][video] on YouTube, but it is also embedded
here if that’s easier for you.
Hello all! I’m going to be trying something new, which I call the
“Async Interviews”. These interviews are going to be a series of
recorded video calls with various “luminaries” from Rust’s Async I/O
effort. In each one, I’m going to be asking roughly the same question:
Now that the async-await MVP is stable, what should we be doing
next? After each call, I’ll post the recording from the interview,
along with a blog post that leaves a brief summary.
After reading boat’s excellent post on asynchronous destructors,
I thought it might be a good idea to write some about async fn in
traits. Support for async fn in traits is probably the single most
common feature request that I hear about. It’s also one of the more
complex topics. So I thought it’d be nice to do a blog post kind of
giving the “lay of the land” on that feature – what makes it
complicated? What questions remain open?
I would like to describe an idea that’s been kicking around in my
head. I’m calling this idea “shepherds 3.0” – the 3.0 is to
distinguish it from the other places we’ve used the term in the past.
This proposal actually supplants both of the previous uses of the
term, replacing them with what I believe to be a preferred alternative
(more on that later).
Caveat
This is an idea that has been kicking around in my head for a while.
It is not a polished plan and certainly not an accepted one. I’ve not
talked it over with the rest of the lang team, for example. However, I
wanted to put it out there for discussion, and I do think we should be
taking some step in this direction soon-ish.
I have been thinking about how language feature development works in
Rust1. I wanted to write a post about what I see as one of the
key problems: too much concurrency in our design process, without any
kind of “back-pressure” to help keep the number of “open efforts”
under control. This setup does enable us to get a lot of things done sometimes,
but I believe it also leads to a number of problems.
I wanted to give an update on the status of the “async-await
foundations” working group. This post aims to cover three things:
the “async await MVP” that we are currently targeting;
how that fits into the bigger picture;
and how you can help, if you’re so inclined;
Current target: async-await MVP
We are currently working on stabilizing what we call the async-await
MVP – as in, “minimal viable product”. As the name suggests, the
work we’re doing now is basically the minimum that is needed to
“unlock” async-await. After this work is done, it will be easier to
build async I/O based applications in Rust, though a number of rough
edges remain.
On internals, I just announced the formation of the
language-design team meta working group. The role of the meta working
group is to figure out how other language-design team working groups
should work. The plan is to begin by enumerating some of our goals –
the problems we aim to solve, the good things we aim to keep – and
then move on to draw up more details plans. I expect this discussion
will intersect the RFC process quite heavily (at least when it comes
to language design changes). Should be interesting! It’s all happening
in the open, and a major goal of mine is for this to be easy to follow
along with from the outside – so if talking about talking is your
thing, you should check it out.
In my previous post, I talked about the idea of mapping the
solution space:
When we talk about the RFC process, we always emphasize that the point
of RFC discussion is not to select the best answer; rather, the
point is to map the solution space. That is, to explore what the
possible tradeoffs are and to really look for alternatives. This
mapping process also means exploring the ups and downs of the current
solutions on the table.
In the talk I gave at Rust LATAM, I said that the Rust project has
always emphasized finding the best solution, rather than winning the
argument. I think this is one of our deepest values. It’s also one
of the hardest for us to uphold.
Let’s face it – when you’re having a conversation, it’s easy to get
attached to specific proposals. It’s easy to have those proposals
change from “Option A” vs “Option B” to “my option” and “their
option”. Once this happens, it can be very hard to let them “win”
– even if you know that both options are quite reasonable.
Lately, the compiler team has been changing up the way that we work.
Our goal is to make it easier for people to track what we are doing
and – hopefully – get involved. This is an ongoing effort, but one
thing that has become clear immediately is this: the compiler team
needs more than coders.
Traditionally, when we’ve thought about how to “get involved” in the
compiler team, we’ve thought about it in terms of writing PRs. But
more and more I’m thinking about all the other jobs that go into
maintaining the compiler. “What kinds of jobs are these?”, you’re
asking. I think there are quite a few, but let me give a few
examples:
I wanted to post a quick update on the status of the async-await
effort. The short version is that we’re in the home stretch for
some kind of stabilization, but there remain some significant
questions to overcome.
Announcing the implementation working group
As part of this push, I’m happy to announce we’ve formed a
async-await implementation working group. This working group
is part of the whole async-await effort, but focused on the
implementation, and is part of the compiler team. If you’d like to
help get async-await over the finish line, we’ve got a list of issues
where we’d definitely like help (read on).
Now that the Rust 2018 edition has shipped, the language design team
has been thinking a lot about what to do in 2019 and over the next
few years. I think we’ve got a lot of exciting stuff on the horizon,
and I wanted to write about it.
Theme for this edition
In 2015, our overall theme was stability. For the 2018 Edition, we adopted
productivity. For Rust 20211, we are thinking of maturity as the theme.
Our goal is finish up a number of in-flight features – such as specialization,
generic associated types, and const generics – that have emerged as key enablers
for future work. In tandem, we aim to start improving our reference material,
both through continuing the great work that’s been done on the Rust reference
but also through more specialized efforts like the Grammar and Unsafe Code Guidelines
working groups.
So for the last couple of months or so, I’ve been hacking in my spare
time on this library named
salsa, along with a number
of awesome other
folks. Salsa
basically extracts the incremental recompilation techniques that we
built for rustc into a general-purpose framework that can be used by
other programs. Salsa is developing quickly: with the publishing of
v0.10.0, we saw a big step up in the overall ergonomics, and I think
the current interface is starting to feel very nice.
In my previous post about Polonius and subregion obligations, I
mentioned that there needs to be a follow-up to deal with
higher-ranked subregions. This post digs a bit more into what the
problem is in the first place and sketches out the general solution
I have in mind, but doesn’t give any concrete algorithms for it.
The subset relation in Polonius is not enough
In my original post on Polonius, I assumed that when we computed a
subtype relation T1 <: T2 between two types, the result was either a
hard error or a set of subset relations between various regions.
So, for example, if we had a subtype relation between two references:
Now that NLL has been shipped, I’ve been doing some work revisiting
the Polonius project. Polonius is the project that implements
the “alias-based formulation” described in my older
blogpost. Polonius has come a long way since that post; it’s now
quite fast and also experimentally integrated into rustc, where it
passes the full test suite.
However, polonius as described is not complete. It describes the core
“borrow check” analysis, but there are a number of other checks that
the current implementation checks which polonius ignores:
To me, 2018 felt like a big turning point for Rust, and it wasn’t just
the edition. Suddenly, it has become “normal” for me to meet people
using Rust at their jobs. Rust conferences are growing and starting to
have large number of sponsors. Heck, I even met some professional Rust
developers amongst the parents at a kid’s birthday party
recently. Something has shifted, and I like it.
At the same time, I’ve also noticed a lot of exhaustion. I know I feel
it – and a lot of people I talk to seem to feel the same way. It’s
great that so much is going on in the Rust world, but we need to get
better at scaling our processes up and processing it effectively.
Continuing on with my “After NLL” series, I want to look at another
common error that I see and its solution: today’s choice is about moves
from borrowed data and the Sentinel Pattern that can be used to enable
them.
The problem
Sometimes when we have &mut access to a struct, we have a need to
temporarily take ownership of some of its fields. Usually what happens
is that we want to move out from a field, construct something new using
the old value, and then replace it. So for example imagine we have a
type Chain, which implements a simple linked list:
This week I’ve been attending SPLASH 2018. It’s already been quite an
interesting week, and it’s only just begun. I thought I’d write up a
quick report on some of the things that have been particularly
interesting to me, and some of the ideas that they’ve sparked off.
Teaching programming (and Rust!)
I really enjoyed this talk by Felienne Hermans entitled “Explicit
Direct Instruction in Programming Education”. The basic gist of
the talk was that, when we teach programming, we often phrase it in
terms of “exploration” and “self-expression”, but that this winds up
leaving a lot of folks in the cold and may be at least partly
responsible for the lack of diversity in computer science today. She
argued that this is like telling kids that they should just be able to
play a guitar and create awesome songs without first practicing their
chords1 – it kind of sets them up to fail.
In my previous post on the status of NLL, I promised to talk about
“What is next?” for ownership and borrowing in Rust. I want to lay out
the various limitations of Rust’s ownership and borrowing system that
I see, as well as – where applicable – current workarounds. I’m
curious to get feedback on which problems affect folks the most.
The first limitation I wanted to focus on is interprocedural
conflicts. In fact, I’ve covered a special case of this before –
where a closure conflicts with its creator function – in my post on
Precise Closure Capture Clauses. But the problem is more
general.
Now that the final Rust 2018 Release Candidate has
shipped, I thought it would be a good idea to do another
update on the state of the MIR-based borrow check (aka NLL). The last
update was in June, when we were still hard at work on getting
things to work.
Rust 2018 will use NLL now
Let’s get the highlights out of the way. Most importantly, Rust 2018
crates will use NLL by default. Once the Rust 2018 release candidate
becomes stable, we plan to switch Rust 2015 crates to use NLL as
well, but we’re holding off until we have some more experience with
people using it in the wild.
Just a quick note that the October 2018 office hour slots are
now posted. If you’re having a problem with Rust, or have something
you’d like to talk out, please sign up!
This is a report on the second “office hours”, in which we
discussed how to setup a series of services or actors that communicate
with one another. This is a classic kind of problem in Rust: how to
deal with cyclic data. Usually, the answer is that the cycle is not
necessary (as in this case).
The setup
To start, let’s imagine that we were working in a GC’d language, like
JavaScript. We want to have various “services”, each represented by an
object. These services may need to communicate with one another, so we
also create a directory, which stores pointers to all the
services. As each service is created, they add themselves to the
directory; when it’s all setup, each service can access all other
services. The setup might look something like this:
This is a report on the first “office hours”, in which we
discussed debugging Rust programs with gdb. I’m very grateful to
Ramana Venkata for suggesting the topic, and to Tom Tromey, who joined
in. (Tom has been doing a lot of the work of integrating rustc into
gdb and lldb lately.)
This blog post is just going to be a quick summary of the basic
workflow of using Rust with gdb on the command line. I’m assuming you
are using Linux here, since I think otherwise you would prefer a
different debugger. There are probably also nifty graphical tools you
can use and maybe even IDE integrations, I’m not sure.
Hello, all! Beginning this Friday (in two days)1, I’m going
to start an experiment that I call Rust office hours. The idea is
simple: I’ve set aside a few slots per week to help people work
through problems they are having learning or using Rust. My goal here
is both to be of service but also to gain more insight into the kinds
of things people have trouble with. No problem is too big or too
small!2
This post examines a particular, seemingly simple problem: given
ownership of a Rc<Vec<u32>>, can we write a function that returns an
impl Iterator<Item = u32>? It turns out that this is a bit harder
than it might at first appear – and, as we’ll see, for good
reason. I’ll dig into what’s going on, how you can fix it, and how we
might extend the language in the future to try and get past this
challenge.
One of the long-standing issues that we’ve been wrestling with in Rust
is how to integrate the concept of an “uninhabited type” – that is, a
type which has no values at all. Uninhabited types are useful to
represent the “result” of some computation you know will never execute
– for example, if you have to define an error type for some
computation, but this particular computation can never fail, you might
use an uninhabited type.
I consider Rust’s RFC process one of our great accomplishments, but
it’s no secret that it has a few flaws. At its best, the RFC offers an
opportunity for collaborative design that is really exciting to be a
part of. At its worst, it can devolve into bickering without any real
motion towards consensus. If you’ve not done so already, I strongly
recommend reading aturon’s excellentblogposts on
this topic.
The RFC process has also evolved somewhat organically over time. What
began as “just open a pull request on GitHub” has moved into a process
with a number of formal and informal stages (described below). I think
it’s a good time for us to take a step back and see if we can refine
those stages into something that works better for everyone.
I’ve been getting a lot of questions about the status of “Non-lexical
lifetimes” (NLL) – or, as I prefer to call it these days, the
MIR-based borrow checker – so I wanted to post a status
update.
The single most important fact is that the MIR-based borrow check is
feature complete and available on nightly. What this means is that
the behavior of #![feature(nll)] is roughly what we intend to ship
for “version 1”, except that (a) the performance needs work and (b) we
are still improving the diagnostics. (More on those points later.)
Ever since the Rust All Hands, I’ve been experimenting with an
alternative formulation of the Rust borrow checker. The goal is to
find a formulation that overcomes some shortcomings of the current
proposal while hopefully also being faster to compute. I have
implemented a prototype for this analysis. It passes the full NLL test
suite and also handles a few cases – such as #47680 – that the
current NLL analysis cannot handle. However, the performance has a
long way to go (it is currently slower than existing analysis). That
said, I haven’t even begun to optimize yet, and I know I am doing some
naive and inefficient things that can definitely be done better; so I
am still optimistic we’ll be able to make big strides there.
This is the second in a series of posts about Rust compiler
errors. Each one will talk about a particular error that I got
recently and try to explain (a) why I am getting it and (b) how I
fixed it. The purpose of this series of posts is partly to explain
Rust, but partly just to gain data for myself. I may also write posts
about errors I’m not getting – basically places where I anticipated
an error, and used a pattern to avoid it. I hope that after writing
enough of these posts, I or others will be able to synthesize some of
these facts to make intermediate Rust material, or perhaps to improve
the language itself.
I’ve decided to do a little series of posts about Rust compiler
errors. Each one will talk about a particular error that I got
recently and try to explain (a) why I am getting it and (b) how I
fixed it. The purpose of this series of posts is partly to explain
Rust, but partly just to gain data for myself. I may also write posts
about errors I’m not getting – basically places where I anticipated
an error, and used a pattern to avoid it. I hope that after writing
enough of these posts, I or others will be able to synthesize some of
these facts to make intermediate Rust material, or perhaps to improve
the language itself.
Breakthrough #2: @nikomatsakis had a eureka moment and figured out a
path to make specialization sound, while still supporting its most
important use cases (blog post forthcoming!). Again, this suddenly
puts specialization on the map for Rust Epoch 2018.
Sheesh I wish they hadn’t written that! Now the pressure is on. Well,
here goes nothing =).
I’ve been thinking a lot about persistent collections lately and
in particular how they relate to Rust, and I wanted to write up some
of my observations.1
What is a persistent collection?
Traditionally, persistent collections are seen as this “wildly
different” way to setup your collection. Instead of having
methods like push, which grow a vector in place:
vec.push(element);// add element to `vec`
you have a method like add, which leaves the original vector alone
but returns a new vector that has been modified:
In my last Chalk post, I talked about an experimental, SLG-based
solver that I wrote for Chalk. That particular design was based very
closely on the excellent paper
“Efficient top-down computation of queries under the well-founded semantics”, by W. Chen, T. Swift, and D. Warren. It
followed a traditional Prolog execution model: this has a lot of
strengths, but it probably wasn’t really suitable for use in rustc.
The single biggest reason for this was that it didn’t really know when
to stop: given a query like exists<T> { T: Sized }, it would happily
try to enumerate all sized types in the system. It was also pretty
non-obvious to me how to extend that system with things like
co-inductive predicates (needed for auto traits) and a few other
peculiarities of Rust.
As part of #Rust2018, I thought I would try to writeup my own
(current) perspective. I’ll try to keep things brief.
First and foremost, I think that this year we have to finish what we
started and get the “Rust 2018” release out the door. We did good
work in 2017: now we have to make sure the world knows it and can use
it. This primarily means we have to do stabilization work, both for
the recent features added in 2017 as well as some, ahem,
longer-running topics, like SIMD. It also means keeping up our focus
on tooling, like IDE support, rustfmt, and debugger integration.
So, as you likely know, we tried something new at the end of 2017. For
roughly the final quarter of the year, we essentially stopped doing
design work, and instead decided to focus on implementation – what we
called the “impl period”. We had two goals for the impl period:
(a) get a lot of high-value implementation work done and (b) to do
that by expanding the size of our community, and making it easy for
new people to get involved. To that end, we spun up about 40 working
groups, which is really a tremendous figure when you think about it,
each of which was devoted to a particular task.
In my last post about chalk queries, I discussed how the query
model in chalk. Since that writing, there have been some updates, and
I thought it’d be nice to do a new post covering the current model.
This post will also cover the tabling technique that scalexm
implemented for handling cyclic relations and show how that enables us
to implement implied bounds and other long-desired features in an
elegant way. (Nice work, scalexm!)
I’ve been hard at work the last month or so on trying to complete the
non-lexical lifetimes RFC. I’m pretty excited about how it’s shaping
up. I wanted to write a kind of “meta” blog post talking about the
current state of the proposal – almost there! – and how you could
get involved with helping to push it over the finish line.
TL;DR
What can I say, I’m loquacious! In case you don’t want to read the
full post, here are the highlights:
For my next post discussing chalk, I want to take kind of a
different turn. I want to talk about the general struct of chalk
queries and how chalk handles them right now. (If you’ve never heard
of chalk, it’s sort of “reference implementation” for Rust’s trait
system, as well as an attempt to describe Rust’s trait system in terms
of its logical underpinnings; see
this post for an introduction to the big idea.)
I recently participated in the GNOME / Rust “dev sprint” in Mexico
City. (A thousand thanks to Federico and Joaquin for organizing!)
While there I spent some time working on the
gnome-class plugin. The
goal of gnome-class was to make it easy to write GObject
implementations in Rust which would fully interoperate with C code.
Roughly speaking, my goal was that you should be able to write code
that looked and felt like
Vala code, but where the
method bodies (and types, and so forth) are in Rust. The plugin is in
no way done, but I think it’s already letting you do some pretty nice
stuff. For example, this little snippet defines a Counter class
offering two methods (add() and get()):
In my previous post, I talked over the basics of how
unification works and showed how that “mathematical version” winds
up being expressed in chalk. I want to go a bit further now and extend
that base system to cover associated types. These turn out to be a
pretty non-trival extension.
What is an associated type?
If you’re not a Rust programmer, you may not be familiar with the term
“associated type” (although many langages have equivalents). The basic
idea is that traits can have type members associated with them. I
find the most intuitive example to be the Iterator trait, which has
an associated type Item. This type corresponds to kind of elements
that are produced by the iterator:
We just released Rayon 0.7. This is a pretty exciting release, because
it marks the official first step towards Rayon 1.0. In addition, it
marks the first release where Rayon’s parallel iterators reach
“feature parity” with the standard sequential iterators! To mark the
moment, I thought I’d post the release notes here on the blog:
This release marks the first step towards Rayon 1.0. For best
performance, it is important that all Rayon users update to at least
Rayon 0.7. This is because, as of Rayon 0.7, we have taken steps to
ensure that, no matter how many versions of rayon are actively in use,
there will only be a single global scheduler. This is achieved via the
rayon-core crate, which is being released at version 1.0, and which
encapsulates the core schedule APIs like join(). (Note: the
rayon-core crate is, to some degree, an implementation detail, and
not intended to be imported directly; it’s entire API surface is
mirrored through the rayon crate.)
So in my first post on chalk, I mentioned that unification and
normalization of associated types were interesting topics. I’m going
to write a two-part blog post series covering that. This first part
begins with an overview of how ordinary type unification works during
compilation. The next post will add in associated types and we can see
what kinds of mischief they bring with them.
What is unification?
Let’s start with a brief overview of what unification is. When you are
doing type-checking or trait-checking, it often happens that you wind
up with types that you don’t know yet. For example, the user might
write None – you know that this has type Option<T>, but you don’t
know what that type T is. To handle this, the compiler will create a
type variable. This basically represents an unknown,
to-be-determined type. To denote this, I’ll write Option<?T>, where
the leading question mark indicates a variable.
For some time now I’ve been interested in better ways to construct
LR(1) parsers. LALRPOP currently allows users to choose between the
full LR(1) algorithm or the LALR(1) subset. Neither of these choices
is very satisfying:
the full LR(1) algorithm gives pretty intuitive results but produces
a lot of states; my hypothesis was that, with modern computers, this
wouldn’t matter anymore. This is sort of true – e.g., I’m able to
generate and process even the full Rust grammar – but this
results in a ton of generated code.
the LALR(1) subset often works but sometimes mysteriously fails with
indecipherable errors. This is because it is basically a hack that
conflates states in the parsing table according to a heuristic; when
this heuristic fails, you get strange results.
The Lane Table algorithm published by Pager and Chen at APPLC
‘12 offers an interesting alternative. It is an alternative to earlier
work by Pager, the “lane tracing” algorithm and practical general
method. In any case, the goal is to generate an LALR(1) state machine
when possible and gracefully scale up to the full LR(1) state
machine as needed.
In my previous post, I
outlined a plan for non-lexical lifetimes. I wanted to write a
follow-up post today that discusses different ways that we can extend
the system to support nested mutable calls. The ideas here are based
on some the ideas that emerged in a
recent discussion on internals, although what I describe
here is a somewhat simplified variant. If you want more background,
it’s worth reading at least the top post in the thread, where I laid
out a lot of the history here. I’ll try to summarize the key bits as I
go.
At the recent compiler design sprint,
we spent some time discussing non-lexical lifetimes,
the plan to make Rust’s lifetime system significantly more advanced. I
want to write-up those plans here, and give some examples of the kinds
of programs that would now type-check, along with some that still will
not (for better or worse).
If you were at the sprint, then the system I am going to describe in
this blog post will actually sound quite a bit different than what we
were talking about. However, I believe it is equivalent to that
system. I am choosing to describe it differently because this version,
I believe, would be significantly more efficient to implement (if
implemented naively). I also find it rather easier to understand.
I want to have a tool that would enable us to answer all kinds of queries about the structure of Rust code that exists in the wild. This should cover everything from synctactic queries like “How often do people write let x = if { ... } else { match foo { ... } }?” to semantic queries like “How often do people call unsafe functions in another module?” I have some ideas about how to build such a tool, but (I suspect) not enough time to pursue them. I’m looking for people who might be interested in working on it!
This last week we had the rustc compiler team design sprint. This
was our second rustc compiler team sprint; the first one (last year)
we simply worked on pushing various projects over the finish line (for
example, in an epic effort, arielb1 completed dynamic drop during that
sprint).
This sprint was different: we had the goal of talking over many of the
big design challenges that we’d like to tackle in the upcoming year
and making sure that the compiler team was roughly on board with the
best way to implement them.
In a previous post, I talked about a proposed approach to drafting the
unsafe code guidelines. Specifically, I want to the approach of having
an executable specification of Rust with additional checks that
will signal when undefined behavior has occurred. In this post, I want
to try to dive into that idea a bit more and give some more specifics
of the approach I have in mind. I’m going to focus on this post on the
matter of the proper use of shared references &T – I’ll completely
ignore &mut T for now, since those are much more complicated
(because they require a notion of uniqueness).
Over the last year or two (man, it’s scary how time flies), I’ve been
doing quite a lot of thinking about Rust’s trait system. I’ve been
looking for a way to correct a number of flaws and shortcomings in the
current implementation, not the least of which is that it’s
performance is not that great. But also, I’ve been wanting to get a
relatively clear, normative definition of how the trait system works,
so that we can better judge possible extensions. After a number of
false starts, I think I’m finally getting somewhere, so I wanted to
start writing about it.
While I was at POPL the last few days, I was reminded of an idea
regarding how to bring more struture to the unsafe code guidelines
process that I’ve been kicking around lately, but which I have yet to
write about publicly. The idea is fresh on my mind because while at
POPL I realized that there is an interesting opportunity to leverage
the “blame” calculation techniques from gradual typing research. But
before I get to blame, let me back up and give some context.
This post is the (long awaited, or at least long promised) third post
in my series on Rayon’s parallel iterators. The previous two posts
were some time ago, but I’ve been feeling inspired to push more on
Rayon lately, and I remembered that I had never finished this blog
post series.
Here is a list of the other posts in the series. If you haven’t read
them, or don’t remember them, you will want to do so before reading
this one:
This post is a continuation of my posts discussing the topic of
associated type constructors (ATC) and higher-kinded types (HKT):
The first post focused on introducing the basic idea of
ATC, as well as introducing some background material.
The second post showed how we can use ATC to model HKT,
via the “family” pattern.
The third post did some exploration into what it would
mean to support HKT directly in the language, instead of modeling
them via the family pattern.
This post considers what it might mean if we had both ATC and HKT
in the language: in particular, whether those two concepts can be
unified, and at what cost.
Unifying HKT and ATC
So far we have seen “associated-type constructors” and “higher-kinded
types” as two distinct concepts. The question is, would it make sense
to try and unify these two, and what would that even mean?
Hello. This post is a continuation of my posts discussing the topic of
associated type constructors (ATC) and higher-kinded types (HKT):
The first post focused on introducing the basic idea of
ATC, as well as introducing some background material.
This post talks about some apparent limitations of associated type
constructors, and shows how we can overcome them by making use of a
design pattern that I call “family traits”. Along the way, we
introduce the term higher-kinded type for the first time, and
show (informally) that family traits are equally general.
The limits of associated type constructors
OK, so in the last post we saw how we can use ATC to define a
Collection trait, and how to implement that trait for our sample
collection List<T>. In particular, ATC let us express the return
type of the iterator() method as Self::Iter<'iter>, so that we can
incorporate the lifetime 'iter of each particular iterator.
So for the end of last week, I was at Rust Belt Rust. This was
awesome. And not only because the speakers and attendees at Rust Belt
Rust were awesome, though they were. But also because it gave aturon,
withoutboats, and I a chance to talk over a lot of stuff in person. We
covered a lot of territory and so I wanted to do a series of blog
posts trying to write down some of the things we were thinking so as
to get other people’s input.
If you visit the site today, you’ll notice it looks quite a bit
different. I’ve decided to switch from my old antiquated Octopress to
a plain Jekyll-based one. The most immediate of this is that Rust code
highlighting looks much better, and I get access to modern
Github-flavored markdown. =) Since I understand plain Jekyll a bit
more, I’ll hopefully also be able to customize the appearance somewhat
– but for now I’m just going with the basic theme.
In my previous post, I talked about how we can separate out
specialization into two distinct concepts: reuse and override.
Doing so makes because the conditions that make reuse possible are
more stringent than those that make override possible. In this post,
I want to extend this idea to talk about a new rule for specialization
that allow overriding in more cases. These rules are a big enabler
for specialization, allowing it to accommodate many use cases that we
couldn’t handle before. In particular, they enable us to add blanket
impls like impl<T: Copy> Clone for T in a backwards compatible
fashion, though only under certain conditions.
I spent a really interesting day last week at Northeastern University.
First, I saw a fun talk by Philip Haller covering LaCasa, which is a
set of extensions to Scala that enable it to track ownership. Many of
the techniques reminded me very much of Rust (e.g., the use of
“spores”, which are closures that can limit the types of things they
close over); if I have time, I’ll try to write up a more detailed
comparison in some later post.
For the past year or so, I and a few others have been iterating on
some tutorial slides for learning Rust. I’ve given this tutorial here
at the local Boston Rust Meetup a few times, and we used the same
basic approach at RustConf; I’ve been pretty happy with the
results. But until now it’s been limited to “in person” events.
That’s why I’m so happy to announce a new site, Into Rust. Into Rust
contains screencasts of many of these slides, and in particular the
ones I consider most important: those that cover Ownership and
Borrowing, which I think is the best place to start teaching Rust.
I’ve divided up the material into roughly 30min screencasts so that
they should be relatively easy to consume in one sitting – each also
has some associated exercises to help make your knowledge more
concrete.
In my previous post, I started discussing the idea of
intersection impls, which are a possible extension to
specialization. I am specifically looking at the idea of
making it possible to add blanket impls to (e.g.) implement Clone
for any Copy type. We saw that intersection impls, while useful, do
not enable us to do this in a backwards compatible way.
Today I want to dive a bit deeper into specialization. We’ll see that
specialization actually couples together two things: refinement of
behavior and reuse of code. This is no accident, and its normally a
natural thing to do, but I’ll show that, in order to enable the kinds
of blanket impls I want, it’s important to be able to tease those
apart somewhat.
As some of you are probably aware, on the nightly Rust builds, we
currently offer a feature called specialization, which was defined
in RFC 1210. The idea of specialization is to improve Rust’s
existing coherence rules to allow for overlap between impls, so long
as one of the overlapping impls can be considered more
specific. Specialization is hotly desired because it can enable
powerful optimizations, but also because it is an important component
for modeling object-oriented designs.
I’ve been thinking about the unsafe code guidelines a lot in the back
of my mind. In particular, I’ve been trying to think through what it
means to “trust types” – if you recall from the
Tootsie Pop Model (TPM) blog post, one of the key examples
that I was wrestling with was the RefCell-Ref example. I want to
revisit a variation on that example now, but from a different
angle. (This by the way is one of those “Niko thinks out loud” blog
posts, not one of those “Niko writes up a proposal” blog posts.)
A little while back, I wrote up a tentative proposal I called the
“Tootsie Pop” model for unsafe code. It’s safe to say that this
model was not universally popular. =) There was quite a
long and fruitful discussion on discuss. I wanted to write a
quick post summarizing my main take-away from that discussion and to
talk a bit about the plans to push the unsafe discussion forward.
The importance of the unchecked-get use case
For me, the most important lesson was the importance of the “unchecked
get” use case. Here the idea is that you have some (safe) code which
is indexing into a vector:
In my previous post, I spent some time talking about the idea of
unsafe abstractions. At the end of the post, I mentioned that Rust
does not really have any kind of official guidelines for what kind of
code is legal in an unsafe block and what is not.What this means in
practice is that people wind up writing what “seems reasonable” and
checking it against what the compiler does today. This is of course a
risky proposition since it means that if we start doing more
optimization in the compiler, we may well wind up breaking unsafe code
(the code would still compile; it would just not execute like it used
to).
The unsafe keyword is a crucial part of Rust’s design. For those not
familiar with it, the unsafe keyword is basically a way to bypass
Rust’s type checker; it essentially allows you to write something more
like C code, but using Rust syntax.
The existence of the unsafe keyword sometimes comes as a surprise at
first. After all, isn’t the point of Rust that Rust programs should
not crash? Why would we make it so easy then to bypass Rust’s type
system? It can seem like a kind of flaw in the design.
This is the third post in my
series on non-lexical lifetimes. Here I want to dive into
Problem Case #3 from the introduction. This is an interesting
case because exploring it is what led me to move away from the
continuous lifetimes proposed as part of RFC 396.
Problem case #3 revisited
As a reminder, problem case #3 was the following fragment:
What makes this example interesting is that it crosses functions. In
particular, when we call get_mut the first time, if we get back a
Some value, we plan to return the point, and hence the value must
last until the end of the lifetime 'm (that is, until some point in
the caller). However, if we get back a None value, we wish to
release the loan immediately, because there is no reference to return.
In my previous post I outlined several cases that we would like
to improve with Rust’s current borrow checker. This post discusses one
possible scheme for solving those. The heart of the post is two key ideas:
Define a lifetime as a set of points in the control-flow
graph, where a point here refers to some particular statement
in the control-flow graph (i.e., not a basic block, but some
statement within a basic block).
Use liveness as the basis for deciding where a variable’s type
must be valid.
The rest of this post expounds on these two ideas and shows how they
affect the various examples from the previous post.
Over the last few weeks, I’ve been devoting my free time to fleshing
out the theory behind non-lexical lifetimes (NLL). I think I’ve
arrived at a pretty good point and I plan to write various posts
talking about it. Before getting into the details, though, I wanted to
start out with a post that lays out roughly how today’s lexical
lifetimes work and gives several examples of problem cases that we
would like to solve.
For the last couple of weeks, my mornings have been occupied with a
pretty serious revamping of LALRPOP’s error message output. I will
probably wind up doing a series of blog posts about the internal
details of how it works, but I wanted to write a little post to
advertise this work.
Typically when you use an LR(1) parser generator, error messages tend
to be written in terms of the LR(1) state generation algorithm. They
use phrases like “shift/reduce conflict” and talk about LR(1)
items. Ultimately, you have to do some clever thinking to relate the
error to your grammar, and then a bit more clever thinking to figure
out how you should adjust your grammar to make the problem go away.
While working on adapting the Rust grammar to LALRPOP, I
found I was wasting a lot of time trying to decrypt the error
messages, and I wanted to do something about it. This work
is the result.
This post is the second post in my series on Rayon’s parallel
iterators. The goal of this series is to explain how parallel
iterators are implemented internally, so I’m going to be going over a
lot of details and giving a lot of little code examples in Rust. If
all you want to do is use parallel iterators, you don’t really have
to understand any of this stuff.
I’ve had a lot of fun designing this system, and I learned a few
lessons about how best to use Rust (some of which I cover in the
conclusions). I hope you enjoy reading about it!
Since giving a talk about Rayon at the Bay Area Rust meetup,
I’ve been working off and on on the support for parallel
iterators. The basic idea of a parallel iterator is that I should be
able to take an existing iterator chain, which operates sequentially,
and easily convert it to work in parallel. As a simple example,
consider this bit of code that computes the dot-product of two
vectors:
Over the last week or so, I’ve been working on an update to
Rayon, my experimental library for data parallelism in
Rust. I’m pretty happy with the way it’s been going, so I wanted to
write a blog post to explain what I’ve got so far.
Rayon’s goal is to make it easy to add parallelism to your
sequential code – so basically to take existing for loops or
iterators and make them run in parallel. For example, if you have an
existing iterator chain like this:
So, aturon wrote this interesting post on an alternative
“virtual structs” approach, and, more-or-less since he wrote it, I’ve
been wanting to write up my thoughts. I finally got them down.
(OK, I lied, one more note: starting with this post, I’ve decided to
disable comments on this blog. There are just too many forums to keep
up with! So if you want to discuss this post, I’d recommend doing so
on this Rust internals thread.)
Around four years ago, when I had first decided to start at Mozilla
research, I had planned to write an LR(1) parser generator. It seemed
like a good way to get to know Rust. However, I found that newborns
actually occupy somewhat more time than anticipated (read: I was lucky
to squeeze in a shower), and hence that never came to pass.
Well, I’m happy to say that, four years later, I’ve finally rectified
that. For a few months now I’ve been working on a side project while I
have my morning coffee: LALRPOP (pronounced like some sort of
strangely accented version of “lollypop”). LALRPOP is an LR(1)
parser generator that emits Rust code. It is designed for ease of use,
so it includes a number of features that many parser generators are
missing:
So, in previousposts, I discussed the pros and cons of two different
approaches to modeling variants: Rust-style enums and C++-style
classes. In those posts, I explained why I see Rust enums and OO-style
class hierarchies as more alike than different (I personally credit
Scala for opening my eyes to this, though I’m sure it’s been
understood by others for much longer). The key points were as follows:
Both Rust-style enums and C++-style classes can be used to model the
idea of a value that be one of many variants, but there are
differences in how they work at runtime. These differences mean that
Rust-style enums are more convenient for some tasks, and C++-style
classes for others. In particular:
A Rust-style enum is sized as large as the largest variant. This is
great because you can lay them out flat in another data structure
without requiring any allocation. You can also easily change from
one variant to another. One downside of Rust enums is that you cannot
“refine” them to narrow the set of variants that a particular value
can have.
A C++-style class is sized to be exactly as big as one variant. This
is great because it can be much more memory efficient. However, if
you don’t know what variant you have, you must manipulate the value
by pointer, so it tends to require more allocation. It is also
impossible to change from one variant to another. Class hierarchies
also give you a simple, easily understood kind of refinement, and
the ability to have common fields that are shared between variants.
C++-style classes offer constructors, which allows for more
abstraction and code reuse when initially creating an instance, but
raise thorny questions about the type of a value under construction;
Rust structs and enums are always built in a single-shot today,
which is simpler and safer but doesn’t compose as well.
What I want to talk about in this post is a proposal (or
proto-proposal) for bridging those two worlds in Rust. I’m going to
focus on data layout in this post. I’ll defer virtual methods for
another post (or perhaps an RFC). Spoiler alert: they can be viewed
as a special case of specialization.
This is the second post summarizing my current thoughts about ideas
related to “virtual structs”. In the last post, I described how,
when coding C++, I find myself missing Rust’s enum type. In this post,
I want to turn it around. I’m going to describe why the class model
can be great, and something that’s actually kind of missing from
Rust. In the next post, I’ll talk about how I think we can get the
best of both worlds for Rust. As in the first post, I’m focusing here
primarily on the data layout side of the equation; I’ll discuss
virtual dispatch afterwards.
One priority for Rust after 1.0 is going to be incorporating some
kind of support for
“efficient inheritance” or “virtual structs”. In order to
motivate and explain this design, I am writing a series of blog posts
examining how Rust’s current abstractions compare with those found in
other languages.
The way I see it, the topic of “virtual structs” has always had two
somewhat orthogonal components to it. The first component is a
question of how we can generalize and extend Rust enums to cover more
scenarios. The second component is integrating virtual dispatch into
this picture.
So there has been a lot of really interesting discussion in response
to my blog post. I wanted to highlight some of the comments I’ve seen,
because I think they raise good points that I failed to address in the
blog post itself. My comments here are lightly edited versions of what
I wrote elsewhere.
Isn’t the problem with objects and leak-safe types more general?
I posit that this is in fact a problem with trait objects, not a
problem with Leak; the exact same flaw pointed about in the blog
post already applies to the existing OIBITs, Send, Sync, and
Reflect. The decision of which OIBITs to include on any trait object
is already a difficult one, and is a large reason why std strives to
avoid trait objects as part of public types.
What’s a 1.0 release without a little drama? Recently, we discovered
that there was an oversight in one of the standard library APIs that we
had intended to stabilize. In particular, we recently added an API for
scoped threads – that is, child threads which have access to the
stack frame of their parent thread.
The flaw came about because, when designing the scoped threads API, we
failed to consider the impact of resource leaks. Rust’s ownership
model makes it somewhat hard to leak data, but not impossible. In
particular, using reference-counted data, you can construct a cycle in
the heap, in which case the components of that cycle may never be
freed.
After reading nrc’s blog post about graphs, I felt inspired to
write up an alternative way to code graphs in Rust, based on vectors
and indicates. This encoding has certain advantages over using Rc
and RefCell; in particular, I think it’s a closer fit to Rust’s
ownership model. (Of course, it has disadvantages too.)
I’m going to describe a simplified version of the strategy that rustc
uses internally. The actual code in Rustc is written in a
somewhat dated “Rust dialect”. I’ve also put the sources to this blog
post in their own GitHub repository. At some point, presumably
when I come up with a snazzy name, I’ll probably put an extended
version of this library up on crates.io. Anyway, the code I cover in
this blog post is pared down to the bare essentials, and so it doesn’t
support (e.g.) enumerating incoming edges to a node, or attach
arbitrary data to nodes/edges, etc. It would be easy to extend it to
support that sort of thing, however.
We’ve recently been doing a lot of work on Rust’s orphan rules,
which are an important part of our system for guaranteeing trait
coherence. The idea of trait coherence is that, given a trait and
some set of types for its type parameters, there should be exactly one
impl that applies. So if we think of the trait Show, we want to
guarantee that if we have a trait reference like MyType : Show, we
can uniquely identify a particular impl. (The alternative to coherence
is to have some way for users to identify which impls are in scope at
any time. It has its own complications; if you’re curious for
more background on why we use coherence, you might find this
rust-dev thread from a while back to be interesting
reading.)
The so-called “unboxed closure” implementation in Rust has reached the
point where it is time to start using it in the standard library. As
a starting point, I have a
pull request that removes proc from the language. I started
on this because I thought it’d be easier than replacing closures, but
it turns out that there are a few subtle points to this transition.
I am writing this blog post to explain what changes are in store and
give guidance on how people can port existing code to stop using
proc. This post is basically targeted Rust devs who want to adapt
existing code, though it also covers the closure design in general.
There has been a lot of discussion lately about Rust’s allocator
story, and in particular our relationship to jemalloc. I’ve been
trying to catch up, and I wanted to try and summarize my understanding
and explain for others what is going on. I am trying to be as
factually precise in this post as possible. If you see a factual
error, please do not hesitate to let me know.
The core tradeoff
The story begins, like all interesting design questions, with a
trade-off. The problem with trade-offs is that neither side is 100%
right. In this case, the trade-off has to do with two partial truths:
I’ve been working on a branch that implements both multidispatch
(selecting the impl for a trait based on more than one input type) and
conditional dispatch (selecting the impl for a trait based on where
clauses). I wound up taking a direction that is slightly different
from what is described in the trait reform RFC, and I
wanted to take a chance to explain what I did and why. The main
difference is that in the branch we move away from the crate
concatenability property in exchange for better inference and less
complexity.
A few weeks back pcwalton introduced a PR that aimed to move the
attribute and macro syntax to use a leading @ sigil. This means that
one would write macros like:
This proposal was controversial. This debate has been sitting for a
week or so. I spent some time last week reading every single comment
and I wanted to lay out my current thoughts.
While on vacation, I’ve been working on an alternate type inference
scheme for rustc. (Actually, I got it 99% working on the plane, and
have been slowly poking at it ever since.) This scheme simplifies the
code of the type inferencer dramatically and (I think) helps to meet
our intutions (as I will explain). It is however somewhat less
flexible than the existing inference scheme, though all of rustc and
all the libraries compile without any changes. The scheme will (I
believe) make it much simpler to implement to proper one-way matching
for traits (explained later).
I am on vacation for a few weeks. I wanted to take some time to jot
down an idea that’s been bouncing around in my head. I plan to submit
an RFC at some point on this topic, but not yet, so I thought I’d
start out by writing a blog post. Also, my poor blog has been
neglected for some time. Consider this a draft RFC. Some important
details about references are omitted and will come in a follow-up blog
post.
As you can see from the diagram, the &mut reference is a unique
reference to the integer. That is, it can’t be copied, and it’s the
only direct pointer to that integer. However, there are multiple
paths to the &mut itself. That’s not the fault of the &mut, it’s
just that uniqueness is a global property. In other words, if I have a
variable p of type &&mut int, then **p is not a unique path,
even though it traverses through a unique reference at some point.
Over time, I’ve become convinced that it would be better to drop the
distinction between mutable and immutable local variables in
Rust. Many people are highly skeptical, to say the least. I
wanted to lay out my argument in public. I’ll give various
motivations: a philosophical one, an eductional one, and a practical
one, and also address the main defense of the current system. (Note: I
considered submitting this as a Rust RFC, but decided that the tone
was better suited to a blog post, and I don’t have the time to rewrite
it now.)
I’ve been thinking about an alternative way to factor the PJS API.
Until now, we’ve had these methods like mapPar(), filterPar() and
so forth. They work mostly like their sequential namesakes but execute
in parallel. This API has the advantage of being easy to explain and
relatively clear, but it’s also not especially flexible nor elegant.
Lately, I’ve been prototyping an alternate design that I call
parallel pipelines (that’s just a working title; I expect the name
to change). Compared to the older approach, parallel pipelines are a
more expressive API that doesn’t clutter up the array prototypes. The
design draws on precedent from a lot of other languages, such as
Clojure, Ruby, and Scala, which all offer similar capabilities. I’ve
prototyped the API on a branch of SpiderMonkey, though the
code doesn’t yet run in parallel (it is structured in such a way as to
make parallel execution relatively straightforward, though).
I recently wrote up a
paper describing the current version of the Typed Objects API. Anyone
who is interested in the current state of the art in that
specification should take a look. It’s not too long and intended to be
an easy read. This is just a draft copy, and feedback is naturally
very welcome – in particular, I expect that before we submit it, the
implementation section will change, since it will be much further
along.
Here is the current state of my thinking with respect to value types
and value objects. Some of you may have seen
Brendan’s slides where he discusses value objects. This post
is about the same topic, but it is focused on just the initial part of
the work – what it means to be a value object and how we could define
value types and integrate them into the standard. I am not going to
discuss new syntax or operators yet. I have thoughts on those too but
I wanted to start by laying out the foundations.
In today’s Rust, there are a number of builtin traits (sometimes
called “kinds”): Send, Freeze, Share, and Pod (in the future,
perhaps Sized). These are expressed as traits, but they are quite
unlike other traits in certain ways. One way is that they do not have
any methods; instead, implementing a trait like Freeze indicates
that the type has certain properties (defined below). The biggest
difference, though, is that these traits are not implemented manually
by users. Instead, the compiler decides automatically whether or not a
type implements them based on the contents of the type.
Today, if you do a mutable borrow of a local variable, you lose the
ability to write to that variable except through the new reference
you just created:
let mut x = 3;
let p = &mut x;
x += 1; // Error
*p += 1; // OK
However, you retain the ability to read the original variable:
let mut x = 3;
let p = &mut x;
print(x); // OK
print(*p); // OK
I would like to change the borrow checker rules so that both writes
and reads through the original path x are illegal while x is
mutably borrowed. This change is not motivated by soundness, as I
believe the current rules are sound. Rather, the motivation is that
this change gives strong guarantees to the holder of an &mut
pointer: at present, they can assume that an &mut referent will not
be changed by anyone else. With this change, they can also assume
that an &mut referent will not be read by anyone else. This enable
more flexible borrowing rules and a more flexible kind of data
parallelism API than what is possible today. It may also help to
create more flexible rules around moves of borrowed data. As a side
benefit, I personally think it also makes the borrow checker rules
more consistent (mutable borrows mean original value is not usable
during the mutable borrow, end of story). Let me lead with the
motivation.
I have been working on making the borrow checker treat closures in a
sound way. I hope to land this patch very soon. I want to describe the
impact of these changes and summarize what we might do in the future.
The high-level idea
The basic idea is that the borrow checker will treat a closure as if
it were constructing a record with one borrowed pointer for each
variable mentioned in the closure.
I’ve been working on Issue #3511, which is an effort to
rationalize the lifetimes of temporary values in Rust. This issue has
been a thorn in the side of Rust users for a while, because the
current lifetimes are rather haphazard and frequently too short. Some
time ago, I did some thinking on this issue and then let it lie
while other things took priority.
Part of the reason that this issue has lasted so long is that the
current trans cleanup scheme is very inflexible. I have a
branch now that rewrites the cleanup system so that it can
handle any rules we would like. The problem I am encountering now, of
course, is that it’s unclear what the rules should be. I want to lay
out the options I see.
I believe I have come to the point where I am ready to make a final
proposal for DST. Ironically, this proposal is quite similar to where
I started, but somewhat more expansive. It seems to be one of those
unusual cases where supporting more features actually makes things
easier. Thanks to Eridius on IRC for pointing this out to me. I
intend for this post to stand alone, so I’m going to start from the
beginning in the description.
Dave Herman and I were tossing around ideas the other day for a
revision of the typed object specification in which we remove nominal
array types. The goal is to address some of the awkwardness that we
have encountered in designing the PJS API due to nominal array types.
I thought I’d try writing it out. This is to some extent a thought
experiment.
Description by example
I’ve had a hard time trying to identify the best way to present the
idea, because it is at once so similar and so unlike what we have
today. So I think I’ll begin by working through examples and then
try to define a more abstract version.
Over the Thanksgiving break I’ve been devoting a lot of time to
thinking about DST and Rust’s approach to vector and object types. As
before, this is very much still churning in my mind so I’m just going
to toss out some semi-structured thoughts.
Brief recap
Treating vectors like any other container. Some time back, I wrote
up a post about how we could
treat vectors like any other container, which would (to some
extent) avoid the need for DST.
In the previous post I elaborated a bit on DSTs and how they could be
created and used. I want to look a bit now at an alternate way to
support the combination of vector types and smart pointers (e.g.,
RC<[uint]>). This approach avoids the use of DSTs. We’ll see that it
also addresses some of the rough patches of DST, but doesn’t work
quite as well for object types.
After posting part 2 of my DST series, I realized that I had
focusing too much on the pure “type system” aspect and ignoring some
of the more…mundane semantics, and in particular the impact of
monomorphization. I realize now that – without some further changes
– we would not be able to compile and execute the second proposal
(which I will dub statically sized typed (SST) from here on
out). Let me first explain the problem and then show how my first
thoughts on how it might be addressed.
In the past, I’ve been quite the champion of dynamically sized types
(DST). Specifically what this means is that things like [T] and
Trait would be “types” in the Rust type system. Lately I’ve been
investing a lot of effort thinking through the ramifications of
offering better support for smart pointers, and in particular how this
interacts with dynamically sized types, and I am no longer persuaded
that DST offer the best way forward. I’m a bit unsure, though, and the
topic is complicated, so I wanted to stop and write up a short series
of posts laying out my thought process thus far. This post will
describe what it would mean to offer DST in more detail. I don’t plan
to give a lot of Rust background, since there’s enough to talk about.
A quick follow-up to my previous post. The approach I suggested
(“generate boxing instructions but bypass them when possible”) is in
some sense pessimistic: we generate the instructions we need for the
worst case and then cleanup. Like many problems in computer science,
it has an optimistic dual. We could generate unboxed data and then
insert boxes where needed. In fact, we have an existing mechanism for
doing that, called the type policies. Basically, there is a phase
where each MIR opcode goes through and examines the types of its
inputs, attempting to reconcile those types with what it needs, either
by boxing or unboxing as needed.
There is currently some ongoing effort to implement the
proposed JavaScript SIMD API in Firefox. The basic idea of the
API is to introduce explicit vector value types called float32x4 and
int32x4. These types fit into the typed objects hierarchy, so you
can create arrays of them, embed them in structs, and so forth.
The semantics of these vectors types is designed to make it possible
for JIT engines to detect and optimize their use. One crucial bit is
that they are values and hence do not have identity. Basically
float32x4 values work like numbers and strings do today – they are
equal if they have the same value. This is quite different from
objects, which may be unequal if their properties are the same (e.g.,
{} !== {}).
Alex Chrichton recently sent a
message to the rust-dev mailing list discussing the fate of
parameter coercion in Rust. I’ve been thinking about this for a while
and feeling conflicted. As is my wont, I decided to try and write up a
blog post explaining precisely what’s under debate and exporing the
tradeoffs.
Historical background
In the interest of clarity, I wanted to briefly explain some
terminology and precisely what the rules are. I refer to “autoref”
as the addition of an implicit &: so converting from T to &T, in
terms of the type. “Autoderef” is the addition of an implicit *:
converting from &T, ~T, etc to T. Finally, “autoborrow” is the
addition of both a & and a *, which effectively converts from
~T, &T etc to &T. “Autoslice” is the conversion from ~[..] and
&[...] to &[...] – if we had a DST-based system, autoslice and
autoborrow would be the same thing, but in today’s world they are not,
and in fact there is no explicit syntax for slicing.
Some astutecomments on a recent thread to rust-dev got me
thinking about our approach to vectors. Until now, we have focused on
having built-in support for vectors via the vector type (~[T]) and
slice types (&[T]). However, another possible approach would be to
move vector support out of the language (almost) entirely and into
standard libraries. I wanted to write out a post exploring this idea;
I find it brings some simplifications and reduces the need for
DST. Seems like an idea worth considering. Consider this a thought
experiment, not exactly a proposal.
I got a lot of feedback on my post about
intermingled parameter lists – most of it negative – and I’ve
been thinking about the issue over the weekend. Truth is, I wasn’t
terribly fond of the proposal myself – making the position in the
list significant feels wrong – but I felt it was the least bad of the
various options. However, I’ve had a change of heart, and thus have a
new “least bad” proposal.
I want to optimize assignments to struct-typed fields in typed
objects. This post is an effort to work through my optimization plan.
The goal
Imagine some code like this:
var PointType = new StructType({x: int32, y: int32});
var LineType = new StructType({from: PointType,
to: PointType});
var line = new LineType();
line.to = {x: 22, y: 44};
The last line in particular is the one I am interested in. Today we
execute this in the most naive way. The code which ion generates looks
something like:
I’ve been hard at work finishing up work on Rust’s
“new” syntax for lifetimes – I put “new” in quotes because
the syntax has been in use for some time, but in rustc itself the
change was only half-baked. In effect, the new syntax was a kind of
“bridge” to code that was designed for the older type system. This
resulted in some artificial limitations: for example, types could only
have a single lifetime parameter, and it had to be named 'self.
Under my pull request, these limitations are lifted. However, in the
process of implementing things, I realized one minor problem with the
new syntax that must be rectified. In this post I describe the problem
and my proposed solution.
I think someone reading this blog would be forgiven for thinking that
I must spend most of my energy thinking about Rust. In fact I spend a
good part of my working hours hammering on PJS. I thought I’d try to
write up a bit of a preview of the things we are working on.
Parallel methods on arrays
Right now, on Nightly Firefox builds, you can use the parallel methods
mapPar, filterPar, and reducePar on normal JS arrays. These work
basically like their sequential equivalents except that they execute
in an undefined order (for reducePar, that can be a more significant
difference, since both the left and right operand might be the result
of a reduction). That means you can write code like:
There is a known bug with the borrowck rules that causes it to
be overly permissive. The fix is relatively simple but it
unfortunately affects some of our Iterator implementations,
specifically those iterators that iterate over &mut values. The
short version is that while it is possible to expose a safe
interface for iterating over &mut values, it is not possible to
implement such iterators without an unsafe block.
After giving this quite a bit of thought, I have come to the conclusion
that we have three options:
OK, after writing the
post on iterators that yield mutable references, and discussing
with some folk on IRC, I remembered something I had forgotten. There
is actually a way to phrase the mutable vector iterator differently
such that it is safe. Actually, the end result still has some unsafe
code, but it takes the form of a simple helper function, and it’s
quite plausible to imagine that code becoming safe eventually. Even
better, the approach generalizes to other data structures.
The following is a draft proposal to support a form of single
inheritance, similar to that found in object-oriented languages. The
goal is to enable servo to efficiently support structures like the
DOM. The proposal is not completely rounded out, but I wanted to put
it up in its current form so as to gather any comments.
In a nutshell, the proposal is to:
Enable structs to extend other structs, meaning that the substruct
inherits all fields of the superstruct, and also enabling a
subtyping relationship between borrowed pointers.
Yesterday Dmitry Lomov and I had a discussion about the typed objects
API. Much of the discussion revolved around the specific issue of
handles. In this post I will summarize the issues we discussed and
review the various design options.
I’ll begin with a summary of what handles are and how they are used in
current APIs; if this is familiar to you (coughDave Hermancough)
you may want to skip ahead to the section “Subtle points”.
As some of you may recall, the fate of function types in Rust has been
somewhat uncertain. It all began when I realized that
we could not soundly permit closures to recurse, which implies
that closures must be tracked in a linear fashion. This caused
interactions with the prior plans we had for
dynamically sized types, and led to severalalternativeproposals. The most radical involved keeping only one closure
type for borrowed closures and then using macros and object types to
represent all other use cases.
Since I last wrote, we’ve made great progress with the work on the
Parallel JS and Typed Objects (nee Binary Data) implementation. In
particular, as of this morning, preliminary support for typed objects
has landed in Mozilla Nightly, although what’s currently checked in is
not fully conformant with the current version of the standard (for
this reason, support is limited to Nightly and not available in Aurora
or Beta builds).
Since the new version of PJS is going to be based on binary data, we
are going to need to have a well-optimized binary data implementation.
Nikhil Marathe has prepared an initial implementation, but
it is limited to the interpreter. I am looking now at how to integrate
binary data into the JIT. The goal is to have accesses get compiled to
very efficient generated code. In this blog post, I specifically want
to cover the plan for integrating our type inference with binary data.
Today I had the honor of giving a presentation on Rust at the
Northeastern University PL Seminar. It was a lot of fun, and I also
had a lot of good conversations afterwards with some of the professors
and PhD students there. For those who are interested, I am uploading
the slides from my talk. The talk takes the same approach
that I would like to use for the next Rust paper. It was kind of a dry
run to see if that approach would be understandable and would flow logically;
overall, I thought it worked reasonably well.
Rust currently has very strong support for concurrency in the form of
actors which exchange messages and do not share memory. However, there
are many tasks for which actors are not a good fit. The unbounded
lifetime of actors means that they cannot safely access
stack-allocated memory from another task, even if it is
immutable. Actors cannot share memory except through the relatively
clumsy (and somewhat expensive) mechanism of Arc structures (which
stands for “atomic reference count”), meaning that if there are large
data structures they can be a pain to access. Arc is also
inapplicable to data structures that transition between mutable and
immutable and back again.
As I alluded in the previous post, I have noticed an interesting
connection between memory management and data-race freedom. I want
to take a moment to elaborate on this, becaause the connection was not
obvious to me at first, but it ultimately drives a lot of the Rust
design decisions.
First, I believe that if you want to guarantee data-race freedom, and
you want to support the cheap transfer of mutable state between tasks,
then you must have a garbage-collector-free subset of your
language. To see what I mean by “cheap transfer of mutable state”,
consider something like double-buffering: you have one drawing and one
display task exchanging buffers (so there are only two buffers in
total). While the drawing task is preparing the next frame, the
display task is busy displaying the current one. At the end, they
exchange buffers. In order to prevent data races in a scenario like
this, it is vital that we be able to guarantee that when the buffers
are exchanged, neither task has any remaining references. Otherwise,
the display task would be able to read or write from the buffer that
the drawing task is currently writing on.
So Ben Blum has doing some investigation into the full
implications of the Sized bound that I proposed as part of the
dynamically sized types post. It’s clear that, if we change
nothing else, the impact of Sized will be somewhat greater than I
thought. He estimates somewhere around 40% of the files in libstd need
at least one Sized bound; the actual number may wind up being
somewhat higher.
It is not entirely clear to me if this is a problem. I imagine that
the number of Sized bounds will be highest in container and other
library code. But it is worse than I hoped. So I wanted to briefly
explore some of the alternatives, assuming that the Sized annotation
burden is too high.
I have been thinking about my previous proposal for fn types. I wanted
to offer some refinements and further thoughts.
On Thunks
I proposed a trait Task for encapsulating a function and the
parameters it needs to run. I don’t like this name because this
concept could be used in other places beyond just tasks. I was
thinking that the proper name is probably Thunk. I quote Wikipedia
for the definition of Thunk: “In computer science, a thunk (also
suspension, suspended computation or delayed computation) is a
parameterless closure created to prevent the evaluation of an
expression until forced at a later time.” (There are, admittedly,
other contrary uses for the term)
I’ve been thinking more about my proposal to split the current fn
type into fn and proc. I have come to the conclusion that
we just don’t need proc at all. I think we can get by with two types:
fn(S) -> T: closures that always reference an enclosing scope
extern "ABI" fn(S) -> t: raw function pointer, no environment
Code that uses @fn or ~fn today could be rewritten to either use a
boxed trait or to use a pair of a user-data struct and an extern fn.
We’ve been making a lot of conceptual progress with the PJS API that
has not been written down anywhere, so I want to cover some of that
work. This post focuses on the integration of parallel methods with
the binary data API. It shows how the new API approach allows users to
avoid allocation for higher efficiency.
Methods, not types
We are moving away from a ParallelArray type and into methods that
will be offered on existing array types. Current plan is to name them
things like pmap (vs normal sequential map). The defined semantics
are similar to the sequential version except that the order of
iterations is undefined, because iterations may occur in parallel (I
described the subset of JS that we expect to parallelize in
a previous post).
So, I didn’t actually mean to post that previous post, I had
intended to think more on the idea. But oh well, cat’s out of the
bag. In any case, I’ve been thinking about the “closures” vs
“procedures” idea that I jotted down there and decided to try and
elaborate on it a bit more, since I find it has a lot of appeal. In
particular I think that the current collection of closure types is
addressing too many distinct use cases and the result is confusing.
I’ve been thinking about what I wrote in my last post regarding
closures and I am beginning to change my opinion about the correct
solution. fn~ just seems so unfortunate. So, besides writing fn~,
what are the other options? I just thought I’d write down a few of
the other ideas I’ve come up with for later reference. Not saying any
of the ideas in this post are good yet.
I realized today that there is an unfortunate interaction between the
proposal for dynamically sized types and closure types. In
particular, in the case of the recurring closure, I described
the soundness issues that arise in our language when closures are able
to recurse.
My solution for this was to make the type system treat a &fn() value
the same way it treats &mut T pointers: they would be non-copyable,
and when you invoke them, that would be effectively like a “mutable
borrow”, meaning that for the duration of the call the original value
would become inaccessible. So in short the type system would guarantee
that when you call a closure, that same closure is not accessible from
any other path in the system, just as we now guarantee that when you
mutate a value, that same value is not accessible from any other path
in the system.
Recently, separate discussions with pnkfelix and graydon have prompted
me to think a bit about “dynamically sized types” once again. Those
who know Rust well know all about the sometimes annoying discrepancy
between a type like ~T (owned pointer to T) and ~[S] (owned
vector of S instances)—in particular, despite the visual
similarity, there is no type [S], so ~[S] is not an instance of
~T for any T. This design was the outcome of a lot of
back-and-forth and I think it has generally served us well, but I’ve
always had this nagging feeling that we can do better. Recently it
occurred to me how we could, though it’s not without its price.
I want to look at an interesting topic: what subset of JavaScript do
we intend to support for parallel execution, and how long will it take
to get that working? As my dear and loyal readers already know, our
current engine supports a simple subset of JavaScript but we will want
to expand it and make the result more predictable.
From my point of view, the subset below includes basically all the
JavaScript syntax that I ever use. There are two primary limitations
that I think people will encounter in practice:
Yesterday I realized that you can violate Rust’s memory safety
guarantees by using “stack closures”, meaning closures that are
allocated on the stack which have can refer to and manipulate the
local variables of the enclosing stack frame. Such closures are
ubiquitous in Rust, since every for loop makes use of them (and
virtually every higher-order function). Luckily, this hole can be
fixed with (I think) very little pain—in fact, I think fixing it
can also help us make other analyses a little less strict.
In my last post about ParallelJS, I discussed the ForkJoin()
intrinsic and showed how it was used to implement the parallel map
operation. Today I want to write about the high-level changes to
IonMonkey that are needed to support ForkJoin(). IonMonkey, of
course, is our JavaScript engine.
Parallel execution mode
To support ParallelJS, we introduce a second mode of compilation
called parallel execution mode. JavaScript compiled in this mode
produces executable code that is suitable to be run in parallel. To
accommodate this new mode, each JSScript* potentially contains
pointers to two IonScript* data structures, one for standard
sequential mode and one for parallel mode.
While working on issue #5656 I encountered an interesting
problem that I had not anticipated. The result is a neat little
extension to the region type system that increases its expressive
power. The change is completely internal to the type rules and
involves no user-visible syntax or anything like that, though there
are some (basically nonsensical) programs that will no longer compile.
Anyway I found it interesting and thought I would share.
These proposals have the same descriptive power as what I described
before, but they are backwards compatible. This is nice.
Object-oriented style name resolution
In the object-oriented, C++-like version of associated items that I
introduced before, the names of associated items and methods were
resolved relative to a type. To see what I mean by this, consider a
(slightly expanded) variant the graph example I introduced before:
I’ve been doing a lot of thinking about Rust’s trait system lately.
The current system is a bit uneven: it offers a lot of power, but the
implementation is inconsistent and incomplete, and in some cases we
haven’t thought hard enough about precisely what should be allowed and
what should not. I’m going to write a series of posts looking at
various aspects of the trait system and trying to suss out what we
should be doing in each case. In particular I want to be sure that our
trait design is forwards compatible: that is, I expect that we will
defer final decisions about various aspects of the trait system until
after 1.0, but we should look now and try to anticipate any future
difficulties we may encounter.
One common criticism of the work on ParallelJS is that the API itself
does not guarantee parallel execution. Instead, our approach has been
to offer methods whose definition makes parallel execution possible,
but we have left it up to the engines to define the exact set of
JavaScript that will be safe for parallel execution.
Now, I definitely think it is a good idea to clearly define the subset
of JavaScript that our engine will be able to execute in parallel. As
I wrote in my preivous post, I want to do this both via
documentation and via developer tools that provide live feedback. In
some cases, I think, the rules will probably depend on type inference
or other dynamic analysis techniques that are subtle and hard to
explain, but live feedback should be helpful in detecting and
resolving those cases.
I am going to write a series of blog posts giving a tour of the
current Parallel JS implementation in SpiderMonkey. These posts are
intended to serve partly as documentation for the code. The plan is
to begin high level and work my way down to the nitty gritty details,
so here we go!
I will start my discussion at the level of the intrinsic ForkJoin()
function. As an intrinsic function, ForkJoin() is not an API
intended for use by end-users. Rather, it is available only to
self-hosted code and is intended to serve as a building block for
other APIs (ParallelArray among them).
The first version of our work on ParallelJS has just been
promoted to mozilla-central and thus will soon be appearing in a
Nightly Firefox build near you. I find this pretty exciting. In
honor of the occassion, I wanted to take a moment to step back and
look both at what has landed now, what we expect to land soon, and the
overall trajectory we are aiming for.
What is available now
Once Nightly builds are available, users will be able to run what is
essentially a “first draft” of Parallel JS. The code that will be
landing first is not really ready for general use yet. It supports a
limited set of JavaScript and there is no good feedback mechanism to
tell you whether you got parallel execution and, if not, why not.
Moreover, it is not heavily optimized, and the performance can be
uneven. Sometimes we see linear speedups and zero overhead, but in
other cases the overhead can be substantial, meaning that it takes
several cores to gain from parallelism. Nonetheless, it is pretty
exciting to see multithreaded execution landing in a JavaScript
engine. As far as I know, this is the first time that something like
this has been available (WebWorkers, with their Share Nothing, Copy
Everything architecture, do not count).
Lately, I’ve been thinking about the ParallelJS API that we want to
expose. In particular, I’ve been considering offering methods on the
normal array type for basic parallel operations. I think this opens
up some interesting doors.
Note: To give credit where credit is due, I should note that a lot
of the ideas in this post originate with other members of the Parallel
JS team (Shu-yu Guo, Dave Herman, Felix Klock). But I don’t want to
speak for them, since we seem to each have our own opinions on the
best arrangement, so I’m writing the post from the first person
singular (“I”) and not a team perspective (“we”). This does not imply
“ownership” of the ideas within.
One of the things that I’ve been working on for some time now is the
proper integration of C functions. As with virtually every other
facet of the design of Rust, we’ve been slowly moving from a model
where Rust tried to hide low-level details for you to one where Rust
offers tight control over what’s going on, with the type system
intervening only as needed to prevent segfaults or other strange
behavior. This blog post details what I consider to be the best
proposal so far; some of the finer points are a bit vague, however.
Rust features destructors and, as of this moment, they are simply not
sound with respect to many other features of the language, such as
borrowed and managed pointers. The problem is that destructors are
granted unlimited access to arbitrary data, but the type system and
runtime do not take that into account. I propose to fix this by
limiting destructors to owned types, meaning types that don’t contain
borrowed or managed pointers.
The current “for protocol” is best explained by giving an example of
how to implement it for slices:
fn each<E>(v: &[E], f: &fn(&E) -> bool) {
let mut i = 0;
let n = v.len();
while i < n {
if !f(&v[i]) {
return;
}
i += 1
}
}
As you can see, the idea is that the last parameter to the each()
method is a function of type &fn(&E) -> bool, which means that it is
given a pointer to an element in the collection and it returns true or
false. The return value indicates whether we should continue
iterating.
In a previous post I outlined some of the options for updating our
lifetime syntax. I want to revist those examples after having given
the matter more thought, and also after some discussions in the
comments and on IRC.
My newest proposal is that we use <> to designate lifetime
parameters on types and we lean on semantic analysis (the resolve
pass, more precisely) to handle the ambiguity between a lifetime name
and a type name. Before I always wanted to have the distinction
between lifetimes and types be made in the parser itself, but I think
this is untenable. This proposal has the advantage that the most
common cases are still written as they are today.
In my last post, I made the case against having a deterministic
semantics. I’ve gotten a fair amount of feedback saying that, for a
Web API, introducing nondeterminism is a very risky idea. Certainly
the arguments are strong. Therefore, I want to take a moment and make
the case for determinism.
Why determinism?
All things being equal, it’s clear that deterministic execution
semantics are preferable. They’re easier to debug and they avoid the
question of browser incompatibilities.
One of the interesting questions with respect to Parallel JS is what
the semantics ought to be if you attempt a parallel operation with a
kernel function that has side-effects. There are basically three
reasonable options:
Deterministic results where possible: The function behaves “as
if” it executed sequentially, executing the kernel from 0 to n,
just like Array.map.
Error: An exception is thrown.
Non-determinstic results: The function behaves “as if” it
executed sequentially, but the items were mapped in an unspecified
order.
The branch currently implements option 3: I believe it is
the most consistent and will yield the best performance. However,
reasonable people can differ on this point, so I want to make my case.
I’ve been thinking for a while that our lifetime notation has too many
defaults which can be more confusing than helpful. A recent spate of
e-mails on rust-dev brought this back to my mind. I’ve been wanting
to take a look at these defaults for a while, so I thought I’d write
up a quick exploration of the “syntactic space”. A warning: this is
not really an exciting post to read. I hope to have a few of those
coming up very soon. This one is mostly just a list of syntactic
options I wanted to document for future reference and to serve as a
starting point for discussion.
I mentioned in my previous post that we are using a single primitive
parallel operation to implement PJs. It turns out that I am not very
satisfied with what we currently have and I have been thinking about
some alternatives. In this post I’ll describe briefly how things
are setup, what problems I see, and then sketch out how I think we
could improve it.
How things work now: %ParallelBuildArray()
The current intrinsic is %ParallelBuildArray(length, func, args...). It
attempts to construct an array in parallel using a pool of N worker
threads. Conceptually, %ParallelBuildArray() allocates an array
result of length length and then instructs each worker thread to
invoke func(result, id, N, warmup, ...args), where:
The blog has been silent for a while. The reason is that I’ve been
hard at work on Parallel JS. It’s come a long way: in fact,
the goal is to land an initial version in the next couple weeks!
One of the very exciting developments has been that we switched to a
self-hosting implementation. Self-hosting is a very cool new
direction for the SpiderMonkey engine being introduced by
Till Schneidereit. The idea is to implement large parts of
the engine itself in JavaScript, similar to projects like
Squeak, Jikes RVM, Maxine, PyPy and numerous
others. As an example, imagine the standard JavaScript function
Array.map. In SM, this is currently implemented with approximately
80 lines of C++ code. This function must handle all sorts of
annoying conditions, such as ensuring that objects are rooted,
checking for interrupts, and using an
optimized call sequence to make it faster to invoke the JS code.
If the implementation were written in JS, however, all of these issues
would be handled automatically by the engine itself, just as they are
for any other JS function.
I’ve been thinking of a radical change we could make to the treatment
of mutability and borrowed pointers in Rust. The goal is to eliminate
all of the error messages about “aliasable, mutable data” that the
borrow checker currently issues. The idea is somewhat inspired by
writing a recent paper on Rust’s current system—writing a paper on
something never fails to get me thinking about how to improve it,
though it sometimes fails to stimulate ideas which are actually
good—and also somewhat inspired by recent conversations on IRC and
in person.
I can’t believe I’m saying this, but I’ve started to think that
Parallel JS (nee Rivertrail) should not demand pure callbacks to
functions like map() and so forth. Rather it should just accept
arbitrary functions. Previously, I thought that it was important that
ParallelArray methods should only accept functions which, at least
in a perfect world, would be safely parallelizable. But I am no
longer so sure why that is an important goal. Here is my reasoning.
I am considering whether we should add a way to borrow something but
retain uniqueness. This would address a shortcoming of the borrowing
system that has been bothering me for some time, and it would enable a
few patterns that are difficult or awkward today.
The Problem
I described the problem in this paper review I wrote, but I will
repeat it here, because it’s relevant, and perhaps people don’t read
and remember every single word that I write. In our system, a ~T
type is always owned. So if you write:
My big goal for 0.5 is to straighten out our function types (yet again). I’ve
been tossing the design for these over in my head since the summer and
I wanted to lay out my plan. This is a variation of something
that Ben Blum and I sketched out on a whiteboard.
Closure type
The closure type will be described something like so. Beware, it’s
got a lot of options. It turns out that there is a very large variety
of things one might want to use closures for, and supporting them
requires a fair number of knobs. I believe that in practice there
will be a smallish set of standard forms (I’ll describe those later).
In any case, this is the closure type in its full generality, with
annotations:
In this post I propose an extension of Rust’s purity rules. The short
version is that pure functions would be allowed to mutate data owned
by their &mut parameters. This extends the current Rust purity
rules which allow pure functions to invoke impure closures so long as
they are an argument to the function. The principle is the same: pure
functions are functions whose side-effects can be completely
determined by examining their parameters (for the more formally minded
among you, this is effectively an effect-parametric system with very
lightweight notation). The rest of the post is an elaboration and
justification of this idea.
I have started work on implementing Rivertrail, Intel’s proposal
for data parallelism in JS. I am excited about this project, it seems
like it’s going to be great fun. The initial version that we produce
is going to be focused on Intel’s specification, but I hope we
can eventually combine it with the more general stuff I’ve been doing
as part of PJs. There is an awful lot of overlap between the two,
though also a few minor differences that will need to be ironed out.
So, the condition that was supposed to ensure termination in my
previous post is most certainly wrong. The idea was to prevent
tautological impls like the following:
impl<A: Foo> A: Foo { ... }
Such impls, given a naive algorithm, would loop infinitely trying to
decide if a type T implemented Foo. You can imagine: it would
ask, “does T implement Foo? Well, if I map A to T then this
impl applies, but only if T implements Foo. Hmm. That puts me
back where I started from. Oh well, better try it again!” Obviously a
less naive algorithm could keep a stack and then fail to execute, but
it was precisely the logic of this stack that I was trying to capture
in that restriction.
I was thinking more about type classes as I walked down the street.
In my prior post I wrote that the rules I proposed resulted
in a system where traits loosely fit the following Haskell template:
class C self a ... z | self -> a ... z where ...
However, I gave two caveats. The first was that due to subtyping we
cannot say that one type precisely determines another, but only that
it puts a bound. The second was that, in any given impl, the value of
a ... z may be a type parameter which does not appear in the self
type. I think I understated the importance of this second caveat.
For example, consider the example I gave for simulating overloading:
Currently, the Rust compiler accepts all manner of trait, impl, and
bound declarations. In fact, it accepts plenty of declarations that
later phases of the compiler are not sophisticated enough to handle.
In other words, the syntax is writing checks the semantics can’t cash.
(An aside: I just love saying that phrase for some perverse reason.
I really wish however that checks, like rotary dial telephones, were
something that younger people vaguely understood but which no longer
had relevance in the modern era. The Swiss Einzahlungschein truly
opened my eyes! Anyhow.)
I have been trying to come up with a reasonable set of rules for
deciding when a pattern binding ought to be a move and when it ought
to be a copy and utterly failing. Simultaneously, pcwalton, brson,
and I kind of simultaneously arrived at an alternate design that tries
to simplify the copy/move distinction. I think that it also solves
the question of when to copy/move pattern bindings in a nice way.
Therefore, I wanted to write up this proposal.
Well, I have not done too well with my
goal of reading a research paper a day on the train (actually
my initial goal was two papers, but seeing as how I’ve failed so
spectacularly, I’ve dialed it back some). However, I’ve decided to
give it another go. I’ve bought a printer now so I can print papers
out (double-sided, no less!) at home (I had initially planned to buy
an iPad or something, but a decent printer is only $100, and paper is
still nicer to read and write notes on…you do the math). As
additional motivation, I’m working again on the paper on Rust’s new
borrowed pointers and so I have to catch up on a lot of related work.
We need to clarify our story on rvalue lifetimes. This is related to
issue #3387 and also various recent and not-so-recent
discussions on IRC.
The basic question is how long an rvalue lives when the program creates
pointers into it. To understand the rough issues, first consider this
program:
let x = foo();
match x {
Some(ref y) => {...}
None => {...}
}
Here, the result of foo() is stored into a local variable. The
match clause then creates a pointer to the interior of this local
variable (i.e., into the stack) called y. But what if we eliminated
the variable x:
One of the things that is sometimes frustrating in Rust is the
inability to define a type that indicates some subset of enum
variants. For example, it is very common to have a pattern like this:
I’ve been slowly learning how type inference works in
SpiderMonkey. As I understand it, SpiderMonkey’s type inference
scheme is the brain child of one Brian “Hack-it”, coder
extraordinaire. You may have seen a recent PLDI publication
on the topic. You may, like me, have read that publication. You may,
also like me, have walked away thinking, “um, I don’t really
understand how that works.” In that case, dear reader, this blog post
is for you. Well, actually, it’s for me, to try and document what I
gleaned from a conversation or two. It it is almost certainly not
entirely accurate and it may or may not be helpful.
I have been working on a change to the definition of mutability in
Rust. This is a much smaller change than my previousthoughtexperiments, which were aimed at achieving better
parameterization (those are still percolating; I think the best
approach is a modified version of the latest proposal where not
all types have mutability but type parameters do…but that’s a
problem for another day with many complications). The goal of these
changes is to enable operations like “freeze” and “thaw”.
Here is my latest stab at a tutorial on borrowed pointers. I know, I
know, enough with the borrowed pointer tutorials already! Hopefully
this will be my last post in this vein for a while. I am much happier
with this version. It is still too long to serve as a chapter in the
general Rust tutorial, but I think it’s more approachable than the
previous attempt, which was more of a reference document. As always,
feedback welcome! I have tried to incorporate what people wrote in the
comments into this version.
One thing I didn’t make clear regarding my last post: I am not
especially satisfied with the way the “tutorial” was turning out. I
use scare quotes here because I think it resembles a reference manual
more than a tutorial. Nonetheless I think there are some sections
that are quite good; and a document like it probably ought to
exist. So I figured I’d post it anyhow and I can mine it for material
later. I intend to start afresh however with something that dives at
the core ideas, which I think are relatively simple.
Here is a (much) more complete draft of the tutorial on borrowed
pointers. It is becoming more in-depth than I intended. I hope to
later extract a much shorter subset. But I thought I’d post what I’ve
got so far.
Borrowed pointers
Borrowed pointers are one of the more flexible and powerful tools
available in Rust. A borrowed pointer can be used to point anywhere:
into the shared and exchange heaps, into the stack, and even into the
interior of another data structure. With regard to flexibility, it is
comparable to a C pointer or C++ reference. However, unlike C and
C++, the Rust compiler includes special checks that ensure that
borrowed pointers are being used safely. Another advantage of
borrowed pointers is that they are invisible to the garbage collector,
so working with borrowed pointers helps keep things efficient.
I had a very interesting discussion with Sriram and Terrence (of
Kilim and ANTLR fame, respectively—two smart
dudes) yesterday. One of the things we talked about was adapting
shared-memory data structures like concurrent hash maps into
an actor setting.
One thing we’ve found when working on Servo is that the temptation to
cheat is enormous. Most of the papers you read about things like
parallel layout just assume a shared memory setting and blithely make
use of data strutures like concurrent hash maps. There is nothing
wrong with such data structures, but if we can avoid shared, mutable
memory it will go a long way towards avoiding bugs I think—as well
as keeping things secure. Even if the bug is mostly correct, data
races and similar subtle errors can open holes for exploitation.
I am trying to mop up some of the remaining work for regions now. One
of the big remaining areas is dealing with function and iface types.
This proposal is certainly influenced by my previous proposals.
However, we have backed away from the idea of dynamically-sized types
for vectors and so I will do the same here.
The design
My current design includes the following kinds of function types
(written as I expect them to commonly be written; some details are
omitted):
This is a draft of (the first section of) a new Rust tutorial on
borrowed pointers (the official name for “regions”). Comments
welcome.
UPDATE: I added a section “Why borrowed?”
Borrowed pointers
Borrowed pointers are one of the more flexible and powerful tools
available in Rust. A borrowed pointer can be used to point anywhere:
into the shared and exchange heaps, into the stack, and even into the
interior of another data structure. With regard to flexibility, it is
comparable to a C pointer or C++ reference. However, unlike C and
C++, the Rust compiler includes special checks that ensure that
borrowed pointers are being used safely. We have done our best to
ensure that these checks mostly occur behind the scenes; but to get
the most out of the system, you will have to understand a bit about
how the compiler reasons about your program.
Last Thursday and Friday I had the good fortune of presenting a paper
of mine at HotPar 2012. The paper is called
Parallel Closures: A New Twist on an Old Idea; it basically
describes what has evolved to become the PJs (Parallel JavaScript)
model, though it does so in the context of a static checker built in
Java.
I really enjoyed the workshop: the presenters were generally very good
and the audience was lively. I also appreciate that they make an
effort to encourage good conversation; for example, at lunchtime the
tables are labeled with topics for discussion (“memory models”, say,
or, “schedulers”). It’s always hard for young folk like myself to get
connected with the older, more knowledgable people in the audience, so
everything helps. (“Hi Hans Boehm, love your work on C++ memory
models”)
This is a proposal to unify the mechanics of alt and destructuring
assignment. It was born out of discussion between erickt, pcwalton,
and I amidst various bugs in the bug tracker but I wanted to float it
around to a larger audience. I’d like to discuss this on Tuesday,
because one of the logical next steps for the regions work is to begin
deciding precisely what to do about the types of identifiers in alts.
OK, I’ve been thinking more about the mutability issue and I think
I have found a formulation that I am happy with. The basic idea is
that we refactor types like so:
T = M T
| X
| @T
| ~T
| [T]
| {(f:T)*}
| int
| uint
| ...
M = mut | const | imm
This no doubt looks similar to some of my other permutations. The key
difference is that before I separated qualified and unqualified types.
This was intended to aid with inference, but in fact it was getting me
into trouble. I realize now there is a different way to solve the
inference problem. But first let me back and explain what inference
problem I am concerned about.
T = Q U
Q = mut | const | imm
U = [T]
| @T
| &T
| { (f : T)* }
| X
| int
| uint
| ...
The interesting case is that of a type variable, denoted as X. I
grouped type variables under the heading of “unqualified types”. But
this is of course incorrect, they are not unqualified types. They can
map to a qualified type (in fact, that’s the whole point of this
exercise). So really the hierarchy ought to be:
Currently, Rust has an effect system but refuses to admit it. In an
effort to broaden the set of things that can be safely done in the
face of extant aliases into the heap, I have been experimenting with a
lightweight extension to Rust’s system. So far I think it is
promising but also no magic bullet.
Background
For those who aren’t familiar with the term, an “effect system” is
basically just a fancy name for tagging functions with some extra
information beyond the types of their arguments and their return type.
I am dissatisfied with how mutability is treated in the Rust type
system. The current system is that a type is not prefixed mutable;
rather, lvalues are. That is, a type T is defined like so:
T = [M T]
| @ M T
| & M T
| { (M f : T)* }
| int
| uint
| ...
M = mut | const | (imm)
Note that there is no type mut int (a mutable integer). This is
logical enough; such a type has little inherent meaning: an integer is
a value, it is not mutable or immutable.
I wanted to bring together the various ideas around vectors and
function types into one post. The goals of these changes are
to achieve orthogonality of the pointer types, so that leading &,
@, and ~ sigils are the only way to indicate the kind of
pointer that is in use;
to help pare down on the proliferation of subtle variantions on
types, such as the 5 different function types currently available.
The proposal
The Rust type system would be described by the following grammar. In
this grammar, I have included all optional portions except for region
bounds. I indicated those types which could have a lifetime bound
associated with them by writing (/&r) in the description (a lifetime
bound indicates the lifetime of any pointers embedded within the type
itself; this is not related to the changes I am discussing here so I
won’t go into detail):
Yesterday I wrote about my scheme for paring down our set of function
types to one type, fn:kind(S) -> T. When I finished writing the
post, I was feeling somewhat uncertain about the merits of the idea,
but I’m feeling somewhat better about it today. I really like the
idea that top-level items have the type fn:kind(S) -> T and that you
therefore give them an explicit sigil to use them in an expression;
this allows us to remove the “bare function” type altogether without
any complex hacks in the inference scheme.
As you loyal readers know, I am on a quest to make the Rust type
system more orthogonal with respect to the kind of pointer in use,
by which I mean that I want to have the three pointer sigils (@,
&, and ~) indicate where memory is located and the other types
indicate what value is to be found at that memory. Right now there
are a few cases where we conflate the two things into one type. The
first, vectors and slices, I discused in a recent post. This post
discusses the second case: function and interface types.
I implemented a simple, non-flow-sensitive version of the reference
checker which I described in my previous post. Of course it
does not accept the Rust codebase; however, the lack of
flow-sensitivity is not the problem, but rather our extensive use of
unique vectors. I thought I’d write a post first showing the problem
that you run into and then the various options for solving it.
Errors
The single most common error involves vec:len(). There are many
variations, but mostly it boils down to code code like this, taken
from the io package:
I’ve been working for the last few days on the proper safety
conditions for borrowing. I am coming into a situation where I am not
sure what would be the best approach. The question boils down to how
coarse-grained and approximate our algorithm ought to be: in
particular, ought it to be flow sensitive? But let me back up a bit, first,
and provide a bit of background.
Background
Rust bucks the “new language” trend by not having a purely
garbage-collected model. We feature things like interior and unique
types which can be eagerly overwritten. This means that we have to be
very careful when we create temporary references to those kinds of
values that these references remain valid.
I’m still thinking about vector and string types in Rust and I think
I’ve decided what I feel is the best approach. I thought I’d
summarize it here and make the case for it. If you don’t know what
I’m talking about, see this post for more background. I’ll
forward this to the mailing list as well; I’m sorry if it seems like
I’m harping on this issue. I just think vectors and strings are kind
of central data structures so we want them to be as nice as possible,
both in terms of what you can do with them and in terms of the
notations we use to work with them.
I’ve been making a point of reading academic papers on the train as I
ride home. It’s so easy to get behind with the sheer quantity of work
that is being produced. Anyway, it occurred to me that I ought to try
and summarize the papers I read on this blog so that I can I remember
my reactions to them.
I’ll start with “Permission Regions for Race-Free Parallelism”, by
Westbrook, Zhao, Budimilic, and Sarkar. The basic idea builds off of
Habanero Java, which is a kind of fork of the X10 language that Sarkar
and his group work on. The basic idea of the paper is to add a
language construct permit which looks like:
I want to do an introduction to the regions system I’ve been working
on. This is work-in-progress, so some of the details are likely to
change. Also, I’m going to try some new terminology on for size:
although it has a long history in the literature, I think the term
“region” is not particularly accurate, so I am going to use the term
“lifetime” or “pointer lifetime” and see how it fits.
After my recent dalliance in
Matters of a Truly Trivial Nature, I’d like to return to
Matters Most Deep and Profound. I’m running up against an interesting
question with regions that has to do with the nature of function types
like fn(&int): up until now, I’ve assumed that this refers to a
function that takes an integer pointer in some region that is
specified by the caller. That is, it is a kind of shorthand for a
type that might be written like fn<r>(&r.int), where the <r>
indicates that the function type is parameterized by the region r.
We’ve been discussing a lot about how to manage vectors and strings in
Rust. Graydon sent out an excellent proposal which allows for a great
number of use cases to be elegant handled. However, I find the syntax
somewhat misleading. I’ve proposed one alternative on the mailing
list, but I now find I don’t like it, so I thought I’d brainstorm a
bit and try to find something better.
For a long time, it was considered fairly obvious, I think, that
syntax didn’t really matter. It was just the surface skin over the
underlying ideas. In recent times, though, the prevailing wisdom has
reversed, and it is now quite common to hear people talk about how
“syntax matters”.
While I don’t exactly disagree, I think that the importance of trivial
syntactic matters is generally overemphasized. It is not a matter of
life and death whether or not semicolons are required to end a line,
for example, or whether parentheses are required in making a call.
I’d like to propose a term for code that has been “over-DRY’d”
(dessicated?). I occasionally run across some method which just seems
horribly complex. Reading it closer, it usually turns out that what
happened is that two or three independent operations got collected
into one subroutine. Perhaps they started out as doing almost the
same thing—but before long, they diverged, and now the subroutine
has grown a hundred parameters and has a control-flow path that
requires a whiteboard and a ultra-super-fine-point marker to follow.
But, just as often, you can tear this routine apart into two or three
routines that read just fine, even if they share a line or two of code
in common. So I’m going to start calling such routines “DOA”, though
the acronym has a bit of a different expansion when used as an
adjective.
One of the questions in our object system is what precisely how
“declared” we want things to be when it comes to interfaces and
implementations. In a discussion on IRC, graydon suggested it’d
be nice to have terms like “duck-typing” defined more precisely in
a Rust syntax, and he is correct. So here is my effort.
The current setup
Currently, implementations must declare precisely what types they
implement. For example, it looks like this:
On the rust-dev mailing list, someone pointed out another
“BitC retrospective” post by Jonathon Shapiro concerning typeclasses.
The Rust object system provides interesting solutions to some of the
problems he raises. We also manage to combine traditional
class-oriented OOP with Haskell’s type classes in a way that feels
seamless to me. I thought I would describe the object system as I see
it in a post. However, it turns out that this will take me far too
long to fit into a single blog post, so I’m going to do a series.
This first one just describes the basics.
First off, I want to welcome Brian Anderson to the Rust blog-o-sphere
(which so far consists primarily of myself). His first post
does a great job of explaining how to use the new for syntax that
was recently added to Rust: this syntax allows for break, ret, and
cont from within user-defined loops, which is very nice.
Reading some of the Hacker News comments
(this one in particular), I wanted to clarify one thing. There
is some concern that this new syntax changes the semantics of ret
when, in fact, it aims to do precisely the opposite.
pcwalton and I (but mostly pcwalton) have been hard at work
implementing regions in Rust. We are hoping to use regions to avoid a
lot of memory allocation overhead in the compiler—the idea is to use
memory pools (a.k.a. arenas) so that we can cheaply allocate the data
needed to process a given function and then release it all in one
shot. It is well known that arenas are great fit for the memory
allocation patterns of a compiler, which tend to produce a lot of data
that lives for the duration of a pass but is not needed afterwards.
Yesterday we had a hackathon/meeting to discuss the overarching design
of Servo, the project to build a next-generation rendering engine. We
didn’t ultimately do much hacking (though we did a little), but mostly
we tried to hammer out the big picture so that we can actually get to
writing code. I wanted to try and write up what I understood as the
consensus (for the moment, anyway).
The big picture
There will be (at least) three large components. Each is basically
operating in independent tasks and the various stages are therefore
largely isolated from one another and able to execute independently
(with certain exceptions, as we shall see):
Cross-crate inlining has come a long way and is now basically
functional (I have yet to write a comprehensive test suite, so I’m
sure it will fail when exercising various corners of the language).
Just for fun, I did some preliminary micro-benchmarks. The results
are not that surprising: removing method call overhead makes programs
run faster! But it’s still nice to see things go faster. We’ll look
at the benchmarks, see the results, and then dive into the generated
assembly. In all cases, I found LLVM doing optimizations that rather
surprised me.
My current implementation of the auto-serialization code generator
requires full type information. This is a drag. First, macros and
syntax extension currently run before the type checker, so requiring
full type information prevents the auto-serialization code from being
implemented in the compiler, as it should be. At first I wanted to
change how the compiler works to provide type information, but after
numerous discussions with pcwalton and dherman, I’ve come to the
conclusion that this is a bad idea: it requires exposing an API for
the AST and for type information and introduces numerous other
complications.
In the last few posts I’ve been discussing various options for
regions. I’ve come to see region support as a kind of continuum,
where the current system of reference modes lies at one end and a
full-blown region system with explicit parameterized types and
user-defined memory pools lies at the other. In between there are
various options. To better explore these tradeoffs, I wrote up a
document that
outlines various possible schemes and also details use cases that are enabled by these schemes.
I don’t claim this to be a comprehensive list of all possible schemes,
just the ones I’ve thought about so far. In some cases, the
descriptions are quite hand-wavy. I also think some of them don’t
hang together so well.
Marijn pointed out to me that our current setup should avoid the worst
of the versioning problems I was afraid of. In the snapshot, we
package up a copy of the compiler along with its associated libraries,
and use this compiler to produce the new compiler. The new compiler
can then compilers its own target libraries, thus avoiding the need to
interact with libraries produced by the snapshot.
Of course, I should have known this, since I have relied on this so
that I can changed the metadata format without worrying about
backwards compatibility. That’s what I get for writing blog posts
late at night.
I’ve been busily implementing the Cross-Crate Inlining stuff, but one
area I haven’t looked at much is versioning. In particular, if we are
going to be serializing the AST, we need a plan for what to do when
the AST changes. Actually, if inlining were only to be used for
performance, we wouldn’t really need to have a plan: we could just
not inline when the AST appeared to be stored in some form we don’t
understand. However, if we fully monomorphize, we will not have that
luxury: without type descriptors, the only way to compile cross-crate,
generic calls will be by inlining.
One commonly requested feature for regions is the ability to return
references to the inside of structures. I did not allow that in the
proposal in my previous post because I did not want to have any
region annotations beyond a simple &. I think, however, that if you
want to allow returning references to the interior of a parameter, you
need a way for the user to denote region names explicitly.
I was talking to brson today about the possibility of moving Rust to a
regions system. He pointed out that the complexity costs may be high.
I was trying to make a slimmer version where explicit region names
were never required. This is what I came up with. The truth is, it’s
not that different from the original: adding back region names wouldn’t
change much. But I’m posting it anyway because it includes a description
of how to handle regions in types and I think it’s the most complete and
correct proposal at the moment.
Brian pointed out to me a nice solution to the Task API problem that I
have overlooked, though it’s fairly obvious. Basically, I had
rejected a “builder” style API for tasks because there is often a need
for the child task to be able to send some data back to its parent
after it has been spawned, and a builder API cannot easily accommodate
this. Brian’s idea was to encapsulate these using futures. It’s
still not perfect but it’s better I think and more composable than my
first, limited proposal. It still requires that the actor pattern be
a separate module.
One of the thorny API problems I’ve been thinking about lately is the
task API for Rust. I originally had in mind this fancy and very
flexible aproach based on bind. When I spelled it out I found it was
very powerful and flexible but also completely unworkable in practice.
So here is a more limited proposal. There is a core task API that
looks something like this:
enum task = uint; // wrap the task ID or whatever
type opts = { ... };
fn default_opts() -> opts;
fn spawn(opts: opts, body: fn~()) -> task;
The options struct will let you control simple things like stack size
and so forth.
I’ve been working on implementing Cross-Crate Inlining. The
major task here is to serialize the AST. This is conceptually trivial
but in practice a major pain. It’s an interesting fact that the more
tightly you type your data, the more of a pain it (generally) is to
work with in a generic fashion. Of functional-ish languages that I’ve
used, Scala actually makes things relatively easy by using a
combination of reflection and dynamic typing (interfaces like
Product come to mind).
One of the things I’d like to do for the iteration library is settle
on a convention for breaking and continuing within loops. There is a
bug on this issue (#1619) and it seems like the general
approach is clear but some of the particulars are less so. So I
thought I’d try to enumerate how code will look under the various
alternatives and then maybe we can settle on one: they’re all fairly
similar. Who knows, maybe just writing things out will settle my
mind.
Cross-crate inlining (CCI) refers to the ability to inline a function
across crate boundaries. In Rust, a “crate” is the unit of
compilation, rather than an individual file as in C or C++. A crate
basically corresponds to a single library or executable, but it may
contain any number of modules and source files internally. CCI is
important for performance due to the ubiquitous use of small methods
like vec::iter() in our source code. Such methods have proven to be
a very scalable way to define iteration abstracts, but performance is
currently somewhat lacking.
It’s been a while since I wrote anything on the blog! A lot has been
going on in the meantime, both in Rust, parallel JavaScript, and
personally…I hate to write a big update post but I gotta’ catch up
somehow!
Rust
First, we made our 0.1 release, which is great. We are now planning
for 0.2. The goal is to make frequent, relatively regular releases.
We’re still in such an early phase that it doesn’t seem to make sense
to literally release every few months, but at the same time we don’t
plan to wait long.
In one of the comments on yesterday’s post,
Tushar Pokle asked why I would champion my model over an
Erlang model of strict data separation. There are several answers to
this question. The simplest answer is that Web Workers already
provide an actors model, though they do not make tasks particularly
cheap (it’s possible to work around this by creating a fixed number of
workers and sending tasks for them to execute).
Lately the ideas for a parallel, shared memory JavaScript have begun
to take shape. I’ve been discussing with variousJavaScriptluminaries and it seems like a
design is starting to emerge. This post serves as a documentation of
the basic ideas; I’m sure the details will change as we go along.
User Model
The model is that a JavaScript worker (the “parent”) may spawn a
number of child tasks (the “children”). The parent is suspended while
the children execute, meaning that it will not process events or take
other actions. Once the children have completed the parent will be
re-awoken.
UPDATE: I found some more complications. Updates inline.
I have been working on and off on allowing block sugar to appear in
Rust expressions and not only statements. For those who do not know
what I am talking about, let me give a bit of context. At the moment,
one can write the following in Rust:
vec::iter(v) { |e|
...
}
which is sugar for the function call:
vec::iter(v, { |e|
...
})
Objectively, there isn’t much difference between the two, but somehow
pulling the {||} out of the parentheses feels much lighter to me.
The original Rust design included iterators very similar to Python’s
generators. As I understand it, these were stripped out in favor of
Ruby-esque blocks, partially because nobody could agree on the best
way to implement iterators. I like blocks, but it seems like it’s
more natural to compose iterators, so I wanted to think a bit about
how one might use blocks to achieve similar things. I’m sure this is
nothing new; there must be hundreds of libraries in Haskell that do
the same things I’m talking about here.
So, I worry that my various posts about Rust give the impression that
I’m dissatisfied with the language. It’s true that there are several
things I’d like to change—and those are what I’ve focused on—but I
want to clarify that I quite like Rust the way it is and I find the
overall feel of the language to be very good. When it comes to the
big decisions, I think Rust gets it right:
In the context of thinking about parallelism for Rust, I have been reminded
of an older idea I had for a lightweight, predictable dynamic race
detection monitoring system based around block-scoped parallelism. I should
think this would be suitable for (an extended version of) a dynamic
language like Python, JavaScript, or Lua. I will write in a Python-like
syntax since I know it best, but I am debating about exploring this
for JavaScript.
I landed a preliminary version of unique closures (which I am currently calling
sendable fns) on the trunk last night. I wanted to briefly document what I did
to alter the design of closures to get this working (of course there is a comment
in the code too, but who reads that?).
Closures in Rust are represented as two words. The first is the function pointer
and the second is a pointer to the closure, which is the captured environment that
stores the data that was closed over. Because of how Rust is implemented, the
closure must also store any type descriptors that were in scope at the point where
the closure was created.
I keep thinking about parallel blocks although I know I probably
shouldn’t; but so long as I write these notes while rustc builds,
everybody wins, right?
Anyhow, pcwalton and dherman yesterday pointed out to me
that const is not exactly one of the most beloved features of C++:
“const-ification” is no fun, and if we’re not careful, Rust could walk
right down that path. To some extent my reaction is, “Well,
something’s gotta’ give.” You can’t have modular static race freedom
without some way to know what function will write what. But
nonetheless they are quite correct.
On a call with other Rust developers, I realized that I was thinking about
unique closures all wrong. I had in mind a total ordering:
fn[send] <: fn <: block
but of course this is not necessary. What is desirable is a partial ordering:
fn[send] <: block
fn <: block
just as ~ and @ pointers can both be aliased using a reference.
Ironically, this is precisely what I proposed in my list of possible
solutions, but I did so using region terminology. Embarrassingly
obvious, in retrospect, particularly as that was Graydon’s original
design I believe. I think I got confused by the total ordering of
kinds into thinking that this should translate to a total ordering of
functions that close over data in those kinds. Anyhow, I will now
work on implementing unique closures in this partially ordered way,
and hopefully things will go more smoothly!
I have been trying to implement unique closures—or sendable
functions, as I prefer to call them—but I realized that there is
a fundamental problem that I hadn’t thought of before. The problem
stems from two contradictory design goals:
Sendable functions should be movable to another task without copying
The various function types should have a subtyping relationship
The first requirement really demands that the sendable function’s
environment be stored with a unique pointer. Otherwise multiple
threads could share access to the same mutable state. Uncool.
I’ve been thinking a lot about “parallel blocks” recently and I am
beginning to think they can be made to work very simply. The main
thing that is needed is a type qualifier const that means
“read-only”. This would be a type prefix with very low precedence,
just like immutable and shared in D. The type const T
would refer to an instance of T that cannot be modified. This is a
deep property, so, given some record types defined like:
One of the last remaining tasks for Rust 0.1 is to find a way to
address issues #1128 and #1038. The key problem is
that, right now, we can only spawn a task with a bare function,
which is to say a function that carries no closure or environment.
Due to the way that Rust is implemented, this even excludes generic
functions. I have been wanting to lift this restriction but have
been stymied by trying to make it accessible.
Marijn asked me what it is that I dislike about parameter
modes. I thought I might as well explain here.
For background, today in Rust a function can declare each parameter in
one of several modes:
By value (++): No pointer is used but the value is not owned by the
callee. Therefore, the callee does not need to free it, for example, or
decrement a ref count.
By immutable reference (&&): a pointer to the variable in the caller’s
stack frame is passed, but the callee cannot use it to make changes.
Can be passed an lvalue or an rvalue.
By mutable reference (&): a pointer to the variable in the caller’s
stack frame is passed, and the callee can use it to reassign the variable.
Can only be passed an lvalue.
By copy (+): A fresh copy of the value is created and the callee must
dispose of it.
By move (-): The value is moved from the caller’s stack frame and the
callee must dispose of it.
I recently implemented a new hashtable module for Rust. This was
actually the first piece of code written in Rust which I started from
scratch. I ran into some difficulties. Some of these are just things
you have to get used to; some might be worth trying to correct. Just
not entirely sure which problem falls into which category yet.
Cyclic types
Due to Rust’s structural types, types cannot reference themselves.
In the hashtable module I was defining, I wanted a linked list of entries.
In C I would write something like:
I have been thinking about unique closures, one of the last blocker items
for the Rust 0.1 release. The original idea of a unique closure was
that it should be a closure that can only access uniquely owned state,
and which can therefore be sent to other tasks. However, I’ve come to
have my doubts about this idea. In particular, a unique closure, like
any closure, is a function, and can therefore be invoked many times:
but this imposes some limits on what such a closure can safely do.
The primary means of parallel programming in Rust is tasks. Our task
support is good: as good or better than any other language I’ve seen
(good support for unique types and unique closures) but we have
virtually no support for intra-task parallelism. The classic example
is iterating over an array and processing each element in parallel.
To be fair, this is a hard problem.
For my PhD, I worked on a language called
Harmonic. Harmonic had a lot of ideas
which I—naturally enough—really like, but most of them are
probably not appropriate for Rust, as they leaned heavily on a
complex, dependent type system. Some of them, however, might apply.
In fact, thanks to unique pointers and interior types, it might be
possible to make the Rust version even more expressive than the
original.
I just posted a draft of a proposal for Rust that aims to eliminate
implicit copies. At the moment, it is not the
final version; there are some flaws I need to correct. For one thing,
I need to address implicit capturing of variables by lambdas.
From the introduction:
This is a proposal for Rust whose purpose is to eliminate implicit
copies of aggregate types, while preserving most other aspects of
the language. Secondary goals include:
One of the better features from functional programming languages are
variant types (a.k.a. algebraic data types). Basically they are a way
of enumerating a small set of possibilities and then making sure that
you handle every possible case. However, in real world use variant
types tend to run into a few annoying problems. While working on the
Harmonic compiler, I found that
Scala’s case classes addressed some of these shortcomings.
My goal in writing Scala code was to never have an assert false to
cover situations I knew could not occur. I did not quite succeed, but
I got really close, much closer than I ever got in any other language.
Mostly where I failed I knew that I could refactor the types but I did
not want to spend the time to do it. In this post I want to explain
how and why the case class approach seems to work better than
traditional variant types. In later posts I’ll cover some of the
other tricks that I ended up using, particularly the approach I used
to having an AST whose shape changed over time.
Wow! The response to the last Rustc Reading Club was overwhelming – literally! We maxed out the number of potential zoom attendees and I couldn’t even join the call! It’s clear that there’s a lot of demand here, which is great. We’ve decided to take another stab at running the Rustc Reading Club, but we’re going to try it a bit differently this time. We’re going to start by selecting a smaller group to do it a few times and see how it goes, and then decide how to scale up.
The next