Cessen's Ramblings

2018 - 10 - 11

Optimal Anki Settings - Take 2

I discovered a significant error in the simulations I ran in the previous post. So I'd like to present the corrected simulations, and also explain more deeply the assumptions and known weaknesses even in these fixed simulations. I also made the charts easier to use. However, I won't re-explain the background, so please see the previous post for that.

Summary for those who don't want to read the whole thing: Using an interval modifier near 140% is probably a reasonable default for language learning with Anki.

The Error(s)

The major mistake I made was failing to account for the additional reviews that you do after lapsing on a card. This, unsurprisingly, has a huge impact on the results. In fact, the amount of extra time you spend on a lapse has a far more significant impact than the initial time you spend creating and learning a new card.

I also had an error that Matt vs Japan caught in his own work, which is that calculating the total number of cards you know isn't quite as simple as just counting the cards in your deck. There's a forumla that the Supermemo people derived to get a correct approximation of known cards at any given point in time, and I am now using that. This also turns out to have a significant impact on the results.

Assumptions and Limitations

Before I jump into the updated results, I want to lay out more explicitly what the assumptions of my simulations are, as well as some limitations you should be aware of.

Anki Settings

I'll actually cover a few variations in the results, but the simulation's assumption about Anki settings is that you are pretty much using the defaults (aside from the Interval Modifier, of course).

The only exception is the lapse "new interval" setting, which determines how much a card's interval is reduced when you lapse on it. The Anki default is to reset the interval to zero, but that seems like a flagrantly bad choice to me: if you already successfully remembered it just one interval prior, there's no reason to drop you back to square one. Matt vs Japan uses 75% as his default, but that also seems not optimal. If your interval grows by, for example, 10x every time you get it right, then dropping it to 75% seems like it doesn't go far enough. Alternatively, if you're only growing by 1.1x, then 75% drops you back too far.

What I've come up with—and this is totally just hand-waving "this seems reasonable" territory—is to configure it so that lapsing twice will reverse a single success. So, for example, if getting a card right increases the interval 4x, then lapsing will halve the interval, because halving something twice results in 1/4 the size, reversing the 4x. For the curious, the way to calculate that is 1.0 / sqrt(N), where N is how much your interval is multiplied on success (4.0 in the example I just gave).

Other than that, it's all Anki defaults (except where noted in the variations). My reasoning for this is that it's not clear what effect all of the settings have on retention etc., so sticking to the defaults gives us some reasonable confidence that the formulas for the interval modifier will apply with some accuracy.

The Simulated User

There are two properties of the simulated user that affect the simulation:

  • How much time they spend creating and learning a new card.
  • How much time they spend per review.

However, for the graphs the only thing that actually matters is the ratio between these two items. Knowing their absolute magnitudes is unnecessary for calculating optimal efficiency, although it may still be interesting.

The assumption I've used is that creating a new card and doing the initial reviews to learn it take a combined time of 120 seconds (or two minutes). And each individual review after that takes 20 seconds. These seem like reasonable numbers to me. Moreover, except at bizarre extremes (e.g. new cards take almost no time) it doesn't appear that the ratio impacted the graphs significantly.

Length of the Simulation

For all the simulations in this post, I use a simulated time of 365 days (one year). Doing it for fewer days or more days does impact the results some, but basing things on one year seems reasonable to me. If your cards grow older than a year, it's not totally clear to me how much you're really getting out of them—at least in the context of language learning. And studying for significantly less than a year doesn't make sense for seriously learning a language.

General Limitations

I alluded to this earlier, but there are things that these simulations don't account for. The biggest one is that it's not at all clear how the following variables impact retention of cards:

  • Increasing/decreasing the number of reviews for initial learning of new cards.
  • Increasing/decreasing the number of additional reviews after lapses.
  • Increasing/decreasing the lapse "new interval" setting.

I mean, it's pretty clear that increasing the first two and decreasing the latter will improve retention, but it's not at all clear how to quantify that, or create formulas for them (or at least I don't know). This is also a limitation of Matt vs Japan's work, and the simulator that he's now using.

Because of that, the retention impacts of these factors are completely unaccounted for in these simulations. This means that the particular choice used for one of these factors, even though it affects the simulation, does not affect it accurately. For example, you could set the lapse new-interval setting to 100000% (1000x multiplier), so that whenever you lapse a card it will launch its interval into the stratosphere. From the simulation's perspective, that's a massive increase in efficiency, because it assumes the retention rate for that card still stays the same. But that's obviously false in reality—in reality that setting would be roughly equivalent to deleting all lapsed cards from your deck.

That's why I'm generally trying to stick to Anki's defaults, or at least "reasonable" settings. And that also means that all results from not only my simulations, but also from Matt or anybody else, should be treated as fuzzy guides, not as an exact science. There's a lot that we're not accounting for, and it's not totally clear how that affects the simulations.

Results

So with that out of the way, here are the results. As in the last post, here's how to interpret the graphs:

  • The vertical axis is the Interval Modifier setting in Anki.
  • The horizontal axis is your personal retention rate at (roughly) default Anki settings.
  • The white strip is 99+% optimal efficiency.
  • The light gray strip is 95+% optimal efficiency.

"Efficiency" in this case means "cards learned per hour of time studying". And studying includes both reviews and time spent creating and learning new cards.

All of these graphs are normalized, and therefore don't reflect efficiency differences between different graphs. They are also normalized individually within each vertical slice of pixels, and therefore also don't reflect differences in efficiency between personal retention rates. The latter is intentional, as it makes it easy to e.g. find your personal retention rate and determine what interval modifier results in your personal optimal efficiency. See the previous post for more details.

The "Reasonable" Graph

I actually have quite a few graphs this time, but this is the one I think most people should use:

It represents precisely the settings and simulated user I described earlier, and I think is a reasonable graph to work from. In particular, it assumes:

  • An average of two minutes spent creating + learning each new card.
  • An average of 20 seconds per review.
  • A single additional review per lapse.
  • A "max lapse" setting of 8 (cards are suspended when they exceed 8 lapses).
  • The 1.0 / sqrt(N) lapse new-interval setting I described earlier.

The main take-away is that an interval modifier of around 140% gets you 95% efficiency in almost the entire 75%-95% personal retention rate range. Which is pretty great! So I think this is probably a good default. But going up as high as 160% also seems quite reasonable. And if you have especially good retention, even as high as 200% might make sense.

But now that we have the "reasonable" graph out of the way, let's start playing with the settings!

Varying Max Lapses

The above graph changes the "max lapse" setting to 4.

And this one changes the "max lapse" setting to 12.

These graphs together illustrate something useful: increasing your max lapse setting beyond 8 doesn't make much difference in efficiency, but lower numbers definitely do!

It's also worth noting (although not illustrated in the normalized graphs) that decreasing your max lapse setting has a negative impact on your efficiency. In general, with a max lapse setting of 8 or higher, you're at 99+% of optimal efficiency, but a setting of zero slashes your efficiency by a factor of 2, depending on your personal retention rate. A setting of 4 gives you 95+% max efficiency.

As far as the simulation is concerned, increasing your max lapse setting always improves efficiency. I think this makes sense. Although it's not in these simulations, I also did some sims where each card had a slightly different difficulty (i.e. individual retention rate), and the variance between cards had to get pretty huge before it wasn't always beneficial to increase max lapses.

So my takeaway is this: beyond a max lapse setting of 8 doesn't really make a difference to efficiency. But feel free to max out the setting if it makes you feel better.

However, there is also a psychological component to studying, and culling out cards you're having a tough time with might make sense. In that case, a setting of 4 probably makes sense, since it's pretty low but still has a minimal impact on efficiency.

Varying Additional Lapse Reviews

This should be taken with a significant grain of salt, as per the "limitations" section earlier in the article. But here are a few graphs varying how many additional reviews are done when you lapse. The "reasonable graph" earlier is "one additional review".

^ Zero additional reviews.

^ Two additional reviews.

^ Three additional reviews.

The differences are pretty stark. Especially at zero additional reviews, it seems like you can get great efficiency with much larger interval modifiers! However, that seems very suspect to me, because the simulation isn't accounting for how these additional reviews may improve retention.

Having said that, because we don't know exactly how much those additional reviews help, any of these graphs are potentially as valid as the "reasonable" one that I presented. This is an open area for additional work. If anyone has any data about the impact of additional lapse reviews, I would be very interested to see it!

Varying Lapse New Interval

Finally, let's see how varying the lapse "new interval" setting (how much intervals are reduced on lapses) impacts things.

^ Matt's 75% setting.

^ Anki's default 0% setting.

As you can see, this setting also has a notable impact. But just like varying the additional lapse reviews, I have no idea how this impacts things in reality—this simulation doesn't account for important (but currently unknown) factors. So this also needs further study!

Wrap-up

I'm reasonably confident in the results from these simulations, except for the limitations and unknown factors I've described throughout this post. If anyone has any insight around those issues, I would be very interested to hear from you!

But for now, I think it is at least reasonable to go with an interval modifier of %140. If you want to use my lapse "new interval" setting scheme, that corresponds to a lapse "new interval" settings of 53%.

2018 - 10 - 04

Optimal Anki Settings

UPDATE: I found a major problem with the below simulations, which I have fixed in a follow up post. Please DO NOT use the graphs or settings advice below. Instead see the follow up post.

(If you're already familiar with all this background and just want to see the results, skip down to the Results section.)

Something I haven't talked about on this blog yet is that I'm learning Japanese.

One of the tools that many people use to aid in learning a new language is an application called Anki. It's a flashcard application with automated review scheduling based on spaced repetition. You can make flash cards in Anki for just about anything, but in the context of language learning it's really useful for moving vocabulary, grammar, etc. into your long-term memory. As I understand it, it shouldn't be your primary study method, but it can accelerate your language learning when used as a supplement to e.g. immersion/input-based approaches.

One of the resources I've found useful in figuring out how to even approach learning Japanese has been the YouTube channel Matt vs Japan, and he recently posted a video suggesting a different approach to configuring Anki's spaced repetition settings.

Traditionally, you try to maximize your retention rates (i.e. minimize forgotten cards), balanced with how much time you're willing to spend studying. But in this video Matt presents a remarkable insight: what matters isn't your retention rate, what matters is how many total cards you memorize per unit of time you spend studying. And it turns out, you can memorize more total cards if you're also willing to forget more cards (up to a point) by making your intervals larger.

Matt took a crack at calculating what those optimal intervals would be. It depends on a variety of factors, so he came up with a formula that people can use to determine their own optimal intervals. However, in the video he also encouraged people to not take it on faith, and instead make sure the math actually works themselves. He is (like all of us) human, after all.

So I decided to take a different approach to solving the same problem: simulation. Partly this is to verify Matt's work, and partly this is because we can take more variables into account with simulation, and potentially get more accurate optimal intervals.

Results

The code for the simulation can be found on my github. I haven't put a license on the repo yet, but please consider it as public domain. I encourage anyone who wants to to mess around with it or build on it to do so. It's written in Rust, so it should be pretty painless to build and run for anyone with basic command line experience.

My simulation makes most of the same assumptions as in Matt's video, so if the base assumptions are wrong, my simulation is wrong as well. So take this with a certain amount of salt.

The fun part of my simulation is that it produces visualizations. For example, this:

The vertical axis is the card's interval factor, and the horizontal axis is the user's retention rate with default Anki settings. The brightness of the pixels represents how many cards you memorize per hour of study: brighter is more, darker is fewer.

(A side-note about "Interval Factor": this is literally the number that your card's interval is multiplied by when you answer "good" on a card. I find this easier to reason about during simulation. Getting the Interval Modifier setting—which Matt talked in terms of—from this number is easy: just multiply by 40. For example, an Interval Factor of 7.5x = 7.5 * 40 = Interval Modifier 300%. This assumes that you leave your new card ease at 250%, as is default in Anki.)

One of the things that is immediately obvious is that having better personal retention is... better. Which shouldn't be surprising. As you get closer and closer to 100% retention, the interval factor matters less and less, and you can crank it up crazy high. In other words, if you never forget anything, you never have to review! So it is completely unsurprising that the brightest pixels in the chart are in the upper-right corner: having 100% retention is super time-efficient!

Of course, no human has 100% memory retention (that I'm aware of). And, in fact, for any given person, the only thing that matters in this graph is the vertical slice of pixels corresponding to their personal retention rate. So I have a modified chart that helps us visualize that:

This is the same graph as before, except that each vertical slice of pixels has been individually normalized so that its own brightest pixel is white. In other words, the white pixels show the curve of optimal interval factors. (Note: the right-most part of the image has a strange discontinuity—this is because the optimal factor goes off the top of the chart.)

One of the hilarious things about this image is the left side: at some point, if you're really bad at memorizing things, super long intervals start to be optimal again. However, I'm pretty sure this is a weakness in the simulation, as its assumptions start to break down. It probably doesn't actually match reality. Nevertheless, I find it funny.

In any case, the most obvious thing about this image is that the optimal intervals curve upwards as the personal retention rate increases. This, again, makes perfect sense given what we know: if you can hold cards in your memory longer, you don't need to review as often, so you can be more efficient with your time at longer intervals.

Another thing worth noting about this chart is that the falloff from white (optimal) is very smooth and gradual. This is important, because it means that the exact settings aren't delicate. You can be off by a decent bit and still be close to optimally efficient.

To drive that last point home even stronger, here is the last (and most useful) chart:

The white strip is the area where you are within 99% percent of optimal efficiency. The light gray strip outside of that is the area within 95% of optimal efficiency.

For example, if your personal retention rate at default Anki settings is 90%, then you can use an interval factor anywhere between 6x and 8x and still be 99% efficient. And if you're willing to go as low as 95% efficiency, you can range between 4x and 11.5x. That's a huge range.

However, keep in mind that this is all a bit fuzzy. There are many assumptions made in this simulation, and the sampling has a small bit of noise in it (which is why the strips aren't smooth curves). So aiming for the middle of the 99% efficiency strip is probably best if you have good data on your retention rate.

None of this is especially revolutionary, so far. Matt already figured most of this stuff out in his video. So this is mostly just validating his formula.

But what I find most interesting about this graph is what it means for people who are just starting out, and don't already have data on their retention rate. In other words: if you have no idea what your retention rate is, what interval factor should you use? This graph helps us answer that question.

Although I don't have any data to back this up, my guess is that most people fall somewhere in the 80-90% retention range, with maybe some outliers going as low as 75%. So if we look at the entire range between 75% and 90%, we can see that a 5x interval factor gives us 95%+ efficiency in the entire 80-90% range, and even at a 75% retention rate you barely squeeze in at 95% of optimal efficiency.

My guess at typical retention rates could be off, but I think it's at least a reasonable guess (if anyone has real data on this, I'd love to know!). So my suggestion is this: when starting out, use a 5x interval factor (or interval modifier of 200%, as per Matt's video). Then you're likely operating within 95% of optimal efficiency. Once you've been using Anki for several months, then you can take a look at your actual retention rate and adjust from there.

Do note, however, that the above graphs have personal retention rates corresponding to default anki settings. So here's another graph for retention rates using the 200% interval modifier:

Once you've collected enough retention data using an interval modifier of 200%, use this graph to find your optimal setting. That is, if you care about squeezing out that last 5% of efficiency.

Final Notes

There's more I'd like to say on this topic, and I've glossed over quite a few things in this post. But it's already quite long, so I'm going to end it here. But I'll mention a couple final things.

First, tweaking the other settings of the simulation impact these graphs a bit. In particular, the "max lapses" setting can have a really significant impact. I set it to 8 lapses for the simulations in this post. In my testing, this closely matches most max lapse settings you might choose, except for very low ones (e.g. 3 or less). Aside from that, I've tried to use fairly "typical" settings for the graphs in this post, and therefore this should be pretty close to accurate for most configurations. But if your settings are especially atypical, I recommend re-running the simulations yourself with your own Anki settings.

Second, as Matt noted in his video, these ideas are untested in the real world. The assumptions and math used in these simulations might not match real human beings. Therefore, take this with an appropriate grain of salt. However, I do think the principle is sound, and I would be surprised if e.g. using an interval factor of 4x (interval modifier 160%) wouldn't be beneficial to efficiency, even if the assumptions break down at more extreme settings like the ones suggested in this post and by Matt in his video. In any case, I plan to use the 5x factor (200% modifier) myself, and see how it goes. If you're up for being a guinea pig, feel free to join me!

2018 - 09 - 18

Politics and Open Source Communities

Bryan Lunduke posted a video earlier this year entitled "Divisive Politics are destroying Open Source". I only just now came across it, and it sparked both thoughts and feelings for me. My thoughts are not, I think, terribly complex, but I would nevertheless like to take a moment to write them out. In particular, as an avid Rust programmer, I'd like to give my perspective on the Rust community, since Bryan briefly touched on that.

Bryan's video, if I can summarize it in a nutshell, covers several cases in the open source world where he feels that leftist politics are seeping into communities and actually making them less inclusive. As far as I can tell, Bryan himself seems left-leaning (though I could be wrong), so that's really interesting to me.

(Also, for context while reading the rest of this post: if I were to label my own socio-political views, I would certainly label myself left. Quite far left, actually, I think. Though I try not to take on labels, at least not in a prescriptive way. In any case, I'm not right-wing, voted for Hillary (though would have preferred Lessig), all of that.)

One of the things Bryan's video got me thinking about is what "inclusive" ought to mean with respect to things like open source communities. Much like the word "art", people will disagree on the specifics of what qualifies as "inclusive". Also like the word "art", I do not intend to claim any ownership over what definition should be assigned to it. Nevertheless, I think there is a positive value behind that word that most people can get behind.

Almost everyone has had experiences where they felt excluded, and typically such experiences aren't positive. And I think most human beings with empathy can agree that creating circumstances where people feel excluded and rejected is generally a bad thing. Sometimes that "badness" is a price we pay for some other more important outcome. For an obvious and unnecessarily silly example, I don't want some uncoordinated Joe Shmoe waltzing about the operating room while surgery is being done on me, no matter how excluded he may otherwise feel. But that exclusion is nevertheless a price. It is something we would like to minimize where possible.

I think this very core human experience--of being excluded, of being rejected--is often overlooked in the way people talk about diversity. A lot of people go on and on about how diversity of backgrounds brings diversity of ideas, how it improves productivity, it improves outcomes, etc. But I think all of these things just aren't the real issue (excepting for in politics or similar where representation is important). The real issue is that being excluded sucks. Being told you can't join the club sucks. Being told that you aren't allowed to participate sucks. Being told that people don't want you sucks. And in some cases, being excluded can actually prevent (or at least make much more difficult) exploring something in your life that you want to explore, or from taking your life in a direction that you want to take it, or from connecting with other human beings. These are the things that actually matter about being inclusive.

And I think there's actually a better word than "inclusive" to describe that value: welcoming. I want communities that I'm a part of to be welcoming. There is a warmth to that word that "inclusive" lacks. It brings to mind images of open arms, of encouragement, of supportiveness. It says, come on in, no matter who you are, and sit down by the fireplace and have a nice cup of hot chocolate with us. We want you here. That is what I want in a community (among many other positive traits).

So how does Bryan's video tie into all this? Well, where I find myself agreeing with Bryan is that I think there are plenty of people on the political left that are really shitty at being welcoming, especially when it comes to people that don't share their politics. Of course, this is not an exclusive trait of the political left. But I feel like the language we use on the left about inclusivity and diversity sometimes serves to obscure whether or not we're really being welcoming.

So far this has all been really abstract, so let's bring in a concrete example from Bryan's video. One of the people he talked about is Ashley Williams. I won't go into the whole debacle, but Ashley is (if I recall correctly) currently the team lead of Rust's Community Team. And she posted things like this on Twitter:

never underestimate the wrath of a mildly inconvenienced white dude (and yes it is all dudes complaining)

I want to take a moment to state in no uncertain terms that I don't think Ashley is a bad person. Honestly, I suspect she's probably a great person. And if you look at her more general involvement in things, you'll see that she's really cool! People vent, and sometimes people vent online. And also, sarcasm/hyperbole often doesn't translate well online. I've had plenty of my own mis-steps.

Anyway... it is nevertheless pretty clear that her tweet is not welcoming. In fact, I'd say it's quite the opposite! If a woman overheard some guys complaining about women (even sarcastically and with hyperbole), I'm pretty sure that woman wouldn't feel very welcome. In fact, I'm quite certain articles have been written on that topic. But also, if I overheard some guys complaining about women in a way analogous to Ashley's tweet above, I would feel unwelcome, even though I am not a woman. And that's because it creates an unwelcoming environment.

Whether we care about white men in particular feeling welcome is something we could discuss (and I would argue that, yes, we should care, because we should care about people, but that's a digression for another time). But regardless, I really don't think we get to pick-and-choose what demographics of people we're welcoming towards. Most empathetic human beings dislike seeing other people being shat on (past traumas or brainwashing notwithstanding), and it creates an uneasiness about whether that kind of behavior is acceptable in general. It's like when you're on a date with someone and they're super nice to you, but they're an asshole to the waitstaff.

Now, where I disagree with Bryan's video is in how dire he makes things out to be. My take on these sorts of situations is generally something like, "Yeah, people can be shitty on the left as well--sometimes really shitty." I don't think the world is going to end. Looking at the Rust community, I think overall it does a pretty good job of being welcoming, including towards white men. Is it perfect? Nope. And I do think there's a tendency on the far left to be unaware (sometimes aggressively, seemingly willfully so) of their empathy blindspots. But nevertheless, the Rust community gets a lot of things right.

My biggest concern is failing to be welcoming to people with different political views. Maybe that sounds strange to some people (especially those that vilify the right, "Good riddance!"). But I don't think that should sound strange. I've seen some politically left jargon used in posts from the core Rust team about community management etc. The Code of Conduct also uses some jargon and framing from the left (e.g. "We don’t tolerate behavior that excludes people in socially marginalized groups," instead of just, "We don’t tolerate behavior that excludes people."). And although that's not explicitly unwelcoming, if you put on the other side's shoes for just a moment it's not hard to see how that gives a feeling that Rust is a politically left community in general. And maybe it is. But I don't think it has to be.

And this comes back to why I think inclusivity is actually important. Excluding people hurts. Excluding people is bad. At best, it's a price we pay for some other gain. So what is the gain in this case?

If we want to maintain a welcoming community, then everyone in the community needs to be welcoming. That sounds like a tautology, but what I'm getting at is I do see the value in pushing people out who are not themselves willing to be welcoming--or at least willing to not get in the way of the rest of the community being welcoming. The first bullet of Rust's Code of Conduct has it totally right:

We are committed to providing a friendly, safe and welcoming environment for all, regardless of level of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.

But I would like to add "political views" to that list, even if only implicitly. If part of someone's political views is literally "black people suck", that obviously won't fly. But neither would it fly if someone's religious beliefs included "gay people should be stoned," even though religion is already on the list. So I guess what I'm saying is, in my mind we should aim for this:

We are committed to providing a friendly, safe and welcoming environment for everyone who is willing to uphold this same commitment, regardless of any other trait they may or may not have.

This is surprisingly subtle, though, because I think there are plenty of people on the far left who think what they're already doing is equivalent. As far as I can tell, they believe that the socio-political views of the left are the only views compatible with being welcoming as above. But they aren't. And neither are they sufficient, as Ashley's tweet above demonstrates. There is something deeper and more human about being genuinely and warmly welcoming to everyone. And it isn't about socio-political views (except those that literally conflict).

This whole post is really rambling, and it's not at all clear that I've even communicated my point very well. But I wanted to put this out there because... dunno, really. Come to think of it, I don't think anyone actually reads my blog.

2018 - 01 - 31

A New Kind of Editor

My first post in four years! And, interestingly, on a similar topic as my last post.

Zed, the editor I raved about in my last post, is now essentially abandoned and unmaintained by its author. This is understandable given the direction his career has taken, but is also a bit unfortunate given its innovative approach to editing.

I've since been bouncing around using some other editors, including Atom, VS Code, and most recently Sublime Text 3. In all three cases I've configured them to function as similarly to Zed as possible, but it's never been quite right. Under the hood, all three editors still assume an "open dirty buffer" model of document management, and this leaks out from time to time.

Despite the shortcomings, I've been mostly okay with this state of affairs. But there are nevertheless some things that annoy me:

  1. Both Atom and VS Code are built on top of web technology. I don't like my editors taking up hundreds of megabytes of RAM with only a couple of text files open. That's just silly. And in Atom's case, it has chuggy performance to boot.
  2. Sublime Text isn't open source. I don't mind paying for software (and happily did in Sublime Text's case), but I always feel nervous about committing my workflow to software that doesn't belong to a larger community.

Because of these annoyances, a few years ago I developed an interest in writing my own editor. This sounds completely silly, and it pretty much is. Nevertheless, the journey of creating my own editor has been really interesting, and I've learned a lot of cool things.

Realistically, I don't think the editor will become anything worth using, but it's a really fun exploration, and I want to share that. I have more to share than will reasonably fit in a single post, so this will be the first post of (probably) many.

Led

My editor is called Led, which can stand for "Lousy Editor", "Loser's Editor", "Laughable Editor", or any number of other derogatory phrases that start with "L" and end in "Editor". In its current state Led is a console-based editor that:

  • Can load text files faster than even Vim. A 1GB file takes approximately 1.5 seconds to load on my machine, vs 2.5 seconds for vim.
  • Can edit said 1GB text file without breaking a sweat. (Including when it's all on one line!)
  • Supports real time word wrapping that you never have to wait for. (Even with the afore-mentioned 1GB all-on-one-line file!)
  • Correctly handles Unicode text, including graphemes and double-width characters. (However, it doesn't do bidirectional text.)
  • Recognizes all Unicode line endings (there are eight of them!).
  • Has basic undo/redo functionality.

All-in-all, it's an extremely bare-bones editor, but that has some really robust characteristics. I would absolutely not want to use it for any real editing tasks, but it feels pretty great that in the realm of bare-bones-only text processing it beats out every other editor I've tested it against. Essentially, Led can open and happily edit any utf8 file you throw at it, as long as it will fit in RAM, no matter how weird. Every other editor I tested (including Vim, Sublime Text, emacs, etc.) chokes in at least some corner cases.

Personally, I think that's pretty cool, and I'm reasonably proud of it. Makes me feel like a good software engineer! But obviously, it has a long ways to go to be useful.

Vision

My lofty vision for Led, which will likely never come true, is basically to create a better Zed. Zed made some great UX decisions, but was implemented on top of web technology. Creating a native-code clone would itself be a nice boost, but in the process of developing Led I've come up with some additional things that I want to explore.

The main thing I want to explore is a more expansive view of what is considered "the thing you're editing". Every editor I've ever encountered considers a text file as the thing you're editing. But I think in reality the thing you're editing is the entire code base.

Now, I'm not saying that code editors aren't multi-file aware. Obviously, most are! In fact, every good code editor I've used allows you to work with some kind of project or directory concept. But editing operations aren't treated at that level. The primary experience I've had that makes this apparent is when you do a project-wide search-and-replace: there's no "undo" operation for that. And that makes such operations pretty scary. For most editors this situation is understandable: an undo operation that applies to multiple files raises a lot of questions, like what the state of all the touched files should be afterwards. Saved? Loaded and dirty? Some other weird special-cased concept?

Zed's approach to editing eliminates most of those questions entirely. Buffers are never dirty, undo is infinite and persistent, etc. With that approach you can treat the entire code-base as a single item that just happens to be subdivided into files. Then operations like multi-file search-and-replace, refactor operations, etc. are like any other editing operation, and can be undone, redone, etc. just like anything else.

In such a system, undo within a single file is just a special case: you're selectively undoing within an area. And you could take that even further: if you can selectively undo within a single file, why not within a selection? I cannot count the number of times I've made some edits in one place in a file, then done some unrelated edits elsewhere in it, and wanted to temporarily undo the first edits while testing the second edits. It actually happens quite a lot. Being able to select an area and undo within that would be great!

With this approach, all operations become something that function within a scope, which can be as large as the entire project, or as small as a tiny selection. This would make everything, not just undo/redo, potentially much more powerful.

The default scope for things should probably still be the currently active file. But this concept unifies a lot of things that are otherwise disjoint and strange. I would like to explore how to make something like this work in practice.