How Flashcards Fail: Confessions of a Tired Memory Guy

You can use flashcards and spaced repetition to memorize almost anything -- so why did I come to hate my reviews? Explore the dark side of this amazing memory system.

[UPDATE: You all left so many amazing comments on this article with awesome solutions! So I’ve compiled my favorite solutions into a whole separate article, “How Flashcards Succeed.” I’ll link to it at the end — don’t miss it!]

In my last post, I shared how spaced repetition flashcards made my memory feel like a superpower. I learned to remember thousands of things that would otherwise have misted away.

How Did I Come to Hate Reviews?

But flashcards depend on daily reviews. And after a few years, I just couldn’t stand the reviews anymore. I started skipping days. Weeks. Months. Today, if I fire up Anki, I have over 2000 cards due.

What went wrong?

At first, I blamed myself. I lacked stamina. Determination. Something.

But the solution isn’t that simple. On reflection, I’ve realized that flashcards and spaced repetition have some inherent dangers and difficulties. We can work around them, but only if we step back and think about them.

Review Clusters and Adding New Cards

The first problem: spaced repetition is uneven. That’s the nature of the math. Some days will have only a few reviews. Others will have a ton of cards. Those huge days can be demoralizing. You fire up Anki, and you have hundreds of reviews.

There are several ways to deal with this.

At a minimum, if you have a ton of cards due, don’t add any new cards that day.

You can also break up your reviews into several short sessions. But I rarely did that. I wanted to get reviews done (more on that later).

Also, Anki has a feature where you can see which days coming up will be big days. If you know tomorrow’s going to be a big day, you could review some of those cards today, ahead of time.

But there’s a deeper problem: depending on how you add new cards today, you could be creating mammoth review days down the road. That math is just too complex for a normal person to do on the fly. I want to know: if I add 20 cards today, are they going to land on some day next week that’s already huge?

This seems like a problem the program could address. It can’t predict exactly how much you’ll need to review. But perhaps it could estimate, and warn you if new cards today will land on a day next week that’s already overbooked.

Boredom Bias

A much more difficult problem is the boredom bias. The more boring a card is for you, (unless it’s boring because it’s easy), the more likely you are to get it wrong.

But the whole point of spaced repetition is to focus your time on the cards you get wrong!

Pretty soon, you’re spending most of your review time precisely on the most boring and/or difficult cards.

Damien Elmes, the lead Anki developer, is aware of this problem. His “leech” feature will remove a card after you miss it a whole bunch of times. But when you have thousands of cards, it can take a long time to purge each leech.

Card Laziness

Of course, you can set leeches to disappear faster. But is that the answer? You don’t want to disappear a leech. You want to figure out how to learn it. Isn’t that why it’s in your deck?

Either it’s important, and you need to fix it, or it’s not important, and it’s been a waste of time from the start.

But fixing leeches is a pain. It’s especially a pain if you’re reviewing with AnkiDroid, or another mobile app.

Yes, Anki on your phone is tantalizing. You can slip life-changing memory building into the dribs and drabs of time that are otherwise “wasted”.

But when you trip up on a leech, the last thing you want to do is wrestle with your tiny onscreen keyboard.

Even if you have a usable mobile keyboard, fixing a leech feels like breaking the flow. I’m trying to review here. Get this done. I don’t want to slow down and … um … think?


Flashcards As Tests

At last we poke the raw nerve. What exactly am I doing when I review flashcards? What’s the “flow” that I don’t want to break by fixing broken flashcards?

Am I taking a test?

I’ve spent about half my life (at this point) in school. School is about tests.

When I started doing flashcards, I was excited about how different they are from tests. Tests are a single snapshot of your recall at a particular time. Unfortunately, most students, including valedictorians, forget almost everything they ever get tested on. Traditional tests are a complete failure.

Flashcards are different, because you maintain knowledge. Spaced repetition ensures that you keep seeing things before you forget them.

But flashcards still feel a lot like school tests. Yes, in theory, it doesn’t matter much whether I get a card wrong, since I’ll see it tomorrow. But in practice, I hated getting cards wrong.

At first, I thought this dislike was an old school hangup. I expected that, in time, I would adjust to this awesome new world of gradeless reviews.

But the opposite happened. As time went on, my hatred for mistakes grew.

Spaced Repetition Collects Your Worst Cards

And because of the “boredom bias”, I was spending more and more time reviewing my least favorite cards, getting them wrong, and knowing I would have to see them again.

I began to amass a mental collection of cards that I could remember missing. The card would come up, and I’d think, “I always miss this.” And I would.

That is amazing. Think about that. Whatever I was doing, it was training my brain to remember that I couldn’t remember this. Somehow, I had spent enough time to make that connection, instead of connecting to the actual fact.

Why? Because I was mostly thinking about whether I’d get it right, not about what I was trying to learn.

Flashcards As Video Game

Flashcard review has an intense gravitational pull towards focusing on the actual flashcards:

  • How many cards you’ve gotten wrong.
  • How many cards are left.
  • How long it’ll take to finish them, so you can get back to your real life.

As a friend put it, flashcards feel like a video game. Wow! They really do. And I like video games, but they’re fundamentally geared towards success. Points. Levels. Defeating obstacles.

Should memorizing be a video game?

No. Flashcards are a huge step forward from tests, but if you’re focusing on success, you’re still fundamentally focused on grades. And that leads to absurdities like, in my case, actually memorizing that you always miss a question.

It also leads to the flashcard flow. I wanted to get through these cards as fast as possible! The last thing I wanted to do is hit the brakes and reformulate some broken card. That felt like losing.

Atomizing and Randomizing Knowledge

Flashcards seemed to disconnect me from the actual knowledge. Instead of immersing myself in the knowledge, I found myself controlling a flashcard slot machine.

The knowledge was atomized and randomized. I’d get a Spanish word, then some obscure fact from Edible Forest Gardens, then a letter in Morse Code.

This randomization is essential to spaced repetition. I needed to see that Spanish word, on that day.

For vocabulary, this might have been fine. Words are tiny bits of information that need to be random. We want to be able to reach for any word at any time.

It’s interesting that both SuperMemo and Anki were developed by programmers who were trying to learn a language. Despite all my problems with flashcards, I still think they could be an essential tool for mastering large quantities of vocabulary.

But what about the facts from books? Does it really help that a particular fact is only associated with the question on one flashcard? Is that really how our minds work?

Flashcards Kill Clustering

No. Our minds use chunks and clusters. You want more connections to each fact, not less.

Flashcards work directly against this mental need. Flashcards atomize each fact completely out of any context, except its context as a random flashcard.

Think about your friends, or even acquaintances. You think of a name, and instantly you get a face, hair color, voice, an outfit, the rooms where you see this person — and that’s all in less than a second. If you focus, you can pull tens, hundreds, probably thousands of discrete facts that cluster around this single person.

Spaced repetition and flashcards kill clustering. Instead of associating multiple facts from the same book with one another, you split them all up. Divide and forget.

For me, almost all this knowledge began to have no other meaning than as a flashcard. The only time I thought about it was when I was testing myself.


In short, this tool for remembering more of reality had morphed into an oppressive, self-contained computer game. The only way to win was to finish as fast as possible.

How can I fix this?

By questioning my underlying assumptions about the whole method.

Next time: Reviewing as Thinking

[UPDATE: As promised, here’s the followup article with an epic list of Anki solutions: How Flashcards Succeed.]

Free Video Course

Enter your email to get my free video course: "How to Remember Anything You Want". I'm so glad you're going to take action and improve your memory for real!

Plus, we’ll stay in touch. I’'ll be honored to keep sharing what I learn and helping you stay on track to unleash your awesome memory.

Spam? Not a chance, no way, ever. And you can unsubscribe any time with a mere click. Powered by ConvertKit

19 Replies to “How Flashcards Fail: Confessions of a Tired Memory Guy”

  1. Flashcard (SRS/Anki) problems
    Dear Bill,

    This is a really interesting post about using SRS flashcards. As a heavy user of Anki to study Japanese here in Tokyo, I have spend countless hours creating and studying cards over the last three years. It is easy to relate to many of the things you said.

    I don’t have much constructive to add in terms of flashcard fatigue or other similar problems. Perhaps we are so eager to learn that we add too many cards too quickly. At the beginning, it seems very doable, but with time, decks can become very large, and they can easily slip out of control.

    In terms of randomizing and clustering, I have had similar problems with the decks I have made from my Japanese – Rosetta Stone materials. Some of the decks are very large, and reviews can feel disjointed and disconnected, all of which can be very discouraging…

    However, recently I tried something that might be helpful. With a deck with almost 3000 cards, I had gotten behind in my reviews to the tune of almost 1500 cards. It seemed hopeless. So I decided to use the tags to only study the cards from the most recent lesson – about 260 cards. This did two things that really seemed to help.

    One was psychological: suddenly the number of reviews did not seem overwhelming, and I felt empowered to study in a dedicated fashion. I know all the other reviews are waiting out there for me, but reviewing now felt like a positive experience.

    The second thing was that now all the cards were from the same lesson: all about food, cooking, tools, building, etc. They were clustered again and helped reinforce each other, again making it all feel good.

    So perhaps the key is to break up large decks into more manageable chunks, both in terms of number and content?

    On a similar note, I have a review deck for the first 2000 kanji that I keep chipping away at. I also have a deck of new kanji with 330 cards and climbing that I am able to zero out every morning – a small victory that keeps me motivated!

    I’d be happy to hear your thoughts on any of this. Good luck with all your memorizing!


    Rich Bailey
    Tokyo, Japan

    1. Rich, thanks for stopping by!
      Rich, thanks for stopping by! I just found your site through Damien’s Twitter feed, and I’ve been looking forward to digging in and working my way through your posts. Now here you are!

      It’s _extremely_ helpful for me to hear that another Anki enthusiast has faced similar problems. What a relief!

      But I think you’ve started to solve these problems too, with the clustering that you describe. What if Anki could do this automatically? Probably, we’ve only begun to understand spaced repetition — imagine if the Anki algorithms took into account cards with similar tags? If we thought of ourselves as reviewing _networks_ of cards, not isolated facts? We might review “easy” cards a bit more often, but find ourselves remembering “harder” cards _much_ more often.

      Still, as I say in my [next post](content/reviewing-thinking), I think the solution (for me anyway) also lies in radically rethinking how I approach the whole process of “reviews”.

      Thanks again for your thoughts!

      1. Bill, I too had problems with
        Bill, I too had problems with clustering, until I found the cure. The trick is to limit the number of new cards that you are seeing to a fairly low number each day. eg. 20. This allows you to build up a back log of new cards and not have to worry about “lumps” nearly as much.

        If you want to get deeper into efficiency for flashcards, Super-memo does just what you describe for clustering, in fact, though careful testing, the author has found roughly the optimum formula for memory. The reason that I use Anki like you do instead is that it’s simpler and less prone to the weird types of failure that the super memo algorithm is.

        1. Thanks for the comment! I
          Thanks for the comment! I agree that limiting the number of new cards is crucial. But I found that, for me, even 20 cards per day would eventually add up to a lot.

          What kind of topics do you study with Anki?

    2. Bill, Rich—
      Bill, Rich—

      I did something similar (though not quite), but I actually started off with two different decks for German words. It depended on the source of the words. I have one deck for words I come across while reading newspapers or listening to the radio, and I have another deck for words that come from a German word frequency list.

      Really, they should all be in the one deck, but I’ve found that it’s much more manageable. I study the decks at different times of the day.

      I had to take a break from studying my cards recently, and one deck built up to around 800 words. So I started a new deck and begun learning this, and I’m slowly pecking away at the reviews in the other deck. Some days I might do a few hundred, other days only twenty reviews.

  2. Leeches
    Hi. As a heavy Anki user (almost 300,000 reviews so far), I’ve faced the same problems as you. When we face a difficult card and try to remember the answer, we begin to travel along the well-established neural pathways that lead us to the wrong answer or to nowhere, and this attempt to remember strengthen those incorrect paths, creating a vicious circle of error. Here’s what I do with leeches now: I put a mnemonic on the question side. If even that doesn’t work, I put the answer on the question side. At that point I’m obviously not testing myself, but I realized that testing myself isn’t the goal. Every time I see that now-easy card, I’m reviewing it anyway. It will probably end up forming a memory. If it’s truly useful, I’ll see it in the real world and will later associate it with that encounter.

    1. Hmm, putting the answer on
      Hmm, putting the *answer* on the card — very interesting! The usual advice is to rework the card, simplify it, perhaps break the material into multiple cards. But your approach intrigues me. Instead of reinforcing that mysteriously broken “vicious circle”, you’re at least seeing the material on regular basis. Thanks so much for sharing this.

      1. Difficult cards
        I guess I’m a little bit too late for the party but another tip, aside from the obvious one of rewriting the card, would be to remove the time limit and take as much time as necessary to answer correctly. It’s better to take longer and answer it right than to answer it wrong and get the wrong circuitry reinforced inside your brain.

        And also, practice only with full concentration. If you’re the kind of person who answers flashcards while walking or waiting for the bus, then at least skip this kind of card when you find yourself on such environments.

    2. Answers on question cards
      Brilliant. I particularly like that you point out, like the OP, that this shouldn’t be about testing.

      Perhaps an automated feature in Anki that instead of just removing it after a certain number of times as a leech, instead prompted you to write a mnemonic or hint that would be added to the question side. If you still keep getting it wrong, the answer gets added to the question side too.

      Of course, this process would ideally also be automatically reversed. After you’d seen the card with the answer enough times, the answer gets removed. When you’ve correctly answered the card with a mnemonic enough times, the mnemonic is removed…

      This is a real scripting issue I guess.But fascinating reading and ideas, people!

    3. Specfic Items that are difficult to remember
      FWIW, I find that for myself when something becomes difficult for me to learn I need actually review it more than once a day. I used to walk around with (physical) notecards of those hard to remember things and review them about 3 times a day at random (when I’m in line etc). After 2 days, I’d have them down cold. HTH and GL

    4. Yes, give yourself hints
      I handle the “difficult cards” in a similar manner. I’ll give myself the first or last “letter” of the answer (say, a hint like “k–” on a Japanese vocab card for 軽蔑), or a cloze style sample sentence, or a definition from a native dictionary on the question side. For words that could easily be mistaken for each other, I add content, or even a note of a word that is NOT the answer for card. (Q: “heart attack (NOT 心臓麻痺, the other word for it), A: “心臓発作”)

      The point of the cards, as you said in the article, is to get them right. So, I figure, why not do whatever I can to help myself get them right? This is a drastically different approach to study from when I was a teenager in high school and expected myself to be able to be able to produce extraordinary answers based on minimalist questions. Anki and other SRS software is quite flexible about editing cards later and about how much info can be displayed on a card. If I find a card difficult, I can add gobs of more content to the question side and usually I’ll find that it has become much easier. If a card takes too long to answer, I’ll rework it to call for a shorter, more instantaneous response. In addition to the extra hints, I think the time spent reworking the card helps retain it: If we spend fifteen minutes with a dictionary or looking up usage examples online, consider what we find, and select some of that information to place on out reworked cards, we’re resorting those missing components–active, critical thinking and networking–to our flashcard study experience.

  3. Some suggestions:
    Some suggestions:

    About the boredom part:
    I only create flash cards for:
    1) vocabulary that I read in books or that I hear around
    2) concepts, ideas or mental models that are core basic knowledge of a given area
    3) high level concepts in my area of expertise.
    4) anything that is too expensive to lookup when I need it

    For leeches, besides finding mnemonics, I also use images from Google Images. It doesn’t work for abstract concepts but it does wonders for concrete words.

    About the clustering issue:

    For vocabulary I go as fast as I can. As soon as I can explain the meaning verbally I move on to the next card.
    For decks with higher level concepts like Algorithms or Game theory every time I review a card I explain it and I let me my mind linger and come up with associations. It’s rare the card where at least 1 or 2 associations new associations don’t pop in my mind including questions. Besides that it’s also usually the case that when I try to explain a concept I find little nuances or questions that I then research. And that builds new associations too.

    This requires time, of course, that’s why I find it important to prune knowledge aggressively. I am generally eager to add new knowledge and overestimate its value. Boredom is actually a godsend for me.

  4. Tip for avoiding Card Failure
    I have been using anki to learn anatomy, physiology, pathology, etc

    In regards to difficult cards and leaches, I create a separate deck called “FIX THESE”. So whenever I am reviewing and do not want to disrupt my flow I simply move obstinate cards into this deck and deal with them later. It can be done pretty quickly on anki droid as well.

    Rewording or reorganizing the card differently is usually enough to prevent future leaching.

  5. Problems with Anki
    Hi–I found your article interesting. I can understand why some of the things that frustrate you about Anki frustrate you. I have had some of the same frustrations, but I think I have solved most of them, or at least come up with a solution that works for me. I will share them with you, addressing your issues one by one.

    Spaces Repetition is Uneven
    The main problem is not that it is uneven, but occasionally overwhelming. If you only have 20 minutes a day to spend on review, a 10 minute day is not a problem, but a 30 minute day is.

    There are always two possible approaches to solving Anki problems. One is to adjust the settings, the other is to adjust how you design cards.

    To adjust your settings, you can limit the number of new cards a day, and the use the time box feature to tell you when to stop reviewing. Limiting new cards will reduce days that go over your preferred limit. Time boxing will tell you when that limit is reached. Then you just stop. Over time the rat will work its way through the snake.

    As far as card design goes, try to be aware of the difficulty of material up front, and break down difficult cards into easier cards, or create redundancy–learning the same material in different ways–for harder material.

    Boredom Bias and Card Laziness
    If you are bogged down by a lot of leaches, that can be a problem. I agree that it would be hard to address leaches on a mobile app, so I would suggest that every week or so you sync with your computer, and then address leaches. You can use the sort feature to find all of them. Then you can decide to delete an item, or improve it by breaking it down, or creating redundancies. Going through this process will help you to create better new cards in the future, as you will see the types of cards and information that give you trouble.

    Even if you do nothing, you can allow leaches to stay in your deck unchanged, and eventually they will be learned. But if something is not high value, delete it. Mainly this comes down to whether it is foundational for other learning, or just a fact that you are unlikely to need in the future.

    Flash Cards as Video Games
    I think I just have a different attitude about this. I am comforted by the fact that right or wrong don’t matter. I appreciate Anki telling me that I no longer know something I thought I had learned. That is the whole point. I think that perhaps the reason that you feel you are “losing” the game is that you are “punished” with additional study time when you miss a lot of cards. Limiting new cards, time boxing, addressing leaches, and creating efficient cards to begin with will make it easier to appreciate failed cards.

    Atomizing Random Knowledge and Flash Cards Kill Clustering
    I see this criticism of Anki a lot. I think the problem with it is that it sets up a false dichotomy. For example, people will talk about how their friend went to Mexico for a year and learned Spanish without ever using a flash card. Actually, the BEST way to learn Spanish is to study flash cards AND practice talking to native speakers, listening to Spanish podcasts, reading Spanish books–using Spanish. They are complementary systems. So don’t only depend on Anki, and don’t over do it.

    As you have said, it is easy to get sucked into the game, and to start thinking of your Anki numbers as your knowledge level. That is because Anki is quantifiable, and having a conversation is not. So you need to schedule the other knowledge synthesizing methods, and make sure they happen. For example, I use Anki for practicing musical knowledge: chords, scales arpeggios. I schedule 15 minutes for Anki, and I schedule 15 minutes for drawing on that knowledge to create musically interesting ideas–jamming.

    I hope you find some of these ideas helpful.

    Anki will never be perfect, but no study method is, and it was never meant to be a “complete” learning system.

  6. You sound like someone who’s
    You sound like someone who’s asking for Supermemo’s Incremental Reading feature.

  7. Revising my method
    My ANKI experience has led me to the same conclusions! There are many types of information that is well suited to ANKI, like discrete facts or vocabulary, not requiring a larger context. But a lot of concepts / skills, need a longer structure/explanation/stories/interpretation/exercises/multimedia for us to grasp and establish a strong understanding. I like the idea of ANKI but have struggled with exactly what you are getting at in your post. I think ANKI can be augmented to work around it though. What i am trying recently, is to keep the information suited for pure memorization in ANKI as individual cards, but then to make longer notes into a powerpoint presentation document, and then link an ANKI card to that note. So when it comes up, you use the link, review the whole card and emmerse yourself in that note, then exit and rate how well you recalled it. This way atomized information can be atomized. And deeper concepts can stay deeper and connected. And all are reviewed with spaced repetition.

  8. I think the problem is…….
    I love this article.

    I think the problem is…..

    Humans are terrible at predicting what knowledge they will need to know.

    The truth is, even in technical professions– a doctor for example, probably should only remember about 1% of what they learned in medical school, 0% in college, and 0% of high school.

    The main thing SRS software taught me is: school is probably almost entirely a scam. At best most of our learning is about getting exposed to subjects and then picking something to learn more in depth. Most happens through osmosis.

    My current theory is to read widely – highlight 1% of that reading — 1% of those highlights I put in a mindmap using then 1% of that material (maybe less) goes into Anki.

    I have only been using SRS to learn sales and economics which are strange use cases.

    I think some of the classic ways of learning – writing about the subject and teaching it to others — can’t be replaced by SRS software. But the creator of SuperMemo’s wacky ideas on the importance of rote learning are probably true – it’s just hard to know what is worth really learning.


  9. Load Balancer
    In case someone else finds this post, here a few tips, because all the points discussed in the post are real. There is no point in pretending these issues don’t exist, but there are also ways around them.

    First, there is a “load balancer” plugin for Anki. This will take care of the peak days with many reviews and spread the reviews out a little bit. Yes, these demoralizing mammoth days are really a thing if the past once you’ve used this for a while. Unfortunately it doesn’t take care of existing peaks, but you could simply “skip” a day by choosing “hard” on all the reviews for a day (without looking at them). It may feel like cheating, but there is more gained than dropping out of Anki.

    Second, you don’t need tags to review similar items, but separate sub-decks. If you create separate decks for categories of words, and put move (drag) them into one, Anki will do the reviews in order/by category. It took me a bit of experimenting which sub-decks are pertinent (it also depends on the number of cards you typically review). I certainly wouldn’t want to mix my say Spanish and Chemistry reviews.

    Third, when I learn a bunch of new items, I add them all to Anki. If there are too many, I add them to a separate (temporary) deck and do say 20 reviews a day. Once there are no new cards left, I move all the cards in this temporary deck to wherever I want them. By that time, the cards are already spread out.

    Fourth, take leeches seriously. At a minimum, you should reformulate the card. Often I delete the card. Yes, I wanted to remember this after learning what it is, but given the effort it takes, I often *choose* not to remember this. This is quite liberating. (And incidentally also what makes Anki different from school exams: *I* can choose what I want to remember, *I* can choose to delete an item.

  10. My solution approach to the anki-burnout problem

    Your post is already quite old, but I stumbled over it several times while googling for anki-related stuff so I decided to leave a comment. I am using anki to study Japanese and am doing the following things to prevent the problems you mentioned:

    I continuously want to focus on new input and going through all the old reviews is tedious and time consuming. Therefore, I maintain multiple small decks (400-800 cards each). Anki discourages you to do that, but it allows to easily skip old cards – just let them accumulate in the old deck.
    On days where I don’t add new cards I sometimes go through the old cards. The good thing about leaving them un-reviewed is, that anki considers the time that is passing. When you review cards after they have been due for some weeks, they will be pushed further into the future.

    It can be quite difficult to ignore the large amount of due cards. I found that I know about 95% of them, but instead of considering that as awesome I often feel the pressure of filtering out the 5% I don’t know. Sometimes I also just feel the pressure of reducing that large number of reviews anki is showing me. Splitting your large collection of cards into smaller decks gives an easy solution to that: Just export the old decks as akpg-files (don’t forget to include the scheduling information and maybe the media). This way you don’t see them anymore but still have the opportunity to review them in the future by importing them again.

    I used to be a language learning perfectionist, which means that I cared about grammar and exactness way too much. I was afraid of not being able to remember a card after seeing that anki will reschedule it in 6 months when I hit “good”. Therefore, I hit “hard”. Getting rid of the aforementioned perfectionism is actually extremely helpful but I am still struggling with that. I try to ignore minor mistakes and found this to be really helpful – It reduces the review load, because I hit good/easy more often. Moreover, I suspend or delete cards that are really easy.

    A general thing that motivates me to ignore mistakes, leeches and all these things likely to cause pressure is the following: When you study a language, especially the grammar but also words and phrases repeat themselves over and over again. So if you choose “easy” despite that small mistake you just did, it is likely that you’ll get many other examples until the card shows up again in the future. And if you completely ignore (=suspend or delete) that hated card leeched for the 1000th time, you’ll create room for new content (which might also illustrate the problem you’re having with that card in a better way).

    That hopefully inspires people who have similar problems. Mabye one last remark regarding the clustering problem you mentioned: I am using only full sentences on my notes, that creates a strong context and makes memorizing new words really easy.

Comments are closed.