I've never changed this setting, but I have found when I learn a new kanji, the mastery levels of terms with it drop. I'm fine with this, but since it seems like the setting isnt working, figured I could bring it up.
When you "add" a new vector to a term, it might look like that term drops to level 1, but it's actually just the new vector dropping the overall level. For example, I have 功 showing an overall mastery of 8, but when I include the kun'yomi, overall drops to 1. I can also reverse it by deselecting the kun'yomi. This is just visual, individual vectors have their own mastery levels.
The same thing happens when you "unlock" the kanji vectors for words.
I haven't tested it, but I assume that setting would actually drop the whole thing to X level. That's my understanding anyway.
And I guess it would drop mastery multiple times for words with more than one kanji you haven't learned yet? For example, if you had learned ほんとう, its mastery would drop twice, when you learn 本, and when you learn 当?
Your levels are not dropping, they're switching out.
When you have no kanji on a word, you are practicing kana study vectors (kana > meaning, meaning > kana). By default (settings), when you learn a kanji, you stop learning those vectors, and switch to kanji ones.
Because you're brand new with the kanji in that word, your kanji vector mastery is 0. Your kana vectors still have their old mastery, they're just not being used, so the mastery level is not shown or used in the average.