I haven't been either, but am suddenly thinking I'd like to finally consider actually doing it. Ha. I've often thought I'd love to go to Tokyo, but it never went beyond that. :( The culture, food and nature all seem amazing.
I say, "Go for it!" The sooner the better, if possible. Of course, one should probably plan a bit more for a trip to Japan, than say, some European destination, but it's doable! I have lots of ideas and advice about it saved on insta! =)