Basically they're just groups of renshuu users that are working with similar materials or have similar interests, etc. I don't think I can explain this too well, but there are different quests and stuff in the study groups and that can help motivate you to study more. Each group also has different activities like their own shiritori rounds and question corner as well as monthly quizes and more specific things like for example in the music lovers study group there's a song translation forum where everyone translates a few different lines of a song.
If there's any you want to join you should definitely join them! I think personally study groups definitely motivate me to keep studying Japanese (mainly because I really want my Japanese to get good enough to be able to translate a bit in one of the song translations in the music lovers study group ) and I feel that they are a great feature of renshuu!