Say I come across a word that I have never seen before, and I don't have my Japanese dictionary with me to help me. The word contains kanji that I already know. How would I know which reading to use if I were to take a stab at guessing what the word is?
unfortunately there are no hard and fast rules: some kanji mostly use the kun-yomi, some mostly use the on, and some will do either with equal frequency. Also you can see both on-yomi and kun-yomi in the same kanji compound, just to make things more interesting! I know this is probably exactly what you don't want to hear, but it's probably best just to remember which kanji tend to use one reading over another and go from there.
As mysticfive said, while there isn't any set rule, there are ways to take an accurate guess at which reading to use.
When a single character is followed by hiragana (okurigana) then you will almost always use the Japanese reading. There are the odd characters that will use the Chinese reading (usually because it only has a Chinese reading), but more often than not, this is the case.
When two or more characters are in a compound for a normal word (i.e. not a person, or place's name), then you will usually use the Chinese reading.
With what 宮本勝利 said about compounds, that's exactly what I was referring to: in a compound, you can find kun/kun pairs (ie 背中 せなか), on/on pairs (ie 日本 にほん or にっぽん), kun/on pairs (ie 切符 きっぷ), and on/kun pairs (can't think of an example off the top of my head for that one :-[) so you really can't be sure of using the on-yomi for compounds.
