The Rabbis were concerned that people directed their hearts toward God in the sense of praying towards the Temple:
"Our Rabbis taught: A blind man or one who cannot tell the cardinal points should direct his heart towards his Father in Heaven, as it says, And they pray unto the Lord. If one is standing outside Palestine, he should turn mentally towards Eretz Israel, as it says, And pray unto Thee towards their land. If he stands in Eretz Israel he should turn mentally towards Jerusalem, as it says, And they pray unto the Lord toward the city which Thou hast chosen." [Babylonian Talmud: Tractate Berakoth Folio 30a]
Paul, here, is dealing with something deeper: the motives and desires of our hearts.