Following your heart is one of the most important things that you can do in this life.  Only you know what is in your heart so stay true to yourself.  And yes, you will be criticized for whatever decision you make anyways so go for it!  In the end, who cares about others opinions?  All that really matters is following what is in your heart.