The only thing that makes the game difficult is the lack of control. This is what people call "Fake Difficulty" and unless your game is meant to be sadistic like I Wanna Be The Guy, having all of your difficulty be fake is a terrible design choice.
This game would be much more pleasant if I could drag and drop instead of clumsily bump things around.