For RPG Maker 2000/2003: Press F4 to toggle windowed/fullscreen
For RPG Maker XP: Pess Alt and Enter together to toggle windowed/fullscreen
For RPG Maker VX/VX Ace: Press F1 and in one of the tabs there should be a tab that says something about fullscreen. Untick that box and close and reopen the game to run windowed.
I hope I helped.