So, to leave town, you have to go to the inn. I have it where you sleep at the inn and then get teleported to the forest from there, since you are leaving there at dawn. There is also a tavern if you want to check that out and the rooms are on the second floor.