You’ll need to exit the Alcazaba in order to visit the Charle’s V palace. Go down Torre de la Vela back to Arms Square and head to the first door on the right to the “Adarve Gardens”, which leads you to the Alcazaba exit. Leaving the Alcazaba, go to the large square building in front, the Palace of Charles V.