Throughout the whole colonial period, the flow of slave ships heading for the archipelago was intense, since cultivation on the island's farms was based on slave labour. The extremely fertile soil and the abundance of waterfalls to turn water wheels, made the Island an excellent site to build sugar mills. The first sugar-mill on the Island was established in 1608 by Francisco Escobar Ortiz.

In the late 17th century there were already 5 mills producing sugar and another 17 producing "aguardente" (a type of sugar cane brandy). There are no records regarding the number of inhabitants or slaves before 1808, when the local settlement was raised to the status of a small town with the name of Villa Bella. There were about 3,000 inhabitants and around 2,000 slaves. The black population increased constantly till 1850 when trafficking in African slaves was prohibited in Brazil. From then on, until 1888, when slavery was abolished in Brazil and all the slaves were freed, Ilhabela profited a lot from the illegal slave trade. The numbers of black people in the region shot up. Each of the 20 farms on the Island was estimated to have 500 slaves, totalling 10,000 in all.

On account of its geography, with the inhabited part of the island being very close to the mainland, and the other part being inhospitable and inaccessible, Ilhabela became a depot for slaves illegally brought from Africa. The slave ships coming straight from Africa used to harbour in Castelhanos Bay. They would unload their cargo at huge anchorage sites specially built for this purpose, such as the old Lage Preta Farm. The black people were then forced to endure the rough-hewn trails over the island's high mountains, on their way toward the farms at the banks of the canal. This illegal trade increased, prospered and made a lot of money for the local farmers who would sell these negroes smuggled to the mainland as if they had been born in their properties.

Out of the Island's ancient stories involving slaves, perhaps the most memorable one is that of the Feiticeira (Sorceress) who ultimately lent her name to the beach on the southern bank of the Channel/Canal. It is said that the owner of São Mathias farm, the main residence of which is nowadays found at the left end of that beach, was a beautiful lonely woman who had accumulated a fortune acting as a fence for the pirates' plunder. The pirates trusted her to sell the goods stolen from the ships they boarded on the high seas. Part of her fortune came from concealing captains of slave ships that after 1850, when the African slave trade was prohibited, began using the Island as their favourite point of entry into Brazil for illegal slaves.

In her old age, the local residents, out of envy, nicknamed her 'the sorceress' because she lived alone at a time when this was unthinkable for a woman. She then decided to hide part of her treasure in the jungle. She went away, at the head of a large caravan of slaves and vanished into the woods for a few weeks, later returning completely alone. She was said to have buried the treasure and killed the men one by one, so that no one would ever know where the riches had been buried. After this incident, she is said to have gone insane and disappeared, leaving behind the mystery of her treasure and the name of her beach.