Everyone avoided the tower. It was believed to have been brought ashore by the Water-lord with the high tide, and they held it in great reverence. And distance.
When the Sea-Men returned, with their strange gods and words, and obvious pride in the tower-- their tower, so they claimed-- they took it amiss, though none could have foreseen what would come of it. Later, with the sword's terror upon them as they were taken inside, some hapless soul could still be heard invoking the Water-lord for mercy, though none came.
They never forgot. Nor their hatred of the grey-eyed men.
***
"...for in the days of the sojourn of Sauron in that land the hearts of well nigh all its people had been turned towards darkness. Therefore many of those who sailed east in that time and made fortresses and dwellings upon the coasts were already bent to his will..." (Of the Rings of Power and the Third Age, The Silmarillion)
"The Men of Númenor were settled far and wide on the shores and seaward regions of the Great Lands, but for the most part they fell into evils and follies. Many became enamoured of the Darkness and the black arts.." (The Window on the West, The Lord of the Rings)
"For with the aid and counsel of Sauron they multiplied... And they sailed now with power and armoury to Middle-earth, and they came no longer as bringer of gifts, nor even as rulers, but as fierce men of war. And they hunted the men of Middle-earth and took their goods and enslaved them, and many they slew cruelly upon their altars." (Akallabeth, The Silmarillion)