While there were no doubt individuals of African descent in the Japan of the 1600s there is also well documented skin color prejudice in Japan from that period in which white appearing skin was a sign of high status and dark appearing skin was a sign of low status... from Wiki; "Japan's Edo period saw the start of a trend of women whitening their faces with rice powder as a "moral duty". That period began in 1603, as a result of civil wars that consolidated power in the Tokogawu Shogunate, the model for Clavel's Toranaga character.

I haven't yet seen the current production, I found the 1980 version unwatchable. If the new series shows Blackthorn's fellow prisoners in the Osaka prison awaiting execution, then an accurate portrayal should show far darker shin than that of the noble class he spends much of the book consorting with. Of course, if black skinned people were such a novelty that people 'broke down the doors of a residence to get a glimpse" of them, then no doubt they were a novelty indeed.