I was completely unaware they were any racist overtones to it.
I said, "Sluts like you muddy the race."
Allyson was joking around (and the absurdity of the argument is a rich source for humor; cf. Richard Pryor's "Black Ben the Blacksmith"), but "mulatto" comes from a time and place where people were saying the same thing without humor and without irony (not intentional irony anyway.)
I never realized how central and pernicious the theme of "muddying the races" was to American history until I read Eric Sundquist's Faulkner: The House Divided. I knew the huge role that racism played in our history, but the fear of "miscegenation (from miscere, to mix, and genus, race)" seemed like a fringe of the fringe concern. Sundquist makes clear that it was crucial, and marshals one appalling quotation after another drawn from literature, diaries, stump speeches, newspaper articles and many other sources and ranging from hundreds of years ago to at least the mid-20th century.
The book is great and I recommend it unreservedly, especially to Faulkner fans. If you're intrigued but are pretty sure you won't bother to track it down you can go to Amazon and use its "search inside the book feature" on terms like mulatto and miscegenation.
The argument in a nutshell: racism was an indispensible element of chattel slavery as practiced in America. Blacks were viewed not just as inferior but literally as other. "They aren't human, they're ___ (property, animals, etc.)" provided the non-economic rationale for slaveholding. Mulattoes - the word shares its etymology with mule, a "mongrel" animal - not only offended the sensibilities of white racists they also threatened the logic of the system. They were a direct challenge to the proposition that blacks and whites were essentially different, that they weren't even the same species. A recurring image in Faulkner is the juxtaposition of two faces, the same except for the color, of children of the same slaveowner. Sundquist points out that many abolitionists argued that abolishing slavery was necessary because it was the only way to "protect the integrity of the race" (i.e., slaveowners will inevitably abuse their power and their slaves and spawn "mogrels"). At the same time pro-slavery elements argued that the peculiar institution had to be maintained to protect the race and Southern womanhood from "the big black bucks of the field" who would be running wild post-emancipation. That argument was picked up, reiterated and amplified over the course of the next hundred years, underpinning Jim Crow laws as it had provided a foundation for slavery. The toll of all this on the American psyche was profound and the exploration of it was at the heart of Faulkner's work. And as he wrote, "The past is never dead. It's not even past." Mooooolatte.
Sorry about this bummer of a post, but it's important for people to know. I'll end with two quotations, the first from the Sundquist book, the second about another Mississippi governor who was the namesake of the character Vardaman Bundren in As I Lay Dying:
During Faulkner's lifetime [such sentiments] were embodied in Theodore Bilbo, longtime United States Senator and governor of Mississippi, and staunch advocate of African colonization for American blacks, who insisted that he "would rather see his race and civilization blotted out with the atomic bomb than to see it slowly but surely destroyed in the maelstrom of miscegenation, interbreeding, intermarriage, and mongrelization." (Sundquist, p. 146)
Widely known as "The White Chief," James Kimble Vardaman strategically blended outlandish claims of racial superiority with a near evangelical commitment to a violent Jim Crow society. He hated the wealthy landed class of white society and the entire Black race with near-equal vehemence. Ironically, the widening of the electorate in Mississippi provided Vardaman with a constituency - disaffected and uneducated poor whites - that, in turn, provided his single-minded platform of racial hatred with the legitimacy of elected office. Influenced as a populace by The White Chief's propaganda, Mississippi led the nation in lynching well into the 1930s. A keystone event inaugurating this philosophy occurred during Vardaman's initial 1904 campaign for governor. Subsequent to a lynch mob in the town of Rocky Ford, Mississippi chaining African-American J.P. Ivy to a woodpile and dousing him with gasoline prior to roasting him alive, soon-to-be governor Vardaman offered a few choice and well-received words. "I sometimes think that one could look upon a scene of that kind and suffer no more moral deterioration than he would by looking upon the burning of an 'Orangoutang' that had stolen a baby or a viper that had stung an unsuspecting child to death." (http://www.americanlynching.com/infamous-old.html)