KEENE, N.H. (AP) — The husband of Transportation Secretary Pete Buttigieg said Thursday he hasn’t heard from Mike Pence in the days since the former vice president mocked his family, but will continue to hold him accountable for his words.

Chasten Buttigieg was asked on ABC’s “The View” whether Pence contacted him after joking at a dinner for journalists and politicians that Pete Buttigieg took “maternity leave” after he and his husband adopted newborn twins.

“Pete is the only person in human history to have a child and everyone else gets postpartum depression,” Pence said Saturday at the Gridiron Dinner.

Chasten Buttigieg, who said he doesn’t expect to hear from Pence, called the remarks “part of a much bigger trend attacking families.”

“I spoke up because we all have an obligation to hold people accountable for when they say something wrong, especially when it’s misogynistic, especially when it’s homophobic, and I just don’t take that when it’s towards my family, and I don’t think anyone else would, especially when you bring a very small, medically fragile child into it,” he said.

Pence defended his comments Thursday night after speaking at a Republican Party dinner in New Hampshire.

“The Gridiron Dinner is a roast. I had a lot of jokes directed to me, and I directed a lot of jokes to Republicans and Democrats,” he told reporters. “The only thing I can figure is Pete Buttigieg not only can’t do his job, but he can’t take a joke.”