Frankie Muniz says he’s never done drugs, and at age 38, “I’ve still never had a sip of alcohol.”

Now a professional race car driver, the former child actor spoke about his

relationship with drugs and alcohol with actress and Jeopardy!

host Mayim Bialik on her podcast Breakdown. Muniz revealed that even though he starred on the hit Malcolm in the Middle from 2000 to 2006, and appeared in a slew of hit movies, he grew up largely outside of the realm of the Hollywood social scene.

“Were you raised in a tradition that did not drink?” Bialik, herself a former TV child star (who started playing the lead on Blossom at age 15) asked.

“Every family member in my family drinks. And that’s what they do for fun,” Muniz replied with a laugh.

“I tried to figure out why I didn’t, because I don’t have a reason. It wasn’t, like, a religion,” he shared, musing that it likely goes back to his “hatred towards time.”

“When I was 15, 16 years old, I felt like I was so old. I had experienced so much. And I had experienced a lot in my life in comparison to most 15 or 16 year olds. but even though I was in the entertainment business, I never saw it, meaning the drug use, any of that stuff. I somehow stayed away,” Muniz said about avoiding the young Hollywood party scene. “I just went to the set and did my thing.”

Frankie Muniz on Malcolm in the Middle.
Deborah Feingold/20th Century Fox TV/Kobal/Shutterstock

“I grew up thinking the only people who smoked pot were criminals…on episodes of Cops,” he said with a laugh. “Now I talk to my dad about it and my mom and they’re like, ‘We smoked pot every single day, you just didn’t know.’”

Now a father himself — he shares son Mauz Mosley with wife Paige — Muniz told Bialik that he was 18 the first time he was around alcohol.

“In my mind, I felt like I had made it so long without it, I was like, ‘Well, I’m not going to start now, even though I was only 18.’”

Frankie Muniz racing in 2016.
Albert L. Ortega/Getty

Muniz — who has struggled with memory loss and had a “mini-stroke” more than ten years ago — recalled people’s reactions when he would turn down alcohol, which he said influenced his decision to abstain.

“I couldn’t believe everyone’s reactions when I said I don’t drink. ‘Oh, we’re at a club, we have a bottle. You want a drink?’ ‘No, I don’t drink. I’m good with water.’ Everyone was either shocked or appalled that I didn’t drink. Or they’d be like, ‘Wow, that’s so strong of you.’”

“I was always kind of like, ‘It’s so weird that people have such a strong reaction to me not drinking,’” he said. “Like, who cares? I don’t care what you do. Why do you care so much about me?’ I don’t know if I made a conscious decision, but it just stuck with me that I was never gonna drink or do anything.”

“But now I’m 38 and I think I’m going to start tonight,” he quipped, adding, “Only kidding.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.