Observations Talmo Pereira From video game bots to leading-edge AI tools

What was your childhood like, and how did it influence your career path?

TP: When I was growing up in rural Brazil, my grandmother and uncle were constantly telling me to get off the computer.

I had built the computer myself, purchasing each part once I could afford it with my allowance. But money was tight in our family, so playing on the computer seemed like a waste of time.

When I was young, my mother left home and moved to the United States to find work. There, she delivered newspapers, cleaned houses, worked as a nanny, and sent as much money as she could back to us in Brazil. She wanted to build a better life for us, though it took longer than we had initially anticipated.

While she was away, my mom was constantly worried that she’d lose me to a street gang, which many of my peers were turning to for income and community. To ease her fears, I became very sheltered and nerdy. I didn’t go out a lot, and we couldn’t afford a lot.

In a way, computer games offered me an alternate life, one where I could live in a fantasyland and befriend other gamers from around the world. And the more time I spent at it, the more I saw ways to make the experience better. I learned to code so that I could automate the boring parts of online role-playing games by building avatars that could perceive the world and choose appropriate behaviors to achieve the objective—which, essentially, is what I still do today.

How is your work today related to your childhood experience with role-playing video games?

TP: My team and I created SLEAP, and we’re frequently surprised by its far-ranging applications. To name just a couple, we are currently using our machine learning-based tool to diagnose Alzheimer’s disease and ALS simply by analyzing body language. In a totally different project, we’re tracking plant root growth to help develop crops that are more resilient to climate change.

Now we’re reverse-engineering biological intelligence by mapping real-world movements onto digital avatars to simulate all sorts of scenarios—which, now that I think about it, is really just a super advanced version of my role-playing game characters back in Brazil.

What challenges did you face when you first moved to the United States?

TP: When I was 16, I was finally able to join my mom in Maryland after nearly 10 years of separation. I thought it would be my land of opportunity, but that wasn’t my initial experience. The teachers in my high school didn’t think I had a future in the sciences. I wasn’t recommended for advanced classes or given the same opportunities as other kids.

I eventually got myself into a high school research program at the National Institutes of Health. That was my big break—a chance to prove myself. I spent the summer working in a neuroscience lab and continued during the school year, even though I had to take two buses and the subway to get there.

When it was time to apply to college, I hit another roadblock. I realized that there was this whole system I didn’t know about, one that my peers had been preparing for since middle school. Nobody told me what I needed to do to get into a top-tier university.

How did you overcome those challenges?

TP: I was initially deeply disappointed by all the rejections. I ended up at the University of Maryland, Baltimore County, which offered fewer research opportunities than some other schools. But I was fortunate to be accepted to the Meyerhoff Scholars Program, an effort to increase diversity among future STEM leaders by supporting students who intend to pursue graduate or medical school.

My life would be totally different without that program. It starts with a highly disciplined boot camp, specifically formulated to bridge the gap in lack of institutional knowledge and give students like me the tools, opportunities, and confidence we need to get into and succeed in top graduate schools.

To supplement my education at UMBC, I also took advantage of summer research programs at the Broad Institute, MIT, and Caltech. I attended my first Society for Neuroscience meeting—a huge annual conference where I got so exhausted from trying to visit every poster that I had to find an empty room to take a nap. The meeting had a major impact on my career, as I woke up to find myself in a lecture by David J. Anderson, who I later interned with and who introduced me to the topics my lab still studies today.

Each of these experiences taught me a lot, but the most valuable part was finally feeling like I belonged. For the first time, I was being treated like an intellectual equal and a colleague. It was very different than my high school experience.

How did you come to invent SLEAP?

TP: By the time I was applying to graduate schools, I had a clear vision of what I wanted to accomplish: I wanted to find a way to look at natural, complex behavior and use computer vision and machine learning to extract and quantify patterns. I just needed a place that would let me do it.

This time, thanks in large part to the Meyerhoff Scholars Program, I had my pick of Ivy League schools. I ended up choosing Princeton University because I loved the vibe. I felt like I belonged, and my interests were being validated. I just wanted to do good science and dive deep into big questions.

Then I saw a documentary about the making of the animated movie Shrek, in which they used motion capture technology in the real world to generate detailed, 3D images. Around that same time, deep learning was coming around, as was markerless capture—the ability to capture movement with a camera without putting invasive tags on the subject.

I proposed this approach to my thesis advisors, but they didn’t think it would be feasible. The problem was “Big Data.” For markerless capture and deep learning to work as a research tool, we would be generating more data than we had the capacity to store, share, and use.

But I did it anyway, just in secret! Within a few months, we had a working product—the precursor to what is now SLEAP. After some initial challenges, it started to take off, and labs around the world have been using it ever since. When I attended my next Society for Neuroscience meeting, we had published SLEAP and it was a totally different experience—people stopped me on the streets to ask about it!

What brought you to the Salk Fellows program?

TP: Despite that success, I was uncertain about my future in academia. Typically, the next step would be to apply for a postdoctoral training position, working in another lab at another university. Then, hopefully, after an unknown number of years, I’d apply for faculty positions. The trouble with that path is you’re living on a relatively low stipend for many years as a trainee and there’s no guarantee that it’ll pay off in the end with a stable, well-paying faculty position. My mother was also beginning to have health issues at that time and was still working as a nanny. It was my turn to help support my family, and I wasn’t convinced that academia was going to be the way to do it.

I had recently done an internship at Google AI, so I was planning to apply to similar industry jobs after I completed my PhD. Then Salk Professor Kay Tye gave a seminar at Princeton and mentioned that she was chairing the search for the next Salk Fellow. The program supported a small number of scientists to skip postdoctoral training and move directly into faculty-level roles. My advisor told her about me, and she encouraged me to apply. I’m thankful to both of them, as well as the postdocs in my lab, who helped me navigate the process of applying to a faculty-level job.

The Salk Fellows program is ideal for me because I get the best of both worlds—the academic freedom to follow my ideas as well as financial and career stability I need to support my family.

What’s next?

TP: Historically, neuroscientists have studied the brain but largely ignored the rest of the body. Yet a large portion of the brain is dedicated to getting the body to move appropriately—toward food and away from danger, for example. To fully understand the brain, we must study it in the context of the whole body.

We’re now doing that by using SLEAP’s output to reverse-engineer the neural codes that produce those movements. In other words, what would have to be happening in the brain for that movement to occur? To do this, we’re building AI-powered digital twins of real-life animals. These “avatars” are opening a whole new set of possible applications. With our digital twins, we can come up with hypotheses and perform virtual experiments. For example, what happens to behavior if we remove a part of the brain? My team just received our first big National Institutes of Health grant to pursue this, and we’re excited to see where it will take us.

I’m also proud to serve as chair of Salk’s Justice, Equity, Diversity, and Inclusion (J.E.D.I.) Council. We’re a group of faculty members, graduate students, postdoctoral researchers, and staff members who support Salk’s Office of Equity & Inclusion and advocate for these issues across the campus and beyond. As you might imagine, given my background, I feel very strongly that supporting J.E.D.I. efforts, including Salk’s Summer Undergraduate Research Fellowship (SURF) program, is of paramount importance.

You never know where the next big idea will come from, and breakthrough science is  often made possible by encouraging new voices and perspectives.