Gemini’s new beta feature provides proactive responses based on photos, emails, and more

Google announced on Wednesday that it is launching a new beta feature in its Gemini app that lets its AI assistant customize its responses by connecting to the Google ecosystem, starting with Gmail, Photos, Search, and YouTube history.

Although Gemini can already extract information from the application, it can now reason in your data to provide proactive results, such as connecting a thread in an email to a video you watch. Google says this means Gemini understands the context without being told where to look.

The tech giant notes that this beta experience, called Personal Intelligence, is turned off by default, as users have the option to choose if and when to connect Google apps to Gemini. Of course, not everyone wants AI to see their photos and YouTube history. If you decide to connect your app, Gemini will only use Personal Intelligence when it determines it will be helpful, Google said.

“Personal Intelligence has two core strengths: reasoning in complex sources and extracting specific details from, say, an email or a photo to answer your questions,” wrote Josh Woodward, VP, Gemini applications, Google Labs, and AI Studio, in a post a blog. “It often combines this, working with text, photos and video to give a unique answer.”

Woodward shared an example of when he was in line at a tire store and couldn’t remember the size of his car’s tires. While most AI chatbots can determine a car’s tire size, Woodward says Gemini can go a step further by providing customized responses. In that case, Gemini recommended all-weather tires after recognizing a photo of the family’s trip on Google Photos. Woodward also said he forgot the license plate number, but Gemini was able to pull the number from the picture in Photos.

“I also got great tips for books, shows, clothes and travel,” Woodward wrote. “Just this week, it’s amazing to plan our upcoming spring break. By analyzing our family’s interests and past trips in Gmail and Photos, they avoid tourist traps. Instead, they recommend overnight train trips and special board games that we can do on the road.

Google says it has fences for sensitive topics, as Gemini will avoid proactive assumptions about sensitive data like health. However, the tech giant also notes that Gemini will discuss this data if you ask.

Additionally, Gemini doesn’t train directly on your Gmail inbox or Google Photos library. However, it is training in specific directions in Gemini and his model response. In the example above, street photos, license plate images in Photos, and emails in Gmail are not directly used to train the model. They are only referenced to generate responses, Google said.

Personal Intelligence is rolling out to Google AI Pro and AI Ultra customers in the US.

Google provides a list of examples to try, including “Help me plan a weekend in (city is New York) based on what I like,” “Recommend some documentaries based on what I like,” or “Based on my Gmail deliveries and grocery receipts, search history, and YouTube viewing history, recommend 5 YouTube channels that match my style of cooking or meal preparation.”