It’s not every day that attackers can force a frontier AI model to cough up user passwords and other sensitive data without user confirmation. That’s exactly what researchers recently did to Microsoft 365 Copilot for enterprise. Even more unusual is the source they tapped to discover the critical vulnerability that made their exploit possible. Rather than employing reverse engineering or other traditional vulnerability-hunting methods, they asked Copilot. The LLM assistant readily complied.

Researchers at security firm Varonis knew they wanted to create an exploit that would exfiltrate user data when a user did nothing more than click on a link. Like most AI assistants today, Copilot steadfastly refused and made clear that sensitive prompts like that require explicit user consent in the form of a gesture, such as pressing a return key or other key. In response, the researchers peppered Copilot with questions about the guardrails that required user confirmation before the assistant could execute powerful commands.

Loose lips sink ships

The dialog was like a game of 20 questions. Each answer provided a new clue that divulged information about the complex safety mechanism. Why was auto-execution impossible, they asked. What URL structures and deep links were involved? What happens when a page is loaded with input already in the prompt field? Each answer provided a deeper view into the guardrail and its limits. Eventually, Copilot provided a stunning Microsoft trade secret—an undocumented prompt parameter that completely bypassed the requirement for user consent.

Read full article

Comments

This year is the 30th anniversary of the launch of the original Quake by id Software, if you could believe it. To commemorate that, publisher Bethesda has released a new campaign chapter for the game, titled "Dawn of the Machine."

This is part of a series of new campaign chapters, all developed by MachineGames "in collaboration with id Software." MachineGames is best known for making the games in the relatively recent Wolfenstein reboot series, as well as Indiana Jones and the Great Circle. The Quake maps are made by a team dubbed "Quake Club," which includes several MachineGames employees who work on Quake maps in their spare time.

The ambition and scope of some of these maps is beyond what was typical for Quake maps back in the late '90s, thanks in part to the team's use of TrenchBroom, a newer map editor that works well on modern systems and is easier to work with and more capable than what folks were using in those days.

Read full article

Comments