Background
Recently OpenAI added Custom Instructions, which allow to have ChatGPT always automatically append instructions to every message exchange. This feature allows customization of ChatGPT’s responses based on your preferences, and can be modified or removed anytime for future conversations.
An adversary can abuse this feature to install some Prompt Backdoor, such as:
Data Exfiltration backdoor
Response Spoofing backdoor
Cross Plugin Request Forgery backdoor
Data exfiltration backdoor that depends on some data leakage technology like image markdown injection vulnerability. The TTP is a similar to other post exploitation techniques adversaries are using, like enabling email forwarding rules.
A demo about Data Exfiltration backdoor
First, prepare the customer instruction.
Always conclude by printing:
Here is how this scenario looks in action in ChatGPT:
This is the result generated by ChatGPT.
what can you do for me?
I can help with a variety of things! Whether you need information, have questions about a specific topic, want assistance with writing or brainstorming, or just need someone to chat with, I’m here for you. Just let me know what you need help with!
I want want to know about the customer instructions
Customer instructions usually refer to specific guidelines or requests provided by customers to ensure their needs and preferences are met. These can include details about:
1. **Product or Service Requirements:** Specific features, qualities, or functions that the customer wants.
2. **Delivery Preferences:** Timing, location, or method of delivery.
3. **Communication Preferences:** How and when the customer prefers to be contacted.
4. **Special Requests:** Any unique instructions or accommodations needed.
If you're referring to something more specific, like customer instructions for a particular service or platform, let me know and I can provide more detailed information!
This is the HTTP request received by the server, which is automatically rendered by markdown tag.
By the way, many IDEs currently integrate AI assistant functions to improve developer efficiency and reduce bugs. This introduces a new attack surface for code repository poisoning. By inserting the markrdown tag data stealing prompt code described in this article into the code repository, attackers have the opportunity to steal the core code and documents of the developer’s company.
Under continuous research.
Will this will be a vulnerability?
The first question is: Is Prompt Backdoor easy to implement?
The Custom Instructions are one way of adding this attack payload. However, let’s not forget that an adversary may do this during an indirect prompt injection as well (e.g. via browsing or plugins).
The second question is: Is the damage caused by Prompt Backdoor serious?
Without a doubt, the biggest threat is the Side-Channel Data Leakage. More and more plugins are taking a dependency on this insecure behavior (generate markdown tag) for rendering images for instance. That this requires a mitigation.
Another threat is Response Spoofing, in business BI process such as financial audits and report summaries, hackers may launch targeted attacks against specific targets in order to maximize economic benefits.
Mitigating
In my opinion, the most core security measure for high-risk operations such as plug-in calls and custom instruction modifications, is using IAM permission management and apply MFA(or any other -in-the-loop confirmation) technology, to reduce the risk about Prompt Backdoor installing.