In 2025 Chat GPT revealed that, each week 1 million people talk about mental health issues with Chat GPT[1]. So, let's take that example to start this article. Lets assume you used Chat GPT exact same way. With Chat History and Training on, you pour your heart onto the AI while it silently listens for months. Personal anxieties, family names and private details are all told into the chat so the AI also remembers and knows the context. Now all this information like your linguistic patterns and even the anonymized fragments of your stories become “weights[2]” in the neural network which OpenAI legally uses all this to further make the AI better[3]. Now question arises how would you delete all this sensitive data? Is merely deleting the chat enough?
AI vs DPDP ACT
DPDP Act comes to rescue to protect the new “oil” of the modern era. Section 12 of the Act introduces right to erasure of the personal data. the provision stipulates deletion of data by a data fiduciary even if the consent to use and process has been granted earlier by the principal. This is an important provision of the act, envisioning the right to be forgotten[4]. But the issue arises with the AI. The provision is perfect when addressing the old way of processing and storing data like databases, cloud information etc. but not machine learning. Deleting data from databases or deleting chat is like picking strawberries from the garden but removing the data from a neural network is like removing strawberries from a smoothie Achieving this level of erasure is technically fraught. Further, this also gives big AI companies a loophole where they can remove the dataset for training but not address the model itself. Due to these concerns of privacy, there is a new paradigm shift called as “machine unlearning[5]” but remain unviable practically[6]. Generally, when trying to remove only a specific data point from a neural network or an AI model the new model performs worse than the previous model with information intact. But this issue can exponentially increase when trying to remove more data. This issue is called by experts as “catastrophic unlearning”.
Solutions:
- Privacy by design
Incorporating the principles of privacy while developing the model can be anonymized. Anonymization is the process of converting the data into anonymous information such that the data principal cannot be identified. But if there is any means which can lead to reidentification then it should be considered as personal data. ICO's "Guidance on AI and Data Protection"[7] also argue that if a model can be used to "re-identify" a person, the model is personal data. Court of Justice of European Union[8] was also of the same opinion. This case pertained to reidentification of Dynamic IP address through the ISP. The court was of the opinion that since there exists a legal route of reidentification therefore it is personal data and not anonymous. This case is also important as puts a question whether the “weights” in the neural network can also be used to reidentify a person. These “weights” can be a vulnerable point to privacy as techniques like model inversion attacks[9] (where original training data is reverse engineered) can be used to trace someone back, also sometimes these models can regurgitate the training text itself[10].
- Algorithmic Disgorgement
This solution can be a final kill switch where the design is itself flawed or illegal. In In the Matter of Everalbum[11] the case pertained a photo-storage app which used the private images of the users to develop facial recognition despite the assurance of not doing so. The Federal Trade Commission of US in this case not only ordered to delete the unlawfully collected photos and videos but also to delete models or algorithms which was made by that data. somewhat similar to mesne profit. But this principal was not followed in Germany's Hamburg Data Protection Authority (HmbBfDI) vs. Clearview AI (2021). Matthias Marx approached Data Protection Authority after he found out that Clearview AI, a company which has used AI to scrape tons of images from web and make a searchable database of biometric profile. In this case the DPA directed the company to delete the biometric hash which was made after analyzing the photos but only of the complainant and not every individual of Germany.
- User's awareness
Using algorithms as therapists[12], reveals a major data privacy issue. Users are sharing vulnerable information about themselves to these algorithms without knowing the implications of it. This shows that the users of these algorithms also need to be made aware of data privacy. Even though the DPDP Act talks about notice to the users, an average user of these models may remain unaware how of their sensitive information is converted into “weights” for future AI to train. Hence, awareness of masses and following good privacy habits also is important.
The introduction of DPDP Act was a much awaited moment coming after long judicial battle to strengthen the Right to privacy enshrined under Article 21. It has finally brought the concept of Right to Privacy into tangible statute. However, with rapid advancement of AI may leave the act redundant. The current state of the law is more apt for databases but not AI. Hence, the comparative analysis of privacy laws becomes important. It becomes important to look at countries like EU and US to see how their law is catching up in the fast-paced times. The companies need to incorporate privacy by design otherwise they risk an all-together discontinuation of the algorithm as illustrated in the article with In the Matter of Everalbum case. Privacy should be in the very DNA of every device that you use.
End Notes & References
https://psychology.org.au/insights/chatgpt-sees-one-million-users-in-mental-distress ↑
“Weights and bias can be interpreted as a system of knobs that we can manipulate to optimize our model — like when we try to tune our radio by turning the knobs to find the desired frequency. The main difference is that in a neural network, we have hundreds if not thousands of knobs to turn to achieve the final result.” From: https://medium.com/@theDrewDag/introduction-to-neural-networks-weights-biases-and-activation-270ebf2545aa ↑
https://help.openai.com/en/articles/7730893-data-controls-faq ↑
Justice K.S. Puttaswamy (Retd.) v. Union of India ↑
https://arxiv.org/pdf/2209.02299 ↑
https://arxiv.org/html/2403.07362v1#:~:text=A widely recognized exact unlearning,set [45, 21] . ↑
Guidance on AI and Data Protection 2025: Specifically the section on Individual Rights (accessible at https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/artificial-intelligence/guidance-on-ai-and-data-protection/) ↑
Patrick Breyer v. Bundesrepublik Deutschland (Case C-582/14) ↑
https://arxiv.org/pdf/1911.07135 ↑
https://www.mdpi.com/2673-2688/5/4/134?utm_source=chatgpt.com ↑
(FTC, 2021) ↑
Ibid 1 ↑
The Author is an LLM Student. Views are Personal