ChatGPT knows everything about its users. It will not be a good steward of that power.
"Many users may not be engaging with chatbots as if they’re tools, and might instead be treating them like a coworker or even as a friend — we tell our friends some very personal details." And they remember.
It probably shouldn’t be a surprise that chatbots like ChatGPT are profiling their users. Still, there are some surprising numbers here:
“A staggering 96 percent of memories were created unilaterally by the chatbot, according to the study, meaning that the users had little overt control over the data points that ChatGPT logged as their interactions wore on. Stored memories were also deeply revealing about users. About 28 percent of the memories examined contained personal data as defined by Europe’s strict General Data Protection Regulation standards, and 52 percent contained rich “psychological insights” about users.”
I think back to Gina Chua and Adiel Kaplan’s three-layer cake analogy for the AI stack. In their analysis, the consumption layer profiles the user; this new report makes clear how deep that profiling goes.
As I said in my commentary then:
“Profile data is accumulated, private, and usually non-portable. The consumption layer also represents the greatest risk to creators from platform business decisions, both in terms of the business models of those platforms and which creators they choose to feature. Open portability standards and protocols — combined with full user control of their data — can scale, create useful competition, and empower users in the process.”
Nothing here dissuades me from that view. To be as useful as possible, AI tools need to dig into our most personal information, make inferences about who we are and what’s interesting to us, and maintain highly personal profiles. Our AI agents are not some kind of trusted friend: they are products owned by a third party that needs to justify a trillion dollar valuation and may not have our best interests at heart. All the major vendors provide services used by the military and law enforcement. What happens if one of those agencies wants to get hold of that profile information? These companies simply cannot be trusted with our most private information, and should not have access to it.
But, of course, legislators have also been negotiated into a position of alignment with AI accelerationism. We need to gather all of this personal information about individuals in a supposed democracy, they argue, because otherwise China will win! These concerns seem to trump the fact that privacy is considered a fundamental human right, for very solid and defensible reasons: it makes dissent, organizing, a free press, and a well-lived life possible.
I am not arguing that there are not legitimate uses for AI or that the technology is not transformative. But there should not be an argument that we should let go of our core human values or the fundamental protections needed to maintain a robust democracy — of which privacy is one — because this tech is cool and we need it to grow really fast and by the way it’s also now propping up our economy. Trading privacy for growth is not a smart long-term strategy: it will bite us.
As individuals, we can protect ourselves by using different tools. But privacy is a group inoculation. If the chatbot is profiling its users, I suspect it’s also profiling the people its users are talking about. We need to demand better behavior from our vendors while taking steps to also demand it from our friends.
