Mind your language: The risks of using AI-powered chatbots like ChatGPT in an organization (2023)

Victor Dey

Mind your language: The risks of using AI-powered chatbots like ChatGPT in an organization (1)

Image Credit: Getty Images

Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More

Millions of users have flocked to ChatGPT since its mainstream launch in November 2022. Thanks to its exceptional human-like language generation capabilities, its aptitude for coding software, and its lightning-fast text analysis, ChatGPT has quickly emerged as a go-to tool for developers, researchers and everyday users.

(Video) Mind your language: The risks of using AI-powered chatbots like ChatGPT in an organization.

But as with any disruptive technology, generative AI systems like ChatGPT come with potential risks. In particular, major players in the tech industry, state intelligence agencies and other governmental bodies have all raised red flags about sensitive information being fed into AI systems like ChatGPT.

>>Don’t miss our newest special issue: Data centers in 2023: How to do more with less.<<

The concern stems from the possibility of such information eventually leaking into the public domain, whether through security breaches or the use of user-generated content to “train” chatbots.


Transform 2023

Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.

(Video) Beyond ChatGPT: Stuart Russell on the Risks and Rewards of A.I.

Register Now

In response to these concerns, tech organizations are taking action to mitigate the security risks associated with large language models (LLMs) and conversational AI (CAI). Several organizations have opted to prohibit the use of ChatGPT altogether, while others have cautioned their staff about the hazards of inputting confidential data into such models.

ChatGPT: A scary AI out in the open?

The AI-powered ChatGPT has become a popular tool for businesses looking to optimize their operations and simplify complex tasks. However, recent incidents have underscored the potential dangers of sharing confidential information through the platform.

In a disturbing development, three instances of sensitive data leakage via ChatGPT were reported in less than a month. The most recent occurred last week. Smartphone manufacturer Samsung was embroiled in controversy when Korean media reported that employees at its main semiconductor plants had entered confidential information, including highly-sensitive source code used to resolve programming errors, into the AI chatbot.

Source code is one of any technology firm’s most closely guarded secrets, as it serves as the foundational building block for any software or operating system. Consequently, prized trade secrets have now inadvertently fallen into the possession of OpenAI, the formidable AI service provider that has taken the tech world by storm.

Despite requests by VentureBeat, Samsung did not comment on the matter, but sources close to the firm revealed that the company has apparently curtailed access for its personnel to ChatGPT.

Other Fortune 500 conglomerates, including Amazon, Walmart and JPMorgan, encountered similar instances of employees accidentally pushing sensitive data into the chatbot.

(Video) The A.I. Dilemma - March 9, 2023

Reports of Amazon employees using ChatGPT to access confidential customer information prompted the tech behemoth to swiftly restrict the use of the tool and sternly warn workers not to input any sensitive data into it.

Knowledge without wisdom

Mathieu Fortier, director of machine learning at AI-driven digital experience platform Coveo, said that LLMs such as GPT-4 and LLaMA suffer from several imperfections and warned that despite their prowess in language comprehension, these models lack the ability to discern accuracy, immutable laws, physical realities and other non-lingual aspects.

“While LLMs construct extensive intrinsic knowledge repositories through training data, they have no explicit concept of truth or factual accuracy. Additionally, they are susceptible to security breaches and data extraction attacks, and are prone to deviating from intended responses or exhibiting ‘unhinged personalities,’” Fortier told VentureBeat.

Fortier highlighted the high stakes involved for enterprises. The ramifications can severely erode customer trust and inflict irreparable harm to brand reputation, leading to major legal and financial woes.

Following in the footsteps of other tech giants, Walmart Global Tech, the technology division of the retail behemoth, has implemented measures to mitigate the risk of data breaches. In an internal memo to employees, the company directed staff to block ChatGPT after detecting suspicious activity that could potentially compromise the enterprise’s data and security.

A Walmart spokesperson stated that although the retailer is creating its own chatbots on the capabilities of GPT-4, it has implemented several measures to protect employee and customer data from being disseminated on generative AI tools such as ChatGPT.

“Most new technologies present new benefits as well as new risks. So it’s not uncommon for us to assess these new technologies and provide our associates with usage guidelines to protect our customers’, members’ and associates’ data,” the spokesperson told VentureBeat. “Leveraging available technology, like Open AI, and building a layer on top that speaks retail more effectively enables us to develop new customer experiences and improve existing capabilities.”

Other firms, such as Verizon and Accenture, have also adopted steps to curtail the use of ChatGPT, with Verizon instructing its workers to restrict the chatbot to non-sensitive tasks, and Accenture implementing tighter controls to ensure compliance with data privacy regulations.

How ChatGPT uses conversational data

Compounding these concerns is the fact that ChatGPT retains user input data to train the model further, raising questions about the potential for sensitive information being exposed through data breaches or other security incidents.

OpenAI, the company behind the popular generative AI models ChatGPT and DALL-E, has recently implemented a new policy to improve user data privacy and security.

As of March 1 of this year, API users must explicitly opt in to sharing their data for training or improving OpenAI’s models.

In contrast, for non-API services, such as ChatGPT and DALL-E, users must opt out if they do not wish to have their data used by OpenAI.

“When you use our non-API consumer services ChatGPT or DALL-E, we may use the data you provide us to improve our models,” according to the OpenAI blog, recently updated. “Sharing your data with us not only helps our models become more accurate and better at solving your specific problem, it also helps improve their general capabilities and safety … You can request to opt-out of having your data used to improve our non-API services by filling out this form with your organization ID and email address associated with the owner of the account.”

This announcement comes amid concerns about the risks described above and the need for companies to be cautious when handling sensitive information. The Italian government recently joined the fray by banning the use of ChatGPT across the country, citing concerns about data privacy and security.

(Video) The Real Danger Of ChatGPT

OpenAI states that it removes any personally identifiable information from data used to improve its AI models, and only uses a small sample of data from each customer for this purpose.

Government warning

The U.K.’s Government Communications Headquarters (GCHQ) intelligence agency, through its National Cyber Security Centre (NCSC), has issued a cautionary note about the limitations and risks of large language models (LLMs) like ChatGPT. While these models have been lauded for their impressive natural language processing capabilities, the NCSC warns that they are not infallible and may contain serious flaws.

According to the NCSC, LLMs can generate incorrect or “hallucinated” facts, as demonstrated during Google’s Bard chatbot’s first demo. They can also exhibit biases and gullibility, particularly when responding to leading questions. Additionally, these models require significant computational resources and vast amounts of data to train from scratch, and they are vulnerable to injection attacks and toxic content creation.

“LLMs generate responses to prompts based on the intrinsic similarity of that prompt to their internal knowledge, which memorized patterns seen in training data,” said Coveo’s Fortier. “However, given they have no intrinsic internal ‘hard rules’ or reasoning abilities, they can’t comply with 100% success to constraints that would command them not to disclose sensitive information.”

He added that despite efforts to reduce the generation of sensitive information, if the LLM is trained with such data, it can generate it back.

“The only solution is not to train these models with sensitive material,” he said. “Users should also refrain from providing them with sensitive information in the prompt, as most of the services in place today will keep that information in their logs.”

Best practices for safe and ethical use of generative AI

As companies continue to embrace AI and other emerging technologies, it will be crucial to ensure proper safeguards to protect sensitive data and prevent inadvertent disclosures of confidential information.

The actions taken by these companies highlight the importance of remaining vigilant when using AI language models such as ChatGPT. While these tools can greatly improve efficiency and productivity, they pose significant risks if not used appropriately.

“The best approach is to take every new development in the raw advancement of language models and fit it into an enterprise policy-driven architecture that surrounds a language model with pre-processors and post-processors for guard rails, fine-tune them for enterprise-specific data, and then maybe even go to on-prem deployment as well,” Peter Relan, chairman of conversational AI startup Got It AI, told VentureBeat. “Otherwise, raw language models are too powerful and sometimes harmful to deal with in the enterprise.”

For his part, Prasanna Arikala, CTO of Nvidia-backed conversational AI platform Kore.ai, says that moving forward, it will be essential for companies to limit LLMs’ access to sensitive and personal information to avoid breaches.

“Implementing strict access controls, such as multifactor authentication, and encrypting sensitive data can help to mitigate these risks. Regular security audits and vulnerability assessments can also be conducted to identify and address potential vulnerabilities,” Arikala told VentureBeat. “While LLMs are valuable tools if used correctly, it is crucial for companies to take the necessary precautions to protect sensitive data and maintain the trust of their customers and stakeholders.”

It remains to be seen how these regulations will evolve, but businesses must remain vigilant and informed to stay ahead of the curve. With the potential benefits of generative AI come new responsibilities and challenges, and it is up to the tech industry to work alongside policymakers to ensure that the technology is developed and implemented responsibly and ethically.

VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.


What are the risks of using ChatGPT? ›

Malicious code generation

However, the chatbot's ability to generate code could also be used for harm. ChatGPT code can be used to produce quick code, allowing attackers to deploy threats quicker, even with limited coding knowledge.

What are the risks of using AI chatbots? ›

But the way these products work—receiving instructions from users and then scouring the internet for answers—creates a ton of new risks. With AI, they could be used for all sorts of malicious tasks, including leaking people's private information and helping criminals phish, spam, and scam people.

What is the disadvantage of AI chatbot? ›

The extent of Resolving Queries

One of the major drawbacks of chatbots is the number of queries it can resolve. At a certain point in time, it will have to connect to an actual human to resolve the issues. They also have limited replies and solutions which can leave a customer unsatisfied.

Should you use ChatGPT? ›

ChatGPT risks

Besides the risks that ChatGPT directly poses to you as a user, there are other major risks you should consider. ChatGPT has the potential to be used by attackers to trick and target you and your computer. For example, fraudsters could use ChatGPT to quickly create spam and phishing emails.

What is the difference between chatbots and ChatGPT? ›

Differences between Chatbot and ChatGPT

✅Personalization and Sophistication: Chatbots are typically pre-programmed with a limited set of responses, whereas ChatGPT is capable of generating responses based on the context and tone of the conversation. This makes ChatGPT more personalized and sophisticated than chatbots.

What are the 4 risks dangers of AI? ›

Some of the biggest risks today include things like consumer privacy, biased programming, danger to humans, and unclear legal regulation.

What are the negative effects of AI? ›

AI systems learn from training data, by which they learn to make decisions. These data may contain biased human decisions or represent historical or social inequalities. Likewise, the bias could enter directly into the programming of the human being who designs the system, based on their own biases.

What are the pros and cons of ChatGPT? ›

Pros of Chat GPT include improved natural language understanding, faster response times, and the ability to generate more natural-sounding conversations. Cons of Chat GPT include difficulty training models to respond appropriately to a wide range of topics and potential bias from the data used to train it.

What are the three advantages and three disadvantages of AI? ›

The advantages range from streamlining, saving time, eliminating biases, and automating repetitive tasks, just to name a few. The disadvantages are things like costly implementation, potential human job loss, and lack of emotion and creativity.

Is ChatGPT safe to use at work? ›

ChatGPT is a third-party system that absorbs information into its data. Even if AI bot security isn't compromised, sharing any confidential customer or partner information may violate your agreements with customers and partners, since you are often contractually or legally required to protect this information.

Is chat GTP safe? ›

No, Chat GPT is not confidential.

Chat GPT logs every conversation, including any personal data you share, and will use it as training data. Open AI's privacy policy states that the company collects personal information included in “input, file uploads, or feedback” users provide to Chat GPT and its other services.

What are the ethical issues with ChatGPT? ›

Many leading cultural commentators have voiced their concerns about the ethical implications of AI language models like Chat GPT. One of the foremost concerns is the issue of bias. AI models learn from the data they are trained on, and if this data contains biases, the AI may inadvertently perpetuate them.

What makes ChatGPT different? ›

ChatGPT is an app; GPT is the brain behind that app

ChatGPT is a web app (you can access it in your browser) designed specifically for chatbot applications—and optimized for dialogue. It relies on GPT to produce text, like explaining code or writing poems. GPT, on the other hand, is a language model, not an app.

Is ChatGPT a new AI chatbot? ›

ChatGPT is a general-purpose chatbot that uses artificial intelligence to generate text after a user enters a prompt, developed by tech startup OpenAI. The chatbot uses GPT-4, a large language model that uses deep learning to produce human-like text.

What are the biggest risks of AI? ›

Here are the top 5 AI risks that business leaders should watch out for:
  • Risk of Disruption. Artificial intelligence will disrupt existing business models and markets like no technology before it. ...
  • Cybersecurity Risk. ...
  • Reputational Risk. ...
  • Legal Risk. ...
  • Operational Risk.
Apr 9, 2023

What is a high risk AI system? ›

Product manufacturers and certain sectors of the technology industry that use artificial intelligence in their products are the key targets of the EU's AI Act. These categories of use of AI are specifically called out as high-risk and they will be subject to a significant new conformity assessment regime.

What are the risks of using AI at work? ›

Over-reliance on AI tools may lead to deskilling of some professions. Increased monitoring may lead to micromanagement and thus to stress and anxiety. A perception of surveillance may also lead to stress. Controls for these include consultation with worker groups, extensive testing, and attention to introduced bias.

What is high risk AI examples? ›

High risk. AI systems identified as high-risk include AI technology used in: critical infrastructures (e.g. transport), that could put the life and health of citizens at risk; educational or vocational training, that may determine the access to education and professional course of someone's life (e.g. scoring of exams ...

What are the risk and benefits of AI? ›

AI can offer many benefits, such as improving efficiency, accuracy, accessibility, and quality of life. However, AI also poses some risks, such as displacing jobs, invading privacy, creating bias, and undermining human dignity.

What are the two types of problems in AI? ›

The most prevalent problem types are classification, continuous estimation and clustering. I will try and give some clarification about the types of problems we face with AI and some specific examples for applications.

What is the negative impact of AI on sustainability? ›

But AI systems also raise sustainability concerns linked to the natural resources they consume such as electricity and water, and the carbon emissions they produce. The rise of deep learning and large language models has also dramatically increased the amount of compute capacity AI systems need.

Why is artificial intelligence a threat? ›

AI could harm the health of millions via the social determinants of health through the control and manipulation of people, the use of lethal autonomous weapons and the mental health effects of mass unemployment should AI-based systems displace large numbers of workers.

What are the cons of ChatGPT in healthcare? ›

One is the risk of data breaches and unauthorized access to sensitive medical information. ChatGPT must be designed with robust security measures, such as encryption, access controls, and user authentication, to protect patient data. Another concern is the potential for bias in ChatGPT algorithms.

What is pros advantage and cons disadvantage? ›

The pros and cons of something are its advantages and disadvantages, which you consider carefully so that you can make a sensible decision. They sat for hours debating the pros and cons of setting up their own firm.

What are the Disadvantages of GPT chat? ›

Cons: ChatGPT is an AI model and therefore cannot provide the same level of human empathy and understanding as a real person. ChatGPT may sometimes provide inaccurate or irrelevant responses, especially when it encounters a question that it has not been trained on.

What is the advantage and disadvantages? ›

A disadvantage is the opposite of an advantage, a lucky or favorable circumstance. At the root of both words is the Old French avant, "at the front." Definitions of disadvantage. the quality of having an inferior or less favorable position. Antonyms: advantage, vantage.

Will AI help the world or hurt it? ›

The creation of a human-level AI would certainly have a transformative impact on our world. If the work of most humans could be carried out by an AI, the lives of millions of people would change. The opposite, however, is not true: we might see transformative AI without developing human-level AI.

What are 2 advantages of AI? ›

What are the advantages of Artificial Intelligence?
  • AI drives down the time taken to perform a task. ...
  • AI enables the execution of hitherto complex tasks without significant cost outlays.
  • AI operates 24x7 without interruption or breaks and has no downtime.
  • AI augments the capabilities of differently abled individuals.

What are the three limitations of AI? ›

However, despite its many advantages, there are also several limitations to the technology that must be taken into consideration… Some of these limitations include the lack of common sense, transparency, creativity, emotion and safety and ethical concerns.

What are three 3 benefits of artificial intelligence AI technology in healthcare? ›


AI-equipped technology can analyse data much faster than any human, including clinical studies, medical records and genetic information that can help medical professionals come to a diagnosis. AI can automate many routine tasks, such as maintaining records, data entry and scan analysis.

Can ChatGPT leak data? ›

Further Investigation Showed ChatGPT Plus Subscriber Information Leak. Upon further investigation, OpenAI discovered another instance of the same bug leaking data from the AI chatbot.

Is ChatGPT safe from hackers? ›

Meta's security team said it found hackers software that claimed to offer ChatGPT-based tools via browser extensions and online app stores that contained malware designed to give hackers access to people's devices, Meta said.

Is ChatGPT better than Google? ›

While both ChatGPT and Google have their own unique capabilities, they are used for different purposes. ChatGPT is a sophisticated AI chatbot that is capable of understanding and responding to natural language, while Google is a powerful search engine that is used for finding specific information on the Internet.

Is ChatGPT traceable? ›

It is now completely possible for universities to detect ChatGPT and many other AI content generators. If work is submitted through a university's learning management system, such as Turnitin, AI and plagiarism detection should happen.

Is ChatGPT ethical for students? ›

Universities all agree this is a form of cheating, and students giving this a go are opening themselves up to serious academic penalties. They're also not forming any original opinions in their future area of expertise and turning in some pretty poorly written assignments.

What are the three main ethical issues? ›

Here are some of the ethical issues in business and real-world cases of how these ethical issues have affected companies.
  • Unethical Accounting. ...
  • Social Media Ethics. ...
  • Harassment and Discrimination. ...
  • Health and Safety. ...
  • Technology/Privacy.

What are the risks of generative AI? ›

Generative AIs may also generate inappropriate or offensive content, which could harm a corporation's reputation or cause legal issues if shared publicly. This could occur if the AI model is trained on inappropriate data or if it is programmed to generate content that violates laws or regulations.

What is the data leak via ChatGPT? ›

According to OpenAI, a vulnerability in the Redis open-source library used by ChatGPT meant that “some users” were able to see “titles from another active user's chat history” as well as potentially being able to view the first message of a new conversation if both users were active at the same time.

What is the biggest risk of AI? ›

The 5 Biggest Threats of Artificial Intelligence for Humans
  • Hallucinations. No, we're not talking about a Hunter S. ...
  • Cyber Security Concerns. AI tools are getting pretty good at coding, which is both good and bad news for developers everywhere. ...
  • Upending the Labor Market. ...
  • Roko's Basilisk.
May 4, 2023

What is the danger of data leakage? ›

Data leaks can reveal everything from social security numbers to banking information. Once a criminal has these details, they can engage in all types of fraud under your name. Theft of your identity can ruin your credit, pin you with legal issues, and it is difficult to fight back against.

What is data leakage risk? ›

The unauthorized transmission of data from an organization to any external source is known as data leakage. This data can be leaked physically or electronically via hard drives, USB devices, mobile phones, etc., and could be exposed publicly or fall into the hands of a cyber criminal.

Will ChatGPT data be updated? ›

GPT-5 will follow in late 2023, according to some reports. It might turn out to be the most important ChatGPT update to date. But OpenAI said in mid-April 2023 that it's not training the nex-gen model.


1. Top 10 ChatGPT Secrets - Use Responsibly!
(Ai Guy)
2. Did ChatGPT Become Useless Already?!
(Simon Høiberg)
3. Language Learning with ChatGPT: Exploring Opportunities and Risks
(Joe Dale)
4. ChatGPT RISKS: 8 Reasons to NOT Trust OpenAI & What to Do About It
(Bill Hauser)
5. Coding using ChatGPT AI broke me
(Nick Chapsas)
6. How AI Could Save (Not Destroy) Education | Sal Khan | TED


Top Articles
Latest Posts
Article information

Author: Errol Quitzon

Last Updated: 28/10/2023

Views: 6142

Rating: 4.9 / 5 (59 voted)

Reviews: 82% of readers found this page helpful

Author information

Name: Errol Quitzon

Birthday: 1993-04-02

Address: 70604 Haley Lane, Port Weldonside, TN 99233-0942

Phone: +9665282866296

Job: Product Retail Agent

Hobby: Computer programming, Horseback riding, Hooping, Dance, Ice skating, Backpacking, Rafting

Introduction: My name is Errol Quitzon, I am a fair, cute, fancy, clean, attractive, sparkling, kind person who loves writing and wants to share my knowledge and understanding with you.