[Valliappan Muthukaruppan]

At Exelon we have started to explore LLMs. We are working with our security team to identify the best use case and what are the security implications involved in using these technologies. OpenAI being a black box model and behind a paywall definitely raises a lot of concern. Some of the concerns we are trying to address:

1. If we send data to the ChatGPT/OpenAI (questions and prompts) will it be stored and reused for training purposes?

2. There is inherent bias in the models so how do we address them especially for customer facing applications?

To address point-1 we are working with Microsoft Azure team who will provide the OpenAI models privately in the Azure cloud platform. We don’t have transparency in to the model but we are at least confident that user data won’t be used for training purposes.

To address point-2 there are some LLM techniques and frameworks to handle these issues. We are still in the exploration stage of how to come up with a use case that is not significantly impacted by the hallucination and bias issues but still provides value to our organization. We are planning to develop customer facing applications once we have confidence in the model with the internal use cases. (Note that OpenAI did rigorous internal testing on ChatGPT by deploying it to their internal teams before public release).

Apart from OpenAI there are multiple other open source models which provides complete transparency into the model. We can have full ownership and control of the model so won’t have to worry about data security issues. The challenge here is navigating the licensing issues with these models. Not all models are viable for commercial usage, so have to do some homework in figuring out the right model. These models are much smaller than the OpenAI models but surprisingly their performance have been on par with the larger models.

At Exelon we are trying to understand how well these smaller models perform compared to OpenAI and what are the benefits of using such models.

——————————
Valliappan Muthukaruppan
Sr. Data Scientist
Exeloncorp
9199170140
——————————
——————————————-
Original Message:
Sent: 05-22-2023 17:22
From: Leslie Cook
Subject: GPT and LLM Questions from Members

Good afternoon, UAI Members!

I have a couple of questions regarding GPT (Generative Pre-trained Transformer), LLM (Large Language Model), and other models developed by OpenAI. Many members are talking about these tools (coming up in almost every conversation I have with members), and have questions for each other. Please find a couple of those questions below submitted to me today. Let’s get a conversation going on this thread on these topics!  

1.) What is your company planning to do with LLMs? Are you exploring options now, or blocking access and waiting until others investigate?

2.) What are some use cases for GPT at your company? Customized searches, report rough draft write-ups, chatbots that have context of your training documents?

On behalf of the members asking, thanks for your input! 

Cheers! 

     Leslie 

——————————
Leslie Cook
Membership, Engagement, & Training Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@utilityanalytics.com
——————————

[Maggie Peacock]

Leslie, 
I appreciate you asking this question and I am interested in hearing what others are doing.

For SERC, representatives from our Data Management, Legal, IT, Business and Security teams (collectively our Data Governance Working Group) are exploring options and are in the process of drafting and providing guidance to staff. For example, our annual confidential information training will include reminders on restrictions of using confidential data with AI tools. Our Data Management procedures have been revised to include information on the legal and ethical consideration when decision making, the need to perform quality control, etc.

As to access, it depends on if the tool conforms to our security policies i.e., our firewall blocks sites hosted outside of the US, so ChatGPT will not always be accessible.

For use cases, there are too many to name! I am an advocate and have used ChatGPT in tandem with other tools when developing content such as presentations, training materials, procedures, emails, etc. It is not the whole of the work product, but it is a useful resource. I am quick to share with others when I have used it and how. Though I am always careful to highlight the challenges, blind spots, and when I personally have seen it get something wrong.

Hope that helps.

——————————
Maggie Peacock
Manager, Analytics and Data Management
SERC
——————————
——————————————-
Original Message:
Sent: 05-22-2023 17:22
From: Leslie Cook
Subject: GPT and LLM Questions from Members

Good afternoon, UAI Members!

I have a couple of questions regarding GPT (Generative Pre-trained Transformer), LLM (Large Language Model), and other models developed by OpenAI. Many members are talking about these tools (coming up in almost every conversation I have with members), and have questions for each other. Please find a couple of those questions below submitted to me today. Let’s get a conversation going on this thread on these topics!  

1.) What is your company planning to do with LLMs? Are you exploring options now, or blocking access and waiting until others investigate?

2.) What are some use cases for GPT at your company? Customized searches, report rough draft write-ups, chatbots that have context of your training documents?

On behalf of the members asking, thanks for your input! 

Cheers! 

     Leslie 

——————————
Leslie Cook
Membership, Engagement, & Training Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@utilityanalytics.com
——————————

[Henry Wu]

We are looking to augment our Semantic Search application with Open AI GPT.

 

Our Semantic Search team started in late 2019. At the time, we wanted to help our Legal and Regulatory department to reduce the time they spent on the discovery process. We looked at language models, e.g., LSI, Word2Vec, GloVe, and BERT to augment traditional search algorithm, which is based on TF-IDF. We wanted to provide a more expert or assistant like experience with our search application. We settled on Azure Semantic Search services, which uses a form of transformer model.

 

When ChatGPT came into the scene, it was no brainer for us to make the case to obtain a preview access to the Azure Open AI services from Microsoft, since we see this as a potential next step in the evolution of our search application and we needed a secured and isolated environment to protect our company’s information. We are testing the services extensively in this isolated and secured environment since we are concerned about hallucinations. We’ve seen enough examples of this to be concerned. We should know by end of Q2 where GPT is appropriate, if at all.

 

Thanks.

 

Henry Wu

Data Science

Performance Mgmt. & Org. Strategy, SoCalGas

(213) 231-9890 (Office)

(626) 374-8932 (Cell)

 

 

Empowering better decisions through data

 

——————————————-
Original Message:
Sent: 5/22/2023 5:23:00 PM
From: Leslie Cook
Subject: GPT and LLM Questions from Members

Good afternoon, UAI Members!

I have a couple of questions regarding GPT (Generative Pre-trained Transformer), LLM (Large Language Model), and other models developed by OpenAI. Many members are talking about these tools (coming up in almost every conversation I have with members), and have questions for each other. Please find a couple of those questions below submitted to me today. Let’s get a conversation going on this thread on these topics!  

1.) What is your company planning to do with LLMs? Are you exploring options now, or blocking access and waiting until others investigate?

2.) What are some use cases for GPT at your company? Customized searches, report rough draft write-ups, chatbots that have context of your training documents?

On behalf of the members asking, thanks for your input! 

Cheers! 

     Leslie 

——————————
Leslie Cook
Membership, Engagement, & Training Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@utilityanalytics.com
——————————

[Duminda Randeniya]

Hello,

 

At Enbridge Gas Inc., we have developed (in-house) language model/s using some of the established concepts used in Natural Language Processing models. These models are predominantly trained on our data and supplemented on large text repositories (corpuses). The reason for the in-house development was some of the same concerns that Exelon mentioned around the data security.

 

As per Use Cases: we started developing these models to classify work order comments and remarks into pre-defined failure classes (for Integrity and Asset Management functions). However, we are expanding on usage of these models. And, currently, exploring some applications within Operations to improve predictability of various maintenance tasks and efficiencies.

 

 

Duminda Randeniya Ph.D., P.Eng.

Specialist – Forecasting & Optimization,

Workload Planning, Operations Support

 

ENBRIDGE
TEL: (416) 758-7936 / Cell: (416) 559-0835
500 Consumers Road, North York, ON – M2J 1P8

enbridge.com
Integrity. Safety. Respect, Inclusion.

 

——————————————-
Original Message:
Sent: 5/24/2023 8:49:00 AM
From: Valliappan Muthukaruppan
Subject: RE: GPT and LLM Questions from Members

At Exelon we have started to explore LLMs. We are working with our security team to identify the best use case and what are the security implications involved in using these technologies. OpenAI being a black box model and behind a paywall definitely raises a lot of concern. Some of the concerns we are trying to address:

1. If we send data to the ChatGPT/OpenAI (questions and prompts) will it be stored and reused for training purposes?

2. There is inherent bias in the models so how do we address them especially for customer facing applications?

To address point-1 we are working with Microsoft Azure team who will provide the OpenAI models privately in the Azure cloud platform. We don’t have transparency in to the model but we are at least confident that user data won’t be used for training purposes.

To address point-2 there are some LLM techniques and frameworks to handle these issues. We are still in the exploration stage of how to come up with a use case that is not significantly impacted by the hallucination and bias issues but still provides value to our organization. We are planning to develop customer facing applications once we have confidence in the model with the internal use cases. (Note that OpenAI did rigorous internal testing on ChatGPT by deploying it to their internal teams before public release).

Apart from OpenAI there are multiple other open source models which provides complete transparency into the model. We can have full ownership and control of the model so won’t have to worry about data security issues. The challenge here is navigating the licensing issues with these models. Not all models are viable for commercial usage, so have to do some homework in figuring out the right model. These models are much smaller than the OpenAI models but surprisingly their performance have been on par with the larger models.

At Exelon we are trying to understand how well these smaller models perform compared to OpenAI and what are the benefits of using such models.

——————————
Valliappan Muthukaruppan
Sr. Data Scientist
Exeloncorp
9199170140
——————————
——————————————-
Original Message:
Sent: 05-22-2023 17:22
From: Leslie Cook
Subject: GPT and LLM Questions from Members

Good afternoon, UAI Members!

I have a couple of questions regarding GPT (Generative Pre-trained Transformer), LLM (Large Language Model), and other models developed by OpenAI. Many members are talking about these tools (coming up in almost every conversation I have with members), and have questions for each other. Please find a couple of those questions below submitted to me today. Let’s get a conversation going on this thread on these topics!  

1.) What is your company planning to do with LLMs? Are you exploring options now, or blocking access and waiting until others investigate?

2.) What are some use cases for GPT at your company? Customized searches, report rough draft write-ups, chatbots that have context of your training documents?

On behalf of the members asking, thanks for your input! 

Cheers! 

     Leslie 

——————————
Leslie Cook
Membership, Engagement, & Training Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@utilityanalytics.com
——————————

[Derek Johansen]

Hey Valliappan – do you have any idea on the timeline for your engagement with Microsoft’s private cloud instance? I’m curious if that’s something that would take a month or two, or 6+ months in the experience you’ve had.

Thanks,

Derek

——————————
Derek Johansen
Black Hills Energy
——————————
——————————————-
Original Message:
Sent: 05-24-2023 08:48
From: Valliappan Muthukaruppan
Subject: GPT and LLM Questions from Members

At Exelon we have started to explore LLMs. We are working with our security team to identify the best use case and what are the security implications involved in using these technologies. OpenAI being a black box model and behind a paywall definitely raises a lot of concern. Some of the concerns we are trying to address:

1. If we send data to the ChatGPT/OpenAI (questions and prompts) will it be stored and reused for training purposes?

2. There is inherent bias in the models so how do we address them especially for customer facing applications?

To address point-1 we are working with Microsoft Azure team who will provide the OpenAI models privately in the Azure cloud platform. We don’t have transparency in to the model but we are at least confident that user data won’t be used for training purposes.

To address point-2 there are some LLM techniques and frameworks to handle these issues. We are still in the exploration stage of how to come up with a use case that is not significantly impacted by the hallucination and bias issues but still provides value to our organization. We are planning to develop customer facing applications once we have confidence in the model with the internal use cases. (Note that OpenAI did rigorous internal testing on ChatGPT by deploying it to their internal teams before public release).

Apart from OpenAI there are multiple other open source models which provides complete transparency into the model. We can have full ownership and control of the model so won’t have to worry about data security issues. The challenge here is navigating the licensing issues with these models. Not all models are viable for commercial usage, so have to do some homework in figuring out the right model. These models are much smaller than the OpenAI models but surprisingly their performance have been on par with the larger models.

At Exelon we are trying to understand how well these smaller models perform compared to OpenAI and what are the benefits of using such models.

——————————
Valliappan Muthukaruppan
Sr. Data Scientist
Exeloncorp
9199170140
——————————

Original Message:
Sent: 05-22-2023 17:22
From: Leslie Cook
Subject: GPT and LLM Questions from Members

Good afternoon, UAI Members!

I have a couple of questions regarding GPT (Generative Pre-trained Transformer), LLM (Large Language Model), and other models developed by OpenAI. Many members are talking about these tools (coming up in almost every conversation I have with members), and have questions for each other. Please find a couple of those questions below submitted to me today. Let’s get a conversation going on this thread on these topics!  

1.) What is your company planning to do with LLMs? Are you exploring options now, or blocking access and waiting until others investigate?

2.) What are some use cases for GPT at your company? Customized searches, report rough draft write-ups, chatbots that have context of your training documents?

On behalf of the members asking, thanks for your input! 

Cheers! 

     Leslie 

——————————
Leslie Cook
Membership, Engagement, & Training Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@utilityanalytics.com
——————————