A strategic question about data stored in Excel files

  • A strategic question about data stored in Excel files

    Posted by Michael Fourman on January 27, 2023 at 10:24 am

    Good morning,

    We are several months into our analytics journey and we have a decision to make on how to best handle data stored in Excel files.  Here is the situation, we have several use cases now (and I suspect many more that will arise) where staff creates there own data and keeps it in Excel files they save on their personal network drive.  We’d like to get this data into Azure while allowing the user to continue to update the files when they need to.  Some will be updated daily, others not as often.  Our current thinking is to have the staff store and work from a common network drive which makes ingestion easy but introduces some security concerns.  We could have them password protect their Excel files to help with this but I’m not sure how that impacts the pipeline.  Could we save the files in Azure and have the users update it there?  Are there other solutions I’ve not described here?  If you’ve already solved this, I’d love to hear more about your solution.  Thanks!

    Mike Fourman

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————

    Cara Gilad replied 1 year, 9 months ago 6 Members · 8 Replies
  • 8 Replies
  • Oleg Kachirski

    Member
    January 27, 2023 at 11:41 am

    Hi Mike,

    If it has to be Excel (for certain workflows Power Apps-based solution could work), it could be stored in SharePoint (Online or on-premise) to enhance user access control, maintain file versioning and perhaps implement a QA workflow in SharePoint prior to ingesting the data into its final destination. The data “sink” (in ADF-speak) could be a data lake in Azure, a SQL db or a Synapse data warehouse among other things. Synapse or Azure Data Factory (ADF) could be used to ingest, transform and enrich the data as needed. There may be more appropriate options based on your IT architecture, use of third-party products, data governance standards, etc. 

    Thanks,

    ——————————
    Oleg Kachirski
    Manager, Global Advisory
    Black & Veatch
    407-970-1034
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-27-2023 10:23
    From: Michael Fourman
    Subject: A strategic question about data stored in Excel files

    Good morning,

    We are several months into our analytics journey and we have a decision to make on how to best handle data stored in Excel files.  Here is the situation, we have several use cases now (and I suspect many more that will arise) where staff creates there own data and keeps it in Excel files they save on their personal network drive.  We’d like to get this data into Azure while allowing the user to continue to update the files when they need to.  Some will be updated daily, others not as often.  Our current thinking is to have the staff store and work from a common network drive which makes ingestion easy but introduces some security concerns.  We could have them password protect their Excel files to help with this but I’m not sure how that impacts the pipeline.  Could we save the files in Azure and have the users update it there?  Are there other solutions I’ve not described here?  If you’ve already solved this, I’d love to hear more about your solution.  Thanks!

    Mike Fourman

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————

  • Michael Fourman

    Member
    January 30, 2023 at 2:21 am

    Thanks, Bert.  I have forwarded this to our Safety group for consideration.

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-19-2023 10:30
    From: Bert Hargesheimer
    Subject: Attention Safety and Analytics Leaders!

    Safety and Analytics Leaders!

    We are looking to build a benchmark group for Safety Metrics within UAI.  We are interested in looking at the traditional metrics, but also at Serious Injuries and Fatalities (SIF).  First we’d like to discuss the SIF definition (Edison has one that is broadly used) and then begin benchmarking to gauge performance.  Please check in with our survey and join us during the February Safety Analytics Community meeting for the SIF discussion and in March we’ll begin looking at PSIF (Potential Serious Injuries and Fatalities).

    Bert Hargesheimer
    Safety Analytics Working Group Co-Lead
    CPS Energy

    ——————————
    Bert Hargesheimer
    VP Operational Support Services
    CPS Energy
    San Antonio TX
    2103534323
    ——————————

  • Michael Fourman

    Member
    January 30, 2023 at 2:47 am

    Hi Doni.  GTC is still early in its analytics journey.  We’ve drafted a data governance document which is currently being reviewed.  I’m not sure we address data classification by name but we do allude to it specifically when we talk about security.  I hope someone is willing to share because I think it would not only help you, but us as well as others.  Thanks for posting this!

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-25-2023 09:46
    From: Doni Gustafson
    Subject: Data Classification and Guidelines

    A few requests/questions:

    1. Policy: Would anyone be willing to share their Data Classification policy, which could include things like the companywide responsibilities for classifying data (e.g., confidential, sensitive, etc.) and a basic classification framework?
    2. Guidelines: I would also appreciate seeing your Data Classification Guidelines or Data Handling Standards that provide guidance on how to classify data and defines handling requirements based on data classifications. 
    3. Manual or Automated: Does your organization classify data manually or automatically? If manually, how are you holding employees accountable and what are your lessons learned? If automatically, can you share what tool you are using as well as lessons learned? 
    4. Contact: If you are willing to have a deeper conversation about the questions above, can you please let me know your contact information? 

    Thank you all for your time and feedback!

    ——————————
    Donielle Gustafson
    Sr Program Manager of Enterprise Data & Analytics
    Black Hills Energy
    1-605-800-1799
    ——————————

  • Leslie Cook (Adm)

    Member
    January 31, 2023 at 6:12 am

    Hello UAI Members!

    Thank you to everyone that has taken the time to participate in the Safety Analytics Benchmarking survey! So far, we have 8 members that have taken the survey. We would love to see at least 30 members take it. 

    We are looking to build a benchmark group for Safety Metrics within UAI. We are interested in looking at the traditional metrics, but also at Serious Injuries and Fatalities (SIF). First we’d like to discuss the SIF definition (Edison has one that is broadly used) and then begin benchmarking to gauge performance.  Please check in with our survey! 

    If you are not in safety within your utility, please take @Michael‘s lead and forward benchmarking survey to your safety group for consideration. Thank you so much, Michael, for forwarding the Benchmarking survey to your safety group. This is an excellent first step to getting involved in this Benchmarking project. I am hopeful that more UAI Utility members forward this on to their safety groups.

    Also, make sure you join us for the February 28, 2023 UAI Safety Analytics Community Conversation, where @Bert from CPS Energy will facilitate a discussion on SIF as follows and review results from the Benchmarking survey. If you are not part of the UAI Safety Analytics Community, you can request to join by completing the following form: https://uaievents.wufoo.com/forms/request-to-join-uai-communities

    February 28, 2023 UAI Safety Analytics Community Conversation
    Session Title: Part 1: Serious Injuries and Fatalities (SIF) Benchmarking and Community Discussion (Part 2 will be on PSIF)
    Session Description: CPS Energy has been exploring safety metrics that help to create a better picture of safety performance in an effort to better inform leaders in where time spent making improvements will be most valuable. There are many ways to look at injury severity; DART, OSHA Severity Rate among others. Serious Injuries and Fatalities (SIF) is an emerging trend with some varying definitions. We’ve done a survey to get information from utilities regarding what metrics they’re using and we’ll share the results and then have a short discussion about interest in longer term benchmarking safety metrics with this group.

    If you have any questions about the benchmarking project, please reach out to me, @Leslie, at lcook@utilityanalytics.com or simply reply to this discussion thread in UAI Connect.

    Thanks!

         Leslie ​​​​

    ——————————
    Leslie Cook
    Membership & Digital Engagement Manager
    Utility Analytics Institute (UAI)
    719-203-8650, lcook@utilityanalytics.com
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-30-2023 14:20
    From: Michael Fourman
    Subject: Attention Safety and Analytics Leaders!

    Thanks, Bert.  I have forwarded this to our Safety group for consideration.

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————

    Original Message:
    Sent: 01-19-2023 10:30
    From: Bert Hargesheimer
    Subject: Attention Safety and Analytics Leaders!

    Safety and Analytics Leaders!

    We are looking to build a benchmark group for Safety Metrics within UAI.  We are interested in looking at the traditional metrics, but also at Serious Injuries and Fatalities (SIF).  First we’d like to discuss the SIF definition (Edison has one that is broadly used) and then begin benchmarking to gauge performance.  Please check in with our survey and join us during the February Safety Analytics Community meeting for the SIF discussion and in March we’ll begin looking at PSIF (Potential Serious Injuries and Fatalities).

    Bert Hargesheimer
    Safety Analytics Working Group Co-Lead
    CPS Energy

    ——————————
    Bert Hargesheimer
    VP Operational Support Services
    CPS Energy
    San Antonio TX
    2103534323
    ——————————

  • Nadia Powell

    Member
    February 1, 2023 at 10:43 am

    Hello Ty-Rell and welcome to UAI!

    I spent time in Alexandria, Louisiana and love the food, culture and people!  Still have many friends there.  Looking forward to connecting with you at the UA Summit.

    nadia

    ——————————
    Nadia Powell
    Director Enterprise Advanced Analytics
    El Paso Electric
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-11-2023 10:46
    From: Ty-Rell Walton
    Subject: Hello Everyone!!

    Hello everyone

    My name is Ty-Rell Walton. I currently work for CenterPoint Energy at the Louisiana office. I’ve been with the company for a few months but, have been working in Data Analysis for three years now. I am a southern boy from the great Cajun State of Louisiana where we love our Gumbo, Boudin, and Mardi Gras! I am a 2019 graduated at Louisiana State University (Go Tigers) in Computer Science. I love to travel, spend time with family, friends, camp, watch my favorite show on Netflix (NCIS) and bowl among other things. I am very adventurous and always looking to try new activities. I am excited to join the UAI Community and look forward to learning, collaborating, and meeting new colleagues in this data analysis world. As we say back home, laissez les bon temps rouler (Let the Good time roll)!

    ——————————
    Ty-Rell Walton
    Data Analyst
    CenterPoint Energy
    ——————————

  • Nadia Powell

    Member
    February 1, 2023 at 10:45 am

    Hi Nolan,

    Is there a recording available for this seminar?  

    nadia

    ——————————
    Nadia Powell
    Director Enterprise Advanced Analytics
    El Paso Electric
    ——————————
    ——————————————-
    Original Message:
    Sent: 11-08-2022 15:53
    From: Nolan Steiner
    Subject: Live Webinar: Powering a Data Catalogue – Avista’s data culture journey (using data catalogue)

    Live Webinar: Wednesday November 16, 2022
    8am Pacific time; 11am Eastern time
    Please join us for an overview of Avista’s experience adopting a data catalogue (to promote ‘analyst productivity’)  

    Avista Utilities is an energy company founded in 1889 based in Spokane WA.   Avista provides electricity to 400,000+ customers and natural gas to 370,000+ customers in WA, OR, ID and MT.  

    Sign up for this event using the attached link.  
    Powering a Catalog: Avista’s Data Culture Journey | Alation Resources

    ——————————
    Nolan Steiner
    Manager, Data Science
    Avista Corporation
    Spokane WA
    ph: 509-495-4479
    ——————————

  • Nolan Steiner

    Member
    February 1, 2023 at 12:22 pm

    Hi Nadia:  Yes…per your request, attached is the link to the webinar recording we did recently for Alation (our journey on our Data Catalogue), for those interested.  It’s about an hour total.  

    https://www.alation.com/resource-center/youtube-all-videos/powering-a-catalog-avistas-data-culture-journey

    Thanks for asking!

    ——————————
    Nolan Steiner
    Data Science
    Avista Corporation
    Spokane WA
    5094954479
    ——————————
    ——————————————-
    Original Message:
    Sent: 02-01-2023 10:44
    From: Nadia Powell
    Subject: Live Webinar: Powering a Data Catalogue – Avista’s data culture journey (using data catalogue)

    Hi Nolan,

    Is there a recording available for this seminar?  

    nadia

    ——————————
    Nadia Powell
    Director Enterprise Advanced Analytics
    El Paso Electric
    ——————————

    Original Message:
    Sent: 11-08-2022 15:53
    From: Nolan Steiner
    Subject: Live Webinar: Powering a Data Catalogue – Avista’s data culture journey (using data catalogue)

    Live Webinar: Wednesday November 16, 2022
    8am Pacific time; 11am Eastern time
    Please join us for an overview of Avista’s experience adopting a data catalogue (to promote ‘analyst productivity’)  

    Avista Utilities is an energy company founded in 1889 based in Spokane WA.   Avista provides electricity to 400,000+ customers and natural gas to 370,000+ customers in WA, OR, ID and MT.  

    Sign up for this event using the attached link.  
    Powering a Catalog: Avista’s Data Culture Journey | Alation Resources

    ——————————
    Nolan Steiner
    Manager, Data Science
    Avista Corporation
    Spokane WA
    ph: 509-495-4479
    ——————————

  • Cara Gilad

    Member
    February 3, 2023 at 9:10 am

    We also use the Sharepoint-hosted solution for Excel files, but I do not have experience with the password protection. You may want to consider changing the way the data is captured from staff – I agree with the PowerApps suggestion. If all updates are appended new rows, without editing old rows, you could use a web-based survey that feeds a sharepoint List, which is a straightforward way to capture data entry from multiple people without them being able to see or edit each other’s responses. But this only works in limited cases.

    ——————————
    Cara Gilad
    Data Scientist
    Exelon
    ——————————
    ——————————————-
    Original Message:
    Sent: 01-27-2023 10:23
    From: Michael Fourman
    Subject: A strategic question about data stored in Excel files

    Good morning,

    We are several months into our analytics journey and we have a decision to make on how to best handle data stored in Excel files.  Here is the situation, we have several use cases now (and I suspect many more that will arise) where staff creates there own data and keeps it in Excel files they save on their personal network drive.  We’d like to get this data into Azure while allowing the user to continue to update the files when they need to.  Some will be updated daily, others not as often.  Our current thinking is to have the staff store and work from a common network drive which makes ingestion easy but introduces some security concerns.  We could have them password protect their Excel files to help with this but I’m not sure how that impacts the pipeline.  Could we save the files in Azure and have the users update it there?  Are there other solutions I’ve not described here?  If you’ve already solved this, I’d love to hear more about your solution.  Thanks!

    Mike Fourman

    ——————————
    Michael Fourman
    Director – Engineering Services
    Georgia Transmission
    ——————————

Log in to reply.