Snowflake or/and Databricks As Cloud Data Platform

  • Snowflake or/and Databricks As Cloud Data Platform

    Posted by Ramesh Sethuraj on September 20, 2023 at 11:15 am

    Hi,

    We have legacy Enterprise Datawarehouse (EDW) On-Prem and Datalake (mainly AMI) on Cloud. Our on-prem EDW is primarily used for Reporting and Dashboarding (Power BI & Cognos) with thousands of users using it day to day and cloud platform used for Advanced analytics, very few Cognos reports running at this time. 

    As part of our new Data Platform strategy, we are in the process of identifying cloud data platform for AEP that can host both EDW and Bigdata.

    Based on our Working Group outcome, we have shortlisted Snowflake and Databricks as our final choice. Before we finalize it, we would like to find out and talk to other utilities who went thru this route, selected the cloud data platform, migrated on-prem EDW workload and successfully using it. Do we have anyone?

    Please let me know if you need more information.

    I appreciate your help.

    Thanks!

    Ramesh

    ——————————
    Ramesh Sethuraj
    IT Architect
    American Electric Power
    Gahanna OH
    614-883-7125
    ——————————

    Kevin Praet (Adm) replied 1 year, 2 months ago 4 Members · 4 Replies
  • 4 Replies
  • Jason Pegg

    Member
    September 20, 2023 at 2:19 am

    Hello!

    Like a product review by someone that never made a purchase, I’m going to give you a response that doesn’t quite answer the question.  :-)  But I’m hoping you still find it useful.

    A few years back, we went through a similar exercise for very similar reasons.  We also narrowed it down to Snowflake and Databricks.  In the end, we ended up choosing AWS technologies – Redshift, Redshift Spectrum, and S3.  For those more comfortable with traditional EDW solutions, Redshift worked well.  For those comfortable with newer technologies, they were able to utilize S3 through tools like Athena.  By storing the data in S3, we were also able to bridge the gap with Redshift Spectrum.

    Like I said – not a direct answer to your question, but there were definitely parallels.  We ended up saving a lot of money along the way as well.  Happy to discuss more if it’s helpful.

    Good luck!

    Jason…

    ——————————
    Jason Pegg
    Data Delivery & Integrations Manager, Domain Architect
    Avista Corp.
    Spokane WA
    509.495.4731
    ——————————
    ——————————————-
    Original Message:
    Sent: 09-20-2023 11:15
    From: Ramesh Sethuraj
    Subject: Snowflake or/and Databricks As Cloud Data Platform

    Hi,

    We have legacy Enterprise Datawarehouse (EDW) On-Prem and Datalake (mainly AMI) on Cloud. Our on-prem EDW is primarily used for Reporting and Dashboarding (Power BI & Cognos) with thousands of users using it day to day and cloud platform used for Advanced analytics, very few Cognos reports running at this time. 

    As part of our new Data Platform strategy, we are in the process of identifying cloud data platform for AEP that can host both EDW and Bigdata.

    Based on our Working Group outcome, we have shortlisted Snowflake and Databricks as our final choice. Before we finalize it, we would like to find out and talk to other utilities who went thru this route, selected the cloud data platform, migrated on-prem EDW workload and successfully using it. Do we have anyone?

    Please let me know if you need more information.

    I appreciate your help.

    Thanks!

    Ramesh

    ——————————
    Ramesh Sethuraj
    IT Architect
    American Electric Power
    Gahanna OH
    614-883-7125
    ——————————

  • Kevin Praet (Adm)

    Member
    September 20, 2023 at 4:17 am

    Hello UAI Members,

    I hope all is well! I want to conclude your Wednesday with a little #wisdom from one of my favorite athletes, Simone Biles! Though this quote is short and simple, I believe it serves as an important reminder that we’re all unique and have more to give.

    See below and, as always, if you have a quote you would like to share in our #WisdomWednesday thread I encourage you to reply below.

    “I’m not the next Usain Bolt or Michael Phelps… I’m the first Simone Biles.”

    ——————————
    Kevin Praet
    Membership Coordinator
    Utility Analytics Institute (UAI)
    Boulder CO
    315-440-3033
    ——————————
    ——————————————-
    Original Message:
    Sent: 09-13-2023 12:01
    From: Kevin Praet
    Subject: Wisdom Wednesday – Inspiration to Get You Through the Week

    Hello UAI Members,

    Happy Wednesday and I hope everyone is doing well! Rather than share one quote this week for #WisdomWednesday, I actually wanted to share a short video that contain several great quotes. For those looking for inspiration, I couldn’t recommend their YouTube channel anymore, there’s a lot of great material!

    As always if there’s a quote you would like to share, I’d love to hear from you! Please comment in the reply section below.

    ——————————
    Kevin Praet
    Membership Coordinator
    Utility Analytics Institute (UAI)
    Boulder CO
    315-440-3033
    ——————————

  • Garrett Granger

    Member
    September 21, 2023 at 10:36 am

    Ramesh, we went with Databricks before hearing about Snowflake from someone at UAI. I am biased, because we already have Databricks deployed. One of our employees used Snowflake at her former, very large private company. From a poll I saw online, the mindshare for Databricks and Snowflake are about the same. Which one serves your users is really going to depend on what type of users you have. I would encourage you to demo both options.

    Few brief pros and cons I have read about the two platforms.

    • Databricks: Targeted more toward data engineers/data scientists, less expensive, faster. Supports SQL, PySpark, R, Scala, open source. PySpark (Python, basically) is natively integrated into the application in a Jupyter notebook-like interface. You can run different languages in the same notebook.
    • Snowflake: More user friendly, more expensive, slower, closed source. Supports GO,  .NET, Node.js, Python. Uses Python worksheets or external Jupyter/Anaconda notebooks and connectors.

    My personal opinion is that Databricks will be more future proof. It is more established, uses less proprietary syntax and Microsoft is trying to emulate it through  Synapse – the code is virtually interchangeable, as they are both based on a Spark platform. Azure is also rapidly gaining marketshare compared to AWS and may become the dominant cloud platform in the future. 

    I would definitely not do both. I would like to hear which platform you actually decide to deploy. 

    ——————————
    Garrett Granger
    Data Analytics Manager
    Georgia Transmission
    770-270-7497
    ——————————
    ——————————————-
    Original Message:
    Sent: 09-20-2023 11:15
    From: Ramesh Sethuraj
    Subject: Snowflake or/and Databricks As Cloud Data Platform

    Hi,

    We have legacy Enterprise Datawarehouse (EDW) On-Prem and Datalake (mainly AMI) on Cloud. Our on-prem EDW is primarily used for Reporting and Dashboarding (Power BI & Cognos) with thousands of users using it day to day and cloud platform used for Advanced analytics, very few Cognos reports running at this time. 

    As part of our new Data Platform strategy, we are in the process of identifying cloud data platform for AEP that can host both EDW and Bigdata.

    Based on our Working Group outcome, we have shortlisted Snowflake and Databricks as our final choice. Before we finalize it, we would like to find out and talk to other utilities who went thru this route, selected the cloud data platform, migrated on-prem EDW workload and successfully using it. Do we have anyone?

    Please let me know if you need more information.

    I appreciate your help.

    Thanks!

    Ramesh

    ——————————
    Ramesh Sethuraj
    IT Architect
    American Electric Power
    Gahanna OH
    614-883-7125
    ——————————

  • Kevin Praet (Adm)

    Member
    September 22, 2023 at 12:07 pm

    Hello UAI Members,

    Happy Friday and I hope all is well! I just wanted to circle back on this questions from last week and see if anyone has experience creating customer engagement scores within their utility and would be willing to connect with a utility team out of Texas to answer a few questions. If you have any insight, please reply to this message below or reach out to me directly at kpraet@utilityanalytics.com. Your help is appreciated as always!

    Thanks again!

    ——————————
    Kevin Praet
    Membership Coordinator
    Utility Analytics Institute (UAI)
    Boulder CO
    315-440-3033
    ——————————
    ——————————————-
    Original Message:
    Sent: 09-13-2023 13:16
    From: Kevin Praet
    Subject: Customer Engagement Score – Who Has Experience Developing?

    Hello UAI Members,

    I’m working with an electric cooperative out of Texas who are in the early stages of creating a customer engagement score. Being in the early stages, they were hoping to connect with other utility teams (preferably other coops) to ask a few questions and get advice on devlopment. Might any members have experience creating customer engagement scores? If so might you be willing to connect directly with our team lead?

    Any insight here is appreciated!

    Thanks for your help,

    ——————————
    Kevin Praet
    Membership Coordinator
    Utility Analytics Institute (UAI)
    Boulder CO
    315-440-3033
    ——————————

Log in to reply.