Ask Your Network
Have a question? Want to bounce ideas off your peers? Need to do a sanity check for a project? Reach... View more
Organizer:
- Organized by
-
-
Member Spotlight: Kaleb Thompson
-
Member Spotlight: Kaleb Thompson
Kaleb is an active UAI member and provides outstanding influence and contributions to the UAI community through any of our multiple channels, including, but not limited to engagement on UAI Connect, volunteering with UAI, involvement and/or leadership in monthly Community Conversations, and/or speaking at UAI events, plus so much. As thanks, we are featuring him in the UAI Member Spotlight. Kaleb Thompson is an analytics professional based in Tulsa, OK. He has worked in a variety of roles such as Data Analyst, Data Engineer and Data Scientist over the last 5 and a half years at ONE Gas Inc. He holds a Master of Science in Business Analytics and Data Science from Oklahoma State University. He now serves as the Supervisor of Enterprise Analytics, supporting the centralized Data Science and Business Intelligence teams.
Kevin: How did you get into the utility analytics industry?
Member: Entirely by chance! When I graduated from college, I really had no idea what I wanted to do. My first professional job was at an energy retailer where I was responsible for enrolling accounts with the various Canadian electric and natural gas utility companies. It was my first real taste of analytics and I soon realized I wanted to grow into a more technical role. I saw a job posting for what would now be called a data analyst at ONE Gas and the rest is history. I’ve spent the last five and a half years helping support and grow the analytics team here.
Kevin: What project have you enjoyed working on most in your career?
Member: About two years ago, we worked on a project to increase visibility into our emergency response incidents. We ultimately created a descriptive solution where our business partners can see how quickly we are responding to emergency incidents, where they are located and any contextual details regarding the incident. We recently added a predictive forecast component that provides insight into how many orders we can expect to get on a given day during a given shift for a geographic area. The project combined data from our work scheduling system, customer information system as well as newly mastered employee hierarchy data from our data governance program. It was a big win for our organization and was a good use case for combining descriptive, diagnostic and predictive analytics together.
Kevin: What has been your biggest professional challenge and how did you work through it?
Member: So far, I would say that my biggest challenge was pioneering the data science program. We have an extremely robust Business Intelligence program that has delivered a lot of value to our business. It can be challenging to delineate “advanced analytics” or “data science” from Business Intelligence and get our business partners thinking in terms of predictive analytics. I think over the last year, we have made significant strides in this area.
Kevin: If you could go back in time and give yourself one piece of advice before entering the utility analytics industry, what would it be?
Member: Understand the implications of change and how to effectively manage it. The highest value initiatives in the data science space require changes in people, process and technology beyond the creation of an algorithm. It’s important to understand how quickly you can implement data science solutions into production.
Kevin: What would you like to talk to fellow colleagues and members about in UAI Connect? Or, what issues or topics would you like to see more discussions on in UAI Connect?
Member: I really enjoy applying new tools and technologies to problems that we are all facing as an industry. I would love to connect with colleagues and members that are using data to solve complicated problems within the natural gas industry.
Kevin: Thank you so much for sharing with us and for being a highly engaged member of UAI!
Do you have any final thoughts, ideas, or comments you would like to share with your fellow UAI members?
Member: I would just like to say thanks to the UAI team for putting together such a great community of analytics professionals. I have certainly received a lot of value from the monthly community meetings, the in-person events (before COVID) and the professional connections that I’ve made up until now. I’m excited to be a part of the community in 2021 and beyond!
——————————
Kevin Praet
Membership Coordinator
Utility Analytics Institute (UAI)
Boulder CO
315-440-3033
—————————— -
5 Replies
-
I asked one of our Meteorologists how he pulls hourly NOAA data and he advised: “My typical avenue for downloading hourly data from NOAA has just been through their website form process Data Search | National Centers for Environmental Information (NCEI) (noaa.gov). However, they recently started limiting the file size, which became very inconvenient if I wanted more than a few years of data. I’ve used the R package called “rNOAA” rnoaa.pdf (r-project.org) recently with some success.”
——————————
Norv Clontz
Director, Data Science Innovation
Duke Energy
Charlotte NC
9803732380
——————————
——————————————-
Original Message:
Sent: 01-05-2021 15:49
From: Lauren Gouveia
Subject: NOAA Hourly Weather Dataset DownloadWe’ve tried a number of sources for collecting weather data including NOAA, DarkSky, Accuweather, deploying our own kits for solar irradiance and weather stations, and through contract with UL (formerly AWSTruepower). We are settling on using UL to provide both (renewable) forecasts and historical weather parameters – this improves consistency with the forecasts and simplifies the data translators (reduced the need for multiple data translators to the different services).
If it helps, the team found the NOAA data to be inconsistent with the reliability and updates. Darksky was easy to use but didn’t seem very accurate. Both NOAA and Darksky were free (with limits). UL has an associated cost but under our contract it includes modeling for Hawaii’s terrain which improved the granularity of the forecast (since our islands are so small). For awhile we had some success with the strategically located solar irradiance kits and weather stations but it requires dedicated resources for maintenance and to keep them calibrated.
Hope this helps!
——————————
Lauren Gouveia
Principal, IT Technical Architect
Hawaiian Electric Companies
Honolulu HI
808-543-7619
——————————Original Message:
Sent: 01-05-2021 11:52
From: Vera Lei
Subject: NOAA Hourly Weather Dataset DownloadYes, we download historic daily data on multiple zones from WSU Agriculture site already.
Thanks,
Vera
BI/ETL Developer
Benton PUD WA
+1509-582-1280Original Message:
Sent: 1/5/2021 11:44:00 AM
From: Ben Ettlinger
Subject: RE: NOAA Hourly Weather Dataset DownloadBefore you buy your own weather station, check out WeatherUnderground. They used to have thousands of local weather stations reporting in, like from schools universities etc. They were bought by the Weather Channel so I’m not sure if they still have that.
I also think Washington State has an atmospheric science dept. NYPA is big into partnering with State Universities as we are a State owned utility. Not sure if tat’s something of interest to your utility.——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-05-2021 10:40
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadBen, thank you for the response. When I say hourly I am referring to historical hourly weather data, so not an hourly forecast.
And yes, we are exploring other services as well for this data so thanks for the recommendation. Another option we may end up exploring is purchasing our own weather station(s) to capture hourly data from.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
+15095548928Original Message:
Sent: 01-05-2021 08:24
From: Ben Ettlinger
Subject: NOAA Hourly Weather Dataset DownloadWhen you say hourly data, do you mean to receive the data hourly, or twice a day with data for every hour going forward. The last I checked the former is not available. NOAA sends out data twice each day after their weather balloon (depending where you are) returns. For NYC its in Upton NY.
There are plenty of services that also provide the data like Accuweather, The Weather channel or https://openweathermap.org/api/hourly-forecast. The difference is that the service apply their own algorithms to the NOAA data and sometimes come up with different predictions.
——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-04-2021 16:53
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadHappy New Year UAI Network!
I am wondering if anyone uses hourly weather data from NOAA, and if so, how have you been able to retrieve this information?
Our team has currently found how to bring in free daily summary data through an NOAA API, but we are seeking hourly data if available (whether it is free or payment needed). I have yet to hear back from NOAA regarding my request, so asking here in case anyone has had prior experience retrieving hourly data.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
—————————— -
Hey Norv,
I think you may have the wrong Freddy Jones. I work in Customer Delivery out of Raleigh.
Thanks,
Freddy
——————————————-
Original Message:
Sent: 1/5/2021 4:57:00 PM
From: Norv Clontz
Subject: RE: NOAA Hourly Weather Dataset DownloadI asked one of our Meteorologists how he pulls hourly NOAA data and he advised: “My typical avenue for downloading hourly data from NOAA has just been through their website form process Data Search | National Centers for Environmental Information (NCEI) (noaa.gov). However, they recently started limiting the file size, which became very inconvenient if I wanted more than a few years of data. I’ve used the R package called “rNOAA” rnoaa.pdf (r-project.org) recently with some success.”
——————————
Norv Clontz
Director, Data Science Innovation
Duke Energy
Charlotte NC
9803732380
——————————
——————————————-
Original Message:
Sent: 01-05-2021 15:49
From: Lauren Gouveia
Subject: NOAA Hourly Weather Dataset DownloadWe’ve tried a number of sources for collecting weather data including NOAA, DarkSky, Accuweather, deploying our own kits for solar irradiance and weather stations, and through contract with UL (formerly AWSTruepower). We are settling on using UL to provide both (renewable) forecasts and historical weather parameters – this improves consistency with the forecasts and simplifies the data translators (reduced the need for multiple data translators to the different services).
If it helps, the team found the NOAA data to be inconsistent with the reliability and updates. Darksky was easy to use but didn’t seem very accurate. Both NOAA and Darksky were free (with limits). UL has an associated cost but under our contract it includes modeling for Hawaii’s terrain which improved the granularity of the forecast (since our islands are so small). For awhile we had some success with the strategically located solar irradiance kits and weather stations but it requires dedicated resources for maintenance and to keep them calibrated.
Hope this helps!
——————————
Lauren Gouveia
Principal, IT Technical Architect
Hawaiian Electric Companies
Honolulu HI
808-543-7619
——————————Original Message:
Sent: 01-05-2021 11:52
From: Vera Lei
Subject: NOAA Hourly Weather Dataset DownloadYes, we download historic daily data on multiple zones from WSU Agriculture site already.
Thanks,
Vera
BI/ETL Developer
Benton PUD WA
+1509-582-1280Original Message:
Sent: 1/5/2021 11:44:00 AM
From: Ben Ettlinger
Subject: RE: NOAA Hourly Weather Dataset DownloadBefore you buy your own weather station, check out WeatherUnderground. They used to have thousands of local weather stations reporting in, like from schools universities etc. They were bought by the Weather Channel so I’m not sure if they still have that.
I also think Washington State has an atmospheric science dept. NYPA is big into partnering with State Universities as we are a State owned utility. Not sure if tat’s something of interest to your utility.——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-05-2021 10:40
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadBen, thank you for the response. When I say hourly I am referring to historical hourly weather data, so not an hourly forecast.
And yes, we are exploring other services as well for this data so thanks for the recommendation. Another option we may end up exploring is purchasing our own weather station(s) to capture hourly data from.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
+15095548928Original Message:
Sent: 01-05-2021 08:24
From: Ben Ettlinger
Subject: NOAA Hourly Weather Dataset DownloadWhen you say hourly data, do you mean to receive the data hourly, or twice a day with data for every hour going forward. The last I checked the former is not available. NOAA sends out data twice each day after their weather balloon (depending where you are) returns. For NYC its in Upton NY.
There are plenty of services that also provide the data like Accuweather, The Weather channel or https://openweathermap.org/api/hourly-forecast. The difference is that the service apply their own algorithms to the NOAA data and sometimes come up with different predictions.
——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-04-2021 16:53
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadHappy New Year UAI Network!
I am wondering if anyone uses hourly weather data from NOAA, and if so, how have you been able to retrieve this information?
Our team has currently found how to bring in free daily summary data through an NOAA API, but we are seeking hourly data if available (whether it is free or payment needed). I have yet to hear back from NOAA regarding my request, so asking here in case anyone has had prior experience retrieving hourly data.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
—————————— -
Hello all,
At CPS Energy (San Antonio, TX) we download NOAA forecast data through their NOMADS prod server, using API calls onto their HRRR and HRRR Subhourly Regional Models. We have a limited vendor subscription for specific geo locations actuals and forecasts that we enhance our models with.
NOMADS-NOAA Operational Model Archive and Distribution System
Kind regards,
Rolando E. Vega, Ph.D., P.E.
Senior Manager, Advanced Analytics and Intelligence (A2I)
Enterprise Advanced Analytics
CPS Energy | 145 Navarro San Antonio, Texas 78205 | MD: 110903
Office: 210.353.2538 | Mobile: 210.243.5807
cpsenergy.com
This email (including attachments) is confidential, may be legally privileged, and is intended for the addressee only. Persons other than the addressees are not authorized to access, rely upon, copy, forward, distribute, disclose, or otherwise use this email. Unauthorized access or use of electronic mail is strictly prohibited and may be a criminal offense. If you are not an addressee, please delete all copies of the message and the attachments and immediately contact the sender.
——————————————-
Original Message:
Sent: 1/5/2021 4:57:00 PM
From: Norv Clontz
Subject: RE: NOAA Hourly Weather Dataset DownloadI asked one of our Meteorologists how he pulls hourly NOAA data and he advised: “My typical avenue for downloading hourly data from NOAA has just been through their website form process Data Search | National Centers for Environmental Information (NCEI) (noaa.gov). However, they recently started limiting the file size, which became very inconvenient if I wanted more than a few years of data. I’ve used the R package called “rNOAA” rnoaa.pdf (r-project.org) recently with some success.”
——————————
Norv Clontz
Director, Data Science Innovation
Duke Energy
Charlotte NC
9803732380
——————————
——————————————-
Original Message:
Sent: 01-05-2021 15:49
From: Lauren Gouveia
Subject: NOAA Hourly Weather Dataset DownloadWe’ve tried a number of sources for collecting weather data including NOAA, DarkSky, Accuweather, deploying our own kits for solar irradiance and weather stations, and through contract with UL (formerly AWSTruepower). We are settling on using UL to provide both (renewable) forecasts and historical weather parameters – this improves consistency with the forecasts and simplifies the data translators (reduced the need for multiple data translators to the different services).
If it helps, the team found the NOAA data to be inconsistent with the reliability and updates. Darksky was easy to use but didn’t seem very accurate. Both NOAA and Darksky were free (with limits). UL has an associated cost but under our contract it includes modeling for Hawaii’s terrain which improved the granularity of the forecast (since our islands are so small). For awhile we had some success with the strategically located solar irradiance kits and weather stations but it requires dedicated resources for maintenance and to keep them calibrated.
Hope this helps!
——————————
Lauren Gouveia
Principal, IT Technical Architect
Hawaiian Electric Companies
Honolulu HI
808-543-7619
——————————Original Message:
Sent: 01-05-2021 11:52
From: Vera Lei
Subject: NOAA Hourly Weather Dataset DownloadYes, we download historic daily data on multiple zones from WSU Agriculture site already.
Thanks,
Vera
BI/ETL Developer
Benton PUD WA
+1509-582-1280Original Message:
Sent: 1/5/2021 11:44:00 AM
From: Ben Ettlinger
Subject: RE: NOAA Hourly Weather Dataset DownloadBefore you buy your own weather station, check out WeatherUnderground. They used to have thousands of local weather stations reporting in, like from schools universities etc. They were bought by the Weather Channel so I’m not sure if they still have that.
I also think Washington State has an atmospheric science dept. NYPA is big into partnering with State Universities as we are a State owned utility. Not sure if tat’s something of interest to your utility.——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-05-2021 10:40
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadBen, thank you for the response. When I say hourly I am referring to historical hourly weather data, so not an hourly forecast.
And yes, we are exploring other services as well for this data so thanks for the recommendation. Another option we may end up exploring is purchasing our own weather station(s) to capture hourly data from.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
+15095548928Original Message:
Sent: 01-05-2021 08:24
From: Ben Ettlinger
Subject: NOAA Hourly Weather Dataset DownloadWhen you say hourly data, do you mean to receive the data hourly, or twice a day with data for every hour going forward. The last I checked the former is not available. NOAA sends out data twice each day after their weather balloon (depending where you are) returns. For NYC its in Upton NY.
There are plenty of services that also provide the data like Accuweather, The Weather channel or https://openweathermap.org/api/hourly-forecast. The difference is that the service apply their own algorithms to the NOAA data and sometimes come up with different predictions.
——————————
Ben Ettlinger
Emerging Technologies
Enterprise Architecture & Engineering Group
New York Power Authority
White Plains, NY
914 681 6496Original Message:
Sent: 01-04-2021 16:53
From: Brett Jacobs
Subject: NOAA Hourly Weather Dataset DownloadHappy New Year UAI Network!
I am wondering if anyone uses hourly weather data from NOAA, and if so, how have you been able to retrieve this information?
Our team has currently found how to bring in free daily summary data through an NOAA API, but we are seeking hourly data if available (whether it is free or payment needed). I have yet to hear back from NOAA regarding my request, so asking here in case anyone has had prior experience retrieving hourly data.
——————————
Brett Jacobs
Business Intelligence Analyst
Benton PUD
WA
—————————— -
Hello UAI Members,
Happy New Year and I hope you all had amazing holidays! The UAI team is in full planning mode for our upcoming UA Summit event (May 3-5 2021) and are looking to our members to join our growing program board as speakers. Speakers should face the same day-to-day challenges as our attendees and members and be able to provide their experiences, advice, and tried and tested strategies. We are looking for presentations to focus on the following topics:
– Analytics Strategy
– Foundational Analytics
– Advanced Analytics
– Emerging TechnologyAs a reminder, UA Summit will be a hybrid event this year meaning both in person and virtual sessions. With your safety being our number 1 priority, speakers have the option of deciding how they want to deliver their session by selecting both in-person and virtual or virtual only on the proposal form. The deadline for submission is Friday, January 8th so we encourage you to submit your proposals ASAP and forward to any interested colleagues.
Thanks and we look forward to hearing from and including you in this great event!
——————————
Kevin Praet
Membership Coordinator
Utility Analytics Institute (UAI)
Boulder CO
315-440-3033
——————————
——————————————-
Original Message:
Sent: 12-16-2020 09:53
From: Leslie Cook
Subject: You’re invited to speak at UA Summit 2021!Deadline to submit your session proposal for UA Summit 2021 is January 8, 2021!
Hello UAI Members,We are looking for dynamic presenters with new ideas to speak at UA Summit 2021, hosted by Entergy. This will be our first ever Hybrid Event, taking place both in-person in New Orleans and virtually.
Speakers should face the same day-to-day challenges as our attendees and members and be able to provide their experiences, advice, and tried and tested strategies. We are looking for presentations to focus on the following topics:
– Analytics Strategy
– Foundational Analytics
– Advanced Analytics
– Emerging TechnologyYour comfort and safety is of the utmost importance to us – speakers have the option of deciding how they want to deliver their session by selecting both in-person and virtual or virtual only on the proposal form.
Why should you speak at UA Summit? To establish yourself as an analytics champion and accelerate the adoption of analytics among other utilities. Speakers receive a complimentary registration to both the in-person and virtual conference and are invited to participate in all event activities.
This is your opportunity to gain visibility for your analytics thought leadership in success. Submit your session proposal before January 8, 2021 to be considered for UA Summit 2021.
We look forward to hearing from you!
Sincerely,
Leslie
——————————
Leslie Cook
Membership & Digital Engagement Manager
Utility Analytics Institute (UAI)
719-203-8650, lcook@endeavorb2b.com
—————————— -
Aloha!
We’re currently in the middle of an evaluation of new tools to handle streaming data from multiple IoT sources. As background, we also utilize PI for generation and transmission (we’re a vertically-integrated IOU) and our smart meter vendor for basic smart meter data collection. That said, we’ve been dipping our toe into the AWS waters for outages where some simple AWS tools (Lambda functions, S3 for storage, Neptune as a graph database of network nodes based on a CIM model) are used to handle PONs and PRNs. We looked at Redshift and Snowflake for data lake capabilities and landed on Redshift since the price point was lower and the capabilities met our needs. (We were in the middle of our Snowflake/Redshift evaluation a year ago when AWS released a slew of new features for Redshift and reduced its pricing. Redshift’s basic functionality then met our needs and we went in that direction.)
Going forward, we’re actively evaluating how we handle timeseries data. We’re evaluating an open source approach (writing a connector to Kafka to then feed a Timescale DB on-prem with an option to host in the cloud) as well as an AWS approach (using IoT Greengrass on-prem and in the cloud, connected to Amazon Kinesis and Amazon Timestream). From there, we would provide data mining capabilities either through direct connection to the streaming database (Timescale or Timestream) via our regular reporting tools (Tableau, QuickSight, and now possibly Grafana).
You were curious if anyone had put together a business case analysis of the different approaches, and while ours is incomplete, I thought I’d at least share where we’re at. We are blessed with many great options in front of us. We are also cursed with many great options in front of us. :-) If you’re interested, I’ll keep you updated on our progress and decision process.
Have a great day!
——————————
Jason Pegg
Domain Architect
Avista Corp.
Spokane WA
509.495.4731
——————————
——————————————-
Original Message:
Sent: 12-15-2020 15:48
From: Rosemary Peh
Subject: Time Series DataAloha e UAI Network!
Another question for you as Hawaiian Electric is grappling with how time series data should be consolidated in order to facilitate enterprise analysis. We currently use OSI PI to archive and consolidate our time series data from disparate EMS and substation systems. We noticed that when meter data management systems were implemented, OSI PI was not necessarily the data consolidation tool of choice and that many chose to consolidate the meter time series data into other data analytics tools like open source no sql databases. However, because we are vested in OSI PI technology and capabilities (and we are not looking to move away from its current use on the generation and substation side), we were thinking that it would make sense to convert to enterprise licensing and to consolidate all time series data to OSI PI – which includes EMS, ADMS, MDMS, weather, power quality and other time series data. The price tag to do so is not “cheap” though. And “cheap” is relative – we are not a large utility and our roll out of meters is not big bang but slow over time. PI pricing is by “tag” which is essentially by data source column (e.g., each meter has 8 registers that collect separate pieces of info so it will need 8 tags per meter). The amount of data is limited to the hardware capacity – of which, if one builds flexibility in its scalability then data volume is then scalable to that design.
However, given today’s advancement in database capabilities (e.g., high scalability, lower cost per use models) and data access technologies, is moving such data that could be housed on services that are based on open source technologies prudent? Hence, my question…
Has anyone done any business case analysis comparing the use of traditional data systems (like OSI PI) to the current offerings from data analytics infrastructure and application service providers (like SnowFlake and Starburst? And if so, does it find that the utilization of these newer technologies a better long term solution – considering the Total Cost of Ownership?
Additionally, what is your utility using to consolidate time series data?
Appreciate your input. Mahalo nui (Thank you) in advance!
——————————
Rosemary Peh
Operations Excellence Director
Hawaiian Electric Company
rosemary.peh@hawaiianelectric.com
1-808-543-4790
——————————
Log in to reply.