[Gary Rector]

Hi Kate,

I think this is a great question because it highlights several competing aspects of using big data, demonstrating why there is no simple answer:

• Regulatory or policy requirements: our organization’s policy is to keep no more than three years of PII data
• Practicality of storage and retrieval: our data lake can handle more than seven-years-worth of AMI 15-minute interval data
• Relevance of historical data: the pandemic changed market behavior
• Statistical and analytical needs: usually, the more data the better 

Trying to thread a solution for all of these needs, I suggest retaining as much granular data as your policies allow. After that period, aggregation or anonymization can enable keeping data for a longer period, depending on your analytical needs.

We have AMI data older than seven years, but only a fraction of our customers had smart meters that long ago. I find that older data is useful for observing historical trends and teasing out possible forces of change, but that is more to satisfy curiosity than business need. Data older than seven years becomes less relevant for current business planning because of market changes, technology changes, and climate changes.

Did I successfully avoid answering your question?

——————————
Gary Rector
Data Scientist
Salt River Project
602-236-6961
——————————
——————————————-
Original Message:
Sent: 01-21-2021 10:42
From: kate lemay
Subject: AMI data retention?

Hey Utility Analytics Friends,

How much AMI data do your retain? We currently have a large archive and are trying to figure out how much is actually useful for analytics. If you have a limited history, do you summarize the farther back history? We are debating between summarizing data past 7 years or purging it. I’d love to learn how others have handled this and if there’s really any use case for AMI data farther than 7 years back.

——————————
kate lemay
Director of Enterprise Analytics
Black Hills Energy
6166343919
——————————