A day that must be finished before breakfast
Yesterday's metering values must be with ElhubNational hub for metering data and market processes in the Norwegian electricity market. before seven in the morning. Everything that happens to them happens in the window between midnight and 06:45: collection from the meters, validationThe first step in VEE - automated checks of metering data against technical and logical limits to detect obviously erroneous values., estimationThe second step in VEE - calculation of missing or rejected metering values based on historical patterns, neighbouring meters or statistical models. of whatever is missing, and transmission.
That is not much time, and it is the same amount of time every single night. The window does not move according to how much has to pass through it.
The chain, link by link
From meter to hub the values pass through four links, and each one gathers up before sending on.
The smart meter records consumption continuously. It does not send every value the moment it arises.
The concentrator asks a group of meters for their values at a fixed interval, typically every 6, 12, 18 or 24 hours, and forwards the batch. Not every rollout uses them: meters with their own cellular connection talk directly to the collection system. Where they exist, they set the pace.
The head-end systemHead-End System - central system for collecting and managing data from smart meters in AMI networks. receives from the field and passes on for validationThe first step in VEE - automated checks of metering data against technical and logical limits to detect obviously erroneous values..
The VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. engine validates, estimates and corrects, typically with the metering values arriving as GS2A text-based Norwegian file format for exchanging metering values, in use in the power sector since long before Elhub. files. It runs in batches because it needs enough CPU and IO to get through, not because anyone finds batches tidy.
ElhubNational hub for metering data and market processes in the Norwegian electricity market. processes a queue in the order the messages arrive.
Batch is not an old habit
It is easy to read this chain as a legacy system waiting to be modernised. That reading is wrong.
The chain is batch for two reasons, both of which still hold. One is regulation: the deadline is set, and a full day must be processed before it. The other is capacity: a system that has to validate millions of values does so faster in batches than one at a time, because disks and processors work better that way.
The discussion about how data is collected at the other end, meaning whether a receiving system should ask or be told, is an entirely different discussion. It changes nothing about the fact that the collection in front of it is batches with a deadline.
What made the window tighter
On 22 January 2024 the settlementTurning metering data into money. Consumption and production per metering point become what each party owes or is owed. basis in ElhubNational hub for metering data and market processes in the Norwegian electricity market. moved to quarter-hourly resolution. Submission of 15-minute metering values had started on 18 September 2023, and grid companies changed the resolution on metering points gradually from October that year.
For the chain this means one thing: four times as many values through the same window.
Four hourly values became sixteen quarter-hourly values per meter per four hours, and a day went from 24 to 96 values per metering pointThe fundamental unit in Elhub - the unique point in the electricity network where energy is measured and settled, identified by an 18-digit GSRN number.. Nothing else was extended. Not the time, not the deadline.
Many manage it. But it is not unusual for VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. software that held the deadline with a good margin on hourly values to no longer chew through the volume in time. That is not a fault in the software, it is a dimensioning done for a quarter of the quantity.
The grid utility data sitting in the queue
The second thing that makes the window tight is more specific, and it does not affect everyone.
Many want to take in grid utility dataMetering data used to operate and plan the grid rather than to settle consumption. alongside the metering values due for settlementTurning metering data into money. Consumption and production per metering point become what each party owes or is owed.. In several older VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. systems the two pass through exactly the same logic. The grid utility dataMetering data used to operate and plan the grid rather than to settle consumption. does not need VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. treatment, but it sits in the same queue, and it takes the place of something that does.
The effect is that the grid utility dataMetering data used to operate and plan the grid rather than to settle consumption. blocks the fastest possible VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. processing simply by being in the way. It is a bottleneck no volume figure reveals, because it is not about how much data there is, but about the order it stands in.
What is worth knowing about your own chain
None of the above is solved by a single purchasing decision. But they all become far easier to handle once three questions have an answer.
Which link sets the pace? The concentrator interval is a floor: no mechanism further down the chain can deliver a value faster than it was collected. If you do not know the interval, you do not know how early the night can actually begin.
Where does the time go inside the window? A window that holds with twenty minutes to spare and one that holds with two look equally green in an operations report. The difference is only felt the night something goes wrong, and then it is decisive.
Does anything share a queue with anything else? Grid utility dataMetering data used to operate and plan the grid rather than to settle consumption. against settlementTurning metering data into money. Consumption and production per metering point become what each party owes or is owed. values is one example. The pattern is general: two data flows with different requirements sharing the same processing logic, where only one of them has a deadline.