Definition
Data cleansing is the work of tidying up a dataset already in use. It differs from its three neighbours: data qualityThe degree of accuracy, completeness and reliability in a dataset - critical for correct billing and balance settlement in the power sector. is the state the dataset is in, validationThe first step in VEE - automated checks of metering data against technical and logical limits to detect obviously erroneous values. is the check that reveals a value breaking a rule, and VEEValidation, Estimation, Editing - systematic process to ensure quality of raw metering data from electricity meters. is the continuous, regulated process applied to metering values. Cleansing is the correction itself, and it usually targets master dataThe authoritative source data in a System of Record (Master) - data owned, maintained and edited in one place, from which all other systems retrieve copies. rather than time seriesMeasurements of the same quantity repeated over time. The order and the timestamp are part of the data, not decoration.: duplicated metering points, addresses that do not match the cadastre, asset components with no link to a facility, naming that has drifted apart between two systems.
In the energy sector the need typically arises in three situations. After a merger, where two companies' registries must become one. Ahead of a migrationProcess of moving data, systems or services from one platform or environment to another., where the old system's tolerance for sloppiness does not exist in the new one. And before a new ElhubNational hub for metering data and market processes in the Norwegian electricity market. role goes live, where data that was good enough internally suddenly has to survive being sent outside.
The word deserves an objection. Cleaning is a one-off act: you wash something dirty, and then it is clean. But data gets dirty again, because the fault usually does not sit in the dataset. It sits in the process that fills it. A registry cleaned last year and dirty again this year was not cleaned badly. It is a registry without maintenance. That is why HEVDNot an acronym, but the Norwegian word "hevd" - a legal right you must keep in continuous use for it to hold. Tvimenning's managed service for data quality, across the entire system landscape. sells data quality and cleansing as a continuous service rather than a project.