There’s no shortage of information about data these days, with good reason. Data is the fuel that enables many modern technologies to operate, including artificial intelligence (AI), business intelligence (BI), the Internet of Things (IoT), and many others. These technologies are foundational to company success in the 21st century, so it’s hard to overstate the importance of data in the business world. But not just any data will do. For the highest level of effectiveness, data must be of the highest quality based on standards set by each unique company.
Additionally, raw data must go through a process known as ETL, which stands for extract, transform, and load, that must also be carried out according to specific standards. The first step in ETL is to extract the data from its original source, such as online retail engines, vehicles, social media, or company databases. Next, it must be transformed to ensure compatibility with other information. Finally, it must be loaded into the target database. The following video explains more about this process.
To achieve the highest quality of data and data transformation, the process must be overseen by knowledgeable data experts. These professionals are known as data quality managers. In the following sections, we explore the role of data in modern organizations, what constitutes data quality, and how a data quality manager can help.
The Importance of Data
“Data is the new oil” is a refrain currently popular in the business community. Some use it to mean that data is comparable to oil because it’s so valuable. While that may be true, the real meaning of the phrase is that data, like oil, has no real value until it is processed into something else. In the case of oil, that’s fuel. In the case of data, it’s usable information that is integrated into a single source and “cleaned” to be consistent and error-free.
Such clean data can help companies derive myriad insights that can give them a fuller understanding of the market, support smart decision-making, and assist them in gaining an edge over the competition. For example, social media mentions can help a business know which products are getting the best response and why. Information about performance in certain locations can help leaders determine where the next brick-and-mortar store should go. And prospect lists can be used to build promotional campaigns and generate sales.
Additionally, data can be used to make decisions about day-to-day operational activities such as hiring, purchasing, and customer management. The more data organizations collect, the more important it becomes. Therefore, the more critical it is that data is of high quality that benefits organizations rather than causes them to make mistakes and misjudgments or commit costly regulatory errors.
Companies across industries rely on specific types of data.
Agriculture
As the world’s population continues to grow, food production is more important than ever. With unpredictable climate conditions, growers must have ways to develop alternative methods. Data can help by showing farmers how to optimize their resources — including machinery, equipment, and fertilizers — for the highest yields. Advanced practices such as vertical farming are linked to the collection of data, which can be analyzed to determine optimal lighting, water, and nutrition.
Education
As education embraces more electronic methods, data can be collected and used to determine which strategies work best, how students and teachers are performing, and how systems can be enhanced. Administrators can create insightful reports that show which students learn better in certain environments or with specific teachers. Course methodologies can be tracked, analyzed, and improved.
Government
The use of data can help public jurisdictions benefit citizens with useful programs that contribute to safety, health, and convenience. For example, to reduce crime, administrators can analyze data to determine which areas need a broader police or community patrol presence. Municipalities can use data collected through smart city technology to improve and integrate it even more.
Healthcare
Data can be used throughout the healthcare system to track patient sentiment and make improvements to their care. It can also be used directly in patient care, such as with wearable trackers that monitor their condition, giving professionals granular information that can be analyzed to adjust recommendations. A similar method can be used to track the effectiveness of medication.
Insurance
By its very nature, the insurance industry benefits from the ability to predict things like which homes are likeliest to be subject to natural disasters, which drivers are likeliest to get into accidents, which customers are likeliest to develop illnesses, and which people are likely to die at what age. The data used for these analyses comes from a variety of sources, including telematics, agent interviews, and social media.
Manufacturing
The manufacturing industry contains multiple opportunities for data gathering and analysis. For example, IoT technology could deploy potentially hundreds of sensors attached to machines to monitor their performance and condition. The data they generate can be used to gain insights into which are in need of repair or replacement, enabling operators to plan and budget according to company policies and priorities.
Marketing
Marketing professionals can get a deep understanding of customers and prospects through the use of data from various sources. The more targeted a marketing campaign is, the more likely it is to succeed. Marketers can push information on the online platforms where their target customers are likely to spend time and use messages and images that align with their values and preferences.
Retail
Retailers use data analytics to understand and predict customer requirements and create new products or develop new service models based on those needs. Customers can benefit from personalized experiences and marketing that are more targeted to their specific tastes, based on information collected from a wide variety of sources. These sources include loyalty programs, credit card transactions, online searches, social media posts, purchase histories, and more.
Securities
Data technologies are used to develop insights both for and about customers. For example, they can be used to get updated information about securities trading, measure buyer sentiment, or predict outcomes. It is also used to perform risk and know-your-customer (KYC) analyses, prevent money laundering, and reduce fraud. Additionally, these technologies can be used to identify criminals within the system.
Transportation
The transportation industry can use data to make vehicle transportation smarter. For example, information about specific intersections can help develop traffic signals that change in response to real-life situations rather than timed intervals. It can also be used to plan routes and connect to smart city technology to increase convenience and safety for citizens and visitors. Companies that rely on logistics can use data to better plan their routes and determine which vehicles to deploy.
Utilities
Power utilities use data across all operations, including power management between what the company provides and what customers generate using distributed energy resources (DERs) such as solar panels. They also use data for service operations, such as restoring power following a weather event. Data analysis can help them know which areas need to be addressed first. Utility customer care teams can use it to determine which products to suggest to customers based on their past use of energy efficiency technology.
What Is Data Quality?
When organizations put effort and expense into acquiring data, they should take the next step to ensure it is of good quality. For example, a marketing list with a high number of duplicate names and bad addresses can lead to wasted efforts on the part of marketing professionals looking to send out a mailing.
When the purpose of such a mailing is to get the highest return rate possible, bad data can decrease that percentage. Even more serious results are possible. According to analytics software provider SAS, “Poor data quality has a significant business cost – in time, effort and accuracy.”
As with other business processes, the best way to achieve high standards is to get direction from the experts. As a comparison, consider the role of IT within a company. Individual workers may know a bit about computer maintenance. But if they try to maintain their own workstations, it may conflict with what others are doing and may even cause damage to the network. That’s why IT professionals manage and direct updates, maintenance, and repairs.
How Can Companies Improve Their Data Quality?
Similarly, companies can improve their data quality through a process known as data quality management. That is, having an individual or team to define what data quality means and develop processes for ensuring it is achieved. SAS offers the following suggestions for approaching data quality:
“Elevate the visibility and importance of data quality.” Create an internal communication campaign to ensure all employees understand the consequences of using inferior data and how they can contribute to avoiding that scenario.
“Formalize decision making through a data governance program.” To ensure consistency in data across the organization, create a group that determines how data will be handled.
“Document the data quality issues, business rules, standards and policies for data correction.” To solidify the work of the governance program, document all applicable rules and policies, and share the documentation throughout the company. Update the guidelines regularly.
“Clarify accountability for data quality.” Create roles for specific team members to be responsible for data quality. Typically, it would be the data manager, which is described below. This person or persons are responsible for setting data quality standards and ensuring they are met.
“Applaud your success.” Measure key performance indicators (KPIs) at the beginning of your process and regularly thereafter to show your progress in data quality improvement.
Gartner offers a few additional pieces of advice:
“Define what is a ‘good enough’ standard of data.” The definition of “good enough” will vary between companies and even between different departments within a company.
“Use data profiling early and often.” Think of data profiling as data maintenance, where you examine data from a particular source and note information about it as well as what can be done to improve it. Data profiling should be performed regularly.
“Design and implement DQ [data quality] dashboards for monitoring critical assets, such as master data.” Dashboards can be used to show a snapshot of data quality at any given time or to demonstrate quality over time. This information is useful for improving existing data quality programs.
“Establish a special interest group for DQ across BUs [business units] and IT, led by the chief data officer team or equivalent body.” The data quality manager should not be working in isolation. Rather, they should be supported by others in the organization with an interest in their activities, including the chief data officer (CDO).
What Does a Data Quality Manager Do?
A data quality manager evaluates, maintains, and manages data quality for an organization. They may work in more specific capacities, such as CRM managers or marketing professionals, but the high-level job typically includes the following duties.
- Creating standards by which data quality can be measured and setting company policy standards
- Developing a company-wide strategy for ensuring high data quality, including operational details, each department’s specific needs, and specific processes for improving data quality
- Reviewing data sets, reports, and other tools to evaluate data quality, ensuring it is error-free and compliant with regulations
- Reporting to upper management on the state of data quality within the company, including information about concerns and possible ways to address them
- Working with individual departments to ensure data is consistent across the organization and meets regulatory standards
- Developing and updating initiatives to oversee data collection, integration, and processing and to improve data management systems
- Recommending procedures for data storage
- Maintaining and updating knowledge about the latest trends in data quality, including new methods for implementing data quality management
- Creating a data dictionary, which is a system that keeps track of data, including data relationships and use cases
- Training other employees on how to comply with standards related to data quality
Your company may need a data quality manager if one or more of the following statements are true.
- You rely on data to make important recommendations to your customers, clients, or patients.
- You rely on data to make important and far-reaching company decisions.
- You receive data from a wide variety of sources and need to ensure it is useful.
- Your data is growing in quantity or is separated into silos.
- Fees and other penalties from regulatory bodies for mismanaged data would pose a significant challenge to your organization.
How to Hire a Data Quality Manager
Data quality managers are in high demand, which means two things. One, it might take some time to find the right one to hire. Two, it’s worth the effort.
Planning
Begin by identifying what you hope to achieve in hiring a data quality manager. It helps to have specific goals or pain points to address. For example, using the list above, you might determine the biggest obstacle to achieving data quality is that team members don’t know how to contribute to it. You could then seek out a data quality manager with plenty of training experience in addition to their other expertise. You should expect to pay a data quality manager an average of $110,000 per year.
Searching
Look for someone who has at least a bachelor’s degree in computer science or related field. Consider whether you require candidates with experience in data quality and certifications in addition to their regular college education. Make sure candidates have the soft skills to accompany their technical know-how. They should be good written and verbal communicators, have experience in developing programs and seeing them through, and understand that their role is to help your company succeed.
Interviewing
The following are some questions from Indeed to ask prospective data quality managers. Keep in mind these questions are specific to the position. You’ll also want to ask more general questions about their background, expectations, and similar matters.
- Do you prefer working with simpler or more complex statistical models?
- What are the criteria for a good data model?
- What is an N-gram and how does it relate to data quality analysis?
- Between simple imputation and multiple imputation, which imputation method do you prefer?
- Can you describe a few key differences between hot-deck imputation and cold-deck imputation?
- What are hash table collisions and how can you avoid them?
- When is a correlogram analysis useful?
- What is a time series analysis?
- Name a few key properties for clustering algorithms.
- Provide an example of collaborative filtering.
Data Quality Ensures Business Quality
As business processes become more dependent on technology, companies require people who understand that technology and can translate their knowledge into supporting business achievement. Data may be related to IT, but it is a separate function. New positions are emerging, even at the highest level, as with the CDO role. The CDO and other data professionals such as a data quality manager can help companies make the most efficient use of one of their most valuable resources — data — resulting in overall business success.
