Recognizing and mitigating these risks is necessary to create a social sector that is more effective, equitable, and efficient.
Data standardization has many, significant, and indisputable advantages for the social sector as well as almost any other industry. Having access to the same kind of data throughout time makes it possible to monitor development and raise responsibility. For instance, to evaluate shifts in giving patterns, my group, Candid, has monitored grantmaking by the biggest foundations for the past 20 years. We were able to show through the data that historically Black schools and institutions are underfunded by philanthropy. Additionally, data standardization opens up possibilities for benchmarking, which enables people and businesses to evaluate how they compare to rivals and colleagues. Large volumes of standardized data can also be used to forecast industry trends. Lastly, and maybe most critically for the social sector, data standardization always results in a reduction of the heavy reporting loads that organizations must bear.
Despite all of its advantages, data is all too frequently presented as a panacea that will enable us to conclusively assess the efficacy of social change initiatives and procedures. The truth is much more subtle and intricate. If unaddressed, the inadvertent outcomes of standardizing data provide noteworthy hazards to attaining a social sector that is more proficient, economical, and fair.
Three Typical Issues in Standardizing Data
Fundamentally, information and facts that are standardized tend to normalize and reflect the lived experiences of those in positions of authority. After all, the people in positions of authority typically establish the norms. For instance, the use of standardized testing in schools was expected to increase the ability to benchmark, monitor progress, and recognize good performance by establishing a fair assessment of students’ knowledge across schools on an annual basis. Nevertheless, numerous studies conducted over several decades have demonstrated that standardized exams frequently harm individuals who differ from the test creators (e.g., in vernacular, in test-taking preferences, or access to resources). Furthermore, research indicates that women’s and persons of color’s success on US standardized tests may not be as accurately predicted.
Valuing particular data points or results over process and understanding is another possible weakness in standardized data, as it might encourage systemic gaming. According to studies, schools and teachers who were apprehensive about standardized testing began teaching to the test, sometimes at the expense of real learning. Another illustration is the increase in p-hacking: Researchers started adjusting analyses, or “hacking,” to increase their chances of finding statistically significant results as more academic journals demanded a standardized threshold of statistical significance (referred to as a “p-value”) to be published. This practice reduces the validity of the findings, though.
Additionally, because people tend to overstate the universality of their data, measures, and methodologies, data standardization may fail to achieve its intended goals. It is simple to believe that people are interchangeable and that persons with various identities, situations, or cultural backgrounds can benefit from the same research findings. Even in cases where they appear harmless, these extrapolations may result in inaccurate and unintentional outcomes. Research has indicated that cultural variations exist in the responses that individuals give to Likert scales, which are survey items that allow respondents to rate their answers on a range of one to seven. Asian respondents are more likely to steer clear of extreme responses, whilst Black and Brown participants are more inclined to select extreme response categories when compared to White respondents. The quick development of new technologies raises the stakes since it creates conditions in which incorrect data extrapolations can result in standards that are harmful and exclusive. For instance, motion detectors calibrated for pale skin may miss dark skin, which might be fatal if autonomous vehicles are unable to identify pedestrians with dark complexion. Women are more likely to have cybersickness when wearing virtual reality headsets made to fit men’s skulls, which is a significant drawback as virtual reality becomes more common in daily life and the job.
These difficulties do not exempt the social sector. The social sector in the United States is practically interlaced with the normalization of the perspectives of those in power. The majority of the resources in the sector are controlled by grantmakers, who also frequently decide what information is gathered for grant submissions and how to assess a program’s effectiveness. Nonprofits may therefore be tempted to prioritize attaining a certain data point, metric, or outcome that grantmakers find valuable—even at the expense of what would most effectively advance their missions and communities. In a similar vein, universalist conclusions may lead to false presumptions that information about a particular program or nonprofit may be extracted from its context and used in another.
Developing More Precise, Fair, and Comprehensive Data Standards
More inclusivity, transparency, and self-reflection are necessary on the road to a data-driven social sector that is more equal, effective, and efficient. It will be necessary for individuals in positions of authority—those who establish the norms—to look beyond the boundaries of their own experience and expertise and to delve deeper than surface-level universal measures. It won’t be quick, easy, or simple. We may, however, mitigate the detrimental impacts of data standardization by taking certain doable actions.
Instead of standardizing the “whole house,” focus on the “building blocks.” Many of the advantages of standardization can be achieved while lowering the dangers associated with overvaluing or applying a particular set of data points uniformly. This is achieved by standardizing little bits of information or data that can then be used in a variety of ways. For instance, we can find smaller components of grant applications that can be standardized rather than imposing a fully standardized grant application procedure. Candid has been testing this strategy with our Candid ad for Demographics. In essence, this project creates a social sector demographic data registry by inviting nonprofits to submit a brief synopsis of important demographic data about their organizations on Candid’s nonprofit profile. This enables grantmakers to import that data into their various grant application processes. This method gives the social sector a means to benchmark and evaluate demographic data while also saving grantseekers the trouble of repeatedly entering demographic data. Simultaneously, it facilitates modification when necessary—for instance, gathering more demographic data if it pertains to the particular award under consideration.
When defining and operationalizing concepts, be precise. While unclear definitions are generally annoying, they become much more crucial to handle when producing standardized facts and information that are utilized by different people in different settings. After all, if everyone is interpreting questions or responses differently, what use is it to collect standardized data? Providing definitions for any jargon or important terminology during the process of gathering, interpreting, and disseminating data is a recommended practice. Candid accomplishes this through our guidelines for collecting demographic data, which offer best practices and common criteria for enterprises to utilize when gathering demographic data internally. The public definitions found in the NTEE IRS Activity Codes, which are a component of the taxonomy system used by the IRS and NCCS to categorize nonprofit organizations, are another excellent example.
To clarify, being explicit about operationalizations is a bit more subtle but no less significant step. Operationalization is the process by which investigators or assessors quantify and turn theoretical ideas into numerical values. For instance, the term “happiness” refers to any positive feeling, such as joy or contentment. This notion, however, can be operationalized in a variety of ways, including scoring answers to journal entries, surveying people about their happiness, and monitoring actions like laughing and smiling. Similarly, NGOs that serve the BIPOC community can be categorized as BIPOC-serving nonprofits in the social sector. This can be operationalized in a variety of ways, though, such as whether it is included in the missions or programs of charitable organizations, if a particular program satisfies a particular proportion of BIPOC representation, or if the local community fulfills a particular percentage of BIPOC representation. In a similar vein, organizations that identify as BIPOC can be described as “BIPOC-led.” However, depending on how “leader” is defined, there are also several ways to operationalize this term. Which one of the board of governors, the CEO, the founder, or the entire executive team is it? What level of leadership is necessary? Building knowledge requires understanding operationalizations, which is true for any discipline. It is impossible to determine without it whether variations in measurements and outcomes are caused by measurement discrepancies or advancements in the field.
Encourage data openness in terms of restrictions and methods. All too frequently, information, data, and research are assumed to be true at face value. Any organization that uses research or data in any way should always make explicit the constraints, the methodology, and any known assumptions that may have affected the results. Specifically, information that serves as a benchmark or standard for the field should be clear and open about the methods used to generate the data, the validity checks that were carried out, and any recommended usage guidelines. End users will be better able to comprehend which viewpoints and norms were employed, what sample was used to generate the data, and what unanswered problems still need to be answered by doing this. The Transparency Initiative of the American Association for Public Opinion Research and the National Institute of Health’s instruction on writing about constraints are just two of the many resources available to data collectors that strive for transparency.
Accept methods of participatory research. Speaking with the groups that these standards will affect directly is crucial when trying to produce standardized data. When creating research and data projects, participatory approaches aim to incorporate and prioritize a wider range of voices and perspectives. This method is useful because it can verify assumptions that the original analysts and researchers may have made based on their own experience, knowledge, or skill. It can also stop the normalization and centralization of the experience of people in positions of power by challenging presumptions and oversimplification. To decide which operationalizations of “BIPOC-led” to include in the research, for instance, a group of nonprofit leaders in New York was gathered for a recent report on BIPOC-led NGOs in the city.
Early and often, challenge presumptions. As cliche as it may sound, questioning presumptions and examining potential biases is crucial to promoting strong data standards and center equity. This might be as easy as compiling a list of inquiries that decision-makers ought to make whenever new data standards are developed or examined.
- Have the most affected parties—end users and program recipients, for example—been contacted?
- Has this been vetted by individuals with diverse experiences and identification groups?
- Why are we requesting this information or posing this specific query?
- How may we approach this issue differently?
- What knowledge do we assume to be “common sense” or take for granted?
No matter how much data we have or how much work we put into standardization, we are still vulnerable to false information, hidden biases, and inaccurate conclusions about the people and communities we are committed to serving because of the human element inherent in the social sector. This is true despite an influx of data analytics and tools. However, the social sector will increase equity and strengthen its capacity to bring about long-lasting, significant change by implementing strategies that include more voices, define terminology clearly, are open about methodology, and lessen burdens and restrictions.